Disclaimer: The following text is taken with impudence from David Wulff's The Psychology of Religion: Contemporary and Classic Views. New York: John Wiley and Sons, Inc., 1991.
Born New York City Jan. 11, 1842 as the eldest in a family of 4 sons and 1 daughter, James' religiously orthodox and aggressively acquisitive (Irish immigrant) Grandfather had become a leading citizen of the state of New York and a millionaire several times over. James father Henry inherited neither the church going Calvinism nor the interest in finance much to his father's disappointment. Henry was a profoundly introspective and religious man that, after several career mis-starts, settled into a life of a professional academic.
After his father's death and the eventual breaking of his Last Will, Henry was at last able to incur sufficient income for a life of study and writing. This also provided him the opportunity to experiment with his children's education. Henry James frequently changed his children's schools or tutors as he swept from one city to another in both the US and Europe. This was obviously a stimulating situation for the children to be brought up in and resulted with their cosmopolitan character. On the other hand, this kind of situation could have resulted with instability and alienation in their lives. Thus we find that the situation incurred a heavy dependence of William on his father Henry in spite of their differences in outlook and religious conviction. This dependency played a role in seriously prolonging his childhood and effecting his writing.
William began his career interests as a promising artist. The choice to do so was particularly difficult as he was talented in both the creative arts as well as science. Some biographers claim that it was the inconsistent family life of always being on the move, his father's educational experiments, a strong need to develop his own identity, and so on that lead to his struggle for a career decision, while, on the other hand, his inability to be decisive and may have had something to do with his subsequent ill health.
James thus decided at 18 to become a painter but less than a year later he changed his mind and never again used his talent in any serious way. Instead, the next fall he enrolled in the Lawrence Scientific School at Harvard College to study chemistry and then comparative anatomy, a pre-medicine course of study.
The motives of this abrupt change in mind as to his career has been the subject of considerable speculation. Some think it may have been due to his ill health--eye trouble experienced as student artist. Other considerations may have played a role--his instructor informing him of the impossibility at that time of making a career in N. America as a painter.
Others, however, suggest that it was James's father that convinced him to enter into science. His father, it is well known, had strong preference for science. Also after James had begun his artistic studies his father had attacks of fainting spells and predicted that he would soon be dead. In a family that is said to have practiced "manipulative invalidism" (...) such a maneuver would finally succeed in causing James to turn to science. Yet James the painter is said to have lived on--filled with violent anger which expressed itself in recurrent physical symptoms of illness. In fact James, at one point, equated the rejection of a career under way with a murdering of the self.
In any event his time at Cambridge was the first that he had spent away from his parents for more than a few days at a time. After 3 terms at Harvard, his poor health forced him to return home for half a year, during which time he read widely in science, philosophy and literature. His return to the scientific school in 1863, came with a shift of program from chemistry to comparative anatomy. A still later shift to medicine was yet to come. This choice came only after months of indecision between considerations of practicality and interest. He decided to enter medicine as it allowed him to continue to pursue his interest in the human body and that it may have appealed to a psychological state he was in at the time (depressive?).
James studied medicine between 1863 and 1869. His studies were interrupted twice--once for a trip to the Amazon collecting samples (he was discouraged by this work and the harsh conditions, etc.) and another time when he went to Germany to study physiology.
His trip to Germany followed a winter of suffering due to eye trouble, digestive disorders, insomnia, painful back weakness, and profound depression. He at last decided that he might find relief in the mineral baths of Germany (a popular medical misconception of that time). Although physically incapable of doing laboratory work, he read extensively the German writings on the nervous system and psychology as well as philosophy and literature. The outcome of these months of reading and writing was a liberal education that he could not have received at Harvard college.
James did not see his time spent as such, and returned to Cambridge weary and discouraged. He then prepared for his medical school examination which he passed without difficulty in 1869. But he turned his back on what others thought was a promising career. The practice of medicine seemed to James to be a dead end.
During the time of invalidism in his parent's home, following is trip to Germany, James took extensive notes on a large number of books--physiology, neurology, psychology and philosophy, as well as German, French and English literature. Underneath the surface of quiet preoccupation, we sense a man who is concerned with a profound melancholy and helplessness that was convinced that he was incapable of any kind of affectionate relationship.
He later had a brush with insanity that he barely escaped (p 472) by turning to scripture texts. This experience was included in his Varieties and credited to an anonymous source. He recounts that one day at twilight, while in a state of "philosophical pessimism and general depression..." he was suddenly overcome by a fear for his existence. This feeling of terror, he writes, was accompanied by the mummy-like image of an idiotic and totally withdrawn epileptic youth whom James had seen in an asylum. He felt such a state could in an instant be his own.
It was as if something hitherto solid within my breast gave way entirely, and I became a mass of quivering fear. After this the universe was changed for me altogether. I awoke morning after morning with a horrible dread at the pit of my stomach, and with a sense of the insecurity of life that I never knew before, and that I have never felt since...[The experience] gradually faded, but for months I was unable to go out into the dark alone.
The texts he turned to that helped him escape insanity were ones like "the eternal god is my refuge' and 'I am the resurrection and the life.' etc. During this time he also found solace in the writings of the French Philosopher Charles Renouvire and the British psychologist Alexander Bain. From Renouvier he obtained a definition of free will that he thought he could provisionally accept: "the sustaining of a thought because I choose to when I might have other thoughts." (NB the basis of the pragmatic theory of truth). Thus like his father, who had undergone a similar experience, he found relief in a new intellectual understanding.
This spiritual crisis has prompted much theorization of its causes--ranging from physical to psychological factors and to cultural and philosophical ones. Much has been written on his depression and its effects on his resulting philosophy and psychology.
In 1872 he received an appointment at Harvard college which provided him the time to study his insights from the illness, that would preoccupy him for the rest of his years. This illness is said to have also "gave [him] a personal intimacy and intensity to the deepest problems that philosophy and religion can present to man's understanding."
He viewed his appointment in physiology as a helpful diversion from his introspective studies which he said bred in him a kind of philosophical hypochondria. At the same time he was having doubts of the scientific method, and came to the opinion that physiology did not provide the key to psychic states. To the contrary, his own experience had taught him that the mind could be approached directly through self-observation. Thus he began on the road to the study of psychology and philosophy--the two fields in which he was to make his most lasting contributions.
In 1878, shortly before his marriage, James was contracted to write The Principles of Psychology, a monumental, two volume work that is still described today as the most provocative and at the same time most intelligible books on psychology that has been published in any language. Ironically this 1400 page work--12 years in the making--marked the end of psychology's domination in his life. Transferred to the philosophy department in 1880, he increasingly gave his attention to philosophical questions, to the pragmatism and radical empiricism that were to be his contribution to American philosophy.
Behind all of his academic interests and titles, it seems that religion was his all prevailing interest. Interestingly enough, no matter how pervasively his religious interests were, they remained unusually muted in his works. To James the interest was in the "personal religious experience" and he considered it to have its "roots and centre in mystical states of consciousness"; yet his own constitution, he adds, "shuts me out from their enjoyment almost entirely." The contempt that Henry James had for any institution ensured that William would have no official church association. Even though James did, as a Harvard professor, regularly attend morning chapel services, he otherwise found himself incapable of any other ordinary form of piety. To James prayer seemed foolish, yet he wrote in the Varieties that prayer was "the very souls an essence of religion." The bible had no personal authority to him; and even though he was open to the possibility of personal immortality, he never keenly believed in it.
To James the religious impulse consists of, it seems, a sympathetic response to the cumulative testimony of other's experiences. His own mystical seed, undeterred by rational criticism, compelled him to recognize in these accounts lies truth. He became famous and was criticized for his openness to whatever phenomena that came his way however much it clashed with medical and scientific orthodoxy. In fact the type of experience that, in less critical minds, inspires such an affirmation of the other world was not entirely foreign to James. There is a description of another mystical experience in James's life. One July evening in 1898, James, while on a camping trip with friends in the Adirondack mountains, found himself in a state of "spiritual alertness." He spent the evening wandering through the moon-lit woods experiencing a ferment of "tumultuous mixture of impressions and memories: of magically illuminated nature, of his wholesome companions and beloved family..." It seemed to him "a regular Walpurgis Nacht"--a discordant meeting within himself of the "Gods of all the nature-mythologies" and "the moral Gods of the inner life." It was an experience that he was never able to capture in words and thus ever remained as a "boulder of impressions" and was remembered as one of the happiest nights of his life.
The immediate reason for writing Varieties was provided by am invitation to give the Grifford lectures at the University of Edinburgh. What James did in these lectures is characteristic of his own life and interests. James maintains that the understanding of another's perspective requires familiarity with that individual's personal life. This seems to be a special rule for James as the key to his influence is said to lie in his personality rather than his expounded principles, his distinctions or the few hypotheses he proposed.
After this religious experience, he suffered a period of invalidity before and during his trip to Europe for the Grifford lectures. During this time he continued to read religious biographies. His illness, however, cause a double postponement of these lectures and it was not until 1900 that he began writing them and even then the progress was agonizingly slow.
The task he set himself would not have been easy even under ideal circumstances. He wanted to  defend experienced against philosophy as being the backbone of the world's religious life and  to make the hearer and reader believe, that, although the specific manifestations of religion may have been absurd (creeds and theories), yet the life of it as a whole is mankind's most important function. To James it was his religious act even if he recognized the inherent impossibility of the task.
In spite of his writing style, his self-deprecatory comments, and wishy-washiness of generalizations the Varieties was an unqualified success. In the first year the lectures were attended by 300 persons each day, in the second year attendance swelled to 400. The book itself was immediately so popular that there was a need of half a dozen reprintings in the first year alone; it is still being printed today.
The Varieties is called quintessentially James in many respects--emphasis on individuality and feeling, sympathetic understanding of human suffering and eccentricity, and predominance of vivid fact over abstract formulation. Yet it contains little of his philosophy which he is said to have given the last decade of his life to. He died after a few years of retirement in August 26, 1910. (Freud's comments of his courage).