As David J. DeLaura and a number of other historians of Victorian intellectual life have pointed out, many Victorian atheists and agnostics abandoned Christianity for a particularly Victorian reason: They found it immoral!
The loss of religious faith in such representative early Victorian aginostics as F. W. Newman (John Henry Newman's brother), J. A. Froude (brother of Newman's close friend, Hurrell Froude), and George Eliot was not due, in the first place, to the usually suggested reasons -- the rise of evolutionary theory in geology and biology and the Higher Criticism of the Bible. Indeed, in each life the dominant factor was a growing repugnance toward the ethical implications of what each had been taught to believe as essential Christianity -- especially the set of interrelated doctrines: Original Sin, Reprobation, Baptismal Regeneration, Vicarious Atonement, Eternal Punishment. 
George MacDonald, for example, left his Congregationalist pulpit because he could not accept that God would damn for all eternity babies who had not been baptised before their death, and similar repugnance proved the straw that broke the camel's back for Ruskin. In St. Paul and Protestantism, Arnold "contemptuously rejects the 'monstrous' vision of a capricious God who deals in election andf predestination and cruelly emphasizes the crass commercial quality of the Puritan catchwords, "covenant," "ransom," "redeem," "purchase," bargain" [DeLaura, 13].
Altholz, Josef L. "The Warfare of Conscience with Theology." The Mind and Art of Victorian England. Ed. Josef L. Altholz. Minneapolis: U. of Minnesota P, 1976. 58-77.
DeLaura, David J. Hebrew and Hellene in Victorian England: Newman, Arnold, and Pater. Austin: U of Texas P, 1969.
Murphy, Howard R. "The Ethical Revolt against Christian Orthodoxy in Early Victorian England." American Historical Review 60 (July 1955): 800-17.
Last modified 1999