Taxonomy was the reason for Darwin's theory

Taxonomy - the science of classifying organisms into putatively natural groups - is often treated as a kind of necessary bit of paperwork without much theoretical import by some biologists. Others think it is the single most important thing to do, usually justifying it in terms of conservation biology, but sometimes in terms of foundational knowledge. One thing that has become clear to me is exactly how foundational taxonomy is. Now, historian Polly Winsor has published a paper in the leading taxonomic journal, appropriately named Taxon, in which she argues, I believe correctly, that Darwin's real problem, and the foundation of what he subsequently did, was taxonomy. I have been beaten to this paper by the excellent Archetype, who is also worth reading.

Polly holds that what set up Darwin's problems was the hierarchical arrangement of organisms in the Linnean system that had effectively swept all before it in the early 19th century. Darwin himself refers to this later in the Origin:

So that we here have many species descended from a single progenitor grouped into genera; and the genera are included in, or subordinate to, sub-families, families, and orders, all united into one class. Thus, the grand fact in natural history of the subordination of group under group, which, from its familiarity, does not always sufficiently strike us, is in my judgment fully explained.

Naturalists try to arrange the species, genera, and families in each class, on what is called the Natural System. But what is meant by this system? Some authors look at it merely as a scheme for arranging together those living objects which are most alike, and for separating those which are most unlike; or as an artificial means for enunciating, as briefly as possible, general propositions,—that is, by one sentence to give the characters common, for instance, to all mammals, by another those common to all carnivora, by another those common to the dog-genus, and then by adding a single sentence, a full description is given of each kind of dog. The ingenuity and utility of this system are indisputable. But many naturalists think that something more is meant by the Natural System; they believe that it reveals the plan of the Creator; but unless it be specified whether order in time or space, or what else is meant by the plan of the Creator, it seems to me that nothing is thus added to our knowledge. Such expressions as that famous one of Linnæus, and which we often meet with in a more or less concealed form, that the characters do not make the genus, but that the genus gives the characters, seem to imply that something more is included in our classification, than mere resemblance. I believe that something more is included; and that propinquity of descent,—the only known cause of the similarity of organic beings,—is the bond, hidden as it is by various degrees of modification, which is partially revealed to us by our classifications. [p413f; Polly quotes a different passage on page 128]

In the paragraph after Polly's quotation, Darwin says something I find very interesting:

The several subordinate groups in any class cannot be ranked in a single file, but seem rather to be clustered round points, and these round other points, and so on in almost endless cycles. On the view that each species has been independently created, I can see no explanation of this great fact in the classification of all organic beings; but, to the best of my judgment, it is explained through inheritance and the complex action of natural selection, entailing extinction and divergence of character, as we have seen illustrated in the diagram. [emphases added]

Darwin had previously, when trying to understand the "affinities", as he put it, of various related species, toyed with the Quinarian ideas of William Macleay and William Swainson, who had developed a kind of ideal numerological system, which had diagrams like this:

Swainson's book, I consider, is the most important for understanding Darwin's concerns.* Yes, he read Lyell and Whewell, but Swainson has a volume-length argument for the necessity to undertake natural systems in classification as a prolegomenon to doing natural history. For instance, he writes:

(105.) If we reflect for a moment on the sort of information which it is the province of natural history to teach, we shall find that all the knowledge of an organised being which it is possible to acquire, is comprised under one or other of the following heads:

1. Its structure and composition.

2. Its properties.

3. Its relations to other beings.

(106.) Hence it naturally follows that a knowledge of species is the true basis upon which the science reposes for its successful prosecution. We cannot combine objects, with due regard to their fitness, until we understand their structure and properties; any more than we can acquire a language, before we become acquainted with its alphabet.

(107.) A knowledge of structure, and of properties, is to natural history, what experience is to other branches of physical science. In either case, we look not to causes, for they are beyond our comprehension; but we look to objects or to facts, which every body, under favourable circumstances, can verify; and which, in consequence, become immutable truths. Upon this basis, therefore, we must commence the study of nature: and proceeding step by step, measuring back our ground when we begin to doubt, yet gaining confidence from every corroborating evidence, we advance from the foundation to the portico. [Swainson, Preliminary Discourse, p166]

Swainson's osculating circles (i.e., "kissing" circles) is a solution to the same problem as Darwin's. However, Swainson thinks that we cannot give causal explanations for the structure of group under group, where Darwin, as Ruse has argued, thinks he should. Swainson does, however, think that taxonomy is the basis for theorising, and this point Polly makes in her paper:

Darwin’s situation to the world-altering revolution that culminated with Isaac Newton. Newton famously supplied the causes of planetary motion, inertia and gravity, and Darwin has been called, with good reason, the Newton of biology. There would be good reason to crown Darwin the Copernicus of biology too, if Mayr is correct to claim that he was the first to posit branching evolution. (People who recall Lamarck’s branching diagram may dismiss Mayr’s surprising claim out of hand, but the presence of extant groups along the trunk and branches of Lamarck’s tree signals how confused was Lamarck’s view, due to the lingering influence of the old chain of being as well as his reluctance to believe in extinction.) Copernicus proposed the sun-centered system to which Newton, 144 years later, supplied the gears. Contrary to the notion that a theory without a causal mechanism is unsatisfactory, Copernicus was willing to believe in a radically new cosmos even though he had no idea of what drove it. Is it possible that a closer look at Copernicus’s reasoning can give us a clue to Darwin’s situation between March 1837 and September 1838 (as well as Wallace’s between 1845 and 1858)?

Darwin was indeed the first to posit tree branching - Lamarck's diagram is more like a road system that many lineages can evolve along, than an evolutionary history. And like Copernicus, Darwin did think the evidence of taxonomy was the Kepler to his new World System. But he was also the Newton to a blade of grass as Kant had denied was possible. He gave mechanisms.

However, while the "Natural System" of "group under group" sets up a problem, it is not solved by natural selection, because as Darwin noted, adaptive characters are least useful for systematists (the terms "taxonomist" and "systematist" really are synonyms, despite later authors trying to make them separate activities). This is because similar modes of life call for similar solutions, and the groups that share these modes of life can be quite distinct. Today we would say that this is because of convergent evolution.

Darwin's greatest insight, to my mind, is the notion of common descent. He overcomes (and explicitly criticises) the Swainsonian intelligent design argument that the order of things can only be ascribed to divine plan, and as Polly says, his reasoning is strictly analogous to that of Copernicus and Newton.

* Swainson's book also contains an eloquent plea for the adequate funding of taxonomy, a plea that has been repeated ever since. And his Treatise also started the work that Wallace was later to develop into biogeography, as well as containing expositions of various systems of taxonomy, including Cuvier's which he held in low regard.

References

Mary P. Winsor (2009). Taxonomy was the foundation of Darwin’s evolution Taxon, 58 (1), 1-7

Swainson, William. 1834. Preliminary discourse on the study of natural history. London: Longman, Rees, Orme, Brown, Green and Longman.
———. 1835. A Treatise on the Geography and Classification of Animals. Edited by R. D. Lardner, The Cabinet Cyclopaedia: Natural History. London: Longman Rees, Orme, Brown & Longman.