Although they live in the ocean like fish, whales are clearly mammals. A reasonably complete series of intermediate fossils have been found for the evolution of whales from terrestrial mammals. A sampling of these transitional species is shown below:
Modern whales have no external hind limbs, and have their nostrils or blowholes on top of their heads. Pakicetus (shown in the figure) lived around 52 million years ago. Although it probably spent considerable time submerged in rivers hunting its prey, it was fully terrestrial. It had long, slender legs, and nostrils at the tip of its snout, like most mammals. Dorudon appeared about 12 million years later, and is maybe halfway to modern whales, being fully aquatic, with tiny vestigial hind limbs and nostrils halfway back on its snout. Whales largely similar to modern species appear after another 12 million years or so. The more modern traits include either capabilities for echolocation via “sonar” or mouths that gulp in volumes of water and then filter it to capture prey. This is fairly fast in terms of typical evolution for changes of these magnitudes.
How did all this occur so fast? It turns out that a lot of the changes in the whale lineage came about by simply deactivating various existing genes. That is a lot easier than having to evolve whole new genes.
The figure below maps out many of the changes in genes that took place in the course of whale evolution. Purple dots mean the genes are still intact, but are down-regulated so that they no longer produce the proteins they originally did. The SHH and HAND2 genes are present in whale genomes but are inhibited from growing large hind limbs like they normally do in mammals. Red dots mean the genes were totally inactivated, due to mutations. Most whales which use echolocation to find and catch prey apparently have little use for regular smelling, and so many of their Olfactory Receptor (“OR genes”) have been deactivated. Baleen whales have no teeth; instead they filter mouthfuls of water through porous, hairlike plates of baleen to capture small prey. Baleen whales still have the genes in their genomes for making teeth, but these genes have become deactivated (See “No enamel, no teeth” in the diagram, again with red dots). Orange dots indicate positive selection to modify existing genes, which again typically involves fewer mutations than evolving whole new genes.
Some differences in genes/expression between whales and terrestrial animals, mapped onto anatomy. Figure from Michael R. McGowen, John Gatesy, Derek E. Wildman, “Molecular evolution tracks macroevolutionary transitions in Cetacea”. Trends in Ecology & Evolution, Volume 29, Issue 6, June 2014, Pages 336-346.
A more recent study (Huelsmann, 2019) identified further instances on how the deactivation or down-regulation of specific genes in whales can explain many of their unique characteristics that are so well adapted to living in the ocean and diving deeply. These include altering blood clotting, protecting lungs from diving stress, and changing melatonin production so as to facilitate sleep patterns.
If you just can’t get enough of whale evolution, see my recent, really long article here.