Odor molecules are detected by the olfactory receptors (hereafter OR) in the olfactory epithelium of the nasal cavity. Each receptor type is expressed within a subset of neurons, from which they directly connect to the olfactory bulb in the brain.^[1] Olfaction is essential for survival in most vertebrates; however, the degree to which an animal depends on smell is highly varied.^[2] Great variation exists in the number of OR genes among vertebrate species, as shown through bioinformatic analyses. This diversity exists by virtue of the wide-ranging environments that they inhabit. For instance, dolphins that are secondarily adapted to an aquatic niche possess a considerably smaller subset of genes than most mammals.^[3] OR gene repertoires have also evolved in relation to other senses, as higher primates with well-developed vision systems tend to have a smaller number of OR genes. As such, investigating the evolutionary changes of OR genes can provide useful information on how genomes respond to environmental changes. Differences in smell sensitivity are also dependent on the anatomy of the olfactory apparatus, such as the size of the olfactory bulb and epithelium.

Nonetheless, the general features of the olfactory system are highly conserved among vertebrates,^[4] and, similarly to other sensory systems, olfaction has undergone fairly modest changes throughout the evolution of vertebrates. Phylogenetic analyses reveal that at least three distinct olfactory subsystems are broadly consistent in vertebrates, and a fourth accessory system (vomeronasal) solely arose in tetrapods.^[4]

Mutations affecting OR genes on the chromosome are primarily responsible for the evolution of smell. OR genes are grouped in clusters along multiple chromosomes and are responsible for coding respective OR proteins. These proteins contain seven transmembrane domains that are responsible for detecting specific sets of odor molecules.^[5] OR genes are located on error-prone regions of the chromosome, and consequently, the DNA of the OR gene is periodically duplicated during crossover.^[6] After this duplication event, one of the two genes may mutate and disable its function, rendering it as a pseudogene. Alternatively, the duplicated copy may mutate without dysfunctionality, and will continue making the same olfactory receptor but with altered structural changes. This protein adjustment can induce a subtle shift in the range of smells an animal can sense. The diversity of smelling genes present in humans today are attributed to multiple rounds of mutations that have occurred throughout vertebrate evolution.^[5][6]

In particular, repeated rounds of gene duplication, deletion, and pseudogene evolution contribute to the variety of OR gene number. Formally known as "birth-and-death evolution", these dynamics are measured by the number of gains and losses from genes in each branch of the phylogenetic tree in question.^[5] Statistical methods can be used to estimate the total number of gains and losses, which can be as large as several hundred per branch of the tree.^[7] Moreover, the number of gains and losses can be enormous even if two extant species possess the same gene number (for example, humans and macaques).

Both adaptation and random events can cause birth-and-death evolution. The probability of gene duplication is dictated by chance events and primarily occurs through unequal crossover; this excludes the rare event of whole genome duplication. Alternatively, the Fixation of duplicate genes can be influenced by natural selection or can occur randomly.^[8] Within the mammalian phylogenetic tree, a large number of gene gains and losses are observed for almost all branches, suggesting that a significant fraction of gene number changes were caused by inactivation events and random gene duplication. This process is known as genomic drift, or "random genetic drift of gene frequencies" in population genetics. Pseudogenes are also subject to genomic drift, since they are rendered as non-functional and are believed to evolve in a neutral manner.^[9]

At least nine groups of vertebrate OR genes have been identified (α,β,γ,δ,ε,ζ,η, and θ), each of which derived from ancestral genes in the most common ancestor of tetrapods and fish.^[5] Specifically, vertebrate OR genes convey an evolutionary pattern of three separate lineages: fish, amphibians, and mammals.^[10]

Vertebrate olfaction was first derived in an aquatic ecosystem, where water was the primary medium for odorants. Lancelets, a class of fish-like marine chordates, are the most distantly related ancestors that share the same olfactory receptors (OR) with humans.^[3] They diverged from our own ancestors approximately 550 million years ago, shortly before the development of the camera eyes and brain. Although they lack an identifiable olfactory organ, lancelets possess 40 olfactory receptor genes that are studded along their flanks to detect odor molecules from their surrounding aquatic environment. Furthermore, many vertebrate-type olfactory receptor genes were found in their genome.^[3] The common ancestor of vertebrates and tunicates, olfactores, had developed the olfactory system more than the lancelets did.

Serving as living descendants that thrived almost half a billion years ago, the study of OR genes in lampreys also provides deep insight into the origins of vertebrate olfaction.^[11] Structural motifs that are characteristic hallmarks of vertebrate ORs were discerned when isolating OR genes expressed in lamprey olfactory organs. Phylogenetic analysis comparing the OR genes of fish and mammals revealed that lamprey ORs diverged from higher vertebrates before the origin of Class I and II genes.^[11] It was also found that the lamprey OR gene repertoire was relatively small. Gene duplication and genetic drift expanded these small repertoires in gene number over evolutionary time, supporting the notion that the lamprey OR sequence represents an ancient OR family.^[5][12]

The vast diversity of olfactory organs is largely attributed to the long evolutionary lifespan of fish.^[13] Representing an evolutionarily older class of vertebrates, fish carry several ancestral OR genes on their chromosome, yet they express wide variation in these genes among individuals and species.^[14] Fish have reduced olfactory systems in comparison to mammals since they only contain approximately 100 OR, whereas most mammals can possess 500 to 1000.

New phylogenomic analysis shows urochordates, rather than cephalochordates, as the sister group to vertebrates.^[15][16] All OR genes were lost in the lineage of urochordata because of the absence of any vertebrate-type OR-like genes. The loss of vertebrate type OR-genes therefore suggests that they may have been lost in the urochordates genomes (lost genes conserved between amphioxus and vertebrates).^[17]

Phylogenetic analysis has delineated two main types of olfactory receptor genes that are traced back to early vertebrate OR history: class I genes, responsible for encoding proteins that detect water-soluble odorants, and Class II genes, associated with the detection of airborne molecules. Both types serve as homologous structures in fish and tetrapods.^[18]

In Xenopus laevis, an amphibia, both class receptors are present, unlike fish or mammals. Their class I receptors are expressed on the lateral diverticulum (LD), a region specialized to respond to water-soluble odorants, and class II receptors are abundantly present in the medial diverticulum (MD) which is specialized at detecting volatile odors.^[18]

All fish perceive odors though nostril-like structures called nares. Most fish lack vomeronasal structures, yet they are still able to detect pheromones through vomeronasal-related genes. In lieu of the organ, fish possess sensory epithelium that contains three types of sensory cells, two of which are responsible for sex pheromones and social cures.^[19] Considering the slow diffusion of odor molecules through water, it is evolutionarily reasonable to possess only one type of cell dedicated to odorants and two cells dedicated to chemical communication.^[2]

Fish primarily use pheromones to facilitate social behavior, such as social and reproductive cues and predator avoidance.^[2] Antipredation pheromones vary across species; some may emit chemical cues that provoke physiological changes (e.g. body depth increase), or cues that promote evasion (release of odor cues of dead conspecifics). Pheromone detection is also highly used in kin identification, survival-enhancing aggregation (e.g. shoaling), and migratory signalling. Sex-selected pheromones are also able to distinguish between male and females for mating and spawning.^[20]

An OR gene radiation co-existed during the time of reptile dominance approximately 200 million years ago. This occurrence would have likely signaled a major speciation event or adaptive shift in the prevalent organisms. The earliest tetrapods thrived in aquatic environments prior to making their initial transition to land. Correspondingly, modern amphibians too began in water before emerging on land as adults. This dual adaptation was selected for since early tetrapods and modern amphibians could reap the resources of both environments. As a result, modern amphibians possess olfactory organs that are specialized in detecting both volatile and water-soluble odors. Odorants in water are detected in a manner that is akin to fish, in which a respiratory pump is responsible for filling their nasal cavity with water.^[21]

Coelacanths are widely referred to as the intermediate between fish and tetrapods since they contain limb-like protrusions. One such species, Latimeria chalumnae, holds particular interest since they contain Class II OR genes that are present in mammals and amphibians but are absent in fish.^[22] Acting as a close living relative to tetrapods, recent studies elucidating the coelacanth OR repertoire state that the common ancestor of mammals and fish had both Class I and II OR genes.

Tetrapods exhibit both a main and accessory olfactory system. The main olfactory sense is derived from the more ancient neural system, broadly present across insects and mammals.^[23] This system is specialized to detect volatile, airborne molecules. The accessory olfactory system is the more recently evolved structure, first appearing in the common ancestor of modern amniotes and amphibians. The detection of non-volatile molecules and chemoreception seems to be the primary function of this system, as the accessory organ (Jacobson's organ) contains vomeronasal receptors responsible for environmental sampling.^[24] Quantitative research has suggested that the relative size of the main olfactory bulb is highly correlated with ecological adaptations, while the relative size of the accessory olfactory bulb is related to sociosexual factors.^[25]

Evolutionary loss of the accessory olfactory system is observed in multiple tetrapod lineages, which is believed to be caused by the overlapping redundancy of function with the main olfactory system.^[26] Correspondingly, the reduction of vomeronasal receptor genes is often attributed to the transition of tetrapods from water to land.^[27] However, with the exception of the archosaur clade, the accessory olfactory system is at least primitively possessed (vestigially) in all higher taxa of tetrapods.^[26]

The skull of the extinct Hadrocodium wui, which are considered reptiles that evolved into the first mammals, has unveiled significant implications of the reptilian olfactory transition. In comparison to their descendants, CT scans of these craniums revealed that the olfactory bulb had increased in size over large timescales. This sequence of events culminated the origin of mammalian olfaction, suggesting an OR gene shift from reptiles to mammals.^[28]

The olfactory genetics of mammals are divergent from the vertebrate lineage when looking at the size of the OR gene family. Alone, this family of genes makes up 1% of the entire active genome and represents the largest gene family for all species.^[29] Therefore, mammals may be evidence of a second adaptation radiation event that occurred. These mammalian OR genes have a tendency to cluster on chromosomal ends or telomeres. Recombination is more rapid at teleomeric sites. These telemetric loci are evidence for OR gene expansion where genes were duplicated quickly.

One study used newly developed phylogeny-based methods to classify well over 10,000 OR genes from 13 different placental mammals. The experimenters separated them into orthologous gene groups (OGGs). What they found was that OR gene lineages, which had experienced more gene duplication, had a weaker purifying selection while Class II OR genes evolved dynamically compared to Class 1.^[30]

Mammals (unlike other tetrapods) utilize a nose to sense volatile odors.^[23] The appearance of nasal turbinates and fossae, which resemble scroll-shaped spongy bones in the nasal passage, is one of the distinctive features of their evolution. These structures are paired on both sides of the midline nasal septum, linking the external nasal opening and the internal nasal aperture together. Evidence for their first appearance are shown in fossils of Therapsida, from which mammals descend. The first true mammals developed additional nasal tissue to carry more neurons that transfer olfactory information to the brain.^[31] Among amniotes, mammals were the first to evolve a complex system of nasal turbinates, which augment the surface area for olfactory epithelium. The complexity of turbinates vary highly across mammalian species, yet a correlation appears to exist among phylogenetic groups rather than the environmental niche.^[32][33]

The skull of Brasilitherium provides useful information when discerning the emergence of a true nose in mammals. Brasilitherium was not a mammal but indeed a synapsid tetrapod that served as a close predecessor to the first mammaliaforms. The emergence of turbinates can be recognized through the observation of this 227 million year old protomammal.^[34]

The nasal concha, or turbinates, is composed of little bones and soft tissue that provide structure to the nose and aid in the perception of smell. Paleontological work on other protomammals has identified specialized ridges lined between the orbital fissures, superior to the maxillary plate. Research has yet to define a correlation between these ridges and the evidence of turbinates, mostly based on the assumption that they were probably cartilaginous and difficult to preserve. The lack of intact skulls in these early mammals also serves as an impediment to study.^[34] The skull of Brasilitherium held more promising results, as it contained a secondary palate separating the nose from the mouth, thereby enhancing skeletal durability and preservation of the turbinate structure. Researchers identified small shards of bone inside the skull's nasal cavity that were assumed to be parts of the turbinates. It was inferred that these structures found in Brasilitherium performed the same roles in modern mammals; specifically, the anterior overlying tissue warmed incoming odorants and the posterior portion was responsible for picking up scent. The complex latter section of the nasal structure suggests that Brasilitherium had a well-developed sense of smell.^[34] The same studies also found a hollow across the secondary plate, implicating the presence of the Jacobson's organ. These observations suggest that this organism possessed features that connect both its tetrapod ancestors and later mammalians. Particularly, the Brasilitherium nose had transitional features that help elucidate the emergence of protruding mammalian noses.^[34]

Structural changes in rhinaria provide suggestive material for phylogenetic analysis. Generally speaking, the rhinarium is present in most species of mammals and is correlated with an acute sense of smell. Therefore, it is likely that this trait emerged in the protomammal stage.^[35] The adaptations that gave rise to this anatomical structure largely reflected the ecological factors that affected the mammals of the period. In most species with a rhinarium, the organ takes the form of a firm pad with dual nostrils, allowing for the processing of both olfactory and tactile information.^[35] As such, the structure of the rhinarium is assumed to be important in the evolution of exploratory behaviour. A common assumption is that higher primates lost the rhinarium secondarily because of their decreasing reliance on olfaction.

One hallmark feature of the Order Primates is the diminished emphasis on olfaction.^[23] Smell is often viewed as a mitigated special sense during the emergence of higher neural function, and correspondingly, olfaction has been increasingly reduced throughout the course of primate evolution. Some researchers relate the de-emphasis of smell to the emergence of complex vision. Accordingly, many primate species convey a large proportion of OR pseudogenes, with the highest levels shown in humans, chimpanzees, and gorillas.^[23] Humans only contain approximately 400 functional genes, contrasted with the 600 OR pseudogenes present within the genome. The disparity is explained by the development of acute vision in Catarrhini (apes and Old World monkeys) 40 million years ago, namely during the period when the Earth became cooler. Trichromatic vision was evolved to enhance long-distance perception and foraging for ripe fruit, reducing the selective advantage of possessing a large OR gene repertoire.^[23]

Contrarily, New World howler monkeys (Alouatta) have independently evolved routine trichromatic vision, yet they still perform pheromone communication and do not exhibit reduced olfactory capabilities. It is suggested that although enhanced vision relaxed the selection of sensitive smell, it did not render olfaction as an unneeded trait, and may have been advantageous in a jungle habitat.^[36]

In primates, olfaction is primarily important for social signalling and dietary strategies.^[23] Ample evidence suggests that olfactory social behaviours, such as sniffing and scent marking, are heavily involved in communicative interactions among primate species. For instance, scent marking with glandular secretions is a prevalent mode of signalling in strepsirrhines. Within the order Catarrhini, all hominoid genera contain specialized cutaneous scent glands (i.e. apocrine glands in the axilla) to detect aromatic cues.^[37] Recent studies have also drawn connections between olfactory function and mate preference across a broad range of primates, including humans.^[37]

During the twentieth century, a myriad of theories were developed for the origin of primate sensory adaptations. Evolutionary "progress" was marked by the adaptive characteristics of visual and tactile structures, yet a reduction in olfactory abilities.^[23] This theory was further elaborated by Le Gros Clark, asserting the reduction of olfactory structures was attributed to the diminished need for smell in arboreal environments.^[38] This theory was later challenged with the existence of arboreal mammals that do not exhibit primate traits that are considered "adaptive" (specifically, reduced olfaction), yet they are still successful within their respective environment.^[39] Furthermore, strepsirrhine primates demonstrate the social behavior of scent-marking in both terrestrial and arboreal habitats.^[40] Arboreal New World primates (Platyrrhini) possess skin scent-glands and also exhibit the same scent-marking traits as do strepsirrhines. Studies have provided a structural explanation for the reduced olfaction in primates, proposing that degeneration of the nasal region resulted from crowding of the nasal cavity in conjunction with progressively convergent orbits.^{[citation needed]}

Aside from an external nose structure, some primates contain a vomeronasal organ to detect odorants of higher molecular weight.^[23] Genomic analysis has asserted that vomeronasal organ receptors became impaired approximately 23 million years ago in primate evolution, before the advent separation of Old World monkeys and hominoids. Furthermore, multiple lines of evidence depict that the entire accessory olfactory system became nonfunctional in pheromonal communication before this divergence took place.^{[23][41][42][43]} Currently, no structure of the vomeronasal organ has been elucidated in Old World monkeys, although it has been shown that an apparent vestigial vomeronasal organ develops but degenerates before birth.^[23] Conversely, human embryos possess a vomeronasal organ that persists, although this feature is vestigial throughout their lifetime.

Primates are phylogenetically divided into Strepsirrhini, species that possess a curly "wet nose" rhinarium, and Haplorhini, those that possess a dry "simple nose" structure. Strepsirrhines are considered to have more primitive features and adaptations because of their preservation of heightened smell.^[44] In haplorrhine primates, the loss of the wet rhinarium and reduced number of turbinates are correlated with their diminished reliance on smell. Accordingly, the dry noses of humans place them under the Haplorhini clade as well.^[44]

More than 900 human OR genes and pseudogenes have been identified.^[45] Close to 63% of the ORs are changed by a random process of pseudogene formation. These ORs constitute 17 gene families, of which only 4 contain more than 100 members each. There are Class I ORs, Fish-like, which form 10% of the human repertoire all on one cluster on chromosome 11. ORs are present on all human chromosomes with the exception of 20 and the male sex chromosome ‘Y'. 80% of all ORs are found in clusters containing anywhere from 6-138 genes.^[45]

A comparative cluster analysis study traced the evolutionary pathway that led to OR proliferation and diversification. The study concluded the following expansion history 1) a Class II OR cluster on chromosome 11 was generated via local duplication 2) Single-step duplication from the cluster on chromosome 11 to chromosome 1 3) A storm of duplication events from chromosome 1 to other chromosomes.^[45]

In recent humans, nasal configuration is mostly associated with nasal bridge elevation and width of the internal nasal cavity. Emergence of such a structure mainly derives from respiratory needs in varying climates; for instance, a large nasal cavity in Neanderthals adjusted for the cold environment and low humidity of that epoch.^[46] In higher primates, the structural reduction of the snout is correlated with diminished priority for olfaction. As such, the human nose displays reduced innervations of the olfactory mucous membrane, decreased snout length, and an overall reduction of complexity of the nasal concha.^[46]

The role of smell has long been viewed as secondary to the importance of auditory, tactile, and visual senses.^[23] Humans do not rely on olfaction for survival to the same extent as other species. Instead, smell plays a heavier role in aesthetic food perception and gathering information on the surroundings.^[1] Nevertheless, humans also communicate via odorants and pheromones, exerting both subconscious and conscious (artificial) scents.^{[citation needed]}

Humans have a high interindividual variation in pseudogenes and OR genes which research attributes to geographical and cultural separation. Selection and cultural practices can conserve OR genes even though it is likely caused by bottleneck effects and geographic isolation.^[47]

A weak positive selection acting on human nucleotide diversity is proposed because of a report that observed genomic segments in a 450kb cluster of olfactory genes found on chromosome 17.^[47] They observed a lower ratio of nucleotide diversity to divergence in intact genes compared to introns and pseudogenes. Additionally a small but significant reduction in variability was observed in the OR gene cluster when compared to other genomic areas.^[47]

Geographically and culturally distinct human populations differ when comparing regions on chromosome 17 (looking at SNPs) in coding regions of 400-kb olfactory receptor gene clusters. These differences are proposed to help with future genotype-phenotype studies such as evaluating the effect of genetic drift on these populations and finding greater functionality in pygmyolfactory receptor genes and pseudogenization.^[47]

3 million SNPs in the HapMap database of four populations was analyzed, and it was found that heterozygotes dominate significantly. Because of the double amount of odorant binding sites it is theorized that heterozygotes are selected for.^[47] This could be an evolutionary response to current human olfactory receptor loss.^[47]

Comparisons between human and mouse OR gene repertoires have been well documented in genomic and phylogenetic analysis. Identifying the orthologous relationship between their genes provides key translating data from mouse studies in understanding human olfaction.^[1] Approximately 63% of the human OR repertoire has degenerated to pseudogenes, whereas mice only exhibit 20% of pseudogenization. In addition, human OR genes lack motifs that are highly conserved in mouse OR genomes, implicating that not all human OR genes encode functional OR proteins. These differences are explained by the reduced reliance of smell in humans in comparison to rodents.^[48] It is still unclear whether the extensive OR repertoires of mice enable them to detect a larger range of odorants than humans. When human OR sequences are analyzed phylogenetically, intact human genes are found in most OR subfamilies.^[1] Assuming that various OR subfamilies bind to different odorant classes, it is likely that humans are able to detect a wide range of smell similarly to mice.^[1]

Similarly to humans, whales and dolphins have experienced an independent loss of functional OR genes. This is demonstrated by the organism's return to an aquatic environment, subsequently experiencing a loss of terrestrial olfactory register.^[49] Olfactory senses are generally more heightened for a terrestrial species than for an aquatic, as airborne volatiles are more important to detect than water-soluble scents in land animals. Support for this theory is exemplified through the fish-to-tetrapod transition, where animals began to populate terrestrial niches and a tremendous expansion of the olfactory system can be observed.^[49]

Once cetaceans evolved from their terrestrial ancestors, the reduction in their olfactory apparatus was primarily clade-specific. Toothed whales (Odontoceti) seem to have lost their olfactory sense completely, whereas baleen whales have shown partial impairment, expressing about 58% OR pseudogenes in their cluster. Some adult tooth whales express 77% OR pseudogenes and are completely devoid of olfactory structures. This occurrence is analogous to that of the blind Southern marsupial mole, in which mutation of the inter-photoreceptor protein was paired with anatomical degeneration of the eyes.^[50]

Two other marine vertebrates, the Steller sea lion and the loggerhead turtle, display a rather low number of pseudogenes with respect to cetaceans. The preservation of their OR genes is correlated with their semi-adaptation to an aquatic habitat, since these organisms still perform many functions on land (e.g. mating, basking). Cetaceans do not depend on terrestrial environments, and have thusly lost a large fraction of their OR repertoire. Echolocation was presumable evolved to compensate for this loss.^[50]

Previous research has elucidated the psychophysical and behavioural characteristics of olfactory adaptation. Adaptation allows for the olfactory system to appropriately respond to the appearance of novel scents or changes, yet it also maintains equilibrium with odorant concentrations in ambient environments. Like other sensory systems, prolonged exposure to an odorant often leads to a stimulus-specific decrease in olfactory sensitivity to that particular odor; however, the absence of further exposure will regain olfactory sensitivity.^[51] Psychophysical analysis has demonstrated both an elevation of odor thresholds and a diminished responsivity to suprathreshold stimulation under the effects of olfactory adaptation. In addition, evidence suggests that the time course of adaptation and recovery relies on the duration and concentration of the odor.^[51]

It is generally understood that olfactory adaptation involves both receptor level and post-receptor components (peripheral and central regions, respectively). The central role in adaptation was first proposed in a model in 1990,^[52] and this model has generated supporting experimental data.^[53] Support for the participation of both central and peripheral systems derives from experiments showing that monorhinal stimulation results in both ipsilateral and contralateral adaptation. However, the degree of adaptation is more profound in the ipsilateral nostril, and recovery appears to be slower.^[51] Psychophysical support for this theory draws from studies that have reported relatively small decreases in peripheral response after repeated stimulation despite significant reductions in behaviorally perceived intensity.

Unlike vision and hearing, olfactory chemical stimulation does not terminate after the removal of the stimulus. As such, although some degree of olfactory adaptation may be attributed to response characteristics of receptors, another influence may be the delay in signal termination within the receptor environment. Consequently, the degree of adaptation may rely on differences in odorant clearance among species, which would include properties of nasal mucociliary, submucosal blood flow, and expiratory desorption.^[54]