Pleiotropic opposing dominance within a color gene block contributes to a nascent species boundary via its influence on hybrid male territorial behavior

Recently, we found that selection within a single gene block associated with plumage color variation has maintained a stable and narrow hybrid zone. Here, we investigated the potential role of plumage signals in moderating a behavioral mechanism of selection. Specifically, we assessed whether two plumage traits are associated with body size among breeding males and if trait mismatch predicted aggressive behavior within hybrid and parental individuals in response to simulated territorial intrusion.

The two plumage signals, cheek and flank coloration, though associated with the same gene block, reflect opposing dominance of SOCC and STOW alleles. We found that both plumage traits significantly predict the body size in the territorial sex (i.e. males). The opposing dominance of the single color gene block resulted in plumage signal discordance in heterozygotes, which in turn was associated with reduced hybrid territorial performance, an important proxy of fitness in this system.

Taken together, these observations point to a single-locus-two-alleles mechanism of incompatibility in shaping a natural species boundary in the early stage of speciation.