Theory suggests that sex-specific selection can facilitate adaptation in sexually reproducing populations. However, sexual conflict theory and recent experiments indicate that sex-specific selection is potentially costly due to sexual antagonism: alleles harmful to one sex can accumulate within a population because they are favored in the other sex. Whether sex-specific selection provides a net fitness benefit or cost depends, in part, on the relative frequency and strength of sexually concordant versus sexually antagonistic selection throughout a species' genome. Here, we model the net fitness consequences of sex-specific selection while explicitly considering both sexually concordant and sexually antagonistic selection. The model shows that, even when sexual antagonism is rare, the fitness costs that it imposes will generally overwhelm fitness benefits of sexually concordant selection. Furthermore, the cost of sexual antagonism is, at best, only partially resolved by the evolution of sex-limited gene expression. To evaluate the key parameters of the model, we analyze an extensive dataset of sex-specific selection gradients from wild populations, along with data from the experimental evolution literature. The model and data imply that sex-specific selection may likely impose a net cost on sexually reproducing species, although additional research will be required to confirm this conclusion.