Eye formation is the result of coordinated induction and differentiation processes during embryogenesis. Disruption of any one of these events has the potential to cause ocular growth and structural defects, such as anophthalmia and microphthalmia (A/M). A/M can be isolated or occur with systemic anomalies, when they may form part of a recognizable syndrome. Their etiology includes genetic and environmental factors; several hundred genes involved in ocular development have been identified in humans or animal models. In humans, around 30 genes have been repeatedly implicated in A/M families, although many other genes have been described in single cases or families, and some genetic syndromes include eye anomalies occasionally as part of a wider phenotype. As a result of this broad genetic heterogeneity, with one or two notable exceptions, each gene explains only a small percentage of cases. Given the overlapping phenotypes, these genes can be most efficiently tested on panels or by whole exome/genome sequencing for the purposes of molecular diagnosis. However, despite whole exome/genome testing more than half of patients currently remain without a molecular diagnosis. The proportion of undiagnosed cases is even higher in those individuals with unilateral or milder phenotypes. Furthermore, even when a strong gene candidate is available for a patient, issues of incomplete penetrance and germinal mosaicism make diagnosis and genetic counseling challenging. In this review, we present the main genes implicated in non-syndromic human A/M phenotypes and, for practical purposes, classify them according to the most frequent or predominant phenotype each is associated with. Our intention is that this will allow clinicians to rank and prioritize their molecular analyses and interpretations according to the phenotypes of their patients.