Oryza sativa, the common cultivated rice, is one of the most important crops for human consumption, but production is increasingly threatened by abiotic stresses. Although many efforts have resulted in breeding rice cultivars that are relatively tolerant to their local environments, climate changes and population increase are expected to soon call for new, fast generation of stress tolerant rice germplasm, and current within-species rice diversity might not be enough to overcome such needs. The Oryza genus contains other 23 wild species, with only Oryza glaberrima being also domesticated. Rice domestication was performed with a narrow genetic diversity, and the other Oryza species are a virtually untapped genetic resource for rice stress tolerance improvement. Here we review the origin of domesticated Oryza sativa from wild progenitors, the ecological and genomic diversity of the Oryza genus, and the stress tolerance variation observed for wild Oryza species, including the genetic basis underlying the tolerance mechanisms found. The summary provided here is important to indicate how we should move forward to unlock the full potential of these germplasms for rice improvement.