The natural world is optically unconstrained. Surface properties may vary from one point to another, and reflected light may vary from one instant to the next. The aim of this work is to quantify some of the physical failures of color vision performance that result from uncertainty. In computational simulations with images of vegetated and nonvegetated outdoor scenes, it is shown that color provides an unreliable guide to surface identity. It is also shown that changes in illuminant may cause colors to no longer match and the relations between individual colors to vary. These failures are generally well described by a measure of the randomness of the colors in scenes, the Shannon entropy. Although uncertainty is intrinsic to the environment, its consequences for color vision can be predicted.