We study the low temperature physics of an ultracold atomic gas in the potential formed inside a pumped optical resonator. Here, the height of the cavity potential, and hence the quantum state of the gas, depends not only on the pump parameters, but also on the atomic density through a dynamical ac-Stark shift of the cavity resonance. We derive the Bose-Hubbard model in one dimension and use the strong coupling expansion to determine the parameter regime in which the system is in the Mott-insulator state. We predict the existence of overlapping, competing Mott-insulator states, and bistable behavior in the vicinity of the shifted cavity resonance, controlled by the pump parameters. Outside these parameter regions, the state of the system is in most cases superfluid.