We report new limits on ultralight scalar dark matter (DM) with dilatonlike couplings to photons that can induce oscillations in the fine-structure constant α. Atomic dysprosium exhibits an electronic structure with two nearly degenerate levels whose energy splitting is sensitive to changes in α. Spectroscopy data for two isotopes of dysprosium over a two-year span are analyzed for coherent oscillations with angular frequencies below 1 rad s-1. No signal consistent with a DM coupling is identified, leading to new constraints on dilatonlike photon couplings over a wide mass range. Under the assumption that the scalar field comprises all of the DM, our limits on the coupling exceed those from equivalence-principle tests by up to 4 orders of magnitude for masses below 3×10(-18) eV. Excess oscillatory power, inconsistent with fine-structure variation, is detected in a control channel, and is likely due to a systematic effect. Our atomic spectroscopy limits on DM are the first of their kind, and leave substantial room for improvement with state-of-the-art atomic clocks.