We present a method to implement ultrafast two-qubit gates valid for the ultrastrong coupling and deep strong coupling regimes of light-matter interaction, considering state-of-the-art circuit quantum electrodynamics technology. Our proposal includes a suitable qubit architecture and is based on a four-step sequential displacement of the intracavity field, operating at a time proportional to the inverse of the resonator frequency. Through ab initio calculations, we show that these quantum gates can be performed at subnanosecond time scales while keeping a fidelity above 99%.