We study fourfold rotation-invariant gapped topological systems with time-reversal symmetry in two and three dimensions (d=2, 3). We show that in both cases nontrivial topology is manifested by the presence of the (d-2)-dimensional edge states, existing at a point in 2D or along a line in 3D. For fermion systems without interaction, the bulk topological invariants are given in terms of the Wannier centers of filled bands and can be readily calculated using a Fu-Kane-like formula when inversion symmetry is also present. The theory is extended to strongly interacting systems through the explicit construction of microscopic models having robust (d-2)-dimensional edge states.