The differences in classification and training performance of three- and four-layer (one- and two-hidden-layer) fully interconnected feedforward neural nets are investigated. To obtain results which do not merely reflect performance on a particular data set, the networks are trained on various distributions, which are themselves drawn from a distribution of distributions. Experimental results indicate that four-layered networks are more prone to fall into bad local minima, but that three- and four-layered networks perform similarly in all other respects.