Insulator-based dielectrophoretic separation of small particles in a sawtooth channel is studied in the limit of dilute concentration. Pathlines for the movements of infinitesimal particles are constructed and the geometric changes of these pathlines are used to establish the criterion for blocking and trapping particles with different physical properties. The sharp corners of the sawtooth channel create much stronger dielectrophoretic force than channels with smooth corners for blocking particle movements. Particle blocking and trapping depend on particle properties and the geometry of the device. It is shown that once the channel geometric aspect ratios are specified, the blocking criterion depends on only a single dimensionless parameter C defined in terms of the particle mobility ratio (dielectrophoretic versus electrokinetic), the applied voltage and the spacing between the teeth. Selective blocking and trapping of particles can be realized by varying the geometry of the channel progressively. High-resolution separation can be achieved by tuning the differential in the parameter C to a desired level.