In the automatic detection for surface defects of optical components, the digs and dust particles exhibit similar features: point-like shape and variable intensity reflectivity. On this condition, these two types with entirely different damages are easily confused so that misjudgments will be induced. To solve this problem, a polarization-characteristics-based classification method of digs and dust particles (PCCDD) is proposed based on the polarimetric imaging technique and dark-field imaging technique. First, a dark-field imaging system equipped with a polarization state generator (PSG) and a polarization state analyzer (PSA) is employed to measure and establish normalized Mueller matrices' datasets of digs and dust particles. And by a nonlinear global search combined with a separability evaluation method, the optimal number of acquisitions and corresponding polarization measurement states of the PSG and the PSA are obtained, as well as the parameters of classification function. Then, multiple polarization images are acquired under the optimal states to extract a multidimensional feature description that relates only to the polarization characteristics of the defect; this subsequently acts as the input vector of the classifier to finally achieve the classification. This method takes full advantage of both the difference in polarization properties between digs and dust particles and the characteristic that the polarization properties of digs are relatively invariant while those of dust particles have a large variability. The classification process involves only simple matrix operations. Compared to the traditional discrimination method based on intensity images, the features obtained by this method have a higher separability. Experiments show that the classification accuracy reaches over 90%. This method can be further applied to the recognition and discrimination of other defects in the field of surface defects' detection.