Diabetes is a major medical problem that imperils public health. Over two thousand years ago, Traditional Chinese Medicine (TCM) called diabetes-related symptoms "Xiaoke" disease. In ancient China, TCM and Chinese herbal medicines were used widely in treating Xiaoke and abundant experience has been accumulated. This article discusses the TCM theory on diabetes and its achievements in the prevention and treatment of diabetes in the past. Using Chinese herbal medicine, recent progress in diabetes therapeutics, including data from clinical trials, are presented. Mechanistic studies from basic research are discussed. Yin-yang balance and a holistic approach of TCM may complement diabetes treatment in Western medicine. With continuous efforts, TCM could play a more important role in fighting this disease.