We have studied the in vitro biological activities and mechanism of action of 1,25-dihydroxyvitamin D3 (1,25D3) and four potent 1,25D3 analogues [20-epi-22oxa-24a,26a,27a-tri-homo-1,25(OH)2D3 (KH 1060); 20-epi-1,25(OH)2D3; 1,25(OH)2-16ene-D3; and 1,25(OH)2-16ene-23yne-D3] on proliferation and differentiation of estrogen receptor-negative (MDA-MB-436, BT-20, SK-BR-3, and MDA-MB-231), estrogen receptor-weakly positive (BT474), and estrogen receptor-positive (MCF-7) breast cancer cell lines. Dose-response studies showed that KH 1060 was the most potent analogue, because it was able to induce differentiation in all seven breast cancer cell lines (measured by lipid staining) and to suppress more than 50% clonal proliferation (ED50) at 10(-10) M in all cell lines, except MDA-MB-436 and BT-20. To explore how these compounds mediated antiproliferative actions, their effects on the cell cycle, on expression of bcl-2 and p53, and on apoptosis were assessed. Five of six cell lines have a mutant p53 gene, whereas MCF-7 has wild-type p53. Immunohistochemical staining showed that the p53 protein was predominantly localized in the nucleus in each of the breast cancer cell lines except for MCF-7, which expressed the protein predominantly in the cytoplasm. After incubation with KH 1060 (3 days; 10(-7) M), expression of bcl-2 protein as determined by immunohistochemical localization was markedly decreased in BT-474, MCF-7, and MDA-MB-231; these same cells were profoundly inhibited in their clonal proliferation and arrested in the G0/G1 phase of the cell cycle when cultured with KH 1060. In contrast, BT-20 and MDA-MB-436 cells that were refractory to the antiproliferative effect of KH 1060 (ED50 < 10(-6) M) had no down-regulation of their bcl-2 expression and no cell cycle changes after exposure to KH 1060. MCF-7 showed morphological changes and DNA fragmentation, indicative of apoptosis after 48 h incubation with KH 1060 (10(-6) M), during which time p53 protein accumulated in the nucleus and decreased in the cytoplasm. In contrast, no apoptosis was detected in three other breast lines (MDA-MB-231, SK-BR-3, and BT-474) that had a mutated p53. In conclusion, the data indicate that KH 1060 is an extremely potent 1,25D3 analogue inducing differentiation of all six breast cancer lines and potently inhibiting clonal growth of four of them with concomitant decreased bcl-2 and cell cycle arrest at G0/G1.(ABSTRACT TRUNCATED AT 400 WORDS)