Localized prostate cancer (confined to the gland) generally is considered curable, with nearly a 100% 5-year-survival rate. When the tumor escapes the prostate capsule, leading to metastasis, there is a poorer prognosis and higher mortality rate, with 5-year survival dropping to less than 30%. A major research question has been to understand the transition from indolent (low risk) disease to aggressive (high risk) disease. In this chapter, we provide details of the changing tumor microenvironments during prostate cancer invasion and their role in the progression and metastasis of lethal prostate cancer. Four microenvironments covered here include the muscle stroma, perineural invasion, hypoxia, and the role of microvesicles in altering the extracellular matrix environment. The adaptability of prostate cancer to these varied microenvironments and the cues for phenotypic changes are currently understudied areas. Model systems for understanding smooth muscle invasion both in vitro and in vivo are highlighted. Invasive human needle biopsy tissue and mouse xenograft tumors both contain smooth muscle invasion. In combination, the models can be used in an iterative process to validate molecular events for smooth muscle invasion in human tissue. Understanding the complex and interacting microenvironments in the prostate holds the key to early detection of high-risk disease and preventing tumor invasion through escape from the prostate capsule.