Background Various cardiovascular risk factors are thought to modify atherosclerosis in a similar fashion (ie, by increasing the magnitude of coronary artery disease [CAD]). However, coronary CT angiography allows precision phenotyping of plaque characteristics through use of radiomics. Purpose To assess whether different cardiovascular risk factors have distinctive contributions to the changes in plaque morphologic features over time. Materials and Methods Individuals with or without HIV infection and cocaine use and without cardiovascular symptoms underwent coronary CT angiography between May 2004 and August 2015. In the current HIPAA-compliant study, the effects of cocaine use, HIV infection, and atherosclerotic cardiovascular disease (ASCVD) risk on the temporal changes (mean ± standard deviation, 4.0 years ± 2.3 between CT angiographic examinations) in CAD structure were analyzed by using radiomic analysis. The changes in radiomic features were analyzed by using linear mixed models, with correction for factors that may change plaque structure: high-sensitivity C-reactive protein level, statin use, positive family history of CAD, and total plaque volume to account for any potential intrinsic correlation between volume and morphologic features. Clusters among significant radiomic features were identified by using hierarchical clustering. Bonferroni-corrected P values less than .00004 (.05 divided by 1276) were considered to indicate significant differences. Results Of 1429 participants, 300 with CAD confirmed at coronary CT angiography were randomly selected (mean age, 48 years ± 7; 210 men, 226 people infected with HIV, 174 people who use cocaine) and 1276 radiomic features were quantified for each plaque. Cocaine use was significantly associated with 23.7% (303 of 1276) of the radiomic features, HIV infection was significantly associated with 1.3% (17 of 1276), and elevated ASCVD risk was significantly associated with 8.2% (104 of 1276) (P < .00004 for all). Parameters associated with elevated ASCVD risk or cocaine use and HIV infection did not overlap. There were 13 clusters among the 409 parameters, eight of which were affected only by cocaine use and three of which were affected only by ASCVD risk. Conclusion Radiomics-based precision phenotyping indicated that conventional risk factors, cocaine use, and HIV infection each had different effects on CT angiographic morphologic changes in coronary atherosclerosis over 4 years. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Schoepf and Emrich in this issue.