Rice, which is the staple food in East Asia, is a source of Selenium (Se) and Cadmium (Cd). The distribution of Se and Cd in soil-rice system is significant to human nutrition and public health. This study is to explore the distribution of Se and Cd in arable land soils and their distribution in polished rice and stalks of Se-rich area. A total of 63 soil samples and 126 rice samples (63 groups of rice grains and stalk samples) were collected from West Hainan Island to determine Se and Cd concentrations. The results suggested the concentration of Se in soil was higher than average level in China, and Cd content was lower than the agricultural land-use threshold of China. The distribution of Se and Cd in arable land soil was primarily determined by diagenesis and mineralization. Se and Cd were more inclined to accumulate in stalks than rice grains, and the contents in polished rice were correlated with that in stalk. Acidification of arable land soil will threaten human nutrition and health for the bioaccumulation factor of Se in polished rice decreased significantly with the decrease of soil pH, while that of Cd in polished rice increased significantly. Therefore, application of lime or alkaline fertilizers in arable land soil of Se-rich area can promote the accumulation of Se in polished rice but reduced the intake of Cd in rice crops.