Lack of biocompatible and effective delivery carriers is a significant shortcoming for siRNA-mediated cancer therapy. To overcome these limitations, selenium nanoparticles (SeNPs) have been proposed for siRNA transfection vehicles. In this study, we synthesized novel RGDfC peptide modified selenium nanoparticles (RGDfC-SeNPs) as a gene vehicle, which was expected to improve the tumor-targeted delivery activity. RGDfC-SeNPs were compacted with siRNAs (anti-Oct4) by electrostatic interaction, which was capable of protecting siRNA from degradation. RGDfC-SeNPs exhibited excellent ability to deliver siRNA into HepG2 cells. siRNA transfection assay showed that RGDfC-SeNPs presented a higher gene silencing efficacy than conventional lipofectamine 2000. The cytotoxicity of RGDfC-SeNPs/siRNA on normal cells was lower than that on tumor cells, indicating that RGDfC-SeNPs/siRNA exhibited selectivity between normal and cancer cells. Additionally, Oct4 knockdown mediated by the selenium nanoparticle transfection arrested HepG2 cells mainly at the G2/M phase and significantly induced HepG2 cell apoptosis. Western blotting results showed that RGDfC-SeNPs/siRNA might trigger Wnt/β-catenin signaling, and further activate a BCL-2 apoptosis-related signaling pathway to advance HepG2 cell apoptosis. In vivo biodistribution experiments indicated that RGDfC-SeNPs/siRNA nanoparticles were specifically targeted to the HepG2 tumors. Most importantly, RGDfC-SeNPs/siRNA inhibited tumor growth significantly and induced HepG2 cell apoptosis via silencing the Oct4 gene. In addition, the results of H&E staining demonstrated that RGDfC-SeNPs/siRNA had negligible toxicity on the major organs of mice. In a word, this study provides a novel strategy for the design of biocompatible and effective siRNA delivery vehicles in cancer therapy.