Although it has been qualitatively demonstrated that surface roughness can reduce the thermal conductivity of crystalline Si nanowires (SiNWs), the underlying reasons remain unknown and warrant quantitative studies and analysis. In this work, vapor-liquid-solid (VLS) grown SiNWs were controllably roughened and then thoroughly characterized with transmission electron microscopy to obtain detailed surface profiles. Once the roughness information (root-mean-square, σ, correlation length, L, and power spectra) was extracted from the surface profile of a specific SiNW, the thermal conductivity of the same SiNW was measured. The thermal conductivity correlated well with the power spectra of surface roughness, which varies as a power law in the 1-100 nm length scale range. These results suggest a new realm of phonon scattering from rough interfaces, which restricts phonon transport below the Casimir limit. Insights gained from this study can help develop a more concrete theoretical understanding of phonon-surface roughness interactions as well as aid the design of next generation thermoelectric devices.