For the most studied nanocrystal system in the literature, experimental results in this paper revealed that formation of either zinc blende or wurtzite CdSe nanocrystals was dominated by the ligand-surface interaction, instead of the interior structure difference. This conclusion was considered to be reasonable, given the very small energy difference between wurtzite and zinc blende CdSe (only 1.4 meV per CdSe unit and about 1000 times smaller than the energy of a single cadmium-ligand bond). Cadmium carboxylate ligands in the form of cadmium fatty acid salts promoted formation of the zinc blende structure. Conversely, cadmium phosphonate ligands with a long hydrocarbon chain favored the formation of the wurtzite structure. The effects of either cadmium carboxylate or cadmium phosphonate ligands were found to play a determining role during both nucleation and growth. Different from our expectation, fatty amine was found to be only a secondary factor for crystal structure determination. With an appropriate choice of capping ligands, it was possible to achieve precise control of the crystal structure of the CdSe nanocrystals in both nucleation and growth for either the zinc blende or wurtzite structure.