This paper proposes an image segmentation method named iterative region growing using semantics (IRGS), which is characterized by two aspects. First, it uses graduated increased edge penalty (GIEP) functions within the traditional Markov random field (MRF) context model in formulating the objective functions. Second, IRGS uses a region growing technique in searching for the solutions to these objective functions. The proposed IRGS is an improvement over traditional MRF based approaches in that the edge strength information is utilized and a more stable estimation of model parameters is achieved. Moreover, the IRGS method provides the possibility of building a hierarchical representation of the image content, and allows various region features and even domain knowledge to be incorporated in the segmentation process. The algorithm has been successfully tested on several artificial images and synthetic aperture radar (SAR) images.