Maximum likelihood sequence estimation (MLSE) offers effective equalizations for bandwidth-limited optical signal on mitigation towards inter-symbol-interference (ISI) impairment. In this paper, we provide the first comprehensive comparisons and analysis of three post-compensation algorithms on the same modeling platform for high spectral-efficiency (SE) optical systems employing the spectral prefiltering. Those algorithms include 1-tap constant modulus algorithm (CMA) and 3-tap MLSE, regular CMA and digital filter with 2-tap MLSE, and constant multi-modulus algorithm (CMMA) with 2-tap MLSE. Furthermore, a novel and effective approach is proposed and verified for the generation of both hard value and soft value at the output of MLSE in order to be compatible with the implementation of soft-decision forward error correction (SD-FEC) decoding process.