We have proposed and demonstrated a transfer learning (TL)-assisted deep learning network (DNN) for nonlinear distortion compensation in optical side-band PAM-4 modulation and direct-detection transmission. Since there exists partial correlation of nonlinear distortions, we can transfer the parameters of the trained DNN to the target model to speed up remodeling and reduce complexity. We conduct experiments to demonstrate the effectiveness of the proposed scheme in Nyquist PAM-4 transmissions. The required iterations or train size with TL can be less than half of that with retraining without any performance degradation in single-channel transmission. We also extend the proposed scheme into multi-channel transmissions. Since all channels co-propagate on the same fiber, the correlation of nonlinear distortions enables TL to share the parameters among different channels and realize fast remodeling with a better starting rather than a stochastic beginning. The experimental results show that the iterations of TL are one-fourth that of retraining without performance penalty in five-channel transmission.