We generated a single chain Fv fragment of an antibody (scFv) with a binding affinity of about 5 pm to a short peptide by applying rigorous directed evolution. Starting from a high affinity peptide binder, originally obtained by ribosome display from a murine library, we generated libraries of mutants with error-prone PCR and DNA shuffling and applied off-rate selection by using ribosome display. Crystallographic analysis of the scFv in its antigen-bound and free state showed that only few mutations, which do not make direct contact to the antigen, lead to a 500-fold affinity improvement over its potential germ line precursor. These results suggest that the affinity optimization of very high affinity binders is achieved by modulating existing interactions via subtle changes in the framework rather than by introducing new contacts. Off-rate selection in combination with ribosome display can evolve binders to the low picomolar affinity range even for peptide targets.