Smartphone fluorescence microscopy has various applications in point-of-care (POC) testing and diagnostics, ranging from e.g., quantification of immunoassays, detection of microorganisms, to sensing of viruses. An important need in smartphone-based microscopy and sensing techniques is to improve the detection sensitivity to enable quantification of extremely low concentrations of target molecules. Here, we demonstrate a general strategy to enhance the detection sensitivity of a smartphone-based fluorescence microscope by using surface-enhanced fluorescence (SEF) created by a thin metal-film. In this plasmonic design, the samples are placed on a silver-coated glass slide with a thin spacer, and excited by a laser-diode from the backside through a glass hemisphere, generating surface plasmon polaritons. We optimized this mobile SEF system by tuning the metal-film thickness, spacer distance, excitation angle and polarization, and achieved ~10-fold enhancement in fluorescence intensity compared to a bare glass substrate, which enabled us to image single fluorescent particles as small as 50 nm in diameter and single quantum-dots. Furthermore, we quantified the detection limit of this platform by using DNA origami-based brightness standards, demonstrating that ~80 fluorophores per diffraction-limited spot can be readily detected by our mobile microscope, which opens up new opportunities for POC diagnostics and sensing applications in resource-limited-settings.