Aspergillus flavus is the main etiological agent for aflatoxin contamination of crops. Its close relative, A. oryzae, does not produce aflatoxins and has been widely used to produce fermented foods. We compared the phylogeny of A. oryzae isolates and L- and S-type sclerotial isolates of A. flavus using single nucleotide polymorphisms in the omtA gene in the aflatoxin biosynthesis gene cluster and deletions in and distal to the norB-cypA intergenic region as phylogenetic signals. Aflatoxin-producing ability and sclerotial size also were weighted in the analysis. Like A. flavus, the A. oryzae isolates form a polyphyletic assemblage. A. oryzae isolates in one clade strikingly resemble an A. flavus subgroup of atoxigenic L-type isolates. All toxigenic S-type isolates closely resemble another subgroup of atoxigenic L-type isolates. Because atoxigenic S-type isolates are extremely rare, we hypothesize that loss of aflatoxin production in S-type isolates may occur concomitantly with a change to L-type sclerotia. All toxigenic L-type isolates, unlike A. oryzae, have a 1.0 kb deletion in the norB-cypA region. Although A. oryzae isolates, like S-type, have a 1.5 kb deletion in the norB-cypA region, none were cladally related to S-type A. flavus isolates. Our results show that A. flavus populations are genetically diverse. A. oryzae isolates may descend from certain atoxigenic L-type A. flavus isolates.