Avian influenza viruses (AIVs) periodically cross species barriers and infect humans. The likelihood that an AIV will evolve mammalian transmissibility depends on acquiring and selecting mutations during spillover. We analyze deep sequencing data from infected humans and ducks in Cambodia to examine H5N1 evolution during spillover. Viral populations in both species are predominated by low-frequency (<10%) variation shaped by purifying selection and genetic drift. Viruses from humans contain some human-adapting mutations (PB2 E627K, HA A150V, and HA Q238L), but these mutations remain low-frequency. Within-host variants are not enriched along phylogenetic branches leading to human infections. Our data show that H5N1 viruses generate putative human-adapting mutations during natural spillover infection. However, short infections, randomness, and purifying selection limit the evolutionary capacity of H5N1 viruses within-host. Applying evolutionary methods to sequence data, we reveal a detailed view of H5N1 adaptive potential, and develop a foundation for studying host-adaptation in other zoonotic viruses.