Selective autophagy is mediated by cargo receptors that link the cargo to the isolation membrane via interactions with Atg8 proteins. Atg8 proteins are localized to the membrane in an ubiquitin-like conjugation reaction, but how this conjugation is coupled to the presence of the cargo is unclear. Here we show that the S. cerevisiae Atg19, Atg34 and the human p62, Optineurin and NDP52 cargo receptors interact with the E3-like enzyme Atg12~Atg5-Atg16, which stimulates Atg8 conjugation. The interaction of Atg19 with the Atg12~Atg5-Atg16 complex is mediated by its Atg8-interacting motifs (AIMs). We identify the AIM-binding sites in the Atg5 subunit and mutation of these sites impairs selective autophagy. In a reconstituted system the recruitment of the E3 to the prApe1 cargo is sufficient to drive accumulation of conjugated Atg8 at the cargo. The interaction of the Atg12~Atg5-Atg16 complex and Atg8 with Atg19 is mutually exclusive, which may confer directionality to the system.