The endolysosomal system is a highly dynamic network of membranes for degradation and recycling. During endosomal maturation, cargo molecules destined for lysosomal degradation are progressively concentrated through continuous rounds of fusion and fission reactions concomitant with inbound and outbound membrane fluxes. Of the cargo molecules delivered to endosomes, about two-thirds are rescued from degradation and recycled for reuse. This balance between degradation and recycling is essential to preserve the proteostatic plasticity of the cell under variable physiological demands. Cargo retrieval from endosomes involves several sorting complexes with stable core compositions that associate with multidomain regulatory proteins, consequently displaying complex interaction networks. The vacuolar protein sorting 29 (VPS29) has emerged as a central scaffold that coordinates the physical assembly of retrieval complexes with regulatory components in what appears to be an elegant solution for regulating distinct retrieval stations. This review summarizes the VPS29-binding partners and its integration into retrieval complexes for endosomal sorting and trafficking.