In the gear fault diagnosis, the emergence of periodic impulse components in vibration signals is an important symptom of gear failure. However, heavy background noise makes it difficult to extract the weak periodic impulse features. Therefore, the paper presents an impact fault detection method of gearbox by combining variational mode decomposition (VMD) with coupled underdamped stochastic resonance (CUSR) to extract the periodic impulse features. First, the adaptive VMD is presented to decompose the vibration signal into several intrinsic mode functions (IMFs), which can automatically determine the appropriate mode number according to the correlation kurtosis (CK) of decomposition results and extract the sensitive IMF component containing the main fault information. Next, the adaptive CUSR method is developed to analyze the selected sensitive IMF component, and the optimal system parameters are obtained by the genetic algorithm using the CK index as optimization objective function. Finally, the periodic impulse features are extracted by the output signal of CUSR system accurately. Experiments and engineering application verify the effectiveness and superiority of the proposed adaptive VMD-CUSR method for extracting the periodic impulse features in gear fault diagnosis compared to other methods.