Controlling defect states in a buffer layer for organic photo devices is one of the vital factors which have great influence on the device performance. Defect states in silicon oxynitride (SiOxNy) buffer layer for organic photo devices can be controlled by introducing appropriate dopant materials. We performed ab initio simulations to identify the effect on doping SiOxNy with carbon (C), boron (B), and phosphorous (P) atoms. The results unveil that hole defects in the SiOxNy layer diminish with the phosphorous doping. Based on the simulation results, we fabricate the small molecule organic photodetector (OPD) including the phosphorous-doped SiOxNy buffer layer and the active film of blended naphthalene-based donor and C60 acceptor molecules, which shows excellent enhancement in the external quantum efficiency (EQE). The results of our charge-based deep level transient spectroscopy (Q-DLTS) measurements confirmed that the EQE enhancement originates from the decrease of the hole traps induced by the reduced hole defects. The method of controlling the defect states in SiOxNy buffer layers by the doping can be used to improve the performance in various organic photo devices.