This paper presents a simple method for compensating the Sagnac phase shift in an interferometric fiber-optic gyroscope (I-FOG) with a piezoelectric modulator. The common advantages of I-FOGs with closed-loop compensation are linearized output characteristics and insensitivity to the light source power, including its time and thermal-induced fluctuations. Whereas closed-loop operation is normally achieved via ramp modulation requiring an electro-optic modulator, all-fiber architectures with a piezoelectric modulator are mostly limited to open loop. Nevertheless, such setups can more conveniently utilize a less expensive single-mode fiber with depolarized light and do not require any custom-made components. The proposed method allows us to combine the advantages of both approaches. Closed-loop compensation is ensured by adding further sinusoidal modulation to the common biasing modulation, such that the Sagnac phase shift is compensated solely at the sampling instants. We describe and experimentally demonstrate the proposed approach, utilizing a test setup to compare our closed-loop solution with open-loop operation. The results denote that the method provides a cost-efficient manner of performance improvement compared to the open-loop I-FOGs based on a piezoelectric modulator.