Turbocharger and negative valve overlap (NVO) strategy are widely used among advanced combustion modes for internal combustion engines. In order to achieve well emission performance, the NVO can be as large as 100 crank angle (CA) degrees, such that the residual gas fraction can be up to 40%. With such amount of residual gas in the cylinder, the trapped unburned fuel is not trivial. It has a significant impact on the combustion process. However, the trapped unburned fuel mass is hard to be measured directly. In this paper, a novel method based on the signals of oxygen fraction is proposed to estimate it. By analyzing the combustion process, dynamic equations for the intake/exhaust manifolds and in-cylinder oxygen fractions, as well as actual fuel mass in the cylinder are constructed. A smooth variable structure filter (SVSF) was designed to estimate oxygen fractions and further the trapped unburned fuel. As a comparison, Kalman filter (KF) and linear matrix inequality (LMI) based linear parameter-varying (LPV) filter were also applied. Robustness properties of the three observers are analyzed based on the theory of input-to-state (ISS) stability. The proposed models and methods and theoretical analysis are validated and compared through a set of simulations in high-fidelity GT-Power environment. The simulation results match well with theoretical analysis that the SVSF has good properties of strong robustness (with a root mean square error (RMSE) of 0.24, comparing with 0.4 of LPV filter and 0.49 of KF, for the unburned fuel estimation).