Estimation of a normal process variance from measurements with large round-off errors

Measurements are sometimes affected by excessively large round-off errors. Small rounding-off may safely be ignored for purposes of statistical inference however large rounding-off may have an effect. The importance of the round-off (δ) is determined by the ratio between the standard deviation s and the instrument's scale step h, δ = σ/h. In this study, the authors estimate σ when δ is small (δ < 0.5) using a variant of the method of moments (MoM). The MoM estimators are compared with the maximum-likelihood estimators (MLE), using simulation. The authors find that the MoM can improve the estimation in terms of mean-square error and bias, especially under circumstances where the MLE method is not accurate or cannot provide a solution.