The shortest comparison
| Metric | What it averages | Main consequence |
|---|---|---|
| MSE | Squared errors | Large errors receive rapidly increasing weight; units are squared. |
| RMSE | Square root of mean squared error | Returns to outcome units while retaining extra sensitivity to large errors. |
| MAE | Absolute errors | Each unit of error has equal linear weight. |
| MAPE | Absolute percentage errors | Scale-free-looking, but undefined at zero and unstable near zero. |
All four are nonnegative and lower is better for the same target, cases, and evaluation design. Comparing RMSE across different outcome scales or datasets is not meaningful without a defensible normalization.
Read RMSE and MAE in original units
If a model predicting a dollar-valued outcome has MAE = $12, the average absolute miss is $12 on the evaluated sample. RMSE = $20 on the same cases says the squared-error summary corresponds to 20 outcome units and suggests that larger misses have substantial influence.
RMSE is not “the typical error” in the same direct sense as MAE. Because errors are squared before averaging, a few large misses can pull it upward. Willmott and Matsuura argue for MAE as a more natural average-error measure while explaining the different weighting embedded in RMSE.1 The choice should follow the error cost, not a slogan that one metric is universally better.
Why MAPE can fail
MAPE divides each absolute error by the observed value. If an observed value is zero, the term is undefined. Near zero, a small absolute miss can become an enormous percentage. The measure is also asymmetric: equal-sized over- and under-forecasts can receive different percentage penalties.2
Before interpreting a reported MAPE, check how zeros and negative values were handled, whether cases were excluded, and whether the implementation multiplied by 100. A clean percentage does not rescue a denominator that is inappropriate for the data.
What counts as good
Use comparisons that preserve the evaluation question:
- A baseline: Does the model beat predicting a historical mean, median, seasonal naive value, or other relevant reference?
- The same held-out cases: Compare models on identical observations and preprocessing.
- Outcome scale: Relate error to the meaningful range, natural variation, or decision tolerance.
- Error distribution: Report quantiles or a residual plot so a mean does not hide a bad tail.
- Uncertainty: Resampling or repeated validation can show how stable the difference is.
Normalized errors can aid cross-series comparison, but the normalization itself becomes part of the metric. State the denominator and why it is meaningful.
R-squared interpretation addresses variation relative to a reference rather than error in outcome units. AIC versus BIC compares fitted candidate models with penalized likelihood rather than held-out predictive error. Use residual plots to see where the errors arise, and the ResultAtlas provenance guide for why “no universal threshold” is a positive conclusion.
A defensible reporting sentence
“On the same held-out cases, the model’s MAE was 8.4 units and RMSE was 13.1 units, compared with baseline values of 11.7 and 17.9. The larger RMSE relative to MAE prompted inspection of high-error cases.”