What Newey-West does to a backtest t-statistic: 18 months became two

A Newey-West standard error accounts for dependence in the series behind a mean. On my engine's 12-month panel, 18 monthly readings with an unadjusted t-statistic of negative 18.8 had an estimated effective sample size near two. A result that had looked decisive failed the engine's evidence floor.

The reading came out of the diagnostic harness I wrote for my factor scoring engine. The harness scores a universe of stocks at each month-end, waits for the forward return over some horizon, and computes a cross-sectional information coefficient per date: the rank correlation between score and what happened next. Average the per-date ICs, divide by the standard error of that average, and you have a t-statistic. The 12-month quality reading came back at negative 0.57 across 18 dates with an unadjusted t of negative 18.8. A quality factor that predicts returns backward is a factor built wrong, and that is the fault I went looking for first.

Why 18 readings are not 18 facts#

The naive standard error is the standard deviation of the per-date ICs, 0.13 here, divided by the square root of 18. That formula assumes the readings are independent draws. A 12-month forward return from a January month-end snapshot covers February through the following January; the next snapshot's window shares eleven of those twelve months. That is about 92 percent overlap in the return windows, though it does not by itself determine the correlation between IC readings. The scores are sticky as well, since a company's quality metrics do not reshuffle month to month, and factor performance can persist across several quarters.

Those are reasons to examine dependence rather than count every monthly reading as an independent draw. Positive autocovariances can make the naive standard error too small and the absolute t-statistic too large. The amount of window overlap alone does not tell me the size of that error; I need the IC series and its estimated autocovariances.

The Newey and West (1987) correction replaces the sample variance with a long-run variance. Take the variance of the series, then add twice the autocovariance at lag one, lag two, up to lag L, each weighted by a Bartlett factor of one minus the lag over L plus 1, so nearby lags count fully and distant ones fade out. The helper in my harness is fourteen lines: the mean, the deviations, the lag-zero variance, a loop that accumulates weighted autocovariances into the long-run variance, then the mean divided by the square root of that variance over n.

I now print a variance-ratio diagnostic on every cell: effective n, calculated as n times the naive variance over the estimated long-run variance, capped at n. For the 12-month panel it came out near two. That estimate fails my evidence floor; it does not turn the series into two literal independent observations or establish an exact reference distribution. A Student-t distribution with one degree of freedom is defined, but assigning that degree of freedom from this diagnostic would need a separate justification. With only 18 dependent readings, a self-selected panel, and scores from two engine versions, I could not treat the result as a reliable basis for reversing the factor.

The lag is a decision, not a default#

Nothing in the formula tells you L. Too small and the correction misses the overlap; too large and the estimate of the long-run variance gets noisy in its own right. The engine's comparator derives L from the horizon: the number of series steps two snapshot dates can be apart and still share a forward window, which is the horizon divided by the median spacing between dates, rounded up, minus one. On month-end dates that is lag 11 for a 12-month horizon and lag 2 for a 3-month one, clamped to n minus 1. When it does not know the horizon it falls back to the automatic Bartlett rule, the floor of 4 times n over 100 to the two-ninths power. A Bartlett estimator truncated at L ignores every lag above L, so a lag chosen without looking at the horizon is a lag that can miss the overlap entirely.

My standalone diagnostic scripts use a different choice for monthly net-of-cost portfolio returns: fixed lag 3, motivated by quarterly score refreshes. The IC series use the comparator's horizon rule. The rule of 0.75 times T to the one-third power, rounded down, gives lag 3 at 64 observations and lag 1 at 18. I had chosen the larger lag when conventions disagreed, but that choice is not automatically conservative. The Bartlett calculation adds signed autocovariances, so adding lags can lower the estimated variance. Sensitivity to justified lag choices belongs beside the result, especially for this short panel; no such sensitivity result is established here.

I told this story with the wrong numbers once#

My June post on backtesting without fooling yourself told a version of it with a different pair: a factor whose naive t was negative four and whose Newey-West t was negative 1.6. The mechanism in that paragraph is right and I would write it the same way again. The numbers came from a point-in-time scoring harness that I later found had two defects, a market-cap field that leaked today's value into historical dates and a renamed API field that left one factor reading a constant. Every negative quality reading that harness produced is void, including the one I quoted. The correction ratio was real; the signal it was correcting was an artifact. I would rather say so here than let the older paragraph stand as the reference.

The clean re-measurement, on a point-in-time mid-cap corpus with the leaks fixed, put the same factor's 6-month IC at plus 0.046 with a Newey-West t of plus 3.5 over 2015 to 2021. Three caveats travel with that number in every record the engine writes. It is an upper bound, because the corpus has no in-window delistings. It is an information coefficient, not a net-of-cost portfolio return, and the engine has not shown that any factor survives transaction costs, which is the bar that decides anything. And in the 2024 to 2025 sub-window, the one with the least survivorship contamination, the same factor's 6-month IC is plus 0.008 with a Newey-West t of plus 0.39, indistinguishable from zero. Two readings compete, and both are open.

The gate that makes the correction unskippable#

A correction you remember to apply is a correction you will forget under deadline. After the 12-month episode I moved the effective-n calculation from a diagnostic printout into the engine's decision protocol. Every IC and t-statistic the engine records now carries an effective-n field. No factor weight is promoted unless the holdout cell that validated it has an effective n of at least 8 at the headline horizon, and none of the revision triggers, the checks that can propose a change to how a factor is scored, fires unless every horizon it names clears the same floor. Below the floor the cell reports itself as immature and not actionable, the same state the engine uses for a signal with no forward record at all. The floor of 8 is provisional and I expect to move it. The rule that a cell below the floor cannot change a weight is not.

The 12-month cell was still immature when I wrote this. One non-overlapping annual window per year is a useful planning approximation, but estimated effective n depends on the observed series and can change as data accumulate. Shorter horizons mature sooner only if their dependence and other evidence checks permit it. Choosing a horizon affects how long I may need to wait before a result can justify a weight change.