The Question That Exposes Weak Quant Models

What Institutional Investors Should Ask Before Allocating to Systematic Strategies

Your due diligence process for quantitative managers likely focuses on performance: backtests, Sharpe ratios, drawdowns, and attribution. It almost certainly does not test whether the variables are structured correctly in relation to the economic forces they are meant to capture.

That gap is not minor. It may be the largest undiagnosed source of risk in systematic strategy evaluation today. This piece gives you one question that closes it. It requires no technical background and can be used in your next manager meeting.

The Pattern

Three allocators at three different institutions described the same scenario to me within a single week. A systematic equity manager added a “quality” overlay to a value strategy. The backtest improved: higher Sharpe ratios, lower drawdowns, cleaner attribution. The allocation is made. Twelve months later, the strategy underperforms the simpler value-only version the allocator replaced.

All three allocators concluded their managers had overfit the model to historical data. But that diagnosis did not fully explain what went wrong.

The quality factor was not an independent variable. It was a consequence of the same forces that drive returns. Including it did not add information. It introduced a distortion that made the backtest look better precisely because it made the model structurally worse.

This is called specification error. López de Prado and Zoonekynd examined 26 widely used Barra factor models in their CFA Institute Research Foundation study and found cases where this type of error flipped the sign of the factor coefficient. In one example, the correct loading on a liquidity factor was +0.08. With the wrong control variable, it became −0.04. The model’s statistical fit improved with the error.

They call this a “factor mirage.” López de Prado later translated these findings for practitioners in an Enterprising Investor blog post.

Where Current Frameworks Stop Short

The CFA community has produced strong tools for quant evaluation. Simonian’s screening framework asks whether factors have economic intuition, whether evidence is robust across subsamples, and how model changes are governed. His question about risk controls gets at whether a strategy delivers what it promises. These are the right instincts.

But even the best existing frameworks focus on what a model does and how it was built. They do not ask why the variables are structured the way they are. Industry-standard due diligence questionnaires (DDQs) ask which factors a manager uses and how they define them. They do not ask why those variables and others deliberately excluded. That gap is where specification error hides.

One Question That Changes the Conversation

“How did you decide which variables to include in your model, and which did you deliberately exclude?”

The value of the question lies in what it reveals. You are not asking for a list of variables. You are asking whether the inclusion and exclusion decisions were grounded in economic reasoning rather than statistical fit alone.

In my conversations with both allocators and managers, the responses fall into three distinct categories.

A strong answer: The manager explains the economic mechanism behind each variable’s inclusion. Crucially, they discuss variables they excluded and why, showing that specification was a deliberate design choice. They distinguish between variables that drive their target factor and variables that result from it. The strongest managers trace a chain of economic causality: how macro forces project onto stock-level signals, and why the model reflects those causal chains rather than mining for correlations.

A standard answer: The manager cites statistical criteria: information ratio, R-squared improvement, significance tests. This is current industry practice. It is not wrong, but it is incomplete. Statistical fit alone cannot distinguish between a variable that belongs in the model and one that introduces distortion while improving fit metrics. This is exactly the trap in the opening story.

A concerning answer takes one of two forms: “We use all available variables and let the model select” signals structural vulnerability to factor mirages. On the other hand, “Our variable selection process is proprietary” may reflect legitimate IP protection. But a manager who cannot explain the reasoning behind their specification, even without disclosing specific variables, cannot demonstrate that the reasoning exists.

Why This Matters Now

Total portfolio approach (TPA) is centralizing factor transparency.  The largest pension funds now require every mandate to be expressed in a common factor language. When your entire portfolio must be understandable at the factor level, the causal validity of those models directly affects capital allocation and risk budgeting.

Factor returns are decaying. McLean and Pontiff (2016) document a 50-58% decline in factor returns after academic publication. As more capital chases published factors, the difference between a well-specified model and a mirage becomes the difference between residual alpha and expensive noise.

The most sophisticated allocators already act on this. ADIA Lab has committed dedicated funding, a $100,000 annual research award, and a global challenge that attracted nearly 2,000 researchers to causal inference in investments.

When the allocator managing a trillion dollars invests in solving this problem, it is worth one question in your next meeting.

CFA Institute’s Standard V(A) requires members to have “a reasonable and adequate basis” for investment recommendations, including understanding the assumptions and limitations of quantitative models. This question — “How did you decide which variables to include in your model, and which did you deliberately exclude?”  — helps meet that standard.

Before Your Next Meeting

Ask one question about why the variables are there and why others are not. The quality of the answer will tell you more about the structural soundness of a quant process than any backtest.

This is the first of four specification risk dimensions I examine in a broader framework covering how managers diagnose performance failures, whether they can explain specific trades, and how sensitive their models are to structural changes. But specification comes first, because if the variables are wrong, nothing downstream can fix it.

This is one dimension of a broader specification risk framework, alongside how managers diagnose performance failures, explain specific trades, and respond to structural change.