Drowning in Data Soup: Why Factor Investing Cannot Be Scientific

The Nature of Financial Data

Late last year, Marcos M. López de Prado, a hedge fund manager and professor who has pioneered machine learning in finance and who is currently global head of quantitative research and development at one of the world’s largest sovereign wealth funds, published a monograph called Causal Factor Investing: Can Factor Investing Become Scientific? In it, he decries what he calls “rampant backtest overfitting and incorrect specification choices” in the academic literature about factor investing, discussing “spurious claims” and “causal confusion.” He then “proposes solutions with the potential to transform factor investing into a truly scientific discipline.”

López de Prado is at his best when debunking academic approaches to finance, specifically the pretension of financial/economic academics to a truly scientific approach. Much of his work along these lines is rehashed in this monograph. But here he falls into the same trap as the academics he debunks with the claim that factor investing can be “a truly scientific discipline.” I have my doubts.

The main problem is that the data factor investors work with is anything but scientific. Earnings, free cash flow, and other financial items are interpretations by accountants of what happens in firms that they work for. They follow regulatory standards, but have plenty of discretion, as well as a vested interest in presenting these items in a positive light. Nothing could be less scientific than this data, and this data is the foundation for all factor investing. While certain accounting items may be more “scientific” than others—interest expense, for instance, is quite quantifiable—so much depends upon completely discretionary items like depreciation rates, classification of costs and salaries (as CapEx, SG&A, R&D, other costs of goods, and so on), estimation of intangibles, and the timing of recognition of cash inflows and outflows. Despite regulatory frameworks like GAAP (generally accepted accounting principles, the US standard) and IFRS (International Financial Reporting Standards, the European standard) that aim to standardize reporting, there are very few financial items that are not infected with the discretionary practices of company accountants.

Factor investors take a quantitative approach to variables that are extraordinarily unreliable. What does that make us? Certainly not scientists. But neither are we soothsayers or literary interpreters. We take this extremely unreliable data and use established statistical methods and employ advanced testing techniques to get meaning out of it.

The only comparison I can come up with is parimutuel betting. A good gambler will take unreliable data and, using a probabilistic approach, place bets that have a somewhat decent chance of paying off. Nobody would confuse a gambler with a scientist, even though both may use advanced statistical techniques. People do confuse investors and gamblers, and there’s a good reason for that.

Discretionary Accounting

Here are a few specific examples of the kinds of discretion accountants can make.

• When valuing inventory, GAAP allows for all three methods (first in, first out; last in, first out; and weighted average cost).
• IFRS allows flexibility as to how interest and dividends are categorized in the cash flow statement, permitting listing under either operating or financial cash flow.
• IFRS allows revaluation of a broad range of assets.
• For software services, GAAP allows additional flexibility in revenue recognition.

I definitely do not want to suggest that GAAP and/or IFRS are flawed. Without them, things would be a thousand times worse. There’s a cost to overstandardization: the nuances in company accounts are important for investors to take notice of. That’s why 95% of the S&P 500 report both GAAP and non-GAAP earnings.

The key point is that financial reporting was not designed for broad data analysis but for properly illuminating an individual firm’s financial results. The standardization that GAAP and IFRS impose are important for establishing some consistency, but that consistency is geared toward comparison of individual firms and industries rather than toward scientific or statistical analysis. And comparison of individual firms and industries falls prey to a very different form of terribly unscientific data.

Industry Classification

Most factor investing necessitates intraindustry comparisons. Earnings yield is meaningful, but much more meaningful if you compare it to other companies in the same industry. But how do you classify companies into industries? Take, for example, Kewaunee Scientific (KEQU), which is currently one of my top holdings, both in my hedge fund and in my personal accounts. The company makes specialized furniture primarily for health care facilities. So is it a health care company? That’s what Compustat/S&P says it is. Or is it a furniture company? That’s what FactSet says it is. The problem goes beyond quirky businesses like Kewaunee. It makes a big difference whether Amazon (AMZN) is classified as a discretionary (cyclical) or a staples (non-cyclical) retail firm, or whether Visa (V) is in the financial or the technology sector. Different data providers classify these companies differently at different times (Compustat classifies Amazon as “discretionary” while FactSet classifies it as “noncyclical”; Compustat used to classify Visa in the tech sector, then moved it to the financial sector). And this brings us to yet another source of uncertainty.

Data Providers

For the sake of argument, I created a ranking system on Portfolio123 (a subscription-based tool that helps research and build systematic quantitative investing strategies) with thirty factors that I’ve been using quite heavily lately. I then compared how companies fared on this system depending on whether the data came from Compustat or FactSet.

One of the many companies with jaw-dropping discrepancies is eBay (EBAY). On FactSet it ranks 86 out of 100 while on Compustat it ranks 25.

So let’s take this apart a bit.

• The first thing I notice is an issue with special items. These are non-recurring pretax items such as moving expenses, severance payments, write-offs, write-downs, reserves for litigation, and so on. They should be taken into account when calculating a company’s net income. Compustat has this figure as –$206 million for 2023 while FactSet has it as +$1.75 billion for the same year. (Compustat lists a couple of dozen items that together make up special items; FactSet calculates it as the difference between extraordinary charges and extraordinary credits.) With a net income last year of $2.75 billion, whether you add back $206 million or subtract $1.75 billion makes a huge difference. According to FactSet, eBay is experiencing excellent earnings growth, since last year’s income is adjusted so radically. According to Compustat, its earnings growth is not so great.
• The next thing I notice is a big discrepancy in the balance sheet accruals. This is traceable to the fact that FactSet shows more than $3.1 billion in other investments and advances this quarter and $1.2 billion the same quarter last year, meaning that eBay has substantially increased its long-term receivables. Compustat, on the other hand, gives N/A for both. So for FactSet, eBay’s accruals are fine; for Compustat, they’re terribly high given that eBay’s cash on hand has decreased substantially in the last year. (This problem is lessened if you use annual figures. Compustat doesn’t usually list quarterly figures for this particular line item, which explains the N/As.)
• eBay’s earnings yield, based on current fiscal year estimates, is a little over 8%. How good is that? Well, normally one compares earnings yield to other companies in the same industry. For Compustat, the industry is multiline retail, which puts them in 7th place out of 19 companies (I’m limiting my scope to companies with sufficient liquidity to trade large amounts). For FactSet, the industry is general merchandise retail, which puts them in 10th place out of 32 companies. So eBay scores a little better with FactSet than with Compustat on earnings yield.
• I look at a lot of aspects of a company’s subsector, including its momentum and its average free cash flow yield; I also compare the inventory change of the company (which in this case is zero) to the inventory change of the subsector. For FactSet, eBay is in the food and staples retail subsector of the staples sector, while for Compustat, it’s in the consumer discretionary distribution and retail subsector, which is in the discretionary sector. Those two subsectors are very different indeed. The first has stronger momentum, higher free cash flow yield, and higher inventory growth. These make very substantial differences to the company rankings.
• According to FactSet, eBay’s operating income growth, comparing the most recent quarter to the same quarter a year ago, is 31%, while according to Compustat it’s only 7%. FactSet lists eBay’s current operating income as $658 million while Compustat lists it as $552 million; there are discrepancies between the numbers for the same quarter a year ago too. I’m not going to pronounce judgment on which data provider is correct: they simply standardize operating income (also called EBIT) in different ways.

eBay isn’t really an outlier here. 15% of the companies in Portfolio123’s “easy to trade” universe have a discrepancy of 20 or more between their FactSet and Compustat rankings (on this thirty-factor system), and that’s not even mentioning those companies that are covered by only one of the two data providers.

Using Unscientific Data to Assess Companies

How does one proceed, then, with assessing companies given the craziness of financial data? I have a few suggestions.

1. Don’t ever think that what you’re doing is scientific, objective, or truly solid. Suspect every conclusion you come to.
2. Check your data. If you come across major discrepancies, investigate them. Try to figure out what the company is reporting and what your data providers are doing with those numbers.
3. The more data you use, the less these discrepancies will matter. If you base your judgment of a company on just a handful of metrics, you’re bound to run into serious problems. Use ten different measures of earnings growth or earnings yield rather than only one; use a wide variety of measures of company quality, stability, and so on; calculate a company’s value using intrinsic value methods (preferably more than one) as well as relative value methods; use two or three different data providers rather than only one; and so on.
4. Hedge your bets with diversity. Use several different systems or strategies at once, provided you’re agnostic about which is better.
5. Concentrate on probability rather than absolutes in your thinking about investing and in your approach to systematization.

Working with financial data is never easy, and it’s made much more complex by the amount of data out there. Add to that the fact that the data itself is unreliable, fuzzy, capricious, and mutable and you’re liable to drown in data soup. Taking a cut-and-dry approach to data like this, simplifying your strategy to its essence, treating data as if it is sacrosanct, or ignoring it altogether is bound to backfire. Instead you have to learn to swim in the data soup, because the more of it you absorb, the better a picture you’ll be able to paint of the companies you’re investing in.

I have based my entire investing career on using financial data in novel ways. Ever since late 2015, I have been investing using ranking systems based on financial data, concentrating on safe, solid, boring, and under-the-radar companies, and as a result I have a CAGR of 42% during that period, without a single negative calendar year. I generally have little use for the kinds of factors that are standard in the financial services industry, the kinds of simple and well-trodden factors that are, for example, the basis of Seeking Alpha’s quant rankings. I have found that

• low profit margins can be predictive of earnings growth,
• the ratio of gross profit to total assets can be more meaningful than ROA,
• if a company’s receivables vary a great deal from quarter to quarter that usually spells trouble,
• you can base a value ratio on taxes paid,
• a company whose sales are turning around can be a much better investment than a company whose sales are growing steadily,
• low share turnover decreases market risk,
• dividend yield means nothing without considering the payout ratio,
• a simultaneous increase in both inventory and gross plant is a bad sign,
• operating cash flow had better exceed net income, and
• intrinsic value can be roughly estimated algorithmically.

My entire philosophy of investing is based on taking a look at every stock I buy or bet against from as many different angles as possible.

But I’m a quant at heart. I spend a hundred times more time and energy on my algorithmic systems than I do looking up the specifics of one company’s financials. The discretionary decisions I make have nothing to do with whether or not I buy or sell a particular stock on a particular day, but are all about improving my system and incorporating into it financial factors. Data matters to me a lot, and it scares the daylights out of me that it could be all wrong.

But what does “wrong” really mean when it comes to financial data? Nothing. Financial data is a mess of biased interpretations, but it’s all we have to work with. Approaching it in a holistic, individualized, and well-reasoned way is your best bet if you want to beat the market consistently.