Are Zillow’s losses an AI failure?

About regimes, optimization and compasses

In my mid 20’s I was facing an unusual opportunity for someone my age. There it was, the detailed plan for the next two years on how I intended to manage a huge amount of other people’s money and develop some nice buildings in the process. All fit to the budget and timeline I myself had estimated. All looked great. I felt like a philharmonic orchestra conductor before a well prepared concert.

The opportunity was unusual as that decision level is not where you usually are without much more experience, and for a good reason. But those were the crazy real-estate bubble years, pre-2007, add a couple of random events and there I was, with just my Architecture studies and a “Very Large Crude Oil Tanker” to sail home, so to speak. Nervous? Not really. That much I trusted my capacity. Even today, I smile remembering it.

But even if overconfident I knew some advice might be good. The problem was there were not that many people I could access to with a proven successful track record in the field. In fact there was only one. Near his 60’s, the man had fought all battles. After my humble approach he agreed to have a look, asked a couple of tricky questions, seemed satisfied with the answers. I felt relieved, but then came the sentence:

“But this is not what will happen, you better keep that very present”

• “ Uh, What do you mean, is it about the estimated timeline? ”
• “ Nah, the timeline is reasonable. ”
• “ So, is it about the budget, is anyone here using mistaken numbers or, worse, deceiving info? ”
• “ Nope, everyone is probably reporting according to their beliefs ” I still was not getting it.
• “ But this will just not happen. It never does. You are not planning buying a yogurt in the corner shop. In two years anything can and will happen. The companies, the laws, the project scope, hidden constraints… it’s all dynamic. Your plan is an incomplete draft. ”

You need a better draft with an additional layer, your layer. And then you will have a compass and nothing more…

• … a compass to locate North this next 2 years. It might seem a compass is not much but there’s no more you can aim to at this scale. It will put you ahead of most”.

The conversation lasted more, an instance of successful “human transfer learning” that generalized well to many situations in the following decade.

Enough about past lifes. How does this little story fit with Zillow’s problem?

As you probably know, ending 2021 Zillow has reported huge losses and cut 25% of their workforce in what is being published as a “Machine Learning” failure. But I don’t think algorithms are to blame here.

The sad truth is that some problems in Machine Learning applied to markets are far from solved, namely regime changes (radical concept shift) and adversarial activity. In its current state Machine Learning can be seen as optimization and interpolation. Can you see the problem of this tool for dealing with price time series?

In markets sometimes “ cheap ” (or any other feature) can mean “ it’s a bargain, buy now ” but also in a different regime “ cheap ” can mean exactly the opposite, “ tomorrow will be cheaper sell asap! ”. If regime is decided by latent factors then no Transformer of LSTM will serve you. Modeling will just optimize whatever objective you define for the training dataset.

In other words, sometimes markets are white and other times are black but unless you can tell which one applies Machine Learning will just either bet blindly on one color or more frequently optimize a gray for the dataset.

What about online learning? Won’t that be better for dealing with regime changes? Well… maybe. But if features dont contain information about regime the most you can hope for is to minimize the reaction lag based on recent observations’ loss structure. But optimality will again depend on the training dataset and the regime change structure it contains so you better got the correct data!

My point here is that as this problems are not currently solved you can not rely on models to behave well under market regime changes.

We shouldn’t call a “Machine Learning failure” what is a risk management failure.

To me Zillow’s losses say not so much about “Artificial Intelligence” as about the difficulty of risk management when you are a human.