In Defense of Zillow’s Besieged Data Scientists

Timothy Chan
Wed Nov 17 2021
FORECASTING ZILLOW ALGORITHMS DATA-SCIENTIST REAL-ESTATE

Opinion

Models Must Be Wielded With Care

One of the many Zillow jokes floating around the internet

Zillow’s instant Buying (aka Zillow Offers) business spectacularly imploded this month. After 3.5 yrs, Zillow made the tough decision to shutter their Zillow Offers business. The final result[1]:

  • 25% workforce reduction
  • 7,000 unsold homes bought for $2.8B (with an expected $304M write-down in Q4).
  • -30% stock drop
  • -$245M in losses for Q3

The scale of this disaster is impressive. Misjudgingly, most of the blame has been pinned on the Data Scientists; those “book-smart know-it-alls” that blindly create and trust their models /s. It’s lazy to mock ZEstimate and the idea that Zillow trusted it with with billions of dollars[2]. It’s also easy to mock the science of forecasting and the notion that Zillow’s data scientists simply used “pip install prophet” (profit) to get a production model[3].

All of this is wrong. Zillow Offers and iBuying is an intrinsically risky business that was recklessly executed by Zillow’s business leaders who wanted to take over the real estate market in just a few years. As evidenced by Zillow’s competitors, Opendoor and Offerpad, iBuying does work! But it needs patience, experience, and a healthy respect for risk. Instead Zillow’s business leaders dove head-first into the shallow-end.

What is iBuying

iBuying is an emerging and somewhat unproven industry. Opendoor (2014) and Offerpad (2015) have been at it for years with growing success. The concept is simple: Allow people to sell their home, instantly at a slight discount while saving them the hassle of a tedious, painful, and risky-selling process. Anyone who has bought or sold a home has had the same realization: The home buying process badly needs modernation and innovation. Even in 2021, it’s still largely a handshake and paper business.

iBuying at its core is not about market speculation nor reality TV-style overnight renovations. Zillow themselves spent less than 2% of their operating costs on renovations (not far from the cost of a paint job). Nor is this an automated business; Zillow sent home inspectors and conducted in-person assessments for each purchase.

Here’s the rub: iBuying has exceptionally slim profit margins. On the $1.2B in homes Zillow sold in Q3, they only made $21.5M in profit… a measly 1.8%. Competitors faired much better, between 7–9% margins during the same period. It’s no coincidence that these are close to total real estate agent’s commissions. And for Zillow, a multibillion dollar tech company, these razor-thin margins are only attractive if it could scale this to a $20–100B business.

Zillow believed iBuying was the future of real estate, and they were going to dominate. They expected this to be a $20B business in just a few years, and needed to aggressively expand in all markets, nationwide. While competitors continued to take a methodical, and cautious approach, Zillow’s executives took on all the risks.

The Data Science Challenge

To be successful, Zillow needed hyper accurate forecasting models which could predict a home’s sale price in the next 3–6 months, within +/- 3% [4]. An overestimate could wipe out profits, or worse, result in >$100M losses. An underestimate would result in rejected lowball offers and cripple Zillow’s ability to purchase homes (ie. low buyer conversion rates). While machine learning-based predictive models are standard practice in tech, housing is also subjected to significant and unpredictable macroeconomic trends. A forecasting rule of thumb says to predict X months into the future, you need 2X months of historical data. Yet that rule only works if your historical data is representative of the future. The ongoing global pandemic, rapid inflation, shifts in consumer behaviors and major swings in home prices ensure that this is not possible: We’re in unprecedented territory.

Image used with permission from Lighter Side of Real Estate

Aggressive Growth and Business Goals Killed Zillow Offers

To aggressively scale the Zillow Offers business, Zillow executives intentionally adjusted their algorithm estimates upwards, which accomplished the goal of increasing buying conversion rates but at higher offer prices [5]. Zillow Offers, coming off a terrific Q2 with 15% gross margins thanks to generous price appreciations was feeling pretty confident and continued to expand. Unfortunately, the market in Q3 reversed and instead of +12% growth, the housing market saw -5–7% drops, resulting in $300M in losses and an expected $245M in write-downs. Is this a failure in algorithms? No, Opendoor contrastingly reported a successful Q3 with $170M in profits and 7.3% in gross margins. As Zillow exits the iBuying business, Opendoor will hold the course and continue to refine their models while treating volatility with great respect. The failure lies not in the models, but those who wield them. Models are not omnipotent, and those who choose to base a business on them need to understand their nuance and limitations.

References

  1. Zillow Inc. (2021). Q3 Financial Report. https://s24.q4cdn.com/723050407/files/doc_financials/2021/q3/Zillow-Group-Q3'21-Shareholder-Letter.pdf
  2. https://twitter.com/ryxcommar/status/1456079677957804038
  3. W.D. “Zillow, Prophet, Time Series, & Prices.” 6 Nov 2021. https://ryxcommar.com/2021/11/06/zillow-prophet-time-series-and-prices/
  4. Vashishta, Vin. “Zillow Just Gave Us A Look at Machine Learning’s Future.” 4 Nov 2021. https://vinvashishta.substack.com/p/zillow-just-gave-us-a-look-at-machine
  5. Clark, Patrick, Natarajan, Sridhar, Perlberg, Heather. “The decision came after the company tweaked the algorithms that power the business to make higher offers, leaving it with a bevy of winning bids just as home-price appreciation cooled off a bit.” — Zillow Seeks to Sell 7,000 Homes for $2.8 Billion After Flipping Halt, Bloomberg. 1 Nov 2021. https://www.bloomberg.com/news/articles/2021-11-01/zillow-selling-7-000-homes-for-2-8-billion-after-flipping-halt
  6. Opendoor Technologies Inc. (2021) Opendoor announces third quarter 2021 financial results. https://investor.opendoor.com/news-releases/news-release-details/opendoor-announces-third-quarter-2021-financial-results

Special thanks to Mike from Lighter Side of Real Estate for the Bob Ross meme.


Try Statsig Today

Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.

MORE POSTS

Recently published

My Summer as a Statsig Intern

RIA RAJAN

This summer I had the pleasure of joining Statsig as their first ever product design intern. This was my first college internship, and I was so excited to get some design experience. I had just finished my freshman year in college and was still working on...

Read more

Long-live the 95% Confidence Interval

TIMOTHY CHAN

The 95% confidence interval currently dominates online and scientific experimentation; it always has. Yet it’s validity and usefulness is often questioned. It’s called too conservative by some [1], and too permissive by others. It’s deemed arbitrary...

Read more

Realtime Product Observability with Apache Druid

JASON WANG

Statsig’s Journey with Druid This is the text version of the story that we shared at Druid Summit Seattle 2022. Every feature we build at Statsig serves a common goal — to help you better know about your product, and empower you to make good decisions for...

Read more

Quant vs. Qual

MARGARET-ANN SEGER

💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...

Read more

The Importance of Default Values

TORE

Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...

Read more
ANNOUNCEMENT

CUPED on Statsig

CRAIG

Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...

Read more

We use cookies to ensure you get the best experience on our website.

Privacy Policy