Solving the One Million Dollar Problem: How to Keep Machine Learning Models Accurate Over Time

The one million dollar problem:

How do you keep a Machine Learning model returning accurate predictions over time?

Companies spend absurd amounts of money yearly to solve this, yet many have no idea where to start.

Here are three examples:

— Santiago (@svpino) April 12, 2023

A model that uses stock prices to make predictions needs updates every second as prices changes.

Netflix's recommendations change as frequently as you and the people around you watch more movies.

Amazon's sales predictions change when customers spend money on the site.

— Santiago (@svpino) April 12, 2023

These problems have something in common:

You have a billion samples but can't use them all for training your model. Instead, you need a slice of the data.

Most people will tell you to get a random slice, but that doesn't work.

— Santiago (@svpino) April 12, 2023

The solution:

You need to "condense" the data while preserving its information.

In other words, use a slice of the data that preserves the original distribution.

But this is easier said than done.

— Santiago (@svpino) April 12, 2023

Here is a paper that proposes a solution using "Coresets": https://t.co/2zjo8P8MHA

Fortunately, you don't have to do this manually!

I work with the team @akridata. Their platform is awesome, and if you work with a dataset of images, you need to check it out!

— Santiago (@svpino) April 12, 2023

If you start with a large dataset, you can get a slice of your data that maintains the underlying distribution with a single click.

Look at the example images:

We can select 1% of the data without losing any information.

The tool just works! pic.twitter.com/Lym0JBbfR5

— Santiago (@svpino) April 12, 2023