History of Scaling Laws in AI Research

“Scaling increases not just the domain-specific capability of AI systems but also their generality. . . . Scale does not just make systems better at individual tasks; counterintuitively, it makes them better at almost everything.”

OpenAI’s scaling laws, used to develop GPT-3, were “clearly a major breakthrough.”

“Pre-training was the thing to scale. It was a particular scaling recipe. The big breakthrough of pre-training is the realization that this recipe is good.”

“So far, the (exponentially) more compute and data you put in, the more intelligence you get out. The effect is so clear and so important that I call the period since 2016 the scaling era of AI.”

Amodei explains that scaling laws are an empirical fact, but on why scaling works so smoothly, “the truth is that we still don’t know.”

Textbook says scaling “completely changed what was possible with deep learning.”

Review predicts deep learning will “take advantage of increases in the amount of available computation and data,” and that new methods “will only accelerate this progress.”