Adaption Labs represents a significant bet against the prevailing wisdom of the AI ​​industry that the best way to create more capable AI models is to make the underlying LLM bigger and train them on more data. As tech giants pour billions into greater training, Hooker argues that the approach is seeing diminishing returns. “Most labs don’t quadruple their model size every year, mainly because we’re seeing architecture saturation,” he said.

Hooker said the AI ​​industry is in a “calculation point” where improvements no longer come from simply building larger models, but by building systems that are more quickly and cheaply adapted to the task at hand.

Adaption Labs is not the only “neolab” (so called because they are a new generation of frontier AI labs after the success of more established companies such as OpenAI, Anthropic, and Google DeepMind) pursuing new AI architectures aimed at cracking continuous learning. Jerry Tworek, a senior researcher at OpenAI, left the company in recent weeks to found his own startup, called Core Automation, and said he is also interested in using new AI methods to create systems that learn continuously. David Silver, former Google DeepMind top researcher, left the tech giant last month to launch a startup called Ineffable Intelligence which will focus on the use of reinforcement learning—where an AI system learns from the actions it takes instead of from static data. This can, in some configurations, also lead to AI models that learn continuously.

Hooker’s startup organizes its work around three “pillars” he says: adaptive data (where AI systems generate and manipulate the data they need to answer a problem on the fly, rather than having to be trained from a large static dataset); adaptive intelligence (automatic adjustment of cost calculation based on the difficulty of the problem); and adaptive interface (learning from the way users interact with the system).

Since his days at Google, Hooker has built a reputation within AI circles as an opponent of the “scale is all you need” dogma of many of his fellow AI researchers. In a widely cited 2020 paper called “The Hardware Lottery,” he argued that AI ideas often succeed or fail based on whether they happen to fit existing hardware, rather than on their inherent merit. Recently, he wrote a research paper called “On the Slow Death of Scaling,” which argued that smaller models with better training methods can outperform larger ones.

At Cohere, he spearheaded the Aya project, a collaboration of 3,000 computer scientists from 119 countries that brings state-of-the-art AI capabilities to dozens of languages ​​where leading frontier models underperform—and does so using relatively compact models. The work shows that creative approaches to data curation and training can pay off at raw scale.

One of the ideas that Adaption Labs is investigating is what is called “gradient-free learning.” All AI models today are very large neural networks consisting of billions of digital neurons. Traditional neural network training uses a technique called gradient descent, which acts like a blindfolded hiker trying to find the lowest point in a valley by taking baby steps and trying to sense if they are going down a slope. The model makes small adjustments to billions of internal settings called “weights” – which determine how much a given neuron emphasizes the input from any other neuron connected to its own output – checking after each step if it is closer to the correct answer. This process requires a lot of computing power and can take weeks or months. And once the model is trained, these weights are locked in place.

To adjust the model for a particular task, users sometimes rely on fine tuning. This involves further training the model on a smaller, curated data set—often still consisting of thousands or tens of thousands of examples—and making further adjustments to the models’ weights. Also, it can be expensive, sometimes running into millions of dollars.

Alternatively, users simply try to give the model very specific instructions, or prompts, on how it should perform the task the user wants the model to perform. Hooker dismisses this as “prompt acrobatics” and notes that prompts often stop working and need to be rewritten when a new version of the model is released.

He said his goal was to “eliminate rapid engineering.”

Gradient-free learning avoids many refinement issues and is easy to engineer. Instead of adjusting all of the model’s internal weights through expensive training, Adaption Labs’ approach changes how the model behaves once it answers a question—what the researchers call “inference time.” The core values ​​of the model remain untouched, but the system can still adapt its behavior based on the task at hand.

“How do you update a model without touching the weights?” Hooker said. “There’s really interesting innovation in the architecture space, and it’s using computation in a very efficient way.”

He mentioned several different ways to do this. One is “on-the-fly merging,” where a system chooses from what is a repertoire of adapters—usually small models trained separately on small datasets. These adapters shape the large, main response of the model. The model decides which adapter to use depending on the question asked by the user.

Another method is “dynamic decoding.” Decoding refers to how a model selects its output from a range of possible responses. Dynamic decoding changes the probabilities based on the task at hand, without changing the model weights.

“We’re moving away from being a model,” Hooker said. “It’s part of the deep idea — it’s based on interaction, and a model has to change (in) real time based on what the task is.”

Hooker argues that the shift to these methods will dramatically change the economics of AI. “The most expensive compute is the pre-training compute, mostly because it’s a huge amount of computing, a huge amount of time. With the inference compute, you get more bang for (each unit of computing power),” he said.

Roy, Adaption’s CTO, brings deep expertise in making AI systems run smoothly. “My co-founder made GPUs very fast, which was important to us because of the real-time component,” Hooker said.

Hooker said Adaption will use the funding from its seed round to hire more AI researchers and engineers and also to hire designers to work on different user interfaces for AI that go beyond the standard “chat bar” used in most AI models.