Name: From the AI We Have to the AI We Need
Uploaded: 2026-06-09
Duration: 18 min

Today's AI stack is delivering big productivity gains, but it continues to suffer from serious shortcomings. The AI GPUs on which the stack rests are not optimized for running trained AI models. They consume too much energy per unit of intelligence. SambaNova is trying to solve that problem with a better AI chip. Meanwhile, AI models do not learn continuously or have true memory. That's what Adaption Labs is working on. In conversation, their chief executives discuss how we get from the stack we have to the stack we truly need. 

Sara Hooker, Co-Founder and CEO, Adaption Labs
Rodrigo Liang, Co-Founder and CEO, SambaNova
Jeremy Kahn, AI Editor, Fortune

From the AI We Have to the AI We Need

Conferences

Right, there we go. um. The title of this panel is from the AI we have to the AI We Need or the AI We Want rather, um, and Sarah, I know you've written it before about like the slow death of scaling and why you think that today's sort of vastly uh gargantuan models are maybe not where we're heading. And uh, and Rodrigo, you built a company that's basically, uh, building uh AI chips to run these huge models but much faster, much more efficiently. So I just want to start with maybe one sentence from each of you, um, you know, do you think that the, you know, Sarah, that today's foundation, uh, LLMs are really, uh, you know, where, where we're going, or do you think we'll look back at this and say this was a detour and a bit of a dead end. Yeah, it's lovely to be here. One sentence. I would say today's models are monolithic. They're stuck in time. So a big goal of how we think about what should be fundamentally different is how do we efficiently learn from the environment, because now all models are moving towards interaction. So now it matters more that last mile performance and Uh, for that you need models which can evolve, otherwise you end up, uh, with massive inefficiencies. Rodrigo, are we going to be in this paradigm of huge models for forever? Uh, well, I, I think the world's heterogeneous. I think scale matters and so these large models are, I, I think, are going to be here for a while, and I think there's going to be plenty of room for new models and more efficient models to come in, um, and, um, have a, have a say in the market, right? And do you, do you think we need evolution across. All parts of the stack and maybe talk a little bit about what Samba Denova is doing on chips. Well, I think we're entering the phase of production, and I think if you think about production and at scale, you've got basically three constraints that people are struggling with. One is the models are getting bigger, at least one class of the models and the ones that, you know, we hear most about, I think cost is a huge part of the problem as you go to scale. That's just too expensive and that's largely driven by the infrastructure that we have. And then of course also about. Power and you know when we, when we, we argue about power and and being able to find uh chips, that's not because we're running small models, that's because the large models are taking a disproportionate number of chips to run and so we've got to tackle those problems. Got it. Sarah, at adaption, you're trying to solve some of the remaining challenges that we really haven't been able to solve with the current transformer-based architectures. Can you talk about what some of those gaps are that adaption's really trying to to fill and to solve for? Yeah, I would say, I mean this question kind of amounts to like how did we get here and like where do we go next. So I would say I would agree that the last 15 years what Rodrigo's saying is true. We've just, we've had a very simple cheap formula. You just build bigger and bigger models, but I think we're at an inflection point with massive urgency to change that curve. I would say there's two reasons. Most people intuitively understand. That you shouldn't just apply the same model to all problems, that probably 90% of problems are very easy. Many of the things that you do, um, in bulk processing, for example, you shouldn't be throwing a massive model at 10%, you need a ton of compute. So that is intuitive to people and things like adaptive compute, how do you do much more flexible model. Behavior is very critical, but the second intuitive thing I think most people grasp is that what matters now is that last mile performance, and that typically means you're in a long horizon task. You have many sequential agent actions, and if you just use a monolithic model, error compounds, so. A lot of what we care about is how do you adapt continuously to new data and how do you speed up changes to model behavior and both of those things are pretty critical to our mission because efficiency is key there. So when you have these massive rollouts, um, if you're just naively doing it, which I think a lot of people are feeling right now as they use a general harness monolithic model, um. You just end up painfully inefficient and that reflects as well in all the discussion I think has been at this conference about what am I going to do about my API bills. Right? And you talk a little bit about continuously learning from data that is one of the big unsolved challenges. Um, number of startups such as yourselves have come up in the last year saying they're going to try to address this. What's your intuition about how you solve for kind of continuous learning? Well, firstly, like what is continuous learning? Why do we care about it? Like a lot of why we care about it is now we have these long horizon tasks. If we're not updating model behavior as we go, you're, you're paying in a very expensive way for just blindly doing a rollout. Um, continuous learning is really the, the, the goal that you should be able to update model behavior without forgetting what you already know. For this, I think it's pretty critical for us that we infer automatically what the task is and are changing model behavior every step. For us, the constraint should be real time because unless it's real time, your cost, and maybe Rodrigo will talk about that, but your cost of even leveraging hardware and kind of offloading things to memory and back. Extremely expensive, so we want algorithmic and hardware codesign and that supercharges and I think differentiates what we're doing is that we actually care a lot about that synergy squeezing performance out of hardware and making it as fast as possible to update model behavior. Arrigo, what do you think about this idea that the model and the chip really need to be paired together and are you working on any of that at Somba Nova with any of the providers? Well, I mean, I, I think there's a lot of affinity between kind of the way that the chips run and the models that people build. Some of them we're focused on inference and we'll focus on. Um, premium inference, which is defined by speed and size and so we think that until the great new models are efficient come, you know, the battleground is in these very large 1 trillion parameter sized models, and you've got to run those much, much more efficiently than we are today to get the cost down. And so just give people a sense of how, how much faster you are than some of the competition now. Yeah, so if you look at a large Model only a 1 trillion parameter model these days, and we run full precision. There are a lot of little ways that you might be able to game it by quantizing the models down, but as Sarah will, will say, you lose accuracy on that, and that's not really the right way for hardware companies to go do this. And the way we do it is we run everything original precision. If great, uh, researchers created a wonderful model, it's not for the hardware guys to come and just, you know, chop the accuracy. And so we run a full precision, uh, and we run about 10 times. Faster at 1/10 the power. And so, but we don't necessarily come and say we do everything. What we do is we take the most popular models that people like and we optimize for them. So that's kind of tying into the synergy you're talking about. We look at what's, what are people struggling with, what do people want real time for, and then we drive the efficiency of those up and get the cost and the power way down. And Rodrigo, I know in, in the data centers that you're deploying into. You talk a little bit about heterogeneous compute. It's not all Sombonova chips. You're also using, you know, some CPUs from others, and you've even integrated with other GPUs for running different model nodes, right? I mean this is something Jensen announced a few months ago, and we actually showcased this last week in Taipei called disaggregated inference. And so you can use Nvidia GPUs in the front just to do the prefill and We run the decode of the exact same models significantly faster and we're getting 2 to 3x better than the Blackwell's on the exact same models and so we think that at scale that's the way to at least bring the cost down. You can get 2 to 3x uh cost reduction, 2 to 3x power reduction, and you get faster tokens for these real-time applications. Great. I want to go out to the audience for questions. Uh, who has questions? This gentleman right here got his hand up really fast, so please tell, please tell, please tell us who you are. Hi, I'm Paul Brody with Nightfall Networks, very interested in the case of uh Sombonova and adaption, especially for Sombonova, when you're talking about 2 to 3x improvement in performance, does that apply for training, or does that also apply to in addition to execution, and does that mostly just work with large language models, or can you put other kinds of models into that technology? So, so when I talk about that, this is really ultimately about inference. So we're inferencing models that can be As a single large model, a 1 trillion parameter model, or an aggregate of agents, and so you can aggregate agents that are doing various different things like Sarah says serially connected, uh, and then in total potentially adding up to 1 trillion parameters that you're switching in and out. But I think those two use cases is what we see the most. And Sarah, I wanted to ask about the question you, you touched on it in your answer to the previous question on um. Often with a, you know, what people really want is, uh, an answer to their question or or they want to run an agentic process at a price point that makes sense for them, and they don't necessarily, not necessarily wedded to one particular model to do that. Uh, and people often are saying, well, I wish I had a way that you could easily route, uh, a question to the right set of models or the right model. Are you looking at this as adoption of how you can potentially route to different kinds of models? Yeah, I think that so candidly, so, so much of what we're looking at is how do we make the full stack adaptable, so um. I think that routing is one, but even things like harness design, how do you automate that? Because everyone now has a different view of a harness, and oftentimes that's the main cost of inefficiency. And also for us it even extends to how do you change the whole innovation cycle. We just released Autoscientist which basically allows anyone to build a frontier model and a radically accelerated timeline. So instead of months it's In 2 days, the goal there is, you know, the people who can build models are actually very narrow. It's probably less than 2000 people who build frontier models. This came up in some of the talks yesterday, but how do we really learn in and teach models some of that knowledge that goes into that optimization process. So for us auto scientists is it automatically. Specifies uh how do you train for a given modality as well as like how does it automate all those choices of harness for a given problem that relates to something that came up earlier how do you do adaptive compute? How do you spend less when you don't need to, um, which is quite fun. um I'll throw in for Rodrigo's point like I just I think this is quite interesting maybe it frames the choice about inference um, so much of current workloads are inference so. As people acknowledge that you don't just want to scale anymore, those are not the returns in the pre-training stage. Most labs, but also most workloads are shifting towards how do you do things test time. So inference, there's a reason we're designing this domain specialized hardware. So I just wanted to add that because I think it's very important as people think about why we have this specialization. Yeah, there's a question. Yeah, one second, wait till the mic gets to you. There we go. Wait, who are you? Tell us who you are. Really interesting one is, you know, you have 1 trillion parameters, and you're designing to cope with that. And that begs the question to me about, well, maybe it will make more sense economically to do more efficient ensembling so you get rid of all of those freaking. But the other is you keep doubling down on inference, and I'm just wondering, I'll use the word in public ontologically, is the meaning of inference changing? As you create a learning loop between, yes, that's a very fun question. I'll share. So traditionally inference is you have a single model, you're basically just optimizing for throughput, and you basically have, you're trying to get squeeze as much performance out of a single model, and you're just passing things through. What's fundamentally different though, and Rodrigo, I think this is also changing the hardware scene critically. Um, is you now are doing agentic long horizon task, which means you have to check in as you go. That is a very painful operation. It means it's basically the whole crisis with memory. So you can think of like how we design hardware and optimize for efficiency as like we have this big oven. We can build the biggest oven in the world, and our only bottleneck is how fast the baker can put stuff in the oven. The baker is memory. What's fascinating. Now is that we're basically taking things out and putting things in the oven multiple times over the course of the recipe. So this makes it so important that you're designing things well. So inference is a different beast. How do you make it efficient and all your choices, like how do you auto optimize that become even more critical. It's also probably Rodrigo can speak to us, that's probably dramatically changing your world, right? I'm guessing. Well, I think it's, you know, to Sarah's analogy, it's actually worse than that. I think we have. 50 ovens, but 50 ovens, and we're actually moving from one oven to another to another and so ideally we, you know, we, we want to find ways to keep things in the oven and not keep opening. It's like the way I actually do turkeys for Thanksgiving. I keep checking in and you shouldn't, you know, so we really want to keep it in there and, uh, and that's what we're, uh, we, we think that the optimizations come from. Got it. Other questions? Other questions, to be honest, otherwise I'll keep going. Anybody? All right, um, uh, this is an interesting, uh, issue around, um. And would you just sort of, how do you, how do you do this though? I mean this is what I want to drill down on. What, what is for Soma Nova, the strategy for, uh, you know, making sure you're optimizing for these really large models? Maybe you don't have to run all those parameters. Are you doing things also on the, on the kind of middleware layer or, um, you know, at the kernel layer to optimize? Well, I think there are two ways to do this. I mean, if you have a hardware cycles, our hardware cycles are 3 to 4 years. Uh, so we're releasing now things that we created 3 years ago, and we've got great minds like Sarah who are inventing things that we could not possibly see 3 years earlier. And so we as hardware vendors, we're, we're having to create general purpose infrastructure, you know, things that allow future innovators to be able to run on our platform. And so we try to create things that allow us to take kernels that people are Going to use and implement them and still get the benefits of new architecture. And so, uh, there are, there are folks that say, hey, I wanna harden existing models, but by the time that hardware shows up, you're, the model is no longer of interest. And so, uh, so our, our view of it is, find the abstraction layer that allows innovators to create those, uh, uh, those architectures and primarily GPU, uh, centric, uh, ecosystems. Training and then allow them to then seamlessly bring it over to accelerators to get the real time and scale. Got it. Well, Rodrigo, I'd be remiss up here if I didn't, if I didn't talk about, uh, you know, Samanova. I'm sure you guys must have watched Cerris go out for its IPO. Um, you know, you guys were in discussions with Intel. Uh, there was talk about a purchase. Ultimately, you formed a very close partnership with them. They did not acquire you. Um, are you looking at IPO now for Samanova? Well, I mean, the market is incredibly hot for for technologies like ours. I think in the end we feel like, you know, there's a market that that requires our technology. And so as, as we build the business and we're able to show consistency in in deployment across a broad range of things, I think that it might be in the future. Got it. And Sarah, I want to ask you. For adaption, you know, you're working on these very interesting problems, yet there's so much attention on just a couple of the frontier labs, Anthropic, OpenAI, and they're sucking in a lot of talent and especially as they're heading towards IPOs soon. A lot of people are very attracted to work there, I'm sure. How are you finding sort of this war for talent as a smaller lab working on a very, very different kinds of architecture? I think when so most smart people like two things. They like very interesting questions and they want to work with people who are talented. I think that adaption has both of those. What I will say is that there's different stages to a company. A company just before IPO attracts a certain type of talent, a company that is at its beginning that resonates with a lot of people because especially if you're building something at the frontier of what's possible, that feels remarkable. It's also a very different type of problem. So the companies that IPO, their identity. To be cheeky has been weight-based, right? Even the discussions we've had here about, you know, it's always about the name of the model. Our, our bet is fundamentally different that the shift and the focus will put the model in the background and that by continuously evolving and interacting. And what matters much more is the critical efficiency of that last mile task. So this is a fundamentally different way of viewing the world. You're not thinking about a monolithic set of weights. Your identity is not the weights. You're thinking about how fast and how efficiently can you get that model to learn from its environment. And I think for many talented researchers that feels like the most important problem they can work on because it's the new era of what's coming next. Which is fun. Yeah, and it's much more about a systems approach to solving a problem as opposed to looking only at the model. 0, 100%. If you, so everything is moving towards interaction, I think existing Frontier Labs is just inertia there because they, their whole trajectory of how they, they shape teams, where they place resources is around stages of training, right? There's a pre-training team, continued learning team. Training RL. This is how the very resources are configured. If you're moving towards interaction, you care about different things. I care a lot about how do we design stuff that plays extremely fast with hardware because we want to update as fast as possible. If I care about interaction, I care about interface because I want to get feedback as quickly as possible to know when our model should update, and this requires a different type of team working much more nimbly and actually. A different type of co-design and that's, I think what attracts a lot of people and makes it a very fun time to be working on innovation again because the formula now is not as simple as like just throw more weights at the problem and that means in some ways it's it's a good era. The era of research is back again. The era of research is back again. All right, that's a good note to end on. We're afraid we're out of time, but thank you very much. Thank you for listening.