Rene Haas, President, IP Products Group, Arm
Andy Hock, Vice President, Product Management, Cerebras Systems
Moderator: Verne Kopytoff, FORTUNE

Fortune Brainstorm A.I. 2021: How next-generation hardware will change the game in A.I.

Conferences

thank you Renee and Andy for for coming appreciate it reminder. We'd love to take questions later in the session. So please raise your hand when we when we start. And also we're taking questions from the virtual audience as well so please don't be shocked. Um So to start uh can you tell us why a safe business side executives should should in the audience should care about specialist ai chips. So I will start. Okay okay. Uh Gosh well I would say you know first off I work for Arm and Arm doesn't actually build chips. We are a supplier of semiconductor intellectual property. So we supply I. P. Two people who build chips and uh I would say when we think about AI as just an umbrella for specialized processing uh we're seeing it explode whether it's in areas around mobile phones, IOT automotive, virtually any application area and the verticals we're now seeing a need for specialist processing. It's really because the devices are getting smarter these harder devices are becoming more intelligent, more intuitive they learn. And the previous generation at least from our standpoint of I. P. That was developed I. E. Generic processing just doesn't cut it. And so we're seeing just an increased demand for specialized computer specialist processing so they need to care because it's going to make their business better. Absolutely make their products better. Okay sounds good sounds important. Um Now there's something in the news about some chip shortage um you know anything about it and and how is that going to impact what uh you know people using Ai are doing or will be able to do in the future. Sure. Um well let me start by saying thank you, pleasure to be here. Uh sorry, this is an Ai computer systems company. We exist to help transform the compute landscape by accelerating, compute for ai by orders of magnitude beyond legacy general purpose architectures to do things like reduce training time and cycle time on R and D and accelerate the ability of organizations like yours to bring new Ai powered applications to market. Um now on the chip shortage that we've heard a little bit about. We we do think about this a lot at rivers and we work very closely with our manufacturing and supply chain vendors to do as well as we can to forecast what's happening and bring those lead times. In. Um recently this year we've announced our second generation system and product called the CSM two. Um we are taking deliveries building and delivering those systems on schedule right now, so no impact for us at the moment. But of course we're keeping an eye on all those lead times but in general, how will it impact your average ai user? Well from the, from the highest level there is a shortage and uh I think everyone in this room is probably felt it in one way or another. Uh the pandemic, I think accelerated a number of megatrends relative to everything going digital. Just look at what it took to register for this conference. I had to have a clear app. I had to take a picture? I had to have a lot of technology that would allow me to safely attend this a micro example of the pervasiveness of this technology everywhere. So, you know, the other thing and I've been in the semiconductor business my my entire career longer than I would like to say, uh there are always booms and busts in this uh in this industry. There are times when there's demand is so high it can be served and then there's a time when we're oversupplied and then there are a lot of times where people will say, well this time is different, this time is different. They're all different in their own way. This one is different in the sense that we're seeing again increased digitalization of everything that we're doing ai is driving that and at the same time what's required from an investment standpoint to to catch up to build new fabs is a huge capital investment and it takes a lot of money. It takes a lot of time and it can't be solved overnight. Um so I think we're going to see the shortage for a bit uh things will correct. They always do, but it does feel like it's we're in it for for quite a bit mm and let's switch to how the what fields that ai chips can benefit now and and ultimately is it is it every industry, is it just some industries? Sure. Do you think? I think we're at the early stages of an evolutionary divergence in the chip architecture space Ai um as Renee noted, has transformative potential across industries from automotive to web, two basic scientific research and healthcare as we heard earlier today for the kinds of organizations that you all run. Um, Yeah. Um and and yourself, I mean, do you think that are their industries that ai chips aren't relevant? No, I believe that it's going to touch literally every single one of the original, the segments Arm and its partners, I think have shipped over 200 billion ships in the 30 years since arm has been accepted the last 25 years. The last, excuse me last year was about 25 billion. I wouldn't want to put a percentage on it, but if I go forward 10 years, I would imagine virtually every single chip that we're shipping has some ai component to it. In other words, Ai workloads are running on those devices. There may be aspects that run on our CPU on a GPU on a dedicated accelerator, but there will be some level of AI and learning taking place on literally everything in the next 10 years. Now, service is known for the super sized chips, how big are they again? So the chip that we've developed is 8.5 by 8.5 inches of unbroken Silicon, as we heard earlier today in the in the conference, the compute demands for large scale deep learning are growing exponentially such that state of the art models often take days or weeks or sometimes even months of compute time to train to convergence even on very capable large scale clusters of many small processors. So there's an opportunity there to make processing more efficient if you can effectively condense the deep learning compute resources of a cluster onto a single chip. And in a sense, that's what we've done at cerebral by bringing all the compute and memory and communication resources all onto one silicon surface, we can then compute not only have higher performance, but unlock that performance with greater efficiency and ultimately the customers would have to build custom machines that would fit these giant ships. Right? So if you don't take a full solutions perspective, that might be true. So it's a reverse. Once we arrived on this large scale chip architecture, our systems team then looked at the solution from the full systems standpoint and took a systems engineering approach to build new packaging, power delivery technologies and cooling to put that wafer scale engine into a standard rack mountable device that can be put into a laboratory cluster data center supercomputer or into a cloud facility so that it can be accessed by many users anywhere, uh who might not be able to lift one of our machines on their own. And when I do you have any thoughts on the super sized chip? Um 88" by 8" is a gigantic chip. So I'll go to the other in the spectrum and I'll go to an area around low power edge devices. We've been working with some companies who have been developing sensors. Think of a very small battery powered sensor that can detect particles and different particles inside a room. So for example, going back to to the covid covid example, we've been working with companies who are trying to do small sensors with Ml inside. So embedded nps that can do some level of machine learning around the data. Imagine this room with those sensors inside where ultimately you could detect what particles are actually inside the room and potentially particles that could be could be harmful. Uh one of the trends that we're seeing really take off is adding this type of machine learning uh an NP you capability to very low battery power devices. This was something for five years ago we weren't having conversations with people. Uh people thought about IOT was a receiver or a transmitter and a small computer but doing machine learning on the edge device that wasn't something people talked about now. We're just seeing all kinds of demand for those applications. You mentioned edge computing. Uh, thoughts on where we are in that trajectory and you know why it's such a good idea. Sure. I think we're uh you know, Renee mentioned earlier this concept of heterogeneous infrastructure where best of breed architectures and machines can be employed to uh to work on different parts of an end and workflow. And I think that vision of the future is maybe not just true in the data center where you might imagine a data center that has uh Cpus gpus F P G a. S high performance devices like ours arm architecture devices. But you can also imagine a heterogeneous spectrum of compute from heavy high performance data centers all the way out to very lightweight applications. More and more we hear from customers that are interested in high performance compute in those intermediate domains like heavy edge processing for example next to a scientific instrument um or processing may be at a ground station for satellite telecommunications or earth observation. Let's talk about the role that software plays in this. How how how key is it for the software to be catered specifically to the chip. Sure. Um I love this question. I come from a application and research background. I think software is absolutely critical in the metaphor of the body. It is the connective tissue between the brains of your ai enterprise and the muscle. The underlying compute um I often joke that most good hardware companies are in fact mostly software engineers and by population. That's actually true. It's a reverse as well. So we've invested a significant amount of resources in building a software stack that makes this new high performance architecture easy to use without change using standard open source ml frameworks. So at the end of the day, you program our system using the same ml frameworks that people are using today, like tensorflow and pytorch. Right. And your thoughts on that? Yeah, I'm gonna resist the opportunity to chant developers develop, I did it anyway, I would know when we think about it, The software component is obviously, you know, to the chinese communists is huge and uh and developer community for this type of area is is just exploding. So one of things that that we think about an armed quite deeply is how do we make it easier for that development community too. Now essentially poured their ml workloads onto the, onto the hardware and again, I'll go back to the Iot example of one that uh doesn't lend itself traditionally very well to a software developer doing work, in other words, there's typically a development board, someone has to build a chip, that board has to be built and then the developer has to work on that board to do their ml workloads. So we've been doing some things about making that happen in the cloud, virtualizing, ml development what we call Ml devops. And that is an opportunity that gives Ml developers to start targeting hardware before that hardware even exists and at a rapid pace. Uh that is taking place with ai and ml uh we think that's, you know, really, really important, we work very closely with the ecosystem uh to do that. And again, I I think the developer story community for for this area is huge. And you know, these comments around tensorflow and pytorch, you need to support those frameworks and make it just easy for these things to be built. Now the googles and apples of the world are increasingly making their own chips, um you know, what does that mean to, you know, consumer devices and and to the, you know what people are, what software people can create. You can take this one. Sure. Yeah, it's it's a it's a trend and I'm in this industry my whole career, I started at at T. I many, many moons ago when T I used to make their own testers there on testing programs are very, very vertically integrated company and obviously they had their own factories and then the industry moved and you had sort of a horizontal ization, if you will of the industry. Now we're moving back a little bit to a vertical ization where large system companies are starting to do their own products, which I think in some cases does make a lot of sense for for what they're doing. Um You know, the one company you mentioned as far as having a closed ecosystem, they do the hardware, they do the software, there's a lot of advantages to that. Um I think we'll see that, you know, I think we're seeing more and more specialization. The companies as you mentioned are very large. They have scale uh these products are really hard to build, there's a scarcity of engineers to build them. So I think over the next number of years we'll start to see that that trend um uh continue, It'll be interesting to see where it goes next 5 to 10 years, but I think in the short to medium term, I think that would be a trend that will continue. Let's go to uh audience questions if anyone wants to raise their hand, we have people with mics around. Um Anyone Up, we got one over here, Go ahead. Hi Angelo Z Tavern Alixpartners. I'm curious to what extent do you work with Academia? And is that important to, to your industry overall or you know, how does that work um with hardware? Yeah, for a really good question for us, it's very important. So again, arms primary products are programmable devices, so we work a lot with universities work a lot with different universities in terms of training programs. Having them program on arm is a really good thing. Uh We're a UK based company. So our headquarters are in Cambridge. So, Cambridge University is one of the ones we work very closely with, but we do it globally uh and and having folks really learning how to use our products in academia and universities is is super critical. Yeah, I'll just, I'll double down on that answer. I think what we see in academics fear today is that research institutions are really pushing the boundaries of what's possible in a I particularly large scale and with novel algorithms that can learn more efficiently and take advantage of novel architectures like these. Um some of our earliest deployments were two basic science research facilities uh like Argon National Labs in the U. S. Department of Energy as well as Pittsburgh Supercomputing Center at Carnegie Mellon University. So I think having academic researchers use and understand where new architectures can deliver the greatest value is incredibly important to our product design. Uh huh. Any more questions? Go ahead. Hi, this is sundry from intel. Um my question is on security. We just had a panel discussion just before this where we were talking about security being a critical element of Ai and how hardware needs to do more to help secure the large ai data sets that are either residing in our memories, whatever we're doing with what is the plan for arm from the hips side to do? Yes, you have your trust zones, you have all that. But I mean is there something more that you see being done, rennie and what do you do instead of risk for security? Sure, I'll start there and nice to see you some very um so we it's a it's an excellent question. Obviously we just announced our new version nine of our architecture arm comes up with a new version of the architecture every eight or nine years and the version that we just came out with. The real notion there was around was around security. We didn't go from, we went from 32 bit to 64 on the earlier version. We didn't go from 64-1, All the things we did here, we're around security and really around hardware features. I was just before we walked on, I heard the two words confidential compute and it's one of the key areas that we're focused on with with version nine. Uh we've added such things such as realms, uh, things that can protect uh the operating kernel from outside malware attacks things of that nature. So a lot of what we've done at arm sunday on on version nine is very focused on security because it has, we believe that obviously um software is what runs on the CPU and the and the infrastructure, but ultimately we have to protect in hardware and we've done a lot of things around the next version there. Yeah, we only have a little bit of time left, but its rivers with multiple layers of security through R. I. P stack on the hardware side. Um and as as Renee mentioned, our our system in particular can be thought of as a as a network attached accelerator. And so also can take advantage of the additional layers of security say that a host CPU or network might offer uh from the application side, I think I'm really excited to see the kinds of neural networks and algorithms that are our colleagues in the cybersecurity space come up with to operate on these new architectures that not only accelerate existing Ai models but offer platforms to develop new kinds of AI models that can help advance the security enterprise overall. All right, well, I want to thank both of you for coming and talking about ships. Thank you. Thanks very much. Thank you that.