Aligned AI

The essential guide to a sector that’s now attracting one out of every four startup investment dollars.

AI-Innovators-2023-Aligned AI

Aligned AI made a name for itself when its content moderation filter crushed OpenAI’s. It captured 97% of problematic responses compared to OpenAI’s 32%, Fortune’s Jeremy Kahn reported. Since then, it has become the first company to show that its AI model can master a challenging AI safety and alignment benchmark based on the simple video game environment CoinRun. Safety has been Aligned AI's calling card from the get go. “You can actually think logically about danger from artificial intelligence, and create technical solutions that make progress on the problem,” says Rebecca Gorman, Aligned AI cofounder and CEO. Gorman and Stuart Armstrong started the Oxford, England-based startup in December 2021 with the idea that capabilities and safety in AI didn’t have to be a zero-sum game. Instead, they believe that the safer AI is, the more effective it will ultimately be, because end users will trust it more.<br />
<br />
&nbsp;<br />
<br />
The company is still in the early stages of its development, with £600,000 (approximately $762,000) in pre-seed money and just six employees. It’s now ramping up hiring of technical roles as it looks to develop partnerships with AI industry players it hopes to provide with safety features. Aligned aims to partner with the major industry players for each use case of AI, like quality assurance or robtoics, Gorman says.

Fortune 50 AI Innovators

Get insights into Aligned AI including business profile, stock information, news, and rankings from Fortune.

Aligned AI Company Profile, Stock Price, News, Rankings

legacy

aligned ai

Companies

Aligned AI | Fortune

Aligned AI Company Profile, Stock Price, News, Rankings | Fortune

As energy security and supply chain resilience dominate global priorities, a new axis of investment and infrastructure is emerging across ASEAN, the Gulf, and China.
From undersea cables to hydrogen ports, and from integrated maritime corridors to electrified freight, the region is laying down the hardware of the 21st-century economy.
How are capital, sustainability, and scale being aligned across borders?
What models of cross-regional cooperation are working—and which need a rethink?
How are Gulf entities positioning themselves not just as investors, but as long-term operators in Asia’s most critical corridors?
And what role will net-zero commitments, digitalization, and geopolitical realignment play in shaping the next phase of energy and logistics growth?

Driving Strategic Growth and Excellence in Energy and Logistics

Conferences

ASEAN is poised to be one of the world’s new economic powerhouses; it’s also one of its most complicated regions, boasting different government types, levels of development, and regulatory frameworks across its ten member nations.
This session will explore high-growth sectors across ASEAN and examine the best practices for market entry.
Speakers will share success stories and case studies, from joint ventures to strategic partnerships, and highlight how they aligned commercial objectives with ASEAN’s broader sustainability and energy transition goals.

Accelerating Strategic Energy Investments Across the ASEAN Region

In today's rapidly evolving workplace, organizations are facing unprecedented challenges in employee mental health and organizational resilience. How can companies effectively evaluate mental health vendors to ensure their solutions are customizable, scalable, and aligned with employee needs while staying within budget? Join three experts in the field for an insightful discussion about how to create an engaging, evidence-based mental health benefits strategy that is a cornerstone of organizational performance and employee well-being—and how to prove its ROI.  

INVESTING IN RESILIENCE: A STRATEGIC APPROACH TO MENTAL HEALTH BENEFITS

Jeremy Barnes, Vice President, Product, Platform AI, ServiceNow Rebecca Gorman, Co-founder and CEO, Aligned AI Raza Habib, Co-founder and CEO, Humanloop Moderator: Jeremy Kahn, FORTUNE

Brainstorm AI London 2024: Boosting Confidence In AI With Employees And Consumers

Photography by Joe Maher / Fortune Brainstorm AI

The biggest impediments businesses are encountering and trying to implement generative A I is around the issue of trustworthiness. These systems just do not seem reliable enough to use in many business contexts. And I'm curious, you know, businesses are very concerned about reputational damage. They're concerned about giving out inaccurate information. But we're going to explore here maybe ways we can try to solve some of those problems and make A I more controllable. And I think control is also a big, a big issue, particularly as we move from A I that can just generate content to A I that's actually going to take actions, which is where everyone says this is probably going, Jeremy. I'm going to start with you at service. Now, how have you tried to address this problem of trustworthiness? You know, you're trying to implement systems for a lot of customers at scale. You know, one of the biggest enterprise, uh tech uh service companies out there. How have you tried to deal with trustworthiness? Yeah. And it, it's something that you need to think about very carefully from the beginning. It's not something that you can bolt on. So I've been thinking about it for quite a number of years. Uh A big part of our approach there is to uh make sure that we understand the actual, uh you know, the, what we're solving for, for the end user. Uh There's a little bit of a, um, you know, a, a false idea that if you say that each of the components of the system you build is trustworthy, therefore, the entire system is, uh that's not entirely true. So you know how you work that to analyze the, the end state of it. Um And that comes really into how we design our products, uh design them, test them and, and build them. Uh So we have a fundamental research lab who's working on issues of uh of trustworthiness, published about 70 papers uh on the, on the subject. Uh We have a set of uh what we call our human centric A I guidelines. So that's ensuring that there is uh your transparency, that it's human centric that we have uh is accountable and that is inclusive as well. So each of our products goes through that kind of design process to ensure that we uh have those built in. And there's just a giant amount of testing, testing of the end uh product. Uh We use it internally before we release it and we've uh trained hundreds of people to uh do this testing. Uh And it's important, it's not just testing the model, it's testing the system end to end so that we can get some kind of, uh, uh, guarantee of, you know, it's not gonna do, do, do certain things because we've tested very carefully around edges to make sure that it is. I was gonna ask about edge cases. I mean, how do you identify those and how do you make sure that, you know, because you might have a system that seems like it's working and then you put in deployment and suddenly a user asked something that you weren't expecting and you get, you get a failure mode, right? Yeah. So, I mean, part of that is addressed with the uh you know, our human centry A I guideline. So we make sure that we have a human in the loop for any consequential decision. So there is the ability for the human to understand what's going on and, and override it. But a lot of it is the, the mentality of testing is you need to figure out, you know, you can't just test that it works. You have to test, you have to know where it's gonna break and then you can put guide uh you know, guard rails in around that. And so it's really, you know how you equip that team to be able to, to do that. So you get the outcome right? Raza, I want to go to human loop. You guys are, you know, creating systems for figuring out reliable prompts for LL MS can you talk about how you do that and, and how you're able to, to implement that because I think a lot of people struggle with that. They, you know, one particular subtle prompt works really well. And then the next week it doesn't work or you change a few words in the prompt and suddenly you get a completely different, different answer. Yeah. So I think, I think Jeremy touched on some of this already, right, in terms of having good processes for testing internally. But maybe if we zoom out for a moment, like it's worth understanding, like what's changed, that means that this stuff is harder than before. And I think fundamentally, if you think about large language models or generative A I compared to traditional software, machine learning, we've gone from a paradigm with traditional software where you run the software and every time you get the same result. So it's very easy to write a test for that. And then with machine learning, OK, it was stochastic, it was random, but at least I could calculate a number. I knew where the accuracy was. I could calculate precision or recall. We had ways of doing this. We're now applying these technologies to things that are much more subjective. And so actually even just saying what good looks like can start to become difficult. And so giving the first thing we do is give people tools to be able to measure performance in a variety of different ways. So building like a suite of different evaluators that are measuring different aspects of how well the system is working both during development and production and then iterating against those with some kind of test set. And, you know, Jeremy, you asked Jeremy asked Jeremy, you know, how do you sort of test for edge cases? And I guess I take two things of that one, you know, in some senses because it's subjective, the correct answer becomes what your users say. The correct answer is for some of these applications, not for factual ones, but if you're writing someone an email or doing something that's creative in nature, you're trying to optimize for end user experience, not for some internal metric. And you find the edge cases to a certain extent like you can cover as much as you want pre production. But what you also need to be able to do is if something goes wrong in production, know about it and if you don't have good evaluations in place, you can't even catch those edge cases to fix them. So I think a lot of people, you know, a worry is deploying systems that have edge cases that they don't know about. And we try to give people the tools to avoid that interesting Rebecca at aligned A I, you're all about sort of trying to create A I that has more conceptual understanding. Can you talk about why that's important and how you're trying to achieve that. Yes. So, um, if you think about artificial intelligence it's, it's kind of made, it made up of three pieces. Uh, there's computer science, there's, uh, statistical science and, and then there's data and, um, sometimes, uh, an artificial intelligence can go wrong because the, the computer science bit of it, um, isn't, isn't completely, uh, reliable, like in the Post, uh, post Master scandal we had here with Fujitsu in the UK. Um And sometimes it can go wrong because of, of um the statistical science, not uh it has inherent limitations. So anyone here who works in artificial intelligence knows about things like concept drift, um and data drift and all models um degrade over time. Uh 95% of models A I models have to be retrained within a year because uh because the world changes, so you might not think this affects large language models, but actually, uh G BT two was out before um the pandemic and it or it came out during the pandemic and it made absolutely incorrect uh statements about what was going on in 2020 which were a bit odd when you were trying to use it in 2020. Um So e even language models uh need need to adjust the world as it changes as our language changes. And how are you trying to, how are you trying to solve that? Um Well, Jeremy, I was gonna mention the third part which is data. Um and uh data is, is kind of the also the another wild card when you come to artificial intelligence because all artificial intelligence is built on data. Um So we make artificial intelligence that um it's our, our goal is to make artificial intelligence that can understand human concepts. Even in situations where the data environment has changed. The environment is is different from, from what it was yesterday. It is different from what humans expected. Human blind spots are. There's always the uh blind spots when humans look at data because we're the ones who collect it. We're the ones who label it, it looks right to us. Obviously, we've CRE created a data set that looks right to us. So the blind spots that artificial intelligence learns from this data is our own blind spots. And so it's very, very hard for humans to identify and recognize. So that's why we need to create artificial intelligence systems that can become aware of its own blind spots and extend that to new environments. And you've had some initial success with this. I mean, if you, I don't know if you want to talk about what you've done with content moderation and some of the results you've had there. But I I think that might be interesting that, you know, this is not complete science fiction people, you know, it's possible to actually use some of these techniques to achieve uh some success. Yeah. So um a uh one of the, the early open source um chat bots, uh open source language models that um that you would know the name of uh they released a chat bot that they had to shut down. And it was because uh they were using open A is content filter and a lot of, of toxicity was getting through. So they asked us if we could do better. So we looked at, at open A is uh content filter. They have it have an open source. Um uh They, they present open source of data sets that they, they use, et cetera. And it turned out that um their content filter worked great 97% or something on, on the um on the data sets they were trained on, but they didn't work well on different data sets of toxic content that, that, that model hadn't been trained on. So, Kin A I created a content filter uh that be the open A I content filter on data sets of toxic content that the artificial intelligence hadn't previously seen. Um I wanna do a quick sort of rapid fire uh set of questions here. One of the big issues is around hallucination uh that people are concerned about. I wanna ask you all whether you think this is solvable and if so on time scale, if you have a prediction for how many years out we are from solving this problem, if it's solvable or maybe you think it's not. So I'm curious, we'll go down the line. Raza. What about you? Do you think hallucination is, is a solvable technical challenge for these elements? Yeah. So I think, I think it is solvable technically, but I also think that it doesn't need to be solved for this to be useful productively. So there are ways we're used to designing user experiences that are fault tolerant. In some way. You go to Google search, you get a ranked list of links, you don't get an answer. And like people who are in perplexity, get citations back now. So I don't think it has to be solved to make it useful. But the reason I'm optimistic that we can solve it is if you look at the, when you train a large language model, it kind of goes through three stages, there's kind of pre training and then there's sort of fine tuning and then reinforcement learning from human feedback at the last stage. And if you look at the models before they are fine tuned on human preferences, they're surprisingly well calibrated. So if you ask the model for its confidence to an answer, that confidence correlates really well with whether or not the model is telling the truth, we then train them on human preferences and undo this. And so the knowledge is kind of there already. And the thing we need to figure out is like how to preserve it once we make the models more steerable. And but the fact that the models are learning calibration in the process already makes me very optimistic that it should be much easier to solve than most people. And what time frame would you guess? Oh, like within a year? Ok. Rebecca. What do you think? Uh, yeah, I think, uh, or not. Oh, yeah, obviously solvable. Right. And, and what time scale do you think? Um, II, I agree with that in a year. Uh, Jeremy. well, from just a model perspective, uh you do have to buy some, in some ways, you don't want them to only produce things that are in the training data. So they kind of need to, you know, it's part of their, their capability. So from a model perspective, uh it's an unsolved question, uh it's not gonna be solvable from a systems perspective. And user application perspective, I agree with rather, it's, you know, it's already solved in, in many cases. And uh when you can build a system around it, uh you know, you have other tools there. So you know, a model which will hallucinate can still be incredibly useful in producing value. Can I can I add one thing to that? It's not always a bug like sometimes it's a feature. So if we want to have models that will one day be able to create new knowledge for us, then we need them to be able to act as conjecture machines. We want them to propose things that are weird and novel and then be able to filter that in some way. And so like in some senses, and also if you're doing creative tasks, actually, like having the model to be able to sort of fabricate things that are going off, the data domain is not necessarily a terrible thing. I'd like to add to my as well. Yeah. So um as someone set up here yesterday, uh foundation models as they're created today are hallucination machines. It's just sometimes the case that their hallucination corresponds with reality. Um So, well, I think it's solvable in a year. I also think that uh well, obviously making the models bigger is, is not going to solve that problem. Would you agree that humans are hallucination machines? That uh that's a philosophical question. II, I would say yes, that we are in a different way. Interesting. I want to go to the audience for questions. Um Please raise your hand if you have a question and we'll get a mic to you and um anybody questions. If not, I'll keep going. There's a question there. And if you could please identify yourself when you stand up, that would be great SCT of real chemistry. So when new technology comes as adaptation, we adapt technology to humans, we also adapt humans to technology and we've seen that happening a lot with technology over the years. So I'm curious about your take on what's the biggest adaptation of humans to generative A I that you expect in the next few years, biggest adaptation of humans to gene A I Jeremy. You take that one first. Well, we saw it happen. Uh You know, when Google first came onto the scene, people just changed the way that they searched in order to, you know, type in queries, which would be achievable. So, uh you know, I think that will lead to uh you know, some of the uh challenges with those systems, you know, gradually as people learn how to steer them, uh you know, I improving um on the flip side, uh we probably don't wanna expect people to adapt. I think that's one of the great things about, you know, that the promise of all this new generative A I technology is it can meet humans where they are instead of humans having to kind of come, you know, if you've used a chatbot from pre gene A A I, you had to, you could get it to work, but you had to really know exactly how to, to phrase things. And so um yes, I think it'll happen but I hope that we don't count on it too much. Interesting. Uh Other questions from the audience, please raise your hand if you have questions. Um I want to ask about the sort of the Air Canada chatbot incident because that's a case where, you know, it was using some sort of rag mechanism, it retrieves a document, it thinks says something but it has not summarized the document correctly and then you have a user, you know, a customer in that case, getting incorrect information and ultimately suing successfully. You know, I think that there's a lot of companies out there, that sort of incident. I, I'm curious, like how you deal with that situation, uh, since you're Canadian as well. Jeremy, I'll go to you first. Um, but yeah, what was, what's your reaction to the incident? Yeah, I'm on, I'm on an air Canada flight tomorrow, so it's gross to my heart. Listen, that's our, that's our kind of our, our core business. And so, you know, there's a um it's a question of how you design these systems. Uh you know, if you're just designing it to just with a positive metric in mind of, you know, is it able to get to the end of the conversation? Not, is it able to, to, uh you know, be accurate and you don't design the system, right? That's kind of what you get and the a lot of these systems here. Yeah, we talk about, you know, prompt engineering and I'd say, you know, the engineering is very much uh in, in inverted uh commas here. Uh We see that already there's this kind of prompt depth where, you know, there are these prompts which don't actually represent what we wanted to do but have been tested on, you know, a small uh set of examples. And so again, it's how you take something which seems to work in a proof of concept. You probably don't just wanna put it straight into production, you know, with real customers who, you know, have expectations and terms and conditions and things like that. And so it's, you know, how you bring it in the enterprise context so that it's able to uh you know, knock off some of those rough edges. That's where that the tough work is. And yeah, that needs to be done. Prompt. That's an interesting concept rather. I mean, is that something you, you've been uh considering or, or? Yeah. No, absolutely. I mean, in some senses, my answer, this feels like a softball question for me because like, how could I, I can, could have fixed this or they should have used human loop, right? Um But yeah, the management of prompts like versioning them, having history of them being able to test them rigorously, being able to have the domain experts be involved in that development. Like I do think that the Air Canada incident was like, completely avoidable. Like there's nothing fundamental about the technology that meant that had to happen. I just don't think that they had done enough around testing of the city. Yeah, I, I really disagree. What, what we've seen in enterprises over the past several months is that they're really cool demos and the really cool prototypes the CIO s brought to their CEO S, they just haven't been able to get them reliable and robust enough to, to bring them into production, at least. So, I, I agree there's a perception of that, but we've done it enough times now that I think it's just a question of like, so how would it, I'm curious on what, what would they should they have done to solve that? So, I think if it had sufficient guard rails in place, like they, they gave the Chatbot way much hard, like wide range than what it should have been able to say. So you can put these things in much narrower rails and like we used to do this with Chatbot systems where you do intent inference and then you kind of like follow up with smaller pieces of usage and they more or less gave people like almost raw access to Chat GPT with a little bit of rag attached and sure that's a dangerous thing to do, but like most people shouldn't do that. Right. Yes, I should have used service now on that. We've had, you know, hundreds of 13 years of work to kind of make that work. And, you know, a lot of it is just an intern builder system and then it's like we wanna push into production. So if you're understanding the, what I could do, so don't just give real access to, to chat GP T. Um, on that note, we're gonna have to end it because we're out of time. But thank you very much for, to my panelists and thank you all for listening. Thank you.

Aligned AI

Videos