Subscribe to Eye on A.I. for expert weekly analysis on the intersection of artificial intelligence and industry, delivered free to your inbox.
Facebook unveiled an ambitious project to create an artificial intelligence system that can understand videos on Friday.
Initial progress on the technology has already improved video recommendations on Instagram, Facebook said. Applying the latest breakthroughs to the app’s Reels, the short videos users can upload and share, made suggestions about what users watch next more relevant.
The project showed a 20% reduction in errors when recognizing the speech in videos, Facebook said. In the future, the company could adapt the technology to automate video captioning and flagging policy violations involving hate speech and incitements to violence, it said in a blog post announcing the effort.
Facebook said the new methods would evemtually make it easier for users to search for and organize content. An example feature could enable someone to find a specific past video by asking Facebook to “show me every time we sang to Grandma.”
Putting brains in computers
The A.I. technology behind the project uses a large neural network, a kind of machine learning software loosely modeled on the human brain. The system teaches itself through a process called self-supervised learning.
The method involves training the A.I. by having it ingest large amounts of unlabeled or unstructured data and asking it to cluster that data into distinct groups by comparing and contrasting its intrinsic properties. This is different from the method that is often used to train A.I. systems, particularly for commercial applications, where a large pool of human-annotated examples are needed to teach a system to make a decision.
Facebook has in recent weeks made a series of announcements about its progress using self-supervised learning. Last week, it debuted a system called SEER (short for SElf-supERvised) that took in more than 1 billion variables and was trained on more than 400 million publicly available Instagram accounts. The technology performed tasks such as object recognition and image classification better than any previous self-supervised A.I. systems, Facebook said.
SEER demonstrated other advantages too. With a little additional training on a relatively small set of labeled data, the A.I. was able to outperform systems that had been specifically trained for those tasks with much larger sets of human-labeled data.
The future is so bright you’ll need shades
The Learning From Video project is even more ambitious since it involves moving images, instead of still ones.
Adding to the complexity, the system must incorporate audio information alongside all the extra visual data. The A.I.’s ability to find similarities in audio proved particularly useful in improving which Reels videos to recommend to Instagram users, Facebook said.
The next step will pose a greater challenge. As Facebook seeks to build a system capable of surfacing digital “memories” based on users’ short text queries, it will have to grapple with inconsistencies between the text description of the scene the user is looking for and the audio found in the video segment. To overcome the hurdle, the system will have to use other information, such as text found on the Internet, to learn what text descriptions might be associated with a set of images and sounds, and then match its findings accordingly.
“This work will allow us to move away from A.I. that requires people to look at and label videos by hand,” Facebook said in its blog post. That sort of automatic categorization of video will be crucial in a not-too-distant future when augmented reality glasses enable consumers to capture exponentially more video than they do even now with smartphones, the company said.
Eventually, people will expect A.I. systems to serve as a kind of extension of their actual memory, Facebook said. They will expect these tools to be able to quickly pull up and play back the past on request.
Of course, it’s no coincidence that Facebook is placing a huge bet on the future of augmented reality. The company has plans for new augmented reality glasses that it is working on in conjunction with glasses company Ray-Ban.
More must-read tech coverage from Fortune:
- Fitbit’s first product since being acquired by Google is just for kids
- How Clubhouse is engaging with Native American communities
- One year later: 15 ways life has changed since the onset of the COVID pandemic
- Commentary: Disinformation attacks are spreading. 4 keys to protecting your company
- T-Mobile wants to share your data with advertisers. Here’s how to opt out
