• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

Iran just crossed Trump's red line for resuming all-out war as fighting continues to escalate with no end in sight

2

'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic

3

Indeed chief economist: Aging Baby Boomers are America's real labor problem, not AI

1

Iran just crossed Trump's red line for resuming all-out war as fighting continues to escalate with no end in sight

2

'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic

3

Indeed chief economist: Aging Baby Boomers are America's real labor problem, not AI
TechTikTok

TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25 times faster than OpenAI

By
Kali Hays
Kali Hays
Down Arrow Button Icon
By
Kali Hays
Kali Hays
Down Arrow Button Icon
October 3, 2024, 9:00 PM ET
A person holding a phone in front of a large TikTok logo
TikTok’s parent company is using a bot that scrapes online content at far greater quantities than rivals. Artur Widak/NurPhoto via Getty Images
Add Fortune on Google for similar content.

ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.

Recommended Video

The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April, according to research from Kasada, a company that specializes in bot management for companies with online data. The existence of the bot was also confirmed by Dark Visitors, which monitors scraper bots.

ByteDance’s bot has quickly become one of the most, if not the single most, aggressive scrapers on the internet, the research shows. It’s scraping data at a rate that’s many multiples of other major companies, such as Google, Meta, Amazon, OpenAI, and Anthropic, which use their own scraper bots to help create and improve their large language or multimodal models, known as LLMs or LMMs.

Sam Crowther, the CEO of Kasada, said since Bytespider showed up, it’s been scraping data at about 25 times the rate of GPTbot, which scrapes data for OpenAI’s ChatGPT platform and underlying models, for instance. Bytespider has been scraping at 3,000 times the rate of ClaudeBot, from Anthropic, which operates the Claude platform.

As the months have gone by, Bytespider has become even more aggressive, according to Kasada. Data shows huge spikes in scraping activity from Bytespider over each of the last six weeks.

Representatives of TikTok and ByteDance did not respond to emails seeking comment.

ByteDance’s aggressive scraping comes despite the possibility of TikTok being banned in the U.S. in the coming months. President Joe Biden has signed legislation that requires ByteDance to sell TikTok, due to national security concerns, or shut it down.

The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data. 

Web scraping goes back decades, mainly by search engines to gather links to web pages. But the rise of generative AI tools has added a new dimension and made the practice a prime source of lawsuits and controversy. People and organizations whose work has been scraped argue their copyright is being infringed in the process. All of the models that underlay generative AI tools were trained on massive amounts of online data, effectively everything available on the web, particularly written information. Tech companies use scraper bots to essentially copy it all for free and put it into their datasets.

“It’s like they’re trying desperately to catch up,” Crowther said of the aggressive scraping being done by Bytespider. Just last year, ByteDance was reportedly so far behind in the generative AI race that it was using OpenAI to help build ByteDance’s own LLM, which is against OpenAI’s terms of service. Earlier this year, ByteDance released a chat-based LLM called Doubao, but work on that model would have been completed prior to the accumulation of more recent training data scraped by Bytespider.

It’s “clear” that ByteDance is at work on a new LLM, according to one person familiar with the company. As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on keywords for ads, basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

A new AI model with data on more recent internet trends and topics could expand and improve TikTok’s search environment further, according to the person familiar with the company’s ambitions. 

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

Are you a TikTok or ByteDance employee or someone with insight or a tip to share? Contact Kali Hays securely through Signal at +1-949-280-0267 or at kali.hays@fortune.com.

About the Author
By Kali Hays
See full bioRight Arrow Button Icon
Add Fortune on Google for similar content.

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • Pinterest icon

Latest in Tech

Andrew McAfee
SuccessCareers
MIT AI expert warns automating Gen Z entry-level jobs could backfire—and cost companies their future workforce
By Preston ForeJuly 19, 2026
3 hours ago
A decade after the ‘Godfather of AI’ said radiologists were obsolete, their salaries are up to $571K and demand is growing fast
AIthe future of work
A decade after the ‘Godfather of AI’ said radiologists were obsolete, their salaries are up to $571K and demand is growing fast
By Marco Quiroz-GutierrezJuly 19, 2026
3 hours ago
America’s math and reading scores tanked after schools ditched textbooks for screens—and AI could worsen the brain rot
AIEducation
America’s math and reading scores tanked after schools ditched textbooks for screens—and AI could worsen the brain rot
By Sasha RogelbergJuly 19, 2026
3 hours ago
Looking back at the World Cup: Fans drank Boston dry, got permanent tattoos, sold out famous BBQ joints, and drove up small business revenue
RetailSports
Looking back at the World Cup: Fans drank Boston dry, got permanent tattoos, sold out famous BBQ joints, and drove up small business revenue
By Catherina GioinoJuly 19, 2026
4 hours ago
Beyond rockets and satellites, SpaceX is quietly building an AI compute business that may become key to its eye-popping valuation
AISpaceX
Beyond rockets and satellites, SpaceX is quietly building an AI compute business that may become key to its eye-popping valuation
By Marco Quiroz-GutierrezJuly 19, 2026
5 hours ago
data
AIData centers
Power companies are using eminent domain to seize land for data centers as 70% of Americans say not in my backyard
By Aaron Walayat and The ConversationJuly 19, 2026
8 hours ago

Most Popular

Iran just crossed Trump's red line for resuming all-out war as fighting continues to escalate with no end in sight
Middle East
Iran just crossed Trump's red line for resuming all-out war as fighting continues to escalate with no end in sight
By Jason MaJuly 18, 2026
17 hours ago
'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic
AI
'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic
By Jason MaJuly 18, 2026
18 hours ago
Indeed chief economist: Aging Baby Boomers are America's real labor problem, not AI
Commentary
Indeed chief economist: Aging Baby Boomers are America's real labor problem, not AI
By Svenja GudellJuly 18, 2026
1 day ago
U.S. companies have finally gotten $71 billion in tariff refunds, but they’re using it to offset inflation caused by the Iran war
Economy
U.S. companies have finally gotten $71 billion in tariff refunds, but they’re using it to offset inflation caused by the Iran war
By Sasha RogelbergJuly 17, 2026
2 days ago
FedEx CEO says we are in the middle of the biggest supply chain shift he’s seen in 35 years: ‘We are the referendum’
C-Suite
FedEx CEO says we are in the middle of the biggest supply chain shift he’s seen in 35 years: ‘We are the referendum’
By Fortune EditorsJuly 15, 2026
4 days ago
Kevin O’Leary claimed opposition to his Utah data center was fueled by Chinese money. Now he and Fox News are being sued for defamation
Law
Kevin O’Leary claimed opposition to his Utah data center was fueled by Chinese money. Now he and Fox News are being sued for defamation
By Marco Quiroz-GutierrezJuly 17, 2026
2 days ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.