• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia
TechBig Data

Survey shows huge popularity spike for Apache Spark

By
Derrick Harris
Derrick Harris
Down Arrow Button Icon
By
Derrick Harris
Derrick Harris
Down Arrow Button Icon
September 25, 2015, 3:49 PM ET
185216772
Servers in the data centerPhotograph by Getty Images

Apache Spark is the Taylor Swift of big data software. The open source technology has been around and popular for a few years. But 2015 was the year Spark went from an ascendant technology to a bona fide superstar.

Over the past summer, Fortune has covered some of Spark’s including IBM’s (IBM) $300 million investment in the technology, and big data company Cloudera’s switch to Spark from Hadoop MapReduce. They’re doing this because Spark is significantly faster and easier than other ways of processing lots of data, and it includes tools (real-time processing, machine learning and interactive SQL, for example) that are well suited for powering business objectives such as analyzing real-time data from connected devices, also known as the Internet of things.

But sometimes numbers speak louder than words (and even actions). One popular number often noted by the Spark community is that its roughly 600 contributors make it the most active project in the entire Apache Software Foundation, a major governing body for open source software, in terms of number of contributors. That’s no small feat considering the number of popular enterprise database and infrastructure projects currently governed by Apache.

And new numbers released this week as part of survey from Databricks, a software startup founded by the creators of Spark, shed some new light on just how popular the technology has become. One of the standout statistics has to do with attendance at user conferences, which are usually a good sign of interest in a technology and who’s using it. In 2015, attendance at Spark Summit events grew 156% to nearly 3,000, and the number of companies represented grew 152% to more than 1,100.

Users that presented at the most recent Spark Summit in June included NBCUniversal, Netflix (NFLX), Uber, Capital One (COF) and Baidu (BIDU). There’s also a set of technology companies including Salesforce.com (CRM) that are embedding Spark into their data analysis software to provide customers with faster processing. While software and web companies are the largest group of Spark users, the Databricks survey says about 15% of Spark users are in more traditional businesses like banking, medicine and telecommunications.

That might help to explain why the number of Spark developers who use Windows nearly quadrupled, going from 6% in 2014 to 23% in 2015. While Mac OS X and Linux have always been popular among Spark users and web developers, they’re not nearly so prevalent inside Fortune 500-type companies.

Really, though, Spark’s success among big companies is just a microcosm of a larger trend over the better part of a decade—including inside Apple (AAPL), the world’s most valuable company. Open source software is getting much better, often outpacing proprietary software in terms of innovation and quality, and chief information officers everywhere are taking note. The saying “No one ever got fired for buying Oracle” once seemed like a truism, but it might soon just seem like an antiquity.

For more about the business value of data analytics, watch this Fortune video:

Sign up for Data Sheet, Fortune’s daily newsletter about the business of technology.

About the Author
By Derrick Harris
See full bioRight Arrow Button Icon

Latest in Tech

OpenAI CEO of Applications Fidji Simo
AIOpenAI
OpenAI aims to silence concerns it is falling behind in the AI race with release of new model GPT-5.2
By Jeremy KahnDecember 11, 2025
1 hour ago
Sam Altman, CEO of OpenAI, speaks to the media as he arrives at the Sun Valley Lodge for the Allen & Company Sun Valley Conference on July 11, 2023 in Sun Valley, Idaho.
AIOpenAI
OpenAI and Disney just ended the ‘war’ between AI and Hollywood with their $1 billion Sora deal—and OpenAI made itself ‘indispensable,’ expert says
By Eva RoytburgDecember 11, 2025
2 hours ago
AIOpenAI
Bob Iger says Disney’s $1 billion deal with OpenAI is an ‘opportunity, not a threat’: ‘We’d rather participate than be disrupted by it’
By Marco Quiroz-GutierrezDecember 11, 2025
3 hours ago
ellison
AIearnings
Oracle drops on disappointing cloud sales, more AI spending
By Brody Ford, Ian King and BloombergDecember 11, 2025
4 hours ago
Kushner
Middle EastM&A
Paramount’s Mideast backing likely runs deeper than $24 billion
By Adveith Nair and BloombergDecember 11, 2025
4 hours ago
Late Apple cofounder Steve Jobs
SuccessCareers
Apple’s Steve Jobs told students to never ‘settle’ in their careers: ‘If you haven’t found it yet, keep looking’
By Emma BurleighDecember 11, 2025
4 hours ago

Most Popular

placeholder alt text
Success
At 18, doctors gave him three hours to live. He played video games from his hospital bed—and now, he’s built a $10 million-a-year video game studio
By Preston ForeDecember 10, 2025
1 day ago
placeholder alt text
Economy
‘Be careful what you wish for’: Top economist warns any additional interest rate cuts after today would signal the economy is slipping into danger
By Eva RoytburgDecember 10, 2025
1 day ago
placeholder alt text
Politics
Exclusive: U.S. businesses are getting throttled by the drop in tourism from Canada: ‘I can count the number of Canadian visitors on one hand’
By Dave SmithDecember 10, 2025
1 day ago
placeholder alt text
Economy
‘Fodder for a recession’: Top economist Mark Zandi warns about so many Americans ‘already living on the financial edge’ in a K-shaped economy 
By Eva RoytburgDecember 9, 2025
2 days ago
placeholder alt text
Success
Netflix–Paramount bidding wars are pushing Warner Bros CEO David Zaslav toward billionaire status—he has one rule for success: ‘Never be outworked’
By Preston ForeDecember 10, 2025
1 day ago
placeholder alt text
Uncategorized
Transforming customer support through intelligent AI operations
By Lauren ChomiukNovember 26, 2025
15 days ago
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • Future 50
  • World’s Most Admired Companies
  • See All Rankings
Sections
  • Finance
  • Leadership
  • Success
  • Tech
  • Asia
  • Europe
  • Environment
  • Fortune Crypto
  • Health
  • Retail
  • Lifestyle
  • Politics
  • Newsletters
  • Magazine
  • Features
  • Commentary
  • Mpw
  • CEO Initiative
  • Conferences
  • Personal Finance
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
About Us
  • About Us
  • Editorial Calendar
  • Press Center
  • Work At Fortune
  • Diversity And Inclusion
  • Terms And Conditions
  • Site Map

© 2025 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.