• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic

2

Power companies are using eminent domain to seize land for data centers as 70% of Americans say not in my backyard

3

‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion

1

'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic

2

Power companies are using eminent domain to seize land for data centers as 70% of Americans say not in my backyard

3

‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion
TechBig Data

Survey shows huge popularity spike for Apache Spark

By
Derrick Harris
Derrick Harris
Down Arrow Button Icon
By
Derrick Harris
Derrick Harris
Down Arrow Button Icon
September 25, 2015, 3:49 PM ET
185216772
Servers in the data centerPhotograph by Getty Images
Add Fortune on Google for similar content.

Apache Spark is the Taylor Swift of big data software. The open source technology has been around and popular for a few years. But 2015 was the year Spark went from an ascendant technology to a bona fide superstar.

Over the past summer, Fortune has covered some of Spark’s including IBM’s (IBM) $300 million investment in the technology, and big data company Cloudera’s switch to Spark from Hadoop MapReduce. They’re doing this because Spark is significantly faster and easier than other ways of processing lots of data, and it includes tools (real-time processing, machine learning and interactive SQL, for example) that are well suited for powering business objectives such as analyzing real-time data from connected devices, also known as the Internet of things.

But sometimes numbers speak louder than words (and even actions). One popular number often noted by the Spark community is that its roughly 600 contributors make it the most active project in the entire Apache Software Foundation, a major governing body for open source software, in terms of number of contributors. That’s no small feat considering the number of popular enterprise database and infrastructure projects currently governed by Apache.

And new numbers released this week as part of survey from Databricks, a software startup founded by the creators of Spark, shed some new light on just how popular the technology has become. One of the standout statistics has to do with attendance at user conferences, which are usually a good sign of interest in a technology and who’s using it. In 2015, attendance at Spark Summit events grew 156% to nearly 3,000, and the number of companies represented grew 152% to more than 1,100.

Users that presented at the most recent Spark Summit in June included NBCUniversal, Netflix (NFLX), Uber, Capital One (COF) and Baidu (BIDU). There’s also a set of technology companies including Salesforce.com (CRM) that are embedding Spark into their data analysis software to provide customers with faster processing. While software and web companies are the largest group of Spark users, the Databricks survey says about 15% of Spark users are in more traditional businesses like banking, medicine and telecommunications.

That might help to explain why the number of Spark developers who use Windows nearly quadrupled, going from 6% in 2014 to 23% in 2015. While Mac OS X and Linux have always been popular among Spark users and web developers, they’re not nearly so prevalent inside Fortune 500-type companies.

Really, though, Spark’s success among big companies is just a microcosm of a larger trend over the better part of a decade—including inside Apple (AAPL), the world’s most valuable company. Open source software is getting much better, often outpacing proprietary software in terms of innovation and quality, and chief information officers everywhere are taking note. The saying “No one ever got fired for buying Oracle” once seemed like a truism, but it might soon just seem like an antiquity.

For more about the business value of data analytics, watch this Fortune video:

Sign up for Data Sheet, Fortune’s daily newsletter about the business of technology.

 

About the Author
By Derrick Harris
See full bioRight Arrow Button Icon
Add Fortune on Google for similar content.

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • Pinterest icon

Latest in Tech

Sumeet Agrawal is VP of Product Management (Data, AI Governance & Context Engineering for Agentic Systems) at Salesforce.
CommentaryAI agents
Salesforce VP on the leaky AI pipeline: why cheaper tokens won’t fix enterprise AI
By Sumeet AgrawalJuly 20, 2026
6 hours ago
Artificial Intelligence technology and futuristic technology transformation
C-SuiteCFO Daily
How CFOs can tell if AI is actually creating value
By Sheryl EstradaJuly 20, 2026
7 hours ago
uk
PoliticsSocial Media
France wants to ban social media for kids — but Brussels says the bill breaks EU law
By Samuel Petrequin and The Associated PressJuly 20, 2026
8 hours ago
ryan
CommentaryLeadership
I’ve interviewed 700 leaders. The ones outsourcing their thinking to AI lose their best people first
By Ryan HawkJuly 20, 2026
9 hours ago
Pharmacy technician Brandy Leroy fills a TPN bag using the ExactaMix Pro Automated Compounding System
Startups & VentureTerm Sheet
The FDA’s peptide vote could create telehealth’s next multibillion-dollar market
By Lily Mae LazarusJuly 20, 2026
9 hours ago
Tarek Mansour, co-founder of Kalshi, in Washington, D.C. on Sept. 29, 2025. (Photo: Kent Nishimura/Bloomberg/Getty Images)
NewslettersFortune Tech
Thanks to $5.7 billion in Kalshi and Polymarket bets, FIFA’s World Cup final may be ‘the largest gambling event in history’
By Andrew NuscaJuly 20, 2026
10 hours ago

Most Popular

'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic
AI
'Dr. Doom' Nouriel Roubini says we're headed for universal basic income or 'some form of socialism' as AI revolutionizes work—He calls that optimistic
By Jason MaJuly 18, 2026
2 days ago
Power companies are using eminent domain to seize land for data centers as 70% of Americans say not in my backyard
AI
Power companies are using eminent domain to seize land for data centers as 70% of Americans say not in my backyard
By Aaron Walayat and The ConversationJuly 19, 2026
1 day ago
‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion
Success
‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion
By Sydney LakeJuly 20, 2026
7 hours ago
Red Lobster’s 37-year-old CEO waited tables before joining the C-suite—he says it was a crash course in managing ‘difficult people and situations’
Success
Red Lobster’s 37-year-old CEO waited tables before joining the C-suite—he says it was a crash course in managing ‘difficult people and situations’
By Preston ForeJuly 19, 2026
2 days ago
Warren Buffett says his $147 billion investing career was an accident: ‘I may be one of the 10 luckiest in the world’
Future of Work
Warren Buffett says his $147 billion investing career was an accident: ‘I may be one of the 10 luckiest in the world’
By Sarah GlodekJuly 20, 2026
13 hours ago
It’s not just Taco Bell lettuce and possible glass in Pillsbury rolls: Food and drink recall events reached a 6-year year-over-year high
Retail
It’s not just Taco Bell lettuce and possible glass in Pillsbury rolls: Food and drink recall events reached a 6-year year-over-year high
By Sasha RogelbergJuly 20, 2026
11 hours ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.