Why organizations fail

October 23, 2013, 6:32 PM UTC

FORTUNE — The technology to see very small things up close showed us we had much wrong about health. The technology to see big things far away showed us we are not the center of the universe.

More recently, a technology called fMRI, that lets us collect images of oxygen use inside an active brain, has shown us that some of our long-held beliefs about human motivation may be wrong.

Matthew Lieberman, one of the founding fathers of a field called social neuroscience, tells this story in his new book, Social: Why Our Brains Are Wired to Connect.

As Lieberman explains, for a long time we believed that people were rational, logical agents, driven by self-interest, greed, and desire. While this is not untrue, it is only half the story. It turns out that people have another driver that is of equal, if not greater, importance: the drive to be social.

The studies tell the story: Giving to charity activates the brain’s reward system more than winning money. Painkillers like Tylenol relieve social pain the same way they relieve physical pain. Being socially rejected can lower your I.Q. score by 20% and cut your GRE score nearly in half. Seeing a friend regularly has the same effect on our well being as making an extra $100,000. Volunteering to help others regularly produces the same increase in well being as making an extra $50,000. When an employee meets a person who benefits from their work, that employee can double their productivity. People will pay $30,000 to be recognized as a high-status employee. And, finally, being socially connected is literally as good for your health as quitting smoking.

MORE: Delta proves that consolidation drives up ticket fares

Clearly, social activity matters more than we have realized. Yet our institutions and organizations, from political systems to hospitals, schools and corporations, have been built based on a different set of beliefs: that people are motivated by money, that physical — not social — health is most important, and that social needs are “nice to have.”

Long ago, researchers discovered that every time people try to solve a problem — do math or anything analytical in nature, including thinking about goals — they activate the lateral region of their brain’s prefrontal cortex. In the 1990s, it was discovered, much to our surprise, that an entirely different system was engaged when we think about other people and their thoughts, feelings, hopes, and fears. In this case, people activate a network of regions within the dorsomedial prefrontal cortex.

The ability to work well with other people in a group depends on our ability to appreciate other individuals’ emotions. A boss who knows what his staff members really want and care about will be able to design a better team environment than one who is simply focused on the elements of a project.

The irony is that human beings are built to mentally “reset” and see the world socially anytime they enter a new situation. However, modern humans tend to value analytical over social thinking, and so we tend to override that natural behavior.

The system for thinking socially and the system for thinking about goals and concepts function like a neural seesaw. When you engage one region, it dampens the activity of the other.

Our organizational environments have systems and processes that nudge people to think rationally rather than socially. In the workplace, if you are in a mindset that discounts social cues, you are going to miss a lot of important information around you and a lot of opportunities for creative problem-solving. We end up thinking that a lot of problems have analytic solutions; you just have to crunch the right numbers. Yet many of the toughest business challenges require social solutions. What does the person, team, or whole organization need to feel good? People who feel good are generally more productive.

We have hired and promoted generations of managers with robust analytical skills and poor social skills, and we don’t seem to think that matters.

How bad is the problem? Recently, I worked with the firm Management Research Group to look into data on 60,000 managers collected over 10 years across four continents. We asked the following question: What percentage of managers could be considered among the top 33% of performers as measured by their ability to focus both on work goals and the needs of other people?

The answer? Only 0.77%. Less than 1% of leaders and managers seem to be reasonably strong in both areas. If we look at just being in the top 50% of performers, we still only get 5%.

A lack of social skills is behind some of the biggest challenges in organizations. Starting from the top, if leaders are not good at understanding others, they are likely to develop a strategy and expect everyone to get on board, without stopping to imagine how others may feel about that plan. In fact, just 30% of change initiatives succeed, according to 15 years of data from McKinsey & Co.

Executives often expect that employees will follow orders and execute a strategy as planned without taking into account the human factors at play. When it comes time to give employees performance feedback, HR departments provide data and expect people to just change without recognizing that criticism is like having someone threaten your life.

MORE: Buffett: Why I didn’t buy the Washington Post

So, what can a positive story look like? Juniper Networks (JNPR) (full disclosure: a client of mine) got rid of performance management rankings entirely, recognizing it created social threats that reduced collaboration. The outcome of this experiment after four years so far? Increased motivation, greater pay differentiation, and wrong people leaving the company faster.

In short: We are deeply social beings, with social needs mattering more than physical needs in many situations. As Lieberman describes in Social, Maslow may have been wrong: Social is not up the pyramid, it is right down there at the base with physical needs. Until this insight makes its way into how we design our institutions, we may continue to see less than 30% of people in our organizations actively engaged in their work, and a number of our most important institutions failing.

David Rock is cofounder of the Neuroleadership Institute, a consultant and author of 
Your Brain at WorkHear Matt Lieberman present his thinking at the NeuroLeadership Summit in Washington, D.C, November 6 and 7. Watch the action via free live streaming if you can’t make it. 

Subscribe to Well Adjusted, our newsletter full of simple strategies to work smarter and live better, from the Fortune Well team. Sign up today.