If there’s one thing that has united Americans and Europeans in the past six months, it’s their attitudes toward refugees. In November 2015, following the Paris terrorist attacks, a Bloomberg News poll found that 53% of Americans were opposed to taking any Syrian refugees whatsoever. The sentiment is even stronger against immigrants and refugees in Europe.

But most economists and political wonks in general have just assumed that the success of the Brexit vote and Donald Trump, both seemingly driven by anti-refugee sentiment, was a result of xenophobia, prevention of terrorism, and/or racism. Turns out, though, there may be something to not wanting outsiders to enter your country, or more specifically your economy.

A new research paper published earlier this money by the National Bureau of Economic Research gives the anti-refugee advocates an economic argument to make as well.

The paper, called “The Labor Market Consequences of Refugee Supply Shocks,” which is by economists George Borjas and Joan Monras, studies the effects of large influxes of refugees on local labor markets in various refugee events during the second half of the twentieth century. They found that depending on the skill level of the immigrating refugees, their arrival can have very damaging effects to wages and employment levels of native workers.

Take for instance the more than 100,000 refugees from Cuba that poured into the United States in 1980, after Fidel Castro declared that Cuban citizens could freely leave the country through the Port of Mariel between April and October of that year. These workers largely settled in the Miami area, with roughly 60% still living there five years after the event. According to Borjas and Monras, the refugee shock caused wages to fall 41% for high school dropouts in Miami, as they were forced to compete with the largely low-skilled workers that poured in from Cuba.

The incident caused the wages of high school graduates, however, to rise by 2%, as they generally weren’t forced to compete with those less-skilled immigrants who nonetheless created increased demand for the local economy.

Another telling example of how massive refugee migrations affect destination countries is the large influx of Soviet Jews to Israel in the early 1990s after the collapse of the Soviet Union. Nearly half a million such immigrants came to Israel at that time, increasing the labor force by a massive 9.7%. But in this case, the Soviet immigrants were actually on average more educated than the typical Israeli, with the largest share of these workers (35.6%) being skilled workers in industry or construction. In this case, the effect on wages were the reverse of the Cuba scenario.

By 1995, wages for high school and university educated workers in industry and construction fell 12% and 39% respectively, though wages still increased for skilled workers in fields where Russian immigrants were less prominent.

This illustrates a logical point, but one that is not part of the normal economic story about immigrants. Generally economists tough immigration as a positive. It brings in more workers, who make more money, spend that money, and grow the economy.

Yet, when you bring in a many workers with one skill set or education level, native workers with a similar skill set will suffer. And this will continue to confound politicians who want to accept, in particular, Syrian refugees for humanitarian and geopolitical reasons. “The humanitarian principles that encourage receiving countries to accept as many migrants as possible have important distributional consequence,” Borjas and Moras write.

This point is especially important for the broader immigration debate in America. Although the 10,000 Syrian refugees that the U.S. has promised to accept are probably not numerous enough to have a large effect on wages, American does accept more than 1 million new, documented immigrants per year. Because the U.S. largely accepts immigrants based on family relationships rather than skill, many of these immigrants are unskilled laborers. This research suggests that in a time of rising inequality, when Americans with lower education levels are already struggling, it may make sense to shift this focus toward accepting more affluent and educated immigrants, so that the negative effects of immigration are borne by segments of the American population that are better off.