By Brad Tuttle @bradrtuttle

A Monmouth University survey released last fall showed that more than half of New Jersey residents want to leave at some point, with 26% saying that it’s “very likely” they’ll move away from the Garden State. The most popular reasons cited for were the costs of housing and property taxes—the high cost of property taxes in particular. “The chief culprit among these costs is the New Jersey’s property tax burden,” Patrick Murray, director of the Monmouth University Polling Institute, explained.

New Jersey isn’t the only state at risk of losing residents to Florida, Pennsylvania, or another state with lower taxes. Stories pop up regularly speculating about the likelihood of homeowners jumping ship from high-tax states like New York and Connecticut as well.

Why is it that some states and municipalities have much higher property tax than their neighbors in the first place? Here’s a rundown of a few major factors.

The community has good schools. Or at least extremely well-funded ones. According to Zillow, the median residential property tax bill in New York’s Westchester County is $13,842, highest in the nation. A Westchester Magazine feature focused on why the leafy, desirable county holds this dubious distinction. The piece draws a comparison to Virginia’s Fairfax County, which is similar in many ways to Westchester: They’re both suburbs of big cities (New York and Washington, D.C.), they have similarly high home values, and they educate about the same number of students in public schools, which in both places have a good reputation.

Yet Westchester spends over $1 billion more to fund its schools, and since property taxes cover the lion’s share of that bill, there’s a big disparity in what homeowners pay. The average residential property tax bill is about $5,500 annually, less than half of what Westchester residents pay (people in Fairfax still complain about property taxes being too high).

The average teacher salary in Fairfax was roughly $67,000 in 2014. In Westchester, the average was estimated at $88,000 in 2013. Benefits and administrative costs add up too—Fairfax County has one superintendent, Westchester has 40—and collectively they translate into bigger burdens on Westchester’s property owners. Defenders of high educator salaries always note that they’re necessary given the high cost of living in the area, and it’s a valid point. After all, teachers, principals, and superintendents must pay local property taxes!

State workers make good money too. By most measures, New Jersey homeowners have the country’s highest property taxes. Tax Foundation data shows that the Garden State has the highest effective property tax rate (percentage of home value) and the highest property taxes per capita. The average property tax bill in the state hit $8,161 in 2014, also tops in the U.S. In fact, one study indicates that less than 1% of American homeowners pay more than $8,000 annually in property taxes.

An Asbury Park Press op-ed published last summer noted that a big reason for the state’s high property taxes is how much the state pays its workers:

The problem lies less with layers of government and excessive numbers of government workers providing services than with the generous salaries and benefits of those who are on the public payroll. Average state worker salaries: highest in the nation. Average teacher salaries: third highest. Public employee health benefit costs: second highest in the nation.

Your state relies heavily on property taxes. The above-referenced editorial also points out that 48% of state and local revenues collected in N.J. come from property taxes, which is off-the-charts high: “No other state derives more than 41 percent of its revenue from that source; the U.S. average is 33.1 percent.”

This state of affairs would be more acceptable to locals if the tradeoff for high property taxes is low taxation in other areas. Indeed, New Jersey has one of the country’s lowest gas taxes, and it’s in the middle of the pack in terms of taxes on wine, spirits, and beer. Unlike many other states, people in New Jersey don’t pay any vehicle property taxes either. Then again, New Jerseyans do pay the second highest state sales tax rate (7%, only California is higher).

Little or no tourism. A recent WalletHub study named Hawaii as the state with the lowest property taxes. New Jersey property taxes are eight times higher than their counterparts in the Aloha State. And a big reason why homeowners get off (relatively) easy in Hawaii is that the state collects so much from outsiders, thanks to high taxes on hotels and other tourism expenses. Likewise, taxes paid by casinos and tourists in Nevadaare often credited as a reason why state property taxes aren’t high.

Little or no industry. The more that industrial and commercial businesses pay in taxes in a state or town, the less it’s necessary for homeowners to cover the government’s tab. According to the Wyoming Taxpayer Association, 69% of property taxes in the state are paid by mineral production businesses. Therefore, residential property taxes can remain low—the state has no income tax either. The city of Marlborough, Mass., recently estimated that it were it not for local commercial taxpayers, the average homeowner would see his property tax bill (now averaging $4,791) shoot up by $1,164 per year.

Your property is worth a bundle. Your property tax bill is based on multiplying the local tax rate times the assessed value of your home. So, generally speaking, the owners of more valuable homes pay more in property taxes. Marin County has the most expensive real estate in California, on average, so it should come as no surprise that it has the highest (or among the highest) average property taxes too. In New Jersey, the 10 towns with the highest property tax bills all averaged over $18,000 per year, and five out of the ten had average residential property values over $1 million.

Or it’s not worth much at all. A recent RealtyTrac reportshows that nationwide, the highest property tax rates were for high-end homes, valued between $2 million and $5 million. That’s not surprising. What is somewhat of a shock, however, is that the second highest effective property tax rate—calculated based on a percentage of a home’s value—was for houses at the extreme low end of the value spectrum, assessed at under $50,000 or less. Granted, owners at the low end aren’t paying big bucks, but in terms of the percentage of the home’s value, property tax rates represent a disproportionate burden.

Your assessment was too high. There may not be much you can do to change your local tax rate—other than move, of course. But you can challenge the assessment on your property. If your appeal results in a lower assessment, your tax bill goes down as well. The National Taxpayers Union estimates that somewhere between 30% and 60% of properties are over-assessed. This guide to disputing your property taxes from This Old House has some of the best advice on the topic we could find.

Watch more business news from Fortune: