The New York-based beauty company, which makes fragrances for stars such as Taylor Swift and Justin Bieber, on Tuesday said it was cutting jobs, and planning to drop some unprofitable businesses and fragrance license agreements as part of a restructuring effort aimed at lowering its overhead costs by $40 million to $50 million annually.

A few years ago, Elizabeth Arden got aggressive in the celebrity fragrance market—its roster now includes fragrances by Britney Spears, Taylor Swift, and Justin Bieber—in an attempt to go toe-to-toe with Coty (COTY). It also has found itself very exposed to the lower-end market after years of pushing into the mass market channel: a couple of years ago, it got dinged when a mass market retailer dramatically cut back on orders. (It didn’t say which one, but Wall Street analysts surmised it was Wal-Mart Stores (WMT).)

And so now it is trying to retrench, dropping some celebrity and designer licenses (it didn’t say which), which are typically lower-priced items, and putting more emphasis on reviving its higher-end namesake brand, along with a more concerted effort to push its prestige products, particularly overseas where Arden has been a latecomer. In May, about the time it instructed its Goldman Sachs bankers to potentially look for buyers, Elizabeth Arden announced its “Performance Improvement Plan,” aimed it helping its get back on its feet after a few years of middling sales. (It has repeatedly had to revise forecasts downward in the past year.)

Another big problem for Elizabeth Arden is that it is very exposed to fragrance at a time companies like Estee Lauder (EL) are pumping millions into research to develop better skin care products, the fastest growing area of the beauty industry.

“In fragrance, there’s been a lot of competition, and it tends to be more discretionary. At the same time, skin care has been getting a lot of investment from beauty companies,” said Morningstar analyst Erin Lash, who covers the beauty industry, though not Elizabeth Arden specifically.