My dad moved from China to a village in Vietnam in the 1930s. He had a small grocery store, and I quit school before sixth grade to help out when he got sick during the Vietnam War. I have one sister and four brothers, and my father feared the South Vietnam army or the Vietcong would take all the boys away if we stayed together. So my oldest brother, who was 17, went to work in a textile factory in Saigon, and my second brother worked in a bakery in Bac Lieu.

The rest of us stayed until the fall of Saigon in 1975. Then we looked for ways to escape. Two of my brothers made it out. I stayed to help the rest of the family. In November 1978 the government allowed us to exchange our property for permission to leave, so we all left in a small boat. It took six nights to get to Malaysia, and along the way, pirates stole everything from us except a few pieces of clothing.

We landed in a village, and the International Red Cross eventually took us to the refugee camp in Pulau Bidong, which had about 53,000 refugees at its peak. The conditions were horrible.

In August 1979 my parents and I immigrated to Fairfield, Calif., with the help of the U.S. Catholic Conference. In 1981 all of us were finally reunited in San Francisco. Even though we had lost everything, we felt blessed.

My brothers worked in restaurants and as newspaper carriers, while their wives became seamstresses. I took English classes during the day and at night worked as a newspaper carrier while earning a degree in accounting from San Francisco State University. For a while seven of us lived in a one-bedroom apartment in the Tenderloin district. We didn’t take trips or spend money like other people. We just saved.

In 1984 a friend told us about a coffee shop that was for sale. We pooled our savings and bought it for $40,000. It was called the Sugar Bowl Bakery, and we kept the name. The seller taught us how to make doughnuts, muffins, and croissants. We began to sell doughnuts to other coffee shops and 7-Elevens and sales grew. Before long we saved $100,000.

In 1986 some of our family wanted to grow the company, and others wanted to grow our savings by investing in real estate. I convinced my brothers we could own real estate and expand the business at the same time, so we looked for a building to buy. We found a second location in Daly City, but no one would give us a mortgage. So I asked the sellers, an older Mexican couple, to finance us. They agreed. We made a down payment of $60,000 and paid them $2,247.46 a month for 10 years.

Seven days a week, another driver and I would deliver doughnuts. I was dating my future wife at the time, and on weekends I’d take her with me to collect money from the shopkeepers, on what I jokingly called our dates. In 1989, I bought a third building for $360,000, this time with a bank loan. My goal was to have each of my brothers managing a store.

I love to network and figure out how to get the next customer, so I started going to culinary events. In 1992, I found out that hotels were outsourcing pastries to save money. So we started selling to Parc 55 at 22¢ a doughnut, and started our food service division with a $500,000 loan from the SBA.

We didn’t have the proper training for making pastries, though. I remember bringing a Chinese cake to one chef, who laughed at me, saying he couldn’t sell Chinese cakes to Caucasian clients who were used to sweeter French pastries. I asked one of my nephews to go to the Culinary Institute of America in Napa Valley to get proper training as a pastry chef. Before he graduated, though, one of my brothers and I tried to improve our croissants by going through a competitor’s dumpsters.
We tried to figure out the ingredients they were using, but it didn’t help. Fortunately, later on, we were able to hire one of their bakers.

Then we started the Ly Brothers Corp., and I became the CEO. I was the only one who could speak English well and think out of the box while pulling everyone together, so my family trusted me to run the company.

After successfully selling to hotels, hospitals, and high-tech businesses in town, I started going to big retailers. I got nowhere. So in 2000 we hired some food brokers to introduce us to Costco. It took us two years to figure out a consistent seller for them: the palmier, which still sells well today.

That’s when I decided to concentrate on selling to high-volume retailers like Safeway, Kroger, and Wal-Mart. At the time we had seven retail locations, and it was difficult to manage the different business models. So after the economy collapsed in 2008, we sold our retail operations and the food service division. We went from 750 product lines with 4,000 SKUs to three product lines with 90 SKUs, so we were able to really lower our costs and achieve a higher margin.

We’re in our best years now. In 2013, President Obama used our story in an immigration-reform speech in San Francisco. I was so sick that day, I couldn’t even talk. But I managed to make it and got to shake his hand.

When I came here, I didn’t speak the language or have any money. I am proud that I’ve taken my family where they hesitated to go years ago. Whenever I mentor young people, I tell them, never give up. Work hard, have a good heart, and be disciplined. Those are the ingredients to success. 

My Advice

Andrew Ly, CEO and co-founder, Sugar Bowl Bakery

Persistence prevails. Whenever a client didn’t want to buy a new product, I would put samples and a personal note in his regular delivery, asking him to try it. If you aren’t persistent and consistent, your competition will take over.

Reward your employees. We issue prizes for good suggestions called brownie points. When enough points are earned, employees can get prizes ranging from a free lunch to an iPad.

Respond to market trends quickly. Sugar Bowl Bakery was one of the first to take trans fat out of our products, and we’re now certified organic, gluten-free, and non-GMO.

This story is from the June 1, 2015 issue of Fortune magazine.