By Kevin Kelleher, contributor

Sorry, small investors. If you want to buy shares in Facebook, Groupon or Twitter, you’re going to have to wait until their IPOs.

That much is certain now that the Securities and Exchange Commission has started looking into the trading of shares in privately held companies. What’s less clear right now is how SEC’s involvement will affect the market for buying and selling of shares in fast-growing startups. While new regulations could shut down the market before it has a chance to take off, it looks more likely that the SEC’s interest could prompt the market to evolve in a way that minimizes risks for investors.

“I think it’s a good thing the SEC is looking into this area,” said Robert Heim, an attorney at Meyers & Heim who previously worked at the SEC. “I suspect we may see more firms that provide these markets registering as broker dealers and falling under a more regulated structure.”

Until last year, such so-called secondary transactions took place in a sleepy little niche in the equity markets. But a new crop of companies like SecondMarket and Sharespost started running eBay-like auctions for equity in private startups, drawing a torrent of interest from institutional investors and retail investors alike for an early entry into the next big tech star.

As demand rose, so did the valuations that the auctions accorded to startups: Facebook’s putative value has doubled from $30 billion in September to $60 billion last month. (Amazon, by comparison, is worth $80 billion.) Reports of surging valuations put the new secondary markets sites on the radar of many investors, but they may also be drawing regulatory scrutiny.

“It boils down to the reality that you have trading in stocks of companies that makes them look like public companies,” said Gary Aguirre, a former SEC attorney now at the Aguirre Law Firm. “Accredited and institutional investors can fare on their own. But if it slips down the food chain to retail investors who are clueless about what they are getting into, then it can get very sticky.”

Securities laws define accredited investors as banks, pension funds or charities with assets exceeding $5 million, as well as individuals with a net worth above $1 million or income exceeding $200,000 a year.  Heim said regulators may want to see how closely the secondary-market firms are vetting investors as accredited before they let them participate in stock transactions.

Who in the secondary market survives an SEC investigation

Any fallout from the SEC’s scrutiny of the private secondary markets is likely to hurt newer entrants who aren’t licensed as broker dealers. Licensed firms with a longer history – such as SecondMarket, founded in 2004, and NYPPEX, which has been in the private secondary markets since 1998 – are likely to weather a shakeout.

SecondMarket has emerged as the largest trader of private stock in big-name web startups. The company began by trading illiquid securities such as restricted stock in public companies and auction-rate securities during the market turmoil of 2008. After a former Facebook employee approached the company about selling some of his shares in the company, SecondMarket began exploring such transactions in April 2009.

SecondMarket charges a transaction fee between 3% and 5% of the proceeds raised from all private equity transactions, which are averaging about $2 million in size. Such fees make up less than half of SecondMarket’s revenue, but is its fastest growing area of its operations. In 2010, the company saw $100 million from transactions in private companies, a figure that rose to $400 million last year.

“The number of buyers is also growing,” says Jeremy Smith, SecondMarket’s chief strategy officer. “They pretty much run the gamut of the investment community. It’s now venture capital funds, hedge funds, asset managers and high net-worth individuals.”

Another hazard of the secondary private market is that poor disclosure, since investors aren’t guaranteed the trove of financial information in a prospectus for a public offering. Smith said SecondMarket sets up online “data rooms” that offer potential buyers access to documents such as articles of incorporation, financial statements or management commentary on the company that buyers can use in valuing a company’s shares.

To encourage companies to share such information, SecondMarket allows the companies to decide who can buy or sell shares. “It gives them a level of control, so that they don’t have a competitor buying the stock,” said Smith. “By making them more comfortable this way, you make them more comfortable revealing information.”

SecondMarket sees the secondary-transaction market growing beyond a niche as private companies in other industries like biotech or consulting start participating. The company is exploring a move next year into primary markets as well, helping companies to sell shares directly to institutional investors as an alternative to venture capital.

The challenge for firms like SecondMarket as they explore new ways to help facilitate equity transactions for private companies is to evolve while remaining in compliance with securities laws. For its part, the SEC will need to update those laws in a way that adapts to those evolutions while still protecting investors.

“I believe the SEC recognizes these firms provide a valuable service that is in compliance with the SEC guidance,” said Heim. “But we’re dealing with securities laws passed in the 1930s and Internet technology made in the 21st Century. The SEC will have a challenge in creating regulations that are up to date and protect investors in these securities.”