On a recent episode of Yahoo Finance’s live morning show, “Shark Tank” investor Kevin O’Leary said he would “never” touch a marijuana stock because it’s a Schedule 1 narcotic — a designation that not even cocaine has in the U.S. (it’s Schedule II).

And most institutional investors are also proceeding with caution with marijuana investments, whether due to the volatility of public cannabis companies or the illegal status of marijuana under federal law, as U.S. and Canadian exchanges list more weed companies.

However, it’s highly unlikely that individual investors will face criminal charges over putting their money in weed stocks, several lawyers told Yahoo Finance.

‘I don’t think anybody should lose any sleep’ investing in weed

It’s true that investing in marijuana companies technically comes with a legal risk, even for retail investors with no connection to pot companies other than owning shares in them. But that legal risk is largely “theoretical” and “remote,” according to Christopher Barry, who chairs the Canada practice group at Manhattan-based law firm Dorsey & Whitney.

Two main types of publicly listed weed companies have cropped up: Those doing business in the U.S. and trading on Canadian exchanges, and those trading on U.S. exchanges but only doing business in Canada (where recreational marijuana is now legal). Investing in the marijuana companies merely doing business in Canada carries even less legal risk than the first type of company, according to Barry.

He noted the Canadian pot company Tilray (TLRY), while working on its IPO in the U.S., hired investment banks that apparently weren’t too concerned about the illegality of pot in the U.S.

“I don’t think anybody should lose any sleep at all over investing in those companies — for legal reasons,” Barry said. “Whether the stock goes up or down is a different question altogether. Those have been and may continue to be volatile investments.”

That assurance likely wouldn’t convince a wary investor like O’Leary, who in a follow-up interview with Yahoo Finance this week reiterated that he has “no interest” in investing in marijuana companies until the U.S. government has removed it from its list of Schedule 1 narcotics and it’s no longer illegal under federal law.

“This is a personal risk assessment,” said O’Leary, a Canadian venture-capital investor who co-founded O’Leary Funds and SoftKey. “I will not take that chance even if it’s .01%.”

He added: “It doesn’t mean that others shouldn’t. I’m not telling others what to do. I’m looking at the facts. The facts are this is an illegal substance.”

‘Conspiracy to violate the Controlled Substances Act’

Technically, investors who put their money in marijuana companies are conspiring to violate the Controlled Substances Act, which regulates the manufacture and distribution of certain substances.

Marijuana is classified as a Schedule 1 substance under the Controlled Substances Act, meaning it has “no currently accepted medical use and a high potential for abuse.” Other Schedule 1 drugs include heroin, peyote, and LSD. While nine states have legalized recreational weed, the federal government says pot is illegal even for medicinal purposes.

As an investor in marijuana stocks, you are technically funding an enterprise that’s illegal under U.S. law. “You have liability for that, even as an investor,” Hilary Bricken, an attorney who writes a weekly column on marijuana policy on Above the Law, told Yahoo Finance.

That’s part of the reason institutional investors have stayed away from marijuana investments, she noted, as they don’t want to expose their clients to that risk. Retail investors are also breaking the law by buying pot stocks, technically. But, Bricken says, “I don’t see the federal government pursuing individual investors.” Quite simply, she said, the feds have lower-hanging fruit to prosecute when it comes to drug crimes.

Unless investors own a significant stake in a company, there’s “little or no risk of prosecution,” according to Dean Heizer, the chief legal strategist and executive director of Colorado marijuana dispensary LivWell. “It’s a theoretical exposure,” he noted. “I’ll give you that.”

In reality, even marijuana companies themselves that operate in states where it’s legal probably won’t be prosecuted if they’re complying with states laws — even though U.S. attorneys could technically do so. As Bricken wrote in Above the Law, “such prosecutions are not likely at the top of their list given the political toxicity of punishing state-sanctioned, voter-approved marijuana businesses.”

For O’Reilly, the “Shark Tank” judge, it’s simply not worth any risk to invest in a company that sells a substance that’s illegal under U.S. law — and he points out that he’s not alone. “There’s a lot of excitement about cannabis,” he told Yahoo Finance in the follow-up interview, “but the truth is a majority of investors won’t touch it.”

Regardless of its legality under federal law, the buzz over weed isn’t likely to die down anytime soon. Four more states have ballot initiatives during next week’s midterm elections that could legalize marijuana for recreational or medical purposes.

See also: The business of weed banking is veiled in secrecy

Erin Fuchs is deputy managing editor at Yahoo Finance.

Follow Yahoo Finance on Twitter, Facebook, Instagram, Flipboard, LinkedIn, and reddit.