If President Obama nominates Larry Summers as the next Fed Chairman he will face the opposition of Nobel prize-winning economist Joseph Stiglitz, the New York Times editorial board and, more importantly, members of his own party in Congress.

Related: Larry Summers is Too Much of a Hothead to be Fed Chair: Kai Ryssdal

At least three Democrats on the Senate Banking Committee oppose Summers, who headed the White House Economic Council in Obama’s first term. That committee will hold confirmation hearings after Obama puts forth his Fed Chief nominee. About a third of the 54 Democratic members of the Senate have signed a letter supporting Janet Yellen, current vice chair of the Fed and the other leading contender, for the job.

Senator Tom Harkin (D-IA), who signed that letter, told the The Wall Street Journal recently, “There's a lot of concern among a lot of Democrats about an appointment of Larry Summers to that long-term position as Fed chairman. He was one of the architects of getting rid of Glass-Steagall, of getting rid of other regulations."

Related: Janet Yellen: Focusing on Her Gender Sets a "Dangerous Precedent," Says Heidi Moore

The support for Yellen is not just anti-Summers. Former FDIC Chair Sheila Bair wrote in a recent opinion piece for Fortune magazine's website: “There is no better qualified candidate to fill Bernanke's shoes when he steps down in January…She was not part of the deregulatory cabal that got us into the 2008 financial crisis…[and] she had a solid record as a bank regulator at the San Francisco Fed and was one of the few in the Fed system to sound the alarm on the risks of subprime mortgages in 2007.”

Summers has some important backers too, including Former Treasury Secretary Bob Rubin, whom he succeeded as Treasury Secretary under President Clinton; Steve Rattner, a former investment banker who headed Obama's auto task force; and Roger Altman, former Treasury Secretary under Clinton and co-founder and chairman of Evercore Partners, a private equity firm.

Related: The $4 Trillion Choice: Why the Stakes Are So High for the Next Fed Chair

But beyond the qualifications of either candidate are pure and simple politics—and that could determine who Obama nominates.

“Some Democrats are really going to put up a battle if he’s chosen,” says Chris Moody, political reporter for Yahoo News, about a Summers nomination. Given that many Congressional Democrats—as well as Republicans—are already opposing Obama’s push to attack Syria for using chemical weapons against its own civilians, the “last thing he [Obama] wants to do is pick another fight with his own party, says Moody. “That might make him hesitant, if he’s looking at the political calculus, to pick Larry Summers.” Confirmation hearings for Yellen, in contrast, would “go a lot more smoothly,” says Moody.