The shrinking job market is also shrinking the universe of stocks that investors should own, says Jim Cramer, host of CNBC's "Mad Money" show and author of the new book, Jim Cramer's Get Rich Carefully.

"Employment growth is the most important thing for profits and the profits have to keep going up," Cramer tells The Daily Ticker.

Related: Why there's no reason to panic over the jobs report

He's very concerned about the stock market now because of the latest jobs data. On Friday the government reported that employers hired only 113,000 more workers in January -- 40% below expectations, more than the 75,000 reported for December. The unemployment rate, however, fell to 6.6% and reportedly for the right reason: people finding jobs rather than dropping out of the workforce. Still 6.6% is a relatively high jobless rate more than four years into an economic recovery.

"I'm very concerned right now," says Cramer, about the U.S. job market. "I am not going to buy into [the reason] that’s it's all weather," referring to one explanation for January's sluggish payroll growth.

Cramer says the weak job market -- 3.6 million remain unemployed for six months or more -- is why retail and auto stocks should do "poorly."

"I'm limited in the themes I like because there is a down-tick in the economy and I would like to see something that tells me it's temporary," says Cramer.

One key factor for the market's future is Washington, and if past is prelude, there's reason to worry, according to Cramer.

After looking at market charts for the past three years, he says "All the big dips are because of Washington...I don't think that's changed...We did get that deal that averted a shutdown and that was very positive...but I never trust Washington. Never."

Related: “We should have a contentious debate” over the debt ceiling: Sen. Isakson

The next big challenge in Washington: the debt ceiling. It was officially reached on Friday, Feb. 7, but the Treasury has enough funds on hand to keep funding the government through Feb. 27, according to Treasury Secretary Jacob Lew. 

Janet Yellen, the new chair of the Federal Reserve, testifies before Congress Tuesday and Thursday for the usual semi-annual presentation to lawmakers about Fed policy.

"She's terrific," says Cramer. "I think the Fed really saved us."

But interest rates are rising in part because the Fed has begun its retreat from aggressive asset purchases, and those rising rates are hurting the financial sector as well as the overall stock market.

"I was thinking Bank of America (BAC) and JPMorgan (JPM) [look] pretty good here, but if the yield curve stays like this you're going to have shortfall," says Cramer.