(Adds White House comment, updates markets)

* Economy contracts 3.5% in 2020

* Fourth-quarter GDP increases at 4.0% rate

* Double-digit growth in Q4 business, residential investment

* Consumer spending slows, inventories increase

* Weekly jobless claims fall 67,000 to 847,000

By Lucia Mutikani

WASHINGTON, Jan 28 (Reuters) - The U.S. economy contracted at its deepest pace since World War Two in 2020 as the COVID-19 pandemic depressed consumer spending and business investment, pushing millions of Americans out of work and into poverty.

Though a recovery is underway, momentum slowed significantly as the year wound down amid a resurgence in coronavirus infections and exhaustion of nearly $3 trillion in relief money from the government. The moderation is likely to persist at least through the first three months of 2021.

The economy's prospects hinge on the distribution of vaccines to fight the virus. President Joe Biden has unveiled a recovery plan worth $1.9 trillion, but some lawmakers have balked at the price tag soon after the government provided nearly $900 billion in additional stimulus in late December.

White House economic advisor Brian Deese said the report from the Commerce Department on Thursday underscored the urgency for Congress to pass Biden's plan, warning that the cost of doing nothing was too high.

"Without swift action, we risk a continued economic crisis that will make it harder for Americans to return to work and get back on their feet," said Deese.

Gross domestic product decreased 3.5% in 2020, the biggest drop since 1946. That followed 2.2% growth in 2019 and was the first annual decline in GDP since the 2007-09 Great Recession.

Nearly every sector, with the exception of government and the housing market, contracted last year. Consumer spending, which accounts for more than two-thirds of the economy, plunged 3.9%, the worst performance since 1932. The economy tumbled into recession last February.

Delays by the government to offer another rescue package and renewed business disruptions caused by the virus restricted GDP growth to a 4.0% annualized rate in the fourth quarter. The big step-back from a historic 33.4% growth pace in the third quarter left GDP 2.5% below its level at the end of 2019.

The economy is expected to return to its pre-pandemic level in the second quarter of this year.

The Federal Reserve on Wednesday left its benchmark overnight interest rate near zero and pledged to continue pumping money into the economy through bond purchases, noting that "the pace of the recovery in economic activity and employment has moderated in recent months."

With the virus still raging, economists are expecting growth to slow to below a 2.0% rate in the first quarter, before regaining speed by summer as the additional stimulus kicks in and more Americans get vaccinated.

"We foresee record-breaking consumer spending growth in 2021 with households benefiting from a watered-down $1.2 trillion version of Biden's rescue plan, vaccine diffusion gradually reaching two thirds of Americans by July and employment accelerating this spring," said Gregory Daco, chief U.S. economist at Oxford Economics in New York.

Stocks on Wall Street rallied as mega-cap technology shares tried to recoup recent losses. The dollar slipped against a basket of currencies. U.S. Treasury prices were lower.

K-SHAPED RECOVERY

Services businesses like restaurants, bars and hotels have borne the brunt of the recession, disproportionately impacting lower-wage earners, mostly women and minorities. That has led to a so-called K-shaped recovery, where better-paid workers are doing well while lower-paid workers are losing out.

The stars of the recovery have been the housing market and manufacturing as those who are still employed seek larger homes away from city centers, and buy electronics for home offices and schooling. A survey by professors at the University of Chicago and the University of Notre Dame showed poverty increased by 2.4 percentage points to 11.8% in the second half of 2020. The sharpest rise since the 1960s boosted the ranks of the poor by 8.1 million people.

Rising poverty was highlighted by persistent labor market weakness. In a separate report on Thursday, the Labor Department said initial claims for state unemployment benefits totaled a seasonally adjusted 847,000 for the week ended Jan. 23. While that was down 67,000 from the prior week, claims remain well above their 665,000 peak during the 2007-09 Great Recession.

Including a government-funded program for the self-employed, gig workers and others who do not qualify for the regular state unemployment programs 1.3 million people filed claims last week.

The economy shed jobs in December for the first time in eight months. Only 12.4 million of the 22.2 million jobs lost in March and April have been recovered. About 18.3 million Americans were receiving unemployment checks in early 2021.

"The labor market is struggling this winter, but better times are ahead," said Ryan Sweet, a senior economist at Moody's Analytics in West Chester, Pennsylvania.

Lack of jobs and the temporary expiration of a government weekly jobless subsidy curtailed growth in consumer spending to a 2.5% rate in the fourth quarter after a record 41% pace in the July-September quarter.

But business investment grew at a 13.8% rate, with spending on equipment rising at a 24.9% pace. Spending on nonresidential structures rebounded after four straight quarterly declines.

Businesses also accumulated inventories last quarter, contributing to GDP growth. But the inventory build pulled in more imports, leading to a larger trade deficit, which subtracted from output. The housing market recorded another quarter of double-digit growth, thanks to historically low mortgage rates. Government spending was weak.

(Reporting by Lucia Mutikani; Editing by Chizu Nomiyama and Andrea Ricci)