Are you better off today than four years ago?

That's a question frequently posed on the campaign trail these days as the country heads towards its next presidential election.

But perhaps there is a much better question to be asking yourselves: Are you better off than you were 16 years ago?

[Related: Paul Ryan a Bold VP Choice but Romney/Ryan Plan Will Hammer the Middle Class]

That's when the median annual household income of Americans peaked at $54,932. By 2011, it had fallen 9% from its height, according to the latest data from the Census Bureau released Wednesday.

"The middle class has stagnated," says Hedrick Smith, former Pulitzer Prize-winning New York Times reporter, now author of "Who Stole the American Dream?" "The average pay of a male worker in America was a little lower in 2011 than in 1978, that's three decades of going nowhere."

[Related: The Betrayal of the American Dream]

This decline of the standard of living for the middle class is hurting the broader economy, says Smith. "We're looking at a slow recovery from our economic collapse of 2008, and one of the main reasons is there's no consumer demand driving growth, causing companies to hire and to expand…people are not being paid enough and haven't been paid well enough for the last three decades." As a result, companies "sitting on a couple of trillion dollars….(are) not investing it," he tells The Daily Ticker's Henry Blodget in the accompanying interview.

[Related: Dear Walmart, McDonald's, Starbucks: How Do You Feel About Paying Your Employees So Little That Most Of Them Are Poor?]

What companies should be doing, says Smith, is paying their workers more. They shouldn't impose a wage freeze like Caterpillar did at a hydraulic parts factory earlier this year despite record profits, says Smith. Unionized workers at the plant then went on strike, but after 3-1/2 months settled for a contract that included a 6-year wage freeze for veteran workers and reduced health care and pension benefits.

"The economy would grow faster if workers were paid more," he says.

[Related: Apple Store Workers Get 25% Raise: Company 'Finally Recognizing Value We Provide,' Employee Says]

It might also grow faster if more of the U.S. economy was based on manufacturing rather than services. Smith cites Germany where 21% of its labor force works in manufacturing compared to 9% in the U.S. and where workers pay over the past 25 years rose five times as fast as in the U.S.

Germany also had a $2 trillion trade surplus while the U.S. had a $6 trillion trade deficit.