A big rally in the bond market has resulted in a sharp drop in mortgage rates. For the week ended Feb. 6, average 30-year fixed rate loans were 4.23%, down from 4.53% at the end of 2013.

And while rates remain above the all-time lows of 3.5% seen in 2012, the recent fall represents a great opportunity for millions of Americans to refinance and lower their costs. But, thus far, refinancing activity remains subdued meaning "a lot of consumers are leaving money on the table," says Steve Deggendorf, a director in Fannie Mae's economic and strategic research group.

[Click here to check home loan rates in your area.]

According to Fannie Mae, between 40%-50% of Americans with a mortgage have never refinanced and have an average mortgage rate around 6%. Since there are roughly 50 million mortgages outstanding, that means 20 million-plus households could potentially refinance. 

According to Fannie Mae's national housing survey, the top reasons people don't refinance are:

• Payments aren't reduced enough.

• Closing costs are too high.

• Don't want to extend term of loan.

• Trust issues with lender.

• Under water on mortgage or other qualification issues.

While these are all legitimate concerns, Deggendorf says most can be addressed. For example, it is possible to get a refinancing with zero or very low closing costs (they will be built into the loan) and underwater homeowners who are current on their loans may still be eligible for the government's HARP program, which runs until the end of 2015.

"Given the large percentage [of Americans] who never refinance, it's really important for them to step back and determine whether there's an opportunity for them to get this big benefit that's potentially out there," says Deggendorf.

The good news here is there's no harm in looking into whether a refi makes sense -- no legitimate lender will charge you to inquire -- and there's no obligation if the numbers don't make sense. So get started by checking rates in your area and go to Yahoo Homes for more information about housing finance.

Aaron Task is the host of The Daily Ticker and Editor-in-Chief of Yahoo Finance. You can follow him on Twitter at @aarontask or email him at [email protected].