Ask the Fool Pay mortgage with your IRA? Q: I'm thinking of paying off my mortgage with money from my IRA. Should I? - B.C., Greensburg, Pa. A: Think twice about it. With a traditional IRA, if you're younger than 59 ½, withdrawals will be taxed at your ordinary income tax rate, and you'll face a 10 percent early withdrawal fee, too.