Many people pay federal income tax on Social Security benefits. How much of your Social Security is taxed depends on how much income you have from other sources in addition to your benefits. One way to tell if you might have a tax bill is to add half your annual benefits to your other income... Liz Weston is a writer at NerdWallet.