The former cabinet minister says he does not want to "cause further embarrassment" by his links to the late convicted paedophile.