The Cabinet Office says the move is "by mutual agreement" but it follows months of negative media reports about his performance.