And he said any suggestion that the comedian had been "forced to quit" Mock The Week or was sacked was "completely untrue".
Mr Sherborne said Boyle's humour was "deliberately challenging".
He said the comic did not object to being criticised but did object to being described as racist.
"Saying 'vile' or 'offensive', or his material is 'vile' or 'offensive', is one thing," Mr Sherborne told the jury. "He realises that that goes with the territory, so to speak. But accusing him of being a racist is an entirely different matter."
Mr Sherborne added: "You can call him 'offensive', you can call him 'tasteless'. That's fine. But he is not racist."