London (CNN) Prince Andrew has been accused of using the n-word at a 2012 meeting in Buckingham Palace by a former senior government aide, a claim that is being vehemently denied by a royal source.

Writing in the London-based newspaper the Evening Standard, Rohan Silva -- a former aide to ex-Prime Minister David Cameron -- said when he asked the Prince whether a government department responsible for trade "could be doing a better job," Andrew responded: "Well, if you'll pardon the expression, that really is the n***** in the woodpile."

A royal source told CNN that the claims are strenuously denied, and that point was made clear in a legal letter to the Evening Standard. CNN has asked the Evening Standard for a response to the letter.

Silva, who is of Sri Lankan descent, is now an Evening Standard columnist. He said it was the second time he had heard the Prince using unacceptable language in a palace meeting, which was attended only by himself, the Prince and a royal aide.

"For a long time after, I kicked myself for not confronting the prince on his choice of words -- and it's something I still regret today. He clearly wasn't taken to task very often by the people around him, which meant offensive language could go unchallenged," the 38-year-old wrote.