Story highlights Carson had been rumored as a potential pick for several cabinet posts

Carson was a key adviser and top surrogate during Trump's presidential campaign

(CNN) Ben Carson declined an offer from President-elect Donald Trump to join his cabinet as Secretary of Health and Human Services, a close Carson adviser told CNN on Tuesday.

Carson confidant Armstrong Williams said the former neurosurgeon and top Trump surrogate will continue to advise Trump from outside the White House but declined Trump's offer to join his administration as a cabinet-level official.

"He's never run an agency and it's a lot to ask. He's a neophyte and that's not his strength," Williams said, despite the fact that Carson vied for the Republican nomination to be the next president of the United States.

Carson had been rumored as a potential pick to head up the Department of Health and Human Services or the Department of Education.

The Trump transition team did not respond to a request for comment.

Read More