Most undergraduates at Harvard live in a ‘House’. When I went to the university in 1947 the place was overcrowded with soldiers and sailors returned from the war. So I spent the first two years in Dudley Hall, the ‘non-residents’ student centre’ in Harvard Yard. My room was number 46: I know this precisely because I have a letter sent to this address by Albert Einstein, dated 3 June 1949, telling me he did not give ‘oral interviews to avoid misinterpretation’. Since Dudley wasn’t considered a House it didn’t have a ‘master’ but a ‘graduate secretary’. Then in autumn 1949 I moved into Eliot House.

My assignment to Eliot – it was an ‘assignment’ – had nothing to do with my intellectual interests. The master, John H. Finley Jr, was a classicist. I do not think he had the remotest interest in science. I thought of him as a snob, the perfect choice for a House with a reputation for lodging ‘preppies’ and members of ‘final clubs’ such as the Porcellian. In an interview with the New York Times he asked: ‘Where else would you find, in one room, the grandson of Matisse, the grandson of Joyce and the great-great-great-great-grandson of God?’ He meant the fourth Aga Kahn. Eliot was then like all the Harvard Houses all male.

Since I was quite sure that Finley didn’t have the foggiest idea who I was, I was astounded when at the end of my senior year he asked me to organise the entertainment for the senior dinner. The only ‘entertainment’ I had been involved in during my college days was to play an exceedingly mediocre trumpet in the Harvard-Radcliffe Symphony Orchestra. I did not write for any of the college papers or take part in any of the dramatic societies. What he had in mind when he chose me I have no idea. He told me that one of the entertainers was going to be a senior who played the recorder but the rest was my problem. Tom Lehrer was in graduate school and had just begun to perform around Harvard. I knew him a bit (we were both mathematicians) and asked if he would perform at our dinner. Al Capp, the creator of Li’l Abner, was a family friend and he agreed to talk. Capp was still in his raucous liberal days, an unguidable missile and just what Finley deserved.

The dinner went much better that I had imagined. After we had suffered through the recorder, Lehrer sang several of his songs. He was wonderful and Capp was so impressed that he had him as a regular guest on a radio programme he then had in Boston. Capp’s talk began: ‘I am glad to be part of an experiment on the effect of beer on the human kidney. I enjoyed the songs about murder, rape and John Finley.’ In a photograph taken at this precise moment Finley has the look of someone who has mistakenly eaten a persimmon.

I played the trumpet because the schools in Rochester, New York, where I grew up, had a programme in which every pupil learned to play a musical instrument. When we moved to New York City during the war I took trumpet lessons from Edward Treutel, for a while the only trumpet teacher at Juilliard but I saw him at the 92nd Street YMHA. He spoke with great admiration of a trumpet player in the NBC Symphony – conducted by Toscanini – named Harry Glantz. One Saturday I snuck into one of Toscanini’s rehearsals at the NBC studios in Radio City. I introduced myself to Glantz and told him I was a student of Treutel. He said he had never heard of Treutel and that ended that. But on another Saturday I snuck into another studio and this changed my life.

There was a large group of African Americans with their instruments close at hand smoking and relaxing. A man with a saxophone at his side was studying a chessboard. I asked him if he would like a game. A man sat down at the piano and played some chords. All the musicians got up and, walking to the stage, played in an ensemble. I had never seen anything like this display of virtuosity. I soon learned that the man who had sat down at the piano was Duke Ellington and they were playing ‘Take the A Train’. The man I had been playing chess with was Al Sears, one of the best tenor saxophone players of his era. He and I became friends.

After the war we moved back to Rochester and Sears spent the night in our house when the orchestra was playing in nearby Binghamton. Not long after I moved into Eliot House the orchestra came to Boston to play. I invited Sears to Sunday lunch at Eliot. I had given no thought to the fact that he was African American. He was a great musician and I was proud that he was my friend. I also had given no thought to what Sunday lunch at Eliot House was all about. You brought your parents and your best girl and everyone was white. People stared at Sears and at me. I didn’t care but Sears did. He was very uncomfortable and left as soon as he could. I never saw him again. He left Duke Ellington that year and died in 1990.

Harvard in general and Eliot House in particular has changed a lot since I left the university in 1957. I do not know who the first African American to be admitted to Eliot House was, or when this came about. But I do know that among its distinguished alumni is the actress and writer Rashida Jones, the daughter of Quincy Jones. I imagine that if she had brought her father to lunch people would have asked for his autograph. At the end of last year the House masters at Harvard unanimously voted to change their title because it smacks too much of slavery. A new title has yet to be chosen.