The first time I saw Jerry Lewis, it was on television with his partner Dean Martin. This was in the late 40s when TV was just beginning – the medium was new and so were Martin and Lewis. We were used to comedy “teams” such as Abbott and Costello, the straight man who fed the lines to the comedian, which grew out of vaudeville. Martin and Lewis took it all to another level. Martin himself was funny, and he was also smooth, romantic, and he sang. They got into a groove and Lewis would take off into pure anarchy. Pretty soon, they were stars of the big screen as well – they started in the My Friend Irma pictures and then they became the main attraction: 14 pictures beginning with At War With the Army in 1950 and ending with Hollywood or Bust in 1956. And at some point in the early 50s – I don’t recall the year – I saw them live at the Paramount theatre in New York.

‘We got to be good friends’ … Martin Scorsese, right, with Jerry Lewis in New York, 2016. Photograph: Kevin Mazur/WireImage

Artists and Models and Hollywood or Bust were directed by Frank Tashlin, who had come out of Leon Schlesinger’s legendary animation unit at Warner Brothers and developed a new kind of movie comedy in the 50s. His pictures were as visually inventive as Schlesinger’s Looney Tunes series and just as liberating, because absolutely everything was fair game. After Martin and Lewis split – we didn’t really know why at the time – Tashlin directed most of Lewis’s first solo pictures, from Rock‑A-Bye Baby in 1958 to Who’s Minding the Store? in 1963.

In 1960, Lewis did something interesting – he started directing his own pictures, with The Bellboy, in which he played an attendant at a Miami hotel who doesn’t say a word until the end of the picture. In order to get a sense of himself in the frame of his own picture, he used a very early version of what has become a standard tool of directing: the video assist.

Dean Martin, Jerry Lewis, Shirley MacLaine and Dorothy Malone in Artists and Models (1955). Photograph: Allstar/Paramount

In 1971, Lewis published a book called The Total Film-maker. The title seems absolutely right to me. When you watch The Nutty Professor or The Bellboy or my favourite, The Ladies Man, you’re watching the work of someone who is using every single aesthetic element of the cinema, sometimes to the point of making film‑making itself into a gag. Lewis definitely learned a lot from Tashlin, but as a director I think he was temperamentally closer to the French director/performer Jacques Tati. In fact, the amazing “dollhouse” set for the women’s boarding house in The Ladies Man, which took up two soundstages at Paramount, reminds me of the elaborate airport set for Tati’s Playtime. Both pictures stretch the possibilities of time, space, sound and colour to their limits, well beyond what we were used to seeing in most comedies. In certain sequences from The Ladies Man, you feel like you’re deep within the mind of Lewis’s character, a caretaker at the boarding house – for instance, the amazing moment when he enters an all-white forbidden room to find a bat lady hanging from a ceiling, who backs him into a wall that gives way to a gleaming white set against a painted blue sky, where Harry James and his orchestra suddenly materialise.

In the early 80s, I got the chance to work with Jerry on The King of Comedy. By that time, he had become a legend in movies and on the late night television talk show circuit, which was a world unto itself – one that I loved and that I was paying homage to in the picture. It was a world in which professionals such as Jack Paar and Johnny Carson and regular guests like Don Rickles and Jerry used all of their performing skills, the sense of timing and rhythm that they’d honed throughout the years to the finest point, to create a feeling of relaxation and intimacy with their audiences. We were looking at the other side of the equation, namely the people in the audience who took the intimacy literally, and the very thin line between love and hatred that could develop when they learned that it was all just a beautiful illusion. I remember Jerry telling us many stories about his own experiences, one of which we incorporated into the picture.

Lewis with Anita Ekberg in Hollywood or Bust (1956). Photograph: Allstar/Paramount

Jerry Langford was an uncomfortable role for him to play, because he was skirting the edges of his own life in absolutely every scene. Sometimes it went beyond that: he was wearing his own clothes, he was playing scenes where he was often expressing his own feelings about showbusiness and celebrity, and at times you didn’t know if you were seeing Jerry Langford or Jerry Lewis. And through it all he was, needless to say, a consummate professional. It was a remarkable and moving experience to watch him at work, improvising with Bob De Niro and the other actors and with his old friend Freddie de Cordova – I felt like I was watching a virtuoso pianist at the keyboard. And he knew his way around live television so well that I asked him to direct some of the actual on-air sequences for The Jerry Langford Show.

The last time I saw Jerry was at a New York Friars Club event. He had, by that time, surpassed legendary status. He had become something like an immortal

I started out in movies in the late 60s, working independently, and I initially thought of making movies as something that you did with friends, the people you always knew were going to be there no matter what – that was the spirit I’d grown accustomed to. I’d made some big-scale pictures and I’d worked with many actors and artists you could call “professional”, but I still felt like I didn’t belong to the world of Hollywood movie-making. When we started The King of Comedy, we were shooting at night, which is always difficult, and we were finding our way in. And there were delays. We had scheduled Jerry to come in, but then we hadn’t reached his scenes, so he was basically sitting in his trailer and waiting for us to be ready for him. On the third night, he asked to see me in his trailer. I sat down, he looked me in the eye, and he spoke very quietly but firmly. “Look, I’m a professional,” he said. “So when you tell me to be here at 9pm, I’m going to be here. You’re paying for my time. All I ask is if you find that by two o’clock in the morning you’re not going to need me, just let me know.” I already had a great deal of respect for Jerry, but at that moment it deepened. It was a moment of clarity for me, because it helped me to realise that you could work with people who weren’t friends, people who came from different worlds who perceived the work differently. Jerry taught me that being a professional was a serious matter.

Lewis, left, with Robert De Niro in The King of Comedy (1982). Photograph: Allstar/20th Century Fox

We got to be good friends, and we saw each other fairly often over the years. A couple of years ago, we did a conversation at the Museum of the Moving Image in Queens. He had been through some serious health problems but he was still coming out, still out there – the consummate professional. Someone asked him what he thought was the most important thing he’d learned in all of his years at work. “If I’m working on a picture and I’m not enjoying it, then I know there’s something wrong.” It surprised me that he responded that way at his age. It sounds very simple, but it’s absolutely essential. Making a movie is tough work. It can be punishing. So you need to enjoy it, every aspect of it. The good and the bad. You need to have that personal connection.

The last time I saw Jerry was at a New York Friars Club event. He was wearing that red blazer, he was 91 years old, and he was still out there. He had, by that time, surpassed legendary status. He had become something like an immortal. It sounds corny, but it’s true. We planned to get together after that, but the last time he was supposed to visit I was told ahead of time that he wasn’t going to be making it. Jerry was cancelling a gig, and that’s how I knew that the end had really come. As I said, he was a true professional who took the greatest pride in his work, whether it was making a movie or doing an interview or hosting a gathering. Seeing him and listening to him was always a source of renewal for me. And in my memory, it always will be.