"The moment when a man comes to paint himself – he may do it only two or three times in a lifetime, perhaps never – has in the nature of things a special significance." So Lawrence Gowing wrote, in an introduction to a 1962 exhibition of British self-portraits. And he is right: self-portraits, whether of men or women, have a particular call on our attention. Take Käthe Kollwitz's etching Self-portrait with Hand to her Forehead, reproduced in James Hall's new book. The head and hand fill the entire plate, leaving no room for anything else. The heavy, repeated lines form dark shadows on and around the head, while the eye under the hand is obliterated by darkness. Yet her face presses forward, as if she were leaning on the kitchen table, offering us, with inescapable intimacy, a memory of the suffering and sadness she witnessed in the poor quarter of Berlin where her husband worked as a doctor. The viewer need know nothing of this: it is all there in her look.

Whatever stance they promote, be it pompous or playful or merely pleasing, self-portraits have much to say. They can show success or indulge in self-mockery; advertise a new aesthetic or celebrate a marriage. "Certainly he likes his face; what is wrong with that?" writes Gowing of the self-portraitist (again confining his thoughts to his own gender). "Naturally he is not without vanity, without a little conceit few men could bear their mirrors of their lives. Moreover, the painter's opinion of himself is a part of his equipment. He needs his conceit, as an opera singer does. Pride and a certain exultation nourish any physical skill." Again, the accuracy of this is borne out by illustrations in Hall's book; most memorably so by James Barry's cool sideways glance, or by the fresh and original pose adopted by the youthful Reynolds as he shades his eyes with his raised arm. But there is much else going on in The Self-Portrait: A Cultural History, as it departs radically from the popular belief that the sole purpose of self-portraits is to bring us into contact with the artist's soul.

It's hard to understand why self-portraits, as a genre, have until now been so little discussed. They include some of the greatest works of all time. Among those featured in this book are Velázquez's Las Meninas and Courbet's The Artist's Studio, as well as such masterpieces as the 1665 self-portrait by Rembrandt at London's Kenwood House, a painting seemingly devoid of any agenda other than what it feels like to carry into old age the weight of being human. Yet despite such riches, this genre has, until now, remained largely overlooked (Laura Cumming's recent book, A Face to the World, is an exception), existing merely as subset within portraiture, which is a relatively under-investigated subject. Perhaps the huge diversity within self-portraiture, and its leaning towards bombast, have kept scholars at bay. Be that as it may, Hall has now created a cultural map of this field, and he charts its development in terms that relate to the interests or intellectual climate of each period.

Nothing about this book is predictable. Hall, while acknowledging that portraiture was pioneered by the Egyptians, Greeks and Romans, argues that a coherent starting point for self‑portraiture is the middle ages, because it was an age preoccupied with personal salvation and self-scrutiny. One illustration, c943-57, shows a tiny St Dunstan prostrate before a giant figure of Christ, seeking absolution by touching his robe. It is one of the many surprises in this survey, for Hall's illustrations, though always deftly chosen in relation to his argument, are often little known. He presents the engraving by Israhel van Meckenem, Self-Portrait With His Wife, c1490, which is indeed as Hall describes it – "one of the most delightful and intense double portraits ever made". The two heads are squeezed together, making a tight fit within the space. Every detail compels attention: the curls escaping from Israhel's cap, the tops of his wife's breasts, pushed above the neckline of her tight dress, or the way the outline of her bonnet gives a heart-shaped frame to her face. Nothing could better explain Hall's claim that, as the 16th century approached, artists began putting themselves more in the picture – physically, socially and stylistically – for portraiture had become more naturalistic and more concerned with the individual.

Hall makes knowing use of contingent details. He is an expert on the various kinds of mirrors which aided the development of self-portraiture. He inevitably focuses at one point on the circular convex mirror on the back wall of the National Gallery's The Arnolfini Portrait and agrees with others that Van Eyck is probably one of the three figures reflected in it. He notes also the inscription on the wall above the mirror, "Jan van Eyck was here", but does not engage with the debate as to whether or not a marriage is taking place; and, if it is, whether or not we can look on the picture as the equivalent to a marriage certificate. Elsewhere, however, Hall accesses vast tracts of knowledge with an easy familiarity, as he travels through the centuries and across Europe to the present day.

It appears that, up until 1490, the production of self-portraits remained modest and spasmodic. During the Renaissance, the genre benefitted from the "heroisation" of the artist, though in time this led on to its polar opposite, the mock-heroic self-portrait. The obvious example of this is the scrunched-up likeness of his own features which Michelangelo gave to the flayed skin, held up by St Bartholomew in the Last Judgment on the end wall of the Sistine Chapel, an identification which many accept, although it was not put forward until 1925. Hall, a specialist on Michelangelo, connects this flayed self‑portrait to Michelangelo's interest in the shedding of skins, be it his own, or the outer "skin" of a marble block which the sculptor removes, in order to reveal the life within. The flaying or shedding of a skin or two becomes an act of cleansing and therefore an "imitation" of Christ's Passion.There are many instances where Hall's arguments take us down unexpected routes. He also moves his survey on through his identification of historical shifts. At one moment he discusses the importance of self-portraitists found in their studios; then, in the 18th century, he observes them buffeted by contrary passions, while in the 19th century they are shown seeking a return home, to more rooted forms of expression. The pursuit of the elusive self, it seems, never ends.

No mention of the "selfie" is made in Hall's discussion of the modern and contemporary period, but he does note that self-portraits today flood the internet and that children at school are required to make them. Van Gogh remains a role model, with his fierce expressiveness. Hall is acute on the tense, short-lived partnership between Van Gogh and Gauguin in the four-room Yellow House at Arles. Van Gogh's decision to portray himself and Gauguin by means of two rush-seated chairs is well-known. But Hall enhances perception by pointing to the way both face inwards, as if towards each other, and therefore ape paired portraits from the Renaissance where each sitter turns slightly towards the other. Van Gogh also introduces the theme of the serial-self-portraitist that continues in the work of Frida Kahlo, Cindy Sherman, Gilbert and George and Antony Gormley, to name just a few, all of whom Hall deals with lightly but tellingly.

There is never a dull passage in this book: the detail is crisply imparted; the content richly arcane at times, but more usually profoundly human; the ideas come freshly coined. Hall manages to retain the intellectual high ground while writing with verve and enthusiasm. It is a creditable achievement, and, like all the best gifts, comes beautifully wrapped, in book production of the highest quality.