Was Sigmund Freud a great medical scientist who uncovered important truths about human psychology, or was he something different - a gifted artist, a philosophical visionary who re-imagined human nature and helped us confront taboos, but whose theories, offered as science, fail under scrutiny? His critics and his devotees see the answer lying at opposite extremes, having him all magician or all messiah. Was he either?

Next month, the first major translations of Freud's work for over 30 years will be published by Penguin. Under the general editorship of Adam Phillips, the 15 volumes will include Freud on the unconscious, on jokes, and on dreams and hysteria. This large endeavour of scholarship prompts serious questions about the nature of Freud's contribution and his legacy.

Hostile criticism of psychoanalytic theory takes different forms. In its early years it met strong opposition from the medical establishment, but it later became fashionable, and quickly evolved into an orthodoxy, despite its fragmentation into opposing schools. In the last few decades it has met renewed opposition, not least from feminists, who have objected to the misogyny and patriarchalism they find in Freudian views, and to the spurious scientific authority accorded the idea that women are inferior because they are hommes manqués, castrated - and therefore lesser - versions of men. Defenders of Freud respond by attributing feminist attacks to "resistance" and sublimated penis envy, both technical Freudian concepts behind which they readily claim shelter.

Criticism from science and philosophy is not so easily deflected by appeal to technicalities, however, because what they question is the very basis of the theory. Karl Popper, Frederick Crews and Adolf Grünbaum figure largely in this attack. Among other things, they charge that the empirical basis of psychoanalysis is inadequate, that its central concepts are untestable, and that its aim - which is to give a complete theory of human nature - is overambitious. Its methodology is inadequate, they argue, because it rests on speculation and subjective insights, not on objective examination of public and repeatable phenomena. It depends on generalisation from single cases or very small samples - much of Freud's data is provided by a few Viennese women together with his own "self-analysis" - and its reasoning relies on analogies, subjective associations, memories real or supposed, puns, and ("unconsciously intentional") mistakes and coincidences . It assumes that mental activity is causally deterministic, and a number of philosophers, among them Wittgenstein, note that Freud also conflates the concepts of an action's causes and the reasons why it was performed.

In all this, a central and unquestioned role is assigned to the "unconscious mind". At base, Freud's theory rests on a claim, which, expressed unadorned and without preamble, looks frankly absurd: that an infant sexually desires its parent of the opposite sex, and is therefore hostile towards, because jealous of, its parent of the same sex; and that because neither the desire nor the hostility is acceptable, these feelings are repressed into the unconscious, as a result of which internal conflicts arise; and that this - the Oedipus complex - is the key to human nature. It is not, note, the key only to pathological human nature: but to human nature as such.

Freud did not invent the notion of the unconscious. By the end of the 19th century it had become a fixture of psychological medicine, accorded prime clinical significance by such practitioners as Jean-Martin Charcot and Joseph Breuer. Freud inherited an elaborate theory of the unconscious stating that many perceptual and cognitive processes occur at subliminal levels, and that what is consciously learned can become automatic and remain non-consciously effective. It also held that many memories and beliefs can be stored in the unconscious entirely without the knowledge of its possessor, but can be recovered by, for example, hypnosis. It further held that the unconscious has a creative, mythopoeic capacity responsible for dreams, stories, symbols and ideas, which when pathological is responsible for delusions and hysterical symptoms. And finally it held that the unconscious is the source of psychic energy, which can be inhibited, sublimated or transferred from one application to another, and that within it can coexist "dissociated" (split-off) sub personalities which manifest themselves in dreams or trances.

Freud's predecessors had relied on hypnosis to gain access to the unconscious, but under the influence of Breuer's experience with "Anna O", Freud adopted instead the "talking cure", the process of eliciting, by the technique of "free association", experience which had been repressed into subliminal regions of the mind. The aim was to release patients from neurosis, conceived as the baleful product of suppressed, and therefore unacknowledged, trauma by effecting a discharge or catharsis of the emotions involved. Freud was struck by the fact that most of his patients seemed to have real or imaginary sexual trauma at the root of their difficulties. When he found them reluctant to tell him about their early sexual experiences and masturbatory fantasies, he diagnosed "resistance".

Notoriously, he initially thought that his patients had indeed been sexually abused as children; later he decided that these were fantasies, expressing "infantile wishes" as described in the Oedipus theory.

A dispassionate evaluator of these views is bound to question Freud's version of the "unconscious mind". Today's cognitive and neurological science agrees that much information is processed in the central nervous system non-consciously, and that learned routines can be performed in the absence of self-awareness, without needing a notion of an "unconscious mind" to explain either phenomenon. The idea of non-conscious processes is very different from that of an "unconscious mind" conceived as a source of psychological motivation. Ideas of repressed trauma, subpersonalities, psychic energies, mythopoeic functions and mental life unknown to its possessor, constitute a speculative and highly questionable mixture. But it is precisely on these concepts that Freudian theory depends.

There are two reasons, one conceptual and one empirical, for scepticism about the unconscious. The conceptual reason is that the very idea of mental life - of motives, emotions, reasons and intentions - seems essentially conscious. It is true that we need a distinction between what philosophers call "occurrent" and "dispositional" mental states, meaning by the former, those that are currently at the centre of attention, and, by the latter, those that are not but could become so. But the Freudian unconscious is a different thing, a realm where emotions, beliefs and motives are in lively play, just as they would be in conscious mental activity, with the difference that if we ever come to know of them, it is only by their indirect effects. To some critics this is like saying that we suffer unfelt pains; hence the expression "unconscious mind" appears paradoxical.

The empirical difficulty is that the evidence for unconscious mentation is at best ambiguous, and that there is no rigorous empirical way of testing claims about it. The objection is not that unconscious phenomena are by definition unobservable, for science admits many unobservable entities, whose existence and properties can only be determined by such indirect means as inference or the detection of remote effects. In these cases, however, there is a principled correspondence between the indirect evidence and the postulated entities, which is repeatable, testable and predictable. In sharp contrast, in the "laboratory" of the analyst's consulting room, where free association of ideas is the means of enquiry, where anecdotal and subjective material constitute the data, and where "suggestion" by the analyst is an ever-present danger, no such empirical control is possible.

To doubts about the reliability of Freud's methods - free association, reliance on subjectivity, the dangers of suggestion - have to be added doubts about the characteristic style of Freud's reasoning, exemplified by his analyses of dreams, and by his claims to be able to recognise intuitively what something "truly means". For example, he reports that during self-analysis he recalled a childhood incident in which he snatched flowers from a girl. "To take flowers from a girl," he states, "means to deflower her." In another case, he diagnoses a patient's impulse to go for a run after each meal as a wish to kill a man whom he sees as a rival for his fiancée's affections. The rival's name is Richard; running after meals is a way of losing weight; "dick" is German for "fat"; Dick is a diminutive of Richard; the runner wishes to get rid of "dick"/Dick; QED.

Why should one accept that these speculative leaps have landed on the right place, or even that there is a right place to land, given that there are any number of competing interpretations one could give? Why might the runner not be wishing to kill himself by burning off more calories than he eats? Or make himself more athletic and therefore more attractive than his rival? Or find that running after meals soothes his anxieties about his fiancée's loyalty? In the flower story, would the young Freud not have snatched what was in the girl's hand if it had not been a bouquet? What if it had been a stick, or an ice-cream? If he had snatched either, what would it have "meant"? Why is it not the snatching that matters, no matter what object is snatched, when deciding how to give an interpretation?

In a letter to Jung, Freud spoke of his "serene confidence" in his methods; few critics can share it. One has only to remember the case of Emma Eckstein. Freud claimed to have discovered a "nasal reflex neurosis" and linked it to excessive masturbation, and diagnosed Emma as suffering from it. He got his colleague Wilhelm Fliess to remove the turbinate bone from her nose. After her nose had bled and suppurated for many days, another surgeon found a mass of surgical gauze left in the wound. Its removal caused a near-fatal haemorrhage. The bleeding continued intermittently for months. Freud meditated on the problem, and concluded that Emma's bleeding was hysterical in nature, caused by her wish to bring him to her bedside.

That Freudian theory is not science is suggested by the important difference between giving a systematic account of it, and doing the same for any of the physical sciences. To explain Freudian theory properly, one has to recount its origins and growth. The standard account proceeds by tracing Freud's thought from his early days as a neurologist, through his associations with Charcot, Breuer and Fliess, to his rejection of the "seduction theory" (which postulated widespread child sex abuse) and its replacement by theories of the Oedipus complex and dreams (his Interpretation of Dreams, published in 1900, is regarded as his greatest book), and thence to the later development of theory to include, for example, the Ego-Id distinction. The variety of further, and often diverging, developments in Freudian theory, carried out by gifted followers, commands extra volumes. But it is only recently that a better understanding of Freud's intellectual adventure has been gained, showing among other things its dependence on his women patients and collaborators.

In view of feminism's strictures on Freud, a certain irony attaches to this fact. Freud's earliest women patients effectively invented psychoanalysis, and the psychoanalytic profession owed its rise, spread, and in time some of its most decisive transformations, to women who began as his patients and later became analysts. The two women who should be central in his life, his mother Amalia Freud - born Amalia Nathanson, she was his father's third wife and 20 years his junior - and his wife Martha, are dim presences; Freud hid the former behind a screen of sincere but nevertheless conventional filial respect, and the latter behind an embargo on his papers which seals them for another century. But the shadowiness of these two is more than compensated by a number of other extraordinary women, who in their different ways are chiefly responsible for transforming psycho-analysis from an oddity on the medical fringe to an explosive force in 20th-century consciousness.

Freud said that psychoanalysis began when his colleague Josef Breuer treated a wealthy and beautiful young hysteric. This was Bertha Pappenheim, "Anna O", who effectively created her own therapy by substituting what she called "the talking cure" - Freud later called it "free association" - for Breuer's standard treatment of hypnosis. (Subsequent research has shown that she was not in fact cured by this means, but lingered in hospital long afterwards; it also shows that her symptoms, which involved partial paralysis, contractures and diplopia, probably stemmed from a physical complaint that was first misdiagnosed and then mistreated; it is now convincingly believed to have been tubercular meningitis, caught while nursing her father as he died from a tubercular abscess. Her illness began soon after his death.) Freud's experience with the woman he called his "teacher", another Anna - Anna von Lieben - confirmed for him that the route to a new understanding of psychology (an aim more important to him than any therapeutic possibilities) lay this way. Anna von Lieben was his patient for six years, and taught him that it takes a talented patient to make a talented analyst.

These early patients helped establish the technique of analysis. The theory emerged more slowly. Its painful growth is reflected in the different paths from Freud's couch taken by women as variously brilliant as Helene Deutsch and Princess Marie Bonaparte when they became analysts in their own right. Deutsch was a qualified psychiatrist who, after emigrating to America, specialised in the emotional life of women. Marie Bonaparte established Freud's theories in France, and later engaged in famous battles with Jacques Lacan over Freud's legacy. She was the great-grand-niece of Napoleon, and very rich. Nicknamed "Freud-a-dit" by French analysts, Marie was troubled by the question of vaginal versus clitoral orgasm, and had two surgical operations to move her clitoris closer to her vagina. When she was contemplating sex with her son she wrote to Freud for advice. Although in general hostile to incest ("in my private life I am a bourgeois", Freud wrote) he diplomatically replied that incest is not always harmful. Marie was, after all, a wealthy and influential follower.

Freud sometimes likened himself to King Lear. In a major respect, the comparison is apt, for his youngest daughter Anna proved crucial both to him and his cause. Although her disputes with Melanie Klein about analysis later split the analytic movement, Anna's tireless loyalty was central to the later flourishing of Freudian orthodoxy. Freud analysed Anna, discussing with her at great length her almost obsessive masturbatory life. The significance of this fact should not be misread: some oppose Freud because of the heavy and disturbingly foetid scent given off by his family life, rather than coming to evaluations of his theories and methods on their merits. Defenders point out that psychoanalysis has always laboured against a heavy weight of convention and prudery.

The swingeing attacks to which Freudian theory is subjected by feminists arise from their scorn for the claim that girls suffer "penis envy" and that vaginal orgasm supersedes clitoral orgasm in the mature woman. According to Freud, girls are dismayed to discovery their genital inferiority to boys, and long to acquire a penis, first by sexually desiring their fathers and then by wishing for children, especially sons, who bring the coveted penis with them. As to sexual maturity, Freud identified the "phallic phase" in girls as an infantile stage involving external genital pleasure, initiated by the nappy-changing mother (whom girls blame for their "castration", hence mother-daughter conflict); so any woman who does not mature by developing a capacity for internal, that is vaginal, orgasm has remained in the infantile stage.

In their robust response to these latter claims, feminist critics have powerful ammunition; empirical research undertaken by, among others, Masters and Johnson, decisively refutes Freud, showing not only that the clitoris is the chief sensory focus in the female pelvis, but that women are capable by its means of many orgasms in sequence, prompting some feminists to argue that women are restricted if they rely on men for their pleasure.

If Freudian theory thus fares ill under critical scrutiny, what explains its power? When Auden described Freud as "not a person but a whole climate of opinion", and Harold Bloom nominated him "the central imagination of our age", there is little hyperbole in the claims. Freudian theory indeed took western 20th-century civilisation by storm. How so? The answer lies in four factors. One is Freud's genius as author and ideologue. Another is the immense attraction of any theory that offers to each individual an explanation of his or her own hidden secrets. A third is the promise that science has at last delivered what there had never before been, namely, a proper theory of human nature. And finally there is the fact that at the centre of the package lay the most delicious, anxious, and titillating of all taboos: sex. Such a combination could hardly fail.

Of Freud's powers as a writer and advocate of ideas, and as a possessor of an extraordinary ability to weave together medical knowledge, some genuine in- sights into the human condition, and a powerful imagination, there can be no question. To read him is to be spellbound. He has the narrative skills of a first-rate novelist, and a knack for devising striking ways to describe the psychological phenomena he studied. It is a characteristic of highly speculative enquiries that the thinkers who most influence them are those who find the most compelling vocabulary - one which offers a new way of expressing and articulating its subject. Freud himself once wrote, in commenting on a book by Jung, "In it many things are so well expressed that they seem to have taken on definitive form". This exactly describes Freud's own talent. His marvellous powers of imagination fed on analogy and metaphor, and annexed the austere terminologies of scientific medicine and psychology to them. This gave them authority. His case studies are highly organised narratives constructed from true-life gossip based on voyeurism - irresistible to human curiosity! - and yet they go further: they add the deeply satisfying dénouement of the kind one has in detective stories, where mystery is unravelled by clever and striking juxtapositions of clues.

The second attraction - that Freud offers each individual a revelation of secrets about himself that he does not himself know - is equally irresistible. The same compound of insecurity and curiosity, anxiety and desire that makes so many resort against their better judgement to fortune-tellers, is at work here; except that here the imprimatur of science makes the proceeding respectable, which is why people will spend far more on their analysts than on their astrologers.

The third attraction is the promised theory of human nature. Religious accounts of fallen man, of humanity as midway between beast and angel, of imperishable souls trapped in disgusting matter and therefore sinful from birth, had lost their grip with many, while at the same time Darwinian views offered no account of why evolution had made man as he is.

In identifying sexual and aggressive impulses as the fundamental human drives, and in specifying their causes, Freud offered an inclusive philosophical psychology. Humans struggle with conceptual bewilderments about themselves and their complex natures; one can see why the appearance of Freud's magisterial new insights seemed as welcome as rain in drought.

And finally there is the fact that sex lies at the core of the story. Freud performed a great service by liberating debate on the matter, but it is questionable whether the importance he assigns it is correct. The hungry always think of food; the fed put eating in its proper place. The accidents of social history are easily mistaken for the essentials of human nature, and this is surely what explains Freud's choice of sexuality as the well-spring of human nature. The surprise is that people do not see how, at most, sex can only be part of a far more complicated story.

Philosophies that capture the imagination never wholly fade. From Animism to Zoarastrianism, every view known to man retains at least a few devotees. There might always be Freudians, and there will always be admirers of Freud's great imaginative and literary powers; these two, as the foregoing remarks suggest, are intimately linked. But as to Freud's claims upon truth, the judgment of time seems to be running against him.