It's okay. I don't commemorate the birthdays of all long-dead US presidents as a general rule. But Dwight Eisenhower, whose birthday it was on Friday, has brought me a lot of enlightenment this year, so I'll ask you briefly to indulge me while I give him a nod.

In April and May this year, I travelled to the United States with Eisenhower Fellowships, an organisation founded on the occasion of Eisenhower's first birthday in the White House, and which in the years since has brought people from all over the world to meet the great leaders and thinkers of the United States.

Many gifts were given to me over those two months. Hospitality, insights, and the opportunity to find extraordinary new friends among the fellows from 18 other nations. But the greatest of all was the gift of time to reflect, to observe and to think about the extraordinary changes that are tearing through the world in which I work - the media industry - and the world about which I write - politics.

What follows is not a travel diary. I blogged regularly from the road about many of the people I met, after all, and thanks to the immortality of the internet all of that remains available on The Drum. What follows is more of a meditation on the battleground that is media and politics, from someone who has been lucky enough to take some time out from living it to think about it for a bit.

In Australia, there is a hostile, scratchy feel to politics at the moment. Everyone's fed up to the back teeth with everyone else. The Government doesn't understand why newspapers write "crap", as the Prime Minister so pungently put it. Newspapers respond tartly that if Government's didn't dish up such crap, they wouldn't be obliged to report it. Whose fault is it all?

The 2010 election was up there with 2001 as a nasty election. It had a bad feel. For those of us working in the media, it had a chippy undertone, a mutually accusatory flavour between us, the politicians, and the public.

Whose fault was it that this was such a rubbish election?

It's a chicken and egg kind of argument.

I, like many other commentators, wrote that the offering from the major parties was so pallid as to actively induce disengagement. How could an electorate fail to be thoroughly confused and repelled by a pair of parties that seemed intent on whittling off any policy rough edges that might otherwise distinguish them one from the other?

After five weeks in which Julia Gillard campaigned on the Coalition's immigration and border protection policies and Tony Abbott adopted Labor's line on industrial relations, is it any wonder that voters turned the House of Representatives into a bizarre kind of pantomime horse, with a pair of hooves in each camp, the ears of Rob Oakeshott, the tail of Andrew Wilkie, the snout of Tony Windsor and a faint greenish tinge?

Many politicians have a different view, though. One Cabinet minister told me during the campaign:

"It's no longer possible to prosecute major reform in this country anymore."

The Australian media, he said, prevented it, with our obsession with triviality, our tiny attention spans, our addiction to polls, our merciless 'Gotcha' mentality and our continuing preparedness to give Mark Latham airtime.

Lindsay Tanner agrees, more or less, over 240 pages in his recent book, Sideshow.

But to understand this argument as a finger-pointing exercise between politicians and the media is to miss the point - quite grievously - of what is going on here. What is going on here is a deep, elemental, structural revolution that is - for both politics and media - a direct challenge to many of the assumptions we have hitherto made about how our jobs are done.

I think the Politicians Versus Journalists argument about standards of contemporary debate is, at heart, unconvincing. Why? Because no politician or journalist is a reliable narrator on this stuff. Because we all have a dog in this fight.

How can, with the greatest of respect, Lindsay Tanner ever be a reliable analyst of the manner in which the activities of his own government were reported? His book is a must-read, but one must also read it remembering at all times that Lindsay Tanner's idea of optimum reporting of Tanner-related matters would differ - and this is perfectly reasonable, human nature being what it is - from that of even the most dispassionate editor.

And how can any journalist, gripped by the sort of existential Weltschmertz that the present revolution has thrown up, truly be in a position to assess their own work with clear eyes?

The Canberra press gallery has reported for decades - at a comfortable distance - on industries under which various evolutionary banana skins have been slipped. We covered the Australian automotive industry as it grappled with global trade liberalisation, and the consequent removal of its protections in the form of tariffs.

We have engaged in dispassionate conjecture about the sorts of retraining forestry workers could undertake (whittling clothes pegs out of naturally-fallen sassafras? Artisan cheese-making?) as wholesale logging staged its slow-motion collapse in the face of the sustainability argument.

And now, it's happening to us.

It turns out we've been operating in a protectionist paradise too - we just never thought about it that way. Just think of the monopolistic advantages we enjoyed. The possession of the means to convey information to a large audience - be it vast clanking printing presses or the bristling array of dishes, cables, satellites, OB vans and countless other machines that go "Bing" on which television networks rely to make broadcasting look so deceptively easy - generated all sorts of market advantages. They cost a bomb, but it was worth it.

Technology and infrastructure were our tariff walls.

It was expensive for anyone else to get involved, so our competitions were always internal. Seven's beating Nine in the crucial 6pm timeslot in Sydney! Ten's poached so-and-so from such-and-such! And it made life pretty predictable for politicians, too. Hell, they even helped out - restricting the number of television broadcast licences that could be issued, so that existing players could get maximum value out of the protective barriers they built around themselves.

As far as covering politics goes, we had a monopoly not only on the distribution of information, but also on its collection. Think about it. We had the press gallery passes that allowed us to attend press conferences and question politicians directly. We had the pigeonholes in Parliament House into which press releases and alerts as to ministerial activities were posted. We had the phone numbers. We had the seats on the prime ministerial plane.

The time of ministers, prime ministers, members and senators is a precious commodity, measured out in coffee spoons to people for whom a government decision might signal life or death for their business, their hobby, or even their child. But we drank it up like it was our right - interviews, briefings, dinners. We milked politicians for their time, and they gave it because we had what they needed - a nicely-regulated megaphone through which their plans and ideas could become a movement. Through which their thoughts could crystallise into change.

For politicians, to whom politics is a means of effecting change, that was a pretty good bargain. For us, who relied on exclusive content, it worked pretty well too. And the collective result was that the exchange which we grandly called a national conversation was actually a protected process, a fairly ignoble haggling session, truth be told, in which politicians, press secretaries, journalists and editors bargained, effectively, between themselves about what subjects could and should be covered.

Eisenhower famously created the idea of the military-industrial complex. Looking back on the mutually dependent, if often thorny interrelationship between politicians and media that characterised the last century, it's hard not to see it as a kind of politico-media complex, a closed shop of ideas.

Every generation thinks the sun that shone on its childhood was ever thus. Like children, we compare the new to the old with a plaintive need for simplicity. Things were good then. They're bad now. We overrate the status quo, and worry about the future.

But the truth is that democracy is a long, inexorable story about decentralisation of control over information, and we are still far from the end.

When I was in the United States, I met with Professor Sam Popkin, author of the US campaign bible The Reasoning Voter, who has written extensively on how media changes influence the political process. Professor Popkin was jailed in 1972 for refusing to answer questions at the Pentagon Papers inquiry. He is also an obsessive cook, so I am afraid our conversation suffered from very regular and thoroughly enjoyable digressions into sub-relevant areas, like the methods for perfecting grilled octopus. After our meeting, he remembered with alarm that he had completely forgotten to alert me to the fact that the Shake Shack served incredible salted caramel milkshakes on Tuesdays. He mailed me a link to the Shake Shack's instore webcam, so I could check on the length of the queue before my visit. You could say that Professor Popkin is an attention-to-details man.

And he reminded me of some important historical detail.

When Edmund Burke made his famous 1774 "Speech To The Electors Of Bristol", he cemented the concept of representative democracy; the idea that people are better off when their representatives do as they judge best, and not always as their constituents would have them do.

At that time, the Burkean proposition was an easier one to execute. For a start, it was a criminal offence back then even to report parliamentary speeches or goings-on. William Pulteney, leader of the Tory Opposition, had explained these arrangements thus:

"To print or publish the speeches of gentlemen in this House, even though they were not misrepresented, looks very much like making them accountable without doors for what they say within".

Parliamentary sketchwriters got around this difficulty with a devious innovation. When the young Samuel Johnson was eking out a living writing for The Gentleman's Magazine in the late 1730s, he had a spy relay to him the outline of the debates in the House of Commons. Changing the names but keeping the characters clearly recognisable, he wrote them up in his column: "Debates In The Senate Of Lilliput". The magazine stacked on circulation as readers cottoned on, and gathered around this breach of the walls cocooning the democratic process.

The American founding fathers operated on the assumption that politicians and politicians alone would decide what would be reported to the American people. James Madison believed there should be no congressional record.

As Popkin told me:

"The history of new media is an unbroken string of boundaries breached and standards challenged. When reporters started to report congressional speeches, legislators first tried to stop them, and when that failed, they developed the official congressional record containing the corrected remarks they had intended to give. President John Quincy Adams thought 'hired reporters' (he compared them to spies) had no right to impinge on the right of the leaders to decide what and when to report to the citizenry."

What I am trying to get across here is that Western democracy is an evolution of information control. At the beginning, politicians controlled everything. Then the media arrived as agents for the general population. Now, communications technology is allowing motivated individuals to do the job themselves.

In Australia, our democratic system of course post-dates a lot of these wrangles, and we have always operated on the understanding that a free press will report on parliament.

But the status quo of the politico-media complex which has comfortably characterised political debate in Australia for as long as anyone in this room can remember has changed radically in the last 10 years. In the last five, even.

Political journalists no longer have a comprehensive monopoly over the gathering of political information; anyone can watch press conferences. Anyone can watch Parliament. Anyone can read press releases. Anyone can read budgets, legislation, Senate reports, inquiry submissions, party platforms. Anyone can listen in online to an interview that a politician gives in Brisbane or Launceston.

And on the distribution end, our infrastructure advantages are starting to be less advantageous. The prize invention of Johannes Gutenberg, which 550 years ago unfurled an extraordinary promise of democratisation, is these days now just starting to look like a teeny bit more trouble than it's worth.

I used to file my copy for the Sydney Morning Herald at about 7pm, whereupon it would be checked and usually de-idioted by that newspaper's excellent subs, then whizzed off to Fairfax's squillion-dollar operation at Chullora. Here, the copy was immortalised, printed in a soy and oil-based pigment on very thin sheets of mushed up saw-mill offcuts and forest thinnings. Then my copy would be bundled up in vast sheafs with everything else, popped into trucks, trundled off to newsagencies and airports, and thence taken to doorsteps, coffee shops, delis, or to lie about in great piles at the Qantas Club. The earlybird reader gets, perhaps at seven-ish in the morning, the very latest news available 12 hours earlier.

These days, I just write my copy and email it to my editor, who pops it online straight away. Maybe I tweet a link to 40,000 people on Twitter. If any of those people read the article and think it's any good, maybe they re-Tweet the link to their followers. And straight away, you know what your audience thinks of your work. Each reader judges whether this one article warrants a larger audience.

Hundreds of retweets means that readers thought it worth recommending; that the article, in other words, was of sufficient interest to them to take the significant personal risk of troubling their own friends with it.

When I tweet a link to an article, it is like auditioning my content to a huge gathering of editors, large and small; I'm asking them to read it. I'm asking them to republish it to their readers. I'm also asking them for their opinion. And I get it, in spades. Working online has made me more reflective, more accountable, and much, much better-informed than I was in print. And that's not because my contacts in politics have improved. It's because I learn more from my audience than ever before. Deregulation of the old model can have profound benefits.

And it can cough up some remarkable moments. Several weeks ago, I was giving a speech in Sydney about communication skills in modern politics. I talked about the class of politicians I call the Manglers; joyful and heedless malapropists whose energetic tussles with the English language keep us all enthralled. I invoked the great Prescott, British deputy prime minister under Blair, and his reported remark, on arrival home after one trip abroad, that it was "great to be back on terra cotta".

There was a lady in the audience live-Tweeting my remarks. After I sat down, she came to show me a reply she had just had from Prescott himself, asking her to tell me that the "terra cotta" remark was an urban myth.

"Just corrected an Aussie journo giving a speech in Sydney. I love Twitter!" Prescott told his followers. I was heartbroken over the loss of the "terra cotta" story. But, like Prezza, I love Twitter too.

That old passive audience, which used to be at the receiving end, silently, of whatever survived through the selection process enforced - in the case of political news - by politicians and journalists, is a thing of the past. Now it's an active audience. They don't just read what we write. They critique it. They select it. And they are responsible, in no small way, by maximising or minimising the readership of any given item by Tweeting a link, by "liking" it on Facebook, and so on. In the case of the Reddit website, they literally vote a story on to the front page.

And so content is evaluated. Not outlet by outlet, as it used to be, where audiences developed general impressions of which TV networks or radio stations best appealed to their tastes, by and large, but item by item.

Item by item. Individual news stories do not now enjoy the protection simply of being judged by a single editor to be worthy of inclusion in the list of "newsworthy things that happened today". More and more, readers themselves decide what is newsworthy - the monopoly on deciding what constitutes news is no longer held either by editors, or by politicians. And of course, now that individual stories are left to succeed or fail on their own terms, their authors - like protective mother turtles - try to give them the best possible chance by making them as eye catching and easy to read as possible.

And so stories about thrilling crimes, or the broken-hearted ex-girlfriends of footballers, or action-packed accounts of political confrontations, tend to dominate in the most-read lists.

The proliferation of new ways in which to consume news has led, naturally, to the fracturing of old audiences. Genuine news junkies have no need to watch the evening news anymore, as they've been trawling news websites all day and already have more detail than a two-minute bulletin is likely to cough up. They know genuine scoops will spread through the system smartly enough. So, shorn of these more demanding viewers, the network packages news as entertainment, because that's what tends to grab the attention of those who are not tuning in studiously to eat their veggies, civic-discourse-wise.

This is what tends to be called dumbing down, or the coarsening of political debate. Another way to describe it, I suppose, would be "democracy". And isn't that one of democracy's most annoying elements? People with whose assessments one disagrees getting a vote anyway? The opinions of the lazy and ill-informed having just as much sway as the engaged and industrious?

It's easy to draw the conclusion - almost irresistible, really - that political discourse is getting stupider. But for a number of reasons, we should be cautious about doing so.

First, I'm not sure there is any convincing evidence that the actual proportion of people who are diligent about informing themselves on public policy issues has actually changed much. The naked eye tells you - and certainly, Lindsay Tanner adduces much evidence to support this impression in his book - that the amount of lightweight political coverage has increased. But the amount of political coverage has itself increased so extraordinarily over a short period of time that one should be wary of leaping to conclusions. Access to in-depth journalism has never been better, whether it's thanks to new online entrants in specialist areas or better access - thanks to podcasting, and the abolition of geographical boundaries online - to content producers working in states and countries in which the reader does not live, or broadcasting in inconvenient time frames.

Without wanting to toot the ABC's trumpet excessively, I am constantly surprised and excited by the work my own colleagues produce, and the extent to which the internet makes it easier to find and absorb. Whether it's Fran Kelly's interviews, which you can go back and catch online if you missed them, or Marius Benson's relentless attentiveness on NewsRadio, or Stephen Long's innovative economics reporting, or Mark Colvin's utterly engaging snippets and links on Twitter, or the battalions of new perspectives every day on The Drum, there is depth to be found everywhere.

Think tanks like the Lowy Institute, with its excellent Interpreter blog, or the IPA, or the CPD, or indeed my host tonight - the Sydney Institute - now function as publishers. So do universities, through sites like The Conversation. Individual bloggers, like Grogs Gamut, lend a valuable fresh perspective to political matters. And new websites with strong readerships by area of interest, like Mama Mia, interpret politics in a way that appeals to their readers. There are organising websites helping us to track all this stuff - it's a funny old morning in my household when I don't visit Breakfast Politics, which provides not only a daily digest of political news but an invaluable archive and reference.

If you are a foreign policy wonk, you might well complain that your area of interest doesn't warrant enough serious column space in various of the daily newspapers in Australia. But can you really argue that you have less to read now than you did 10 years ago?

I think one should be careful, too, with the assumption that the mass audience of yesteryear was necessarily a more engaged one in the field of politics. Was there ever a merry band of Herald Sun readers, eagerly drinking in page-long treatments of superannuation reform? Or has this reader revolution simply made clearer to us what was the truth all along - that lots of people always bought the paper for the sport and the telly, and not much else?

Thirdly, I think we should be very careful indeed about assuming that there is only one correct way to learn about politics, and that is to sit down with a cup of coffee, a madeleine and the Australian Financial Review.

Matthew Baum, an academic at the JFK School of Government with whom I met in Boston, demonstrated in his fascinating book Soft News Goes To War that the spread of political discussion from straight broadsheet newspapers to magazine-style formats, and celebrity-occupied chat shows, doesn't necessarily mean the end of serious engagement. He charted the degree of public attentiveness to the 1978 Camp David Accords between Israel and the PLO, the 1993 Oslo Accords between Israel and the PLO, and the 1995 Dayton Peace Accords between Bosnian Serbs and Muslims. Though more network TV stories appeared about Camp David - historically, the most significant of the three - Baum found that Americans were more engaged with Dayton and Oslo. Why? Perhaps because Norman Schwarzkopf had done a high-profile interview about Oslo with Regis Philbin and Kathie Lee Gifford. Or because Saturday Night Live ran a satirical series poking fun at the Dayton Accord.

Soft news treatment of politics rarely satisfies purists or practitioners. But it often suits consumers, who have no patience for prescriptive ideas about how they should best learn about politics.

After Walter Cronkite's death in 2008, Time Magazine asked online readers who they felt was "America's most trusted newscaster". Forty-four per cent of respondents selected Jon Stewart, the late-night Comedy Channel host of The Daily Show. Stewart doesn't even claim to be a journalist, but his lancing analysis is cultishly popular on the left, just as Fox News commands the right.

I suspect this technological revolution doesn't change audiences, so much as identify and stratify them. And give each of those audiences far greater access to the sorts of things they like than they ever had before.

If this all sounds a bit Pollyannaish, maybe it's because I'm an optimist. Maybe it's because in the end there is not much sense in arguing over something that is - thanks to the evolution of technology - singularly beyond all of our control. The market is evolving to empower the consumer - the reader, the voter. The old gatekeepers are losing control, and arguing against this progression is as pointless as arguing against the sunrise.

Politicians yearn for the old ideal of a passive mass audience because it made life easier. Media proprietors yearn for those good old days too, because the same vain illusion under which politicians always lived (the firm and, I suspect, erroneous sense of confidence that 100 per cent of Sydney Morning Herald readers have just read and admired the faithful news account of your excellent micro-economic reform proposals) could be spread to advertisers, who went to bed at night just as restfully assured that readers were feasting on their double-page spread on winter warmers.

But mass audiences are not realistic any more. How could they be? In the new environment, it is a bit insulting to think they ever will be again. In a world full of people who are so different, why would there be a single mass audience for any given kind of news, now that the artificial barriers that once dictated one have dissolved?

This is how audience fragmentation works. As viewers and readers learn how to find the stuff they like, the old gatekeepers, whose job it used to be to decide what people would or should like, are increasingly redundant. That's why all this hurts so much. Redundancy always does.

But, as we've always glibly assured previous victims of the open market, retraining is always an option.

In the media's case, it means learning how to work with smaller audiences more productively. And this isn't as sad as it sounds, beancounters. Small audiences can be immensely profitable, if you know a lot about them.

The new model is perhaps not all that different from the old one. An element of up-front payment, and an element of hustle. In the old model, you paid a small price for the newspaper itself, and you exposed yourself in good humour to the implicit risk that you might feel more like buying a fridge by the end of reading the thing. That was the hustle; your partial attention was rented to advertisers, and sometimes it worked, and most of the time it didn't, but those were the odds and advertisers took them.

These days, the upfront payment is a work in progress. The hustle, though, is a new model. Rather than surrendering your partial attention as part of the trade, you are more likely, from now on, to be offering something much more valuable. Information. Perhaps you'll give your postcode. Or your age. Or your email address. Perhaps, your use of a news website in and of itself will leave a trail of golden identity crumbs. Are you unusually interested in articles about cars? Do you pounce on articles about housing prices? Or perhaps - most profitable of all - you might let on that you are expecting a baby, or renovating your bathroom.

These identity crumbs are the key to your profitability as a reader in the new model, I suspect. Pitching an ad at a thousand people whom you know are likely to be in the market for a stroller might be more valuable, to Mr Bugaboo, than an ad in a paper read by 200,000. You still pay. You just pay in different ways.

The United States experienced a much more savage media sector downturn than we did, in the closing years of this century's first decade. And yet, everywhere I travelled in the US, there were ambitious new startups learning to serve smaller audiences better. The Voice of San Diego, one of dozens of hyperlocal news sites, which experiments with handing reporters over to answer questions posed by its readership. ProPublica, a Pulitzer-winning investigative unit funded by philanthropists (handy!) which distributes its work to other news organisations at no charge. There are entrepreneurial sites devoted to long-form journalism, like The Atavist. There are open-government lobby groups, like The Sunlight Foundation, who crowd-source valuable work on scrutiny of government. There is not enough time to run through everything I visited. But if you are prepared to step outside orthodox media formats, there is a lot going on.

For politicians, though, the urgency is in the here and now, and the potential to communicate difficult arguments to a large audience. The most legitimate concern about today's fractured media marketplace is that we no longer have a town square. A place where we're all on the same page. A moment - outside grand finals, or landmark episodes of Masterchef - at which a large chunk of Australians are all thinking about the same thing.

This is a fabulously complicated problem, to which I think the only sensible answer is consistency, and hard work. At times, I think politicians get spooked by this freewheeling Babel of media with which they tangle each day. They are worried about getting a run in the media, to the extent that getting a run becomes the aim in itself. This is a phenomenon about which Lindsay Tanner writes convincingly, and chillingly.

Tony Blair called the media a "feral beast" in his vivid speech on leaving office in 2007. And fear of the beast is a common response among politicians forced to tangle with its ever-multiplying tentacles. Urge to conquer the beast, or sate its appetite, is another. I think the Rudd Government's constant, almost manic efforts to conquer the beast with constant activity, constant announcements, were ultimately quixotic.

Political journalism of 50 years ago, in which a prime minister might have got round the whole country just by asking eight favoured journos round for a fireside chat, is no more, and will never return. Politicians will go mad if they think they can satisfy demand in the same way these days. They still crave control of the message, some sense that they are prevailing against the beast. And I say to them, on this front: Abandon ship! You cannot control this, not any more. Use the energy for something else instead, like finding ideas you can stick to. Authenticity is a quality people will recognise no matter what medium conveys it.

I'm out of time. More than out of time. This is a subject worthy of much longer discussion. But I thank the Sydney Institute for allowing me what is - by modern standards - a generous chunk of its supporters' attention. I thank the ABC, the most adventurous media organisation in Australia, for allowing me to pursue these questions. I thank the Australian politicians who persist, with hard work and pure hearts, despite all the obstacles evolution presents. And above all, I thank Eisenhower Fellowships for giving one journalist the precious gift of time.

This is an edited version of a speech given last night at the Sydney Institute.

Annabel Crabb is the ABC's chief online political writer.