This month, the US chain Walmart bought the startup Social Calendar, one of the most popular calendar apps on Facebook, which lets users record special events, birthdays and anniversaries. More than 15 million registered users have posted over 110m personal notifications, and users receive email reminders totalling over 10m a month.

Of course, when a Social Calendar user listed a friend's birthday or details of a holiday to Malaga, she or he probably had no idea the information would end up in the hands of a US supermarket. But now it will be cross-referenced with Walmart's own data, plus any other databases that are available, to generate a compelling profile of individual Social Calendar users and their non-Social Calendar-using friends.

The second decade of the 21st century is epitomised by Big Data. From the status updates, friendship connections and preferences generated by Facebook and Twitter to search strings on Google, locations on mobile phones and purchasing history on store cards, this is data that's too big to compute easily, yet is so rich that it is being used by institutions in the public and private sectors to identify what people want before they are even aware they want it.

The most important thing for data holders in the Big Data age is the kind of information they have access to. Facebook's projected $100bn value is based on the data it offers people who want to exploit its social graph. Its holdings include more than 800m records about who's in a user's social circle, relationship information, likes, dislikes, public and private messages and even physiological characteristics.

Google's recent privacy policy change has integrated the various accounts an individual maintains, creating a single profile that includes intentions from its search engine and the connections identified from its social network Google+; preferences and interests from mail, documents or YouTube; and location from its maps and mobile phone operating system.

Aggregated, this data can prove powerful. "Given enough data, intelligence and power, corporations and government can connect dots in ways that only previously existed in science fiction," said Alexander Howard, government 2.0 correspondent at the technology publisher O'Reilly Media.

In a trend that is remarkably similar to the plotline of Philip K Dick's Minority Report, Big Data is being used to predict social unrest or criminal intent. For example, Pax, an experimental system developed by the documentary maker and historian Brian Lapping, predicts the conditions for uprisings using aggregated search terms in different regions of the world. The analysed intelligence is then sold to governments, which can act accordingly.

The systems used to parse, synthesise, assimilate and make sense of the information are starting to make sophisticated connections and learn patterns. Big Data proponents view this as an opportunity to observe behaviours in real time, draw real-time conclusions and affect real-time change. Yet their conclusions can trip into areas that require human sensibilities to truly understand their implications.

In one recent high-profile example, a Minneapolis man discovered his teenage daughter was pregnant because coupons for baby food and clothing were arriving at his address from the US superstore Target. The girl, who had not registered her pregnancy with the chain, had been identified by a system that looked for pregnancy patterns in her purchase behaviour. "Data can say quite a lot," said Howard. "Though one has to be very careful to verify quality and balance it with human expertise and intuition."

In an infamous case in 2006, anonymised search terms released into the public domain by AOL were quickly de-anonymised, identifying individual searchers. And last month, police in New York used a photo from Facebook in combination with their own photo files and facial recognition software to arrest a man for attempted murder.

"People give out their data often without thinking about it," said the European commission vice-president Viviane Reding. "They have no idea that it will be sold to third parties." So users continue to populate databases such as Social Calendar with increasingly valuable personal information that, as commercial property, can be transferred to a new company with a different privacy ethos.

Privacy is not about control over personal data, according to the web theorist Danah Boyd, but the control individuals think they have. "People seek privacy so that they can make themselves vulnerable in order to gain something: personal support, knowledge, friendship," she said at the WWW conference in 2010. Increasingly, people are gaining services that deliver value, relevance and connection – as Google and Facebook do – in exchange for their personal information.

Expectations of privacy are being renegotiated. "When I grew up in Greensborough, Alabama, the population was 1,200," said Jim Adler, chief privacy officer and general manager of data systems at the information commerce firm Intelius. "If you cut school, everyone knew it by dinner. The expectation of privacy was low.

"Now, the expectation of privacy that we've had before Big Data, and our parents had, has been pulled away."

To some degree, this is happening because web users and web developers may not share a universal sense of what is and what is not private. As Boyd put it, privacy is contextual. An individual may be willing to share what they had for breakfast on Twitter, divulge where they are via FourSquare or record every keystroke made on their computers since 1998, but they wouldn't want information about their health or their children's whereabouts made public.

It becomes even more complicated when the users of software systems and architectures are a global population but the privacy expectations have been put in place by primarily US services. "Our expectations of privacy in the US versus Europe are very different," said Adler. "We are currently negotiating which is more important: the rights of the individual or the rights of knowledge."

In the EU, Reding has campaigned for the "right to be forgotten", already part of the 1995 data protection directive, which establishes by law that private data is the property of the individual and must be deleted from a system on request at any time. "More and more people feel uncomfortable about being traced everywhere, about a brave new world," she said. Information held by public bodies, however, remains exempt.

Reding's motivation is primarily to maintain a business ecosystem friendly for foreign investment. "This isn't about the reputation of the individual," she explained. "It's about the reputation of the companies. Data is their currency.

"What we're aiming for is privacy by design," she said. Companies should initiate a hallmark system that informs users that the privacy policy adheres to the guidelines. This, she argued, would ensure that people continue to share their data.

Sceptics like Adler argue that the right to be forgotten is flawed because it ignores how social boundaries are currently being negotiated in the Big Data world. "The ability to delete personal information means that you lose the potential for lessons learned," he said. "If you can step away and erase something someone says that is stupid or hurtful, you lose an element of accountability."

Yet if, as Reding maintains, 80% of British citizens are already concerned that data held by companies will be used for purposes other than the reason it was collected, there may be a shift in how much information people are willing to share.

The weakest link is the technology itself. The Target pregnancy case demonstrates that machines can pick up patterns in ways that may have unexpected consequences for individuals. The people who design the systems that collect and analyse the data are now responsible for thinking about data privacy and projecting future outcomes, and may – because they're human – get it wrong.

"These technologies are as neutral as guns," said Adler. "The Big Data guys who want to send you coupons when you're pregnant – because they're nerdy and technologists – probably don't realise that pregnancy is a sensitive issue." And sensitivities shift throughout an individual's lifespan and, more broadly, social norms shift over time.

Fundamentally, privacy means the same thing in an era of Big Data as it always has, but the capacity of machines to capture, store, process, synthesise and analyse details about everyone has forced new boundaries. It is unlikely that people will stop sharing data in exchange for services that are viewed as valuable.

Big Data offers undeniable opportunities, but requires a delicate balance between the right to knowledge and the right of the individual. Privacy norms will demand that new systems of trust be built into technology design.