Three years ago the satirical website The Onion ran an article with the headline “Woman Stalked Across 8 Websites by Obsessed Shoe Advertisement.” Everywhere she went online, this fictional consumer saw the same ad. “The creepiest part,” she says in the story, “is that it even seems to know my shoe size.” The piece poked fun at an increasingly common — if clumsy — digital marketing technique. But today its gentle humor seems almost quaint. Technology has advanced far beyond the browser cookies and retargeting that allow ads to follow us around the internet. Smartphones now track our physical location and proximity to other people — and, as researchers recently discovered, can even do so when we turn off location services. We can disable the tracking on our web browsers, but our digital fingerprints can still be connected across devices, enabling our identities to be sleuthed out. Home assistants like Alexa listen to our conversations and, when activated, record what we’re saying. A growing range of everyday things — from Barbie dolls to medical devices — connect to the internet and transmit information about our movements, our behavior, our preferences, and even our health. A dominant web business model today is to amass as much data on individuals as possible and then use it or sell it — to target or persuade, reward or penalize. The internet has become a surveillance economy.
What’s more, the rise of data science has made the information collected much more powerful, allowing companies to build remarkably detailed profiles of individuals. Machine learning and artificial intelligence can make eerily accurate predictions about people using seemingly random data. Companies can use data analysis to deduce someone’s political affiliation or sexuality or even who has had a one-night stand. As new technologies such as facial recognition software and home DNA testing are added to the tool kit, the surveillance done by businesses may soon surpass that of the 20th century’s most invasive security states.
Illustration by Michael McQuaid
The obvious question is, How could consumers let this happen? As a behavioral scientist, I study how people sometimes act against their own interests. One issue is that “informed consent” — the principle companies use as permission to operate in this economy — is something of a charade. Most consumers are either unaware of the personal information they share online or, quite understandably, unable to determine the cost of sharing it — if not both.
It’s true that consumers do gain some benefits from all the data gathering, such as more meaningful advertising and better customer service, pricing, and potentially even access to credit. But companies urgently need to find a way to balance the benefits with privacy protection. Consumer advocates are raising alarm bells about invasive digital practices. Public outcries ensue each time a scandal hits the headlines, whether it involves Equifax’s loss of sensitive personal information about tens of millions of people or Russian operatives using social media to manipulate the votes of Americans. Internet privacy experts who not too long ago were viewed as cranks on the fringe now testify before Congress and headline conferences. In Europe major legislation to protect user privacy has already passed. We’re starting to see signs of a widespread “techlash,” which could have profound implications for firms that use consumers’ data. It’s probably no coincidence that Facebook saw its valuation plummet roughly 20% after it publicly suggested it might scale back on some data collection.
At the same time, consumers don’t reward companies for offering better privacy protection. Privacy-enhancing technologies have not been widely adopted. People are generally unwilling to pay for privacy-enhancing technologies and even if they do, will pay only modest amounts. Though some might take this as evidence that people simply don’t care about privacy, I’ve come to a different conclusion: People do care, but as I’ll explain, several factors impede their ability to make wise choices.
If both sides continue on these diverging trajectories, the surveillance economy may be headed for a market failure. The good news is that policymakers can help. The first step is to understand how people make decisions about the privacy of their personal information and how they can be induced to overshare.
How Consumers Got in So Deep
Let’s be frank: People are bad at making decisions about their private data. They misunderstand both costs and benefits. Moreover, natural human biases interfere with their judgment. And whether by design or accident, major platform companies and data aggregators have structured their products and services to exploit those biases, often in subtle ways.
Impatience. People tend to overvalue immediate costs and benefits and underweight those that will occur in the future. They want $9 today rather than $10 tomorrow. On the internet, this tendency manifests itself in a willingness to reveal personal information for trivial rewards. Free quizzes and surveys are prime examples. Often administered by third parties, they are a data-security nightmare, but many people can’t resist them. For instance, on the popular “real age” website, people divulge a large amount of sensitive health information in exchange for the immediate “benefit” of knowing whether their “biological age” is older or younger than their calendar age. Consumers gain zero financial reward for such disclosures. They may be vaguely aware of the potential costs of providing such information (at its extreme, higher insurance premiums down the road), but because those downsides are vague and in the future, they’re disregarded in exchange for a few minutes of fun.
Impatience also prevents us from adopting privacy controls. In one experiment, people who were setting up a digital wallet were offered a service that would secure (that is, encrypt) their purchase transaction data. Adding the service took a few additional steps, but only a quarter of people successfully went through them. The vast majority were unwilling to trivially inconvenience themselves by following a onetime simple process in order to protect their data from abuse down the road.
Data “transactions” are often structured so that the benefits of disclosure are immediate, tangible, and attractive, while the costs are delayed and more amorphous — and in these situations, our impatience tilts us toward disclosure. Mobile credit-card stations, for instance, email you receipts and make transactions fast and paperless. But the costs of companies’ capturing your email address and other personal information come later. Sensitive data, such as your name, demographics, and location, is amassed and shared or sold, and in all likelihood you are eventually barraged with targeted marketing. Although some of those ads may be welcome, others may be annoying or intrusive. And some fear that in the future consumer data may even be used in more-impactful ways, such as credit score calculations — and possibly lead to discriminatory “digital redlining.”
The endowment effect. In theory people should be willing to pay the same amount to buy a good as they’d demand when selling it. In reality, people typically value a good less when they have to buy it. A similar dynamic can be seen when people make decisions about privacy.
In one study, Alessandro Acquisti, George Loewenstein, and I offered consumers one of two gift cards: a $10 “private” card that wouldn’t track their purchases, or a $12 card that would. In some cases the study’s subjects were given the option to “buy” privacy by trading the $12 tracked card for the $10 untracked card. In other cases they were given the option to “sell” their privacy by trading the $10 card for the $12 one. In either situation, privacy cost $2. Surely your willingness to forgo $2 to protect your privacy should not be affected by which card you’re initially handed. But, in fact, almost 50% of people were willing to give up privacy for $2, but fewer than 10% were willing to pay $2 to get privacy.
This implies that we value privacy less when we have to acquire it. So something as simple as whether our information is by default public or private can have enormous implications; we’re far more amenable to sharing it when the default is public. More broadly, the disparity is consistent with the way privacy breaches generate outcry while privacy gains aren’t met with commensurate jubilation. It may also set the stage for vicious cycles of privacy erosion: Breaches make information increasingly public. And when our information is public, we value our privacy less, in turn making us more comfortable with parting with it.
Firms have made loose privacy defaults, well, the default for the tech industry. Here are just a few examples: In November 2016, Uber changed its preset options to allow it to track users at all times. (It changed them back in September 2017 after facing criticism.) On the social payments app Venmo, transactions are public by default. Google automatically stores your location when you merely open its Maps app; opting out is confusing, if not downright misleading.
Users’ ability to opt out is also often obfuscated. In a recent white paper, Senator Mark Warner (D-VA) highlighted how Facebook’s mobile app used defaults to “deceptively prod users into consenting to upload their phone contacts to Facebook (something highly lucrative to Facebook in tracking a user’s ‘social graph’).” The first screen on the mobile app gives the impression that consent to sharing contacts is the sole choice. Only when users click on a “learn more” button do they discover (if they scroll down and look carefully) that they can opt out.
Illusion of control. People share a misapprehension that they can control chance processes. This explains why, for example, study subjects valued lottery tickets that they had personally selected more than tickets that had been randomly handed to them. People also confuse the superficial trappings of control with real control. In a study on receptiveness to behaviorally targeted ads, Tami Kim, Kate Barasz, and I found that people are more comfortable with third-party data sharing, a practice they ordinarily deem invasive, when they have a sense of control — even if what they seem to control has nothing to do with the ads they see or the data shared. People can be put at ease by something as irrelevant as a reminder that they can choose their profile pictures. Related research suggests that people are overconfident about their ability to control their own security in cyberspace. In a recent survey conducted by Experian, 56% of respondents mistakenly believed that the risk of identity theft decreases over time, and 10% believed that they weren’t at risk because their finances were weak.
While some efforts to grant consumers more control over their data are meaningful, I also see instances in which their privacy concerns may be placated by an illusory sense of control. Consider the Network Advertising Initiative’s Consumer Opt Out site. On it people are informed of the companies that are customizing ads for their browsers and can select which companies’ to opt out of. When I used the service, I opted out of 72 firms’ ads. I felt in control. But when I checked the fine print, I learned that my choice only prevents the specific companies from delivering targeted advertisements; it doesn’t necessarily stop me from being tracked — a fact that’s easily forgotten because I no longer see those targeted ads — the very thing that could remind me my data is being collected.
Desire for disclosure. This is not a decision-making bias. Rather, humans have what appears to be an innate desire, or even need, to share with others. After all, that’s how we forge relationships — and we’re inherently social creatures. In one study, even people who were very concerned about their privacy went on to readily divulge personal information to a chat bot. Unloading your secrets has psychological and physical benefits. When strangers are paired up in lab experiments and prompted to disclose personal information with each other, they build greater rapport. Keeping a journal in which you share your worries can improve physical health, while keeping secrets can reduce well-being. And a neuroscientific study found that when people disclose information about themselves, it activates the reward regions in their brains; in the same experiment, people even passed up monetary rewards for the chance to answer personal questions.
Our orientation toward disclosure is also apparent in how we perceive those who abstain: We view people who withhold with contempt. For example, as my research with Kate Barasz and Mike Norton shows, we dislike and distrust those who avoid answering personal questions even more than those who reveal damaging information about themselves. In one experiment, participants indicated greater interest in hiring a job candidate who admitted to having done drugs than someone who had withheld the answer to a question about drug use.
Online, the boundaries between social and commercial transactions are increasingly blurred. For example, on virtually all social media platforms, ads resemble noncommercial posts. Though there may be other reasons for this practice (the wish to make ads less intrusive, for instance), it also gives ads the feel of social posts, which I suspect helps trigger people’s desire to disclose and keeps privacy concerns at bay. Similarly, casual, unprofessional-looking interfaces induce self-disclosure too, even though such interfaces are often indicative of poorer privacy protections.
Indeed, heightening the desire to disclose appears to be central to many social media sites, right down to the perpetual “What’s on your mind?” prompt on Facebook. Online retailers have been adding similar social elements to sales processes, such as robot chat agents designed to build rapport with consumers. The structure of Venmo’s site mirrors that of social media sites. Users build their social graph by adding contacts; those contacts’ transactions are displayed prominently in a newsfeed. That makes financial transactions feel like social transactions, turning something that people would ordinarily keep private into something that they not only are comfortable sharing but potentially want to share. Though consumers may get some value out of sites’ social aspects, those aspects can also make the risks of disclosure less apparent.
False sense of boundaries. In off-line contexts, people naturally understand and comply with social norms about discretion and interpersonal communication. Though we may be tempted to gossip about someone, the norm “don’t talk behind people’s backs” usually checks that urge. Most of us would never tell a trusted confidant our secrets when others are within earshot. And people’s reactions in the moment can make us quickly scale back if we disclose something inappropriate.
But in the online world, the rules are different. We often don’t get the same rich, visceral feedback that tempers our behavior in the off-line world. We may have the illusion, for instance, that we’re disclosing information only to a select group of people, such as the friends in our social media feed. People get into trouble when they post rants (say, about their employer) meant for a small subset of their friends, forgetting the broader audience that can see those disclosures (say, their boss and colleagues).
We’re easily seduced by the seeming ephemerality of digital interactions, too. My research with Reto Hofstetter and Roland Rüppell has found that temporary sharing technologies that allow messages to disappear, such as Snapchat and Instagram Stories, lead people to make uninhibited disclosures. Yet the damage to their reputations is potentially long lasting. Most of us wouldn’t dream of making a profane gesture in a professional meeting just because such an act would be fleeting. But online, perhaps because we often receive only impoverished feedback, the promise of ephemerality goads us into oversharing.
Complexity and the Coming Conundrum
We’ve unpacked both the consumer and producer sides of the surveillance economy, but underlying it all is a factor of increasing importance: Complexity.
Do you know how cookies work? Do you understand how information on your browsing history, search requests, Facebook likes, and so on are monetized and exchanged among brokers to target advertising to you? Do you know what’s recorded and tracked when you ask your digital assistant to do something? The answer is probably no. That’s a problem.
A key tenet of any functioning market is “buyer beware.” But online, weighing the risks against the benefits of sharing can feel like an act of futile metaphysics. How much privacy have you lost when firms track your location — and what is the value of that privacy? Is it worth the added convenience of a GPS navigation tool? What should a consumer be “paid” for allowing continuous location tracking? Moreover, the behind-the-scenes “plumbing” of the surveillance economy is so byzantine and opaque that it’s effectively impossible for consumers to be adequately informed.
There is also no way to know what all third parties are doing, or will do, with your data. Although Facebook has been tightening oversight of apps as well as their access to user data, the fact is that many apps have been selling user information obtained on Facebook, and consumers can’t possibly have known where their data would end up when they agreed to the apps’ terms and conditions. Suppose that a couple of years ago you clicked on a Facebook link for a quiz like “What ’80s Movie Best Represents Your Life?” The administrator of the quiz could have gathered some Facebook data — birthday, friends, things you’ve clicked on, likes, locations, groups you belonged to, where you’d checked in — and made it available to third parties by wrapping it in JavaScript, circumventing privacy protections offered by most web browsers. A data broker could have collected that parcel of information and sold it for use in targeted ad campaigns. It would be effectively impossible for you to figure out how your data moved through the advertising ecosystem or identify the brokers or agencies involved.
There’s also nothing stopping your friends from sharing information on your behalf. In a study by economists Susan Athey, Christian Catalini, and Catherine Tucker, people readily disclosed their friends’ email addresses in exchange for free pizza.
Even when consumers actively seek to uncover what personal information about them has been shared and with which entities, firms are not always forthcoming. When users click on Facebook’s “Why am I seeing this ad?” feature (which is difficult to find), the explanations they’re given are sometimes uselessly generic (for example, “One reason you’re seeing this ad is that Rothy’s wants to reach people who may be similar to their customers”).
Even if all the players and transactions in the surveillance economy were widely understood, consumers would still sometimes find it impossible to know what they were actually disclosing, because discrete pieces of data can be synthesized to form new data. For example, it is possible to identify someone by knowing the dates and locations of just four of his credit card transactions. A person’s Social Security number can sometimes be predicted by her birth date and birthplace, meaning that if a consumer provides her birth date to an entity that already knows her birthplace, she may unwittingly divulge her Social Security number. What the consumer reveals is not just a function of what the consumer decides to reveal. It’s also determined by what the receiver knows about that consumer.
Algorithms and processing power now make it possible to build behavioral profiles of users without ever having to ask for their data. Their mere presence in someone’s social network or comment on someone else’s social feed can be harvested to predict and profile. This phenomenon creates entirely new conundrums: If a company profiles a consumer using machine learning, is that profile subject to the regulatory rules of personally identifiable information? Does the consumer have any right to it? Should a company be allowed to use such techniques without the consent of the targets, or at all? No one knows.
For all these reasons, privacy decision making is incredibly complex. And when people perceive decisions to be overwhelmingly complex, they are prone to disengage. Anyone faced with the decision to “accept” online terms of use can relate. They are typically so long that you cannot reasonably expect anyone to read them, let alone understand them. One study estimated it would take Americans 54 billion hours annually to read the privacy policy of every new website they visit.
So most consumers respond by throwing their hands up and agreeing to terms that would give them pause if they understood them. Mobile game app users, for example, might be surprised to learn that they have “consented” to allow some of these apps to share their personal data with third parties, for any reason whatever. Some even have access to people’s microphones, which they use to record audio even when the app is not in use — and even though that information is not used in the game itself. “Super-apps” like China’s WeChat, which has one billion users, have far-reaching access to personal data, including social media posts, bank and credit card details, financial transactions, and even voice data. By technically providing information on the costs and benefits of information sharing and having consumers “agree” to it, these and other digital platforms maintain a kind of plausible deniability.
Complexity makes it hard to fix the surveillance economy without breaking the system entirely. Though that’s a possible outcome, it’s not a good one. Data gathering doesn’t have to be a bad deal for internet users. Consumers have gained enormous benefits from it and from major platform companies such as Alphabet and Facebook. However, the fact that so much of the surveillance economy operates surreptitiously and by default suggests that tech companies have reason to fear consumers might not opt in if they truly understood the bargain that “free” technologies entailed.
Moreover, while consumers surf in the dark, firms have a much better understanding of their own costs and benefits. Expenses for tracking technology and data brokers and the lift in sales from more finely targeted ads can be calculated with precision. Firms thus have an informational advantage over consumers. As any economist will tell you, that asymmetry on its own suggests a market failure and thus invites regulatory intervention.
Restoring Balance
In the 1960s the U.S. and other governments began to systematically write product safety regulations after it became clear that consumers couldn’t properly assess risks — such as the danger of riding in a car without a seat belt and the chance that a soda bottle might explode — and that firms weren’t motivated to address them. Scholars have argued that in such situations it makes sense for regulators to shift risk onto those best able to manage it: the makers of the products. Given that consumers face similar challenges in evaluating privacy risks, lawmakers should consider taking this approach with regulations about personal data collection.
Of course, any regulatory response will prompt skeptics to point out the thorny issues we haven’t yet begun to understand well. Here are just a few of the questions they’re apt to raise:
- To what extent do people own their personal data?
- Should people have an expectation of privacy in public spaces, or is anything they do in public fair game for surveillance?
- Is the online realm a public space?
- What is the value of privacy? Can it even be calculated?
- What is the value of personal information? Can it even be calculated?
- What information is mine to control — does it include AI-generated predictions about my behavior?
- What are the costs of enforcing privacy regulation? Do they outweigh the benefits?
Some argue that it may be too late to protect consumers’ personal data, because it has already been fed into machine-learning tools that can accurately infer information about us without collecting any more. Despite machine learning’s impressive capabilities, this is not yet the reality. Firms continue to have great interest in obtaining consumer data. But even if predictive AI capabilities do dampen the demand for consumer data, regulation could place basic limits on what firms could do with those predictions (by, say, preventing health insurers from using them to discriminate against applicants who might have medical problems).
Though the details of such regulation are beyond the scope of this article, the research I’ve described does provide some broad guidance about what is likely to work. First, the goal should not be simply to make it harder to share or to unilaterally increase firms’ barriers to consumer data. Such an approach would be overly simplistic, because firms and consumers alike have much to gain from sharing information. Regulators should also be aware of the costs of restricting information flow — for example, the potential to impede innovation.
In Europe, the recent GDPR privacy law requires firms to get consumers’ opt-in consent to harvest personal information. This is laudable because it addresses issues with defaults, though at the cost of annoying and inconveniencing consumers. And when people are repeatedly faced with decisions about opting in or out, they can become desensitized, which is hardly a recipe for thoughtful choices. So some of the same factors that make data collection ripe for intervention also make designing regulations about it particularly challenging.
A common approach is to require firms to give consumers information on the relevant costs and benefits of sharing and to tell them about data breaches. But as I’ve noted, research points to the limits of this approach. It’s unlikely to solve the problem given that users don’t read privacy policies and, despite the media uproar, don’t take much action when they learn of breaches. (Indeed, the majority of Facebook users stayed on the platform after the Cambridge Analytica scandal broke.)
A related approach is to use regulation to directly reduce risks to consumers by, say, placing specific restrictions on what personal data firms can collect and how they can use it, and handing out penalties for noncompliance. In the United States, there is no national law regulating the collection and use of personal data. Some basic ground rules do seem to be in order. In Massachusetts, for example, companies must encrypt personal data that flows over public networks. And California’s groundbreaking new Consumer Privacy Act imposes several rules on firms; for example, businesses that sell consumers’ data must allow users to opt out of such sales without penalty.
But a problem with this approach is that it can lead to the “whack a mole” problem, whereby firms find loopholes to wriggle out of while complying with the letter of the law. For example, California’s new privacy law forbids differential treatment of consumers who exercise their privacy rights — unless it “is reasonably related to the value provided by the consumer’s data” — a possible loophole for firms to exploit. And workarounds may be particularly easy to find in the digital space, where firms are quite nimble; a quick tweak in wording of a privacy policy can have enormous consequences.
So the real promise of government intervention may lie in giving firms an incentive to use consumers’ personal data only in reasonable ways. One way to do that is to adopt a tool used in the product safety regime: strict liability, or making firms responsible for negative consequences arising from their use of consumer data, even in the absence of negligence or ill intent. Relatedly, firms that collect our personal data could be deemed, as legal scholars Jack Balkin and Jonathan Zittrain have argued, “information fiduciaries” — entities that have a legal obligation to behave in a trustworthy manner with our data. Interventions such as these would give firms a sincere interest in responsibly using data and in preempting abuses and failures in the system of data collection and sharing (because otherwise they’d face financial penalties).
To be sure, many difficult questions need to be answered first. For example, how would damages be determined? Although the harm done by disclosure cannot be calculated with precision, it could be estimated. Terry Bollea (also known as “Hulk Hogan”) was awarded $115 million in compensatory damages when Gawker violated his privacy by posting a sex tape of him where millions could see it. (Full disclosure: I worked as a consultant to Bollea’s team on this case.)
Another challenge is proving harm; because this is hard to do in the privacy sphere, some have cogently argued, the courts would have to accept the notion of probabilistic damages. Also, what constitutes “reasonable” versus “unreasonable” data use? That’s difficult to articulate, but it’s often the kind of thing you know when you see it. And a key aim of regulation would be to serve as a deterrent and prevent irresponsible use of data in the first place.
A common concern with regulation is that it can reduce competition. The cost of compliance is disproportionately burdensome for small players, so the net effect of regulation can be greater market power for large incumbents. But there is reason to believe that this pitfall would be less likely if firms were given an interest in behaving in a trustworthy manner. First, companies with deep pockets would be disproportionately targeted by those seeking damages. Second, this approach is conceivably less restrictive to new entrants because it need not require the large up-front investment in compliance that direct approaches typically do.
Regulation is not a panacea for the surveillance economy. It will surely introduce some new issues. There’s also more to gaining consumers’ trust than merely following the law. But if we draw on insights from behavioral science and accept that consumers are imperfect decision makers rather than perfectly rational economic actors, we can design better regulation that will help realize the benefits of data collection while mitigating its pitfalls — for both firms and consumers alike.The Big Idea
