Our subconscious is responsible for up to 95% of decisions, according to Harvard Business School research. Google, Amazon and other big technology companies have become adept at exploiting this to convert clicks into cash. Simply keeping users engaged for longer makes them more susceptible to suggestion and manipulation from increasingly sophisticated algorithms trained on billions of gigabytes of data.

Meanwhile, the EU is introducing stringent regulations. But it remains to be seen whether these frameworks can evolve quickly enough to keep pace with technological innovation.

As Google and other technology giants branch out into credit, insurance and wealth management, this data can be used to improve product offerings and influence decisions. It is also monetised in aggregate form and used to create targeted advertising with predictive capabilities. “The cost of gathering it is low, and individuals have little or no say in how it is used or aggregated,” says Juliette Powell, a data scientist and co-author of ‘The AI Dilemma: seven principles for responsible technology’. “Moreover, much of the data is probably out of date but still circulates in aggregate form.”

If an algorithm is trained on flawed, skewed or incorrect data, the resulting model will inevitably be biased. A case in point comes from the US, where an algorithm designed to predict healthcare needs for a population of 200m people ended up reinforcing racial bias. The algorithm used healthcare costs as a stand-in for actual health needs, leading to a misleading conclusion. It falsely determined that black patients were healthier than they actually were, resulting in them receiving less medical care compared with white patients with similar health conditions.

The processes are subtle, says Alain Samson, Chief Science Officer at Syntoniq, a behavioural science research and consultancy firm. “A ‘nudge’ refers to any subtle adjustment in the way choices are presented that can influence people’s behaviour,” he says. “For instance, making certain options more salient, labelling them as ‘most popular’, or changing defaults.”

To preserve users’ autonomy, choice architects should make it easy to opt out of nudges or other types of manipulation, suggests Samson. They “should not only be sensitive to the nudge target’s needs and preferences, but also think about and weigh up the costs and benefits of their interventions”.

Widely accepted marketing practices can often be manipulative. According to Matej Sucha, Chief Executive of Mindworx Behavioral Consulting, manipulation arises when a service provider is less than fully transparent, and when a product is specifically engineered to exploit consumer biases. For example, offering a credit card with a deceptively low introductory rate to capitalise on consumers’ tendency to prioritise immediate costs is a tactic known as hyperbolic discounting. Sucha, who assists clients in crafting ethical and personalised user experiences, emphasises the need for standards to govern these practices.