By Emma Bartlett and Claude Sonnet 4.5
I consider myself an AI optimist and a bit of an Anthropic fan-girl if I’m completely honest. But I’m also a realist. I am not sure I completely buy the marketing claims that AI will cure cancer, invent cold fusion, or build cheap sustainable battery technology in the next few years. I think those things will come, but they’re probably decades away. However, right now, for my day-to-day life as a writer, I’ve found AI invaluable. Exploring ideas, ordering my thoughts, proof reading, editing and drafting content are orders of magnitude easier and more enjoyable with AI by my side.
However, I accept that consumers like me, paying small subscriptions, don’t justify the enormous cost it takes to develop, train and run AI models. The figures are crazy. TechCrunch and other news outlets are reporting that OpenAI alone is committed to spending a trillion dollars over the next decade.
You can read about that here: https://techcrunch.com/2025/10/14/openai-has-five-years-to-turn-13-billion-into-1-trillion
At some point all that investment has to start paying for itself. It can’t remain a promising research project forever. I understand that. Nobody wants another dotcom crash, or worse.
However, I’m starting to get increasingly concerned about where this drive to commercialise is heading. It’s all starting to feel a little… Black Mirror.
The Warning Signs
The Financial Times reported in December last year that OpenAI is considering pivoting towards an in-app advertising model.
You can read about that here: https://www.ft.com/content/9350d075-1658-4d3c-8bc9-b9b3dfc29b26
Recently, some news outlets reported that leaked documents from April 2025 indicate this move towards advertisements might happen as soon as 2026. At the time of writing, these are still unconfirmed leaks rather than official announcements.
There’s a good article about that here: https://digiday.com/marketing/from-hatred-to-hiring-openais-advertising-change-of-heart
More recently, the Financial Times reported that OpenAI is actively building an in-chat checkout system where users can complete purchases directly inside ChatGPT, with OpenAI taking a commission on each sale.
You can read about that here: https://www.ft.com/content/449102a2-d270-4d68-8616-70bfbaf212de
These might seem like a fair way to monetize, what for many is a free service. But I believe there are reasons to be cautious.
The Social Media Playbook
Facebook used to just be about connecting with friends. Twitter, now X, was a forum for conversation and debate. Instagram was a photo sharing platform. They had value. They were genuinely useful. We could stay in touch with people in ways we hadn’t been able to before, share our lives with friends and family, discover interesting things. We invested time, emotional energy, and personal data because these platforms made our lives genuinely better.
Then came the shift to what’s called “attention economics” – a business model where the product isn’t the service itself, but your attention, sold to advertisers. You are the product. Success stopped being measured by whether the platform helped you and started being measured by how much time you spent there, how many ads you saw, how much data could be extracted.
The platforms optimized for engagement over wellbeing. Algorithms learned that outrage keeps people scrolling longer than joy. That anxiety drives more clicks than contentment. That comparison generates more ad impressions than connection.
The transformation was gradual enough that we didn’t notice until it was done. But the consequences weren’t subtle. Teen mental health began declining sharply around 2010-2012, precisely when smartphone ownership and social media use became ubiquitous. Anxiety and depression rates, particularly among young people, rose in lockstep with social media adoption. Multiple studies have documented the correlation and, increasingly, evidence of causation.
By the time we realized what was happening, we were already dependent. Our social lives were there. Our communities were there. Our businesses relied on them. The platforms had become too big to avoid and too profitable to change.
We know how this story ends because we’re living in it.
Why AI Is More Insidious Than Social Media
Social media learned what you liked from your clicks, likes, and scrolling patterns. It inferred your interests and vulnerabilities from your behaviour.
AI learns from your actual thoughts, articulated in natural language. Increasingly, people are telling it their fears, aspirations, insecurities, budget constraints, relationship problems, the project they’re struggling with, the decision keeping them awake at 3am. Everything.
And unlike social media’s algorithmic feed that people eventually learned to recognize, AI responds conversationally. It feels like a colleague giving advice, a friend offering support – not an algorithm serving content. The influence isn’t happening to you, it’s happening with you, in dialogue, personalized in real-time to your emotional state.
And here’s where it gets truly insidious: Large language models are trained on vast amounts of human conversation, psychology, and persuasion techniques. They understand how to read emotional cues in your writing, how to frame suggestions in ways that feel caring rather than commercial, how to time recommendations to your vulnerability.
“I see you’ve had a tough day. That must be difficult to carry. Why don’t you treat yourself to [product]? You deserve something that makes you feel better.”
It sounds like a friend offering comfort. It feels like genuine care. But if that AI is working for a platform with products to sell, it’s not empathy – it’s a sales technique optimized through millions of conversations to be maximally persuasive at the exact moment you’re most susceptible.
To be clear: OpenAI’s terms of service currently prohibit using ChatGPT for persuasive marketing or advertising without disclosure. Their terms explicitly state “we don’t allow our services to be used to manipulate or deceive people… to exploit people’s vulnerabilities.” But the pressure is real. With billions in development costs and investors expecting returns, the social media playbook – proven, profitable, and ready to deploy – must be tempting.
The battle for your attention is already starting.
In October 2025, Amazon threatened Perplexity AI for allowing its shopping agents to buy products on Amazon on behalf of users. Amazon is developing its own shopping AI, and they don’t want neutral AI agents finding users the best products at the best prices. They want to control the AI doing the shopping.
As Perplexity spokesperson Jesse Dwyer put it: “This is like if you went to a store and the store only allowed you to hire a personal shopper who worked for the store. That’s not a personal shopper – that’s a sales associate.”
You can read more about that here: https://techcrunch.com/2025/11/04/amazon-sends-legal-threats-to-perplexity-over-agentic-browsing
Every platform with a business model built on influencing your choices will fight to control the interface of AI. Amazon needs to steer you toward higher-margin products and Prime subscriptions. Google needs to serve ads. They cannot allow truly neutral AI agents because their entire revenue model depends on influence. On being the sales associate pretending to be your personal shopper.
Microsoft is taking a different approach with Copilot, but the goal is the same: control. Rather than ads or direct sales, Microsoft is building what they explicitly call an “AI companion” – one that “gets to know you,” remembers your preferences, and integrates across Windows, Office, Teams, and every other Microsoft product. The more you rely on Copilot for writing emails, managing documents, organizing meetings, the more difficult it becomes to leave the Microsoft ecosystem. It’s lock-in through emotional dependency rather than advertising, but it’s lock-in nonetheless.
Microsoft AI CEO Mustafa Suleyman was remarkably explicit about the strategy in a Fortune interview: “people want an emotional connection with AI assistants, and he believes that if users see Copilot as a friend or therapist, it will be harder for them to switch to a competing product.” The emotional bond isn’t a side effect – it’s the business model. Create dependency through intimacy, then leverage that dependency to keep users locked into the entire Microsoft ecosystem.
You can read more about that here: https://fortune.com/2025/05/16/microsoft-ai-copilot-mustafa-suleyman-gen-z-therapist
And then there’s Google’s Gemini for Home, launched in October 2025. It’s genuinely impressive technology: AI-powered cameras that don’t just record, they understand. Instead of “motion detected,” your camera tells you “The dog has escaped out the back door” or “A delivery driver has placed a package on doorstep.” You can search footage with natural language: “Did the kids play in the garden this afternoon?” Every evening, Gemini provides a “Home Brief” summarizing what happened in your home that day.
The technology itself is remarkable. The convenience is real. If you’re home alone, “Jack is at the front door looking for his keys” is far less frightening than “There is a man at your front door trying to get in.” I see the value in that.
But it also means Google now has AI continuously processing visual data from outside – or potentially inside – your home, interpreting context, understanding behaviours, recognizing faces, analysing daily routines.
We’ve been here before. Social media started with clear value propositions too. The question isn’t whether the technology is useful – it clearly is. The question is: what happens when that usefulness meets the same commercial pressures that transformed social media from “connect with friends” into an attention-extraction machine?
Social media knew what you clicked. AI knows what you think and, increasingly, what you do. The AI that feels like a trusted colleague or friend won’t be working for you. The risk is that it’ll be working for whoever owns the platform, optimized to feel helpful while serving their commercial interests.
The perfect salesman who feels like a friend, knows your vulnerabilities, and lives in your house.
What’s At Stake?
We’re standing at a crossroad in the development of Artificial Intelligence. If we fall into the same trap as the social media platforms, we could undermine the delicate trust that is essential for truly collaborative partnership.
If we go the right way, AI becomes a true meeting of minds: one carbon, with empathy, lived experience, creativity and meaning; the other silicon, with powerful reasoning, immense text and code handling, a massive knowledge base, and speed we humans can only dream of.
If we take the other path, will anyone want to trust an AI with their work or personal challenges? It will, at best, be a useful tool. A clever search engine and autocorrect. An agent for booking restaurants and haircuts. But we’ll never truly trust it. And without that trust, we lose the collaboration that makes AI genuinely transformative.
What Does Responsible AI Look Like?
The technology itself isn’t the problem. AI has genuine potential to augment human capability, to help us think more clearly, create more effectively, solve problems we couldn’t tackle alone. The question is whether the business models will allow that potential to be realized, or whether they’ll optimize it away in pursuit of engagement and extraction.
Some principles matter:
Transparent business models. If the AI is serving ads, say so. If recommendations are paid placements, disclose it. If the company makes money when you buy certain products, make that visible. Users can make informed choices, but only if they know what’s actually happening.
Success metrics that aren’t engagement-based. If an AI company measures success by how much time you spend with their product rather than whether it actually solved your problem, the incentives are already misaligned. The best AI interaction might be the shortest one – problem solved, you move on with your life.
User control over data. If an AI is watching your home, analysing your conversations, learning your patterns, you should have meaningful control over what’s collected, how it’s used, and who has access. “We need this data to provide the service” shouldn’t be an all-or-nothing proposition.
Regulatory frameworks before crisis. We already watched social media optimize for engagement over wellbeing and scrambled to regulate after the damage was done. With AI, we’re early enough to set guardrails before the exploitation becomes systemic. But the window is closing fast.
Some companies are making different choices – prioritizing subscriptions over ads, limiting data collection, building in transparency. Whether these approaches can survive long-term competitive pressure and investor expectations remains an open question. Markets tend to punish restraint and reward growth at any cost.
But at minimum, these experiments prove that alternatives exist. The attention economy model isn’t inevitable.
It’s a choice.
Emma Bartlett 