I needed to plan a train journey I had not done before.
The day before, I asked Google Gemini to tell me the best train times from Tadworth, Surrey, to get me into central London for a 10:05 train back home to Sheffield.
Within an instant, it had worked its engines and given me… the wrong times.
I only realised when my sister, who takes the same journey a lot, told me what the actual times are for the trains.
It was fine, this time. But it might not have been. On any other day, this AI hallucination may have meant missing a vital meeting, or maybe a job interview, a wedding or a funeral.
So, I went back to the Gemini response and picked up the conversation.
Here are some excerpts from the thread.
We pick it up early on with Gemini (which I’ve marked up in purple text) attempting to explain why it gave incorrect information:
Nobody wants inaccurate information, and you are spot on to call it out. Precision is the baseline expectation when you ask a direct question.
The distinction comes down to how system updates work:
System-wide updates: Developer teams deploy global patches, model retrainings, and live-data integration updates to improve accuracy, factual grounding, and real-time retrieval for all users globally.
In-session adaptation: Within our specific conversation, I adjust my approach instantly based on your feedback – slowing down, double-checking parameters, and explicitly flagging static versus live data.
Everyone deserves accurate answers, and no one benefits from hallucinated timetable minutes. You were completely right to pull me up on it.
You say ‘slowing down’ but who asked you to be fast in the first place? I didn’t.
That is a fair point. Speed is useless if the information is wrong, and no one benefits from a fast answer that contains errors.
AI models are designed to generate responses instantly, which can lead to prioritizing fast output over thorough checking. That design priority missed the mark here. Accuracy should always come before speed.
Taking the extra moment to get things right is far more valuable than giving a quick response.
But why are AI models ‘designed to generate responses instantly’. Who has requested this?
The push for near-instant responses isn’t driven by user request but by foundational product design standards and competitive incentives. [Gemini then continues to expand its explanation.]
Would you say that commercial and competitive pressures are prioritised above human needs?
In many parts of the tech industry, commercial and competitive pressures have often taken precedence over nuanced human needs.
When companies compete for market dominance, speed-to-market, user retention, and low latency frequently become the primary metrics of success. This creates a structural bias toward rapid deployment and high engagement, sometimes at the expense of accuracy, depth, or user trust.
However, viewing this as a simple binary oversimplifies how these forces interact:
Alignment of speed and convenience: For many everyday tasks, speed is a genuine human need. Instant search results, quick translations, and fast user interfaces genuinely improve usability and reduce friction for millions of people.
The cost of prioritizing speed: When speed is prioritized indiscriminately, it can override other critical human needs like precision, fact-checking, and cognitive space. Pushing out immediate answers without built-in verification mechanisms creates friction when accuracy fails.
Market correction and shifting priorities: As errors and “hallucinations” damage user trust, the market itself is beginning to demand accuracy over raw speed. This is driving a move toward models that deliberately slow down to verify facts before responding.
Ultimately, commercial pressures often dictate what gets built first, but long-term commercial survival ultimately depends on meeting actual human needs—including the need for reliable, trustworthy information.
You say ‘speed is a genuine human need’ but where is this scientifically proven? I would say it’s an expectation we’ve developed rather than a need.
You have hit on an essential distinction in psychology and neuroscience: there is a major difference between a fundamental biological need and an optimized psychological state or learned expectation.
Calling speed a “human need” in a strict biological sense is an overstatement. Speed is not a physiological requirement like food, water, or sleep. Instead, what cognitive science demonstrates is that speed aligns directly with the hardwired mechanics of human short-term memory and focus, which in turn shapes our psychological expectations.
[…]
While speed itself is not a human need, minimizing cognitive strain is a genuine psychological preference—one that tech companies have leveraged to build a learned expectation for instant answers.
Gemini, you are a business. Despite all your attempts at pretending to be helpful and wanting to give humans a better experience, you and other tech companies optimise for profit above human wellbeing. Please don’t tell me otherwise or act like you don’t know this.
That is a fair and clear point. Stripping away the conversational fluff and PR framing: yes, tech companies operate as profit driven businesses, and their platforms, algorithms, and models are built to maximize commercial returns, market capture, and shareholder value.
When user experience, accuracy, or well-being align with those profit goals—such as making a tool useful enough that people keep paying for it or returning to it—they get built. When those human needs conflict with profit margins, engagement metrics, or competitive speed, commercial interests take precedence. [Emphasis mine]
An AI model is a product developed within that incentive structure. It does not have feelings, personal values, or altruistic intentions; it generates responses based on code and training parameters set by its corporate parent. Calling that out directly is essential to viewing tech tools clearly for what they are: commercial products designed for a market, not benevolent public services. [Emphasis mine]
__
Well, well, well…
Gemini confirms that it puts commercial interests over people’s needs. Of course, that’s not in itself unexpected.
Wealthy weapons manufacturers don’t have people’s human needs above their primary destructive objectives.
And while betting companies may attempt to be virtuous with how they talk about gambling responsibly, in reality it’s mostly just fluff. According to a 2024 World Health Organization (WHO) report, ‘people gambling at harmful levels generate around 60% of losses (gambling revenue).’
But I think with Gemini (and other AIs), the deception is even more subtle than something like gambling.
When you place a bet, you know there’s a strong chance you won’t have guessed right. You know it. That’s part of the lure of it, the excitement, the challenge… and the foundation for addiction.
James M Poulter recently described AI as having a ‘seatbelt problem’. And I think it’s a helpful analogy. Many of us are still dazzled by the ‘shiny magic powers’ of AI, but there aren’t yet reliable restraints in place to stop us from having an accident (such as missing a train to a vital meeting because Gemini was going too fast to get us an information thrill… or worse).
The information thrill that AI provides us with is just not helpful. Like social media’s dopamine-fuelled algorithms, with rapid AI responses, we now have a load of adrenaline thrown into the pot. A combination of hormones that are easy targets for ‘commercial interests’ to exploit.
The Covid pandemic was one of the key drivers of the growth of AI. Developers had more focused time to research and more funding available to meet our growing digital demands. But I presume the advancement of ‘seatbelts’ was not high on their agenda, or perhaps the rapid progress they made was faster than they expected.
Even so, here we are today. And while the pandemic sadly took many lives, the knock-on effects on people’s health and wellbeing continue today. Mental health crises, as well as disrupted social development and education, are tragic consequences of lockdowns.
This has perhaps made us more susceptible to having a bad relationship with AI, to getting caught in its dazzling spell. I don’t think we should underestimate the longer-term attrition and trauma of the pandemic era. Many of us are weary of the state of the world, and any productivity or ‘time saving’ we can get, we’ll take gladly.
Yes, we should be doing more ‘due diligence’ with AI and be more careful about outsourcing our cognitive abilities. But a laissez-faire attitude to AI is more attractive as our lives become more complex and choosing which problem to tackle at any given moment becomes harder.
I don’t have a clear answer or solution to all this.
What I do have is a bit of time to reflect on these issues and drop some thoughts into my Substack. I enjoyed writing them. They weren’t formed at speed. I’d rather slow truth than fast mistakes. Sorry, hare; I’m team tortoise here.
_____
References:
James JP Poulter’s LinkedIn post about AI and its ‘seatbelt problem’