Feedback loops and the limits of responsibility

Feedback loops in machine learning are often presented as a problem that arises in particular domains, with familiar examples in recommendation systems and predictive policing. These examples are usually treated as special cases. In this piece, I argue the opposite: feedback loops are the default condition of deployed machine learning systems, not a pathological edge case. Feedback loops are structurally inevitable once models are deployed into the world because of how we frame, evaluate, and govern these systems.

Feedback loops in machine learning are often presented as a problem that arises in particular domains, with familiar examples in recommendation systems and predictive policing. In these cases, a model’s predictions change the reality the model is operating in, leading to undesirable outcomes such as narrowing of recommendations or the exacerbation of existing biases.

These examples are usually treated as special cases. In this piece, I argue the opposite: feedback loops are the default condition of deployed machine learning systems, not a pathological edge case. Feedback loops are structurally inevitable once models are deployed into the world because of how we frame, evaluate, and govern these systems.

Accuracy can validate harmful systems

A recent paper by Balcarcel and colleagues provides several clear examples from the ICU. A positive feedback loop can occur when predicted probability of survival is used to determine who receives a resource-intensive therapy. Patients with low predicted probabilities are not offered a potentially life-saving intervention and are therefore more likely to die. The model’s prediction leads to a decision that makes the prediction more likely to be true.

The model appears to be performing well because its predictions are accurate, but its use leads to more deaths. Do we really want to call this a data science success story?

Crucially, if the model is later retrained on data that includes patients denied therapy because of their predicted risk, the features that originally led patients to be assigned lower survival probabilities will become even more strongly associated with poor survival. This embeds bias by reinforcing the model’s own past decisions.

Negative feedback loops can also arise. Patients predicted to be at risk of hypertension may receive treatment as a result. If the intervention is effective, many of the patients flagged by the model will not go on to develop hypertension. Paradoxically, the model will now appear less accurate precisely because it helped prevent the outcome it was predicting.

In both cases, model accuracy becomes decoupled from the outcomes we actually care about. Accuracy isn’t just uninformative here; it’s actively misleading about whether the model is doing good or harm.

What’s particularly striking in both examples is how foreseeable these dynamics are. So how do feedback loops like these keep getting missed?

Prediction is also intervention

The general term for these feedback loops is performative prediction. Once deployed, models inevitably interact with human behaviour, organisational incentives, institutional norms, and formal decision rules about eligibility, priority, and access. Behaviour changes, data changes, and the system begins to learn from a world it is actively reshaping.

Somewhat appallingly, the phenomenon was only given a name in 2020, and most data scientists continue to live in blissful ignorance.

This is a problem, as feedback loops are the norm in machine learning. In most real-world settings, we want models to interact with and change the world in some way. Yet even when models are designed to support decisions, they’re typically built and evaluated within a narrow technical frame that treats them as passive generators of predictions.

Even when algorithms are explicitly intended to support decision making, they’re rarely evaluated in the context of the decisions actually made. Evaluation tends to stop at model performance, rather than extending to how decisions change, how reliance evolves, or what alternatives disappear.

This is exactly how feedback loops emerge. We treat what happens downstream as out of scope rather than as part of the system itself. But in practice, models are not bystanders in a fixed world. Once a prediction is used to trigger action, it begins to shape behaviour and incentives. At that point, prediction and intervention collapse into each other.

Treating models as passive observers is a governance choice, not a neutral technical description. This choice affects how we think about the system, what effects we consider in scope, and where responsibility is assumed to belong. Conveniently, it means that worrying about how model predictions actually drive real-world outcomes becomes someone else’s job — and in practice, often no one’s.

Accountability falls through the cracks

Feedback loops don’t appear by accident. They’re a direct consequence of drawing system boundaries around the prediction rather than its effects. When machine learning models are treated as solutions to narrow technical problems and are then thrown over the fence into a complex social context, accountability vanishes.

The boundary between the technical and social components of the system determines where responsibility is assumed to live. Model developers are accountable for accuracy, and decision-makers are accountable for individual decisions. Outcomes arising from feedback loops, however, are neither purely technical failures nor clearly attributable decisions. They end up unrecognised and unowned.

Feedback loops are often framed as technical limitations with technical solutions, but they also reflect institutional incentives. Evaluating real-world impact is hard, politically uncomfortable, and difficult to attribute. It raises questions about responsibility that many organisations are not prepared to answer.

We want our models to change the world, but we don’t want to take responsibility for the changes they produce. As a result, systems can be judged successful while quietly reshaping behaviour in ways that were never intended, monitored, or owned.

Addressing feedback loops requires governance

Many proposed responses to performative prediction focus on technical fixes, but they miss the core issue. The problem is not that models are inaccurate or miscalibrated. It is that they are evaluated as isolated prediction problems rather than as components of systems that actively shape behaviour and decisions.

If feedback loops are taken seriously, use must be treated as part of the system by default. That has concrete implications for data science work.

Before deployment, a data scientist should be able to specify:

• How predictions are expected to be used, including what decisions they will inform, who will see them, and what actions are likely to follow.
• What behavioural responses are likely, such as deference, adaptation, or gaming, and how these would change the data the system later observes.
• What success and failure would look like at the level of outcomes, not just predictive accuracy.
• What the model should not be used for, and where its outputs are likely to become misleading or harmful.

Many organisations cannot currently answer these questions, and some have structured their workflows so that they never have to.

Although some of these raise organisational questions cannot be resolved by individual data scientists, we have a key role in shaping where the boundaries of the machine learning system are drawn and therefore where responsibility stops. Treating use, adaptation, and behavioural responses as out of scope is not a neutral choice. It determines what organisations remain accountable for and what harms never become visible at all.

Final thoughts

Feedback loops aren’t a quirk, but the predictable outcome of how we typically define, evaluate, and govern machine learning systems. The mistake isn’t technical. It’s drawing responsibility boundaries that end neatly at the point where the consequences begin.