Ten years ago I was a new lawyer working in welfare rights at a community legal centre when Robodebt arrived.

Most know the story. An automated debt recovery scheme. Hundreds of thousands of unlawful debts. Then a royal commission followed by public apologies, compensation payments, careers ended, reputations damaged.

But when I think about Robodebt, I think about people whose lives were destroyed.

I think about the clients who sat across from me clutching debt notices they did not understand. They assumed if a government department said you owed money, you must have done something wrong.

One of the most troubling aspects of Robodebt was that so many warning signs existed. Community legal centres, welfare rights advocates, academics and social security specialists all raised their concerns.

Yet the scheme continued for four years.

Here we are again

It’s no longer 2014. I’m now the CEO of the same community legal centre. We operate disability advocacy services, disability discrimination legal services and NDIS appeals advocacy, among other services.

I’m a decade older, hopefully wiser, certainly more experienced and less willing to accept assurances that “there will be human oversight” simply because a government says there will be.

That is why I am increasingly concerned by the direction of NDIS reform.

The government has defended its move to bring more automated decision-making into the NDIS and future social services programs.

Let me be clear – the NDIS requires reform. Any system managing tens of billions of dollars in public expenditure requires scrutiny.

Worryingly, recent reporting has focused on the increasing use of computer-assisted assessments, digital profiling and standardised planning tools within the NDIS.

We are repeatedly told that humans remain involved in decision-making. Well, perhaps they do, but here are the most important questions:

• Can participants understand how decisions have been reached?
• Can they access the information relied upon?
• Can they challenge assumptions?
• Can they meaningfully exercise review rights?
• Can they do so without requiring specialist legal assistance?

These questions are important because a human signature at the end of an automated process is not the same thing as meaningful human oversight.

We learned that lesson during Robodebt. The concern is not technology itself, but the understanding that disability is not a spreadsheet and human lives do not fit neatly into categories.

A child with autism is not simply a data point. A person living with psychosocial disability is not an algorithmic classification.

A family navigating multiple disabilities, housing pressures, school challenges, workforce participation and rising costs of living cannot be adequately understood through standardised digital profiling.

Disability is inherently human and individual. Indeed, the more complex a person’s circumstances become, the more important human judgement is.

Risk not financial. It’s human

One of the enduring tragedies of Robodebt was that harm was treated as collateral damage.

The royal commission heard evidence about people who experienced severe anxiety, financial hardship and, in some cases, suicide.

The scheme affected approximately 450,000 Australians, while the NDIS directly affects more than 760,000 people, as well as their families, carers and support networks.

The scale alone should give us pause.

At our service, we are already seeing the consequences. We see exhausted parents and carers terrified about what will happen next.

We see people scared to ask for a review, because they don’t want to rock the boat and end up with no support at all.

We see participants struggling to understand decisions that profoundly affect their daily lives – some who have gone from 24 hour, one-on-one care to four-six hours of “drop-in support”, forcing them to ration toilet access and basic care.

Occasionally we hear things that stay with you long after the file is closed.

One client thanked our team after we successfully appealed a decision. They told us prior to the appeal they had begun researching voluntary-assisted dying, despite not being eligible.

I share that story carefully because fear, uncertainty and the loss of essential support can have devastating impacts.

The people affected by these reforms are not budget lines. They are human beings trying to live meaningful lives.

We have already sat through one royal commission where names were read into the public record and families were left to explain the human cost of government failure.

Ten years from now we should not be asking why nobody saw the warning signs. They are here now.

The question is whether we have the institutional courage to change course before more harm is done. Because once systems become sufficiently complex, automated and opaque, they begin to serve themselves rather than the people they were created to support.

That was the lesson of Robodebt. Not simply that the debts were unlawful or that governments make mistakes, but that when transparency, due process and human judgment are sacrificed in the pursuit of efficiency, it is vulnerable people who pay the price.

Zoe O’Neill is CEO of Sussex Street Community Law Service in Western Australia