Transparency is infrastructure

I’ve spent the last couple of months turning scattered, inconsistent FOI disclosure logs into a single searchable archive. This post examines the gap between mandated publication and meaningful transparency, and who ends up doing the work to close it.

Nothing piques my interest more than systems that are intended to promote transparency but fail in ways that are entirely foreseeable.

Freedom of information is an excellent example. While others have written at length about the deficiencies of Australia’s FOI system, in this post I’ll focus on a more specific aspect that’s been rather neglected: FOI disclosure logs.

What disclosure logs are for

In Australia, agencies and ministers are required to publish documents released under FOI on a disclosure log, unless doing so would be “unreasonable.” (Naturally, “unreasonable” is left somewhat ambiguous.) The intent is to make released information available to the public rather than just the original requester, and to reduce duplicate requests. Disclosure logs are supposed to support the broader purpose of the FOI Act, which treats government information as a public resource and aims to increase scrutiny of government decision-making.

The formal requirement to publish released documents is relatively unusual. Many other countries have freedom of information or right to information laws, but few mandate publication of released information. On paper, our system is more transparent than most.

Technically public, practically useless

I’ve spent the last couple of months turning scattered, inconsistent disclosure logs into a single searchable archive. Having spent considerable time fighting my way through the disclosure logs of dozens of agencies, I can report that the implementation of disclosure logs is not consistent with the goal of increasing transparency.

The most significant problem is that logs are fragmented. Often even finding an individual agency’s disclosure log is challenging, which makes locating and searching multiple agencies’ logs so frustrating as to be effectively infeasible. For a particular topic it might be obvious which agency’s log to check, but for broader topics it can be difficult to know where to look and easy to miss documents. Even if you find the logs you need, individually checking each one is hard because many don’t have search functionality.

Many agencies include only documents released in the most recent few years, so the disclosure log isn’t necessarily a permanent record. This is because the FOI Act doesn’t actually specify how long information should be up for. In some cases agencies link to older logs archived on Trove, but those archived versions are often incomplete and contain many dead links.

The FOI Act says you must have an FOI disclosure log, but it doesn’t prescribe what information it should contain or what form it should take. Predictably, this means every agency has independently solved the problem of creating and maintaining a disclosure log, and each has landed on an entirely different solution. This makes comparing or combining information from different logs (or even the same log at different points in time) very hard.

This fragmentation and inconsistency means that FOI disclosure logs fail to support transparency in the way they’re intended to. Every agency has built a fragment of what is needed, but no one has done the work to make them collectively useful.

From scattered to searchable

These frustrations led me to build FOI Forest, which brings together the FOI disclosure logs of dozens of agencies into one place where they can be searched, browsed, and compared. In many cases the records on FOI Forest go back further than those on agencies’ own websites, as I’ve collected archived records from Trove and the Wayback Machine to fill in gaps. Where the released documents themselves are available, they’re directly linked.

So far there are 43 agencies’ logs on the site (covering over 80% of non-personal FOI releases), with more to come. New records are added automatically every weekday. You can explore the site yourself or download the full dataset of over 25K records to do whatever you want with it.

This makes it possible to answer questions that were previously unanswerable without enormous effort. Take Robodebt: a search for the term finds 41 records across 16 agencies. This includes documents from the obvious suspects like Services Australia, NACC, and the Department of Social Services. But importantly it also includes agencies I wouldn’t have thought to check, like the Department of Health, Disability and Ageing and the Department of Education. Despite being published, those documents are unlikely to be found. This is the gap between publishing information and making it useful.

Even I’ve been surprised by how useful it is simply to have everything in one place that I can search. The information feels alive and available in a way it didn’t when it was scattered across dozens of sites. What was missing wasn’t the information itself, but the infrastructure to make it usable.

The infrastructure problem

But what does it say that making already public information findable feels revelatory?

FOI Forest is a workaround that’s operating well downstream of the problem. That it was built by one person in their spare time tells us where the problem actually lives. The work of making public information useful doesn’t belong to anyone: the FOI Act says agencies must publish information, but it doesn’t require anyone to make it findable or usable.

No agency can solve this on its own, because the problem isn’t located within any single agency. Fragmentation is by definition a system-level problem.

Even the OAIC’s attempt at a solution is limited: their disclosure log hub includes links to the FOI disclosure logs of over 240 ministers and agencies. The hub gestures towards the problem without being able to solve it. The problem of fragmented logs can only really be solved by bringing them together in one place, not by simply telling people where they can find the fragments.

This is neither a new problem nor one unique to FOI. Very similar criticisms regarding fragmentation have been made of other mandated transparency mechanisms such as AI transparency statements. And AustLII was created over 30 years ago to address the much more difficult problem of fragmented legal information and remains an essential piece of legal infrastructure. That a project run from outside government has taken on this critical infrastructure work for decades tells us that this problem is structural and durable.

Meaningful transparency requires sustained, deliberate work. It requires infrastructure that has to be built and maintained. A system that mandates information be published without mandating it be findable and usable creates the appearance of transparency while leaving the substance of it to others. And that arrangement is precarious.

Transparency is too important to leave to volunteers.