The Census has already changed – we’re going to need to live with it

Recent announcement about end of the Census have stirred more questions that we thought would pop up about this topic. The government has announced that from 2030, an administrative data-driven process will produce Census-like data annually, supplemented by more detailed surveys. Despite the shift from a more than century-long process, the need to change appears inevitable.

Although some have raised concerns about the changes, it’s important to recognise that things have already changed. Trying to cling on to the idea that the traditional Census is the best way to ensure we accurately count people and deliver vital statistics conveniently forgets that the current Census doesn’t achieve that. Not even close. Instead, to ensure we count as many people as possible, and have the detail we need to make better informed decisions, we need to embrace the shift towards administrative data and ensure we start, today, collecting the data we’ll need in the future so it’s reliable in years to come. But simply “sticking to our guns” on Census as we know it would deliver increasingly expensive, and increasingly poor, data about modern New Zealand.

There seems to be a view among some academics and statisticians commenting about the Census announcement that the Census still today provides a full-coverage Census of the population. It doesn’t. Instead, the traditional Census is already supplemented by administrative data collected by other agencies, to fill the gaps from people who don’t participate in the Census. In trying to argue that we cannot give up on a full coverage Census ignores the fact that this has already happened, and we haven’t had reasonable coverage for over a decade from Census alone.

Infometrics analysis of Stats NZ data shows that the proportion of people counted in the Census from filling out their Census form has dropped markedly, and hasn’t recovered. As Chart 1 shows, around 95% of people in the 1996, 2001, and 2006 Census were counted because they filled out their individual Census form. In 2013, that eased slightly to around 92%. Come 2018, and the individual form response rate plunged to below 82%, and in 2023 only recovered slightly to around 85%.

The more general Census Response Rate (which includes people who were mentioned in a dwelling form even if they didn’t fill out their individual form) plunged too, to 87%. In Census 2023, around 750,000 people were not counted at all, whatsoever, based on a Census response on paper or online. That’s more than the population of Wellington, Christchurch, and Dunedin urban areas combined.

If we’re relying on a Census to provide full coverage data, we’re already well missing the mark. Importantly, the administrative data held by government agencies helped fill in the blanks, achieving a total response rate of nearly 99%. Supposing that those that didn’t complete the traditional Census was a mix of both uninspired busy people, and people that deliberately didn’t want to participate, administrative data ensured that these people were able to be included.

The considerably, and ongoing, lower Census response rate comes despite more funding to support the Census. Running the Census has become an increasingly expensive endeavour, with little to show for the additional spending. Infometrics analysis of Stats NZ appropriations data shows that Census costs have been soaring, from $72m for Census 2013, to $120m for Census 2018, then to an eye-watering $325m for Census 2023. Census spending has increased from $17 per person to $65 per person over a decade. For those that actually complete the Census, the costs are even higher, rising from around $18 per person to nearly $77 per person.

Despite the considerably larger cost of delivering the Census, there’s not much of a turnaround in the response rate. The cost to deliver the Census appears set to continue to rise over time, despite lower than wanted response rates.

What’s abundantly clear is that the time of the Census as we know it is gone. That ship has sailed. To plug the already existing gap, administrative data has become increasingly important, and is less expensive to utilise.

Although some view the Census as a sacrosanct investment that dictates substantial amounts of government spending, the pace of changes in the economy means we need to shift towards a more frequent update. Waiting for data every five years, and taking two years for the data to come out, means we can take a long time to recognise a shift. That’s why we already use a range of other datasets and surveys to ensure that needs are met, and monitored more rapidly. Information from the various government agencies we interact with about where we live will be just as good, but a lot more frequent, than filling out a form - or not - every half a decade.

That’s not to say that administrative data as it stands today is perfect - it’s anything but. Yet it provides considerable insight – in a more cost effective, detailed, and timely way than before. Now’s the times to get New Zealand’s administrative data ready for a post-Census world.

We know, for example, that household data leaves some clear gaps. Occupation data, looking at the jobs people do rather than the industry they’re employed in, isn’t really collected outside the Census. Ethnicity data isn’t regularly produced in detail either. These challenges are solvable – in our view, IRD needs to start to collect occupation and ethnicity data within the next year, so we have a baseline to compare to over time, and more data and linking of households should be done to ensure reasonable comparability to Census data series.

We’ve got to adjust, because the Census coverage as we know it has changed forever. There might not be a perfect replacement, but given the drop in Census response, we need to work quickly to find a solution. We’ve tried more money, and that failed. There is no going back to the full coverage Census many seem to think still exists. Instead, to ensure we count everyone we can, we need to embrace administrative data, work together to fill the gaps and ensure the administrative coverage increases, and continue to deliver vital statistics about the country and its people.