Government authorities are responsible for registering, storing, and providing personal data about citizens. So, how can they ensure that this data is stored securely, while also providing quick response times for data requests and enabling entities to efficiently register data?

This question was already being asked by the Dutch government authority at the centre of this article more than 30 years ago, long before terms like data governance and data management were mainstream. The answer they arrived at back then already reflected key principles that closely align with these concepts today.

The client context: laying the groundwork for data governance

The authority went on to develop an extensive ‘Data Dictionary’, a metadata management system covering all databases, tables, and columns within the organisation. This was enriched with technical metadata (datatype, location, indexes, keys, etc.) and functional metadata (descriptions, security classification, business process, business rules, business owner).

To ensure the Data Dictionary remained up to date, a development cycle was introduced in which registering all metadata was a prerequisite for rolling out database changes. All external data requests had to be handled via XML messaging, with the message’s XSD stored in the Data Dictionary. This made it mandatory to define a clear lineage from each XSD element to its corresponding attribute in the database. If proposed database changes were deemed to undermine performance, or if new systems threatened the integrity of the data landscape, the governing body responsible for the Data Dictionary was both allowed and expected to reject them.

A strong solution indeed, but how could it be sustained?

Two and a half years ago, ALTEN Nederland and SDG, its data expert solution, was asked to help make the system “future-proof” in the face of fundamental changes on the horizon. Data creation and storage were shifting towards cloud and hybrid solutions, development teams were moving to a self-managing and agile way of working, and the growing demand for data from the user base posed a significant challenge.

Our assignment: enabling agile and governance to coexist

What initially appeared to be a simple request for technical support to respond to these challenges ultimately became a broader assignment for ALTEN: helping the client preserve the core principles behind their Data Dictionary while adapting them to a changing way of working. As development teams transitioned towards self-managing, agile practices, the existing development cycle, in which registering metadata in the Data Dictionary was mandatory, began to erode. The transition created a risk that governance requirements would be bypassed altogether.

We identified that this tension was not merely procedural, but conceptual. More fundamentally, the principles of the agile and self-managing way of working contradict the postulations of the data management system. The agile approach promotes a highly dynamic style of development, in which developers devise database changes on the fly.

How, then, can these worldviews coexist?

To address this, ALTEN and the client’s Data Architect team deliberately shifted the focus of data management from enforcement to enablement. Instead of relying on rigid controls, our teams worked to embed data management principles as part of the client’s development culture itself. By clearly demonstrating the added value of good data management and by giving teams greater autonomy in how they applied these principles, we helped create a model in which agile development and sustainable data governance could effectively coexist.

From theory to practice

The following example, while simplified, reflects a real-world situation and helps illustrate this approach in practice. An independent team was tasked with developing a software solution that operates relatively separately from the client’s existing systems. The team worked with newer, agile ways of working, which, at first glance, seemed to conflict with the client’s established data management principles. More specifically, the team deemed integration with authority-wide systems unnecessary and at odds with its focus on tool autonomy and domain-specific development.

However, a new requirement arose: an obligation to provide data externally, from the solution to the government. As the team was required to position its solution within a broader organisational context, it became subject to the rules and constraints governing that wider environment. For instance, the authority’s Data Provision Policy stated that data could only be shared externally through the Central Data Provision body. In addition, all externally provided data had to be registered in the Data Dictionary and classified by the legal team as ‘approved for external use’.

As the team began to feel lost in the contradictions between its old and new purposes, ALTEN stepped in to help bring clarity. Drawing on experience from multiple data governance approaches, our teams helped the developers better understand the intent behind the existing data management framework and how it could support, rather than constrain, their way of working.

Implemented technical solutions

At this stage of the assignment, ALTEN is fulfilling the objective stated in the client’s original request: integrating self-managing and agile ways of working with a central data governance system. The following examples illustrate some of the technical solutions we’ve provided to address the challenges described above:

• The creation of an automatic synchronisation between Azure DevOps pipelines and the Data Dictionary, enabling a more agile approach for cloud-oriented development teams.
• The implementation of a meta-driven way of working on the newly developed analytical data platform in Databricks by integrating the Data Dictionary and Unity Catalogue via Open Data Contracts.

Conclusion

The reader may be disappointed in finding no concrete answer to the questions of data management in this article. This is not a story that advocates for a specific technical solution. Even a robust and sophisticated solution like the one described here is conditional on the belief in the solution. Therefore, less tangible and more complex, the driving force behind a good data management system is a business culture which acknowledges the importance and added value of data governance as a concept.