This article is taken from a free whitepaper produced in collaboration with data and insight specialists REaD Group which explores the scale of ‘bad data’, its implications for businesses and what can be done about it. Click here to access.

It’s inevitable that the data landscape will continue grow in complexity as new digital technologies create new data streams to add to the myriad which already exist. AI, VR and voice-recognition technologies are just three areas of innovation which are evolving at a rapid pace.

Regulations will have to adapt. Not only to datatech innovations and the ethical questions they pose, but also to meet the demands of a savvier and more data literate consumer.

The problem of maintaining data quality is already huge. Organisations believe poor data quality is responsible for an average of $15 million per year in losses, according to oft-quoted research by Gartner. While it’s impossible to verify such a figure, it gives us some idea of the scale of the problem facing businesses today.

And the challenge may yet become more daunting, as the same Gartner report states:

“This is likely to worsen as information environments become increasingly complex — a challenge faced by organisations of all sizes. Those with multiple business units and operations in several geographic regions, and with many customers, employees, suppliers and products will inevitably face more severe data quality issues.”

Consequently, the need for effective data cleaning will become ever more important. All personal datasets become outdated and inaccurate over time, it’s just a matter of when. But how quickly your data decays will depend on many factors, from the consumers or industries you serve, to the countries you do business in.

Automated data cleansing is key

It’s perhaps unsurprising that according to CrowdFlower, data scientists spend 60% of their time organising and cleansing data. However, this time can be massively reduced by API-driven data cleansing technologies which update and clean data as near to real-time as possible.

This auto-clean capability is likely to play a fundamental role in ensuring businesses can draw on useful data in a safe and secure way.

Consider the European perspective. GDPR Article 5.1 (d) specifies that personal data shall be accurate and, where necessary, kept up to date. This means every reasonable step must be taken to ensure that inaccurate personal data is erased or rectified without delay, depending on the purposes for which they were processed.

In the post-GDPR age, a prompt and effective means of data cleansing has become a must-have. Automated data-cleansing technologies are the only realistic solutions for removing the time-sapping process of manually identifying and rectifying personal datasets. In turn, this frees up an organisation’s data specialists while also enhancing information security.

There are automated solutions currently on the market which provide close to real-time update capabilities. For example, REaDConnect – REaD Group’s automation solution – flags updates as soon as changes in its cleansing database are detected. This means that when a contact moves house or goes through another life event which affects the data accuracy, REaDConnect enables businesses to action an update to the data according to their business rules and processes.

A key element is the use of an API which provides access to a secure, scalable environment to detect and process changes in the end user’s database – reducing the need for data movement and associated data security.

Solving the data governance vs data management conundrum

Data governance and data management are often talked of as separate concepts with contradictory objectives. Ursula Cottone, Chief Data Officer (CDO) of Citizens Bank said that the word ‘data governance’ “bring into play concepts like ‘control’, ‘security’ and ‘compliance’” while ‘data management’ conjures ideas such as ‘usage’, ‘functionality’ and ‘innovation’.

They resemble sworn enemies rather than happy playmates. Yet harmonising the two should be a major objective of all data-driven organisations today.

Where does the solution lie?

Powerful integrative data cleansing tools are required which support both governance and management processes.

Let’s look at the issue of governance first:

Thanks to Article 5 (e) of the GDPR, personal data must be stored for a strict time period. When that time is up, the data must be securely disposed of. The best and most efficient way to do this is via a tech solution which automatically highlights data of concern. This can play an important role in an organisation’s compliance capability and reduce the risk of exposure to financial penalties – now and in the future.

And for data management:

It’s an obvious point but worth reiterating: if you want useful data it has to be accurate. For example, if you’re sending direct mail to prospects (B2B or B2C), you want to be sure you have the right names, addresses and other relevant details. Possessing the right data means greater ROI and zero regulatory risk; the wrong data leads to wasted budget and potential run-ins with a data regulator.

For both purposes automated data cleansing technology is the enabling factor.

Prevention is the future

Data protection regulations are still maturing. Regulators are still striving to address the ethical questions posed by Big Data and the various (and advancing) technologies which are evolving to extract meaning and make use of it.

But the direction of travel is clear: regulations will evolve to give ever-greater protections to individuals and their right to privacy as new technology continues to pose new ethical dilemmas.

Bart Willemsen, Senior Director Analyst, Gartner says:

“Multiple countries are implementing regulations inspired by the GDPR principles, a movement that is likely to continue into the foreseeable future. These privacy requirements dramatically impact an organisation’s strategy, purpose and methods for processing personal data. Furthermore, breaches of these requirements carry financial, reputational and regulatory implications.” 

For data-driven organisations, this puts increasing pressure on the ability to manage and govern data. The regulatory landscape has accelerated to a place where preventing ‘age-related’ data storage error is the requirement, and not remedial action.

The knock-on effect is that technology-driven data cleansing solutions need to be capable of acting swiftly to solve potential data storage issues before they happen. Hence, a rapid ‘scan-and-flag’ API-driven solution which can sift through all the relevant data points will not only drive better business results but also play a key role in ensuring compliance as data protection regulations evolve.

Want to find out more data cleansing tools and techniques?  Download our free whitepaper produced in collaboration with data and insight specialists, REaD Group: Are you addicted to bad data? Get clean and stay clean.