The more a company knows about its customer’s wishes, needs and habits and the more that company is able to tailor its proposition accordingly, the greater the value it will eventually provide for its customers. We all know that there are countless examples where defective, fragmented, or just plain poor customer data cause unnecessary costs, decrease in revenue, employee dissatisfaction or frustation, damage of the corporate image and many other unsdesirable or painful consequences.

Customer data quality and integration problems impact every area of the value chain of organisations. Far too often companies have a multiple view of their customers. Customer Data Integration (or MDM for Customer Data) is the key to providing companies with a single view of their customer. According to Gartner, Customer Data Integration (CDI) is a combination of technology, services and processes to deliver an accurate, timely and complete view of the customer across multiple channels, lines of business, departments and divisions drawing customer data from multiple sources and systems.

I think that the real “miracle” of CDI lies in the automated, intelligent matching of customer records. Mind you, I’m not questioning the importance of the various CDI-processes (for example, I think that the post of my colleague Ramon de Noronha on the creation of “golden” records is majorly important), I’m just  saying that -whenever the integration of customer data is an issue- intelligent, automated  matching is the key prerequisite for success.

The quality of your customer data integration solution is only as powerful as the quality of your matching engine. If  this statement intrigues you, I strongly advise you to read the white paper “High Precision Matching at the heart of Customer Data Integration“. Enjoy!

The term Golden Record is closely related to Customer Data Integration or MDM for Customer data. It refers to the “single truth” which has been created or calculated from all those duplicate customer records from different systems. This post is not about finding or tagging all those duplicate records. There all kinds of ways to find them using advanced statistical methods, fuzzy matching etc.

But what do you once you have found the duplicates. How do you create the best possible customer data out of all gathered elements?

First of all we have to define what is meant by the Golden Record. We at Human Inference use the acronym ACCU, short for Actual, Correct, Complete and Unique. Ofbviously, we want one unique record. That’s why we use matching or identity resolution software. But Actual, Correct and Complete are less absolute, they can be interpreted in a subjective manner. You can have never-ending discussions about it, build the most complex business-rules ever etc. But I prefer to start with simply determining the superlative of Actual, Correct and Complete. In other words the most actual, the most correct and the most complete data-element or attribute “wins” and makes it to the Golden Record. Let’s take the following example, two almost identical records are gathered from two different systems (A & B).

The basic rule is that only Correct data will make it into the Golden Record. So, if you can validate data please do so. For instance you can check social security, bank account and credit card numbers using algorithms. You can validate email addresses. Using postal reference data, it is also possible to verify the correctness of addresses. The most difficult is to validate names. Extensive knowledge is needed to check whether names of persons and organizations are valid.

In my own experience and opinion you should always discard incorrect data, or let it be corrected by a data steward. In the end nobody should be in doubt whether a Golden Record has been established using doubtful data.

The next step is to examine attribute (field) by attribute. So using the example from above.

So using validation techniques to distinguish the correct data from incorrect data and determining the length of each attribute in the provided records will result in the following Golden Record:

Mister J.F. (John) Miller

26 Spring Gardens

Manchester, Lancashire, M2 1BB

United Kingdom

Even if you have a lot more of attributes in your Golden Record, this method still works. Determine the correct data and use only correct data. And using the function Length (LEN) to determine the “most complete” data. Most complete simply refers to consisting of the most characters. If the source systems also provide dates for “date entered” and “date last changed” you can use this to determine what the most recent data is. The most recent data is determined by formulas like MIN (”CurrentDate” minus “”Last Changed Date”).

I believe this method will lead to a very usable Golden Record in 90 to 95% of all cases. Only when you have to deal with complicated data, for instance father and son living on the same address and having the same initials it becomes much more complex. I am curious which rules-of-thumb and methods you use when calculating the Golden Record. Please put your ideas in the comments.

Last week I travelled to India and just at that time one of the largest deduplication projects in the world had been accepted. The project is to provide every Indian citizen with a Unique Identification Number.

Main goals are (amongst others):

• Make life easier for the citizens by diminishing the number of ID documents they have now
• Minimize the fraud possibilities for several projects and welfare schemes
• Possibility to share information between different disciplines and organizations

Anyone who has ever been in India knows that you absolutely need to take into account the variety of cultural aspects in that huge country. In Western Europe it is already very difficult to deduplicate all kinds of citizen data, given all the languages and cultural aspects. I think, however,  that the degree of difficulty is even worse in India, where not all citizens have a registered birth certificate, most will have their first official registration from school, some do not have a last name, addresses are not always that trivial (euphemism), and the whole country is used to the fact that typos are allowed in names, because in one area Shrivastava is actually the same as Srivastava (without ‘h’).

As long as human beings are involved in the deduplication, most things work fine. The problem starts if the human factor is diminished and cold computation takes over. Ask the average Indian colleague about his/her experience to get a visa for the US, or even a plane ticket, given his/her original ‘official’ documents. Indian officials do have experience with this, because it is not the first time this project is launched.

Why will it be successful now? In my opinion, because they have asked Nandan Nilekani as head for the Unique Identification Authorithy of India, and with his experience in both business and IT, his global knowledge on how to tackle such huge problems, he seems to be the best foundation for this project. What is lacking is a serious data quality solution, with knowledge of the cultural aspects in India to bridge all information from the citizens, stored in various data sources to that one unique number. I do actually know a company that can solve that!

Our experience in Western Europe is that  even after the introduction of such unique numbers, you will face fraud, multiple citizens or persons sharing the same ID. And the system needs to be aware of that. Never assume that a Unique ID is really unique! The moment you forget that, the system is open for criminals.

• Having the right contact details will reduce waste of natural resources because we bring the deliveries immediately at the right place, and it’s not only the deliveries that can be optimized, we can also avoid that deliveries get lost and natural resources are actually piped for /dev/null !
• By valueing our data through deduplication we can in general avoid to spoil needless energy – both by humans and other resources – and use the sparse energy only for those who actually need it. Here I feel the same remark as David in his blog. There comes a moment in the near future, with an rising energy prices and increasing emission penalties, that that  aspect will win in the equation from the  actual spoil of goods and human energy.
• Saving resources is now also done by concentrating or centralizing services – optimizing the service per energy unit. For data we see this happening in the Virtualization of data amd Master Data Management technologies. Strong place in your centralizing strategy will be the role of your data quality – that will bring your real value

I encourage you all to think out-of-the-box how data-value can help to make it a better world for the future. But I’m afraid that in this economic climate the short term is ruling and not the long(er) term.

A happy new year and be sure to visit DataValueTalk.com in 2009 for continued data value.

Regardless of the purpose of the single customer view and the associated architectural design there’s one clear common divisor: at the heart of every single customer view solution a powerful matching engine is necessary. Within the matching engine the secret of a high quality single customer view is concealed. The quality of the single customer view and the resulting business values depends on the power of the matching technology. If, for instance, the matching engine is not capable of recognizing that “BMW” and “Bayerische Motorenwerke” is actually the same customer you will still accrue unnecessary costs or miss value creating opportunities (opportunity costs) within your company. Because the quality of the matching engine is so important for the final value the single customer view solution is able to deliver, we distinguish “standard matching” from “High Precision Matching”. This whitepaper explains the need for a High Precision Matching engine at the heart of any single customer view solution. 

David Cearley at Gartner comes with a fascinating other angle. Basically he sees virtualization also as strategic technology to virtualize the data. And by that twist, data quality and data governance appears annoyingly in the middle of your radar screen. In order to use this strategy for your operational excellence, to eliminate the number of redundant data on your real storage devices, and make a virtual layer between your applications and this virtual data storage, you need to be sure that all your applications can work seamlessly with that virtual data.

To significantly reduce your amount of redundant data you need to make it more agnostic. Your apps will still expect the data in a specific format, while your virtual data is stored in another way. You need tools to convert it quickly – back and forward – from the standard virtual way (e.g. first-name, prefix, last-name, zip-code, thoroughfare, etc) to the specific application way (e.g. name in one field, zip-code, street).

Virtualization of data is not only about deduplication of the data! The challenge is more on how to use the virtual data in your existing applications in a non-invasive way. This can be done by using good data quality tools for your cleansing and deduplication, and for your – back and forward – standardisation.

In general I agree with the top 10 of technological strategies, be there some slight personal priority adaptations, but let’s focus on that in later blogs. The missing point is in my opinion the lack of emphasis on risk mitigation, and I do realize that things changed since October 2008. Which technologies can we adopt to avoid that we provide services, products, at the end money to the wrong contacts, or that we are sure to deliver it to the right contacts. The technology strategy of Master Data Management, Know your customer, Single View of X, or how we call it, will need our attention in 2009!

The added value of an integrated customer view depends strongly on the quality of that integrated customer view. Every organization that is seriously planning to create a single customer view should ask itself the following question: “What determines the quality of my customer view and so the accompanying level of added value?”

Prior to answering this question we need to take one step back. Why does not every organization have a single view of the customer? The cause lies in the fact that many organizations have their customer data spread across multiple systems all facilitating separate business processes. Additionally customer data is often highly polluted, fragmented and incomplete.

So it appears that the data itself plays a crucial role in the lack of an integrated customer view. Or more accurately, the better the data – the better the customer view.  And the better the matching of customer records across separate systems the better the integrated customer view.

So Data Quality and Matching (Identity Resolution) determine in large parts the quality of the integrated customer view and the added value that it delivers. Take a look at this demo showing a step-by-step approach how to build a single customer view and get a better idea of the role of Data Quality and Matching within this process.

It must have been around 2002, that I was discussing the Return On Investment of Data Quality solutions with one of the founders of Human Inference, Norbert Mergen. While discussing the well known benefits: less return mail, more effective campaigns, reduction of debitor risk, single customer view, … I brought another subject at the table: isn’t it strange that we really do ROI calculations on such an obvious need? Did you ever create a fence in the garden and question the ROI of a hammer? We published on this matter in dutch back in 2002 in the CRM Marketing Centre and included the hammer discussion. And now, in 2008, it is so interesting to see that many people nowadays have put the same questions, reading the blog of Jack Vinson ”Stop thinking ROI, think success!” Anyway, it may not convince your management, so you will still need to do the maths, but just bringing the subject to the table may help you getting your data quality project going.