As I was sitting on a terrace in Barcelona during my recent holiday, I found a copy of the Indenpendent, the well-known British newspaper. Having all the time in the world, I started reading and I came across  this article about the North Yorkshire Police storing data of more than 180,000 people, including their date of birth and ethnicity. The vast majority of these people had given this information voluntarily and had not committed any crime.

When privacy campaigners questioned the need for compiling such a database, a police spokesperson answered: ” The system is used by many police forces in the UK and internationally to record all information relevant to policing, everything from details of arrested individuals, suspects, victims, witnesses and sources of information as well as addresses, phone numbers and vehicles. The information logged and cross-referenced in the system is absolutely vital to allow us to provide the effective policing service that the people of North Yorkshire and the City of York demand.”

I think that this is a very dangerous comment. What about the possibility of mixing up data of witnesses and criminals? How do the police forces create an unique view on their “customers”? What will be the consequences of so called “database errors”?

Of course I understand that the police forces all over the world need information to do their work properly and to prevent crime and other undesirable behaviour. But reading a comment like the above, I really wonder whether law enforcement agencies are  really aware of the essential role of data quality in modern police work.

When the Thomas family from Ohio embarked on a recent trip from Cleveland to Minneapolis, they were in for a huge, but unpleasant surprise. It appeared that 6-year old Alyssa Thomas’ name was on the Homeland Security no-fly list; a list that is used to prevent individuals with known or suspected ties to terrorism from flying. The girl’s father, Santhosh Thomas, states that the worst thing his daughter has ever done, is probably been mean to her sister, but that this should hardly be a matter for the Department of Homeland Security.

The Thomases were eventually allowed to fly that day, but they were told to contact Homeland Security to clear up the matter. Now Alyssa just received a letter from the government, notifying the six-year-old that nothing will be changed and they won’t confirm nor deny any information they have about her or someone else with the same name.

According to the Transportation Security Administration, the body responsible for the Secure Flight List,  it is likely that Alyssa never had problems flying before. The TSA used to check only international passengers’ names against the no-fly list, but since earlier this month has decided to check domestic passengers as well. To us data quality professionals, this very much sounds like a case of incorrect identity matching…. Could it be that Homeland Security is in need of some intelligent tools?

For more information realated to this subject, please read the blogposts Your name is too “common”…. and Attempted bombing Christmas Day could have been prevented!

An increasing number of companies have to deal with data from the world’s fastest emerging economy: China. And the big question in this issue is of course: How can we compare these “strange” Chinese characters with our own writing set? 

Grammar and character set of our Western alphabet-languages (such as English, French, Dutch or German) differ tremendously from Mandarin Chinese (which is the language spoken by most in the People’s Republic of China and abroad. Mandarin is a tonal language with an ideographic character set. Almost all characters have a semantic and a phonetic component. The different pithch in the pronunciation eventually determines the signification

Complicated? Definitely. But what about the other way around? Have you ever thought about the difficulties the Chinese have to face when trying to convert their language into meaningful English?

This phenomenon is sometimes hilariously being illustrated by the many public signs in China used to inform foreign visitors or to help them finding their way around.

This is truly a delightful side-effect of internationalization. …. 

The German sinologist Oliver Lutz Radtke christened these linguistic attempts “Chinglish” and collected many examples, which can be found virtually everywhere: on hotel room doors, on road signs along the highways, shampoo bottles and t-shirts. A small anthology:

• A warning sign for a steep slope: “Please, watch your slip”
• To avoid all misunderstandings, on the inside of a taxi door: “Don’t forget to carry your thing”
• A sign above a store entrance, to let our fantasy run free: “Welcome to presence”

Although this is all very funny, from a data quality point of view, this definitely leaves a thing or two to consider. For example: What should we think of fault-tolerance with regard to typo’s when we think of entering Chinese customer data into a database? What is the influence of typo’s in an ideographic writing set on searching, matching, enriching and correcting customer data?

There is still a lot of work to be done in international data quality. For more information, check out the Human Inference website on HIquality Name Worldwide.

Have you ever tried to get contact details in and out of a CRM system, and ended up with a bigger mess? I have. The concept is easy: store all information about prospects and customers in one system, allowing you to have your communication efforts streamlined.

Reality, however, is harder: contact details entered on your website should be fed to the system automatically. Sending your periodic newsletter should be based on the details in your CRM system. Not to mention dealing with information on bounces. Integrating your CRM system(s) with mass mailing, campaign management and self service portals is helpful, but for some reason the major means of transporting lead and customer information still seems to be Excel… Leaving you with the necessity to mass import results, new contacts and changed information.

What you really want is the ability to throw whatever information you have at the system, and let the system determine if you already know someone and take care of adding information to already existing contacts. In addition you do not want to spend time on maintaining information you have. Actually, wouldn’t it be nice if you would be able to have your website automatically add leads to your CRM system (e.g. via the Salesforce Web2Lead function), but without  adding duplicates?

The good news is that this can be done. There is tooling out there that keeps your contact data up-to-date, prevents adding duplicate records and makes dealing with contact data easy.

Everybody who has ever been on holiday in France has probably had a neighbour named Gaston, Jacques, Louis, Claire or Françoise . We are used to those first names, they evocate the “France profonde”, sleepy villages at the end of a road, films of Pagnol or Rohmer. Walks along the Seine in de shadow of “Notre Dame” in the spring. Coffee at a terrace of the Boulevard Saint-Germain where an obsequious garçon, named Marcel, is looking at your girl friend or wife in a way you dot not really appreciate. This particular image of France is in danger. In a few years our total frame of reference could have disappeared.

Nowadays French parents let their imagination go freely when they are choosing first names for their children. Looking at recent entries in the civil registry, you will find rather unusual first names like Bulle, Héribert, Loeva, Hermès, Evolène, and Argan.
These first names have all kind of origins. For example, they can be a combination of first names (Timéo, which is derived from Timothée and Théo),or they are different writing forms of known first names (Lilou becomes Lee-Lou). We can also find names from Greek or Celtic mythology or even from literature, like Arwen, a character from the novel Lord of the Rings.

This interest for uncommon first names will of course have consequences for the processing of French data, especially if you take into consideration that these “new” first names, with a frequency less than 3000, are now in the majority. But this diversity will not necessary be a curse. Maybe we will be delivered from ambiguous names, of which we never know whether it is a first name or a surname. Consider the 3 most common surnames in France are Martin, Bernard and Thomas. 
But don’t worry; in order to keep the challenge going when you process French data, names like Jacqueline-Germain, Jean Marie Marie Luce, Louise Alexandrine will of course not disappear entirely.
And next time you will be in France, enjoying a salade Niçoise with a cold glass of rosé, overlooking a harbor, where small fishing boots are dancing on the lazy waves, you just will have to get used to the fact that the waitress’s first name is not Marie but Fanchon or Eole.

Source:  Le Parisien 19-02-2010 and “Les 4 000 plus beaux prénoms rares “, de Stéphanie Rapoport, chez First, 8,90 €

The Norwegian Fødselsnummer (Birthnumber) is an 11-digit number with 2 control digits. The 10-th digit is a control digit calculated with a weighted modulo 11 variant over the first 9 digits. The 11-th digit is a control digit calculated with another weighted modulo 11 variant over the first 9 digits combined with the 10-th control digit.

As in other countries also this number is based on the date of birth. The first 6 digits represent the birth date as “ddmmyy”. Problem with a 6-digit date is that you cannot identify the century – is a Fødselsnummer starting with 121009 someone born in 1909 or 2009? The Norwegian government has solved this by grouping the following 3 individual digits (individual number) in groups representing a certain era. If you are born between 1854-1899, then your individual number must be between 500 and 749, born between 1900-1999 then your number lies between 000 and 499, and for those born recently between 2000-2039 then your number lies between 500 and 999. With some exceptions for those with an individual number between 900 and 999.

Like in other countries the odd individual numbers are given for males, the even for females.

Be aware that by validating national personal identification numbers, like the Fødselsnummer, that contain a date part you cannot rely only on the control digits. The Fødselsnummer “31046812355” is completely valid if we look to the 10th and 11th control digit – however the birth date April 31 in 1968 did never occur!

Professional matching engines are becoming more and more intelligent. Within Human Inference, we also see that our matching techniques are capable of using more and more intelligence, and needless to say that we incorporate and use this intelligence in our engines in order to adopt to the way that humans do their matching.

Traditional data quality or matching engines were based on atomic string comparison functions like match-codes, phonetic comparison, Levenshtein string distance, n-gram comparisons or similar functions. These kinds of functions are relatively easy to implement and to use although a significant amount of plumbing is needed to get reasonable results. Open source projects like the Lucene search engine, and variants, provide a solid and proven set of these functions. The drawback of these functions is that it’s not always clear for what purpose one needs to utilize a particular function. An even larger issue is the fact that these low-level DQ functions cannot distinguish between apples and oranges – you end up comparing family names with street names. We still see that,  for example BI vendors,  claim to provide data quality functionality, while they only provide these atomic comparisons.

Within Human Inference we have been developing matching engines that look beyond these primary functions for years. Engines capable of identifying given names, surnames, family names, postal codes, titles,  initials, etc. The true benefit of this approach is that matching results are significantly higher, because you are comparing apples with apples and oranges with oranges. The glueing or plumbing in this approach to validate street or family names is completely under the hood for the data stewards. With a correct set of reference data, the right mix of atomic functions and – not the least – vivid domain knowledge, these matching engines are capable of quickly and adequately finding duplicates – beyond the ones that have simple typos.

The complexity in matching apples starts if you take into account the variants in apples, or to speak in Data Quality terminology, in case you take into account that per country or region people have more or less subtle differences in using names, streets, measurements and writing sets.

The moment you value these differences you also recognize new opportunities. You will notice that by looking at an apple, you get information on oranges. By looking at the name Белоусовa (Beloussowa), you might recognize a family name and that you’re dealing with a female. By looking to the number 681012-2355, you might recognize that this is a valid Swedish personnummer, and that the birth date of this male is October 12, 1968. By looking to an email like Winfried.vanHolland@humaninference.com you might recognize a given name “Winfried”, that you’re dealing with a male, that he has surname “van Holland” and that he is working for a company called Human Inference, and I leave it up to you from which country he originates…. By retrieving additional information out of obvious information, the matching moves beyond the apples and oranges, and becomes easier, faster and more accurate.

The Finnish national personal identification number is the Henkilötunnus, aka Hetu or Ht, has the following format – ddmmyyc999C. For details how to calculate the control character, I refer to the overview blog on National Identification Numbers.

Validating  the Hetu 270368A172X shows that it is indeed a correct number. The number 270368172 generates  indeed 29 for the modulo 31 proof, represented by control character “X” in the checksum list. The number shows that this is the 86-th girl born on the 27th of March 2068.

The latter might is exactly the start for the discussion on validity. Althought the number itself is well formed, and passes all the automatic checks, dealing with this number in a data quality assessment will raise your digital eyebrow. In the data quality world we will nowadays say that this Hetu is a wrong Hetu, that it cannot be correct.

So always use a bit of human inference when dealing with finnish national personal identification numbers.

The national personal identification number in the Netherlands is called the Burgerservicenummer (or abbreviated with BSN, introduced since november 2007). It is a 9-digit number where the number can be validated by a weighted 11-proof. Basically all the digits become a weighting factor and by calculating the sequential digits with their weight the final result must exactly be divisible by 11.

A nice effect of this weighted 11-proof is that there are at least 2 digits different between 2 individual numbers. You need to perform at least 2 changes to come from one number to another – it might be that there are 2 completely different digits (e.g., 112682765 and 112682777) or the you need to swap one digit and change another (e.g., 427096509 and 427096510).

Mathematically it might still be that there are two succeeding numbers like 427096169 and 427096170, which still need 2 changes to come from the one to the other.

This effect helps in preventing mistakes while typing these numbers, you need to make more than one mistake and some bad luck to get exactly a number that matches the proof.

For those who like statistics, there are exactly 90909090 possible combinations – which in itself is a nice number but doesn’t match the proof. The first possible number is 000000012 (assuming that 000000000 is not used), the last is 999999990.

For more on Personal Identification Numbers I refer to another summary blog on European numbers or to a handsome presentation

The struggle of some countries has to do with historical reasons and with privacy aspects. Unique identifiation is not always used in favour of the community. And some of the used identification systems contain privacy-sensitive information, among others date of birth, gender and/or place of birth, where older systems might even contain religious or other privacy-senitive information.

A wide range of countries use the combination of date of birth, gender identification and the political region where you are born. In such a mechanism it is most common that part of the identification number is a 2-digit or 3-digit serial number to identify the unique male or female born on a specific date (or born on a specific month). Some countries provide odd serial numbers for male, and even for female. Bulgaria is the only one that wants “odd” females. Some countries like to divide on range (0-499 male, 500-999 female).  And some countries like Norway make nice combinations to include the century of birth or period of birth in the serial number.

This ‘number’ generation brings the effect that pretty soon you will encounter the maximum number of citizens that the system can handle on a specific day. Some systems run out of numbers if there are more than 500 males or females born on a day. The Denmark system encountered that situation in 2007, where due to immigration the population exceeded the system for January 1st 1965! The Denmark system (CPR-nummer)  has a 3-digit serial number where one of the digits is also the control digit (diminishing the possible numbers than from 500 to less than 50).

Remarkable to see what some countries are doing to solve the ‘century’ issue, people with the same ID but born in the 19th, 20th or 21st century, they add 20 or 40 to the month. Same is true for foreigner identification, e.g. Sweden that is adding 60 to the day of birth. Or again Sweden that is adding 20 to the month to distinguish persons from organisations.

If you want to see the details on these systems you might watch http://prezi.com/csnv3cynv4ai/ or http://en.wikipedia.org/wiki/National_identification_number. Be prepared, definitely there have been PhDs around to invent these systems.