Have you ever tried to get contact details in and out of a CRM system, and ended up with a bigger mess? I have. The concept is easy: store all information about prospects and customers in one system, allowing you to have your communication efforts streamlined.

Reality, however, is harder: contact details entered on your website should be fed to the system automatically. Sending your periodic newsletter should be based on the details in your CRM system. Not to mention dealing with information on bounces. Integrating your CRM system(s) with mass mailing, campaign management and self service portals is helpful, but for some reason the major means of transporting lead and customer information still seems to be Excel… Leaving you with the necessity to mass import results, new contacts and changed information.

What you really want is the ability to throw whatever information you have at the system, and let the system determine if you already know someone and take care of adding information to already existing contacts. In addition you do not want to spend time on maintaining information you have. Actually, wouldn’t it be nice if you would be able to have your website automatically add leads to your CRM system (e.g. via the Salesforce Web2Lead function), but without  adding duplicates?

The good news is that this can be done. There is tooling out there that keeps your contact data up-to-date, prevents adding duplicate records and makes dealing with contact data easy.

Everybody who has ever been on holiday in France has probably had a neighbour named Gaston, Jacques, Louis, Claire or Françoise . We are used to those first names, they evocate the “France profonde”, sleepy villages at the end of a road, films of Pagnol or Rohmer. Walks along the Seine in de shadow of “Notre Dame” in the spring. Coffee at a terrace of the Boulevard Saint-Germain where an obsequious garçon, named Marcel, is looking at your girl friend or wife in a way you dot not really appreciate. This particular image of France is in danger. In a few years our total frame of reference could have disappeared.

Nowadays French parents let their imagination go freely when they are choosing first names for their children. Looking at recent entries in the civil registry, you will find rather unusual first names like Bulle, Héribert, Loeva, Hermès, Evolène, and Argan.
These first names have all kind of origins. For example, they can be a combination of first names (Timéo, which is derived from Timothée and Théo),or they are different writing forms of known first names (Lilou becomes Lee-Lou). We can also find names from Greek or Celtic mythology or even from literature, like Arwen, a character from the novel Lord of the Rings.

This interest for uncommon first names will of course have consequences for the processing of French data, especially if you take into consideration that these “new” first names, with a frequency less than 3000, are now in the majority. But this diversity will not necessary be a curse. Maybe we will be delivered from ambiguous names, of which we never know whether it is a first name or a surname. Consider the 3 most common surnames in France are Martin, Bernard and Thomas. 
But don’t worry; in order to keep the challenge going when you process French data, names like Jacqueline-Germain, Jean Marie Marie Luce, Louise Alexandrine will of course not disappear entirely.
And next time you will be in France, enjoying a salade Niçoise with a cold glass of rosé, overlooking a harbor, where small fishing boots are dancing on the lazy waves, you just will have to get used to the fact that the waitress’s first name is not Marie but Fanchon or Eole.

Source:  Le Parisien 19-02-2010 and “Les 4 000 plus beaux prénoms rares “, de Stéphanie Rapoport, chez First, 8,90 €

The Norwegian Fødselsnummer (Birthnumber) is an 11-digit number with 2 control digits. The 10-th digit is a control digit calculated with a weighted modulo 11 variant over the first 9 digits. The 11-th digit is a control digit calculated with another weighted modulo 11 variant over the first 9 digits combined with the 10-th control digit.

As in other countries also this number is based on the date of birth. The first 6 digits represent the birth date as “ddmmyy”. Problem with a 6-digit date is that you cannot identify the century – is a Fødselsnummer starting with 121009 someone born in 1909 or 2009? The Norwegian government has solved this by grouping the following 3 individual digits (individual number) in groups representing a certain era. If you are born between 1854-1899, then your individual number must be between 500 and 749, born between 1900-1999 then your number lies between 000 and 499, and for those born recently between 2000-2039 then your number lies between 500 and 999. With some exceptions for those with an individual number between 900 and 999.

Like in other countries the odd individual numbers are given for males, the even for females.

Be aware that by validating national personal identification numbers, like the Fødselsnummer, that contain a date part you cannot rely only on the control digits. The Fødselsnummer “31046812355” is completely valid if we look to the 10th and 11th control digit – however the birth date April 31 in 1968 did never occur!

Professional matching engines are becoming more and more intelligent. Within Human Inference, we also see that our matching techniques are capable of using more and more intelligence, and needless to say that we incorporate and use this intelligence in our engines in order to adopt to the way that humans do their matching.

Traditional data quality or matching engines were based on atomic string comparison functions like match-codes, phonetic comparison, Levenshtein string distance, n-gram comparisons or similar functions. These kinds of functions are relatively easy to implement and to use although a significant amount of plumbing is needed to get reasonable results. Open source projects like the Lucene search engine, and variants, provide a solid and proven set of these functions. The drawback of these functions is that it’s not always clear for what purpose one needs to utilize a particular function. An even larger issue is the fact that these low-level DQ functions cannot distinguish between apples and oranges – you end up comparing family names with street names. We still see that,  for example BI vendors,  claim to provide data quality functionality, while they only provide these atomic comparisons.

Within Human Inference we have been developing matching engines that look beyond these primary functions for years. Engines capable of identifying given names, surnames, family names, postal codes, titles,  initials, etc. The true benefit of this approach is that matching results are significantly higher, because you are comparing apples with apples and oranges with oranges. The glueing or plumbing in this approach to validate street or family names is completely under the hood for the data stewards. With a correct set of reference data, the right mix of atomic functions and – not the least – vivid domain knowledge, these matching engines are capable of quickly and adequately finding duplicates – beyond the ones that have simple typos.

The complexity in matching apples starts if you take into account the variants in apples, or to speak in Data Quality terminology, in case you take into account that per country or region people have more or less subtle differences in using names, streets, measurements and writing sets.

The moment you value these differences you also recognize new opportunities. You will notice that by looking at an apple, you get information on oranges. By looking at the name Белоусовa (Beloussowa), you might recognize a family name and that you’re dealing with a female. By looking to the number 681012-2355, you might recognize that this is a valid Swedish personnummer, and that the birth date of this male is October 12, 1968. By looking to an email like Winfried.vanHolland@humaninference.com you might recognize a given name “Winfried”, that you’re dealing with a male, that he has surname “van Holland” and that he is working for a company called Human Inference, and I leave it up to you from which country he originates…. By retrieving additional information out of obvious information, the matching moves beyond the apples and oranges, and becomes easier, faster and more accurate.

The Finnish national personal identification number is the Henkilötunnus, aka Hetu or Ht, has the following format – ddmmyyc999C. For details how to calculate the control character, I refer to the overview blog on National Identification Numbers.

Validating  the Hetu 270368A172X shows that it is indeed a correct number. The number 270368172 generates  indeed 29 for the modulo 31 proof, represented by control character “X” in the checksum list. The number shows that this is the 86-th girl born on the 27th of March 2068.

The latter might is exactly the start for the discussion on validity. Althought the number itself is well formed, and passes all the automatic checks, dealing with this number in a data quality assessment will raise your digital eyebrow. In the data quality world we will nowadays say that this Hetu is a wrong Hetu, that it cannot be correct.

So always use a bit of human inference when dealing with finnish national personal identification numbers.

The national personal identification number in the Netherlands is called the Burgerservicenummer (or abbreviated with BSN, introduced since november 2007). It is a 9-digit number where the number can be validated by a weighted 11-proof. Basically all the digits become a weighting factor and by calculating the sequential digits with their weight the final result must exactly be divisible by 11.

A nice effect of this weighted 11-proof is that there are at least 2 digits different between 2 individual numbers. You need to perform at least 2 changes to come from one number to another – it might be that there are 2 completely different digits (e.g., 112682765 and 112682777) or the you need to swap one digit and change another (e.g., 427096509 and 427096510).

Mathematically it might still be that there are two succeeding numbers like 427096169 and 427096170, which still need 2 changes to come from the one to the other.

This effect helps in preventing mistakes while typing these numbers, you need to make more than one mistake and some bad luck to get exactly a number that matches the proof.

For those who like statistics, there are exactly 90909090 possible combinations – which in itself is a nice number but doesn’t match the proof. The first possible number is 000000012 (assuming that 000000000 is not used), the last is 999999990.

For more on Personal Identification Numbers I refer to another summary blog on European numbers or to a handsome presentation

The struggle of some countries has to do with historical reasons and with privacy aspects. Unique identifiation is not always used in favour of the community. And some of the used identification systems contain privacy-sensitive information, among others date of birth, gender and/or place of birth, where older systems might even contain religious or other privacy-senitive information.

A wide range of countries use the combination of date of birth, gender identification and the political region where you are born. In such a mechanism it is most common that part of the identification number is a 2-digit or 3-digit serial number to identify the unique male or female born on a specific date (or born on a specific month). Some countries provide odd serial numbers for male, and even for female. Bulgaria is the only one that wants “odd” females. Some countries like to divide on range (0-499 male, 500-999 female).  And some countries like Norway make nice combinations to include the century of birth or period of birth in the serial number.

This ‘number’ generation brings the effect that pretty soon you will encounter the maximum number of citizens that the system can handle on a specific day. Some systems run out of numbers if there are more than 500 males or females born on a day. The Denmark system encountered that situation in 2007, where due to immigration the population exceeded the system for January 1st 1965! The Denmark system (CPR-nummer)  has a 3-digit serial number where one of the digits is also the control digit (diminishing the possible numbers than from 500 to less than 50).

Remarkable to see what some countries are doing to solve the ‘century’ issue, people with the same ID but born in the 19th, 20th or 21st century, they add 20 or 40 to the month. Same is true for foreigner identification, e.g. Sweden that is adding 60 to the day of birth. Or again Sweden that is adding 20 to the month to distinguish persons from organisations.

If you want to see the details on these systems you might watch http://prezi.com/csnv3cynv4ai/ or http://en.wikipedia.org/wiki/National_identification_number. Be prepared, definitely there have been PhDs around to invent these systems.

 

Lack of understanding of the complexity of international names caused a near-accident successfully prevented by the Dutchman Jasper Schuringa.

 On Flight 253, on its way from Amsterdam to Detroit, a passenger tried to explode the airplane. This passenger was not called John Smith, or Peter Johnson. No, his name was a little more complicated: Umar Farouk Abdulmutallab. Easy to misspell, and that is exactly what happened. A misspelling of the name of Umar Farouk Abdulmutallab resulted in the State Department believing he did not have a valid U.S. visa.

 We love damage control, not prevention

Now we are introducing bodyscan devices, to detect at the airport what could have been detected earlier, if only we would have applied proper technology to detect the misspelling of the name. Government and also ompanies I speak with seem to be satisfied with simple old-fashioned comparison algorithms to “comply”. If we compare the name against the watchlist or against database X, we have done our duty…..

Well, I don’t believe so! It is your responsibility as a governmental organization or company, to act responsible and do the best you can to prevent terrorism. This is not: “let’s do the absolute minimum to comply to the regulations”. This sad example again shows this is irresponsible. A safer world starts with responsible behavior and thinking about prevention. One of the keys for prevention is using sophisticated name-search technology. This will keep us from introducing more expensive solutions for damage control, or worse: me having to fly naked the next time when I visit the USA.

It becomes more challenging if there is no longer a link between nickname and official name. That may happen, for example, if people move from one cultural region to another where also other writing sets are used. Take for example my chinese friend 高为民, whose Latin name would be Gao Weimin (family name first), but the moment he works in Europe or the US he is using the Latin variant William Gao. There is no common relation to the name William and Weimin both in Latin or Chinese and it they are no phonetic variants of each other.

Recently, I have read a very impressive book from Dave Eggers, called `What is the What´. It gives you a good insight in one of the current problem areas of the world and how people try to survive there. Achak Denk is one of the so-called Lost Boys from Sudan. During his live in Sudan, in refugee camps and finally in the US he is officially using differnt names. That has nothing to do with purposely trying to mystify his identity, but more with receiving an identity from your environment – at that time and place. He is born as Achak, baptized as Valentino, and later on using the name Dominic or Dominic Arou and  Marialdit. Of course there are people calling him nick names as ‘Sleeper’ or ‘Gone Far’ but at certain periods in his life he is officially using completely different names. This makes automatic matching of persons, or even manual matching, challenging and keeps it interresting.

I would recommend the book to everyone who wants to learn about what is happening in our world, and especially those interested in names (don’t forget to study all the names in the last Section of the book).

In the evolution of information technology Gartner provided a new term as ultimate goal to reach: Pattern-Based Strategy.

As you were reaching for the  final destination in your ultimate journey to transform bits and bytes to real information, again you encounter a new optimum. Pattern-Based Strategy, as described by Yvonne Genovese et al. can be identified as the last era in all the eras of  IT-value add. Basically, the level of control identifies in which of the era you currently operate – from tight control and pure automation in the ‘old’ days via augmentation, e-commerce/Web 1.0 and web 2.0 to the highest era called – Pattern-Based Strategy.

In Pattern-Based strategy you make use of the collective intelligence, not only collect data inside and outside your enterprise from those you know, but pro-actively collect information from the crowd, analyse this information and see if you can recognize the patterns on which you know how to respond. Try to be fast on weak signals so that you are the first in that market.

As in Business Intelligence (where you look in your rear mirror to make decisions for the future) and Business Activity Monitoring, within Pattern-Basted Strategies (where you look around and even let others look around while you make decisions) it only works if you can compare data, if you can make information out of your data. In order to make the right company decisions, to steer well, you need to rely on the information that you use. And although we like the new approach behind Pattern-Based Strategy, we as Human Inference, still see enterprises struggle with their Business Intelligence because of their data quality. We see them starting with Business Activity Monitoring and fail because of their data quality. And now we see them start dreaming  of Pattern-Based Strategy. Before you act on that dream, solve the data quality – there are solutions available.

For those who already tried to combine information from social communities, the lack of standards there, prevents that you can easily collect that collective intelligence for your patterns. As Human Inference,  we value your data, we  identified these issues and provide data quality tools to enrich this collective intelligence in order to transform it to trusted information for your strategy.