Business Insight

Churn Models - KDD 2009

The objective of the world-level datamining completion KDD-2009 was a very common task inside the telecom industry and accurately represents the kind of tasks that are encountered in "real life" (in opposition to the purely "abstracts" tasks that are usually proposed in such academic competitions).

For the KDD2009, the final ranking is based on the average quality/accuracy (AUC) of 3 predictive models that are built using data coming from the "Orange" company (Orange is the number one of the French Telecom). The 3 predictive models to develop for the KDD2009 competition are one Churn model, one Upselling Model and one "Propensity-to-buy" (appetency) model.

The competition was a real "fight-to-the-death" because everybody wanted to demonstrate his superiority on "real world tasks". At the KDD2009, there were:

  • 1299 registered teams
  • 7865 entries
  • 46 countries:
    Argentina Germany Malaysia South Korea
    Australia Greece Mexico Spain
    Austria Hong Kong Netherlands Sweden
    Belgium Hungary New Zealand Switzerland
    Brazil India Pakistan Taiwan
    Bulgaria Iran Portugal Turkey
    Canada Ireland Romania Uganda
    Chile Israel Russian Federation United Kingdom
    China Italy Singapore Uruguay
    Fiji Japan Slovak Republic United States
    Finland Jordan Slovenia
    France Latvia South Africa
As usual for such completion, your ranking is based on the (average) accuracy of the (3) predictive models (cross-selling, up-selling, churn) that you create, the higher accuracy, the better ranking. Here is a chart that illustrates the accuracy of the top 100 contestant: the RED dot is the accuracy obtained with TIMi:
KDD2009 rankings

I guess that the accuracy obtained with other "main stream" predictive analytics software is somewhere around 0.77 (AUC=77%). Although all major datamining re-sellers participated almost certainly to the completion, they did not reveal their final ranking... So we will never know for sure the accuracy of their tool... I wonder why....

These competition results place the "TIMi suite" as the best "commercially available" predictive datamining tool in the world (the very few teams that managed to obtain a slightly better results than TIMi were all using software prototypes that are not available to the public). These results were obtained at a vendor-neutral world-level competition organized by qualified university researchers in the datamining field.

The added accuracy of the TIMi predictive models represents a tremendous difference in ROI for a telecom operator. I also encourage you to read this paper : comparison_ROI_IBM_TIMi.pdf ... that explains, using the KDD2009 case, in more details how higher accuracy means higher ROI.

Next: Predictive modelling for Loyalty programs

Test the TIMi Suite for Free!