On the morning of Thursday, March 11, 2004, exploding bombs ripped through four packed commuter trains in Madrid, Spain, killing 191 people. As police investigators sifted through the devastation, they determined that cell phones were used to trigger the bombs.
That evidence fueled the fire behind a directive from the European Union (EU) to require telephone, mobile and Internet service providers to store and keep detailed records from phone calls and Internet activity to aid police and anti-terrorist investigations. Investigators used mobile call detail records (CDRs) for investigations in the aftermath of the Madrid train bombings, and CDRs also proved critical for investigating the bombs that tore through London on July 7, 2005.
The ensuing regulation, known as the EU Data Retention Directive, requires telecom operators and Internet service providers to keep essential information about the time, date and location of any communication (phone calls, e-mails, text messages, etc.) that takes place over their networks. To comply with the Directive, detail records must be stored and retained – without being altered – for six to 24 months, or even longer depending on regulations of individual member countries. The Directive covers wireless or wireline call detail records, SMS logs, e-mail logs, proxy server logs and logs from network servers that assign dynamic IP addresses to individual PCs and devices in order to be available for aiding in criminal and terrorist investigations.
It all aids national security, but to comply with the EU Data Retention Directive, service providers must deal with big issues, like storing and protecting huge event data volumes, and implementing powerful search and analysis tools to quickly meet law enforcement requests for records related to a specific person or group.
The Surging Flood of CDR/IPDR Data
Every day, telcos, mobile operators and Internet service providers generate call detail and IP detail records – time-stamped, chronologically sequenced data records that capture information about an event like a phone call, text message or e-mail. In many cases, providers can generate hundreds of millions of events per day. As they comply with the mandate to store the data up to 24 months, they must cope with billions of records and hundreds of terabytes or more of total data to maintain and manage. The sheer data volume can overwhelm traditional relational databases and network storage infrastructures.
These records must be captured, placed in long-term storage, and protected from any potential changes that might corrupt the files. Managing the tremendous data volume creates a huge challenge, and with IT budgets already strained by the grim economy, many providers are looking for new approaches to help them stay within tight operational budgets.