Log management is not a new concept. In the 1980s it was the primary mechanism for fault analysis and management of computer systems. The sheer success of log data nearly killed it off before it really took off. The cacophony of log formats and the sheer volume of messages generated – up to 40 terabytes a month, or 100,000 log messages every second, for a mid-sized organisation – makes it virtually impossible for any human being to realistically track logs. Subsequently, based on SNMP alerts and other event data, including selected error log messages, large-scale event management systems such as HP OpenView emerged as the new kings of fault detection.
For a while, all was good; but then came the big compliance demands and legislation, including PCI DSS which mandates that in order to prove compliance, organisations must have knowledge of specific events and activities – for example, be able to track and monitor all network resources and cardholder data. They must also allocate responsibility for checking security log files for breaches, be able to report on the findings and take action to remedy any issues raised. In short, organisations need to implement better security best practices to protect themselves and customer data.

Making a comeback
These compliance drivers, along with increased security concerns, have revived log file management, which otherwise may not have made it back. The requirement to track user activity, provide forensic data which could stand up in court, and identify potential insider and outsider intrusions and transgressions of corporate networks has resulted in a new, updated form of log file analysis. There are now technologies making easy work of capturing, analysing and storing the huge volumes of log data, allowing organisations to access the information in a similar fashion to using a search engine.
Now, in addition to aiding in compliance, virtualisation and the ever-increasing cost of downtime in our networked economy has resulted in system and network administrators rediscovering log data and the value it delivers. In surveys, over 70% of organisations confess their primary budget for log management still comes from compliance. However, this same group admits that for years now, 70% of their use of log data has been driven by operational needs such as fault detection and problem isolation. This is no surprise, because operations use cases can drive true log management ROI.
One minute of downtime could cost millions, so if automating log management can help to accelerate problem isolation, then companies are willing to pay. If giving helpdesk employees access to normalised
log data can offload expensive
third-level support personnel, that is even better.
A new dawn has broken for log management in operations and there are even more opportunities for the channel. ●