||Add To My Personal Library
November 18, 2005
Vol.27 Issue 46|
Page(s) 33 in print issue
LMI In The Enterprise
Analyzing Logs Was Never This Exciting
Poring over log data to find out what caused a system crash or network attack can be a dull and annoying task. But the necessity is growing. Every second, small to medium-sized enterprises are generating up to hundreds of thousands of log data messages from firewalls, servers, routers, and applications. Some enterprises are even producing up to 40TB of critical information every year. However, logs are often overlooked because of differences in formatting, complexity of information, and a lack of sound methods to deal with arrant volumes of information. So the need to automate ways to deal with this data is growing at the same time. These pressing realities have generated a new industry called LMI (log data management and intelligence).
The Log Predicament
In a report called The Log Management Industry, the SANS Institute defines the log management predicament as such: As computers become more numerous and regulation compliance becomes more a part of daily life, some system administrators are finding that log management is becoming a problem. The scripts and manual processes that have historically been used by 80% of the market need to be upgraded. Log issues tend to snowball as the size of a company grows.
LogLogic, an LMI solution provider, has its own view of present log management industry hurtles. The companys very existence is based on tackling some of these issues. LogLogics mission is: To remove the complexity of collecting, storing, reporting, and alerting on any log data thereby simplifying compliance tasks, reducing costs, and minimizing complexity and risk.
Dominique Levin, vice president of product management at LogLogic, shares her view on automation and integration through log management. "The end-goal is to fully automate compliance and business processes, utilizing information in log data. For example, if you fire an employee, you want your workflow system to talk to your log management systemyou want the log data message (that confirms that the employee's user accounts were deleted) to automatically close a trouble ticket in the workflow system. We are working on an open log services interface to make this integration possible.
The LogLogic Approach
There are three key technologies that power the LogLogic solution: First is a high-performance, distributed architecture that is highly scalable and secure. This platform lets the company collect 100% of all log data from any device with high data integrity. Secondly, it has proprietary alerting, search, and reporting algorithms that are built on top of this architecture. Essentially, it uses spare horsepower to come up with more intelligent algorithms to automate log analysis. Levin says, For example, we have machine learning that powers our Log Learning alerts, we have super fast response time for our Agile Reports, and we have Google-like search algorithms. Thirdly, the company has the first open log services architecture that provides log routing and a Web services-like interface to enable business process automation and automated policy validation.
But the main question is this: Why should SMEs be interested in participating in log management at this level? For starters, todays compliance and regulatory demands alone are calling for better log management. From time to time, enterprises are called on by lawyers and regulators looking for compliance information regarding staff (and ex-staff) network and information access issues. Because enterprises are becoming increasingly regulated environments, they have a real responsibility for protecting and being able to produce log files.
Levin notes the necessity for SMEs to be concerned with log management. She says, There are two main reasons: 1) Its mandated by regulation such as HIPAA, SOX, [and] PCI (credit card processing standard), and 2) It saves time and money. Most companies are using homegrown log management servers, which are ineffective and inefficient. Imagine searching the Web without having Google or Yahoo!; that is what our customers face. They are looking for critical information in log data, but every question can take days to answer. Yet they cant give up because log data contains mission-critical information to substantiate regulatory compliance and to validate security and privacy policies.
The Google Comparison
We thought Levins Google comparison as it relates to log management was interesting, so we asked her to elaborate a bit. She says, Did you know that up to 25% of all enterprise data is log data? That is a lot of information currently being overlooked or underutilized. Like Google, we make this data useful and accessible for the first time. Levin says her company delivers tailored information to many different stakeholders in the organizationfrom the systems administrator interested in troubleshooting to the legal counsel interested in forensic evidence. She continues, Also, like Google, we have an API that third-party applications such as SIEM (security information and event management) and network management products can use to subscribe to a feed of relevant log data.
Levin says there is much happening in this space, but most solutions, she says, are convoluting the marketplace. She notes, Many people confuse SIEM with Log Management and Intelligence. That is a mistake because SIEM is really no more than one of many applications that makes use of log data. SIEM is focused on a data reduction problem: by placing agents on target devices, they filter out strings from the logs to help with things like reducing false positives from IDS/IPS systems. Levin says this is all very important but also very different from building a complete audit trail around 100% of all logs. She says 90% or more of log management is being done with homegrown processes, where combinations of products are being used for monitoring and alerting.
Because the current legal and regulatory climate is making increased demands on IT departments, the log management industry should be around for a good while yet. Many companies will continue to have a growing need for log storage and processing as infrastructure and computing equipment use continues to grow. LogLogic claims that it has the first comprehensive automated LMI solution (an easy-to-use, plug-and-play appliance that is nonintrusive and doesnt require agent technology). And with a company mantra like 10-minute install, 10 seconds to reporting, where can an SME go wrong?
by Chris MacKinnon