Tuesday, November 20, 2012

ABCs of log file analytics

A. Aggregate data from all log files - All log files are not the same - Most people think of sys logs when they think of log files - no thats not all. Logs from product and software companies are bundles containing many files each of different types and formats. Some are time series data of events but others can be stats, session information, usage metrics, configuration etc. 

B. Bring in structure - Log files contain many different types of events, configuration status and statistical information and parsing them manually or using standard scripting language is just not enough. You need a way to create a model for analyzing a system as a whole to determine impact analysis of changes, correlating events to changes, understanding long term trends etc. 

C. Correlation of events to changes. Collecting logs and bundles and doing a simple search is one simple first step. But you cannot get to the root cause if you cannot correlate events to changes in the configuration. For example you may be seeing file system errors in your log files. You may want to see performance charts over the time period of these errors and then look at changes to the configuration in the same time period - doing all this manually is tedious since the data is in many formats in many files. What if you could see a list of changes that happened to the system automatically popup every time you get an error or a performance blip. 

With Glassbeam you can process not just single log files but logical bundles or collections of log files across various formats containing disparate sections, You can the define a structure using SPL( Semiotic Parsing Language) a DSL ( Domain Specific Language) for reusing the mapping between raw data and the intelligence you want to derive across multiple versions of your files. Finally Glassbeam makes use of this structure from parsed data and the raw log data to derive correlations and providing pre-defined apps for visualizing the relationships between various components in your stack. This ability to shorten the time to insight from raw data is leveraged by customers like IBM, Aruba, Polycom and others. Find out if your issues match our solutions.

No comments:

Post a Comment