ABCs of log file analytics

A. Aggregate data from all log files - All log files are not the same - Most people think of sys logs when they think of log files - no thats not all. Logs from product and software companies are bundles containing many files each of different types and formats. Some are time series data of events but others can be stats, session information, usage metrics, configuration etc. 

B. Bring in structure - Log files contain many different types of events, configuration status and statistical information and parsing them manually or using standard scripting language is just not enough. You need a way to create a model for analyzing a system as a whole to determine impact analysis of changes, correlating events to changes, understanding long term trends etc. 

C. Correlation of events to changes. Collecting logs and bundles and doing a simple search is one simple first step. But you cannot get to the root cause if you cannot correlate events to changes in the configuration. For example you may be seeing file system errors in your log files. You may want to see performance charts over the time period of these errors and then look at changes to the configuration in the same time period - doing all this manually is tedious since the data is in many formats in many files. What if you could see a list of changes that happened to the system automatically popup every time you get an error or a performance blip. 

With Glassbeam you can process not just single log files but logical bundles or collections of log files across various formats containing disparate sections, You can the define a structure using SPL( Semiotic Parsing Language) a DSL ( Domain Specific Language) for reusing the mapping between raw data and the intelligence you want to derive across multiple versions of your files. Finally Glassbeam makes use of this structure from parsed data and the raw log data to derive correlations and providing pre-defined apps for visualizing the relationships between various components in your stack. This ability to shorten the time to insight from raw data is leveraged by customers like IBM, Aruba, Polycom and others. Find out if your issues match our solutions.

Big Data and DSL ( Domain Specific Language )

Domain Specific Languages have been around for a long time – a great example of a DSL is SQL for RDBMS. A DSL is differentiated from a general-purpose language such as C, Java or Python since a DSL is geared towards a specific domain.

In our space of log management a DSL serves many purposes. Log files, especially multi-structured log files, contain very rich information – not only at a system level but also at a business and feature level. This information is “logged” not in one file but is spread across many files of many file types. Providing a simple search solves specific problems that IT is interested in but the usefulness of a simple search on log files stops at that. The higher value of business intelligence from log files requires a DSL. Lets see what the benefits of a DSL are.

1. Describes arbitrary text layout for automated parsing


2. Describes the context or meaning of the text in order to express more than that which is explicitly stated


3. Automatically creates an efficient structured schema


4. Defines semantics for search and browse of log files


5. Defines application specific tags – example “trend-able” attributes, Status and configuration etc.

Having this rich definition allows for a wide range of applications that can be built out of log files. For example knowing which attributes are status, configuration and trends allows an app to treat them differently and use them appropriately inside the application.

SPL™ is a DSL for machine data that enables companies like Aruba and IBM to mine their logs and enable a wide set of people inside their enterprise ranging from support and services to sales and product management to leverage this data.

More in upcoming posts on how SPL™ enables rapid development of Big Data applications for enterprise and IT.

Machine data benefits sales and service

Most people think of sysadmins and IT when they think log files. We at Glassbeam think sales operations and services! Why? Because machine data provides the truth and nothing but the truth about your customers and how they use your products. This information is extremely valuable to enterprises.

How can sales people benefit from log data?

Most systems( think storage boxes or wireless devices or medical equipment or aircraft engines) produce bundles of data - some logs have error and support related information. Some have system configuration, information on which knobs have been turned on and usage information( how long was the call and how many participants were on a call for example). Glassbeam can consume all that disparate multi-structured data and make sense of it all through its SPL(Semiotic Parsing Language).

So based on machine log information, Glassbeam can highlight which customers would run out of licenses by feature based on their past 3 months of usage, plot when they will run out and flag the account as needing follow up.

A customer may be running out of their service entitlements ( if you think your CRM system has upto date customer information think again and check your log files). Glassbeam can dashboard the customers who will be eligible for contract renewals and provide a 3 month snapshot of the customer to enable a renewal sales opportunity.

Technical sales people can quickly look at a customer's configuration before they go on a sales call. They may find that a customer has changed a configuration recently that has resulted in slowness and be prepared to address their customer concerns with actual data during their visit.

There are many more and I will showcase such examples in the following posts but Big Data from machine logs are a gold mine for businesses. Smart companies are already exploiting this to their competitive advantage.

What is the value of Big Data?

A very relevant post. This is why we focus on delivering business value rather than abstract theories like machine learning or arbitrary algorithms on big data. In many companies today, projects get started based on technology or buzz words and while these efforts are important from an R&D point of view, they are not substitutes for proper evaluation of business benefits from big data projects. C level executives need to commit to a business outcome and determine spend on big data projects based on business ROI.

Our customers are line of business executives in large companies and they understand the need to extract answers and solutions from their big data projects.

Business Impact of Machine Data Analytics in a Fortune 100 Account

I just returned from a field visit meeting key executives at one of our leading customer site.  Talk of a global account with hundreds of sales, field engineers, product specialist, support and services people scattered across all geographies around the world supporting their blue chip accounts.  And what do they use as ONE singular Big Data app to get the REAL truth from mining machine / log data from their installed base every day – nothing else but Glassbeam!  It was real encouraging to see that we are providing the most transparent and seamless information coming directly from the machines (remember, machines do not lie!) and helping these folks make the purest form of data driven decisions around support troubleshooting, product usage analysis, account up-sell information etc, etc.   Sky's the limit when it comes to finding end use cases for such information, of course the pre-cursor of which is to rightly parse and store the information in a structure that is amenable for deeper analytics on this class of machine data.

One of the examples that was highlighted in this mtg was when a customer experienced severe performance issues with their storage box.  Instead of running around in circles and trying to find what changed on the system configuration, which incidentally is one of the largest causes of support headaches, a product field engineer used Glassbeam application to see latest configuration of the entire system and could drill down and compare its latest RAID Adapter card configurations to last known values in one singular view.  All this took a few minutes and led directly to a conclusion on what may have gone wrong at this customer site.  Such quick analysis when assimilated across hundreds of cases per month, I am sure, provides tremendous competitive advantage, not to mention potential cost savings through more efficient support operations with Glassbeam.  In addition to all this, Glassbeam lets such customers avoid further damage to their installed base by running a quick report on similar faulty configurations at other end user sites (and taking proactive action there as well).

All in all, I would say it was a great experience sitting with our customers and learning about the value they are able to garner from Glassbeam Big Data Apps in their daily operations.

Loading log data for analytics

We all have situations where we wished we could instantly analyze data in unstructured formats. For example machine log data or web log data, or log data from applications. Over the years tools for analyzing weblogs (for example Google analytics) and most recently some machine data search tools( Splunk)  have made the task easier. However there are files with multi-structured data with each section having multiple formats. Further in many enterprises, log data need to be combined with other structured data to make sense. For example support logs from devices or software applications are typically associated with CRM and installed base data such as tickets, bug numbers, customer and case information etc. There is a need for a way to not only structure this data but associate existing structured data with machine data. Glassbeam has been doing this for some time now with large enterprise customers.

Our overall approach is using our patent pending SPL™ for defining the semantics & semiotics of the unstructured data and creating a platform and set of applications that can then create a normalized structure and a pre-defined set of applications to visualize that data.

As an example lets say you want to see all customers on a particular version of your product with a specific known defect and you want the trends of how often this defect as affected other attributes of the product such as performance over time and you want to create rules to alert any time such events happen. This involves parsing machine log data, combining it with a knowledge base and integrating with a CRM application. Glassbeam dramatically reduces the time to deliver this business value by providing pre-built platform components as well as a pre-defined yet configurable set of applications to report on such data.

Machine data and customer intelligence

Game companies like Disney and Zynga perfected the art and science of analytics by collecting log data from games, as they are played, and using the data to better understand their users. They gained a huge competitive edge by using the log data to tailor their products. They gained valuable knowledge on how the product was being used by their customers, pinpoint performance issues preventing the users from loading a game etc.

In the business-to-business world there are hundreds and thousands of devices, which periodically send back, log information providing information on usage, errors, various configuration parameters etc. This information is a gold mine but it’s hard to extract the gold since the log files are esoteric and not fully structured. It takes a lot of time and effort to understand the meaning and context of the data in the files. Glassbeam has been working with large enterprises over the past three years perfecting a solution to extract gold from dirt using its patent pending SPL™

.

Now, companies like IBM and Aruba have instant insights into their users and usage of their products. For example a high tech manufacturer selling complex systems, can now know real-time if a customer is close to their license limit and proactively engage the customer for upsells. There is a real time time instant dashboard, available to all employees and execs,  showing the status of all machines and devices sending data, their versions, licenses, usage patterns and enabled features. Before the sales person goes to the account, she is armed with all possible information about the account, mined from machine logs using Glassbeam and delivered in easy to use dashboards. All discussions are now based on facts rather than assumptions.

 

Leveraging machine data to reduce support costs

Most progressive support organizations are now moving to leverage Big data to become proactive. They want to put behind the days when support teams were always behind the curve, with the customer knowing about problems much before support knows about it. Further its takes days or weeks for support teams to understand what is going on based on logs uploaded. Tools to analyze logs and determine possible issues have helped but they are mostly single user tools which help highlight simple keywords and when a log file consists of bundles of files with multiple sections and formats a simple search does not help.

According to a recent survey among the support groups of 3 of the world’s top selling storage vendors, it takes an average of 12 hours to identify and resolve any issue. Of this, up to 30% of the time is spent in determining root cause - finding, organizing, and making sense of the glut of diagnostic data coming back from the product.

At Glassbeam we have been working on some very interesting solutions that dramatically improve support productivity. What if you could parse multiple sections of a log file or a set of files, whatever format they in, and apply business rules on data within the file to determine if there were known issues. As an example – say your customer uploads a log file into you salesforce.com instance( or whatever CRM you use) while creating a case. What if the log file can automatically be parsed, a quick summary of current configuration shown, rules from your knowledgebase applied to the file to determine possible issues and even recommend solutions based on previously known cases? That's a potential savings of 11 hours and 55 minutes per case!

At Glassbeam we are applying big data to solve thorny problems for the enterprise and its executives.

Looking at the softer side of execution

Many people ask me what are some of the core values (or I call them guiding principles) for Glassbeam.   We have a list of 10 things here.  But if I were to pick top 3 that make the most impact, interestingly they all relate to the "people" side of the equation.  Here they are:

Hire the best...

We should hire only the BEST and no one else. This is very true especially as we are at a critical growth phase in our evolution. “A” players tend to hire A+ players because they care about their reputation and know that is the only way to excel and achieve higher goals. “B” players end up hiring similar caliber or “C” players, and “C” players go down the rank to build a team of “D” players because they avoid tackling tough problems. By being careful and deliberate on hiring the best and the best only, we avoid a deck of cards that does not take anyone anywhere.

Never give up...

Never give up, retool if necessary, but never ever give up. Start-­-ups are like swimming in an ocean where tides come and go. By having the courage of constantly trying and not giving up, we tend to dramatically increase our probabilities of success. Markets change, strategies change, products change, but if one is clear on the core values of why we are in this together, then each downturn is an opportunity to excel at the next upturn of events.

You cannot do everything...

It is very easy to want all and not be able to focus on one or two important things that can drive product success in the market. Therefore, we owe it to ourselves to constantly keep reminding each other on making clear choices and tough decisions. This is truer for Glassbeam since we deal with data end to end as an end user application. We have to make a conscious effort to focus only on a few things that we do the best and leave others for our customers and partners to derive value on their own.

Moneyball for sales and service

With the buzz around Big-Data it is important to keep in mind what can be done with the data. For those of us who have read or watched Money Ball it is apparent that you can use insights derived from data and combine that with your intuition to make better decisions.

Glassbeam gives you the tools to make such decisions in your business, based on unfiltered machine data.

Once products are sold to customers, the product manufacturer has very limited visibility into how a customer uses the product. The customer feedback is always filtered through the words of the field person or the call center agent or the customer themselves. And not all data is captured. This is not the way the internet companies work. Web based product companies are constantly collecting data on consumer usage and mining the data to make their web pages better, to provide a better experience to their customers or to help realize additional value to simply provide a better performance by knowing and acting on issues proactively. Physical product manufacturers are reactive because the tools to collect and mine log data are not geared towards business users.

Any product executive needs to be on top of how customers are configuring and using their product and find issues before their customer does. As one of our customers put it, “It is embarrassing when my customers know more about what is happening to my product before I do”.  Many product manufacturers log or collect tons of data but don't have tools to make sense of the data being collected.

With Glassbeam, sales and service people can arm themselves with detailed customer intelligence before an account call – based on actual machine reported data. Everything can be logged and should be logged – machine usage, performance metrics, specific configuration and customer specific settings, various events across all the components that make up the storage box. Glassbeam’s customers are realizing the power of data by capturing machine chatter and mining the data for to make better decisions on support models, account management, product roadmaps and licensing.

Skills needed for big data analysis?

A recent article in Forbes opined that Big Data Analysis required Data Scientists and Quant/Excel jockeys. We agree - to an extent.

While these skills are important for an organization that wishes to do everything in-house, there are companies like us that are obviating the need for acquiring this talent. Especially if you are a manufacturer of computer-centric technology products you don't need to look beyond our SaaS-based offering to easily collect and analyze data.

What's really important is to adopt a practice, nay culture, of building products designed to collect and send back operational data. That's the battle half-won. Leave the rest to us.

And think twice before posting that Excel guru req!

McKinsey weighs in on Big Data

The Q4, 2011 issue of McKinsey Quarterly focused exclusively on big data and its implications for various industries moving forward (Free registration required). Makes for a very interesting read!

Some key takeaways:

-- Industries with the most potential for reaping benefits from big data include Finance, Insurance, Transportation and Warehousing, and Health Care.

-- Big data is spawning new business models - in some cases companies that genuinely embrace big data are turning into big data consultants for other firms

-- The majority of the economic surplus from big data is being garnered by consumers - in the form of reduced prices, better information etc

-- Big Data is putting up big numbers: A beverage manufacturer improved forecasting accuracy by 5%; retailers reducing the number of items it stocks by 17% and so on.

-- And perhaps most importantly for this economy, Big Data Analytics is expected to generate 140K-190K additional specialist jobs, as well as create the need for an additional 1.5 million managers!

Way to go!

So, what is SPL?

Simply put is stands for Semiotic Parsing Language. But, more importantly it is the core of our technology.

SPL is an intuitive language that describes the structure and semantics of a class of documents. For semi-structured data, as is typical with our customers, it means we can utilize the structure inherent in the data and reflecting it in a high-performance and highly normalized data warehouse.

The SPL description of a class of documents and its semantics is passed through an interpreter to generate the database Data Definition Language (DDL) for staging, database DDL for the final warehouse, DDL and Data Markup Language(DML) for metadata to generate the UIs, and internal representations to perform a parse, a transactional or bulk load, and ETL transformations.

Please contact us if you need more technical details on our technology.

Our customer-onboarding process

We've developed an effective process for bringing new customers on board.

While a detailed description of each phase is beyond the scope of a blog post, here's an outline:

1. Discovery and Documentation - document amongst other things what aspects of a log file need to be processed.
2. Analysis - essentially study log files and the feasibility of parsing relevant sections with SPLi
3. Design - here's where we identify and codify the relationships between various objects.
4. Development - develop cool SPL code to parse our client's data.
5. Testing
6. Deployment - including setting up the infrastructure and populating it with existing/historic data.
7. Provisioning - ensuring user access to the system.

Keen to try it out? Start off with a Jumpstart

Glassbeam and support teams

We're in the midst of finalizing a case study featuring how one of our customers used Glassbeam to make it's support organization more efficient. Will be sure to update you when we launch the case study, but can share some key findings,

One of the greatest benefits of using Glassbeam was the usage of threshold events and automatically opening cases in CRM systems (hitherto done by L1 support personnel). This resulted in substantial cost savings.

Further, the client was able to support a growing install base without increasing headcount in a linear fashion. Thereby, accruing more revenues without corresponding increase in cost - all adding up to revenue and margin increases!

Bottom lime was a 20% reduction in costs. And, a more satisfied and productive support team.