Enterprise application integration
Logs and privacy
Many enterprise services rely on logs. The most commonly known example is technical customer support resolving end-users issues based on log information. As simple and straightforward this may seem to implement, there are two hard-to-solve problems: how to guarantee the privacy of end-user data and how to manage changes in log structure?
One of the first things to think about is how logs are accessed. Collected logs are generally long text files, often compressed and the most common approach for reviewing them is to use command line tools and text editors. This requires a direct access to the log repository, which is complex to manage and hard to scale. Customer service needs to have skills, that require training. Often this is solved by delegating actual search and analysis to technically skilled admins. However, the problem of user data privacy is still there, as admins can access entire logs containing data from all customers without any audit trail. This is a problem, especially under the GDPR which amongst other things requires retaining access evidence of customer data.
Parsing the point-and-click
Another way of solving this is putting an application layer in front of log search. What could be simpler than setting up a web form with a few fields with customer id, time period, etc, and then searching log records according to these parameters (and keep an audit log, too)? Sound simple except doing that first requires PARSING these logs, i.e extracting data elements according to the expected structure of a log record. An in-house developed application can only solve this with a static parser. And here comes the second problem: how to manage changes in log structure? This is a particular concern with application logs that change very often [https://www.spectx.com/articles/processing-machine-generated-data].
This will result in a hard dependency between your main business applications and applications parsing their logs. That is, changes in business applications cause a need for changes in the parsing application.This inevitably leads to increased implementation cycles. Not to mention the increased complexity in an already complex enterprise application environment.
Replace manual labour with SpectX
SpectX is a perfect addition to enterprise applications for avoiding expensive complexity. It is also the perfect middleware to replace time and effort spent on repetitive manual log analysis tasks. The queries can be executed over RESTful API which serves as an integration point for enterprise applications. Built-in detection for changes in the underlying data structure and powerful pattern matching language allow quick change detection and adoption. The information retrieved from logs can be strictly limited based on the rights of the user. Naturally, all the activities are audit-logged.
Back to solutions