Digging in Logs to Debug Critical Government Infrastructure 

Marek Mägi
Marek Mägi, Head of Software Development Department
 
The availability and integrity of systems here at RMIT is critical for finances in Estonia. To illustrate, every spring, most of the country comes and presses a button we host to declare their taxes. As our daily application log volumes are immense, SpectX fits like a glove.
 
A significant number of our processes revolve around logs. To keep everything running smoothly, we need to be proactive in detecting errors and figuring out the root causes.
The data we use for this is diverse and voluminous. There are different requirements, access rights issues, etc. Logs contain sensitive information, tax secrets and PII. Essentially, the user base is the whole country - private persons as well as organisations.
 
RMIT opted for SpectX because it fit perfectly with our existing log architecture. Most of our applications are in clusters, and the endless volumes of logs cannot stay at the production server. So we move them to hot/cold storage depending on priority. When looking for a log analysis solution, it was ideal to leave the logs as they were - gzipped in a file system - and SpectX allowed us to do precisely that. SpectX' processing speed is stunning - we can solve most of our use cases, search and filter the data, get a full picture across different application nodes without moving or uncompressing the logs.
 
Essentially, we can now weave the whole data flow from logs into a single view. The users of our systems interact with many applications via the SSO. SpectX sews these steps together and gives us a full picture of the sessions, and we can easily search for particular error messages. As SpectX integrates with AD, we're also using it for data access management. What’s more, SpectX allows us to perform the auditing of log management itself.
 
As there are a lot of teams using SpectX, we've found it useful to use Git to manage SpectX' patterns (schemas) and queries. There are two files that each admin has to add to their application: the location of logs and the pattern to parse it. A  bamboo job will then run and give input to the SpectX' resource tree, i.e. nothing gets lost and is overwritten if anybody happens to tweak files directly in the tree.
 
One of the most prominent business cases for us when using SpectX is the Personal Data Usage Monitor. It is one of the fundamental principles of the E- Estonia to give our citizens a comprehensive view of how their personal data has been processed by the government. Now, the existing architecture proposed by the Estonian central IT-authority would require us to build an X-road service on top of every application.
SpectX helps us to get away with much less work and provide more accurate feedback to citizens. What we’re doing is 1) taking our audit log across all the applications that we otherwise should build an X-road service on 2) parsing and filtering the necessary fields with SpectX 3) implementing business rules (e.g. queries made for criminal investigation should not be exposed) and 4) lastly creating an x-road service on top of this SpectX view.
 
Our logs are big data - too big for moving and copying. This is why SpectX’ solution was ideally cut out for us - the compressed logs can stay in the server, and we can still have the full picture.

About RMIT
RMIT (Information Technology Center of the Ministry of Finance) is an Estonian government agency providing ICT services for the Ministry of Finance of Estonia, Tax and Customs Board of Estonia, Statistics Estonia, State Shared Service Center and to most of the Ministry of Culture administration domain. The services include workstation management, central services, business services and software development, accompanied by expertise in quality management and information security.

Back to case studies