THE CHALLENGE

A client has a highly complex Platform that is built upon a large number of services and processes, each of which generated its own log, thus making it difficult to use them for statistical purposes, monitoring or problem solving.

The system for consulting these data logs was based on a web application developed ad hoc to filter and list the information.

THE SOLUTION

The solution consisted in the implementation of a centralized log system and the collection of records of API REST calls to services (outgoing from the system and incoming to the system) to a specific Database.

As a result, the log information would be available at a single point eliminating intermediate processes so that:

  • The overall efficiency of the system was improved and the possibility of errors was reduced.
  • The info exploitation could be done more easily and efficiently.
  • More sophisticated and efficient statistical graphing and reporting tools could be used, eliminating maintenance and evolutionary developments of its associated visualization website, with the consequent cost savings.

TECHNOLOGICAL SOLUTION

ELK Stack was used for the solution, by providing the Elasticsearch search and analytics engine in which to store all the log information. Using Kibana, users can view data in charts and graphs from that stored information.

The different Platform processes and services (implemented in Java) would use the Appenders of the Log4J library (library for recording transactions at runtime) to centrally dump the information of logs and Rest API calls in Elasticsearch.

Due system requirements, an Asynchronous Appender was chosen as it improves efficiency by executing log dump operations in a different Thread than the business flow from which it is performed.

Finally, users could easily consult and monitor the information through the Dashboard visualizations (tables and graphs) created in Kibana.