Worker - let's make it work

https://reaction-engine.bitbucket.io

Let's start quickly what the worker is for: we have many business applications to be observed which can be done with the reader worker. What it does is to read the log file of application if a known pattern (like .*OutOfMemoryError: Java heap space.*) occurs there it will report it to the Reaction Engine as a possible incident. The engine will make a decision and it might start an execution flow (chain of commands) to remedy to problem which will be executed sequentially by the executor worker on one or more hosts.

So the workers seem pretty important, don't they? Ok, so let's install them first.
What has to be done to make them work on a host?

1. install the ZIP file from HERE
   wget https://bitbucket.org/ric_flair_wcw/reaction-storage/downloads/worker-1.0.zip -P /local/reaction
2. unzip it to a folder and add execute permission to manage_*.sh
   unzip /local/reaction/worker-1.0.zip -d /local/reaction/
   chmod a+x manage_*.sh
3. configure it

The first 2 steps are cakewalk and the 3rd one isn't difficult either.

So the config can be found in conf/worker.yml.
The most important settings that have to be taken care of are as follows:

• host_name: Reaction system will link this worker to the system record stored in system administration of the management application with this host_name value. For example: if you define a system in the management app with the host name ADNLT653 then the same value has to be set for the host_name too in the config file. It can be anything but highly recommended to use the real name of the host.
• rest_call.engine_endpoint: It is the endpoint URL of Reaction Engine. For example: http://localhost:8080/reaction-engine
  

And that's all! There are many other settings in the config file that are for refining the worker but only these two have to be set at the beginning.

In the manage_*.sh files the Java home (J_HOME) variable has to be set. If the JAVA_HOME is set correctly then no alteration is needed.
For Linux the EXEC (path to jsvc) and the USER (the user to run the daemon) variables have to be checked too.


The last check should be to have a look at the credential file file and set the password there properly. The same password (and username) has to exist in Reaction Engine too.

I recommend the following places to get more info about how to install worker:

• https://reaction-engine.bitbucket.io/doc.html#2_1_2_How_to_install
• https://reaction-engine.bitbucket.io/doc.html#A_4
• download the Dockerfile from HERE and have a look into it how the worker is installed

Worker - what does it do exactly?

https://reaction-engine.bitbucket.io

First question is why we need the workers at all? Without them the Reaction Engine wouldn't be able to be notified about incidents and OS commands (the parts of the remedy process) couldn't be executed.

Let's assume there are 5 applications that have to be monitored and fixed and 2 of the applications have 4 instances (the remaining 3 applications reside in separate server machines). It means that the reader workers have to be installed on 2 * 4 + 3 = 11 server machines.
It seems a lot but one reader worker can monitor more applications (more precisely it can read more log files that belong to the business applications) on the host machines where it was installed to. Every log reading process runs in a separate thread.

When the reader worker reported a possible incident and it turns out in the Reaction Engine that it is really a problem that has to be dealt with then the engine will start an execution flow (which is a series of OS commands). These commands will be executed serially by the executor worker.
How can the executor worker get the command to be executed? All the executor workers poll the Engine (i.e. call a specific REST service every x seconds) and only that executor will receive the command in the REST response where the OS command has to be executed. After the execution of the OS command the result / output will be sent back to the Engine and the next command (if it was successful) will be called by an executor worker.

So the reader / executor workers are like the ants which serve the Reaction Engine (the queen ant) with information but they are vital to make the whole Reaction system work.

In the next blog I will explain how to make the worker work.