Optimizing app monitoring process. Or how to fetch app logs from the server with one command.

This short article describes my experience in optimization of the development process. I will describe how do I fetch server logs into my local machine in one command.

In my previous article, I describe a way to automate most of the remote server operations. In this article, I will describe some additional useful abilities of shell script programming. If you do not see my previous article, please take a look first with this LINK.

Initial task

In current work become a need to monitor server logs from time to time. And as you might have guessed this issue can be solved in two ways — common simpler way, and more interesting automated way.

A simpler way is to connect to the server via ssh or ftp and take a look at the server logs to analyze them.

Second, the more interesting automated way is preparing shell script which will do most work in one command. The idea is to fetch logs to the local machine and make analytics. This way of logs processing will reduce some time to connect the server and fetch them manually.

How I solve the task

I prepare the next algorithm to solve my task

Create (if not created before) local folder for server app logs

Copy server logs folder into the local destination folder through the network

Rename copied folder with operation timestamp

After these three simple steps, server logs copy will appear on my local machine and I can analyze them with convenient for me way, with any UI editor instead of CLI editors. As I see read thousands of lines with logs in cli interface is not the best idea.

From theory to practice

First of all, I declare variables which will be used in my script. I prefer to move values to variables because it makes my script more clear and flexible. Also, it can be reused in a similar case with just changing some values in a variables declaration block.

DESTINATION_FOLDER='logs_dump'

DUMP_FOLDER_NAME=`date +"%Y-%m-%d_%T"`

SSH_KEY_PATH="./key.pem"

SERVER="<user>@<server_address>"

SERVER_PROJECT_FOLDER="<project_folder_absolute_path>"

SERVER_LOGS_FOLDER="<logs_folder_path>"

DESTINATION_FOLDER — the folder which logs folder will be copied to on local machine.

DUMP_FOLDER_NAME — the new name for coped logs folder based on the operation timestamp.

SSH_KEY_PATH — path to a permission key file.

SERVER — ssh address to connect the remote server.

SERVER_PROJECT_FOLDER — path to a project folder on the remote server.

SERVER_LOGS_FOLDER — path to a logs folder based on project folder path. In my case, logs will be stored in a separate folder in the project directory.

The second action

Create (if not exists) local folder for logs and change directory to it.

mkdir -p "$DESTINATION_FOLDER"

cd "$DESTINATION_FOLDER"

The third action

Copy logs folder from a remote server to a created on prev step folder

scp -r -i $SSH_KEY_PATH $SERVER:$SERVER_PROJECT_FOLDER/$SERVER_LOGS_FOLDER ./

For these purposes, I use the scp command. It allows copy files between two destination points when one of them — remote host. For example copy file or folder from a local machine to a remote server or in reverse direction.

Option “-r” allows to make a recursive copy of entire directories.

Option “-i” select the file from which the identity (private key).

Fourth action

Rename copied logs folder with an operation timestamp value.

mv $SERVER_LOGS_FOLDER $DUMP_FOLDER_NAME

After described above steps one of the routine tasks in my project become automated.

Whole script code:

#==================VARIABLES_SECTION=======================

DESTINATION_FOLDER='logs_dump'

DUMP_FOLDER_NAME=`date +"%Y-%m-%d_%T"`

SSH_KEY_PATH="./key.pem"

SERVER="<user>@<server_address>"

SERVER_PROJECT_FOLDER="<project_folder_absolute_path>"

SERVER_LOGS_FOLDER="<logs_folder_path>"

#==========================================================



echo ======================================================

echo Create sever logs dump...

echo ======================================================



mkdir -p "$DESTINATION_FOLDER"

cd "$DESTINATION_FOLDER"



echo Fetching server log files...



scp -r -i $SSH_KEY_PATH $SERVER:$SERVER_PROJECT_FOLDER/$SERVER_LOGS_FOLDER ./

mv $SERVER_LOGS_FOLDER $DUMP_FOLDER_NAME



echo fetched...

echo ======================================================



echo Dump folder $DESTINATION_FOLDER/$DUMP_FOLDER_NAME

echo ======================================================

Conclusions

In this article, I describe my experience of automatization of the task in a way of coping some files between local and remote machines tough network. This ability helps me with the server logs discovering, but if it will be required this approach also can be used for any other task, when copying files through the network are required.

If this article was useful for you, you can leave you claps here (even 50 times) or ask me a question if something not clear.