Implentation of Automation scripts

Overview

This document provides information on the implementation of Tech stack installation automation script, which aids in clear understanding on how this script was made.Understanding the implementation of this scripts also helps in any future modifications and any future bugs related to version updates.

Script Structure

Here’s an overview of the repository structure:

  • Devops-automation-scripts
    • ConfigurationFiles
      • install.conf
      • configuration.conf
      • databases.conf
      • haproxy.conf
      • haproxyErrors.conf
      • monitoring.conf
    • <projSpecific>ConfigurationFiles
      • samefiles as in ConfigurationFiles directory
    • TechStackInstallation.py
    • ConfigureScript.py
    • StartStopScript.py
    • MonitoringSetup.py
    • tomcat.service
    • haproxy.service
    • prometheus.service
    • prometheus.yaml
    • mysql_exporter.service
    • node_exporter.service
    • blackbox-exporter.service
    • mimir.service
    • mimir.yaml
    • pushgateway.service

ConfigurationFiles:

Usage

  • A folder/directory needs to be created with <project_name>ConfigurationFiles which should contain all the files from the default ConfigurationFiles directory and project specific changes need to be done for the files in this project specific configuration files and specify this project name and required application names while running the TechStackInstallation.py

install.conf

  • Variables required for installation parameters such as installation paths, application versions, network interfaces

    • Variables are divided into sections based on thier relation(such as application specific).
    • Each section is declared with thier unique names enclosed in Brackets(’[ ]’) and each section is differentiated with the given declaration and an empty line.
    • Variables are mentioned in the format <varName> = <varValue> (without quotes).
    • This file will be accessed by TechStackInstallation.py(when project name is specified as command-line argument) to process the variables and declare them in run-time(this will be explained clearly in the later sections).

configuration.conf:

  • Variables such as hostnames, ports to be allowed through firewall, configuration file changes, host file additions required for configuration of installed applications as per the project’s requirements are specified here in the same format as in the install.conf

  • This file is intended to be used by the ConfigureScript.py when passed with the project name.

databases.conf

  • This file consists variables related to database schemas, users, and base data file paths and passwords for a specific project.

haproxy.conf

  • This file consists the default configuration contents of the haproxy.cfg and it will be used in the installation process, where the file will then be moved to given haproxy configuration directory on the server.

haproxyErrors.conf

  • This file consists a dictionary like data with keys as filenames for status code and value as content for the html structure to be returned when this error occured and the content will be displayed to client through HAproxy

monitoring.conf

  • Similar to install.conf and configuration.conf, this file consists of variables required to install the monitoring stack with required parameters also specific to requirements of the project.

Script files

TechStackInstallation.py

  • Each application that needs to be installed is defined as an individual function inside this script and involves inclusion and declaration of required variables such as versions, urls, hostname, file paths, required passwords. These variables are declared at runtime using exec() function by passing each line from the .conf files provided and as each line is declared as a <var> = <value> and when exec() function interprets it, it will be declared as a variable. The value is declared as a string by enclosing the value between double quotes while passing the line to exec() funtion.

  • This script accepts project name and application names as command-line arguments for applications to be installed. The specified project name with suffix as COnfigurationFiles folder needs to be present in the directory where the script is being run. So, that it accepts the project specific parameters from the files provided. (Make sure to change the variables for the project’s requirements)

  • Example::

    python3 TechStackInstallation.py eev haproxy tomcat mysql scylla

ConfigureScript.py

  • It follows the same structure as TechStackInstallation.py while declaring the variables required to configure the installed Applications.

  • The variables from configuration.conf will be read here and gets declared to use in the configuration script.

  • Variables with values such as .sql file names, and other schema file names and basedata files. And usernames and passwords required to authenticate with mysql and cql shells.

  • Example:: python3 ConfigureScript.py eev confiration.conf

StartStopScript.py

  • This script is designed to help in starting and stopping installed tech stack in an order databases, messaging queues, tomcat server, haproxy load balancer.

  • This script accepts a single command-line argument which is of any action from start, stop, restart.

  • Example:

    python3 StartStopScript.py start

MonitoringSetup.py

  • The script is coded just as TechStackInstallation.py which reads it’s parameter variables from ConfigurationFiles/monitoring.py and it is defined with functions each installing exporters and grafana, prometheus.

  • This script accepts command-line arguments to install required exporters or tools based on provided arguments.

  • Example:

    python3 MonitoringSetup.py monitoring.conf node_exporter mysql blackbox-exporter pushgateway grafana prometheus grafana-mimir haproxy-exporter

Service FIles:

  • Service files for haproxy, tomcat are created before hand with variable substitutions inplace to provide for custom path configurations for installed binaries and environments.

  • Service files for several exporters(node_exporter, mysql_exporter, blackbox-exporter, mimir, prometheus, pushgateway) are also defined here to copy to the /etc/systemd/system path for correct installation.

  • Base configuration files for prometheus, mimir in yaml format are pre-defined to be modified in the script while configuring them as per our requirements defined in the conf files for the applications or for data source: prometheus

  • The service files defined with variable subsituttions are then get called in the script files where every variables gets initialized from the conf files and the service files will then be substituted by the .format() function to place respective values in place.