Nodes Creation

Panchayatseva: 5-Node Cluster Setup

Panchayatseva is a high-performance, scalable system deployed on a 5-node cluster.

1. Infrastructure Setup

1.1 Node Installation

Server Configuration: The cluster consists of three nodes, node-1 is installed on server1.

  • Hardware Requirements:

    • CPU: 3
    • Memory: 10GB
    • Disk: 128GB

  • Operating System: Ubuntu-22.04

  • IP: 172.21.0.61

  • Port: 59222

Step-1: Create node-1 using below command.

sudo virt-install \
  --name pssb1avm001 \
  --os-variant ubuntu22.04 \
  --vcpus 3 \
  --memory 10240 \
  --location /data1/os-iso/ubuntu-22.04.4-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
  --network bridge=virbr20,model=virtio,mac=0a:01:e1:20:00:01 \
  --network network=virho10,model=virtio,mac=0a:01:e1:10:00:01 \
  --disk size=128,pool=adm-images \
  --uuid 00000000-0000-0000-0000-0a01e1200001 \
  --graphics none \
  --extra-args 'console=ttyS0' \
  --debug

Step-2: Disk partition for the node.

  • /: 25 GB
  • /boot: 1 GB
  • SWAP: 8 GB
  • /opt: 34 GB
  • /data: 60 GB

Step-3: User Setup

  • Full Name: Devops Admin sayukth
  • Server Name: pssb1avm001
  • User: devopsadmin
  • Password: <1!Devxxxx>

Note:

  • Select the SSH server option during the installation process.
  • Add User to Sudoers: Use visudo to add the following line at the end of the file: 
    devopsadmin ALL=(ALL) NOPASSWD: ALL
  • Make static the IP address of this node through winbox.

1.2. Tech Stack Installation

1. Clone the GitHub Repository

Execute the following command to clone the repository containing the automation scripts:

git clone https://github.com/DevOps-Model/DevOps-Automation-Scripts.git

2. Navigate to the Repository

Change directory to the cloned repository:

cd DevOps-Automation-Scripts

3. Copy the resource files to the temp

Copy the psorbit resource files to the temp directory:

cp -r pssbConfigurationFiles/sb-resources-conf/ /tmp/sb-resources-conf

4. Run the TechStackInstallation Script

Execute the TechStackInstallation.py script to install the required tech stack:

python3 TechStackInstallation.py pssb java maven haproxy tomcat git scylla redpanda mysql

5. Configure the Services

After installing the tech stack, configure the services using the following command:

python3 ConfigureScript.py pssb configuration.conf
  • Start the mysql and scylla services.
sudo systemctl start mysql@bootstrap
sudo systemctl start scylla-server

6. Run Base Data

Once the services are configured, run the following command to initialize the base data:

python3 ConfigureScript.py pssb databases.conf

Finally, the tech stack is completely installed, configured, and initialized with the base data.

1.3 Hardening the Node

  1. Go to the Hardening folder inside the repository:
cd Hardening
  1. Execute the script to harden the node:
bash ubuntu.sh
  1. If you want to verify the hardening score, run the following command:
bash checkScore.sh
  1. To increase the hardening score, refer to the below documentation

2. Generate Nodes for the Cluster

PSSB is a 5-node cluster with nodes distributed across multiple servers and a local data center.

Node Server/Location
Node 1 Server-1
Node 2 Server-2
Node 3 Bare metal (Local DC)
Node 4 Server-2
Node 5 Server-1

Node 2 and 4 Setup

Generate additional nodes for the cluster from a base node in server-2.

1. Copy the disk image of node-1.

  1. Node must be in shutdown state
virsh shutdown pssb1avm001
  1. Copy the disk image of node-1 from server-1 to server-2.
scp /data2/adm-images/pssb1avm001.qcow2 itadmin@172.21.0.32:/tmp

2. Run the Script to Generate Remaining Cluster Nodes

Create and execute the following Python script to generate two nodes for the cluster in server-2:

  1. Copy the below content save it as generate_nodes.py file.
import subprocess

def run_command(command):
    print(f"Running command: {command}")
    subprocess.run(command, shell=True, check=True)

emp_suffix_list = ['002', '004']

ip_prefix = '172.21.'
ip_list = ['0.62', '0.64']

mac_list = ['02', '04']

project_name = 'pssb1avm'

source_disk_img = '/tmp/pssb-node01.qcow2'
source_xml = '/data1/guest-xml/dev-template.xml'

base_firstboot_script = '/root/scripts/startup-script.sh'

base_uuid = '00000000-0000-0000-'

for i in range(len(emp_suffix_list)):

    emp_id = emp_suffix_list[i]

    target_name = f'{project_name}{emp_id}'

    target_disk_img = f'/data2/adm-images/{target_name}.qcow2'
    target_xml = f'/data2/guest-xml/{target_name}.xml'
    target_firstboot_script = f'/root/scripts/{target_name}.sh'

    target_bridge_mac = f'0a:01:e1:30:00:{mac_list[i]}'
    target_host_mac = f'0a:01:e0:10:00:{mac_list[i]}'

    target_uuid = base_uuid + target_bridge_mac.replace(':', '')

    content = ""
    with open(source_xml, 'r') as file:
        content = file.read()
        content = content.replace('dev-ein0034', target_name)
        content = content.replace('0a:01:0d:30:de:35', target_bridge_mac)
        content = content.replace('0a:01:0d:10:de:35', target_host_mac)
        content = content.replace('/data1/guest-images/dev-template.qcow2', target_disk_img)

    with open(target_xml, 'w') as file:
        file.write(content)

    run_command(f"cp {source_disk_img} {target_disk_img}")
    
    script_content = ""
    with open(base_firstboot_script, 'r') as script:
        script_content = script.read()
        script_content = script_content.replace("mac-id", target_uuid)
        script_content = script_content.replace("iptobereplaced", ip_prefix + ip_list[i])

    with open(target_firstboot_script, "w") as start_script:
        start_script.write(script_content)

    run_command(f"virt-sysprep -a {target_disk_img}")
    run_command(f"virt-sysprep --hostname {target_name} -a {target_disk_img} --firstboot {target_firstboot_script}")
    run_command(f"virsh define {target_xml}")

  1. Run the above script.

    python3 generate_nodes.py

Node 3 Setup (Bare Metal Device)

  1. Create a Bootable USB:
    • Use Ubuntu 22.04 Live Server to create a bootable pendrive.
  2. Install the OS on the device using the following details:
    • Full Name: DevOps Admin Sayukth
    • Server Name: pssb1abm003
    • Username: devopsadmin
    • Password: <1!Devxxxx>
    • IP Address: 172.21.0.63
  3. Follow above techstack installation procedure upto hardening the node.

Node 5 Setup (VM on Server-1)

  1. On Server-1, follow the same installation steps as Node 1.
  2. Use the following details:
    • Server Name: pssb1abm005
    • IP Address: 172.21.0.65

Start the Nodes

Once the nodes are generated, start them using the following command:

virsh start <node_name>

Example:

In Server-01:

virsh start pssb1avm001
virsh start pssb1avm005

In Server-02:

virsh start pssb1avm002
virsh start pssb1avm004

Make the IPs Static in Winbox

Log in to the router through Winbox and assign static IPs for the generated nodes.

Update Hostnames in the Nodes

Log in to the nodes and update their hostnames in the /etc/hosts file:

vi /etc/hosts

Add or modify the following entries:

172.21.0.61    pssb1avm001
172.21.0.62    pssb1avm002
172.21.0.63    pssb1abm003
172.21.0.64    pssb1avm004
172.21.0.65    pssb1avm005