PSSB Gluster Filesystem setup

Setting Up a 5-Node GlusterFS Replication File System

Process of setting up a GlusterFS volume with 5 nodes: pssb1avm001, pssb1avm002, pssb1abm003, pssb1avm004, and pssb1avm005. Each node will have 10GB of storage for the brick, and the volume will be replicated across all 5 nodes.

Node and Disk Configuration

  • Node Names:

    • pssb1avm001 - 172.21.0.61 (KVM Node 1)
    • pssb1avm002 - 172.21.0.62 (KVM Node 1)
    • pssb1abm003 - 172.21.0.63 (Baremetal Node)
    • pssb1avm004 - 172.21.0.64 (KVM Node 2)
    • pssb1avm005 - 172.21.0.65 (KVM Node 2)

  • Brick Path:

    • For KVM nodes: /export/vdb/brick
    • For Baremetal node (pssb1abm003): /data/export/vdb/brick

  • Mount Point: /data/pssb

  • Disk Capacity per Node: 10GB per brick

Step 1: Install GlusterFS on All Nodes and configure

  1. Install GlusterFS on all 5 nodes:

    • On each node, run the following commands:
     add-apt-repository ppa:gluster/glusterfs-11
 apt update
apt install glusterfs-server

  2. Configure port usage:

    • On each node, replace the content in the file /etc/glusterfs/glusterd.vol with:
    volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option transport.socket.listen-port 24007
option ping-timeout 0
option event-threads 1
#option lock-timer 180
#option transport.address-family inet6
option base-port 49152
option max-port  49252
end-volume

  3. Configure firewall rules:

    ufw allow 49152:49252 && ufw allow 49152
ufw reload
ufw status | grep -E '^(491|2400)'

  4. Start and Enable GlusterFS on each node:

    systemctl enable --now glusterd

  5. Verify the GlusterFS service is running:

    systemctl status glusterd

Step 2: Create Physical Disks on KVM Hosts

  1. For KVM Host 1 (Nodes pssb1avm001 and pssb1avm002):

    • Create a disk for each node: On first KVM host(syhydsrv01), execute the following command to create a 10GB raw disk image: 
      qemu-img create -f raw /data1/d_disks/pssb1avm001-brick.img 10G
qemu-img create -f raw /data1/d_disks/pssb1avm002-brick.img 10G

  2. For KVM Host 2 (Nodes pssb1avm004 and pssb1avm005):

    • Create a disk for each node: On the second KVM host(syhydsrv02), execute the following command to create a 10GB raw disk image: 
      qemu-img create -f raw /data1/d_disks/pssb1avm004-brick.img 10G
qemu-img create -f raw /data1/d_disks/pssb1avm005-brick.img 10G

Step 3: Attach the Disk to Each VM

You can attach the disk via Cockpit UI (optional) or directly by editing the VM’s XML file.

Using XML Procedure:

For each node VM, follow these steps:

  1. Edit the VM’s XML configuration using the command:

    virsh edit <vm_name>

  2. Add a disk entry for the newly created disk under the <devices> section, replacing the path with the appropriate disk image:

    <disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/data1/d_disks/<brick_name>.img'/>
  <target dev='vdb' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>

  3. Save the file and exit.

  4. Restart the VM to apply the new disk configuration:

    virsh reboot <vm_name>

Step 4: Format the Attached Disk on Each Node

  1. Check the attached disk:

    lsblk

  2. Format the disk with the XFS file system:

    mkfs.xfs -i size=512 /dev/vdb

  3. Mount the disk to the /export/vdb directory:

    mkdir -p /export/vdb && mount /dev/vdb /export/vdb

Step 5: Persistent Mount Configuration

  1. Add the mount configuration to /etc/fstab:

    echo "/dev/vdb /export/vdb xfs defaults 0 0" >> /etc/fstab

  2. Mount all devices:

    mount -a

Step 6: Create the brick Directory on the Attached Disk on each node

  1. Create the brick directory to store the volume data: 
    mkdir -p /export/vdb/brick

Step 7: Create the New GlusterFS Volume

While creating select any arbitrary node to run the below commands

  1. Create the volume: Use the following command to create the GlusterFS volume, including all the nodes and their respective bricks: 
    gluster volume create <volume_name> replica 5 \
pssb1avm001:/export/vdb/brick \
pssb1avm002:/export/vdb/brick \
pssb1abm003:/data/export/vdb/brick \
pssb1avm004:/export/vdb/brick \
pssb1avm005:/export/vdb/brick

Step 9: Start the Volume

  1. Start the GlusterFS volume: 
    gluster volume start <volume_name>

Step 10: Verify the Volume Information

  1. Check the volume information: 
    gluster volume info

Create client mounts and mount them on each node:

Create the client mount directory on each node is necessary but creating the directories inside the volume is not necessary and ownership as the volume is already started and syncs the required directories on every brick

  1. Create client mount directory along with required directories for application code on the volume:

    mkdir -p /data/pssb
mkdir -p /data/pssb/archive
mkdir -p /data/pssb/reports
mkdir -p /data/pssb/health_monitor
chmod a+rx /data

  2. Give ownership to the tomcat directory

    chown -R tomcat:tomcat /data/pssb

Mount the volume to the client mount point

  1. Mount the volume into this directory for each node: On node01:

    mount -t glusterfs pssb1avm001:pssb_dfs /data/pssb

    On node02:

    mount -t glusterfs pssb1avm002:pssb_dfs /data/pssb

    On node03:

    mount -t glusterfs pssb1bvm003:pssb_dfs /data/pssb

    On node04:

    mount -t glusterfs pssb1avm004:pssb_dfs /data/pssb

    On node05:

    mount -t glusterfs pssb1avm005:pssb_dfs /data/pssb

  2. Make the mounts persistent on each node using:

Persistent Mounts on Each Node

To ensure the GlusterFS mounts persist across reboots, add the following entries to the /etc/fstab file on each node:

  • On node01:

    pssb1avm001:pssb_dfs /data/pssb glusterfs defaults,_netdev 1 0

  • On node02:

    pssb1avm002:pssb_dfs /data/pssb glusterfs defaults,_netdev 1 0

  • On node03:

    pssb1bvm003:pssb_dfs /data/pssb glusterfs defaults,_netdev 1 0

  • On node04:

    pssb1avm004:pssb_dfs /data/pssb glusterfs defaults,_netdev 1 0

  • On node05:

    pssb1avm005:pssb_dfs /data/pssb glusterfs defaults,_netdev 1 0

  1. Open the /etc/fstab file on each node using a text editor :

    sudo vi /etc/fstab

  2. Add the respective entry for the node.

  3. Save the file and exit.

  4. Test the persistent mount by running:

    sudo mount -a

    This command will mount all entries from the /etc/fstab file without requiring a reboot.

  5. Verify the mount using:

    mount | grep /data/pssb

It is crucial to perform reboots for all nodes after the setup is completed to ensure mounts are persistent.