PSAP-LXC Redpanda Cluster

Redpanda Cluster Setup

Cluster Configuration

Node Details
  • Node 1: psap1alxc001 - 172.21.0.75
  • Node 2: psap1alxc002 - 172.21.0.76
  • Node 3: psap1alxc003 - 172.21.0.77

Setup Steps

1. Install Redpanda

  1. Ensure Redpanda is installed on all nodes.
  2. Install rpk (Redpanda CLI) for managing and monitoring the cluster.

2. Configure the Initial Node (Bootstrap Node)

  1. Update the configuration file (/etc/redpanda/redpanda.yaml) on psap1alxc001 with the following:

    empty_seed_starts_cluster: true
seed_servers: []

  2. Allow all ports to communicate between the redpanda nodes

    sudo ufw allow 9092/tcp 
sudo ufw allow 8082/tcp 
sudo ufw allow 8081/tcp 
sudo ufw allow 9644/tcp 
sudo ufw allow 33145/tcp
sudo ufw reload

  3. Start Redpanda on the bootstrap node:

    systemctl start redpanda

  4. Verify the cluster status:

    rpk cluster info

3. Configure Joining Nodes

For each additional node (e.g., psap1alxc002, psap1alxc003, etc.), update the configuration file (/etc/redpanda/redpanda.yaml) with the cluster details:

empty_seed_starts_cluster: false
seed_servers:
    - host:
        address: psap1alxc001
        port: 33145
    - host:
        address: psap1alxc002
        port: 33145
    - host:
        address: psap1alxc003
        port: 33145

  1. Start Redpanda on each joining node:

    systemctl start redpanda

  2. Check the cluster status on each node to ensure they have joined:

    rpk cluster info

4. Verify Cluster Status

After all nodes are configured and started, verify the cluster:

  1. Use rpk cluster info to monitor the cluster: 
    rpk cluster info
  2. Identify the leader node (marked with * in the output).

Example output:

root@psap1alxc001:~# rpk cluster info
CLUSTER
=======
redpanda.08c1a93b-ae52-4776-ac8f-c21e1c2f04dd

BROKERS
=======
ID    HOST          PORT
0*    psap1alxc001  9092
1     psap1alxc002  9092
2     psap1alxc003  9092

TOPICS
======
NAME                         PARTITIONS  REPLICAS
__consumer_offsets           3           3
ps_ap_analytics              9           1
ps_ap_asio_consumer          1           1
ps_ap_bulkdata               9           1
ps_ap_cachedependencyremove  9           1
ps_ap_govtscheme             9           1
ps_ap_invoice                9           1
ps_ap_notify_email           9           1
ps_ap_notify_sms             9           1
ps_ap_notify_whatsapp        9           1
ps_ap_paymentstatus          9           3
ps_ap_reportgenerate         9           1

5. Update Configuration on the Bootstrap Node

Once all nodes are joined, update the redpanda.yaml file on the bootstrap node to include the seed servers:

Restart the Redpanda service:

systemctl restart redpanda

Maintenance and Troubleshooting

Cluster Health Monitoring

  • Use the following commands to monitor the cluster:
    • Cluster Info: rpk cluster info
    • Cluster Health: rpk cluster health

Log Cleanup

To delete logs and manage disk space:

sudo journalctl -xeu redpanda --vacuum-files=1

Leader Election

Redpanda uses the Raft Consensus Algorithm for leader election. If a leader fails, the remaining brokers will elect a new leader.