MC Redpanda Cluster

Redpanda Cluster Setup

Here’s a clear and structured documentation for setting up a Redpanda Cluster based on the details provided.

Cluster Configuration

Node Details
  • Node 1: msgcloud-prod-node01 - 172.21.0.55
  • Node 2: msgcloud-prod-node02 - 172.21.0.56
  • Node 3: msgcloud-prod-node03 - 172.21.0.57

Setup Steps

1. Install Redpanda

  1. Ensure Redpanda is installed on all nodes.
  2. Install rpk (Redpanda CLI) for managing and monitoring the cluster.

2. Configure the Initial Node (Bootstrap Node)

  1. Update the configuration file (/etc/redpanda/redpanda.yaml) on msgcloud-prod-node01 with the following:

    empty_seed_starts_cluster: true
seed_servers: []

  2. Allow all ports to communicate between the redpanda nodes

    sudo ufw allow 9092/tcp 
sudo ufw allow 8082/tcp 
sudo ufw allow 8081/tcp 
sudo ufw allow 9644/tcp 
sudo ufw allow 33145/tcp
sudo ufw reload

  3. Start Redpanda on the bootstrap node:

    systemctl start redpanda

  4. Verify the cluster status:

    rpk cluster info

3. Configure Joining Nodes

For each additional node (e.g., msgcloud-prod-node02, msgcloud-prod-node03, etc.), update the configuration file (/etc/redpanda/redpanda.yaml) with the cluster details:

empty_seed_starts_cluster: false
seed_servers:
    - host:
        address: msgcloud-prod-node01
        port: 33145
    - host:
        address: msgcloud-prod-node02
        port: 33145
    - host:
        address: msgcloud-prod-node03
        port: 33145

  1. Start Redpanda on each joining node:

    systemctl start redpanda

  2. Check the cluster status on each node to ensure they have joined:

    rpk cluster info

4. Verify Cluster Status

After all nodes are configured and started, verify the cluster:

  1. Use rpk cluster health to monitor the cluster: 
    rpk cluster health
  2. Identify the leader node (marked with * in the output).

Example output:

CLUSTER
=======
redpanda.6faf56a9-50f3-4f39-8ac3-cf0943d9c5b4

BROKERS
=======
ID    HOST                  PORT
0*    msgcloud-prod-node01  9092
1     msgcloud-prod-node02  9092
2     msgcloud-prod-node03  9092

TOPICS
======
NAME                           PARTITIONS  REPLICAS
__consumer_offsets             3           3
mc_prod_analytics              1           3
mc_prod_cachedependencyremove  1           3
mc_prod_msgbot                 100         3
mc_prod_notify_email           1           3
mc_prod_notify_sms             1           3
mc_prod_notify_whatsapp        1           3
mc_prod_reportgenerate         1           3

5. Update Configuration on the Bootstrap Node

Once all nodes are joined, update the redpanda.yaml file on the bootstrap node to include the seed servers:

Restart the Redpanda service:

systemctl restart redpanda

Maintenance and Troubleshooting

Cluster Health Monitoring

  • Use the following commands to monitor the cluster:
    • Cluster Info: rpk cluster info
    • Cluster Health: rpk cluster health

Log Cleanup

To delete logs and manage disk space:

sudo journalctl -xeu redpanda --vacuum-files=1

Leader Election

Redpanda uses the Raft Consensus Algorithm for leader election. If a leader fails, the remaining brokers will elect a new leader.