Home > DevOps > Infrastructure As a Code > Cluster and Node Setup > PSSB > Redpanda Cluster Setup

Redpanda Cluster Setup

Here’s a clear and structured documentation for setting up a Redpanda Cluster based on the details provided.

Cluster Configuration

Node Details

Node 1: pssb1avm001 - 172.21.0.61
Node 2: pssb1avm002 - 172.21.0.62
Node 3: pssb1abm003 - 172.21.0.63
Node 4: pssb1avm004 - 172.21.0.64
Node 5: pssb1avm005 - 172.21.0.65

Setup Steps

1. Install Redpanda

Ensure Redpanda is installed on all nodes.
Install rpk (Redpanda CLI) for managing and monitoring the cluster.

2. Configure the Initial Node (Bootstrap Node)

Update the configuration file (/etc/redpanda/redpanda.yaml) on pssb1avm001 with the following:
```
empty_seed_starts_cluster: true
seed_servers: []
```

Allow all ports to communicate between the redpanda nodes

sudo ufw allow 9092/tcp comment "Allow Kafka API for Redpanda"
sudo ufw allow 8082/tcp comment "Allow HTTP Proxy for Redpanda"
sudo ufw allow 8081/tcp comment "Allow Schema Registry for Redpanda"
sudo ufw allow 9644/tcp comment "Allow Admin API and Prometheus metrics for Redpanda"
sudo ufw allow 33145/tcp comment "Allow internal RPC for Redpanda"
sudo ufw reload

Start Redpanda on the bootstrap node:
```
systemctl start redpanda
```
Verify the cluster status:
```
rpk cluster info
```

3. Configure Joining Nodes

For each additional node (e.g., pssb1avm002, pssb1abm003, etc.), update the configuration file (/etc/redpanda/redpanda.yaml) with the cluster details:

empty_seed_starts_cluster: false
seed_servers:
  - host:
      address: pssb1avm001
      port: 33145
  - host:
      address: pssb1avm002
      port: 33145
  - host:
      address: pssb1abm003
      port: 33145
  - host:
      address: pssb1avm004
      port: 33145
  - host:
      address: pssb1avm005
      port: 33145

Start Redpanda on each joining node:
```
systemctl start redpanda
```
Check the cluster status on each node to ensure they have joined:
```
rpk cluster info
```

4. Verify Cluster Status

After all nodes are configured and started, verify the cluster:

Use rpk cluster health to monitor the cluster:
```
rpk cluster health
```
Identify the leader node (marked with * in the output).

Example output:

CLUSTER
=======
redpanda.da0b4ef6-d859-41cf-8cb9-be990e1c256c

BROKERS
=======
ID    HOST         PORT
0     pssb1avm001  9092
1*    pssb1avm002  9092
3     pssb1avm004  9092
4     pssb1avm005  9092
5     pssb1abm003  9092

5. Update Configuration on the Bootstrap Node

Once all nodes are joined, update the redpanda.yaml file on the bootstrap node to include the seed servers:

empty_seed_starts_cluster: false
seed_servers:
  - host:
      address: pssb1avm001
      port: 33145
  - host:
      address: pssb1avm002
      port: 33145
  - host:
      address: pssb1abm003
      port: 33145
  - host:
      address: pssb1avm004
      port: 33145
  - host:
      address: pssb1avm005
      port: 33145

Restart the Redpanda service:

systemctl restart redpanda
systemctl enable redpanda

Maintenance and Troubleshooting

Cluster Health Monitoring

Use the following commands to monitor the cluster:
- Cluster Info: rpk cluster info
- Cluster Health: rpk cluster health

Log and Data Cleanup

To delete logs and manage disk space:

sudo journalctl -xeu redpanda --vacuum-files=1
sudo rm -rf /data/ms/rpc/redpanda/data/

Leader Election

Redpanda uses the Raft Consensus Algorithm for leader election. If a leader fails, the remaining brokers will elect a new leader.

Adding a New Node to the Redpanda Cluster

To add a new node to an existing Redpanda cluster, follow the steps below. The process involves updating the configuration on all existing nodes, as well as configuring the new node to join the cluster.

1. Update Configuration on Existing Nodes

On every node in the cluster, update the Redpanda configuration file (redpanda.yaml) as follows:

Set empty_seed_starts_cluster: false.
Add the new node’s address to the seed_servers list. The updated configuration should look like this:

empty_seed_starts_cluster: false
seed_servers:
  - host:
      address: pssb1avm001
      port: 33145
  - host:
      address: pssb1avm002
      port: 33145
  - host:
      address: pssb1abm003
      port: 33145
  - host:
      address: pssb1avm004
      port: 33145
  - host:
      address: pssb1avm005
      port: 33145
  - host:
      address: pssb1avm006
      port: 33145

After updating the configuration on all existing nodes, restart each node one by one to ensure they apply the changes.

2. Update Configuration on the New Node

On the new node, also update the redpanda.yaml configuration file:

Set empty_seed_starts_cluster: false.
Add the same seed_servers list with the addresses of all nodes in the cluster.
Ensure the following configuration is present:

empty_seed_starts_cluster: false
seed_servers:
  - host:
      address: pssb1avm001
      port: 33145
  - host:
      address: pssb1avm002
      port: 33145
  - host:
      address: pssb1abm003
      port: 33145
  - host:
      address: pssb1avm004
      port: 33145
  - host:
      address: pssb1avm005
      port: 33145
  - host:
      address: pssb1avm006
      port: 33145

3. Allow Ports on the New Node

Follow the procedure to allow the necessary ports on the new node as per your Redpanda setup (e.g., 9092, 8081, 8082, 33145, etc.).

Ensure these ports are open on the new node using the ufw (Uncomplicated Firewall) commands:

sudo ufw allow 9092/tcp
sudo ufw allow 8081/tcp
sudo ufw allow 8082/tcp
sudo ufw allow 33145/tcp

4. Start the New Node

After updating the configuration and allowing the necessary ports, start the Redpanda service on the new node:
```
sudo systemctl enable --now redpanda
```

5. Verify Cluster Status

Once the new node is started, verify that it has successfully joined the cluster by running the following command:
```
rpk cluster info
```
The output should show the new node as part of the cluster.

This process ensures that the new node is properly added to the existing Redpanda cluster and can communicate with the other nodes.