Ha-proxy Exporter

HAProxy Exporter

In a HAProxy Exporter dashboard in Grafana, you typically have multiple panels that provide insights into the performance and status of your HAProxy load balancer. Here’s an explanation of some common panels you might find in a HAProxy Exporter dashboard.

Metrics

Frontend Metrics :

  • haproxy_frontend_http_responses_total: Frontend metrics categorize responses by type (1xx to 5xx) and “other” from requests initiated by clients.

  • haproxy_frontend_bytes_in_total: Total bytes received by HAProxy at the frontend, including all incoming traffic from clients.

  • haproxy_frontend_bytes_out_total: Total bytes sent out by HAProxy at the frontend, encompassing all outgoing traffic to clients.

  • haproxy_frontend_connections_total: This metric tracks the total number of connections handled by HAProxy at the frontend, indicating the overall demand and traffic received from clients.

  • haproxy_frontend_http_requests_total: Tracks the total number of HTTP requests processed by HAProxy at the frontend, helping gauge overall client activity and load.

  • haproxy_frontend_request_errors_total: Counts errors encountered during request processing at the frontend, including timeouts, connection issues, and other errors that impact client interactions.

  • haproxy_frontend_requests_denied_total: Measures the number of requests denied by HAProxy at the frontend, often due to configured ACLs (Access Control Lists) or rate limits.

  • haproxy_frontend_status: This metric typically indicates the operational status or health of the frontend services configured in HAProxy. It may provide details such as whether the frontend is up and running, experiencing issues, or if it’s in a maintenance mode.

  • haproxy_frontend_denied_connections_total: This metric counts the total number of connections to the frontend that were denied by HAProxy. These denials are typically due to access control policies, such as ACLs (Access Control Lists) or connection limits.

  • haproxy_frontend_connections_rate_max: This metric represents the peak rate of new connections per second handled by HAProxy at the frontend. It indicates the highest level of incoming connection activity over a specific period, helping administrators understand the maximum load capacity at the frontend.

  • haproxy_frontend_responses_denied_total: This metric counts the total number of responses from HAProxy’s frontend that were denied to clients. These denials usually happen due to configured rules or limits on response delivery, providing insights into resource management and potential issues affecting client interactions.

  • haproxy_frontend_http_requests_rate_max: This metric represents the highest rate of incoming HTTP requests per second handled by HAProxy at the frontend. It indicates the peak level of request traffic over a specific period, helping administrators understand the maximum load capacity at the frontend.

  • haproxy_server_http_responses_total: This metric typically counts the total number of HTTP responses generated by each individual backend server managed by HAProxy. It helps in monitoring server-specific performance metrics, such as response rates and load balancing effectiveness.

Backend Metrics :

  • haproxy_backend_http_responses_total: Backend metrics categorize responses by type (1xx to 5xx) and “other”, but from servers, reflecting the backend’s handling of requests.

  • haproxy_backend_bytes_in_total: Total bytes received by HAProxy at the backend, comprising all incoming traffic from servers.

  • haproxy_backend_bytes_out_total: Total bytes sent out by HAProxy at the backend, covering all outgoing traffic to servers.

  • haproxy_backend_connection_errors_total: This metric counts the total number of errors related to connections encountered by HAProxy at the backend, providing insights into issues like timeouts, connection failures, or other backend-related problems affecting communication with servers.

  • haproxy_backend_redispatch_warnings_total: Indicates how often requests were redirected to another backend server due to load balancing decisions or server health checks.

  • haproxy_backend_retry_warnings_total: Tracks instances where HAProxy retried requests at the backend due to initial connection failures or other transient issues.

  • haproxy_backend_response_errors_total: Counts errors encountered while processing responses at the backend, such as HTTP 5xx server errors or timeouts.

  • haproxy_backend_current_queue: Shows the current number of requests queued and waiting for backend server resources, highlighting potential congestion or performance issues.

  • haproxy_backend_http_requests_total: Measures the total number of HTTP requests handled by HAProxy at the backend, offering insights into server workload and demand patterns.

  • haproxy_backend_status: This metric reflects the status or health of the backend servers configured in HAProxy. It can indicate whether backend servers are operational, reachable, or experiencing errors.

  • haproxy_backend_connection_attempts_total: This metric counts how many times HAProxy tries to connect to backend servers. It shows the demand on backend resources and helps identify connection problems or server availability issues.

  • haproxy_backend_connection_reuses_total: This metric counts how many times HAProxy reuses connections to backend servers instead of creating new ones. It helps optimize resource usage and can indicate efficiency in handling backend communications.

  • haproxy_backend_max_queue: This metric indicates the maximum number of requests that were queued and waiting for backend servers to become available in HAProxy.

  • haproxy_backend_responses_denied_total: This metric counts the total number of responses from backend servers that were denied by HAProxy. These denials typically occur due to configured rules or limits on response handling, providing insights into resource management and potential issues with backend server responses

  • haproxy_backend_check_up_down_total: Tracks the total number of times HAProxy has detected a backend server as either “up” or “down” during health checks. It helps monitor the availability and status changes of backend servers.

  • haproxy_backend_weight: Represents the weight assigned to a backend server in HAProxy’s load balancing configuration. It determines the proportion of client requests that are directed to each server, influencing load distribution.

  • haproxy_backend_uweight: Represents the user-weight assigned to a backend server in HAProxy. It is similar to haproxy_backend_weight but can be adjusted manually by administrators to influence load balancing decisions without changing the actual weight setting.

Time Metrics

  • haproxy_process_start_time_seconds: This metric indicates the start time of the HAProxy process in seconds . It helps determine how long the HAProxy instance has been running.

  • haproxy_backend_max_connect_time_seconds: This metric measures the maximum time taken by HAProxy to establish connections to backend servers. It helps identify performance bottlenecks or issues affecting the timely establishment of connections, crucial for optimizing backend server interactions.

  • haproxy_backend_total_time_average_seconds: This metric calculates the average time taken by HAProxy to handle requests from start to finish with backend servers

  • haproxy_backend_max_total_time_seconds: This metric indicates the maximum time taken by HAProxy to handle a request from start to finish with backend servers. It helps identify peak performance issues or delays in backend server responses, critical for optimizing service delivery and user experience.

  • haproxy_backend_response_time_average_seconds: This metric measures the average time taken by HAProxy to receive a response from backend servers after forwarding a request. It provides insights into the typical latency of backend server responses, aiding in performance monitoring and optimization efforts.

  • haproxy_backend_max_response_time_seconds: This metric indicates the maximum time taken by HAProxy to receive a complete response from backend servers. It helps identify peak response delays or performance issues affecting backend server communications.

  • haproxy_backend_queue_time_average_seconds: This metric calculates the average time requests spend queued and waiting for backend server resources in HAProxy. It provides insights into request queuing efficiency and helps optimize backend server utilization.

  • haproxy_backend_max_queue_time_seconds: This metric represents the maximum time requests spent queued and waiting for backend server resources in HAProxy. It helps identify peak queuing delays or potential bottlenecks affecting backend server responsiveness.

Session Metrics

  • haproxy_frontend_sessions_total: Total number of sessions (connections) handled by HAProxy at the frontend.

  • haproxy_frontend_denied_sessions_total: Total number of sessions denied by HAProxy at the frontend, often due to configured limits or ACLs.

  • haproxy_backend_sessions_total: Total number of sessions (connections) handled by HAProxy at the backend.

  • haproxy_backend_max_sessions: Maximum number of concurrent sessions allowed for backend servers in HAProxy.

  • haproxy_backend_limit_sessions: Current limit on the number of concurrent sessions allowed for backend servers in HAProxy.

  • haproxy_frontend_max_sessions: Maximum number of concurrent sessions allowed for the frontend in HAProxy.

  • haproxy_frontend_limit_sessions: Current limit on the number of concurrent sessions allowed for the frontend in HAProxy.

  • haproxy_backend_max_session_rate: Maximum rate of new sessions per second allowed for backend servers in HAProxy.

  • haproxy_frontend_max_session_rate: Maximum rate of new sessions per second allowed for the frontend in HAProxy.

  • haproxy_frontend_limit_session_rate: Current limit on the rate of new sessions per second allowed for the frontend in HAProxy.


Grafana Dashboard Panels

Basic General Info Row

All HTTP responses Panel All HTTP responses, provides different categories of HTTP responses for both frontend and backend connections in HAProxy. It tracks informational, successful, redirection, client error, server error responses, and other response types for both frontend and backend connections.

  • Metrics Used: haproxy_frontend_http_responses_total, haproxy_backend_http_responses_total.

Incoming/Outgoing bytes Panel The “Incoming / Outgoing bytes” panel in Grafana monitors data traffic for HAProxy. It tracks incoming bytes to the frontend (IN Front) and outgoing bytes from the frontend to clients (OUT Front), along with incoming bytes to the backend (IN Back) and outgoing bytes from the backend to clients (OUT Back). These metrics provide essential insights into data flow and network performance within HAProxy.

  • Metrics Used: haproxy_frontend_bytes_in_total, haproxy_frontend_bytes_out_total, haproxy_backend_bytes_in_total, haproxy_backend_bytes_out_total.

Number of connections Panel It displays the current count of client connections to HAProxy (Front) and tracks errors encountered when connecting to backend servers (Back errors).

  • Metrics Used: haproxy_frontend_connections_total, haproxy_backend_connection_errors_total

Requests and Responses Panel It counts how many requests come from clients to the frontend, notes any errors or denied requests due to setup limits. On the backend, it watches for problems like servers being re-sent requests due to connection problems, retries when servers aren’t available, logs any errors from backend servers when responding, and shows how many requests are waiting to be processed.

  • Metrics Used: haproxy_frontend_http_requests_total,haproxy_frontend_request_errors_total, haproxy_frontend_requests_denied_total, haproxy_backend_redispatch_warnings_total, haproxy_backend_retry_warnings_total, haproxy_backend_response_errors_total, haproxy_backend_current_queue, haproxy_backend_http_requests_total

Active Sessions Panel It tracks active sessions between clients and the HAProxy frontend, indicating current connection usage. Similarly, it monitors active sessions between HAProxy and backend servers, highlighting connections actively managing requests and responses.

  • Metrics Used: haproxy_frontend_current_sessions, haproxy_backend_current_sessions

Basics General Status Row

All Status Panel It indicates the current number of frontend servers (Front Up) that are operational and ready to handle client requests. Similarly, the “Back Up” metric shows how many backend servers are functioning and available for HAProxy to forward requests to, ensuring a reliable flow of data between clients and servers. These metrics are essential for monitoring and maintaining the stability and performance of HAProxy.

  • Metrics Used: haproxy_frontend_status, haproxy_backend_status

Started Panel It displays the duration since servers or HAProxy instances were last started, indicating their current operational continuity. This metric is essential for assessing server reliability, uptime performance, and identifying any interruptions in service within the specified timeframe.

  • Metrics Used: haproxy_process_start_time_seconds

Throughtput/Connections Row

Front - Incoming / Outgoing bytes Panel The “Front - Incoming / Outgoing bytes” panel in HAProxy’s Grafana dashboard tracks data flow for the HTTPS frontend on port 443. It shows incoming data bytes (IN https_443_frontend) from clients to the frontend, and outgoing data bytes (OUT https_443_frontend) from the frontend to clients.These metrics are essential for monitoring and analyzing data traffic handled by the frontend.

  • Metrics Used: haproxy_frontend_bytes_in_total, haproxy_frontend_bytes_out_total

Back - Incoming / Outgoing bytes Panel The “Back - Incoming / Outgoing bytes” panel in HAProxy’s Grafana dashboard monitors data flow for the backend on port 8080. It tracks incoming data bytes (IN http_8080_backend) from the frontend to the backend servers and outgoing data bytes (OUT http_8080_backend) from the backend servers to the frontend. These metrics are essential for understanding backend data traffic, analyzing server load.

  • Metrics Used: haproxy_backend_bytes_in_total, haproxy_backend_bytes_out_total

Front - Connections successful / denied Panel The “Front - Connections successful / denied” panel in HAProxy’s Grafana dashboard tracks the connection status for the HTTPS frontend on port 443. It shows the number of successful connections (Successful https_443_frontend) established between clients and the frontend, and the number of denied connections (Denied https_443_frontend) due to security or configuration rules.

  • Metrics Used: haproxy_frontend_connections_total, haproxy_frontend_denied_connections_total

Back - Connections attempts / errors Panel The “Back - Connections attempts / errors” panel in HAProxy’s Grafana dashboard monitors connection activities for the backend on port 8080. It tracks the number of connection attempts (Attempts http_8080_backend) made by HAProxy to backend servers and logs any connection errors (Error http_8080_backend) that occur. These metrics are crucial for understanding the reliability and efficiency of connections to backend servers.

  • Metrics Used: haproxy_backend_connection_attempts_total, haproxy_backend_connection_errors_total

Front - Maximum observed number of connections per second Panel The “Front - Maximum observed number of connections per second” panel in HAProxy’s Grafana dashboard tracks the peak rate of incoming connections to the HTTPS frontend on port 443. It shows the highest number of connections per second (Max https_443_frontend) observed over a specified period. This metric is crucial for monitoring the workload on the frontend, identifying peak traffic times, and ensuring that HAProxy can handle incoming connections efficiently without exceeding capacity limits.

  • Metrics Used: haproxy_frontend_connections_rate_max

Back - Connections reuses Panel The “Back - Connections reuses” panel in HAProxy’s Grafana dashboard monitors how often connections to backend servers on port 8080 are reused instead of being closed and reopened for new requests. It shows the average (Mean), most recent (Last *), maximum (Max), and minimum (Min) number of times connections have been reused over a given period.

  • Metrics Used: haproxy_backend_connection_reuses_total

Queues Row

Back - Queued requests not assigned to any server Panel The “Back - Queued requests not assigned to any server” panel in HAProxy’s Grafana dashboard tracks the number of requests waiting in a queue to be processed by backend servers on port 8080 but have not yet been assigned to any available server.

  • Metrics Used: haproxy_backend_current_queue

Back - Max queued requests not assigned to any server Panel The “Back - Max queued requests not assigned to any server” panel in HAProxy’s Grafana dashboard tracks the peak number of requests waiting in a queue to be processed by backend servers on port 8080, but which have not yet been assigned to any available server.

  • Metrics Used: haproxy_backend_max_queue

Requests / Responses Row

Back - HTTP requests OK / Denied Panel It tracks the rate of successful HTTP requests (Total http_8080_backend) processed by the backend, indicating how effectively requests are handled. The metric for denied requests (Denied http_8080_backend) shows instances where requests were rejected or not processed due to configuration restrictions or limitations.

  • Metrics Used: haproxy_backend_http_requests_total, haproxy_backend_requests_denied_total

Front - HTTP requests OK / Error / Denied Panel The “Front - HTTP requests OK / Error / Denied” panel in HAProxy’s Grafana dashboard provides detailed metrics on how HTTP requests are managed by the frontend on port 443. It includes three key metrics: Total https_443_frontend, which shows the rate of successful HTTP requests processed by the frontend; Error https_443_frontend, tracking any errors encountered during request processing; and Denied https_443_frontend, indicating HTTP requests rejected due to security rules or capacity limits.

  • Metrics Used: haproxy_frontend_http_requests_total, haproxy_frontend_request_errors_total, haproxy_frontend_requests_denied_total

Back - HTTP responses denied Panel The “Back - HTTP responses denied” panel in HAProxy’s Grafana dashboard tracks instances where backend servers on port 8080 have denied or rejected HTTP responses. This metric, labeled as Denied http_8080_backend, shows the number of responses that were not sent back to clients due to various reasons such as server-side errors, security restrictions, or configuration limitations.

  • Metrics Used: haproxy_backend_responses_denied_total

Front - HTTP responses denied Panel The “Front - HTTP responses denied” panel in HAProxy’s Grafana dashboard monitors instances where the frontend on port 443 has denied or rejected HTTP responses. This metric, labeled as Denied https_443_frontend, indicates the number of responses that were not sent back to clients due to reasons such as security rules, request limitations, or frontend capacity constraints.

  • Metrics Used: haproxy_frontend_responses_denied_total

Back - Error / Redispatch / Retry Panel The “Back - Error / Redispatch / Retry” panel in HAProxy’s Grafana dashboard provides insights into the handling of backend operations on port 8080. It includes three key metrics: Redispatch http_8080_backend, which indicates instances where HAProxy re-dispatches requests to different backend servers due to connection issues or load balancing decisions; Retry http_8080_backend, tracking the number of times HAProxy retries requests to backend servers after initial failures; and Error http_8080_backend, which counts occurrences of errors encountered during the processing of requests by backend servers.

  • Metrics Used: haproxy_backend_redispatch_warnings_total, haproxy_backend_retry_warnings_total, haproxy_backend_response_errors_total

Front - Maximum observed number of HTTP requests per second Panel The “Front - Maximum observed number of HTTP requests per second” panel in HAProxy’s Grafana dashboard tracks the peak rate of incoming HTTP requests to the frontend on port 443. This metric, labeled as Max https_443_frontend, shows the highest number of HTTP requests handled per second over a specified time period.

  • Metrics Used: haproxy_frontend_http_requests_rate_max

Times Row

Back - Max connection time Panel The “Back - Max connection time” panel in HAProxy’s Grafana dashboard monitors the maximum connection time for backend operations on port 8080. This shows the highest duration it took for HAProxy to establish connections with backend servers over a specified period.

  • Metrics Used: haproxy_backend_max_connect_time_seconds

Back - Avg. total time Panel The “Back - Avg. total time” panel in HAProxy’s Grafana dashboard displays the average total time taken for backend operations on port 8080. This reflects the average duration from when HAProxy receives a request until it completes receiving the response from the backend servers.

  • Metrics Used: haproxy_backend_total_time_average_seconds

Back - Max total time Panel The “Back - Max total time” panel in HAProxy’s Grafana dashboard tracks the maximum total time taken for backend operations on port 8080. This represents the longest duration from when HAProxy receives a request until it completes receiving the response from the backend servers over a specified period.

  • Metrics Used: haproxy_backend_max_total_time_seconds

Back - Avg. response time Panel The “Back - Avg. response time” panel in HAProxy’s Grafana dashboard shows the average response time for backend operations on port 8080. This indicates the typical duration from when HAProxy sends a request to a backend server until it receives the complete response.

  • Metrics Used: haproxy_backend_response_time_average_seconds

Back - Max response time Panel This panel shows the maximum response time for backend operations on port 8080. The metric, labeled as http_8080_backend, represents the longest duration from when HAProxy sends a request to a backend server until it receives the complete response.

  • Metrics Used: haproxy_backend_max_response_time_seconds

Back - Avg. queue time for last 1024 successful connections Panel This panel displays the average queue time for the last 1024 successful connections to backend servers on port 8080. The metric, labeled as http_8080_backend, measures the average time requests spend in the queue before being processed by available backend servers.

  • Metrics Used: haproxy_backend_queue_time_average_seconds

Back - Max queue time Panel This panel indicates the maximum queue time observed for requests waiting to be processed by backend servers on port 8080. The metric, labeled as http_8080_backend, shows the longest duration that requests have spent in the queue before being handled.

  • Metrics Used: haproxy_backend_max_queue_time_seconds

Responses by HTTP code Row

Front - HTTP responses code Panel This panel shows the distribution of HTTP response codes received by the frontend on port 443. It categorizes responses into various ranges: 1xx, 2xx, 3xx, 4xx, 5xx, and others.

  • Metrics Used: haproxy_frontend_http_responses_total

Back - HTTP responses code Panel Similar to the frontend panel, this panel monitors HTTP response codes received from backend servers on port 8080. It categorizes responses into 1xx, 2xx, 3xx, 4xx, 5xx, and other categories, providing administrators with visibility into how backend servers handle requests and their response patterns.

  • Metrics Used: haproxy_backend_http_responses_total

Server - HTTP responses code Panel This panel focuses on specific backend servers, such as app01, and their distribution of HTTP response codes. It categorizes responses into 1xx, 2xx, 3xx, 4xx, 5xx, and others, allowing administrators to monitor the performance and response behavior of individual backend servers within the HAProxy infrastructure.

  • Metrics Used: haproxy_server_http_responses_total

Sessions Row

Front - Number of sessions Panel The “Front - Number of sessions” panel in HAProxy’s Grafana dashboard provides insights into session management for the frontend on port 443. It includes two main metrics: Total https_443_frontend, which indicates the average number of active sessions handled by the frontend, representing concurrent connections from clients; and Denied https_443_frontend, which tracks instances where session requests were denied or not processed due to various reasons such as capacity limits or security rules.

  • Metrics Used: haproxy_frontend_sessions_total, haproxy_frontend_denied_sessions_total

Back - Number of sessions Panel The “Back - Number of sessions” panel in HAProxy’s Grafana dashboard provides insights into session management for backend operations on port 8080. It tracks the average number of active sessions, represented by Total http_8080_backend, indicating concurrent connections between HAProxy and backend servers.

  • Metrics Used: haproxy_backend_sessions_total

Back - Maximum observed number of active sessions and limit Panel The “Back - Maximum observed number of active sessions and limit” panel in HAProxy’s Grafana dashboard provides crucial insights into the workload and capacity of backend operations on port 8080. It shows the highest number of concurrent active sessions observed between HAProxy and backend servers over a specified period, reflecting peak demand and utilization,also the maximum capacity or threshold set for active sessions, ensuring backend servers do not exceed their designated limits.

  • Metrics Used: haproxy_backend_max_sessions, haproxy_backend_limit_sessions

Front - Maximum observed number of active sessions and limit Panel The “Front - Maximum observed number of active sessions and limit” panel in HAProxy’s Grafana dashboard provides essential insights into the capacity and workload of the frontend on port 443. It indicates the highest number of concurrent active sessions observed between clients and HAProxy frontend servers over a specified period, showing peak demand and utilization and the maximum capacity or threshold set for active sessions on the frontend, ensuring that it does not exceed its designated limits.

  • Metrics Used: haproxy_frontend_max_sessions, haproxy_frontend_limit_sessions

Back - Maximum observed number of sessions per second Panel The “Back - Maximum observed number of sessions per second” panel in HAProxy’s Grafana dashboard tracks the peak rate of session creation between HAProxy and backend servers on port 8080. It indicates the highest number of new sessions established per second over a specific monitoring period.

  • Metrics Used: haproxy_backend_max_session_rate

Front - Maximum observed number of sessions per second and limit Panel The “Front - Maximum observed number of sessions per second and limit” panel in HAProxy’s Grafana dashboard provides insights into the rate of new session creations and the capacity limits of the frontend on port 443. It shows the highest number of new sessions established per second between clients and the HAProxy frontend during a specified monitoring period, reflecting peak demand and utilization and the maximum allowable rate or threshold set for session creations on the frontend.

  • Metrics Used: haproxy_frontend_max_session_rate, haproxy_frontend_limit_session_rate

Health and Weight Row

Back - UP->DOWN transitions Panel The “Back - UP->DOWN transitions” panel in HAProxy’s Grafana dashboard monitors the transitions of backend servers on port 8080 from an operational (“UP”) to a non-operational (“DOWN”) state. This counts instances where backend servers are marked as unavailable due to issues such as failures, maintenance, or configured health checks.

  • Metrics Used: haproxy_backend_check_up_down_total

Back - Service weight Panel The “Back - Service weight” panel in HAProxy’s Grafana dashboard displays the current weight assigned to backend servers on port 8080. This indicates the relative distribution of traffic load among multiple backend servers within HAProxy configurations. The weight value represents the proportion of requests that each server receives compared to others in the backend pool.

  • Metrics Used: haproxy_backend_weight

Back - Service user weight Panel The “Back - Service user weight” panel in HAProxy’s Grafana dashboard indicates the current user-assigned weight for backend servers on port 8080. This reflects the customized weight values administrators assign to individual servers within the backend pool. Unlike automatic service weights based on HAProxy’s load-balancing algorithms, user weights allow administrators to manually adjust traffic distribution according to specific business needs, server capabilities, or application requirements.

  • Metrics Used: haproxy_backend_uweight

Status Row

The status row in HAProxy’s Grafana dashboard provide essential information about the operational state and errors of both frontend and backend servers. The Front - Status panel for https_443_frontend and the Back - Status panel for http_8080_backend indicate the servers’ operational status. The Front - Internal errors panel shows error metrics for the frontend , while the Back - Internal errors panel shows error metrics for the backend. The Back - Active servers panel tracks the number of active backend servers, and the Back - Backup servers panel monitors the backup servers in use. These panels collectively help in monitoring and managing the server statuses and potential issues within HAProxy.

By server - Throughtput / Connections Row

The panels under Server - Incoming / Outgoing bytes in HAProxy’s Grafana dashboard show the bytes sent and received for the backend server on port 8080 labeled as http_8080_backend / app01. The Server - Connections panel displays the number of currently in-use and estimated connections for the same backend server. The Server - Connections attempts / error panel tracks the number of connection attempts and errors encountered. The Server - Number of connections reuses panel measures how often connections are reused. Finally, the Server - Idle connections panel shows metrics for available, limit, and safe idle connections for the backend server, indicating how connections are managed when not actively in use. These panels provide a detailed overview of server throughput and connection handling within HAProxy.

By server - Requests / Responses / Queues Row

Server - HTTP responses denied in HAProxy’s Grafana dashboard show denied HTTP responses for the backend server on port 8080 labeled as http_8080_backend / app01. The Server - Error / Redispatch / Retry panel tracks the number of redispatches, retries, and errors encountered for the same backend server. The Server - Queued requests panel displays the current number of requests that are queued and waiting for processing. Finally, the Server - Max queued requests and limit panel shows the maximum number of queued requests observed and the limit set for queued requests.

By server - Times Row

The Server - Average connection time panel gives insights into how long it typically takes to establish connections with the backend server. The Server - Max connection time panel highlights the longest observed time for a connection to be established. The Server - Average response time panel shows the typical time it takes for the server to respond to requests, while the Server - Max response time panel indicates the longest response time recorded. The Server - Average total time panel reflects the overall processing time for requests, and the Server - Max total time panel shows the longest processing time observed. The Server - Average queue time for last 1024 successful connections and Server - Max queue time panels monitor how long requests wait in the queue, providing a complete view of server performance and efficiency.

By server - Sessions Row

The Server - Number of active sessions panel tracks the current active connections with the backend server, offering visibility into how many sessions are actively in use. The Server - Failed header rewriting warnings panel indicate issues or warnings related to header rewriting failures. The Server - Number of sessions per second and limit panel provides metrics on the rate at which new sessions are established with the server, alongside any predefined session limits.

By server - Health and Weight Row

Service weight defines the load-balancing weight assigned to each backend server, ensuring fair distribution of requests. Users weight reflects custom weights influencing traffic distribution among backend servers. UP->DOWN transitions tracks server availability changes, indicating when servers go offline or come back online. Checks failures monitors any issues encountered during health checks performed on backend servers. Checks duration measures the time taken for health checks to complete, assessing server health and responsiveness. Throttle percentage indicates the proportion of requests being intentionally limited or slowed down to the backend server. Seconds since last session shows the time elapsed since the last request was routed to a specific backend server, revealing server activity and usage patterns.

By server - Status Row

The Server - Status panel for http_8080_backend / app01 indicates the current operational status of the backend server instances, showing whether they are up and available (1 denotes up, 0 denotes down). Server - Internal errors tracks any internal errors encountered by the backend server, providing insights into potential issues affecting performance. Server - Failed DNS resolutions monitors failures in DNS resolution processes, highlighting connectivity issues that may impact server accessibility.


Alert rules

HaproxyHighHttp4xxErrorRateBackend

  • Expression: ((sum by (proxy) (rate(haproxy_server_http_responses_total{code="4xx"}[1m])) / sum by (proxy) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5
  • Summary: HAProxy high HTTP 4xx error rate backend
  • Description: Alerts when backend HTTP 4xx error rate exceeds 5%.

HaproxyHighHttp5xxErrorRateBackend

  • Expression: ((sum by (proxy) (rate(haproxy_server_http_responses_total{code="5xx"}[1m])) / sum by (proxy) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5
  • Summary: HAProxy high HTTP 5xx error rate backend
  • Description: Alerts when backend HTTP 5xx error rate exceeds 5%.

HaproxyHighHttp4xxErrorRateServer

  • Expression: ((sum by (server) (rate(haproxy_server_http_responses_total{code="4xx"}[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5
  • Summary: HAProxy high HTTP 4xx error rate server
  • Description: Alerts when server HTTP 4xx error rate exceeds 5%.

HaproxyHighHttp5xxErrorRateServer

  • Expression: ((sum by (server) (rate(haproxy_server_http_responses_total{code="5xx"}[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5
  • Summary: HAProxy high HTTP 5xx error rate server
  • Description: Alerts when server HTTP 5xx error rate exceeds 5%.

HaproxyServerResponseErrors

  • Expression: (sum by (server) (rate(haproxy_server_response_errors_total[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100 > 5
  • Summary: HAProxy server response errors
  • Description: Alerts when server response errors exceed 5%.

HaproxyBackendConnectionErrors

  • Expression: (sum by (proxy) (rate(haproxy_backend_connection_errors_total[1m]))) > 100
  • Summary: HAProxy backend connection errors
  • Description: Alerts when backend connection errors exceed 100 per minute.

HaproxyServerConnectionErrors

  • Expression: (sum by (proxy) (rate(haproxy_server_connection_errors_total[1m]))) > 100
  • Summary: HAProxy server connection errors
  • Description: Alerts when server connection errors exceed 100 per minute.

HaproxyBackendMaxActiveSession>80%

  • Expression: ((haproxy_server_max_sessions >0) * 100) / (haproxy_server_limit_sessions > 0) > 80
  • Summary: HAProxy backend max active session > 80%
  • Description: Alerts when backend active sessions exceed 80% of the limit.

HaproxyPendingRequests

  • Expression: sum by (proxy) (rate(haproxy_backend_current_queue[2m])) > 0
  • Summary: HAProxy pending requests
  • Description: Indicates pending requests in the backend queue.

HaproxyHttpSlowingDown - Expression: avg by (instance, proxy) (haproxy_backend_max_total_time_seconds) > 1 - Summary: HAProxy HTTP slowing down - Description: Alerts when backend response time exceeds 1 second.

HaproxyRetryHigh - Expression: sum by (proxy) (rate(haproxy_backend_retry_warnings_total[1m])) > 10 - Summary: HAProxy retry high - Description: Indicates high retry warnings in the backend.

HaproxyHasNoAliveBackends - Expression: haproxy_backend_active_servers + haproxy_backend_backup_servers == 0 - Summary: HAProxy has no alive backends - Description: Alerts when no backend servers are alive.

HaproxyFrontendSecurityBlockedRequests - Expression: sum by (proxy) (rate(haproxy_frontend_denied_connections_total[2m])) > 10 - Summary: HAProxy frontend security blocked requests - Description: Indicates high rate of blocked frontend connections.

HaproxyServerHealthcheckFailure - Expression: increase(haproxy_server_check_failures_total[1m]) > 0 - Summary: HAProxy server healthcheck failure - Description: Alerts when server health check failures occur.