Home > DevOps > Infrastructure As a Code > Monitor Per Client Data Usage

Monitor Per Client Data Usage

Prerequisites to Set Up Monitoring Data Usage Per Client

Step 1: Configure SSH for Easier Script Editing and Accessing Router Features

Follow these steps to enable SSH on your MikroTik router for seamless access:

Log in to Winbox using Wine on your system.
Once logged in, navigate to the IP menu in the Winbox dashboard.
Select the Services tab from the dropdown options.
Locate the SSH service in the list. It should appear with a greyed-out background if disabled.
Double-click on the SSH row to open its configuration window.
Fill out the required details in the new window:
- Port: Specify the port number for SSH (default is 22).
- Available From: Enter the IP address range or specific address of the allowed users.
Click the Enable button to activate the SSH service.
Generate an SSH key pair from your system if you don’t already have one:
- Run the following command in your terminal:
```
ssh-keygen -t ed25519
```
- Save the private key in a secure location, as losing it can result in losing access to your router.
- Upload the generated public key from your system (usually located in ~/.ssh/id_ed25519.pub), to the Menu -> Files section by clicking on the Upload button on the Files tab.
Configure any additional SSH settings:
- Under IP > SSH, ensure Port Forwarding is set to No for enhanced security.
Configure the SSH service for secure access:

Navigate to IP > SSH.
Select the Import Host Key option to enhance security.
Select the file you uploaded in the previous steps by selecting the file uploaded in the dropdown.

To log in to the router via SSH:
- Open your terminal and use the following command:
```
ssh <router_username>@<router_ip> -p <port>
```
  the username would usually be Sayukth Replace <router_username> with the username you use to log in to the router (e.g., admin), <router_ip> with the router’s IP address, and <port> with the SSH port configured on the router (default: 22).

Step-2: Enable API service on the router:

Go to IP -> Services and then double click on the api option and fill out the relevant fields such as port, Available from with port as 8728(default) and Available from with the IP address of the monitoring server that access the api.
Click on the Enable button and apply button to save changes.

Step-3: Add Mangle script on the router:

Go to System -> Scripts and then click on the + button and then fill in details like Owner as user Sayukth or the one you are logged in with and name with add_mangle_rules and tick the policies read, write and fill in the source tab with the following script:

:foreach arpEntry in=[/ip arp find] do={
    :local ipAddress [/ip arp get $arpEntry address];
    :put $ipAddress;
    :local macAddress [/ip arp get $arpEntry mac-address];
    :put $macAddress;
    :local dhcpEntry [/ip dhcp-server lease find where active-address=$ipAddress]
    :if ($dhcpEntry = "") do={
        :put "This doesn't work for this IP";
    } else={
        :local hostname [/ip dhcp-server lease get $dhcpEntry host-name];
        :put $hostname;
        :if ($hostname = "") do={
            :set hostname "UnknownHost";
            :put $hostname;
        }
        :local mangleRuleEntry 
        :local existingDownloadRule [/ip firewall mangle find where dst-address="$ipAddress" and in-interface="ether1"]
        :put $existingDownloadRule
        :local existingUploadRule [/ip firewall mangle find where src-address="$ipAddress" and out-interface="ether1"]
        :put $existingUploadRule
        :if ($existingDownloadRule = "") do={
            /ip firewall mangle add action=passthrough chain=forward dst-address=$ipAddress in-interface=ether1 comment=("Download " . $hostname)
            /log info ("Added download rule for " . $hostname . " IP " . $ipAddress)
        } else={
            /log info ("Download rule already exists for " . $hostname . " IP " . $ipAddress)
        }
        :if ($existingUploadRule = "") do={
            /ip firewall mangle add action=passthrough chain=forward src-address=$ipAddress out-interface=ether1 comment=("Upload " . $hostname)
            /log info ("Added upload rule for " . $hostname . " IP " . $ipAddress)
        } else={
            /log info ("Upload rule already exists for " . $hostname . " IP " . $ipAddress)
        }
        /log info ("Mangle Rules Updated for HostName " . $hostname . " IP " . $ipAddress);
    }   
}

Try clicking on Run script to see entries in log and make sure if the setup is clear.
Goto Scheduler under System and click on + button to add a schedule for the before created script to make it execute for every 5 minutes. Configure parameters to be in similar as below:

Step-4: Setting up mktxp exporter

Install mktxp exporter on the monitoring instance:

pip install mktxp

Use mktxp edit -ed vim to edit the default mktxp configuration file and modify the parameters as given in the following config file and make sure of port to be the same as in the router’s api configuration and host as the router’s IP address:

## Copyright (c) 2020 Arseniy Kuznetsov
##
## This program is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License
## as published by the Free Software Foundation; either version 2
## of the License, or (at your option) any later version.
##
## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
## GNU General Public License for more details.


[winbox]
    # for specific configuration on the router level, change here the defaults values from below
    hostname = 172.21.0.1

[default]
    netwatch = True
    capsman = True
    no_ssl_certificate = False
    ipv6_firewall = False
    use_comments_over_names = True
    poe = True
    installed_packages = True
    queue = True
    monitor = True
    connection_stats = True
    kid_control_assigned = False
    dhcp_lease = True
    lte = False
    firewall = True
    ipsec = False
    enabled = True
    kid_control_dynamic = False
    capsman_clients = True
    plaintext_login = True
    ipv6_pool = False
    ssl_certificate_verify = False
    ipv6_route = False
    neighbor = True
    route = True
    pool = True
    use_ssl = False
    switch_port = False
    dhcp = True
    wireless = True
    user = True
    connections = True
    bgp = False
    ipv6_neighbor = False
    interface = True
    public_ip = True
    wireless_clients = True
    check_for_updates = False
    hostname = 172.21.0.1
    username = mktxp_user
    password = mktxp_user_password
    remote_dhcp_entry = None
    remote_capsman_entry = None
    port = 8728

Log into the mikrotik router using SSH which has been configured above and run the following commands to setup credentials for the router api(Make sure to match the password in mktxp config and in the following command):
```
/user group add name=mktxp_group policy=api,read
/user add name=mktxp_user group=mktxp_group password=mktxp_user_password
```
Check if the installation is successful by using mktxp print -en winbox -dc, you should see similar output with much more entries depending on the router:

2024-11-18 19:34:40 Connection to router winbox@172.21.0.1 has been established
+---------------------------+--------+-------------------+--------------+----------------+---------------+
|         host_name         | server |    mac_address    |   address    | active_address | expires_after |
+===========================+========+===================+==============+================+===============+
| elitical-in-dev1a-node01  | dhcp1  | 0A:01:E9:20:DE:04 | 172.21.7.49  |  172.21.7.49   |     7m59s     |
| devops-ein0034-vm003      | dhcp1  | 0A:01:0D:20:D3:34 | 172.21.4.56  |                |               |
| devops-ein0034-vm002      | dhcp1  | 0A:01:0D:20:D2:34 | 172.21.4.55  |                |               |
| devops-ein0034-vm001      | dhcp1  | 0A:01:0D:20:D1:34 | 172.21.4.54  |                |               |
| elitical-tmpl-1           | dhcp1  | 0A:01:E9:20:00:02 | 172.21.4.41  |                |               |
| devops-ein0031-vm003      | dhcp1  | 0A:01:0D:20:D3:31 | 172.21.4.36  |                |               |
| devops-ein0031-vm002      | dhcp1  | 0A:01:0D:20:D2:31 | 172.21.4.35  |  172.21.4.35   |     8m55s     |
| devops-ein0031-vm001      | dhcp1  | 0A:01:0D:20:D1:31 | 172.21.4.34  |  172.21.4.34   |     6m14s     |
| elitical-devbox-ein0064   | dhcp1  | 0A:01:E0:20:01:25 | 172.21.4.14  |                |               |
| HPE-MCF97CJWXX            | dhcp1  | BC:D0:74:04:4A:54 | 172.21.2.83  |  172.21.2.83   |     5m21s     |
| I2223                     | dhcp1  | 14:DD:9C:CC:83:C9 | 172.21.2.72  |  172.21.2.72   |     5m54s     |
| V2338                     | dhcp1  | F2:4A:87:BC:37:DA | 172.21.2.71  |  172.21.2.71   |     8m5s      |
| DESKTOP-JHLE30H           | dhcp1  | 90:32:4B:2E:A3:B7 | 172.21.2.69  |  172.21.2.69   |     1m56s     |
| 08:00:27:EF:34:90         | dhcp1  | 08:00:27:EF:34:90 | 172.21.2.67  |  172.21.2.67   |     9m28s     |
| 08:00:27:9C:6F:7B         | dhcp1  | 08:00:27:9C:6F:7B | 172.21.2.61  |  172.21.2.61   |     6m14s     |

Setup scrape config for prometheus to be able scrape metrics from mktxp by entering the following configuration in /etc/prometheus/prometheus.yml in the scrape_configs section and reload the prometheus service

  - job_name: 'mktxp'
    static_configs:
      - targets: ['172.21.0.45:49090']

Setup mktxp export as a service by using vi /etc/systemd/system/mktxp_exporter.service and enter the following content:

[Unit]
Description=MKTXP Exporter

[Service]
User=root
ExecStart=mktxp export

[Install]
WantedBy=default.target

Enable and start the above service using:

systemctl daemon-reload
systemctl enable --now mktxp_exporter.service

Setup grafana dashboard for mktxp_exporter by importing the dashboard on grafan UI using the dashboard ID: 13679.
As this dashboard doesn’t include the datausage panel, we can use the following query in the query box, make sure to use the Unit as gigabytes from the right panel to correctly display the usage on the graph: Query-A: label_replace(increase(mktxp_firewall_mangle_total{name=~".*Download.*", name!~".*Global.*"}[1d]) / 1024 / 1024 / 1024 * -1, "name", "$1", "name", ".*Download (.*)")

Query-B: label_replace(increase(mktxp_firewall_mangle_total{name=~".*Upload.*", name!~".*Global.*"}[1d]) / 1024 / 1024 / 1024, "name", "$1", "name", ".*Upload (.*?) \\| .*")

dashboard_queries