TCGraph - visualise Traffic Control statistics using RRDtools

networking tcgraph perl tc qos

TCgraph is very similar to mailgraph: it stores the statistics from TC into a RRD database and use a CGI perl script to displays those statistics as RRD graphs.

At the end, you will obtain something like this :

Our system is Debian Lenny. We have 3 main components:

1. tcgraph.rrd : The RRD database, that will contain the statistics and do the data manipulation
2. tcparsestat.pl: The Perl script that will query the TC statistics and store the result into tcgraph.rrd
3. tcgraph.cgi: The Perl CGI script that will generate graphs from tcgraph.cgi and display it to the end user

And, because an image is worth a thousand words:

Let's assume we have the following QoS policy:

The script is available here: qos-tc-script.zip

We need to create a RRD database designed to store bandwidth usage value for each of these classes.

To create and manipulate a RRD database, you need the following package:

In the script below, each DS entry designates a class, and each RRA entry designates a storage policy.

#! /bin/bash

rrdtool create tcgraph.rrd --start now --step 60 \
DS:interactive:COUNTER:120:0:786432 \
DS:tcp_acks:COUNTER:120:0:786432 \
DS:ssh:COUNTER:120:0:786432 \
DS:http_s:COUNTER:120:0:786432 \
DS:file_xfer:COUNTER:120:0:786432 \
DS:downloads:COUNTER:120:0:786432 \
DS:default:COUNTER:120:0:786432 \
RRA:AVERAGE:0.5:1:10080 \
RRA:MAX:0.5:1:10080 \
RRA:LAST:0.5:1:1 \
RRA:AVERAGE:0.5:60:1440 \
RRA:MAX:0.5:60:1440 \
RRA:AVERAGE:0.5:720:732 \
RRA:MAX:0.5:720:732

What this scripts do is the following:

1. Generate a RRD database called 'tcgraph.rrd' that starts now and have an update frequency of 60 seconds
2. Create, for each class, a counter that will have a maximum value of 786432 (768kbps)
3. Create, for each class, 3 storage policy:

• RRASET 1 keep 1 record for 1 minute for 7 days (10080 records)
• RRASET 2 keep 1 record for 60 minutes for 60 days (1440 records)
• RRASET 3 keep 1 record for 720 minutes for 366 days (732 records)

→ RRA MAX and RRA LAST are additional storages that keep the max and last value for a period

Make sure you have the package

# aptitude install rrdtool

And launch the script. It will create a file called tcgraph.rrd that we need to store into /var/www/tcgraph/.

Given our policy, the TC output will look like this :

# tc -s class show dev eth0
 
class htb 1:99 parent 1:1 leaf 199: prio 7 rate 109000bit ceil 768000bit burst 2Kb cburst 1983b
 Sent 2275664 bytes 1821 pkt (dropped 0, overlimits 0 requeues 0)
 rate 26320bit 2pps backlog 0b 0p requeues 0
 lended: 1306 borrowed: 515 giants: 0
 tokens: 137986 ctokens: 19369
 
class htb 1:1 root rate 768000bit ceil 768000bit burst 1983b cburst 1983b
 Sent 8947521 bytes 7293 pkt (dropped 0, overlimits 0 requeues 0)
 rate 164248bit 16pps bac
[....]

So, to get the bandwidth usage of each class, we need to parse the “Sent” value of each class. This can be done with Perl using the following script.

#! /usr/bin/perl -w

#######################
# tcparsestat.pl
# --------------
# read class statistics from tc command line
# convert them from bytes to bits
# store them into a RRD database
# --------------
# j. vehent - 04/2010
#######################
 
use strict;
use RRDs;
 
use Proc::Daemon;
Proc::Daemon::Init;
 
my $rrdfile = "/var/www/tcgraph/tcgraph.rrd";
my $logfile = "/var/www/tcgraph/tcgraph.log";
my $updatefreq = 60;
 
while(1)
{
        # define list of classes to check with default value = 'U'
        # ('U' means unknown in RRD tool langage)
        my %classlist=(
           10 => 'U',
           20 => 'U',
           30 => 'U',
           40 => 'U',
           50 => 'U',
           60 => 'U',
           99 => 'U'
           );
 
        my %valuelist = %classlist;
 
        # get statistics from command line
        open(TCSTAT,"tc -s class show dev eth0 |") || die "could not open tc command line";
                
        # look for specified classes into command line result
        while(<TCSTAT>)
        {
           chomp $_;
           # do we have class information in this line ?
           foreach my $class (keys %classlist)
           {
              if ($_ =~ /\:$class parent/)
              {
                 # If yes, go to the next line and get the Sent value
                 my $nextline = <TCSTAT>;
 
                 my @splitline = split(/ /,$nextline);
 
                 # multiplicate by 8 to store bits and not bytes, and store it
                 $valuelist{$class} = $splitline[2]*8;
 
                 # do not check this specific class for this time
                 delete $classlist{$class};
              }
           }
        }
 
        my $thissecond = time();
 
        # update line is :
        # <unix time>:<statistic #1>:...:<statistic #n>
        my $updateline = time().":$valuelist{'10'}:$valuelist{'20'}:$valuelist{'30'}:$valuelist{'40'}:$valuelist{'50'}:$valuelist{'60'}:$valuelist{'99'}";
        RRDs::update $rrdfile, "$updateline";
 
        if (defined $logfile)
        {
           open(TCGLOG,">>$logfile");
           print TCGLOG "$updateline\n";
           close TCGLOG;
        }
 
        close TCSTAT;
 
        # sleep until next period
        sleep $updatefreq;
}

There are a few important items in this script that you need to modify according to your QoS Policy.

I stores my files in the public web server directory, but you might want to change that:

my $rrdfile = "/var/www/tcgraph/tcgraph.rrd";
my $logfile = "/var/www/tcgraph/tcgraph.log";

The script needs a list of the classes to parse. In my example, there are 7 classes (from 10 to 99). I simply declare those classes and set a default value at 'U'. In theRRD language, 'U' means that the value is unknown, so if for whatever reason the script cannot parse the corresponding class, it will use the 'U' value when updating the database.

        my %classlist=(
           10 => 'U',
           20 => 'U',
           30 => 'U',
           40 => 'U',
           50 => 'U',
           60 => 'U',
           99 => 'U'
           );

The update line must match the previous classlist:

        # update line is :
        # <unix time>:<statistic #1>:...:<statistic #n>
        my $updateline = time().":$valuelist{'10'}:$valuelist{'20'}:$valuelist{'30'}:$valuelist{'40'}:$valuelist{'50'}:$valuelist{
'60'}:$valuelist{'99'}";

With all of this, your script should be ready to be launched. Simply make sure that you have the following libraries :

# aptitude install librrds-perl libproc-daemon-perl

And launch it from the command line. You can check the parsed values from the log file /var/www/tcgraph/tcgraph.log.

# /usr/bin/tcparsestat.pl
 
# tail -f /var/www/tcgraph/tcgraph.log
1271339880:326104:3585984:2871760:87736:603777088:0:36184008

Good to go.

<note tip>You can try to produce a graph directly using the RRDtool command line. Try with the following script : buildtcgraph.sh.tar

It takes two argument: the name of the image to produce, and the period in seconds. For example

# sh buildtcgraph.sh current.png -3600
797x244

creates a file called current.png that contain the statistics for the last hour. </note>

Below is a simple script to put into /etc/init.d/tcparsestat that I use to launch the daemon when the system starts. Don't forget to put a symbolic link into /etc/rc2.d/

#!/bin/sh
#
### BEGIN INIT INFO
# Provides:          tcparse
# Required-Start:    $syslog
# Required-Stop:     $syslog
# Should-Start:      $local_fs
# Should-Stop:       $local_fs
# Default-Start:     2
# Default-Stop:      0 1 6
# Short-Description: starts/stops tcparse
# Description:       starts and stops tcparse,
#                    a graphing daemon for trafic control
### END INIT INFO
 
DAEMON=/usr/bin/tcparsestat.pl
NAME=tcparsestat.pl
DESC="trafic control graphing tool"
 
if [ $(id -u) != 0 ]; then
  echo "You should run this program as root"
  exit 1
fi
 
[ -x "$DAEMON" ] || exit 0
 
. /lib/lsb/init-functions
 
case "$1" in
        start)
                log_begin_msg "Starting $DESC:" "$NAME"
                start-stop-daemon --start --nicelevel 15 --exec $DAEMON -- --daemon --uid=root >/dev/null || return 2
                log_end_msg $?
        ;;
        stop)
                log_begin_msg "Stopping $DESC:" "$NAME"
                start-stop-daemon --stop --name $NAME
                log_end_msg $?
        ;;
        restart|force-reload)
                $0 stop
                sleep 1
                $0 start
        ;;
        *)
                echo "Usage: $NAME {start|stop|restart|force-reload}" >&2
                exit 3
        ;;
esac

Once upon a time, a geeky sysadmin from the beautiful country of switzerland got bored and wrote MailGraph to produce graph of the activity of his Postfix server.

TCgraph is similar to mailgraph in many ways, except that I couldn't automate it as much as mailgraph. The main reason to this is that all QoS policies are different, unlike postfix's log files that are all the same.

Anyway, Mailgraph displays graphes using a Perl script that is called using CGI on the web server. I personnally use Nginx to handle the HTTP request and fcgiwrap to execute the script(howto available here).

Now, below is a modified version of mailgraph.cgi, called tcgraph.cgi. The version below works with the QoS policy defined earlier, but if you have special needs, we will look at how to modify it.

#!/usr/bin/perl -w
 
# tcgraph -- traffic control graphing tool
# Julien Vehent - 04/2010
# inspired from Mailgraph: David Schweikert <david@schweikert.ch> 
# released under the GNU General Public License
 
use RRDs; 
use POSIX qw(uname);
 
my $VERSION = "20100415";
 
my $host = (POSIX::uname())[1]; 
my $scriptname = 'tcgraph.cgi'; 
my $xpoints = 500;
my $points_per_sample = 3; 
my $ypoints = 110;
my $ypoints_err = 50; 
my $rrd = '/var/www/tcgraph/tcgraph.rrd';      # path to where the RRD database is 
my $tmp_dir = '/tmp/tcgraph'; # temporary directory where to store the images 
 
my @graphs = ( 
   { title => 'Last Hours', seconds => 3600*4,   },
   { title => 'Last Day',   seconds => 3600*24,   }, 
   { title => 'Last Week',  seconds => 3600*24*7, }, 
   { title => 'Last Month', seconds => 3600*24*31,     }, 
   { title => 'Last Year',  seconds => 3600*24*365, },
); 
 
sub rrd_graph(@) 
{
   my ($range, $file, $ypoints, @rrdargs) = @_; 
   my $step = $range*$points_per_sample/$xpoints;
   # choose carefully the end otherwise rrd will maybe pick the wrong RRA: 
   my $end  = time; $end -= $end % $step;
   my $date = localtime(time);
   $date =~ s|:|\\:|g unless $RRDs::VERSION < 1.199908; 
 
   my ($graphret,$xs,$ys) = RRDs::graph($file,
      '--imgformat', 'PNG',
      '--width', $xpoints, 
      '--height', $ypoints,
      '--start', "-$range",
      '--end', $end,
      '--vertical-label', 'bits/s', 
      '--lower-limit', 0,
      #'--units-exponent', 0, # don't show milli-messages/s 
      '--color', 'BACK#333333', 
      '--color', 'SHADEA#000000', 
      '--color', 'SHADEB#000000', 
      '--color', 'CANVAS#000000', 
      '--color', 'GRID#999999', 
      '--color', 'MGRID#666666',
      '--color', 'FONT#CCCCCC', 
      '--color', 'FRAME#333333',
      #'--textalign', 'left',
      #'--lazy', 
      $RRDs::VERSION < 1.2002 ? () : ( '--slope-mode'), 
 
      @rrdargs,
 
      'COMMENT:['.$date.']\r', 
   );
 
   my $ERR=RRDs::error; 
   die "ERROR: $ERR\n" if $ERR; 
}
 
sub graph($$)
{
   my ($range, $file) = @_;
   my $step = $range*$points_per_sample/$xpoints;
   rrd_graph($range, $file, $ypoints,
      "DEF:interactive=$rrd:interactive:AVERAGE",
      'AREA:interactive#ffe400:interactive:STACK', 
      'GPRINT:interactive:MAX:\tmax = %6.2lf%Sbps', 
      'GPRINT:interactive:LAST:\tlast = %6.2lf%Sbps',
      'GPRINT:interactive:AVERAGE:\tavg = %6.2lf%Sbps\n',

      "DEF:tcp_acks=$rrd:tcp_acks:AVERAGE",
      'AREA:tcp_acks#b535ff:tcp_acks:STACK', 
      'GPRINT:tcp_acks:MAX:\tmax = %6.2lf%Sbps', 
      'GPRINT:tcp_acks:LAST:\tlast = %6.2lf%Sbps',
      'GPRINT:tcp_acks:AVERAGE:\tavg = %6.2lf%Sbps\n', 
       
      "DEF:ssh=$rrd:ssh:AVERAGE", 
      'AREA:ssh#1b7b16:ssh:STACK',
      'GPRINT:ssh:MAX:\t\tmax = %6.2lf%Sbps', 
      'GPRINT:ssh:LAST:\tlast = %6.2lf%Sbps', 
      'GPRINT:ssh:AVERAGE:\tavg = %6.2lf%Sbps\n',
 
      "DEF:http_s=$rrd:http_s:AVERAGE", 
      'AREA:http_s#ff0000:http_s:STACK',
      'GPRINT:http_s:MAX:\tmax = %6.2lf%Sbps',
      'GPRINT:http_s:LAST:\tlast = %6.2lf%Sbps',
      'GPRINT:http_s:AVERAGE:\tavg = %6.2lf%Sbps\n', 
 
      "DEF:file_xfer=$rrd:file_xfer:AVERAGE", 
      'AREA:file_xfer#00CC33:file_xfer:STACK',
      'GPRINT:file_xfer:MAX:\tmax = %6.2lf%Sbps', 
      'GPRINT:file_xfer:LAST:\tlast = %6.2lf%Sbps', 
      'GPRINT:file_xfer:AVERAGE:\tavg = %6.2lf%Sbps\n',
 
      "DEF:downloads=$rrd:downloads:AVERAGE",
      'AREA:downloads#7316f2:downloads:STACK', 
      'GPRINT:downloads:MAX:\tmax = %6.2lf%Sbps',
      'GPRINT:downloads:LAST:\tlast = %6.2lf%Sbps',
      'GPRINT:downloads:AVERAGE:\tavg = %6.2lf%Sbps\n',
 
      "DEF:default=$rrd:default:AVERAGE",
      'AREA:default#bcdd94:default:STACK', 
      'GPRINT:default:MAX:\tmax = %6.2lf%Sbps',
      'GPRINT:default:LAST:\tlast = %6.2lf%Sbps',
      'GPRINT:default:AVERAGE:\tavg = %6.2lf%Sbps\n' 
 
   );
}
 
 
sub print_html() 
{
   print "Content-Type: text/html\n\n";
 
   print <<HEADER;
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 
<title>Traffic Control statistics for $host</title>
<meta http-equiv="Refresh" content="300" /> 
<meta http-equiv="Pragma" content="no-cache" />
<link rel="stylesheet" href="tcgraph.css" type="text/css" /> 
</head> 
<body>
HEADER
 
   print "<h1>Traffic Control statistics for $host</h1>\n"; 
 
   print "<ul id=\"jump\">\n";
   for my $n (0..$#graphs) { 
      print "  <li><a href=\"#G$n\">$graphs[$n]{title}</a>&nbsp;</li>\n";
   } 
   print "</ul>\n"; 
 
   for my $n (0..$#graphs) { 
      print "<h2 id=\"G$n\">$graphs[$n]{title}</h2>\n"; 
      print "<p><img src=\"$scriptname?${n}-n\" alt=\"tcgraph\"/></p>\n";
   } 
 
   print <<FOOTER;
<hr/> 
<table><tr><td>
<a href="http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:networking:tcgraph">TCgraph</a> v.$VERSION
by <a href="http://jve.linuxwall.info">Julien Vehent</a></td>
<td align="right">
<a href="http://oss.oetiker.ch/rrdtool/"><img src="http://oss.oetiker.ch/rrdtool/.pics/rrdtool.gif" alt="" width="60
" height="17"/></a> 
and <a href="http://david.schweikert.ch/">Mailgraph</a></td> 
</td></tr></table>
</body></html> 
FOOTER
}
 
sub send_image($)
{
   my ($file)= @_;
 
   -r $file or do {
      print "Content-type: text/plain\n\nERROR: can't find $file\n";
      exit 1;
   };
 
   print "Content-type: image/png\n";
   print "Content-length: ".((stat($file))[7])."\n";
   print "\n";
   open(IMG, $file) or die;
   my $data;
   print $data while read(IMG, $data, 16384)>0;
}
 
sub main()
{
   my $uri = $ENV{REQUEST_URI} || '';
   $uri =~ s/\/[^\/]+$//;
   $uri =~ s/\//,/g;
   $uri =~ s/(\~|\%7E)/tilde,/g;
   mkdir $tmp_dir, 0777 unless -d $tmp_dir;
   mkdir "$tmp_dir/$uri", 0777 unless -d "$tmp_dir/$uri";
 
   my $img = $ENV{QUERY_STRING};
   if(defined $img and $img =~ /\S/) {
      if($img =~ /^(\d+)-n$/) {
    my $file = "$tmp_dir/$uri/tcgraph_$1.png";
    graph($graphs[$1]{seconds}, $file);
    send_image($file);
      }
      else {
    die "ERROR: invalid argument\n";
      }
   }
   else {
      print_html;
   }
}
 
main;

This script assumes that the tcgraph.rrd database is located in the same directory. Since those data are not confidential, I personally put everything into /var/www/tcgraph/, but it's up to you.

The script also needs write access to a temporary directory.

Those two variables are configurable at the beginning of the script:

my $rrd = 'tcgraph.rrd';      # path to where the RRD database is           
my $tmp_dir = '/tmp/tcgraph'; # temporary directory where to store the images    

You can modify the periods covered by each graph in the table @graphs. I use 5 graphs that cover the following periods: 4 hours, 1 day, 1 week, 1 month and 1 year.

my @graphs = (                     
   { title => 'Last Hours', seconds => 3600*4,   },            
   { title => 'Last Day',   seconds => 3600*24,   },             
   { title => 'Last Week',  seconds => 3600*24*7, },             
   { title => 'Last Month', seconds => 3600*24*31,     },             
   { title => 'Last Year',  seconds => 3600*24*365, },                
);   

The sub-function sub graph contain the patterns to print. This is basically a list of our previously defined classes with some printing rules.

If you have to modify the list of classes you want to graph, it must be done there.

<note important>note that the names of the DEF value must match the one defined in the RRD database.</note>

Example:

sub graph($$)                      
{                        
   my ($range, $file) = @_;                  
   my $step = $range*$points_per_sample/$xpoints;                
   rrd_graph($range, $file, $ypoints, 

      "DEF:interactive=$rrd:interactive:AVERAGE",              
      'AREA:interactive#ffe400:interactive:STACK',             
      'GPRINT:interactive:MAX:\tmax = %6.2lf%Sbps',             
      'GPRINT:interactive:LAST:\tlast = %6.2lf%Sbps',                
      'GPRINT:interactive:AVERAGE:\tavg = %6.2lf%Sbps\n',    

• DEF is the name of the entry, we tell the script it can be found in $rrd (the tcgraph.rrd file) at DS 'interactive'
• AREA describes the printing mode and the color. Here we use a stacked graph (values are summed on top of each other) and a yellow color.
• GPRINT is a command that print additional values below the graph. We use it to display the raw values of MAX, LAST and AVERAGE for the period.

The script displays most of the content, but if you want it to look like, don't forget to copy the CSS file tcgraph.css next to it:

Here is the file:

*     { margin: 0; padding: 0 }
body  { width: 600px; background-color: white;
	font-family: sans-serif;
	font-size: 12pt;
	margin: 0 auto }
h1    { margin-top: 20px; margin-bottom: 30px;
        text-align: center; color: #c30 }
h2    { background-color: #ddd;
	padding: 2px 0 2px 4px }
hr    { height: 1px;
	border: 0;
	border-top: 1px solid #aaa }
table { border: 0px; width: 100%; color: #c30 }
img   { border: 0 }
a     { text-decoration: none; color: #09f }
a:hover  { text-decoration: underline; }
#jump    { margin: 0 0 10px 4px }
#jump li { list-style: none; display: inline;
           font-size: 90%; }
#jump li:after            { content: "|"; }
#jump li:last-child:after { content: ""; }

Providing that:

• your CGI configuration is working
• your RRD database is correctly configured and the script can access it
• the TMP directory is writable and the script can access it
• and a few other fantasies that I forgot about but you're most welcome to comment

You should be able to enjoy the visualization of the following page :