RRDTool 95th for Joint In & Out

See Also:
An explenation of 95th percentile, mean, media and mode: /index.php?page=95th-percentile-and-mean-median-mode
Notes on 95th percentile accuracy with rrdtools: /index.php?page=95th-percentile-accuracy-with-rrdtools

Simple bash script to generate a 95th percentile value from an RRD file. Cacti for example, doesn't get them from rrdtool, it generates the values itself as rrdtool isn't actually very good (accurate) at doing this for seperate in & out value. As for a joint in & out value, rrdtool can't actually produce such a value. See here for tests with rrdtool (only) and the 95th percent calculations. This script outputs the 95th percentile value in Kbps.

 

Update 1

Below is the original all Bash script that is slow. Here is a Bash script that is moslty the same, except in the middle an external Perl script is called which speeds things up by an order of magnitude. rrd_faster.sh rrd_organise.pl

The whole thing should be re-writtin in Perl.

 

Update 2

Here is a PHP function which does the same as the old BASH script below, but in PHP which is faster than BASH; check95th_threshold.php.txt. To clarify, this is taking the largest of two 5 minute samples, and presents a Mbps value.

 

95th.sh

#!/bin/bash 

# jwbensley@gmail.com
# Generate 95th percentile value from an rrd file, using two data streams
# "traffic_in" and "traffic"out" as is typical of Cacti. For Observium
# this would be "INOCTETS" and "OUTOCTETS". Change as required.

# Example
# ./95th.sh 1358502298 1358761498 /var/lib/cacti/rra/customer-cpe_traffic_in_11664.rrd

# This script will generate a single 95th percentile figure for
# a given rrd file. It will pull all in and out values for a
# given time period and dump them to a text file. Next it checks
# line by line, which value is bigger (in or out traffic) and
# dumps the larger to a separate file. NaN values are replaced
# with the last recorded value. This final list is sorted,
# and the value 5% down from the top is taken, as is typical
# of 95th percentile billing.

# Blank out a file and dump a given time range of values from rrd file
echo "" > rrd_values
rrdtool fetch -s $1 -e $2 $3 AVERAGE > rrd_values
# Remove the titles
sed -i 1,+1d rrd_values

# Blank out a new file;
# Now we are going to iterate through each time period dumped
# from the rrd file and see which value is bigger, in traffic
# or out traffic, then dump the bigger of the two values in 
# to this new file
echo "" > rrd_values_max

# Sometimes rrd logs "nan", we will replace those values
# with the value at the last working interval, same as
# PRIV function in rrdtool
LASTIN=0.0
LASTOUT=0.0

file="./rrd_values"
while IFS= read -r line
do
	if [ "`echo $line| awk '{printf $2}'`" == "nan" ]
	then
		INNOW="$LASTIN"
	else
		INNOW=`echo $line | awk '{printf "%f", $2}'`
		LASTIN="$INNOW"
	fi
        if [ "`echo $line| awk '{printf $3}'`" == "nan" ]
        then
		OUTNOW="$LASTOUT"
	else
		OUTNOW=`echo $line | awk '{printf "%f", $3}'`
		LASTOUT="$OUTNOW"
	fi
	echo | awk -v n1=$INNOW -v n2=$OUTNOW '{if (n1>n2) printf ("%s\n", n1); else if (n2>n1) printf ("%s\n", n2); else printf ("%s\n", n2);}' >> rrd_values_max
done < "$file"

# Now we have filled ./rrd_values_max with the MAX value from each
# time period, sorted then in desending order
sort -g -r rrd_values_max > rrd_values_max_sorted
# Remove any blank lines (typically one on the end)
sed '/^$/d' rrd_values_max_sorted > rrd_values_max_sorted_trim
# Count the number of lines of rrd values
count=`grep -c ^ rrd_values_max_sorted_trim`
# Get 5 percent of the line count value
NUM=$(echo "scale=4; ($count/100)*5" | bc)
# Round it off
NUM=$(awk -v n1=$NUM 'BEGIN {print int(n1)}')
# Now read the value from the line, 5% down from the top
# of the line count of the file
NUM=`awk -v n1=$NUM 'NR==n1' rrd_values_max_sorted_trim`
# Again, round that off
NUM=$(awk -v n1=$NUM 'BEGIN {print int(n1)}')
# Converts to Kbps
NUM=$(echo "($NUM*8)/1000" | bc)
echo -e "$NUM"

# Clean up
rm rrd_values
rm rrd_values_max
rm rrd_values_max_sorted
rm rrd_values_max_sorted_trim 

---