Description:
The log-collector gets data from remote log-server's. This way the
whole log-file doesn't have to be transfered. The "protocol",
if it deserves that name, is very simple. The collector sends a request,
which looks like (you can type it in via telnet):
LOGFILE /wherever/the/logfile.is
varname function pattern
...
GO
The directives are all in UPPERCASE. They are LOGFILE
,
GO
, DEBUG
and TEST
. The LOGFILE
directive
tells the log-server
which file to read. The GO
directive starts the request. DEBUG
causes some extra remote debugging output, and TEST
makes the
log-server
operate in test mode. In test mode it doesn't update the
last-read position for that log-file, so you won't lose any data when testing.
The other lines are telling the log-server
what data to collect.
The first "word" is the variable name to be returned. The next is the function
to be applied (from count
, sum
, average
, min
, max
,
first
, last
) and lastof
. The rest of the line is a perl-style regex.
Note1: The lastof
function is different from the others, in several
respects. First, it takes multiple regex's. Second, it saves the current
value of the variable in a context file with the same name as the context file
for the log-file with '-' and the variable-name appended. The way it works is
to return the number of the pattern which was last matched (starting with 1).
If none of the patterns have ever matched, it will return zero, until one
of them matches.
Note2: Except for the count function, the regex must contain a
(parenthesized) expression which will match a number, to which the function
will be applied.
For example, the line:
rootlogins count ROOT LOGIN
would return data for a variable called rootlogins. The value would be
the count of the records in the specified logfile which had the string
'ROOT LOGIN' in them.
The pattern can be much more complicated, for example (from the httpdlog rrd):
bytes sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)
This looks through a standard web-server log-file and extracts the bytes
transferred and adds them up to produce the total number of bytes
transferred for successes in that sample period.
How to make RRDs that use the log-collector
It's easiest to explain by example. Look at the beginning of the
rrd httpdlog
, copied here:
source log
step 300
data requests GAUGE:600:0:U count (GET|POST)
data success GAUGE:600:0:U count \sHTTP/\d\.\d"\s+2\d\d
data bytes GAUGE:600:0:U sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)
To form the requests to be sent to the log-server, the log-collector takes the
DS name, e.g. success
, and the last part of the line after all the
DS definition count \sHTTP/\d\.\d"\s+2\d\d
, combines the two and sends:
success count \sHTTP/\d\.\d"\s+2\d\d
Note that the pattern can include magic cookies as of remstats
version 0.12.2.
The line in the host config-file must specify the log-file name, like:
httpdlog /var/log/httpd/access_log
This permits the same code to get information from the same log format, no
matter where it's stored on various machines. Since log-file locations
seem to vary a lot by host, I put the log-file specification in the host
config-file, rather than in the rrd config-file.
[