Topitall

Introduction; John Belshaw [topitall@eircom.net]

Topitall is a Unix and Windows- NT and 2000 system monitoring tool designed to run as an agent, or daemon, to gather, and display as html, time series graphs of the most useful system resource information. Topitall uses the scripting language, Perl, common unix system tools, and the Win32 PerfLib library to gather system information every minute. The performance data can be displayed using the Round Robin Database [RRD] utility as a series of time-series graphs. The parameters collected are basic system parameters such as memory, network, swap and cpu usage, per process or application, and file system usage. Topitall is designed to be quite easy to install and configure and so can be used as a "shrink-wrapped" performance monitoring tool which has the big advantage that it works in the same way on many Unix platforms and windows- NT and 2000.

Stop Press ! : Topitall 2.4a, with NT4 and windows 2000 Client released!

If you want to know to know how your Unix, or NT4 boxes are spending their time, whilst you are not watching, for example when you are off sleeping, eating or sailing to far off places then Topitall could be for you !

Topitall also can be easily configured to generate alerts by syslog and email based on any of the collected parameters and so act as a simple Enterprise System Monitoring [ESM] tool. For example, Topitall could be used to detect system resource problems such as stopped or runaway processes, over or under used file system capacity, or when a process or application has exceeded a system resource threshold. What more do you really need ?

There are two main functions of Topitall, a topitall client and a topitall server.
The client agent gathers system performance statistics and process information, generates alerts, and stores the data in a database containing one years data. This agent also prepares html files and gif graphs of the data so it can be viewed on the client machine if you have a web server configured on the machine. The alerts generate emails and syslog entries and can be configured to run user defined perl scripts which can attempt to take corrective action to restore the normal running of the machine. The client machine can run in standalone mode or can send the alerts and a selected subset of the data to a central topitall server.

The topitall server collects data from topitall clients and generates a state database of data, and alerts which can currently be browsed with web cgi-scripts. This includes a monitor screen with one button per host coloured to represent the state of the host, according to how recent an alert has been received, and whether data is currently being received from the client. This button gives access to the host screen which has a list of the last ten alerts, again colour coded and a table of the data available for that host. Each of the data items can be browsed as a graph at tme scales of an hour, day, week, month or whole year.

Topitall Client

So what do the graph's look like ? here is an example with just three graphs, a normal page will have over thirty graphs, plus three graphs per configured process/application.

The client will also generate emails automatically, amd syslog entries. A simple mechanism is made to allow a use written perl subroutine to be invoked when an alert is triggered.

Here is an example Alert email :

Subject:
jcbhp named : N <1
Date:
Fri, 28 Dec 2001 08:54:03 GMT
From:
John Belshaw

Named has stopped, check slave server for dns !

Automatic Mail from Topitall

Here is an example syslog entry

Dec 28 08:51:08 jcbhp Topitall: Alert named N <1

Topitall Server

Here are mock-ups of the screens, so the links do not work here ;-(

Topitall Host Monitor

Gerthild	homer86	JamesCaird	Asterix	Obelix	Mail	dns1
dns2	jcbhp

No Connect

Alert

Alert <2T

Alert <5T

In this screen the top line of buttons are one per client, the colours represent the state of the host according to the key in the bottom line.

Topitall : homer86

homer86
2002 Feb 23 21:34	homer86 5/5 Disk /dev/dsk/c0d0s0 >=65	user@hostname	Boot disk getting full
2002 Feb 23 21:34	homer86 5/5 named N <1	jbelshaw@localhost	Named has stopped, check slave server for dns !
2002 Feb 23 21:39	homer86 5/5 System LoadAv <5	user@hostname	System is getting busy
2002 Feb 23 21:39	homer86 5/5 Disk /dev/dsk/c0d0s0 >=65	user@hostname	Boot disk getting full
2002 Feb 23 21:39	homer86 5/5 named N <1	jbelshaw@localhost	Named has stopped, check slave server for dns !
2002 Feb 23 21:44	homer86 5/5 System LoadAv <5	user@hostname	System is getting busy
2002 Feb 23 21:44	homer86 5/5 Disk /dev/dsk/c0d0s0 >=65	user@hostname	Boot disk getting full
2002 Feb 23 21:44	homer86 5/5 named N <1	jbelshaw@localhost	Named has stopped, check slave server for dns !
2002 Feb 23 21:48	homer86 5/5 System LoadAv <5	user@hostname	System is getting busy
2002 Feb 23 21:48	homer86 5/5 Disk /dev/dsk/c0d0s0 >=65	user@hostname	Boot disk getting full
2002 Feb 23 21:48	homer86 5/5 named N <1	jbelshaw@localhost	Named has stopped, check slave server for dns !

CPUidle	CPUsystem	CPUuser	IOin	IOout	LoadAv	Mem
MemBuff	MemCache	MemFree	MemSwap	NProc	Nrun	SwapIn
SwapOut	Uptime	Users	_dev_hda5	eth0RXok	eth0TXok	gnomeCpu
gnomeMem	gnome N	httpd Cpu	httpd Mem	httpd N	named Cpu	named Mem
named N	perl Cpu	perl Mem	perl N

homer86

Monitor

In this screen the ten most recent alerts can be seen colour coded in the upper table. The lower table contains colour coded buttons to link to the performance graphs per parameter.

homer86 CPUuser

Hour

Day

Week

Month

Year

homer86

Monitor

This is an individual performance parameter graph which can be generated by time period.

Performance

The topitall agent is designed to run at a very low load and so not interfere with other applications. The size of the client data directory is about 3MB, but it will not grow any bigger. The server gathers data from all clients and so the load will scale with the number of clients, the amount of uploaded data, the update frequency, and the number of alerts generated; all of which can be configured per client. I estimate that one topitall server, say a Sun Ultra 10, could possibly handle up to 200 clients.

Work In Progress

Create a Veritas Volume manager performance plug-in.
Build a Java front end
Develop and NT/Win2000 client

Comments

I wrote version 1 of topitall whilst bored sailing to the Seychelles on my laptop using RedHat Linux 6.2, and so I had limited resources to distract me. If you like Topitall tell all your friends and get them to install it. If you have any comments or constructive criticism please send them to topitall@eircom.net and I will try and answer them. If you have any success customizing the config files for particular applications I would welcome seeing them and I will include them in future example files with an acknowledgement.

Restrictions

Topitall is distributed under the artistic License, so you can freely download it, use it and extend it as long as you respect my copyright. See the license file here.