This section describes how to use the sys_debug.sh script as a system diagnostic and troubleshooting tool. The sys_debug.sh script is designed to watch a process (or processes) and send a Jabber alert for log events or alert for consumption of high system resources.
sys_debug.sh differs from sys_mon.sh (in the previous section) in that sys_debug.sh is designed to run continuously as a troubleshooting tool. It monitors a process or resources that a process is using. Thus, sys_debug.sh would be useful for watching processes that are running abnormally, or for monitoring new daemons that are being put into production — as a way to ensure that the new daemons do not consume abnormal resources. Usage is based on the type of monitoring to be performed:
Usage: ./sys_debug.sh -a|-f|-p [test_param(s)]
Usage Options: ./sys_debug.sh [-s cycle_seconds] [-u exit_seconds]
[-q] [-i alert_interval] -a|-f|-p test_params
The -a argument specifies monitoring of available system RAM, where the the '[test_param]' is available system RAM in kilobytes. sys_debug.sh will send an alert if available RAM falls below this threshold. Technically, this does not watch a specific process; however, the intention is to ensure that a process or process group is not consuming too much memory.
The -f argument specifies monitoring of a log file, where '[test_param(s)]' are the full path the log file and a search string. sys_debug.sh will send an alert if the search string is written to the log file. On alert, the last 5 lines of the log file will be reported; however, the number of lines to report can be modified with the optional -n argument. tail and grep can be used to watch log output; what this script does is give you a real time remote alert when a specified string is written to a log. This would be useful for watching for intermittent errors.
The -p argument specifies monitoring of a process via ps , where '[test_param(s)]' consist of a choice (--pid | --ppid | -C | --User) of process filtering followed by a ps header name and threshold. sys_debug.sh can monitor any ps output that returns an integer, and it will alert when ps reports that a process — as filtered by PID, PPID, Command Name, or User — rises above the stated threshold. Multiple ps headers and thresholds are acceptable.
See the help (sys_debug.sh -h) for more detailed usage information, including information about optional arguments. See the ps man page for more information about ps headers (STANDARD FORMAT SPECIFIERS).
The sys_debug.sh script can be downloaded from the tools folder. Once downloaded, you will need to chmod it (chmod a+x sys_debug.sh) in order to run it. sys_debug.sh relies on jabber_alert.pl.
sys_debug.sh requires setup for the JID of the alert sender and recipient, in addition to a password for the sender. Open sys_debug.sh in an editor, and you will see the user configuration near the top of the file. You should edit the file to provide a recipient_jid, sender_jid and sender_pw. You may enter other optional information below.
When using sys_debug.sh, you may wish to used a fixed-width font, such as courier, in your Jabber client so that output columns align properly.
sys_debug.sh must contain a Jabber user password. See the warning in the help message. sys_debug.sh can be tested against a process running on the workstation on which it is installed by testing it against low thresholds. In this example, sys_debug.sh is run to check if any processes running for the user me are consuming more than 4% memory, 3% CPU or have caused more than 10,000 page faults:
./sys_debug.sh -p --User me %mem 4 %cpu 3 majflt 10000
When run, you should receive an alert message that looks something like this:
Subject: Monitor Alert
[16:56:16] <monitor_user@somedomain.com>
Alert: Alert Message
Time: Mon, 2 Aug 16:56:14 UTC: -4.00
User: me
Host: machine.somedomain.com
Service: Process Alert from ./sys_debug.sh
------------Beg-Summary------------
Thresholds for majflt, %cpu, %mem reached:
( %mem over 4 )
PID CMD %mem
11227 vmware-vmx 18.4
11229 vmware-vmx 18.4
11230 vmware-vmx 18.4
11231 vmware-vmx 18.4
11232 vmware-vmx 18.4
11233 vmware-vmx 18.4
11234 vmware-vmx 18.4
11236 vmware-vmx 18.4
11237 vmware-vmx 18.4
( %cpu over 3 )
PID CMD %cpu
11237 vmware-vmx 19.0
( majflt over 10000 )
PID CMD majflt
4422 enlightenment 3893626
11227 vmware-vmx 25091
11237 vmware-vmx 44909
------------End-Summary------------
Subject: Monitor Alert
Note that ps does not report combined memory usage by a parent PID. As far as I know, this is a limitation of Linux versions lower than 2.6.
sys_debug.sh can be used to monitor a new Jabberd 2 installation. After putting a Jabberd 2 server into production use, you might run sys_debug.sh to watch for high memory or CPU usage:
./sys_debug.sh -p --User jabber %mem 20 %cpu 20
Run ./sys_debug.sh -h for more examples.
© 2003 Will Kamishlian and Robert Norris
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/1.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
|
|
|
|
|