SARMON - SAR + NMON for Solaris

CPU	Disk	Network

You probably know nmon for Linux and AIX if you come to this page... It is a very simple and nice system monitoring and reporting tool developed by IBM engineer Nigel Griffiths. Recently (July 2009) nmon Linux has been released to the OpenSource community.

NMON has for its reporting aspect many tools to represent the captured data. The main one is "nmon analyzer", to be downloaded from http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/. This Excel macro loads a raw nmon file and generates graphs. I find Excel a perfect tool to manipulate the captured data and render as wish.

For large systems, with number of disk devices greater than 254, nmon analyzer has been edited to a XXL purpose, for Excel 2007 of higher. See bellow for more information.

For more information on this tool and its creator Nigel:

• NMON at IBM DeveloperWorks http://www.ibm.com/developerworks/aix/library/au-analyze_aix/
• NMON Wiki https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/nmon
• NMON Analyser Wiki https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/ Power%20Systems/page/nmon_analyser
• NMON at SourceForge http://nmon.sourceforge.net

Working sometimes on Solaris, I could not find its equivalent for reporting purpose, especially the ability to setup the tool easily, and to get numerous OS raw measurements and graphs on Excel (as opposed to PDF or custom graphing tool).

So I decided to write such a tool, and I found the easiest way was to start from SAR tool (http://docs.sun.com/app/docs/doc/816-5165/sar-1?a=view) and to add few hooks in order to render system activity in NMON file format.

Sarmon also supports fully RRD output.

CPU	IOStat Service Time	Processes Wait Times

No warranty given or implied when using sarmon.

Table of Content

• Architecture
• Project Ground Rules
• Source Code
• Download SARMON
• Fields
• Environment Variables
• RRD Support
• VxVM Support
• Nmon analyzer XXL
• How to Skip SA File Generation
• How to Test SARMON
• How to Install SARMON
• How to Compile SARMON
• Report Issues or Request Enhancements
• Up-Coming Enhancements
• Version History

Architecture

Sarmon supports Solaris 10 and 11. Solaris 10 early versions require sarmon early release ('es' version).

Sadc, the sar daemon which captures OS activity, has been modified to output also the nmon file. If sadc generates a file called for example sa17, then another file called sa17.hostname_yymmdd_hhmm.nmon is generated too.

sadc output native file format is not changed.

All sarmon code has been placed into two separate files (sarmon.c and sarmon.h) with most of its methods and variables being static. Any hook method placed in sadc.c will have its name prefixed by sarmon_ to avoid any confusion. There are currently 5 hooks (init, snap, close, sleep and one to capture usage per CPU) in sadc.c.

Additionally prstat project code has been used with out any change to log statistics per process and for accounting per zone or project. At the end of prstat.c, some code has been added to output statistics in nmon format.

Also iostat partial code has been used too in order to render mount points and NFS name to the raw block device name.

"Linux" OS is recognized by the analyzer via the "AAA,Linux" line inside the nmon file.

Project Ground Rules

The project will follow the following rules for its design and implementation:

• Minimum change in original SAR project code. Only few hooks shall be added to process nmon features outside original code. One key reason is that any change of sar project can be merged in minutes
• sarmon is an extension to sar, so any command parameter, feature and output shall remain unchanged
• sadc output raw file format shall not be changed. This means any data structure required for extending sar (i.e. monitor each CPU) shall be carried within sarmon code, and shall not be placed in raw sadc files
• sarmon can provide more monitoring feature, output shall be part of nmon report
• sarmon nmon reporting shall be compatible with nmon file format (well, not formally document thought!), so that tools such as "nmon analyzer" can process the file. Currently it has been tested with version 33D, 33e, 33f and 43a
• sarmon does not need to run as root

Source Code

Original SAR source code has been downloaded from OpenSolaris, under "Common Development and Distribution License" license. Base code version is build 130 (realligned on build 146 today). Original source code locations can be found at:

• http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/sa/
• http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/prstat/
• http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/stat/c...

SourceForge project at http://sourceforge.net/projects/sarmon/

As few parts are reuse of OpenSolaris, the code uses few private APIs. Over time, I will do my best to remove them over time. Namely as for v1.12: zone_get_id, getvmusage, di_dim_fini, di_dim_path_dev, di_dim_path_devices, di_lookup_node.

Download SARMON

Fields

Environment Variables

Since sarmon follows sadc syntax, there is no room to alter sarmon behavior from the command line. Environment variables is the mechanism choosen in replacement.

export NMONDEVICEINCLUDE="sd0 sd0,a sd0,h nfs1"

RRD Support

Sarmon since v1.02 supports RRD output (tested with v1.2.19, can be downloaded from http://sunfreeware.com/). To enable this feature set the environment variable NMONRRDDIR to an existing directory prior to starting sarmon. For example:

export NMONRRDDIR=/var/adm/sa/sa12rrd

Sarmon will then output 5 files in a append mode. So if the files already exist, then new lines are added at the end

• genall: script which executes the 3 rrd_ create, update and graph scripts. Execute this script to generate the graphs
• rrd_create: to create the RRD databases
• rrd_update: to insert new values to the databases
• rrd_graph: to generate graphs
• index.html: load with your browser to view graphs

For a 1 day case (288 measurements), generation of all graphs shall not exceed 10 seconds.

RRD files can be processed real time with the FIFO file approach, for example

mkfifo /var/adm/sa/sa12rrd/rrd_update

VxVM Support

Sarmon since v1.06 can output VxVM volume IO statistics by aggregating disk IO statistics. It is important to understand that statistics (IOPS, KB in and out, etc) are the aggregation of all disks composing that volume. For example assuming a RAID-1 plex, if an application writes 4KB of data, sarmon KB written reports 8KB, result of 2 writes of 4KB to 2 disks.

Sarmon obtains VxVM configuration by running automatically the following command:

/usr/sbin/vxprint -Ath

Output and device mapping is included in BBBP worksheet.

In the case a disk belongs to multiple volumes via multiple subdisks, sarmon estimates that the load of that volume is in proportion to the size of each subdisk (subdisk relevant fields are only LENGTH and DEVICE from the configuration). In such a case, the volume will be flagged as estimated (est.) to remind this assumption.

VxVM statistics gathering is activated when NMONVXVM environment variable is set.

Nmon Analyzer XXL

Nmon analyzer supports a maximum of 255 disks. For larger systems, statistics for all disks won't be available. More importantly, large system total IOPs (xfers) value is not calculated correctly, SYS_SUMM and DISK_SUMM content is then not correct.

To check if a system requires the adjusted version of the analyzer (and the necessary Microsoft Excel 2007 up), just check any DISK* tab on the normal Excel output. If the column IU contains data (IV being the Totals column), then it is required. For example:

NOT FOUND: sarmon_smallXL2.PNG

As of 8-jun-2013, "nmon analyser v33f XXL.xlsm" is deprecated and "nmon analyser v34a-sarmon1.xls" should be used instead. In this new version based on last nmon analyser 34a, for each DISK* worksheet, only top usage (by WAvg.) 255 disk devices are listed in descending order. Though the total is actually calculated for all devices. This design comes from an Excel limitation I found, it is unable to graph on a worksheet having more than 256 rows and 256 columns at the same time. Any workaround is welcome.

In the new nmon analyser, the PROC worksheet contains a new graph for the kthr variables.

How to Skip SA File Generation

Most admins would continue to rely on the OS bundled sar file generation while adding the nmon file generation, sarmon generated sa file is not necessary. There are 2 ways to skip sa file generation:

1. Run without filename and pipe the output to /dev/null. Nmon file is generated following the hostname_yymmdd_hhmi.nmon naming format. For example: ./sadc 60 10 > /dev/null
2. Set the environment variable NMONNOSAFILE

How to Test SARMON

For this, just download the binaries and put sadc inside any location. Then run the command './sadc 5 4 tst1' which will take 20 seconds (4 snapshots, 5 seconds in between) to run. This will output 2 files, tst1 and tst1.hostname_yymmdd_hhmi.nmon. You can then process the nmon file via the nmon analyzer Excel macro.

How to Install SARMON

Once sarmon has been tested successfully, there are (at least) three ways to install SARMON, the first one now being recommended:

1. One minute setup: download _opt_sarmon.zip, and as root unzip inside /opt. Add the following 2 entries inside root crontab, that's it! The folder contains a README file for more information. sa1daily and sa1monthly shell scripts may need minor adjustments depending on your environment (nmon file location, file retention, VxVM used or not, etc). sa1daily and sa1monthly can be started any time (i.e. after a server reboot), the script automatically calculates the end of day or month

1 0 * * * /opt/sarmon/sa1daily &
2 0 1 * * /opt/sarmon/sa1monthly &

2. Place the entire bin/ directory content at any location, for example under a standard UNIX user home directory or /usr/local/sarmon, modify sa1 script with correct path and possibly some specific sarmon environment variables. Then setup crontab for that user to run sa1 daily. Refer to /usr/cmd/sa/README or UNIX manual of sar for instructions. For example to run sarmon daily, with snapshots every 10 minutes, add the following entry to crontab of that standard UNIX user (avoid using root)

0 0 * * * /usr/local/sarmon/sa1 300 288 &

3. (not recommended) Replace /usr/lib/sa/sadc, /usr/bin/sar and timex by the ones inside the bin/ directory. Make sure you take a backup of the original executables!

How to Compile SARMON

SARMON is currently being developed and tested with GCC. Makefile.master has been updated at few locations, search for keyword 'SARMON' to locate the changes.

1. Install gcc if not present. Binary can be downloaded from http://sunfreeware.com/ or from Solaris installation disk. Code has been tested with gcc v3.4.6 (i386) and v3.4.3 (i386, sparc), both on Solaris 10 10/09 and 1/13. According to ON documentation, one needs to build a higher version, which makes the task hard. Hence step 5 is required to support an old gcc version
2. Install ON build tools SUNWonbld-DATE.PLATFORM.tar.bz2. Binary can be downloaded from http://hub.opensolaris.org/bin/view/downloads/on . Specificaly, on SPARC I use http://dlc.sun.com/osol/on/downloads/20090706/SUNWonbld.sparc.tar.bz2 and on i386 I use http://dlc.sun.com/osol/on/downloads/20091130/SUNWonbld.i386.tar.bz2. Simply unzip then install the package (bunzip2 SUNWonbld.xxx.tar.bz2, then tar -xvf SUNWonbld.xxx.tar, then pkgadd -d onbld)
3. Place source code, for example /a/b/sa
4. Setup environment variables as bellow (ksh syntax)

export PATH=/usr/bin:/usr/openwin/bin:/usr/ucb:/usr/ccs/bin
export MACH=`uname -p`
export CLOSED_IS_PRESENT=no
export CW_NO_SHADOW=Y
export SRC=/a/b/sa/src/usr

5. If building for SPARC (to support old gcc 3.4.3)

export CW_GCC_DIR=/a/b/sa/sparcgcc

6. Due to some incorrect inclusion (at least on Solaris 10 10/09), you may have to modify the file /usr/include/sys/scsi/adapters/scsi_vhci.h and comment out lines that include mpapi_impl.h and mpapi_scsi_vhci.h
7. To change compilation from 64 to 32 bit, change src/usr/Makefile.master (line 315 onward) from -m64 to -m32
8. Go to the correct directory

cd /a/b/sa/src/usr/cmd/sa

9. Build the code

make

Report Issues or Request Enhancements

Just click on the "Contact" link inside the top left box. In case of issue, I am glad to track down what went wrong and get sarmon fixed ASAP.

Up-Coming Enhancements

RRD Support:

• TOP graph
• NET: summary of all interfaces (r & w kb/s)
• System summary: CPU %busy and summary disk IO /s
• IO Summary: summary disk IO r+w kb/s and summary disk IO /s

Better handling of SIGTSTP (ctrl-Z) signal

Removal of some Solaris private APIs in the code (zone_get_id in prtable.c, getvmusage in prstat.c, di_dim_fini, di_dim_init in dsr.c, di_dim_path_dev, di_dim_path_devices, di_lookup_node in dsr.c)

Version History

ld.so.1: sadc: fatal: relocation error: file sadc: symbol enable_extended_FILE_stdio: referenced symbol not found