General Download Contact Order Login

Sampler information & download

The Muldex sampler is a type of program frequently referred to as an agent or collection daemon, which runs on the hosts you wish to monitor.

At this time, we offer a sampler for modern GNU/Linux x86-based systems only. If you have a system you would really like Muldex to sample from, tell us about it.

The sampler is a modular extensible application. All of the sampling capabilities come from dynamically loaded modules which are described below. Much time has been spent carefully designing and implementing these modules, the protocol, and sampler application to make it as robust, resilient, efficient and unobtrusive as possible.

All modules, the sampler program, and supporting libraries (codec and protocol layers) have been implemented in the C programming language due to the excellent speed, efficiency, and control this language provides.

Modules included

cpu.so
CPU utilization. Samples User, Nice, System, SoftIRQ, IRQ, and IOBusy proportions of CPU usage on a per-CPU basis.
diskio.so
Disk activity. Samples WriteWait, ReadWait, IOBusy, Sectors read, Sectors written, Reads issued, Reads merged, Writes completed, Writes merged, and IO's in progress on a per-disk basis. Also samples Sectors read, Sectors written, Reads issued, and Writes completed on a per-partition basis.

*** Supports "hot-plugging" of disks and partitions, no sampler restart or reconfiguration is needed when adding or removing disks or partitions.

*** Filters disks which though visible at the operating system have not experienced any IO since boot. This is done to conserve states in your Muldex service, it would be wasteful to monitor 16 ramdisks which are never used for example. If any activity occurs on filtered disks they will immediately be unfiltered.

files.so
File resource statistics. Samples Files allocated and free quantities.
inodes.so
INode resource statistics. Samples INodes allocated and free quantities.
kevents.so
Significant kernel events. Samples the following rates: Interrupts/sec, Context switches/sec, and Forks/sec.
loadavg.so
Load average. Samples the 5, 10, and 15 minute load averages.
mem.so
Memory usage. Samples how much memory is being used for which purposes system-wide. Some of the more interesting fields include: Total KiB, Free KiB, Buffered KiB, Cached KiB, SwapCached KiB, Active KiB, Inactive KiB, LowFree KiB, HighFree KiB, Dirty KiB, WriteBack KiB, PageTables KiB, and VmallocUsed KiB.
netif.so
Network interface activity. Samples receive and transmit rates of bytes, packets, errors, drops, and FIFO errors, receive frame errors, and more on a per-interface basis.

*** Supports "hot-plugging" of interfaces, no sampler restart or reconfiguration is needed when adding or removing interfaces.

*** Filters interfaces which though visible at operating system have not experienced any activity since boot. This is done to conserve states in your Muldex service, it would be wasteful to monitor extra idle interfaces. If any activity occurs on filtered interfaces they will immediately be unfiltered.

vm.so
Virtual memory statistics. Another memory-related sampler, with almost no overlap with the mem.so module. This module samples swapping in & out rates, paging in & out rates, memory distribution within the VM subsystem, and kernel-accounted VM event rates, including NUMA-related events on NUMA systems.

Installation & configuration

General host-side configuration

The sampler uses two configuration files, one which stores the subject ID of the host you are running the sampler on, the other stores the credentials for the sampler to use when authenticating with the Muldex service.

By default, the sampler looks in the following locations for these two files: 

   /etc/muldex/id.conf
   /etc/muldex/sampler.conf

You may change the configuration directory from /etc/muldex by setting MULDEX_CONF_DIR to your preferred configuration directory in the sampler startup script sampler.sh.

The id.conf file should contain just one line: 

   SUBJECTID=id_number
The sampler.conf file should contain just two lines: 
   USER=muldex_user
   PASSWORD=muldex_password
Muldex subject ID's are allocated and labelled, users created, their passwords set, and access control lists defined via the Muldex web interface.

Once you have the two configuration files created properly, the sampler simply needs to be untarred and the sampler.sh script executed.

Additional tips

It is not recommended that you run the sampler as root but instead as either a regular user created for this purpose or some other limited user account which can access data from /proc.

Running as root enables the sampler to use real-time scheduling and locked memory pages which can help minimize the deviation of the sample rate and prevents the sampler from being paged out and unable to run during periods of high memory pressure or other heavily loaded situations. This is however something you can achieve through resource limits on a regular user account just as well without the risk of another root process.

It is also advised that you restrict the sampler.conf file to only the sampler process (at least chmod o-rwx sampler.conf), as it contains the information needed to connect to your Muldex service.

Integrating the sampler startup into the boot process can be as simple as adding a call to sampler.sh to rc.local, or similar init script. Common usage has included wrapping it with 'su' to switch to a regular user before invoking sampler.sh, and using 'nohup' for backgrounding the process.

Disabling specific modules

Inside the sampler tree you will find the directory: 
    muldex-sampler-linux-2.0-i386/lib/samplers/enabled
Within this directory there are symbolic links which refer to the sampler modules they enable. By default, we distribute the sampler with all modules enabled so this directory is fully populated. Disabling a module is as simple as removing the symbolic link before starting the sampler. Enabling a module consists of restoring the symbolic link before starting as well.

Here is an example of the default everything-enabled directory contents: 

    lrwxrwxrwx 1 root root 25 Oct 18 17:46 cpu.so -> ../available/linux/cpu.so
    lrwxrwxrwx 1 root root 28 Oct 18 17:46 diskio.so -> ../available/linux/diskio.so
    lrwxrwxrwx 1 root root 27 Oct 18 17:46 files.so -> ../available/linux/files.so
    lrwxrwxrwx 1 root root 28 Oct 18 17:46 inodes.so -> ../available/linux/inodes.so
    lrwxrwxrwx 1 root root 29 Oct 18 17:46 loadavg.so -> ../available/linux/loadavg.so
    lrwxrwxrwx 1 root root 25 Oct 18 17:46 mem.so -> ../available/linux/mem.so
    lrwxrwxrwx 1 root root 29 Oct 18 17:46 kevents.so -> ../available/linux/kevents.so
    lrwxrwxrwx 1 root root 27 Oct 18 17:46 netif.so -> ../available/linux/netif.so
    lrwxrwxrwx 1 root root 24 Oct 18 17:46 vm.so -> ../available/linux/vm.so
For example if you only wanted to sample CPU usage, delete all the symbolic links except cpu.so, then start the sampler.

Determining and reducing state consumption

Integrated into the sampler is a state consumption meter which can be used before you have a Muldex account. The purpose of this is to assist in determining which modules you want enabled and how many states your servers will consume with those modules enabled. A necessary step for determining your particular state capacity requirements.

To run the sampler in meter mode, simply run the meter.sh script inside the sampler tree. This causes the sampler to print the states as they would be consumed and what they were consumed by, along with summary counts. You may disable & enable modules as mentioned above and re-run the meter to see how the changes effect the consumption.

Download

Muldex GNU/Linux 2.6+ sampler version 2.2 for i386 - See the included README file for more information
© 2008 Xedlum Inc.