Zenoss Developer's Guide Developers 08 102010 3.0 V01

Zenoss_Developers_Guide_08-102010-3.0-v01

Zenoss_Developers_Guide_08-102010-3.0-v01

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 146 [warning: Documents this large are best viewed by clicking the View PDF Link!]

Zenoss Developer's Guide
Copyright © 2010 Zenoss, Inc., 275 West St. Suite 204, Annapolis, MD 21401, U.S.A. All rights reserved.
This work is licensed under a Creative Commons Attribution Share Alike 3.0 License. To view a copy of this license, visit http://
creativecommons.org/licenses/by-sa/3.0/; or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105,
USA.
The Zenoss logo is a registered trademark of Zenoss, Inc. Zenoss and Open Enterprise Management are trademarks of Zenoss, Inc. in the
U.S. and other countries.
Flash is a registered trademark of Adobe Systems Incorporated.
Java is a registered trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Linux is a registered trademark of Linus Torvalds.
Oracle and the Oracle logo are registered trademarks of the Oracle Corporation.
SNMP Informant is a trademark of Garth K. Williams (Informant Systems, Inc.).
Sybase is a registered trademark of Sybase, Inc.
Tomcat is a trademark of the Apache Software Foundation.
Windows is a registered trademark of Microsoft Corporation in the United States and other countries.
All other companies and products mentioned are trademarks and property of their respective owners.
Part Number: 08-102010-3.0-v01
iii
1. Introduction ............................................................................................................................................. 1
1.1. Overview ...................................................................................................................................... 1
1.1.1. Key Tenets ........................................................................................................................ 1
1.2. Architecture and Technologies ....................................................................................................... 2
1.2.1. User Layer ......................................................................................................................... 3
1.2.2. Data Layer ......................................................................................................................... 3
1.2.3. Process Layer .................................................................................................................... 4
1.2.4. Collection Layer ................................................................................................................. 4
2. Getting Started ........................................................................................................................................ 5
2.1. Working with the Source Code ...................................................................................................... 5
2.1.1. Getting the Source Code .................................................................................................... 5
2.1.1.1. Getting Subversion for the Appliance ........................................................................ 5
2.1.2. Keeping Code Updated ...................................................................................................... 6
2.1.3. Getting Patches ................................................................................................................. 6
2.1.4. Style Guidelines ................................................................................................................. 6
2.1.4.1. Docstrings ............................................................................................................... 6
2.1.5. Generating Diffs for new Fixes ............................................................................................ 8
2.1.6. Submitting a Fix ................................................................................................................. 8
2.2. Development Toolchain Requirements ........................................................................................... 8
2.2.1. Appliance ........................................................................................................................... 8
2.3. Programming Techniques ............................................................................................................ 10
2.3.1. Calling Methods Using REST ............................................................................................ 10
2.3.1.1. How to Call Methods Using REST .......................................................................... 10
2.3.1.2. Sending an Event .................................................................................................. 11
2.3.2. Miscellaneous Notes ......................................................................................................... 13
2.3.2.1. pkg_resources ....................................................................................................... 13
2.3.2.2. urllib2 Workarounds ............................................................................................ 13
2.4. zendmd: Command-line Access to the Device Management Database (DMD) ................................. 14
2.5. Programming Documentation ....................................................................................................... 16
2.5.1. Python ............................................................................................................................. 16
2.5.2. Zenoss API ...................................................................................................................... 16
2.5.3. Other Resources .............................................................................................................. 16
2.5.4. Contributing to the Documentation .................................................................................... 16
3. ZenPacks .............................................................................................................................................. 17
3.1. Overview .................................................................................................................................... 17
3.2. Creating a ZenPack .................................................................................................................... 17
3.2.1. ZenPack Names ............................................................................................................... 17
3.2.2. Specifying Dependencies .................................................................................................. 18
3.2.3. Locating ZenPack Source Outside of Zenoss ..................................................................... 18
3.2.4. Community ZenPack Subversion Access ........................................................................... 18
3.3. ZenPack Structure and Contents ................................................................................................. 18
3.4. Developing the ZenPack ............................................................................................................. 21
3.4.1. Base ZenPack Class ........................................................................................................ 21
3.4.2. Storing Objects in the ZODB ............................................................................................. 21
3.4.3. Providing DataSource classes ........................................................................................... 21
3.4.4. Monitoring Template Checklist .......................................................................................... 22
3.4.4.1. Data Sources ........................................................................................................ 22
3.4.4.2. Data Points ........................................................................................................... 22
3.4.4.3. Thresholds ............................................................................................................ 23
3.4.4.4. Graph Definitions ................................................................................................... 23
3.4.4.5. Graph Points ......................................................................................................... 23
3.4.5. Providing Performance Collector Plugins ........................................................................... 23
3.4.6. Referencing Collector Plugins in ZenPacks ........................................................................ 23
3.4.7. Providing Daemons .......................................................................................................... 24
Zenoss Developer's Guide
iv
3.4.8. setuptools and the zenpacksupport ................................................................................... 24
3.5. Building and Distributing ZenPacks .............................................................................................. 24
3.5.1. Migrating between versions ............................................................................................... 25
3.5.2. Converting older ZenPacks to ZenPack eggs ..................................................................... 25
3.6. Development Mode ..................................................................................................................... 25
3.6.1. Source ZenPacks ............................................................................................................. 25
3.6.2. Converting .egg Files to Development Mode ...................................................................... 25
3.7. Where to Get More Information ................................................................................................... 26
4. Zenoss Data Stores ............................................................................................................................... 27
4.1. Zope Object Database (ZODB) .................................................................................................... 27
4.2. MySQL Event database ............................................................................................................... 29
4.2.1. Connecting to the Database .............................................................................................. 29
4.2.2. MySQL in 60 Seconds ...................................................................................................... 29
4.3. Python Pickle Files ...................................................................................................................... 30
4.4. Round-Robin Database ............................................................................................................... 31
5. Events ................................................................................................................................................... 33
5.1. Understanding an Event Entry ..................................................................................................... 33
5.1.1. Event Design ................................................................................................................... 33
5.2. Sending an Event ....................................................................................................................... 33
5.3. Adding an Event Class ................................................................................................................ 34
5.3.1. Add to ZenEventClasses .................................................................................................. 34
5.3.2. Add the class to the import XML ....................................................................................... 34
5.3.3. Write a migrate script ....................................................................................................... 35
6. Configuration Property Management ....................................................................................................... 36
6.1. Adding a Configuration Property .................................................................................................. 36
6.1.1. Adding a Configuration Property to an Event ..................................................................... 36
6.1.2. Adding a Configuration Property to a Device ...................................................................... 36
6.2. Migrating the Configuration Property Code ................................................................................... 36
7. Creating New Jobs ................................................................................................................................ 38
7.1. Job Requirements ....................................................................................................................... 38
7.2. Running a Job ............................................................................................................................ 38
7.3. Life Cycle of a Job ...................................................................................................................... 38
7.4. Shell Command Jobs .................................................................................................................. 39
7.5. Logging ...................................................................................................................................... 39
8. Device Management .............................................................................................................................. 40
8.1. Adding Devices Programatically ................................................................................................... 40
8.1.1. Using a REST call ............................................................................................................ 40
8.1.2. Example: Using an XML-RPC Call from Python ................................................................. 40
8.1.3. XML-RPC Attributes ......................................................................................................... 40
8.2. Editing Device Information ........................................................................................................... 41
8.2.1. Using a REST call ............................................................................................................ 41
8.2.2. Using an XML-RPC Call from Python ................................................................................ 41
8.3. Deleting A Device ....................................................................................................................... 42
8.3.1. Using a REST call ............................................................................................................ 42
8.3.2. Using an XML-RPC Call from Python ................................................................................ 42
8.4. Checking If A Device Exists ........................................................................................................ 42
8.4.1. Using a REST call ............................................................................................................ 42
8.4.2. Using an XML-RPC Call from Python ................................................................................ 42
8.5. Exporting a Device List ............................................................................................................... 43
9. Extending the Model .............................................................................................................................. 44
9.1. Add a ZenModel Relationship ...................................................................................................... 44
9.1.1. One-to-One (1:1) Relationships ......................................................................................... 44
9.2. One-to-Many (1:N) Relationships ................................................................................................. 45
9.3. Many-to-Many (M:N) Relationships ............................................................................................... 46
Zenoss Developer's Guide
v
9.3.1. One-to-Many (1:N) Container Relationships ....................................................................... 47
9.4. Zenoss XML Schema .................................................................................................................. 48
9.4.1. object ............................................................................................................................... 50
9.4.1.1. Example ................................................................................................................ 50
9.4.1.2. Attributes ............................................................................................................... 50
9.4.1.3. Children ................................................................................................................ 50
9.4.2. objects ............................................................................................................................. 51
9.4.2.1. Example ................................................................................................................ 51
9.4.2.2. Children ................................................................................................................ 51
9.4.3. property ........................................................................................................................... 51
9.4.3.1. Example ................................................................................................................ 51
9.4.3.2. Attributes ............................................................................................................... 52
9.4.4. tomany ............................................................................................................................. 52
9.4.4.1. Example ................................................................................................................ 52
9.4.4.2. Attributes ............................................................................................................... 52
9.4.4.3. Children ................................................................................................................ 52
9.4.5. tomanycont ...................................................................................................................... 52
9.4.5.1. Example ................................................................................................................ 52
9.4.5.2. Attributes ............................................................................................................... 53
9.4.5.3. Children ................................................................................................................ 53
9.4.6. toone ............................................................................................................................... 53
9.4.6.1. Example ................................................................................................................ 53
9.4.6.2. Attributes ............................................................................................................... 53
9.4.7. link .................................................................................................................................. 53
9.4.7.1. Example ................................................................................................................ 53
9.4.7.2. Attributes ............................................................................................................... 53
9.5. Zenoss Permissions .................................................................................................................... 53
9.5.1. Adding New Permissions .................................................................................................. 54
9.5.2. Assigning Permissions to a Method ................................................................................... 54
9.5.3. Checking Links ................................................................................................................. 54
10. Zenoss Daemons ................................................................................................................................. 55
10.1. Twisted Network Programming Overview .................................................................................... 55
10.1.1. Understanding NJobs, Driver and DeferredList ................................................................. 55
10.1.1.1. DeferredList ......................................................................................................... 55
10.1.1.2. NJobs ................................................................................................................. 56
10.1.1.3. Driver .................................................................................................................. 56
10.1.1.4. A Simple Example ............................................................................................... 57
10.2. Zenoss Daemon Overview ......................................................................................................... 59
10.3. zenhub: Daemon to ZODB management .................................................................................... 60
10.3.1. Daemon to ZODB management ...................................................................................... 60
10.3.2. Heartbeats and other Events ........................................................................................... 61
10.3.3. Pluggable Daemon Services ........................................................................................... 61
10.4. ZenRender and Graphs ............................................................................................................. 61
10.5. Developing a Daemon ............................................................................................................... 62
10.5.1. Command-line Options ................................................................................................... 62
10.5.2. Add the Daemon Control Script ....................................................................................... 62
10.5.3. Set Up ZenHub Communications .................................................................................... 63
10.5.3.1. Registering Services with the Hub ........................................................................ 63
11. Add a Performance Daemon ................................................................................................................ 64
11.1. Overview ................................................................................................................................... 64
11.2. DataMaps ................................................................................................................................. 64
11.3. Performance Collection .............................................................................................................. 66
11.3.1. Connecting Collectors and Services ................................................................................ 66
11.4. Creating a New Collector ........................................................................................................... 66
Zenoss Developer's Guide
vi
11.4.1. Constructor .................................................................................................................... 66
11.4.2. Getting a List of Devices ................................................................................................. 67
11.4.2.1. Thresholds .......................................................................................................... 68
11.4.3. fetchConfig() ................................................................................................................ 69
11.4.4. Collector's ZenHub Service ............................................................................................. 70
11.4.5. Miscellaneous Functions ................................................................................................. 70
11.4.6. Collect the Performance Data ......................................................................................... 71
12. Adding a Device Type .......................................................................................................................... 74
12.1. Overview ................................................................................................................................... 74
12.2. Add the MIB ............................................................................................................................. 74
12.3. Add a Device Organizer ............................................................................................................ 74
12.4. Create a Modeler ...................................................................................................................... 75
12.4.1. Verify the SNMP connectivity and OIDs ........................................................................... 76
12.4.2. Common SNMP Issues ................................................................................................... 76
12.4.3. Modeler Code ................................................................................................................ 76
12.4.4. Testing the Modeler ........................................................................................................ 79
12.5. Create a Performance Collector ................................................................................................. 79
12.5.1. Performance Data Collector Code ................................................................................... 79
12.5.2. Writing Your Own Command Parser ................................................................................ 80
12.6. Create the Template ................................................................................................................. 82
12.6.1. Create the DataSource ................................................................................................... 82
12.6.2. Create a Threshold ......................................................................................................... 82
12.6.3. Create a Graph .............................................................................................................. 82
12.7. Map Events .............................................................................................................................. 82
12.8. Adding SSH Monitoring Tests .................................................................................................... 83
12.8.1. Overview ........................................................................................................................ 83
12.8.2. Modeling Plugin Test Data .............................................................................................. 83
12.8.2.1. Test Data for an ObjectMap .................................................................................. 83
12.8.2.2. Test Data for a RelationshipMap .......................................................................... 84
12.8.2.3. Test Data for a List of Data Maps ......................................................................... 84
12.8.3. Data Point Parser Test Data ........................................................................................... 84
12.8.3.1. Test Data for Device-Level Parsers ....................................................................... 84
12.8.3.2. Test Data for Component Parsers ......................................................................... 85
12.8.4. Running the Tests .......................................................................................................... 85
13. Extending the User Interface ................................................................................................................ 86
13.1. About Zenoss UI Technologies .................................................................................................. 86
13.1.1. HyperText Markup Language (HTML) .............................................................................. 86
13.1.2. Cascading Style Sheets (CSS) ........................................................................................ 86
13.1.3. Zope 2, ZPT and TAL ..................................................................................................... 86
13.1.4. ZPT and Macro Expansion for TAL (METAL) ................................................................... 87
13.1.5. JavaScript / AJAX ........................................................................................................... 87
13.1.6. JavaScript Library: Ext JS ............................................................................................... 87
13.2. Customizing the Navigation Bar ................................................................................................. 87
13.2.1. Example: Simple HTML Page ......................................................................................... 88
13.2.2. Example: Simple TAL and METAL Page .......................................................................... 88
13.3. Customizing the Logo ................................................................................................................ 89
13.4. Zope 2 Page Templates, TAL and METAL and Zenoss ............................................................... 89
13.4.1. Tips ............................................................................................................................... 91
13.5. Zope 3 Views Explained ............................................................................................................ 92
13.5.1. The Zope 2 Way ............................................................................................................ 92
13.5.2. The Zope 3 Way ............................................................................................................ 93
13.6. Other Customizations ................................................................................................................ 95
13.6.1. Adding Tabs ................................................................................................................... 95
13.6.2. Adding a Dialog .............................................................................................................. 97
Zenoss Developer's Guide
vii
13.6.3. Adding a New Menu or Menu Item .................................................................................. 99
13.6.4. Creating a Table Using ZenTableManager ..................................................................... 100
13.6.5. Creating an Editable Table ............................................................................................ 102
13.6.6. How to Save Properties via an Edit Screen .................................................................... 102
13.7. Creating a Dashboard Portlet ................................................................................................... 104
13.7.1. Create a ZenPack ........................................................................................................ 104
13.7.2. Write the Python Back-End Code .................................................................................. 105
13.7.3. Write the JavaScript Portlet ........................................................................................... 106
13.7.4. Register the Portlet ....................................................................................................... 111
13.8. Debugging Tips ....................................................................................................................... 112
14. ZenPack Conversion Tasks for Version 3.0 ......................................................................................... 113
14.1. About ZenPack Conversion ...................................................................................................... 113
14.1.1. What Has Changed? .................................................................................................... 113
14.1.1.1. Redesigned Pages ............................................................................................. 113
14.1.2. Updating Page Templates ............................................................................................. 114
14.1.3. Updating Page-Level Dialogs ........................................................................................ 114
14.1.4. Updating Data Sources ................................................................................................. 115
14.1.4.1. Modify the MANIFEST.in File .............................................................................. 115
14.1.4.2. Create an Interface for the Data Source .............................................................. 115
14.1.4.3. Create an info Object ......................................................................................... 116
14.1.4.4. Write an Adapter ................................................................................................ 117
14.1.5. Updating Thresholds ..................................................................................................... 117
14.1.6. Custom "Add Device" Widget ........................................................................................ 117
14.1.7. Custom Columns on the Component Grid (Device Summary) .......................................... 118
15. Reports .............................................................................................................................................. 120
15.1. Adding a Report ...................................................................................................................... 120
15.2. Plugins .................................................................................................................................... 121
15.3. Adding Export Buttons to Reports ............................................................................................ 121
16. Migrating Zenoss Code ...................................................................................................................... 123
16.1. Introduction and Steps ............................................................................................................. 123
16.2. How It Works .......................................................................................................................... 123
16.3. What You Write ....................................................................................................................... 123
16.3.1. Implement cutover() ...................................................................................................... 124
16.3.2. Supporting Code ........................................................................................................... 124
16.3.3. Testing and Deployment ............................................................................................... 124
17. Testing .............................................................................................................................................. 125
17.1. Zenoss Unit Tests ................................................................................................................... 125
17.1.1. Introduction .................................................................................................................. 125
17.1.2. doctest Testing ............................................................................................................. 125
17.1.3. Zenoss' Test Runner ..................................................................................................... 126
17.1.3.1. An Example Unit Test ........................................................................................ 127
17.1.4. Integrating With Buildbot ............................................................................................... 130
17.1.5. JavaScript Test Framework ........................................................................................... 130
17.2. Functional User Interface Testing ............................................................................................. 130
17.2.1. Introduction .................................................................................................................. 130
17.2.2. Installing and Running .................................................................................................. 130
17.2.2.1. Installing and Configuring Mac OS X ................................................................... 130
17.3. Where to Get More Information ................................................................................................ 131
A. Event Database Dictionary ................................................................................................................... 132
B. TALES Expressions ............................................................................................................................. 133
B.1. Examples ................................................................................................................................. 133
B.1.1. ping ............................................................................................................................... 133
B.1.2. DNS forward lookup ....................................................................................................... 133
B.1.3. DNS reverse lookup ....................................................................................................... 133
Zenoss Developer's Guide
viii
B.1.4. snmpwalk ...................................................................................................................... 133
B.2. TALES Device Attributes ........................................................................................................... 133
B.3. TALES Event Attributes ............................................................................................................. 134
Glossary .................................................................................................................................................. 136
1
Chapter 1. Introduction
1.1. Overview
The Zenoss system provides full stack coverage of networks, servers, applications, services, and virtualization.
Functionally, it provides complete operational awareness by combining discover and inventory, availability and per-
formance monitoring, event management, and reporting.
At its highest level, the system comprises these major areas:
• Discovery and configuration
• Performance and availability
• Fault and event management
• Alerting and remediation
• Reporting
Zenoss unifies these areas into a single system with a modern, interactive Web user interface.
Figure 1.1. High-Level view
1.1.1. Key Tenets
Zenoss was designed with these important ideas at its core:
•Modeling
The system's model enables it to understand the environment in which it operates. Through sophisticated and
detailed analysis, Zenoss determines how to monitor and manage complex IT environments. The core of the
Introduction
2
standard model describes basic information about each device's operating system and hardware. The model is
object-based, and is easily extended through object inheritance.
•Discovery
With a sophisticated model, manual input and maintenance of data is challenging. To address this challenge,
Zenoss uses discovery to populate the model. During discovery, the system accesses each monitored device in
your infrastructure and interrogates it in detail, acquiring information about its components, network integration,
and dependencies.
•Normalization
Because Zenoss collects information from different platforms and through different protocols, the amount and
format of available information varies. For example, file system information gathered from a Linux server differs
from similar information gathered from a Windows server. Zenoss standardizes the data gathered so that you
can perform valid comparisons of metrics gathered by different methods and for different systems.
•Agentless Data Collection
To gather information, Zenoss relies on agent-less data collection. By communicating with a device through one
of several protocols (including SNMP, SSH, Telnet, and WMI), it minimizes the impact on monitored systems.
•Full IT Infrastructure
Unlike other tools, the system's inclusive approach unifies all areas of the IT infrastructure--network, servers,
and applications--to eliminate your need to access multiple tools.
•Configuration Inheritance
Zenoss extends the concept of inheritance in object-oriented languages to configuration. All core configuration
parameters (configuration properties) and monitoring directions (monitoring templates) use inheritance to de-
scribe how a device should be monitored. Inheritance allows you to describe, at a high level, how devices should
be monitored. It also supports ongoing refinements to the configuration. (For detailed information on inheritance
and templates, refer to the chapter titled "Properties and Templates.")
•Cross-Platform Monitoring
Zenoss monitors the performance and availability of heterogeneous operating systems (including Windows,
Linux, and Unix), SNMP-enabled network devices (such as Cisco), and a variety of software applications (such
as WebLogic and VMware).
•Scale
You can deploy the system on a single server to manage hundreds of devices. The Enterprise version allows
you to manage large, distributed systems by using horizontal scaling of its collectors.
•Extensibility
The system's extension mechanism, ZenPacks, allow for rapid addition and modification to customize your
environment.
1.2. Architecture and Technologies
The following diagram illustrates the system architecture.
Introduction
3
Figure 1.2. Architecture
Zenoss is a tiered system with four major parts:
• User layer
• Data layer
• Processing layer
• Collection layer
1.2.1. User Layer
Built around the Zope Web application environment, the user layer is manifested as a Web portal. It uses several
JavaScript libraries, Mochi Kit, YUI, and extJS to provide a rich application experience.
Through the user interface, you access and manage key components and features. From here, you can:
• Watch the status of your enterprise, using the Dashboard
• Work with devices, networks, and systems
• Monitor and respond to events
• Manage users
• Create and run reports
The user layer Interacts with the data layer and translates the information for display in the user interface.
1.2.2. Data Layer
Configuration and collection information is stored in the data layer, in three separate databases:
•ZenRRD - Utilizing RRDtool, stores time-series performance data. Because RRD files are stored locally to each
collector, no bottlenecks result from writing to a single database as new collectors are added.
•ZenModel - Serves as the core configuration model, which comprises devices, their components, groups, and
locations. It holds device data in the ZEO back-end object database.
•ZenEvents - Stores event data in a MySQL database.
Introduction
4
1.2.3. Process Layer
The process layer manages communications between the collection and data layers. It also runs back-end, periodic
jobs, as well as jobs initiated by the user (ZenActions and ZenJobs).The process layer utilizes Twisted PB (a bi-
directional RPC system) for communications.
1.2.4. Collection Layer
The collection layer comprises services that collect and feed data to the data layer. These services are provided by
numerous daemons that perform modeling, monitoring, and event management functions.
The modeling system uses SNMP, SSH, and WMI to collect information from remote machines. The raw information
is fed into a plugin system (modeling plugins) that normalizes the data into a format that matches the core model.
Monitoring daemons track the availability and performance of the IT infrastructure. Using multiple protocols, they
store performance information locally in RRD files, thus allowing the collectors to be spread out among many col-
lector machines. Status and availability information, such as ping failures and threshold breaches, are returned
through ZenHub to the event system.
For more information about system daemons, see the appendix in the Zenoss Administration guide titled "Daemon
Commands and Options."
5
Chapter 2. Getting Started
2.1. Working with the Source Code
2.1.1. Getting the Source Code
If all that you would like to do is browse through the source code, then go to the Trac/Subversion page at:
http://dev.zenoss.com/trac/browser
The version control system used by Zenoss is Subversion. Subversion has excellent documentation in the form of
an O'Reilly book. For the moment, we will just provide the minimum number of commands to get started.
The absolute latest version of Zenoss can be accessed directly through the Subversion repository. This code should
not be used for production purposes as there are changes actively being made which may not have been thoroughly
tested.
From a command line prompt, go to a directory where you want the source code delivered. Here's a sample com-
mand to get the source code:
$ svn co http://dev.zenoss.org/svn/trunk/Products
This will create a directory called Products in the current directory and check out the source code. This repository
is readable anonymously, so no credentials are required.
To see which other portions of the code are available, such as ZenPacks or support utilities, you can look by using
the following Subversion command:
$ svn ls http://dev.zenoss.org/svn/trunk
Other tools are available that can be used to view or check out the source code for different platforms. See the
Subversion Web site for more details.
2.1.1.1. Getting Subversion for the Appliance
The rPath appliance does not ship with the svn binaries, but you can still obtain them.
Procedure 2.1. Installing Subversion on Appliances
1. Edit the /etc/conaryrc file:
• For the Community version, look for the line that looks like this:
installLabelPath zenoss-project.zenoss.loc@zenoss:core-2.3
Change the above line to this (note that this should be all one line and has been modified to make it look
better in print):
installLabelPath zenoss-project.zenoss.loc@zenoss:core-2.3
conary.rpath.com@rpl:1
• For the Enterprise version, look for the line that looks like this:
installLabelPath zenoss-project.zenoss.loc@zenoss:enterprise-2.3
Change the above line to this (note that this should be all one line and has been modified to make it look
better in print):
installLabelPath zenoss-project.zenoss.loc@zenoss:enterprise-2.3
conary.rpath.com@rpl:1
Getting Started
6
2. Now you should be able to obtain the subversion package by using the conary update command:
[root@localhost ~] conary update --resolve subversion
For more information about rPath commands, see their documentation wiki. There are also a set of blog entries
Conary Uncorked has been put together by a dedicated rPath user that introduces some of the conary commands
much more gently.
2.1.2. Keeping Code Updated
The following command, issued from the base directory where you checked out the Zenoss code, will update all
code from that directory and all subdirectories and bring it up to date with what is current in the Subversion repository
(and therefore apply all of the current patches to the code you checked out previously):
$ svn update
If you have modified any code in this directory, these changes will be merged with the latest code updates. If there
are differences that Subversion cannot automatically resolve, Subversion will tell you that there is a problem by
showing the updated file is in conflict (for example, showing a 'C' beside the file when you run svn status).
You can tell if you have modified any of the files in the checked-out directory by typing the following:
$ svn status
If you are interested in modifying only one file, you can specify that one file:
$ svn udpate filename
2.1.3. Getting Patches
For issue tracking, bug reports and linking patches to bug reports, Zenoss uses Trac to manage issues. The Zenoss
Trac server is found at this location:
http://dev.zenoss.com/trac/report
You can click Search at the top right of the page and enter a search term to look for keywords in the tickets. This
will then present you with the ability to search for changesets (for example, Subversion revisions), trouble tickets,
or the Wiki.
Alternatively, from the start page you can click on the Custom Query which will allow you to view the results from
your customized query.
Once you have found a patch that applies to your system, use the zenpatch command to apply the patch. (As
mentioned previously, if you use the svn update commands, you will already be at the latest patched level.)
$ zenpatch revision_number
2.1.4. Style Guidelines
These following guidelines are targeted at Python files. HTML files, Zope Page Template (ZPT) files, shell scripts,
etc should adhere to these as much as is reasonable and conventional in those languages. Currently, we follow
Guido's Style Guide for Python Code which is detailed in PEP 8 (Python Enhancement Proposals).
Any style conventions that stray from PEP-8 should be annotated in this document.
2.1.4.1. Docstrings
Every method and function definition within Zenoss should include a docstring. The docstring is usually composed
of two parts: the explanatory text and the doctest code. The explanation usually includes a description of all or most
of the following aspects of the function:
Getting Started
7
• The function's purpose
• The context in which the function is usually called
• What parameters it expects
• What it returns
• Any side effects of the function
This explanatory text should scale in size with the complexity and significance of the function.
The second part of the docstring is the doctest section. This is composed of zendmd commands and expected
output from those commands. The commands are run as part of the testing process and output is compared to the
output lines. This code serves two primary purposes. First it is a working example of how the function should be
called and what it returns. Second it serves as a basic test to ensure the function is not horribly broken. This is not
intended as a replacement for unit tests. Thorough testing of boundary cases and unusual situations still belongs in
unit tests whereas the doctests are much simpler and more instructional in nature.
Docstrings begin on the line immediately following the function definition and are indented one level from the def-
inition. The first and last lines of the docstring are three double quotes and a newline. One blank line separates
the description from the epydoc section. epydoc can take specially formatted text in the docstrings and use them
to create API documentation. The Zenoss API documentation is located on the Zenoss Web site and is updated
every release.
Another blank line separates the epydoc section from the doctest section. The code for the function begins on the
line immediately following the docstring. For example:
def TruncateStrings(longStrings, maxLength):
"""
Foo truncates all the strings in a list to a maximum length.
longStrings is any iterable object which returns zero or more
strings. maxLength is the length to which each element from
longStrings should be truncated.
@param longStrings: an iterable object which returns zero or more strings
@type longStrings: Python iterable
@param maxLength: max length of each element in longStrings
@type maxLength: int
@return: longStrings in the same order but possibly truncated
@rtype: list
@todo: Add more epydoc attributes!
>>> from Products.SomeModule import TruncateStrings
>>> TruncateStrings(['abcd', 'efg', 'hi', ''], 3)
['abc', 'efg', 'hi', '']
>>> TruncateStrings([], 5)
[]
"""
return [s[:maxLength] for s in longStrings]
The easiest way to create the doctest portion is from within zendmd. Except for the indentation, the docstring should
exactly match commands and output from a zendmd session.
Use the available epydoc fields where they are applicable. Some of the useful common fields are:
Commonly-used epydoc fields
@param param_name Describe the parameter
@type data_type Data type of the parameter
@return Describe the return value
Getting Started
8
@rtype Data type of the return value
@permission Zope permission that the method requires
@todo Todo for this method
Note
Within the description section of the docstring, you may use the string DEPRECATED on its own line to denote that
the method is deprecated.
2.1.5. Generating Diffs for new Fixes
Once you have determined how to fix something, or have found a way to add a feature, modify the source code in
your checkout directory. Once that is complete, generate a diff starting from the base of the checkout directory.
To generate a diff of all files in the current directory and all subdirectories:
$ svn diff > mychanges.diff
To produce a diff for just a single file:
$ svn diff source_file > mychanges.diff
2.1.6. Submitting a Fix
Zenoss accepts user contributions using the following procedure:
1. Complete the form to allow Zenoss to accept your code.
2. Create a ticket in our ticketing system.
3. Add the keyword contribute to the ticket.
4. Attach your patch (in diff format) or code to the ticket.
Note
All contributions will be accepted under the terms of the Zenoss Contribution Agreement.
2.2. Development Toolchain Requirements
There are a number of other tools that are required to build Zenoss from source (a toolchain). Among them are a
C compiler, the make command, and other associated tools.
2.2.1. Appliance
The Zenoss appliance is based on the rPath Linux 1 (rp11) distribution.
Troves (like the gcc toolchain) that are not available on the Zenoss update repository server are generally available
from install labels, such as:
conary.rpath.com@rpl:1
The trove candy store is rBuilder Online. Zenoss recommends that you obtain an account there. It provides good
search capabilities for packages of interest, and offers forums to assist with appliance-specific questions.
For a gcc toolchain, try this as the root user:
# conary update --resolve autoconf automake make which \
Getting Started
9
--install-label="conary.rpath.com@rpl:1"
# conary update --resolve gcc=conary.rpath.com@rpl:1 \
--install-label="conary.rpath.com@rpl:devel"
The binutils trove should already be on the box.
An actual install sequence looked like the ouput below. If the --info switch is used, it is possible to see if everything
is going to resolve nicely. And if you are really paranoid, use the --test flag which runs through the update but does
not commit the result.
# conary update autoconf automake make which --resolve --info \
--install-label="conary.rpath.com@rpl:1"
Install autoconf(:data :doc :runtime)=2.59-7-0.1
Install automake(:data :doc :runtime)=1.9.6-3-0.1
Install m4(:runtime)=1.4.3-4-0.1
Install make(:doc :locale :runtime)=3.80-7.2-1
Install which(:doc :runtime)=2.16-3-0.1
# conary update autoconf automake make which --resolve \
--install-label="conary.rpath.com@rpl:1"
Including extra troves to resolve dependencies:
m4:runtime=1.4.3-4-0.1
Applying update job:
Install autoconf(:data :doc :runtime)=2.59-7-0.1
Install automake(:data :doc :runtime)=1.9.6-3-0.1
Install m4(:runtime)=1.4.3-4-0.1
Install make(:doc :locale :runtime)=3.80-7.2-1
Install which(:doc :runtime)=2.16-3-0.1
# conary update --info --resolve gcc=conary.rpath.com@rpl:1 \
--install-label="conary.rpath.com@rpl:devel"
Install gcc(:devel :devellib :doc :lib :locale :runtime)=3.4.4-9.4-1
Install libgcc(:devellib)=4.1.2-11-1[~!gcc.core]
# conary update --resolve gcc=conary.rpath.com@rpl:1 \
--install-label="conary.rpath.com@rpl:devel"
Including extra troves to resolve dependencies:
libgcc:devellib=4.1.2-11-1
Applying update job:
Install gcc(:devel :devellib :doc :lib :locale :runtime)=3.4.4-9.4-1
Install libgcc(:devellib)=4.1.2-11-1[~!gcc.core]
Generally try to find something on the rpl:1 branch name and do not mix rpl:2 stuff with the rpl:1 stuff. In some
cases, you may have to resort to pulling a trove from the rpl:devel branch if it cannot find it elsewhere. That's what
happened above when trying to resolve the libgcc dependency for the gcc trove. Adding the extra --install-label
option was necessary so that libgcc could be found. How could you know it was on rpl:devel? Go to rBuilder Online
and search for that package and it should tell you.
If you want to see where the files for a trove are installed:
# conary q trove_name --lsl
[code]# conary q gcc --lsl
...
lrwxrwxrwx 1 root root 3 2004-07-07 17:04:44 UTC /usr/bin/cc -> gcc
-rwxr-xr-x 1 root root 81452 2006-06-19 18:02:30 UTC /usr/bin/gcc
-rwxr-xr-x 1 root root 16134 2005-10-15 07:22:42 UTC /usr/bin/gccbug
...
Lastly, conary makes it relatively easy to run-away if you're not happy with a trove you've installed. Use conary
rblist to see what packages have been committed to the conary stack.
# conary rblist | more
Getting Started
10
r.3:
installed: gcc(:devel :devellib :doc :lib :locale :runtime)
conary.rpath.com@rpl:1/3.4.4-9.4-1
installed: libgcc(:devellib) conary.rpath.com@rpl:devel/4.1.2-11-1
r.2:
installed: autoconf(:data :doc :runtime) conary.rpath.com@rpl:1/2.59-7-0.1
installed: automake(:data :doc :runtime) conary.rpath.com@rpl:1/1.9.6-3-0.1
installed: m4(:runtime) conary.rpath.com@rpl:1/1.4.3-4-0.1
r.1:
updated: info-raa-web(:user) products.rpath.com@rpath:raa-2/1-1.1-2 ->
1-1.3-2
...
Here is how you would remove the gcc trove that was just installed:
# conary rb r.3
Applying update job:
Erase gcc(:devel :devellib :doc :lib :locale :runtime)=3.4.4-9.4-1
Erase libgcc(:devellib)=4.1.2-11-1[~!gcc.core]
# conary q gcc
gcc was not found
Be careful which troves you remove!
2.3. Programming Techniques
2.3.1. Calling Methods Using REST
REpresentational State Transfer (REST) is a method of marshaling data types and calling functions using HTTP.
Zope supports a number of different Remote Procedure Call (RPC) mechanisms, including REST.
This section describes some more advanced Zenoss concepts that we have encountered as the product has rolled
out. Some may be appropriate for your environment. Usually they require at least a little coding experience, but
they are really not that hard.
2.3.1.1. How to Call Methods Using REST
Zenoss' Web interface will let you run any method of any object by using a simple URL. Calls are in the following
format:
USERNAME:PASSWORD@MY_ZENOSS_HOST:8080/PATH_TO_OBJECT/METHOD_NAME?ARG=VAL
where:
•USERNAME is the user with rights to view this information.
•PASSWORD is the user's password.
•MY_ZENOSS_HOST is the hostname or IP address of your Zenoss instance
•PATH_TO_OBJECT is the full path of the object you want to access
•METHOD_NAME is the object's method you want to run
•ARG is the method's parameter name
•VAL is the method's parameter value
The following example provides the most recent load average of a Linux server:
Getting Started
11
http://USERNAME:PASSWORD@MY_ZENOSS_HOST:8080/zport/dmd/
Devices/Server/Linux/devices/angel/getRRDValue?dsname=laLoadInt5_laLoadInt5
Note these things about this URL:
• /zport/dmd/Devices/Server/Linux/devices/angle is the full path to the object you want to access.
• getRRDValue is the method in the Device object you want to run.
• dsname is a parameter to the getRRDValue method.
• laLoadInt5_laLoadInt5 is the value of dsname, which is the name of the data source we are interested in
Watching the URLs as you browse the Web interface can give you a place to start searching.
2.3.1.2. Sending an Event
Events can be sent to Zenoss through the Web interface as well as through using zensendevent, but also through
a programmatic interface.
2.3.1.2.1. Using a REST Call
Sending an event through a rest call can be done by a simple web get. In this example we will use wget to send an
event. If you use wget don't for get to escape the "&" or wrap the URL in single quotes.
[zenos@zenoss $] wget --auth-no-challenge 'http://admin:zenoss@MYHOST:8080/zport/dmd/ZenEventManag-
er/manage_addEvent?
device=MYDEVICE&component=MYCOMPONENT&summary=MYSUMMARY&severity=4&eventclass=EVENTCLASS'
2.3.1.2.2. Using XML-RPC
To send an event to Zenoss using XML-RPC you will first need to create a dictionary (in Perl a hash) that will
represent the event. Zenoss will need at a minimum the following fields:
Event fields
device the name of the device from which this event originates
component the sub-component of the device (for example, eth0 or http)
summary the text message of the event
severity an integer between 0 and 5 with higher numbers being higher severity. Zero is clear.
You can send an event to Zenoss via an interactive session with the Python interpreter as follows:
>>> from xmlrpclib import ServerProxy
>>> myurl= 'http://admin:zenoss@MYHOST:8080/zport/dmd/ZenEventManager'
>>> serv = ServerProxy( myurl )
>>> evt = {'device':'mydevice', 'component':'eth0',
... 'summary':'eth0 is down','severity':4, 'eventClass':'/Net'}
>>> serv.sendEvent(evt)
See below for examples in other languages.
2.3.1.2.3. Example Usage in Other Languages
Please note that we are a Python shop and may not be able to answer specific questions about XML-RPC clients
written in other languages.
2.3.1.2.3.1. Perl
Send an event via Perl using RPC::XML::Client
Getting Started
12
require RPC::XML;
require RPC::XML::Client;
$serv = RPC::XML::Client->new('http://YOURZENOSS:8081/');
%evt = ('device' => 'mydevice2', 'component' => 'eth1',
'summary' => 'eth1 is down', 'severity' => 4);
$args = RPC::XML::struct->new(%evt);
$serv->simple_request('sendEvent', $args);
2.3.1.2.3.2. Ruby
This is an example of an Interactive Ruby (IRB) session (the returns have been omitted for the sake of clarity). Note,
however, that the Ruby standard library is under active development in general, and specifically, the XML-RPC lib
in Ruby is not stable. As of Feb 2007, there is a great deal of on-going discussion regarding XML-RPC in Ruby by
Ruby developers and contributors. The following is known to work in previous versions of Ruby:
require "xmlrpc/client"
url='user:pass@http://YOURZENOSS:8080/zport/dmd/DeviceLoader')
server = XMLRPC::Client.new2( url )
evt = {'device' => 'mydevice3', 'component' => 'eth2',
'summary' => 'eth2 is down', 'severity' => 4}
server.call('sendEvent', evt)
2.3.1.2.3.3. PHP
<?php
include("xmlrpc.inc");
function ifInOutBps($host, $port, $user, $pass, $device, $interface) {
$ifInOctets = 'ifInOctets_ifInOctets';
$ifOutOctets = 'ifOutOctets_ifOutOctets';
# base url $url = '/zport/dmd/Devices';
# message $msg = new xmlrpcmsg(
$device.'.os.interfaces.'.$interface.'.getRRDValues', array());
$xifInOctets = new xmlrpcVal($ifInOctets);
$xifOutOctets = new xmlrpcVal($ifOutOctets);
$xifOctets = new xmlrpcVal(array($xifInOctets, $xifOutOctets), 'array');
$msg->addParam($xifOctets);
# client $clt = new xmlrpc_client($url, $host, $port);
# $clt->setCredentials($user, $pass);
# get response $rsp = $clt->send($msg);
# any error? if ($rsp->faultCode()) {
die('ifInOutBps - Send error: '.$rsp->faultString().'
'); }
# convert to data structure $dst = xmlrpc_decode($rsp->serialize());
return(array('in'=>$dst[$ifInOctets]*8, 'out'=>$dst[$ifOutOctets]*8));
}
?>
Getting Started
13
2.3.1.2.3.4. Java
This example uses the Apache XML-RPC library and Java 6 to send an event to the Zenoss server.
Required jars on the classpath (all available from the Apache download):
•xmlrpc-client-3.1.jar
•ws-commons-util-1.0.2.jar
•xmlrpc-common-3.1.jar
import java.net.URL;
import java.util.HashMap;
import org.apache.xmlrpc.client.XmlRpcClient;
import org.apache.xmlrpc.client.XmlRpcClientConfigImpl;
public class JavaRPCExample {
public static void main(String[] args) throws Exception {
XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
url= "http://MYHOST:8080/zport/dmd/ZenEventManager"
config.setServerURL(new URL(url));
config.setBasicUserName("admin");
config.setBasicPassword("zenoss");
XmlRpcClient client = new XmlRpcClient();
client.setConfig(config);
HashMap<String,Object> params = new HashMap<String,Object>();
params.put("device", "mydevice");
params.put("component", "eth0");
params.put("summary", "eth0 is down");
params.put("severity", 4);
params.put("eventClass", "/Net");
client.execute("sendEvent", new Object[]{params});
}
}
2.3.2. Miscellaneous Notes
2.3.2.1. pkg_resources
Should one need to use pkg_resources, it would normally be imported like this:
import pkg_resources
To avoid the mysterious warning
_xmlplus UserWarning
use the following import line:
import Products.ZenUtils.PkgResources
2.3.2.2. urllib2 Workarounds
There is a bug in the standard Python urllib2 library that prevents HTTPS requests through a proxy from working.
This affects ZenWebTx and any other Python code that might attempt to make HTTPS calls. Zenoss installs a
Python egg named httpsproxy_urllib2-1.0 which provides modified versions of the Python httplib and urllib2
Getting Started
14
modules. These replacement modules are used anytime Zenoss code imports httplib or urllib2. More information
regarding this module is available at PyPi.
Directions for configuring your environment to use an HTTP and HTTPS proxy are available in Zenoss Extended
Monitoringin the chapter on ZenWebTX.
2.4. zendmd: Command-line Access to the Device Management
Database (DMD)
Zenoss uses the Zope database (ZODB) to store its information. Since the ZODB is an object-oriented database,
this is not organized by tables, rows and columns, but by objects. The object that Zenoss uses to store the basic
model of your network is in the Device Management Database (DMD) object.
You can access the DMD through an interactive, programmable interpreter: zendmd. zendmd is the Python inter-
preter, with a handle to the database stored in the default namespace, and a few handy functions.
To start zendmd and see how the interpreter works, use the following commands:
$ zendmd
>>> 1 + 2
3
>>> len('hello there')
11
>>> for i in range(5):
... print i
0
1
2
3
4
These are all basic Python interpreter features. zendmd adds in a reference to the root of the object tree which is
known as dmd. You can see this root name in the URLs used to refer to objects when using Zenoss from the browser.
There is a built-in function that can be used to find devices.
$ zendmd
>>> print dmd
<DataRoot at /zport/dmd>
>>> find('localhost.localdomain')
<Device at /zport/dmd/Devices/Server/Linux/devices/localhost.localdomain>
The find() function also takes wildcards:
>>> find('local*')
<Device at /zport/dmd/Devices/Server/Linux/devices/localhost.localdomain>
You can perform scripting at the command prompt. For example, we can count the number of interfaces on our
device:
>>> d = find('local*')
len(d.os.interfaces())
5
You can inspect the objects:
>>> d.getManageIp()
'127.0.0.1'
Getting Started
15
for i in d.os.interfaces():
... for a in i.ipaddresses():
... print a.name(), a.getIpAddress()
eth0 192.168.1.148/24
You can perform low-level checks such as re-indexing all the objects:
>>> reindex()
Or check/repair relationships on all devices:
>>> for d in dmd.Devices.getSubDevices():
... d.checkRelations(repair=True)
...
Finally, after making changes you can commit them to the database:
>>> commit()
or synch against the database and restore the old state to your interpreter, reverting any changes:
>>> synch()
Zendmd can be used to automate repetitive tasks. For example, you can enter in a large list of devices. First, create
a text file containing the names of those devices:
$ cat >lotsOfDevices.txt
device1
myhost.mydomain.com
host2.mydomain.com
^D
Of course, the data could come from an inventory list or other database. Then, you can use the dmd to process
the file:
$ zendmd
for line in file('lotsOfDevices.txt'):
... d = dmd.Devices.Server.Linux.createInstance(line.strip())
... commit()
... d.collectDevice()
You can feed zendmd commands on stdin:
$ zendmd < AddDevices.py
You can also import scripts:
$ zendmd
import MyScripts
MyScripts.loadDevices(dmd)
If you want to create a stand-alone command, reading the $ZENHOME/ZenModel/zendmd.py file is a good start.
The full list of zendmd names is described below.
zendmd Name Description
dmd Device Management Database, the root persistent object
app The Zope Application, the root of the database
zport Zenoss Portal, the portal that contains Zenoss
find() Look up devices by name, and by address; supports wildcards
Getting Started
16
zendmd Name Description
devices Equivalent to dmd.Devices
sync() Revert the objects in zendmd back to the state in the ZODB
commit() Push object changes to the persistent store
abort() Undo any object changes and refresh from persistent storage
me a reference to the machine running zendmd, if it can be found
reindex() recreates the indexes against the objects
login() sets the security context of the given user
logout() removes any security context
Table 2.1. zendmd Names and Descriptions
2.5. Programming Documentation
2.5.1. Python
If you are new to Python here are a few resources to get you started:
• The official Python documentation contains a tutorial and the reference guide for the standard libraries that ship
with Python. Zenoss currently uses Python 2.6.
• Dive Into Python is an excellent book if you are familiar with other programming languages and contains lots
of great examples.
2.5.2. Zenoss API
As mentioned previously, more detailed information is gathered using the epydoc documentation system, and the
results are in the Application Programming Interface (API) documentation.
2.5.3. Other Resources
Discussion regarding development of Zenoss takes place on the Zenoss forums, at:
http://community.zenoss.org/community/forums
2.5.4. Contributing to the Documentation
If you find errors or omissions in the documentation, you can submit a ticket (see Section 2.1.6, “Submitting a Fix”)
or send an e-mail to docs@zenoss.com.
17
Chapter 3. ZenPacks
3.1. Overview
A ZenPack is a package that adds new functionality to Zenoss. For basic information on ZenPacks see the chapter
titled "ZenPacks" in Zenoss Administration. The following information pertains to the creation of more complex
ZenPacks that contain skins, Python classes, and daemons.
ZenPacks are packaged as Python eggs, which are the standard mechanism for packaging and distributing code.
Note
The zenpack command should be used for installation and removal of ZenPacks, not the easy_install command
that is frequently used with non-ZenPack Python eggs.
The use of dotted names for ZenPacks (see Section 3.2.1, “ZenPack Names” below) was also introduced in this
version. Zenoss 2.2 and later supports installation and use of pre-2.2 ZenPacks, but all new ZenPacks are created
in the new format. This document relates to ZenPacks created in the new style. For documentation on ZenPacks
predating Zenoss 2.2, please see previous versions of this document and Zenoss Administration.
Zenoss currently does not support installation or use pre-2.2 ZenPacks. If you have older ZenPacks that you want to
convert to egg-style ZenPacks, see the section titled Section 3.5.2, “Converting older ZenPacks to ZenPack eggs”.
3.2. Creating a ZenPack
ZenPacks can be created through the Zenoss user interface:
1. From the navigation menu, select Advanced > Settings.
2. Select ZenPacks in the left panel.
The list of loaded ZenPacks appears.
3. Select Create a ZenPack from the Action menu.
4. Enter the ZenPack name, and then click OK.
The ZenPack is created on the file system at $ZENHOME/ZenPacks/zenpackid and installed on the system.
3.2.1. ZenPack Names
ZenPack names consist of at least three strings joined by periods. The first of these strings is always "ZenPacks."
Each of these strings must start with a letter and contain only letters, numbers and underscores. The reason for this
naming scheme is that the ZenPack will set up namespaces in Python that reflect these names. There is a Python
namespace called ZenPacks. Within that namespace are packages representing the second part of all the installed
ZenPack and so on. So for example if you have a ZenPack named ZenPacks.MyCompany.MyZenPack then it can be
imported in Python (and zendmd) as:
import ZenPacks.MyCompany.MyZenPack
A data source class provided by this example might be accessed as:
from ZenPacks.MyCompany.MyZenPack.datasources.MyDataSourceClass \
import MyDataSourceClass
The advantage of these namespaces is that they help prevent namespace conflicts among different organizations
authoring ZenPacks. So if a third party wants to develop an HTTP monitoring ZenPack, then they could name it
ZenPacks.OurCompany.HttpMonitor and it would not conflict with the ZenPacks.zenoss.HttpMonitor Core ZenPack.
ZenPacks
18
3.2.2. Specifying Dependencies
The ZenPack edit page allows you to specify versions of Zenoss with which your ZenPack is compatible, as well as
dependencies on other ZenPacks. The first item in the Dependencies section of the page is the version of Zenoss
that is required. If that field is blank then your ZenPack can be installed under any version of Zenoss version 2.2
or later. If you enter a specific version number then the ZenPack will run only under that exact version of Zenoss,
this is usually not desirable. The most typical version requirement is to specify that the ZenPack is compatible
with any version of Zenoss equal to or greater than a specific version. The syntax for this is ">=X" where X is the
minimum version the ZenPack requires. For example, if a ZenPack requires Zenoss version 2.2.1 or greater the
version specification would be >=2.2.1.
Below the Zenoss version specification is a list of all other ZenPack eggs installed. Old-style (non-egg) ZenPacks
cannot be listed as dependencies and do not appear in this list. If your ZenPack requires another ZenPack to be
installed, then select the option in the Required column next to the required ZenPack. You also can give a version
specification for each ZenPack you require.
3.2.3. Locating ZenPack Source Outside of Zenoss
For any non-trivial ZenPacks we recommend maintaining the source code somewhere other than $ZENHOME/Zen-
Packs. There are a couple reasons for this:
• Performing a zenpack --remove deletes the ZenPack's directory from $ZENHOME/ZenPacks. If you do not have
the files copied in another location you can easily lose all or some of your work.
• If your ZenPack source is maintained in a version control system it is frequently easier to keep the code within
a larger checkout directory elsewhere on the filesystem.
To move a ZenPack source directory out of $ZENHOME/ZenPacks you can simply copy the directory to the new location
then run install again using the --link option. This will remove the $ZENHOME/ZenPacks/YourZenPackId directory.
cp -r $ZENHOME/ZenPacks/YourZenPackId SomeOtherDirectory
zenpack --link --install SomeOtherDirectory/YourZenPackId
3.2.4. Community ZenPack Subversion Access
There is a Community ZenPack development site at:
http://community.zenoss.org/community/developers/zenpack_development
This site hosts Subversion source code control access to all contributed Community ZenPacks. Accounts are grant-
ed by request and offered to ZenPack contributors. The goal of this site is to encourage ZenPack development and
open up improvements to all ZenPacks to a greater audience.
The Community ZenPack development site contains instructions for:
• working with Community ZenPacks from Subversion
• building and modifying ZenPacks
• converting old-style ZenPacks to Python egg ZenPacks
3.3. ZenPack Structure and Contents
This section describes the files and directory structures that make up most ZenPacks. A more detailed source of
information about Python eggs, entry points and other technical details of building eggs is found here.
ZenPacks
19
Note
The $ZENHOME/Products/ZenModel/ZenPackTemplate directory contains the template files and directories used
when Zenoss creates a ZenPack. If you decide to change these files, note that these changes will not be pre-
served across upgrades.
A ZenPack has the concept of a namespace, so that multiple people or organizations can create similar ZenPack
names without their code colliding with each other. In this example, the name of the ZenPack is ZenPacks.pkg.zpid,
where pkg is the package name and zpid is the ZenPack id.
In the $ZENHOME/ZenPacks/ directory, you will find the directory ZenPacks.pkg.zpid with the following contents (ab-
breviated for clarity):
build
build/bdist.linux-i686
build/lib
build/lib/ZenPacks
...
dist
dist/ZenPacks.pkg.zpid-version_id-py2.4.egg
INSTALL.txt
README.txt
setup.py
ZenPacks
ZenPacks/__init__.py
ZenPacks/pkg
ZenPacks/pkg/__init__.py
ZenPacks/pkg/zpid
ZenPacks/pkg/zpid/__init__.py
ZenPacks/pkg/zpid/daemons
ZenPacks/pkg/zpid/datasources
ZenPacks/pkg/zpid/datasources/__init__.py
ZenPacks/pkg/zpid/lib
ZenPacks/pkg/zpid/lib/__init__.py
ZenPacks/pkg/zpid/libexec
ZenPacks/pkg/zpid/migrate
ZenPacks/pkg/zpid/migrate/__init__.py
ZenPacks/pkg/zpid/modeler
ZenPacks/pkg/zpid/modeler/__init__.py
ZenPacks/pkg/zpid/modeler/plugins
ZenPacks/pkg/zpid/modeler/plugins/__init__.py
ZenPacks/pkg/zpid/objects
ZenPacks/pkg/zpid/objects/objects.xml
ZenPacks/pkg/zpid/parsers
ZenPacks/pkg/zpid/parsers/__init__.py
ZenPacks/pkg/zpid/reports
ZenPacks/pkg/zpid/services
ZenPacks/pkg/zpid/services/__init__.py
ZenPacks/pkg/zpid/skins
ZenPacks/pkg/zpid/skins/ZenPacks.pkg.zpid
ZenPacks.pkg.zpid.egg-info
ZenPacks.pkg.zpid.egg-info/entry_points.txt
ZenPacks.pkg.zpid.egg-info/namespace_packages.txt
ZenPacks.pkg.zpid.egg-info/not-zip-safe
ZenPacks.pkg.zpid.egg-info/PKG-INFO
ZenPacks.pkg.zpid.egg-info/SOURCES.txt
ZenPacks.pkg.zpid.egg-info/top_level.txt
ZenPacks
20
This directory is created by Python when the ZenPack is exported to an egg file or when it is installed from
source. This directory can safely be deleted at any time if you wish and need not be kept within any version
control system.
This directory is created when the ZenPack is exported to an egg file. The egg file is initially created within
here then copied to the $ZENHOME/export directory. This directory can safely be deleted at any time if you wish
and need not be kept within any version control system.
This file contains parameters for use by setuptools and distutils in creating eggs and doing source installs.
Zenoss creates an appropriate setup.py when a ZenPack is created. ZenPack developers should usually edit
this information through the ZenPack edit page within Zenoss rather than directly in the setup.py file.
Any time a ZenPack is saved or exported via the GUI Zenoss will modify certain values at the top of the setup.py
file. The lines that Zenoss modifies are clearly commented and segregated at the top of the file. If you wish to
make changes to setup.py you can safely do so as long as you leave those lines intact.
This directory mirrors the dotted name structure of your ZenPack name. For example, if your ZenPack name is
ZenPacks.MyCompany.MyZenPack then this directory will contain a directory named MyCompany which will contain
a MyZenPack directory. This last directory with the same name as the last part of your ZenPack name is where
most of the ZenPack code resides. The structure of that directory is very similar to that of previous non-egg
ZenPacks.
This is the directory whose name is that of the last part of your dotted ZenPack name.
This file contains any code that needs to be executed when the ZenPack is loaded. Zenoss loads all installed
ZenPacks on startup. Typically this file contains a few lines that will register a skins directory if the ZenPack
provides one. Also, if this class contains a class named ZenPack then on installation Zenoss will create an
instance of that class rather than the base ZenPack class in the object database.
See below for more details on providing daemons in ZenPacks.
See below for more details on providing data source classes in ZenPacks.
This directory is intended to hold any 3rd party modules or other code your ZenPack depends on. A module
named Foo in this directory would be imported with
import ZenPacks.MyCompany.MyZenPack.lib.Foo
This directory is intended to hold plugins, such as Nagios-style or Cacti-style plugins.
See below for more details on migrating between versions of your ZenPack.
See below for more details on providing modeler plugins in ZenPacks.
Database objects such as device classes and monitoring templates that are added to the ZenPack via the GUI
are exported to an objects.xml file in this directory. When the ZenPack is installed on another system those
objects will be copied into that object database.
This directory contains any command parsers provided by the ZenPack. See the section that discusses new
platform command parsers for more details.
This directory contains any report plugins provided by the ZenPack.
Zenoss daemons usually communicate with zenhub to retrieve their configuration, send events, and write
performance data. If a ZenPack provides a daemon then it typically will also provide a ZenHub service for that
daemon. See the section on ZenHub for further details.
This directory contains any skins directories that should be added to Zope. Note that this contains directories
of skins, not the skin files themselves. If you include skins directories make sure that the __init__.py file in
the directory above skins is registering this directory. (The default __init__.py file provided in new ZenPacks
does this for you.)
This directory contains files which describe the egg meta-data. This is created when the egg file is generated
or the ZenPack is installed from source. This directory can safely be deleted at any time if you wish and need
not be kept within any version control system.
This file is updated every time a ZenPack is edited and saved. ZenPack developers should normally not edit
this file manually.
ZenPacks
21
3.4. Developing the ZenPack
3.4.1. Base ZenPack Class
$ZENHOME/Products/ZenModel/ZenPack.py contains the base ZenPack class. When a ZenPack is installed Zenoss
inspects YourZenPackId/ZenPacks/..../LastPartOfName/__init__.py to see if it contains a class named ZenPack.
If it does then Zenoss instantiates it, otherwise Zenoss instantiates the base ZenModel.ZenPack.ZenPack class. That
instance is then added to the dmd.ZenPackManager.packs tree.
There are several attributes and methods of ZenPack that subclasses might be interested in overriding:
Interesting ZenPack properties and methods
packZProperties is a mechanism for easily adding configuration properties. packZProperties is a
list of tuples, with each tuple containing three strings in this order:
• name of the configuration property
• default value of the configuration property
• type of configuration property (such as string or int)
Zenoss will automatically create these when the ZenPack is installed and remove
them when the ZenPack is removed. See ZenPacks.zenoss.MySqlMonitor for an
example of this usage.
install(self, app) parais called when the ZenPack is installed. If you override this be sure to call
the inherited method within your code.
remove(self, app, leaveOb-
jects)
is called when the ZenPack is removed. As with install(), make sure you call
the inherited method if you override.
3.4.2. Storing Objects in the ZODB
ZenPacks can provide Python classes for objects that will be stored in the object database. The most frequent
example of this is DataSource subclasses. When a ZenPack is removed those classes are no longer accessible
so the objects in the database are broken. (Zeo needs to have the appropriate Python class in order to unpickle
an object from the database.) In previous versions of Zenoss there was not an easy way to associate instances
of a ZenPack-provided class with the ZenPack that provided the class. As a result ZenPack removal could easily
cause broken objects to remain in the database. If Zope had already loaded a class into the interpreter the objects in
question might continue to function until Zope was restarted, making diagnosis of such problems even more difficult.
In Zenoss 2.2 and later the ZenPackPersistance class aims to remedy this problem. Any Python class pro-
vided by a ZenPack should subclass the ZenModel.ZenPackPersistence.ZenPackPersistence class. Zenoss
maintains a catalog of all ZenPackPersistence instances in the database. When a ZenPack is removed,
the catalog is queried to determine which objects need to be deleted. Any ZenPack-provided Python class
that might be instantiated in the object database should subclass ZenPackPersistence and define ZEN-
PACKID in the class as the name of the ZenPack providing the class. For an example of this see the
ZenPacks.zenoss.MySqlMonitor.datasources.MySqlMonitorDataSource ZenPack.
3.4.3. Providing DataSource classes
ZenPacks can provide new classes of DataSources by subclassing the ZenModel.RRDDataSource.RRDDataSource
class. If you include only one DataSource class per file, name the modules after the class the contain (for
example, MyDataSource.py contains the class MyDataSource), and place those modules in the ZenPack's da-
ta sources directory then they will automatically be discovered by Zenoss. If you want to customize this be-
havior take a look at the ZenPack.getDataSourceClasses() function. See the ZenPacks.zenoss.HttpMonitor and
ZenPacks.zenoss.MySqlMonitor ZenPacks for examples of ZenPacks that provide custom DataSource classes.
ZenPacks
22
When creating a custom DataSource class one of the first decisions you have to make is whether you want zencom-
mand to process these DataSources for you or whether you will provide a custom collector daemon to process them.
The zencommand daemon is a very versatile mechanism for executing arbitrary commands either on the Zenoss
server or on the device being monitored, processing performance data returned by the DataSource and generating
events in Zenoss as appropriate. zencommand expects the command it executes be compatible with the Nagios
plug-in API. Specifically two aspects of that API are of most importance:
•Return code -The command should exit with a return code of 0, 1, 2 or 3. See here in the Nagios plug-in API
for more detail.
•Performance data -- If the command returns performance data then that data can be pulled into Zenoss by
creating DataPoints with the same names used in the command output. See here in the Nagios plug-in API
for more detail.
If you want zencommand to handle instances of your custom DataSource class then several methods in RRD-
DataSource are of particular interest:
•getDescription(self) - This returns a string describing the DataSource instance. This string is displayed next to
the DataSource on the RRDTemplate view page.
•getCommand(self, context, cmd=None) - This returns the string that is the command for zencommand to ex-
ecute. context is the device or component to be collected. If you need to evaluate TALES expressions in the
command to replace things like ${dev/id} and so forth you can call the parent class's getCommand() and pass
your command as the cmd argument. (cmd will not be passed into your method, it exists specifically for sub-
classes to pass their commands to the parent for TALES evaluation.)
• checkCommandPrefix(self, context, cmd) - Zenoss will check the string you return from getCommand() to see if it
is a relative or absolute path to a command. If the string starts with '/' or '$' then Zenoss assumes it is absolute.
Otherwise the configuration property zCommandPath from the context is prepended to the cmd string. You can
override checkCommandPrefix() if you want to alter this behavior.
Make sure that your DataSource subclasses also subclass ZenPackPersistence and list it first among the parent
classes. See the section on ZenPackPersistence.py for more details.
3.4.4. Monitoring Template Checklist
Monitoring templates are one of the easiest places to make a real user experience difference when new features
are added to Zenoss. Spending a very small amount of time to get the templates right goes a long way towards
improving the overall user experience.
3.4.4.1. Data Sources
• Can your data source be named better?
• Is it a common metric that is being collected from other devices in another way? If so, name yours the same.
This makes global reporting much easier.
• camelCaseNames are the standard. Use them.
• Never use absolute paths for COMMAND data source command templates. This will end up causing problems on
one of the three platforms we deal with. Link your plugin into zenPath('libexec') instead.
3.4.4.2. Data Points
• Using a COUNTER? You might want to think otherwise.
• Unnoticed counter rollovers can result in extremely skewed data.
• Using a DERIVE with a minimum of 0 will record unknown instead of wrong data.
• Enter the minimum and/or maximum possible values for the data point if you know them.
• This again will allow unknown to be recorded instead of bad data.
ZenPacks
23
3.4.4.3. Thresholds
• Don't include a number in your threshold's name.
• This makes people have to recreate the threshold if they want to change it.
3.4.4.4. Graph Definitions
• Have you entered the units? Do it!
• This will become the y-axis label and should be all lowercase.
• Always use the base units. Never kbps or MBs. bps or bytes are better.
• Do you know the minimum/maximum allowable values? Enter them!
• Common scenarios include percentage graphing with minimum 0 and maximum 100.
• Think about the order of your graph points. Does it make sense?
• Are there other templates that show similar data to yours? If so, you should try hard to mimic their appearance
to create a consistent experience.
3.4.4.5. Graph Points
• Have you changed the legend? Do it!
• Adjust the format so that it makes sense.
•%5.2lf%s is good for values you want RRDTool to auto-scale.
•%6.2lf%% is good for percentages.
•%4.0lf is good for four digit numbers with no decimal precision or scaling.
• Should you be using areas or lines?
• Lines are good for most values.
• Areas are good for things that can be thought of as a volume or quantity.
• Does stacking the values to present a visual aggregate make sense?
3.4.5. Providing Performance Collector Plugins
When providing performance collectors in a ZenPack (for example, Nagios-style plugins), the suggested method
for referencing the collector in the Command Template area is the following TALES expression:
${here/ZenPackManager/packs/ZenPacks.pkg.zpid/path}/libexec/myplugin.sh
3.4.6. Referencing Collector Plugins in ZenPacks
While modeler plugins are stored in the ZenPack's modeler/plugins directory, collector plugins are, by convention,
stored in the libexec directory. Because Zenoss can be installed in multiple ways, and a ZenPack's directory name,
when installed, includes a version number, Zenoss offers a more portable and "future-proof" way of referencing a
plugin.
In the Command Template section of the data source, you can reference a plugin by using a TALES expression,
such as:
${here/ZenPackManager .....}.../file.sh
For example:
${here/ZenPackManager .....}.../MyCollectorPlugin.sh ${dev/manageIp} ${dev/zSnmpCommunity} OtherParameters
ZenPacks
24
After adding the monitoring template containing the data source to a ZenPack, and then exporting the ZenPack, the
ZenPack's object/objects.xml file will contain an entry similar to:
<property .....>
${here/ZenPackManager .....}.../MyCollectorPlugin.sh ${dev/manageIp} ${dev/zSnmpCommunity} OtherParameters </property>
3.4.7. Providing Daemons
ZenPacks can provide new performance collectors and event monitors. This is a somewhat complex undertaking,
so before deciding to write your own daemons make sure that zencommand and a custom DataSource class won't
fit your needs (see Section 3.4.3, “Providing DataSource classes” above.) Any file in a ZenPack's daemons directory
is symlinked in $ZENHOME/bin when the ZenPack is installed. Also, the Zenoss script that controls the core daemons
will attempt to manage your daemon too. So a zenoss start, for example, will attempt to start your daemon as well
as the core daemons.
Custom daemons usually subclass the ZenHub.PBDaemon.PBDaemon class. This class provides the basic framework
for communicating with zenhub. See the section "Writing a Performance Collector" for more details.
3.4.8. setuptools and the zenpacksupport
Zenoss requires a Python module called setuptools to create and install eggs. The setuptools module is installed
by the Zenoss installer in the $ZENHOME/lib/python directory. Zenoss also provides a module named zenpacksupport
which extends setuptools. The zenpacksupport class defines additional metadata that is written to and read from
ZenPack eggs. This metadata is provided through additional options passed to the setup() call in a ZenPack's
setup.py file. Those arguments are:
compatZenossVers This is the version specification representing the required Zenoss version from the
ZenPack's Edit page.
prevZenPackName This is the name of the old-style (non-egg) ZenPack that this ZenPack replaces. If a ZenPack
with this name is installed in Zenoss then it is upgraded and replaced when this ZenPack is
installed. For example, if HttpMonitor is installed and then ZenPacks.zenoss.HttpMonitor is
installed (which has prevZenPackName=HttpMonitor) then ZenPacks.zenoss.HttpMonitor
will replace HttpMonitor. All packable objects in the database that are included in Http-
Monitor will be added to ZenPacks.zenoss.HttpMonitor instead. A migrate script is usually
required to set __class__ correctly on instances of ZenPack-provided classes in the object
database. The ZenPacks.zenoss.HttpMonitor ZenPack has an example of this in its migrate
directory, in the ConvertHttpMonitorDataSources.py file.
3.5. Building and Distributing ZenPacks
From your ZenPack's page in the GUI select the Export ZenPack... menu item to create an egg file. The file is first
created in your ZenPack's dist directory then copied to the $ZENHOME/export directory.
You can optionally also download the egg file through your web browser when doing the export. As part of the export
process Zenoss exports database objects to the objects/objects.xml file in your ZenPack source directory. If you
don't need to update the objects.xml file you can create the egg from the command line instead:
cd YourZenPackDirectory
python setup.py bdist_egg
This creates the egg file in the ZenPack's dist directory.
Users who install your egg file will not be able to edit the ZenPack or re-export it. These functions require the setup.py
file which is not usually distributed within the egg file itself. In most cases this is desirable because end-users should
usually not be making changes and redistributing a different version of your ZenPack than the one you developed.
ZenPacks
25
There are times when you want to allow others to develop a ZenPack with you. In these cases you must provide
them with the entire source directory, not just an egg file.
3.5.1. Migrating between versions
Any time a ZenPack is installed Zenoss looks in the ZenPack's migrate directory for steps whose version is
greater than or equal to the version of the ZenPack being installed. Migrate steps are classes that subclass
ZenModel.ZenPack.ZenPackMigration. This mechanism allows ZenPacks to modify items in the object database that
were created by previous versions of the ZenPack and need updating. The ZenPacks.zenoss.MySqlMonitor Core
ZenPack includes good examples of how migrate steps are written.
3.5.2. Converting older ZenPacks to ZenPack eggs
Zenoss 2.2 and later includes a new script called eggifyzenpack which automates much or all of the process of
converting a pre-2.2 ZenPack to an egg ZenPack. The script is in $ZENHOME/bin so is usually on the zenoss user's
path already. The --newid option is required and specifies the new name of the ZenPack. (See the section above
on ZenPack names.) the sole positional argument to eggifyzenpack is the current name of the installed ZenPack
to be converted. Zeo must be running prior to invoking the script.
eggifyzenpack --newid ZenPacks.MyCompany.MyZenPackName MyOldZenPackName
This will create a ZenPack with the name given with --newid in $ZENHOME/ZenPacks. The old ZenPack that
was converted is uninstalled and removed from $ZENHOME/Products. ZenPacks converted in this way have
PREV_ZENPACK_NAME in their setup.py set to the name of the old ZenPack that they replace. When a user with the
old ZenPack installed installs the new egg ZenPack it will be processed as an upgrade and the older ZenPack will
be removed.
3.6. Development Mode
New ZenPacks can be created in Zenoss by navigating to Advanced > Settings > ZenPacks and select-
ing Create a ZenPack from the Action menu. This creates the ZenPack on the file system at $ZENHOME/Zen-
Packs/ZenPacks.community.YourZenPack and installs it into Zenoss. You may then proceed to add device classes,
templates, and MIBs to the ZenPack. (This is known as "development mode" for the ZenPack.) Once you are happy
with your ZenPack, you can export it for others to use. However, once you install a freshly exported .egg ZenPack
on another system (or uninstall and re-install your new ZenPack) you can no longer add things to the ZenPack.
3.6.1. Source ZenPacks
If you have the source for the ZenPack available you can simply attach to the source tree. Assuming that the source
directory is ZenPacks.community.YourZenPack, install the ZenPack with the following commands:
zenpack --link --install ZenPacks.community.YourZenPack
zopectl restart
Your ZenPack should now be usable and back in development mode. Changes made to the ZenPack will be per-
sisted back to the source tree, you may still export and download as necessary. When you are satisfied with your
changes, you may commit them back to the Subversion repository.
3.6.2. Converting .egg Files to Development Mode
If you wish to convert an already installed ZenPack, or to install and convert an .egg ZenPack, follow these steps.
1. Install the .egg as you would normally.
2. Restart Zope with the command:
zopectl restart
ZenPacks
26
3. Copy the ZenPack development files into the .egg's directory (the CONTENTS directory is unnecessary):
cp $ZENHOME/Products/ZenModel/ZenPackTemplate/* \
$ZENHOME/ZenPacks/ZenPacks.community.YourZenPack-1.0.2-py2.4.egg/
4. You can now make any modifications to the ZenPack, such as updating the version number or adding new
device classes.
5. Go into the ZenPack (from Advanced > Settings, select ZenPack from the left panel, and then click the ZenPack).
6. Export the ZenPack (select Export ZenPack from the Action menu). The changes will be persisted to the new
.egg and the file system.
Note
There is a minor bug in the export and download functions. The new version saved in the export directory will
have the correct name with all the updates (for example, ZenPacks.community.YourZenPack-1.0.3-py2.4.egg).
If you choose to export and download the ZenPack, it will have the original name despite the updated version
(for example, ZenPacks.community.YourZenPack-1.0.2-py2.4.egg) or it may fail to download. Use the version in
the export directory.
3.7. Where to Get More Information
Discussion regarding development of ZenPacks takes place on the Zenoss Community forums, at:
http://community.zenoss.org/community/zenpacks
27
Chapter 4. Zenoss Data Stores
There are a few data stores used by Zenoss:
Data Stores
ZODB Object-oriented database for Python objects
MySQL The Event database where event information is stored.
Pickle files Python pickle files are used to cache information otherwise obtained from zenhub.
RRD files Round Robin Database that stores performance information.
Figure 4.1. Datastores Overview
4.1. Zope Object Database (ZODB)
The ZODB is an object-oriented database used by Zope to store Python objects and their states. For example,
modelers maintain information about devices and their configuration in the ZODB.
Zenoss uses ZEO, which is a layer between Zope and the ZODB. ZEO allows for multiple Zope servers to connect
to the same ZODB. The ZODB is started and stopped by zeoctl.
Note
ZODBs can be clustered using ZEO, but Zenoss Enterprise customers should contact Customer Support before
investigating clustering.
Here is a simple example of using transactions in the ZODB:
Zenoss Data Stores
28
...
import transaction
...
trans= transaction.get()
# Determine that bad things have happened
if bad_thing:
trans.abort()
# ... any other cleanup required inside the function eg 'return'
# Life is good!
# NB: Username or program name -- it's just a text field
trans.setUser( "zenmodeler" )
trans.note( "Added good things to xyz object" )
trans.commit()
The setUser() and note() functions are responsible for creating entries that can be found under the Modifications
tab or menu-item.
There are restrictions on what data can be stored, specifically data types that can be pickled. Basic Python data
types such as strings, numbers, lists and dictionaries can be pickled, but Python code objects cannot be pickled.
In addition, files and sockets cannot be pickled.
Note
The ZODB cannot detect changes to mutable types like lists and dictionaries. In order for changes to be detected,
not only is commit() afterwards, but you must explicitly tell the ZODB about the change by modifying a Persistent
objects _p_changed attribute.
# The following imports shouldn't be required in code
# as it should already be taken care of for you. These are
# included merely to explicitly show the class dependencies.
import ZODB
from Persistence import Persistent
import transaction
...
class myExampleClass( Persistent ):
"""
An example class to be used to demonstrate the use of the
modifying a list and then notifiying ZODB that work needs
to be done through the _p_changed attribute.
"""
def __init__(self):
"""
Initializer
"""
self.mylist= []
def addToMyList( self, listItem ):
"""
Track the listItems that we need
"""
self.mylist.append( listItem )
self._p_changed= True # Notify ZODB
transaction.commit()
As a general rule, use commit() whenever you want other processes to have access to your database changes. So
if a daemon is collecting and Zope needs to do something with the data, run commit() first from the daemon.
This should be enough information to get you started. See ZODB for Python Programmers for more details.
Zenoss Data Stores
29
4.2. MySQL Event database
MySQL is an open-source relational database that Zenoss uses to store events. Configuration information about
the MySQL database can be maintained by going to Events > Event Manager from the navigation bar when you
are logged in as a user with ZenManager privileges.
MySQL-level performance tweaking can substantially improve Zenoss' ability to handle events. One tool that can
be used to improve your database performance is MySQLTuner (http://blog.mysqltuner.com/).
If you need a connection to the MySQL events database, here is how to retrieve a connection and how to put it
back into the pool.
DbConnectionPool is hidden and is accessed through DbAccessBase. It follows the Singleton design pattern, so it'll on-
ly actually create one DbConnectionPool. It extends the Python class Queue, so DbConnectionPool is also a synchro-
nized queue and should be thread-safe. DbAccessBase is extended by EventManagerBase>?, so if you have access to
the ZenEventManager (located at /zport/dmd/ZenEventManager) you'll have the ability to get a database connection.
4.2.1. Connecting to the Database
First you'll need to get an instance of ZenEventManager OR an instance of a class that extends DbAccessBase. Within
Zenoss, a ZenEventManager should already be instantiated.
Next is the try block which should include ANY database calls. This is where you'll get a connection from the pool
with the connect() method. You may pass this around to other methods or create a cursor and make some database
transactions. The try block MUST be completed with a finally block that includes the close() method. You MUST
pass the connection object to the close() method. This will ensure that even if the code within the try breaks, we
are not leaking database connections. If you create more than one connection (ie more than one connect() call in
your try block) you will need to have a corresponding close() call. There is ALWAYS a one-to-one relationship
between connect() and close() calls.
Here is a block of code that illustrates best practices for using the DbConnectionPool
...
zem = self.dmd.ZenEventManager
try:
conn1 = zem.connect()
conn2 = zem.connect()
curs1 = conn1.cursor()
...
curs2 = conn2.cursor()
...
# do work
...
curs3 = conn1.cursor()
...
finally:
zem.close(conn1)
zem.close(conn2)
...
...
Take a look at EventManagerBase.py for some examples of code using the DbConnectionPool.
4.2.2. MySQL in 60 Seconds
To start an interactive session with MySQL, run the mysql as the zenoss user. The following example is from a
default install of Zenoss where there is no password for the MySQL root user.
$ mysql -uroot
Zenoss Data Stores
30
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 17799
Server version: 5.0.45 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
Once we've logged into MySQL, we can see the various databases and see the tables that are available. The events
database is maintained by Zenoss.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| events |
| mysql |
| test |
+--------------------+
4 rows in set (0.03 sec)
mysql> use events;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+------------------+
| Tables_in_events |
+------------------+
| alert_state |
| detail |
| heartbeat |
| history |
| log |
| status |
+------------------+
6 rows in set (0.00 sec)
From here we can determine what information is in what table. For instance, the log table.
mysql> describe log;
+----------+-------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-------------+------+-----+-------------------+-------+
| evid | char(25) | NO | MUL | | |
| userName | varchar(64) | YES | | NULL | |
| ctime | timestamp | NO | | CURRENT_TIMESTAMP | |
| text | text | YES | | NULL | |
+----------+-------------+------+-----+-------------------+-------+
4 rows in set (0.00 sec)
4.3. Python Pickle Files
Python's native storage for storing data is called a Pickle. Pickle files are used by zenperfsnmp for caching configura-
tion information gathered from zenhub. This is a performance enhancement for dealing with startup communications
with zenhub, as larger sites with hundreds or more devices could experience enough of a delay during initialization
that Zenoss would have difficulty functioning until the configuration information had been gathered. Every update
from the Zenoss server (which is dealt with by zenhub) causes zenperfsnmp to update the pickle files.
The pickle files are kept in the $ZENHOME/perf/Devices/devicename/ directory, and are named collector-
config.pickle. These pickle files are only read during startup and are periodically recreated, so it is safe to delete
them, and it is not necessary to back them up.
Zenoss Data Stores
31
4.4. Round-Robin Database
RRD is used by Zenoss to store and graph performance collection data. These data files have a fixed format that
is decided at their creation time, and record data points at set intervals. This data is later consolidated into coarser
time units (so as to reduce the total size of data files) and the RRD toolset also contains code to create graphs.
A few other interesting facts:
• Zenoss is a gold-level sponsor of RRD.
• The Renderserver sends RRD graphics to Web browsers.
Zenoss Data Stores
32
Figure 4.2. RRD Overview
33
Chapter 5. Events
5.1. Understanding an Event Entry
From a Python programming perspective, an event is essentially a dictionary of keyword/value pairs that gets passed
up to zenhub to be stored and parsed. A description of the standard fields used in Zenoss can be found in the
section titled Event Database Dictionary.
From the user's perspective, the events can be found in the event console. To view an event's information, dou-
ble-click it in the list of events. The standard keyword and value pairs are presented to the user in the Details area.
5.1.1. Event Design
There are a few requirements for events:
• Event objects need to be persisted in the MySQL database.
• On queries from within Zope these queries must use the Zope security mechanisms to allow controlled access
to the data.
• Events must be constructed outside of a Zope framework as well.
To meet these requirements there are three types of event:
Event an event that lives outside of a Zope context and can go in and out of MySQL.
ZEvent event in a Zope context inherits from Event and has a subset of its fields populated as defined
by resultFields in a getEventList() query.
ZEventDetail full event information (all fields, detail, and log)
5.2. Sending an Event
Events can be created through a number of different ways:
• From the command line (zensendevent)
• Through the user interface (Add Event)
• By daemons, which convert their messages into events (such as zentrap)
• From daemons and programs that have detected error conditions
• From an external source (using, for example, XML-RPC)
Regardless of what program generates the event, or from which protocol the event is sent to Zenoss, the following
fields (at a minimum) should be specified:
Event fields
device the name of the device from which this event originates
component the sub-component of the device (for instance eth0, http, etc)
summary the text message of the event
severity an integer between 0 and 5 with higher numbers being higher severity. Zero is clear. Note that for
Python code, that mappings to names are provided (see example below).
Here is an example using Python from within a program that connects to zenhub:
# Import severities (eg Clear, Debug, Info, Warning, Error Critical) and
Events
34
# some event classes into our namespace
from Products.ZenEvents.ZenEventClasses import *
class exampleClass(PBDaemon):
def examplefunc( self ):
event= {}
event[ 'component' ]= 'eth0'
event[ 'severity' ]= Warning
event[ 'summary' ]= 'eth0 is down'
event[ 'message' ]= 'Received error code 0xa7 from listen()'
self.sendEvent( event, device='mydevice' )
Using XML-RPC in Python:
from xmlrpclib import ServerProxy
myurl= 'http://admin:zenoss@MYHOST:8080/zport/dmd/ZenEventManager'
serv = ServerProxy( myurl )
evt = {'device':'mydevice', 'component':'eth0', 'summary':'eth0 is down',
'severity':4, 'eventClass':'/Net'
}
serv.sendEvent(evt)
Tip
Some suggested non-standard fields for adding to your event are:
resolution Describe a method of fixing the situation that might have caused the event, or suggest a course
of action for diagnosing the condition.
explanation Describe in more detail the impact of this event on the computing environment. For instance,
does the condition which generates this event prevent a service from starting or being moni-
tored?
5.3. Adding an Event Class
Event classes can be added easily through the UI. If you need to use an event class internally, however, you need
to make sure that class will always be available, which involves several more steps.
5.3.1. Add to ZenEventClasses
Add a definition of the name of your new event class to Products/ZenEvents/ZenEventClasses:
...
My_New_Class = "/My/New/Class"
Now your event class is centralized and can be imported wherever you need to use it, e.g.:
...
from Products.ZenEvents.ZenEventClasses import My_New_Class
...
if thing.evclass == My_New_Class:
...
5.3.2. Add the class to the import XML
Several event classes are imported from XML by zenload just after the ZODB is created. To include your new event
class in this import, add an <object> element describing it to Products/ZenModel/data/events.xml. Be sure to nest
it inside the classes that already exist, if appropriate. For example, if your new class is "/Status/NewClass", you
would add it inside the <object id='Status'> that already exists:
...
Events
35
<object id='Status' module='Products.ZenEvents.EventClass' class='EventClass'>
<!--This event exists already -->
...
<object id='NewClass' module='Products.ZenEvents.EventClass' class='EventClass'>
<!--This is your new event -->
</object>
></object>
5.3.3. Write a migrate script
Since your code is no longer backwards-compatible, you need to add the new event class to databases that have
already been created by writing a migrate script. (See the section on migrating for more detailed information). Create
a new script in Products/ZenModel/migrate with an unique name (here neweventclasses.py). Here's an example:
__doc__="""Add new classes to EventManager
"""
...
import Migrate
...
class NewEventClasses(Migrate.Step):
version = Migrate.Version(1, 1, 0) # Replace this with the correct version
def cutover(self, dmd):
dmd.Events.createOrganizer("/My/Event/Class")
dmd.Events.createOrganizer("/My/Event/Class2") # Add multiple new classes in the same migrate script
dmd.ZenEventManager.buildRelations()
NewEventClasses()
Next, add your migrate script to Products/ZenModel/migrate/__init__.py:
...
import neweventclasses
Now
zenmigrate --dont-commit
to make sure your class is created properly.
Once you're satisfied with your changes, make the changes permanent with zenmigrate.
$ zenmigrate
36
Chapter 6. Configuration Property
Management
6.1. Adding a Configuration Property
6.1.1. Adding a Configuration Property to an Event
In EventClass.py...
...
def buildZProperties(self):
edict = self.getDmdRoot("Events")
edict._setProperty("zNewProperty", "default value")
edict._setProperty("zNewIntegerProperty", -1, type="int")
edict._setProperty("zNewFloatProperties", 10.01, type="float")
edict._setProperty("zNewListProperty", ["default value", \
"another default value"], type="lines")
edict._setProperty("zNewBooleanProperty", False, type="boolean")
...
Adding a new property to the EventClass is as easy adding a new line to the buildZProperties method. You need
to set a new property at the "Events" level.
6.1.2. Adding a Configuration Property to a Device
In DeviceClass.py
...
def buildDeviceTreeProperties(self):
devs= self.getDmdRoot("Devices")
...
devs._setProperty("zNewProperty", "default value")
devs._setProperty("zNewIntegerProperty", -1, type="int")
devs._setProperty("zNewFloatProperties", 10.01, type="float")
devs._setProperty("zNewListProperty", ["default value", \
"another default value"], type="lines")
devs._setProperty("zNewBooleanProperty", False, type="boolean")
...
...
Adding a new property to the DeviceClass is as easy adding a new line to the buildDeviceTreeProperties method.
You need to set a new property at the "Devices" level.
6.2. Migrating the Configuration Property Code
Create a new file in $ZENHOME/Products/ZenModel/migrate/zNewProperty.py
__doc__='''
Add zNewProperty to DeviceClass.
'''
Configuration Property Management
37
import Migrate
class zNewProperty( Migrate.Step ):
version= Migrate.Version(1, 1, 0)
def cutover(self, dmd):
if not dmd.Devices.hasProperty( "zNewProperty" ):
dmd.Devices._setProperty( "zNewProperty", "default value here" )
zNewProperty()
When a zenmigrate is executed, this code will create the new configuration property for all Devices. Do not forget
to update the Migrate.Version to your current working version. For more information on migrating: see the section
on Chapter 16, Migrating Zenoss Code.
38
Chapter 7. Creating New Jobs
Creating a Job class to encompass an asynchronous process is fairly straightforward. A simple subclass defining
a single method is usually all that is required.
7.1. Job Requirements
Jobs should subclass Products.Jobber.jobs.Job. At a minimum, a Job must implement its own run() method, which
should perform the actions specific to the job and call back to the finished() method reporting success or failure.
Example 7.1. A Job that cleans up the history table in the events database
from Products.Jobber.jobs import Job
from Products.Jobber.status import SUCCESS, FAILURE
class CleanHistoryJob(Job):
"""
Delete all events of a certain age from the
history table.
"""
def __init__(self, agedDays=7):
self.agedDays = agedDays
super(CleanHistoryJob, self).__init__()
def run(self, r):
zem = self.dmd.ZenEventManager
try:
zem.manage_deleteHistoricalEvents(
agedDays=self.agedDays)
except:
self.finished(FAILURE)
else:
self.finished(SUCCESS)
7.2. Running a Job
dmd.JobManager is a tool that, predictably, manages jobs. To add a job to the queue to be run by the zenjobs
daemon, call dmd.JobManager.addJob, passing in the job class as the first argument, followed by arguments to the
job's constructor. For example, to run the example CleanHistoryJob:
dmd.JobManager.addJob(CleanHistoryJob, agedDays=7)
7.3. Life Cycle of a Job
When zenjobs runs a Job, it calls the start() method, which calls run() and returns a Deferred that will fire when
the Job finishes; setup steps that can't happen in run() for whatever reason should occur here. run() should, as
mentioned above, call finished(); Jobs that require post-run actions may override finished() to provide them.
Creating New Jobs
39
Example 7.2. A Job that sends an email when starting and finishing
class EmailSendingJob(Job):
def start()
self.preRun()
return super(EmailSendingJob, self).start()
def run(self, r):
# Do whatever
self.finished(SUCCESS)
def finished(self, r):
self.postRun()
return super(EmailSendingJob, self).finished(r)
def preRun(self):
sendEmail("Job %s is starting" % self.id)
def postRun(self):
sendEmail("Job %s has finished" % self.id)
A Job may provide an interrupt() method that halts the job. The implementation of this method in the base class
does nothing at all.
7.4. Shell Command Jobs
Products.Jobber.jobs.ShellCommandJob is a useful base class for Jobs that run commands in a child shell. Sub-
classes should set the cmd attribute on the instance. A ShellCommandJob can also be scheduled directly, passing in
a list representing the command as the first argument.
Example 7.3. A Job that models a device in the background
from Products.Jobber.jobs import ShellCommandJob
class ModelDeviceJob(ShellCommandJob):
def __init__(self, devname):
self.cmd = ['zenmodeler', 'run', '-d', devname]
super(ShellCommandJob, self).__init__()
# Or, to run a command as a one-off:
def modelDevice(dmd, devname):
dmd.JobManager.addJob(ShellCommandJob,
['zenmodeler', 'run', '-d', devname])
ShellCommandJob's implementation of the interrupt() method kills the child process (kindly, if possible).
7.5. Logging
Jobs can write text to disk so that it is accessible by other processes, using a specialized LogFile object. Simply
call Job.getLog() to get the log, then use log.write(text) to write a line. This LogFile is streamed to the UI in
the job detail view.
40
Chapter 8. Device Management
8.1. Adding Devices Programatically
You can add devices to Zenoss through its user interface and through a programmatic interface. This section pro-
vides more information about adding devices through the programmatic interface.
8.1.1. Using a REST call
Adding a device through a rest call can be done by a simple Web get. For example:
$ wget --auth-no-challenge
'http://admin:zenoss@MYHOST:8080/zport/dmd/DeviceLoader/loadDevice\
?deviceName=NEWDEVICE&devicePath=/Server/Linux'
Note
When using wget, you must escape ampersands (&) and wrap the URL in single quotes.
The result of this command will be the log of auto-discovery. Look in this log for the string "NEWDEVICE loaded!"
to see if the add was successful. Possible failure messages are: "NEWDEVICE exists" and "no snmp found."
8.1.2. Example: Using an XML-RPC Call from Python
This example shows how to add a device by using Python. Because XML-RPC can be used from any language,
your use may differ. The important information is the base URL in ServerProxy, passing positional parameters, and
calling loadDevice on your proxy object.
>>> from xmlrpclib import ServerProxy
>>> url = 'http://admin:zenoss@MYHOST:8080/zport/dmd/DeviceLoader'
>>> serv = ServerProxy(url)
>>> serv.loadDevice('NEWDEVICE', '/Server/Linux')
You can check on the device with another XML-RPC call:
>>> from xmlrpclib import ServerProxy
>>> cp = 'Devices/Server/Linux/devices'
>>> url = 'http://admin:zenoss@MYHOST:8080/zport/dmd/%s/NEWDEVICE' % cp
>>> serv = ServerProxy(url)
>>> print serv.getManageIp()
8.1.3. XML-RPC Attributes
XML-RPC Attributes Description
deviceName the name or IP of the device. If it's a name it must resolve in DNS
devicePath the device class where the first "/" starts at "/Devices" like "/Server/Linux" the default
is "/Discovered"
tag the tag of the device
serialNumber the serial number of the device
zSnmpCommunity SNMP community to use during auto-discovery if none is given the list zSnmpCom-
munities will be used
zSnmpPort SNMP port to use default is 161
Device Management
41
XML-RPC Attributes Description
zSnmpVer SNMP version to use default v1 other valid values are v2
rackSlot the rack slot of the device.
productionState production state of the device default is 1000 (Production)
comments any comments about the device
hwManufacturer hardware manufacturer this must exist in the database before the device is added
hwProductName hardware product this must exist in the manufacturer object specified
osManufacturer OS manufacturer this must exist in the database before the device is added
osProductName OS product this must exist in the manufacturer object specified
locationPath path to the location of this device like "/Building/Floor" must exist before device is
added
groupPaths list of groups for this device multiple groups can be specified by repeating the attribute
in the URL
systemPaths list of systems for this device multiple groups can be specified by repeating the attribute
in the URL
statusMonitors list of status monitors (zenping) for this device default is "localhost"
performanceMonitor performance monitor to use default is "localhost"
discoverProto discovery protocol default is "snmp" other possible value is "none"
Table 8.1. XML-RPC Attributes and Descriptions
8.2. Editing Device Information
Devices can be edited through the UI but also through a programmatic interface. This how to will describe editing
device info using that interface.
8.2.1. Using a REST call
Editing device info through a rest call can be done by a simple web get. In this example we will use wget to add a
device. If you use wget don't forget to escape the "&" or wrap the URL in single quotes.
$ wget --auth-no-challenge 'http://admin:zenoss@MYHOST:8080/zport/dmd/Devices/\
Server/Linux/devices/MYDEVICE/manage_editDevice?serialNumber=MYSERIALNUM\
&tag=MYTAG'
The result of this command will change the Serial Number to MYSERIALNUM and the Tag to MYTAG for device, MYDEVICE.
8.2.2. Using an XML-RPC Call from Python
This is an example of how to edit device info using Python. Because XML-RPC can be used from any language
feel free to use your favorite. What is important here is the base URL in ServerProxy, passing named parameters,
and calling editDevice on your proxy object.
>>> from xmlrpclib import ServerProxy
>>> url = 'http://admin:zenoss@MYHOST:8080/zport/dmd/Devices/'
'Server/Linux/devices/MYDEVICE'
>>> serv = ServerProxy(url)
>>> serv.manage_editDevice('MYTAG', 'MYSERIALNUM')
Here is the signature of manage_editDevice() from Device.py
Device Management
42
def manage_editDevice( self, tag="", serialNumber="",
zSnmpCommunity="", zSnmpPort=161, zSnmpVer="v1",
rackSlot=0, productionState=1000, comments="",
hwManufacturer="", hwProductName="",
osManufacturer="", osProductName="",
locationPath="", groupPaths=[], systemPaths=[],
statusMonitors=["localhost"], performanceMonitor="localhost",
priority=3, REQUEST=None):
8.3. Deleting A Device
Devices can be deleted through the UI but also through a programmatic interface.
8.3.1. Using a REST call
Deleting a device through a rest call can be done by a simple web get. In this example we will use wget to delete a
device. If you use wget don't forget to escape the "&" or wrap the URL in single quotes.
$ wget --auth-no-challenge 'http://admin:zenoss@MYHOST:8080/zport/dmd/Devices/\
Server/Linux/devices/MYDEVICE/deleteDevice'
The result of this command will delete the device MYDEVICE.
8.3.2. Using an XML-RPC Call from Python
This is an example of how to delete a device using Python. Because XML-RPC can be used from any language
feel free to use your favorite. What is important here is the base URL in ServerProxy, passing named parameters,
and calling deleteDevice on your proxy object.
>>> from xmlrpclib import ServerProxy
>>> cp = 'Devices/Server/Linux/devices'
>>> url = 'http://admin:zenoss@MYHOST:8080/zport/dmd/%s/NEWDEVICE' % cp
>>> serv = ServerProxy(url)
>>> serv.deleteDevice()
8.4. Checking If A Device Exists
Devices can be checked for existence through the UI but also through a programmatic interface. This how to will
describe how to check if a device exists using that interface.
8.4.1. Using a REST call
Checking if a device exists through a rest call can be done by a simple web get. In this example we will use wget to
check of the existence of a device. If you use wget don't for get to escape the "&" or wrap the URL in single quotes.
$ wget --auth-no-challenge 'http://admin:zenoss@MYHOST:8080/zport/dmd/Devices/Server\
/Linux/devices/MYDEVICE'
If this command results with an exit code of 1 and a server response code of 404, then MYDEVICE does not exist
in Zenoss. If this command results with an exit code of 0 and a server response code of 200, the MYDEVICE does
exist in Zenoss.
8.4.2. Using an XML-RPC Call from Python
This is an example of how to check if a device exists using Python. Because XML-RPC can be used from any
language feel free to use your favorite. What is important here is the base URL in ServerProxy.
Device Management
43
>>> from xmlrpclib import ServerProxy
>>> cp = 'Devices/Server/Linux/devices'
>>> url = 'http://admin:zenoss@MYHOST:8080/zport/dmd/%s/NEWDEVICE' % cp
>>> serv = ServerProxy(url)
>>> try:
>>> serv.getId()
>>> exists = True
>>> except:
>>> exists = False
8.5. Exporting a Device List
To export a device list:
1. Go to the ZMI:
http://localhost:8080/zport/dmd/Devices/manage
2. Make a script object called getMyDeviceList().
3. Put the following line into the body of the script:
return [ d.id for d in context.getSubDevices() ]
4. Call it like this:
http://localhost:8080/zport/dmd/Devices/getMyDeviceList
Alternatively, enter the following line to return all device IP addresses:
return [ d.manageIp for d in context.getSubDevices() ]
You can call this method from different parts of the tree to limit the list of devices:
http://localhost:8080/zport/dmd/Devices/Server/Linux/getMyDeviceList
44
Chapter 9. Extending the Model
9.1. Add a ZenModel Relationship
The ZenRelations class allows Zope objects to form bi-directional relationships. There are four different types of
relationships possible:
ONE_TO_ONE only one object at each end of the relationship
ONE_TO_MANY classic parent-child relation, no containment objects have different primary paths
ONE_TO_MANY_CONT one-to-many containment relation (but bi-directional)
MANY_TO_MANY many objects on both ends of relationship
Figure 9.1. ZenRelations
9.1.1. One-to-One (1:1) Relationships
Example of 1:1 Server to Admin Relationship
...
from Products.ZenRelations.RelSchema import *
...
class Server(Device):
...
_relations = (
("admin" , ToOne(ToOne, "Admin", "server") ),
) + Device._relations
...
...
class Admin(TestBaseClass):
...
_relations = (
("server", ToOne(ToOne, "Server", "admin")),
)
...
...
...
The Server object is an example of a class that inherits from Device. According to this relationship there can be only
one Admin assigned to a Server and only one Server assigned to an Admin. This relationship is created by:
Extending the Model
45
Importing ToOne from Products.ZenRelations.RelSchema.
Appending a two-item tuple to the _relations attribute
The first item in the tuple is a "string" object which is the local name
The second item in the tuple is a "RelSchema" object which represents the relationship to another class. In
this case the ToOne constructor creates/returns that "RelSchema" object
ToOne constructors takes three parameters:
• The first parameter is a "type" object, "remoteType" which represents the relationship from another class.
The "type" should be of a class derived from RelSchema
• The second parameter is a "string" object, "remoteClass" which is the class name of the relative. In this
case it is again a ToOne relationship.
• The third parameter is a "string" object, "remoteName" which the remote name of itself.
Appending a complementary two item tuple to the _relations attribute in the relative class.
9.2. One-to-Many (1:N) Relationships
This is a real example which illustrates a one-to-many relationship between one Location and many Devices.
From Device.py
...
from Products.ZenRelations.RelSchema import *
...
class Device(ManagedEntity, Commandable):
...
event_key= portal_type = meta_type = 'Device'
default_catalog= "deviceSearch" #device ZCatalog
relationshipManagerPathRestriction = '/Devices'
...
_relations = ManagedEntity._relations + (
("location", ToOne(ToMany, "Location", "devices")),
)
...
From Location.py
...
from Products.ZenRelations.RelSchema import *
...
class Location(DeviceOrganizer):
...
# Organizer configuration
dmdRootName = "Locations"
portal_type = meta_type = event_key = 'Location'
_relations = DeviceOrganizer._relations + (
("devices" , ToMany(ToOne,"Device","location")),
)
...
According to this relationship there can be only one Location assigned to a Device but more than one Device
assigned to a Location. This relationship is created by:
Importing ToOne and ToMany from Products.ZenRelations.RelSchema.
Appending a two-item tuple to the _relations attribute
The first item in the tuple is a "string" object which is the local name
Extending the Model
46
The second item in the tuple is a RelSchema object which represents the relationship to another class.
RelSchema constructors takes three parameters:
• The first parameter is a "type" object, "remoteType" which represents the relationship from another class.
The "type" should be of a class derived from RelSchema
• The second parameter is a "string" object, "remoteClass" which is the class name of the relative.
• The third parameter is a "string" object, "remoteName" which the remote name of itself.
Appending a complementary two item tuple to the _relations attribute in the relative class.
9.3. Many-to-Many (M:N) Relationships
This is a real example from Device.py which illustrates a many-to-many relationship between many Devices and
many Device Groups.
...
from Products.ZenRelations.RelSchema import *
...
class Device(ManagedEntity, Commandable):
...
event_key = portal_type = meta_type = 'Device'
default_catalog = "deviceSearch" #device ZCatalog
relationshipManagerPathRestriction = '/Devices'
...
_relations = ManagedEntity._relations + (
("groups", ToMany(ToMany, "DeviceGroup", "devices")),
)
...
From DeviceGroup.py
...
from Products.ZenRelations.RelSchema import *
...
class DeviceGroup(DeviceOrganizer):
...
# Organizer configuration
dmdRootName = "Groups"
portal_type = meta_type = event_key = 'DeviceGroup'
_relations = DeviceOrganizer._relations + (
("devices", ToMany(ToMany,"Device","groups")),
)
...
According to this relationship there can be more than one Device assigned to a Device Group and more than one
Device Group assigned to a Device. This relationship is created by:
• Importing ToMany from Products.ZenRelations.RelSchema.
• Appending a two-item tuple to the _relations attribute
• The first item in the tuple is a "string" object which is the local name
• The second item in the tuple is a RelSchema object which represents the relationship to another class. In this
case the ToMany constructor creates/returns the RelSchema object.
The RelSchema constructors take three parameters
Extending the Model
47
• The first parameter is a "type" object, "remoteType" which represents the relationship from another
class. The "type" should be of a class derived from RelSchema
• The second parameter is a "string" object, "remoteClass" which is the class name of the relative. In this
case it is again the ToMany relationship.
• The third parameter is a "string" object, "remoteName" which the remote name of itself.
• Appending a complementary two-item tuple to the _relations attribute in the relative class.
9.3.1. One-to-Many (1:N) Container Relationships
Device to Hard Drives
This is a real example which illustrates a one-to-many relationship between one DeviceHW and many HardDrives
where a DeviceHW object contains HardDrives.
From DeviceHW.py
...
from Products.ZenRelations.RelSchema import *
...
class DeviceHW(Hardware):
...
meta_type = "DeviceHW"
...
_relations = Hardware._relations + (
("harddisks", ToManyCont(ToOne, "HardDisk", "hw")),
)
...
From HardDisk.py
...
from Products.ZenRelations.RelSchema import *
...
class HardDisk(HWComponent):
...
portal_type = meta_type = 'HardDisk'
...
_relations = HWComponent._relations + (
("hw", ToOne(ToManyCont, "DeviceHW", "harddisks")),
)
...
According to this relationship there can be only one DeviceHW assigned to a HardDisk but more than one HardDisk
assigned to a DeviceHW. This relationship is created by:
1. Importing ToOne and ToManyCont from Products.ZenRelations.RelSchema.
2. Appending a two-item tuple of to the _relations attribute
1. The first item in the tuple is a "string" object which is the local name
2. The second item in the tuple is a RelSchema object which represents the relationship to another class.
RelSchema constructors take three parameters
1. The first parameter is a "type" object, "remoteType" which represents the relationship from another class.
The "type" should be of a class derived from RelSchema
2. The second parameter is a "string" object, "remoteClass" which is the class name of the relative.
3. The third parameter is a "string" object, "remoteName" which the remote name of itself.
3. Appending a complementary two-item tuple to the _relations attribute in the relative class.
Extending the Model
48
Specifying the remoteClass in a Relationship
The remoteClass parameter can be specified in a relationship by two methods.
("admin", ToOne(ToOne, "Admin", "server"))
In the example above "Admin" is the remote class on the relationship. For this to work properly the module "Admin"
must be in the python path and it must contain a class named "Admin".
This behavior can be modified by using the attribute zenRelationsBaseModule. For instance if Admin was located
in the path Products.ZenModel you could set zenRelationsBase = "Products.ZenModel". Now the remote class is
in the module Products.ZenModel.Admin and the class must be Named "Admin".
If you wish to put multiple classes into one module and use them in relations you can add the class name to the end
of the remoteClass value. For instance "Admin.Test" would access the module Admin with the class Test.
If the two classes in a relation are in a different packages then you can use the fully qualified path to the
class. For instance here are the definitions of two classes in different packages: Products.ZenWidgets.Menu and
Products.ZenModel.DeviceOrganizer.
In Products.ZenWidget.Menu.py
...
class Menu(ZenModelRM):
...
_relations = (
("deviceOrg",
ToOne(ToManyCont,
"Products.ZenModel.DeviceOrganizer",
"menus")),
)
...
In Products.ZenModel.DeviceOrgaizer.py
...
class DeviceOrganizer(ZenModelRM):
...
_relations = (
("menus",
ToManyCont(ToOne,
"Products.ZenWidget.Menu",
"deviceOrg")),
)
...
9.4. Zenoss XML Schema
This XML schema describes the output of the zendump command.
<?xml version="1.0" encoding="UTF-8" ?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="link">
<xs:complexType>
<xs:attribute name="objid" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="object">
<xs:complexType>
<xs:choice>
<xs:element ref="object" />
Extending the Model
49
<xs:element ref="property" />
<xs:element ref="tomany" />
<xs:element ref="tomanycont" />
<xs:element ref="toone" />
</xs:choice>
<xs:attribute name="module" type="xs:NMTOKEN" use="required" />
<xs:attribute name="class" type="xs:NMTOKEN" use="required" />
<xs:attribute name="id" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="objects">
<xs:complexType>
<xs:sequence>
<xs:element ref="object" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="property">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:NMTOKEN" use="required" />
<xs:attribute name="visible" use="optional">
<xs:simpleType>
<xs:restriction base="xs:NMTOKEN">
<xs:enumeration value="True" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="mode" type="xs:string" use="optional" />
<xs:attribute name="setter" type="xs:NMTOKEN" use="optional" />
<xs:attribute name="select_variable" use="optional">
<xs:simpleType>
<xs:restriction base="xs:NMTOKEN">
<xs:enumeration value="lineTypes" />
<xs:enumeration value="rrdtypes" />
<xs:enumeration value="sourcetypes" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="tomany">
<xs:complexType>
<xs:sequence>
<xs:element ref="link" />
</xs:sequence>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="tomanycont">
<xs:complexType>
<xs:sequence>
<xs:element ref="object" maxOccurs="unbounded" />
</xs:sequence>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
<xs:element name="toone">
<xs:complexType>
<xs:attribute name="objid" type="xs:string" use="required" />
Extending the Model
50
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
</xs:schema>
9.4.1. object
<xs:element name="object">
<xs:complexType>
<xs:choice>
<xs:element ref="object" />
<xs:element ref="property" />
<xs:element ref="tomany" />
<xs:element ref="tomanycont" />
<xs:element ref="toone" />
</xs:choice>
<xs:attribute name="module" type="xs:NMTOKEN" use="required" />
<xs:attribute name="class" type="xs:NMTOKEN" use="required" />
<xs:attribute name="id" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
9.4.1.1. Example
<object id='deleteActionRuleWindows' module='Products.ZenModel.ZenMenuItem'
class='ZenMenuItem'>
<property type="text" id="description" mode="w" >
Delete Rule Windows...
</property>
<property type="text" id="action" mode="w" >
dialog_deleteActionRuleWindows
</property>
<property type="boolean" id="isglobal" mode="w" >
True
</property>
<property type="lines" id="permissions" mode="w" >
('Change Alerting Rules',)
</property>
<property type="boolean" id="isdialog" mode="w" >
True
</property>
<property type="float" id="ordering" mode="w" >
80.0
</property>
</object>
The object element is an XML representation of a Zope object. The example above is the XML representation of
a ZenMenuItem object.
9.4.1.2. Attributes
• id - the unique identifier for the object instance
• class - the classname of the object instance
• module - the module in which this object's class is defined
9.4.1.3. Children
• object - an object may also have objects as children
• property - (see property element section below)
• tomany - (see tomany element section below)
Extending the Model
51
• tomanycont - (see tomanycont element section below)
• toone - (see toone element section below)
9.4.2. objects
<xs:element name="objects">
<xs:complexType>
<xs:sequence>
<xs:element ref="object" />
</xs:sequence>
</xs:complexType>
</xs:element>
9.4.2.1. Example
<objects>
<object id='deleteActionRuleWindows' module='Products.ZenModel.ZenMenuItem'
class='ZenMenuItem'>
<property type="text" id="description" mode="w" >
Delete Rule Windows...
</property>
</object>
</objects>
The object element is an XML representation of a Zope object. The example above is the XML representation of
a ZenMenuItem object.
9.4.2.2. Children
• object - the objects element may also have object as children
9.4.3. property
<xs:element name="property">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:NMTOKEN" use="required" />
<xs:attribute name="visible" use="optional">
<xs:simpleType>
<xs:restriction base="xs:NMTOKEN">
<xs:enumeration value="True" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="mode" type="xs:string" use="optional" />
<xs:attribute name="setter" type="xs:NMTOKEN" use="optional" />
<xs:attribute name="select_variable" use="optional">
<xs:simpleType>
<xs:restriction base="xs:NMTOKEN">
<xs:enumeration value="lineTypes" />
<xs:enumeration value="rrdtypes" />
<xs:enumeration value="sourcetypes" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>>
9.4.3.1. Example
<property type="float" id="ordering" mode="w" >
80.0
Extending the Model
52
</property>
The property element represents a property of an object in Zope. The example above represents an "ordering"
property of an object. The value of the "ordering" property is 80.0 and is of type float.
9.4.3.2. Attributes
• id - the unique identifier of this property
• type - the datatype of the property's value
• visible - an optional boolean, a flag used to display or hide the property
• mode - read/write permission of this property
• setter - the name of the method to set this property
• select_variable - the name of the list which hold the possible values of this property
9.4.4. tomany
<xs:element name="tomany">
<xs:complexType>
<xs:sequence>
<xs:element ref="link" />
</xs:sequence>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
9.4.4.1. Example
<tomany id='devices'>
<link objid='/zport/dmd/Devices/Server/Linux/devices/MYDEVICE'/>
</tomany>
The tomany element represent a ToManyRelationship object in Zope. The example above is of the "devices" to
many relationship on an object.
9.4.4.2. Attributes
• id - unique name of the to many relationship
9.4.4.3. Children
• link - (see link element below) These links are the XML representations of the references to related objects
9.4.5. tomanycont
<xs:element name="tomanycont">
<xs:complexType>
<xs:sequence>
<xs:element ref="object" maxOccurs="unbounded" />
</xs:sequence>
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
9.4.5.1. Example
<tomanycont id='instances'>
<object id='dropbear' module='Products.ZenEvents.EventClassInst'
class='EventClassInst'>
<property type="string" id="eventClassKey" mode="w" >
Extending the Model
53
dropbear
</property>
<property type="int" id="sequence" mode="w" >
1
</property>
...
</tomanycont>
9.4.5.2. Attributes
• id - the name of the to many cont relationship
9.4.5.3. Children
• object - the tomanycont element may have objects elements as children, these subobjects are the XML repre-
sentations of these related objects
9.4.6. toone
<xs:element name="toone">
<xs:complexType>
<xs:attribute name="objid" type="xs:string" use="required" />
<xs:attribute name="id" type="xs:NMTOKEN" use="required" />
</xs:complexType>
</xs:element>
9.4.6.1. Example
<toone id='perfServer' objid='/zport/dmd/Monitors/Performance/localhost'/>
The toone element represents a ToOneRelationship on an object. The example above is a toone relationship named
"perfServer". It represents a device's relationship to only one performance server "localhost."
9.4.6.2. Attributes
• id - the name of the toone relationship of an object
• objid - the path to the related object
9.4.7. link
<xs:element name="link">
<xs:complexType>
<xs:attribute name="objid" type="xs:string" use="required" />
</xs:complexType>
</xs:element>
9.4.7.1. Example
<link objid='/zport/dmd/Devices/Server/Linux/devices/MYDEVICE'/>
The link is a reference to another object element rather than a new instance of an object element.
9.4.7.2. Attributes
• objid - is the path to the object
9.5. Zenoss Permissions
In this example we'll be adding a new permission named "Example Permission", assigning it to a method, then
checking for that permission.
Extending the Model
54
9.5.1. Adding New Permissions
1. Add the new permission to $ZENHOME/Products/ZenModel/ZenossSecurity.py
ZenossSecurity.py is a file where all the string constants for Zenoss permissions are held. By adding this line
to ZenossSecurity.py we've made a new constant that will be used to assign to a method.
ZEN_EXAMPLE_PERMISSION='Example Permission'
2. Now that we have a "name" for the permission available, we should add the permission to Zope. In $ZEN-
HOME/Products/ZenModel/ZentinalPortal.py there is a class named PortalGenerator. There is a method
named setupPermissions() defined in PortalGenerator.
Here you'll see a group of calls to manage_permissions. Add a new line to this method that adds your new
permission.
mp(ZEN_EXAMPLE_PERMISSION, [ZEN_MANAGER_ROLE, MANAGER_ROLE], 1)
The first parameter is the permission. In this example the permission being managed is
ZEN_EXAMPLE_PERMISSION. The second parameter is the list of default roles assigned to the permission.
In this example ZEN_MANAGER_ROLE and MANAGER_ROLE are set as defaults. The third argument is the
acquired flag. When the flag is set to true, the permissions will be acquired in addition to the ones specified.
3. To make your permission official you'll need to use this permission. Apply your newly added permission to a
method. See the next section on assigning permissions to a method. Your permission must be declared and
used by a method to make it a valid permission.
9.5.2. Assigning Permissions to a Method
1. Import your new permission:
from Products.ZenModel.ZenossSecurity import *
2. Import ClassSecurityInfo. In most cases we have set ClassSecurityInfo to security
from AccessControl import ClassSecurityInfo
security = ClassSecurityInfo()
3. Above the method definition add this line of code
security.declareProtected(ZEN_EXAMPLE_PERMISSION, 'exampleMethod')
def exampleMethod(self):
...
The first parameter to declareProtected() is the permission to be set on the method. In this case the permission
is ZEN_EXAMPLE_PERMISSION. The second parameter is the name of the method. In this case the name of the
method is exampleMethod().
9.5.3. Checking Links
1. To check permission on a object, call checkRemotePerm().
self.checkRemotePerm(ZEN_EXAMPLE_PERMISSION, foo)
The first parameter is the permission to check. In this case the permission is ZEN_EXAMPLE_PERMISSION. The
second parameter is the object being checked. In this case the name of the object is foo. This call will check
if foo has the ZEN_EXAMPLE_PERMISSION.
55
Chapter 10. Zenoss Daemons
10.1. Twisted Network Programming Overview
Zenoss relies heavily on the Twisted network Python libraries. Twisted provides an asynchronous, layered network-
ing stack that is used by Zenoss for daemon communications as well as for contacting devices. The main Twisted
documentation can provide a more detailed background.
One of the central concepts in Twisted is not a multi-threaded design, but an asynchronous design. This means that
it is event-driven (the next function to be called depends on what data is received) with co-operative multi-tasking
(such as a badly behaved function that sleeps or takes a long time to execute can stall an entire application). The
unit of co-operative multi-tasking is a deferred object. A simplified overview is that a Twisted program starts a bunch
of deferred tasks and then waits for timers to expire and network events to happen.
Daemons communicate with ZenHub via Twisted Perspective Broker (PB), which is a library for transferring objects
over the network. The most important PB concepts for our purposes are these:
• Methods that start with remote_ are callable from the daemons.
• There are restrictions on what type of objects can be passed back and forth between the service and the dae-
mon. Passing native Python types is supported, as well as some support for more simple objects (classes
without methods). Simple objects can be marked using the PB method pb.setUnjellyableForClass() to help
accomplish this goal.
10.1.1. Understanding NJobs, Driver and DeferredList
Writing scalable, single-threaded communications servers requires an event-driven programming approach. Small,
simple I/O steps are connected by callbacks, rather than normal control flow. For example, instead of just sending a
request and waiting for the response you have to create the request, queue it for delivery, send it when the network
flow-control says it has space, wait for the response, reading it piecemeal, as it arrives, and then correlating it to
the sent message. Fortunately, we use a comprehensive library that performs many of these steps for us, so the
underlying steps are not as small. But, once you have queued your request, you must head back to the main event
loop so that I/O from many different parts of your application can complete in a reactive manner. The fundamental
callback mechanism is the Twisted library's Deferred. There are three common tasks that our data collectors perform
in an asynchronous environment. They are:
1. Perform these tasks, in any order, and report to me when they are complete.
2. Perform this long list of tasks, but do not do more than N of them at a time.
3. Perform a sequence of related activities in the correct order.
10.1.1.1. DeferredList
Lets say you need to perform I/O requests in parallel, and you don't care which finishes first, so long as they all
complete before the next step. For this problem, we gather up the deferreds from each step as we initiate it, and we
hand them to a DeferredList. Once they have all fired (with callbacks or errbacks) the DeferredList will return a list
of the results, along with a boolean value indicating success or failure.
from twisted.internet.defer import DeferredList
d1 = task1()
d2 = task2()
d3 = task3()
d = DeferredList([d1, d2, d3])
d.addCallback(printResults)
def printResults(results):
Zenoss Daemons
56
for success, value in results:
if success:
print "Callback successful:", value
else:
print "Errback: ", value
Each task runs in parallel, completing at its own pace. This approach is useful for knowing when a number of
unrelated requests have completed. For example, fetching the initial configuration may have several requests that
are not interrelated. These may be done in parallel, so long as they all complete before collection begins.
10.1.1.2. NJobs
Each collector can overwhelm existing resources if it does not limit itself. For example, file descriptors in a process
are normally limited to approximately a thousand. Unless you change the operating system's default it is not possible
to talk to more than a thousand devices at one time if each requires its own file descriptor. So, we normally wish
to a talk to as many as we can concurrently, but not so many that we run out of local resources. NJobs takes a
callable that takes a single argument and returns a deferred, and a sequence of items, along with a value N, such
that only N of the callables are outstanding at each time.
from Products.ZenUtils.NJobs import NJobs
jobs = NJobs(10, collectDevice, devices)
d = jobs.start()
d.addCallback(printResults):
def printResults(results):
for result in results:
print "Result is", results
The callable is called on the sequence list in the order given, but each call may complete out-of-order. Therefore,
the results may also have a different order than the input sequence. NJobs prevents us from having to write a built-
in limit to each type of asynchronous collector.
10.1.1.3. Driver
The most difficult to understand of the asynchronous tools that uses Deferreds is Driver. First let's understand the
basic problem. We have a sequence of asynchronous activities we want to link together, but each step requires
some intervening computation or organization. If the activities were synchronous, they might look like this:
config = readConfig()
self.updateConfig(config)
for d in self.config:
clearStatus(d.id)
collect(self.config)
sendHeartbeat()
Each of these steps must be completed in order. Using just deferreds we might right something like this:
d = readConfig()
d.addCallback(updateConfig)
def clearStatuses(self):
d = DeferredList([clearStatus(d.id) for d in self.config])
d.addCallback(collect)
d.addCallback(heartbeat)
d.addCallback(clearStatuses)
The interleaving of synchronous calls (the for loop) and asynchronous calls twists the code around the callback
mechanism. There is a mechanism in Python that can be used to straighten out a convoluted sequence of actions
to produce a stream of results. Like a tokenizer, which uses yield to produce tokens as they have been discovered
in an input stream, Driver uses yield to produce deferreds as they come up. Driver consumes the deferreds and
resumes computation when they complete. So lets see what this code looks like when we yield a deferred whenever
we have one:
Zenoss Daemons
57
yield readConfig()
self.updateConfig(results)
for d in self.config:
yield clearStatus(d.id)
yield self.config()
yield sendHeartbeat()
What remains is very much like the normal synchronous control flow, except the result from the deferreds are
missing. The value results in the 2nd line of the example is a stand-in for some mechanism to get the results of
the last deferred that was returned by yield.
Here's the example in a more complete fragment:
from Products.ZenUtils.Driver import drive
def cycle(driver):
yield readConfig()
self.updateConfig(driver.next())
for d in self.config:
yield clearStatus(d.id)
driver.next()
yield self.config(); driver.next()
yield sendHeartbeat(); driver.next()
drive(cycle)
So, when we drive one of these deferred-generating-sequences, we get a reference to the driver. The driver keeps
the last value returned by a deferred result, so that it is available to the iterator. Construction is difficult to understand,
but understanding is not necessary to use Driver. If you have a sequence of code, where deferreds keep cropping
up and preventing your workflow from, well, flowing, you can use Driver to make flow like the synchronous version.
First, you need a generator which takes a single argument. If you don't have one, you can make one right in the
body of the function:
def f(a, b, c, d):
def inner(drive):
yield g(a, b, c, d)
drive.next()
return drive(inner)
Next, just yield the deferreds as they come up, and get the result with driver.next(). It's good to call driver.next()
even if you don't use the result, because if the result was an exception, driver.next() will throw the exception.
Finally, drive returns a deferred, so be sure to perform callback handling on it. The callback value of the deferred
is the last value from the last deferred.
drive(function).addBoth(self.handleResult)
10.1.1.4. A Simple Example
The following code is a simple example of the usage of a Twisted client / server code as well as the Zenoss driver()
code.
#! /usr/bin/env python
__doc__= """
Simple example of using ZenUtils Driver and Twisted Perspective Broker (PB).
Sums all of the numbers that are given as command line arguments by repeatedly
calling a remote add method on the server-side object.
"""
from twisted.spread import pb
from twisted.internet import reactor
import Globals
from Products.ZenUtils.Driver import drive
Zenoss Daemons
58
class Server(pb.Root):
"""
This is the server-side object.
"""
def __init__(self, port):
"""
Listen on the specified port.
@param port: the TCP/IP port to listen on
@type port: positive integer
"""
reactor.listenTCP(port, pb.PBServerFactory(self))
def remote_add(self, x, y):
"""
Add the two parameters together and return the result.
@param x: first operand
@type x: number
@param y: second operand
@type y: number
@return: the sum of x and y
@rtype: number
"""
return x + y
class Client(object):
"""
This is the client-side object.
"""
def __init__(self, port, numbers, callback):
"""
Connect to the server and drive the sum method.
@param port: TCP/IP port number on which a server is listening
@type port: positive integer
@param numbers: numbers to add
@type numbers: list of numbers
@param callback: a callable that accepts an argument
@type callback: Twisted callback object
"""
self.numbers = [int(n) for n in numbers]
self.clientFactory = pb.PBClientFactory()
drive(self.sum).addCallback(callback)
reactor.connectTCP('localhost', port, self.clientFactory)
def sum(self, driver):
"""
Get the root object. Call the remote add method repeatedly keeping
track of the total.
This is a Python iterable.
@param driver: a driver of the iterables
@param driver: Zenoss driver() class
@return: deferred to track the returned number
@rtype: Twisted deferred object
"""
yield self.clientFactory.getRootObject()
root = driver.next()
total = 0
for n in self.numbers:
yield root.callRemote('add', total, n)
Zenoss Daemons
59
total = driver.next()
def main(numbers):
"""
Assign a port. Create the client and server. Run the reactor.
@param numbers: numbers to add
@type numbers: list of numbers
"""
port = 7691
# Add the server to the reactor
Server(port)
def callback(total):
"""
A simple callback to return the total and stop the reactor
@param total: the total, as returned by the server
@param total: number
"""
print total
reactor.stop()
# Add the client to the reactor
Client(port, numbers, callback)
reactor.run()
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
main(sys.argv[1:])
else:
print 'Usage: %s <number> [number...]' % __file__
10.2. Zenoss Daemon Overview
There are a few general types of daemon types in Zenoss:
Types of Daemons found in Zenoss
zenhub Each instance of zenhub opens a connection to the ZODB. All other daemons connect to
the hub in order to receive and transmit changes to the ZODB.
modeler daemons These daemons attempt to construct a model of devices and networks using Zenoss ob-
jects, and associate components with devices to prepare for performance data collection.
collector daemons Collector daemons are concerned with gathering performance data for each of the mod-
eled components and storing the results in RRD files. The RRD data is always stored lo-
cally to the host that runs the collector daemon.
event daemons An event daemon converts messages received from devices using whatever method the
device supports, and converts the messages into Zenoss events.
zenrender A render server takes a request for an RRD graph, renders the graphic and sends the
graphic back. A render server will be found where collectors run, as the collectors generate
the RRD files.
Zenoss Enterprise users also have the option of using Distributed Collectors, which can create hubs and collectors
on different hosts in order to monitor devices. With Distributed Collectors there may be multiple zenhub daemons
(one per hub, naturally), and for a host with collector daemons there will also be a renderserver.
Zenoss Daemons
60
From a programming perspective, most daemons will choose one of the following classes:
Class Features
CmdBase Logging and option parsing
ZenDaemon Logging and option parsing, daemon
ZCmdBase Logging and option parsing, daemon, ZODB connection
PBDaemon Logging and option parsing, daemon, PB communica-
tions
10.3. zenhub: Daemon to ZODB management
The zenhub daemon (aka the Hub or ZenHub) is a single-threaded and asynchronous daemon that provides the
following features:
• Connections between daemons and the ZODB for persistent object management (for example, configuration
loading). Writes to the ZODB are synchronous operations.
• Connections between daemons and the MySQL event database for events and event management. Writes to
MySQL are synchronous operations.
• Connections between daemons and performance data in RRD files
• Pluggable Daemon Services
• User-interactive RRD graph fetching (such as renderserver functionality)
• Loading configuration
Figure 10.1. ZenHub, Daemon and the ZODB
The Hub (as of Zenoss version 2.3) can be split out some of its tasks by creating workers (a configuration file option).
Requests from collectors are farmed out to the worker processes to spread out some of the load.
Note
Propagating configuration changes and fetching RRD Data is not pushed through workers! This means that large
configuration downloads will still affect the user experience. Some sort of caching on the daemon's side may be
necessary for large sites.
10.3.1. Daemon to ZODB management
The zenhub daemon manages updates to the object database (ZODB) to any daemons that connect to zenhub
(in practice this means all Zenoss daemons). The Hub watches for changes to the ZODB database (for example,
Zenoss Daemons
61
the use of the commit() function) and initiates change notifications to any affected daemons. zenhub also provides
daemons access to the object database for loading configuration items and posting events.
10.3.2. Heartbeats and other Events
Another management function that zenhub provides is the ability to send notifications (for example, Zenoss events).
An event will be provided from the daemon to the Hub which then stores the event in the event database (ie a
MySQL table) and then the event is processed according to any mappings that match the event. In this way an
event generated by an error condition can be cleared by another event.
Each daemon should post an event when it is shutdown, so that the console is kept informed of intentional shut-
downs. However, these events should be cleared by matching start events. Start/shutdown events should only be
sent when the server is daemon-ized.
Each daemon should post a periodic Heartbeat event. If a heartbeat event is not updated the Zenoss GUI will
indicate a problem with the daemon. Ideally, a daemon only sends a heartbeat event after each successful operating
cycle (after each performance data collection). It is not acceptable to just post events in a separate thread or timer
unless that thread also does some minimal testing for internal status and health.
If the daemon cannot talk to the Hub (zenhub is down) then events are queued up. When communications are
restored the queued events are then delivered.
10.3.3. Pluggable Daemon Services
To implement these features, zenhub has a collection of Services that it is willing to provide to other daemons.
A daemon will connect and request a particular Service. ZenHub will create that Service, and send future object
change notices to the Service, which in turn can decide how best to notify the daemon. Some daemons, such as
zenping, have a very simple configuration that doesn't change very often. Others, such as the zenperfsnmp, have
a much more complex configuration that must be kept up-to-date with model changes.
Each Service is implemented as a class that zenhub can import. Using Twisted's Perspective Broker (PB) facilities,
a daemon can request that the Hub perform some action (ie a class method) and return the results to the daemon,
and vice versa. In other words, the Service acts as the interface between the daemon and the Hub. The services
directory in a ZenPack directory structure is where the Service class is kept.
10.4. ZenRender and Graphs
ZenRender provides access to RRD files (and rrdtool) stored on a remote collector from a user's browser, and
allows this even with firewalls. Zenrender can implement rendering methods via PB and HTTP.
ZenHub maintains a connection from zenrender, so an HTTP request to ZenHub and back through to the remote
zenrender is an option. zenrender can implement all the RenderServer methods via HTTP requests, too.
You can use the following default URLs to get a graph:
Default URL Description
http://hostname:8080/zport/RenderServer The Zope RenderServer (original mechanism)
http://zenoss:8090/collector ZenHub, where collector is the name of the collector de-
fined in the model. This port number can only be changed
by editing the Render hub service.
http://hostname:8091 A direct reference to zenrender at the given hostname.
The port number is configurable at each zenrender serv-
er.
Zenoss Daemons
62
10.5. Developing a Daemon
10.5.1. Command-line Options
Each daemon should support:
$ mydaemon start
This should daemon-ize the new daemon, running it forever in the background.
$ mydaemon stop
This should find the collector and stop it with a graceful shutdown.
$ mydaemon run
The new daemon should run for one cycle (if it has a cycle), and should not daemon-ize and log to stderr.
Thankfully most of this infrastructure is taken care of for you. Should you require more command-line options, here's
how you should take advantage of the existing code:
from Products.ZenHub.PBDaemon import PBDaemon
class myclass(PBDaemon)
...
def buildOptions(self):
"""Build our list of command-line options
"""
PBDaemon.buildOptions(self)
self.parser.add_option( '--newoption',
dest='dest_var', action="store_true", default=False,
help="Do something really interesting")
The option formats are as specified in the Python optparse library.
Other features taken care of with the Zenoss daemon infrastructure is reading from configuration files, the --genconf
flag (which produces a configuration file populated with all options, comments and default values) as well as the --
genxmltable flag (which produces a DocBook XML table showing command-line switches). As other features can
be added to the base class, if you follow this recommendation there are more things your daemon gets for free.
Note
The code to allow commands to get command-line option values out of a config file in $ZENHOME/etc/ currently
can only set values on lower-case options. Please be aware of this when you create new command-line options.
10.5.2. Add the Daemon Control Script
The daemons directory should contain a file with the name of your daemon (the one that should appear under the
Daemons selection in Advanced > Settings). This file is an executable shell script which should contain the following:
#! /usr/bin/env bash
. $ZENHOME/bin/zenfunctions
MYPATH=`python -c "import os.path; print os.path.realpath('$0')"`
THISDIR=`dirname $MYPATH`
PRGHOME=`dirname $THISDIR`
PRGNAME=mydaemon.py
Zenoss Daemons
63
CFGFILE=$CFGDIR/mydaemon.conf
generic "$@"
Of course, the PRGNAME and CFGFILE variables don't necessarily need to be contain the same name as the daemon.
However, keeping the same name will certainly make things much less confusing.
The mydaemon.py file is assumed to live at the base of the ZenPack.
10.5.3. Set Up ZenHub Communications
The basics of daemon communications are these.
Procedure 10.1. Daemon to ZenHub Communication Steps
1. A daemon connects to ZenHub. The raw mechanics of this are handled by the PBDaemon classes so we don't
need to explicitly code anything.
2. The daemon requests specific Services by name from ZenHub. The Services are classes either already known
to ZenHub or classes provided in the services directory in a ZenPack and are loaded by ZenHub at runtime.
3. The daemon calls remote_ methods on the Service objects from ZenHub to receive configuration information
or perform other work.
4. The Services can also call remote_ methods on the daemon to provide updates, etc.
10.5.3.1. Registering Services with the Hub
The services directory needs to be created at the base directory of your ZenPack. Included in this directory is the
__init__.py file. The __init__.py can be empty, but it must exist or any service class files cannot be loaded by
zenhub.
zenhub imports Services (a daemon-to-Hub interface class) and the daemons can then use their own Service
to perform actions. Look for the example closest to your needs from the $ZENHOME/Products/ZenHub/services/
directory as well as from other ZenPacks (such as HelloWorldZenPack or ZenJMX).
A basic Service class can be found in the Products.ZenHub.HubService.HubService class. More complex daemons
doing data collection may want to subclass Products.ZenHub.PerformanceConfig.PerformanceConfig instead to
take advantage of some additional infrastructure there.
64
Chapter 11. Add a Performance Daemon
11.1. Overview
Zenoss is designed to be an extensible platform for integrating new performance collectors. Basically, this should
be a simple matter of getting the list of devices and sending/receiving data over the network to collect new values.
Essentially, this is what every collector does.
Each collector should post values to RRD files and execute thresholds against those updates. The Python class
RRDUtil supports writing values to RRD files. The Python class Thresholds will simplify the execution of thresholds
on each RRD update.
Data collection needs to work in a wide variety of networking infrastructures, so it needs to have acceptable per-
formance in light of high latency wide-area networks. Collectors should intentionally interleave requests to multiple
devices to reduce the overall time necessary to walk the list of devices. Collectors should not overload a single
device by sending multiple outstanding requests to that device.
In order to debug collection, the collector should be capable of logging detailed debugging output at each step of
collection, as well as posting events about collection failure. In particular, logging raw values and errors from devices
helps find errors in post-processing. Any performance information about total devices collected, or total collect time
should be posted at the informational level (above debug).
Since the collectors are generally going to run long-term, cached values and other stored and pre-computed values
should be periodically purged in order to synchronize the collectors' state with the real world, as well to eliminate
possible memory leaks.
If the collector monitors device components as well as whole devices, it may be necessary to load the device
configuration information in an incremental way. If it takes 30 minutes to gather the configuration information, this is
simply too slow and unresponsive. The collector should load its configuration information incrementally, performing
collection against those devices it knows about. It can cache the configuration information persistently to provide
a larger "initial set" of configuration upon start-up.
Many collectors benefit from "pre-failing" their devices. They get the list of devices presently marked down by the
ping tester, and they skip those devices during collection. This eliminates unnecessary longer delays as collectors
run against devices that are just unreachable.
11.2. DataMaps
Zenoss divides data collection into two parts: modeling, and performance collection. During the modeling, or discover
step, the external world is sampled through a series of plug-ins. The result of the discovery step is a generic "Map":
a nested data structure that mimics the structure of the components within a device.
Add a Performance Daemon
65
Figure 11.1. Modeling Overview
For example, we can query the list of network interfaces on a device using SNMP. We will map that into a data
structure to mimic the path on the device:
{ 'os' : { 'interfaces' { 'eth0': { 'type': 'ethernetCsmacd',
'speed': ... }
{ 'eth1': { 'type': 'ethernetCsmacd',
'speed': ... }
These dictionaries of collected data are called DataMaps. There is a set of recursive functions that walk the maps
and apply the values to the device, creating components and setting values on them. In this way, a remote collector
can push updated configuration back to the central database without concern as to what the current configuration
is, and what exactly should be updated.
The Zenoss plugins are specialized to easily create these maps. Typically they consist of a single method process()
to transform SNMP query results into DataMaps. The plugin specifies the SNMP tables to be scanned, and the
process method is used to transform the results into data maps. Some plugins can test their applicability to a specific
device. For example, the plugin may only be appropriate if the device supports SNMPv2, or has a particular agent
OID. These plugins have a "test" method which is run before the plugin is used by the modeler.
SSH plugins, which are very much like SNMP plugins, transform output of various commands into data maps. For
example, the output of the Unix df command is transformed into a map to create and update file system information.
Add a Performance Daemon
66
11.3. Performance Collection
Modeling updates the object database model with information about what data to collect. As an example, if the
modeler detects three network interfaces, it creates slots for each network interface, and each of these slots is
referenced by an index. It is now up to the data collector to fill each of these slots with performance data.
When the performance collectors read their configuration, the devices are matched against templates, and each
template contains each data sources (for example, which data points (such as SNMP OIDs) and their slot to collect)
and thresholds. In addition, any information necessary to read the performance data (such as configuration prop-
erties that contain login information) is retrieved. This information is usually organized by device, and is loaded by
the collector when it is started.
When devices change configuration (and therefore change the performance data that needs to be collected), the
model must be refreshed either with an explicit selection of Model Device on the device, or by the periodic runs of
a modeler (such as zenmodeler).
11.3.1. Connecting Collectors and Services
All collectors (and the modelers) are sub-classed from PBDaemon. PBDaemon will automatically connect to zenhub and
re-connect as needed. It provides an easy-to-use Event Service.
The configuration format and API for getting and updating any specific collector will depend on the Service it uses.
There are a few caveats about forwarding configuration to collectors:
1. Change notifications are very "bursty."
2. A sequence of updates in a burst will often update the same object many times.
3. The configuration for thousands of devices can take a long time to extract. The configuration should be pushed
or pulled incrementally.
Caveats 1 and 2 mean that we often delay sending updates by several seconds to reduce the number of changes
sent. Caveat 3 makes for complex exchanges between a service and the collector. There are classes to support
delayed evaluation of configuration (Procrastinator). There is support for determining the type of object change:
the deletion of a device, the update of a template, and the update of a monitor's configuration (PerformanceConfig).
11.4. Creating a New Collector
For this section, we will contemplate a new collector that will collect ping performance data. We will want to create
a new data source type with several built-in data points, such as Average Ping Time, and Fastest Ping Time.
11.4.1. Constructor
The following example is a simple network ping-performance collector. It relies on the availability of fping to perform
the actual ping test.
class pingperf(RRDDaemon):
initialServices = RRDDaemon.initialServices + [
'ZenPacks.zenoss.PingPerf.PingConfig'
]
configCycleInterval = 20*60
pingCycleInterval = 5*60
The class pingperf is derived from a base class that supports writing to RRD files. It is a also PBDaemon, which
means that it will connect to ZenHub to fetch its configs and post events. PingConfig is the module/class that will be
loaded in ZenHub to satisfy zenperf's configuration requests. We also configure reasonable default values for two
cycles: the time between configuration refreshes and the time between ping tests.
Add a Performance Daemon
67
def __init__(self):
RRDDaemon.__init__(self, 'pingperf')
self.devices = {} # device id -> ip address
self.running = False
The constructor for this class calls the base's constructor, passing our name. We will need to hold the configuration
between cycles, so we initialize an empty configuration. If the ping testing takes longer than one configuration cycle,
we won't want to start a second test. We set a flag to note that we aren't running a ping test (yet).
When the base class is started, it attempts to connect to ZenHub and get remote references to the services is
will use. Most collectors have two services: EventService and a collector-specific service that scans the model
for configuration. Our service will be PingConfig. After the service reference are loaded, the base class calls a
connected() method.
def connected(self):
def inner(driver):
log.debug("fetching config")
yield self.fetchConfig()
driver.next()
driveLater(self.configCycleInterval, inner)
drive(inner).addCallbacks(self.pingDevices, self.errorStop)
This method uses a technique to serialize a callback chain. See the ZenUtils/Driver.py for details on how this
works. The effect is that the config is loaded with the fetchConfig() method, and the inner function is called repeat-
edly after configCycleInterval seconds.
Once the inner function completes the first time, it either calls pingDevices() on success or errorStop() on failure.
11.4.2. Getting a List of Devices
When the collector connects, and requests its config from the Service, the service will walk the list of all the devices
for that monitor, and extract out the ping data sources:
def remote_getDevices(self):
config = []
monitor = self.dmd.Monitors.Performance._getOb(self.name)
for dev in self.monitor.devices():
for templ in dev.getRRDTemplates():
dataSources = templ.getRRDDataSources('Ping')
if dataSources:
break
else:
continue
config.append(
(dev.id, # name of the device
dev.getManageIp(), # the IP to ping
dev.getThresholdInstances('Ping')
# any thresholds on the ping
)
)
To make this configuration load incremental, the Service can send just the name of the devices to load, and then
the collector can use a different method to load the configuration of each device at a later time. For such a simple
configuration, it may not be worth the extra complexity.
When this code is placed into a class that is a sub-class of HubService, it can be loaded by name, when the collector
loads it services. PBDaemon will automatically connect you to this service, if the name of the service is provided
in the class configuration.
The call to get this configuration in our new collector looks like this:
d = self.getService('some.package.PingService').callRemote('getDevices')
Add a Performance Daemon
68
d.addCallback(self.startCollection)
Note
1. PBDaemon has already connected you to the service some.package.PingService class.
2. getDevices becomes remote_getDevices in the hub.
3. The protocol for getting configurations is anything you like: you can control both sides of the communications.
4. Requests and responses are asynchronous and will involve callback objects.
5. The communications are heavily dependent on the Prospective Broker (PB) library in Twisted. Please refer
to the Perspective Broker (PB) documentation for how the calls to remote objects work.
11.4.2.1. Thresholds
As each collector reads updated performance data it will evaluate any thresholds associated with those updates.
The classes representing those thresholds must be loaded before the thresholds may loaded evaluated. So, each
collector asks ZenHub for the names of all of the thresholds that can be monitored and imports them for future use.
The management of Thresholds within the collector is complex. There exists a class (Thresholds) to manage the
thresholds and transform performance updates into events.
11.4.2.1.1. Complex Thresholds
A complex threshold allows Zenoss to produce an event:
• When user time and system time is over 80%
• When value A is 80% of value B
• On a different RRD consolidation function from AVERAGE
• When a file system is X% full, and a critical event is Y% full
Figure 11.2. Complex Thresholds
Add a Performance Daemon
69
Thresholds are not “min/max value checkers” but “transformers of values into events”. As new values come in,
the Threshold will look at the value and determine if an event is warranted. Because of the inheritable template
mechanism, we have two separate tasks for Thresholds. The first is to represent the configuration for a threshold
within the template. A value like “80” in the case of “File System at 80% full” is part of the configuration. However,
when applied to a context, such as file system “C:\\” on device “WINXYZ” the value becomes “96000 blocks”. The
value “96000 blocks” needs to transfer from the Zenoss object model, to the collector, so that values can be evaluated
with the given context, without referring to the entire object model.
This leads us to separate thresholds into two components: one that hold the configuration and user intent, and
another that can travel as part of the collector configuration to the collector. This “Threshold with Context” object
is then executed when new values for data points are collected. The first type of threshold (for configuration) is
called ThresholdClass, and the second type, which evaluates a value with context is called a ThresholdInstance.
The Zenoss data model will load ThresholdClass classes from Zenoss and installed ZenPacks. These objects are
responsible for creating the ThresholdInstance objects that are sent via the collector configuration for evaluation in
the collector. Templates refer to derived versions of ThresholdClass, which when given a context, create Thresh-
oldInstance objects.
To reduce the effort when writing a performance collector, support classes are used to hold ThresholdInstances
and map updates to data points into threshold evaluation and event generation. The classes MinMaxThreshold and
MinMaxThresholdInstance replaced the previous Threshold and flattening mechanism defined for data points and
collectors in Zenoss version 2.0.X.
Presently, collectors are generally ignorant of context (device, or component), and almost certainly ignorant of
DataSources and DataPoints. They are given the parameters necessary to fetch a value and store it into an RRD file.
ThresholdsInstances wish to work on distinguished DataSource/DataPoint names within a context. So, to map from
RRD files back to Thresholds, we use the RRD filename. When a collector updates a file, it notifies the Thresholds
class (the utility class for all collectors to hold threshold information). This class maintains a mapping of file names
to Threshold and DataPoint. Eventually, it might be worth translating the collectors so that they know about context
and DataPoint.
Known Problems with Complex Thresholds
To send classes from Server to Client, the client has to expect and approve them. We will need to transfer
the list of approved ThresholdInstance sub-classes before a client can load those thresholds. The collector
will then have to approve and import these sub-classes at start-up.
11.4.3. fetchConfig()
Let's look at fetchConfig():
def fetchConfig(self):
'Get configuration values from ZenHub'
def inner(driver):
yield self.model().callRemote('getDefaultRRDCreateCommand')
createCommand = driver.next()
yield self.model().callRemote('propertyItems')
self.setPropertyItems(driver.next())
self.rrd = RRDUtil(createCommand, self.pingCycleInterval)
yield self.model().callRemote('getThresholdClasses')
self.remote_updateThresholdClasses(driver.next())
yield self.model().callRemote('getCollectorThresholds')
self.rrdStats.config(self.options.monitor,
Add a Performance Daemon
70
self.name,
driver.next(),
createCommand)
devices = []
if self.options.device:
devices = [self.options.device]
yield self.model().callRemote('getDevices', devices)
update = driver.next()
if not isinstance(update, dict):
log.error("getDevices returned: %r" % update)
else:
self.devices = update
return drive(inner)
Here the same drive/inner technique is used to serialize a bunch of asynchronous remote method calls. The base
class provides a method called model() which returns a remote reference to the collector-specific configuration class.
We call several remote methods, most of which are inherited from a base ZenHub service class.
We must get the default RRD create command. Then we copy the collector properties, which provides updated
values for pingCycleInterval and configCycleInterval. In order to execute thresholds, we need to know the set of all
threshold classes and get them imported. After the threshold classes are installed, we have to get the thresholds for
this collector. These thresholds do not belong to the data points to be collected (ping response time), but for values
like "total cycle time" that are based on the collectors performance.
Finally we call the remote method getDevices() which returns a mapping of device id to IP address. We make
allowances for the simple one-device invocation:
pingperf -v 10 -d someDevice
11.4.4. Collector's ZenHub Service
Here's our ZenHub service:
from Products.ZenHub.services.PerformanceConfig import PerformanceConfig
class PingConfig(PerformanceConfig):
"""
A very simple service for fetching device data
"""
def getDeviceConfig(self, device):
return (device.id, device.getManageIp())
def sendDeviceConfig(self, listener, config):
listener.callRemote('updateDevice', config)
def remote_getDevices(self, devices):
result = {}
for d in self.config.getDevices():
if not devices or d.id in devices:
result[d.id] = d.getManageIp()
return result
Most of the implementation for this class is in the base class. The base class determines the devices affected when
database changes occur. It then uses the methods getDeviceConfig and sendDeviceConfig to figure out how to
send the changes to the collector.
11.4.5. Miscellaneous Functions
Back to the collector, here are the methods that are called by ZenHub to update the collector with changes:
Add a Performance Daemon
71
def remote_deleteDevice(self, doomed):
log.debug("Async delete device %s" % doomed)
try:
del self.devices[doomed]
except KeyError:
pass
def remote_updateDevice(self, cfg):
log.debug("Async config update for %s", cfg.name)
d, ip = cfg
self.devices[d] = ip
11.4.6. Collect the Performance Data
The only method left in our simple collector is to actually ping some devices, post the timings to a configuration file,
send any resulting events, and send a heartbeat.
def pingDevices(self, ignored=None):
def inner(driver):
reactor.callLater(self.configCycleInterval, self.pingDevices)
if not self.options.cycle:
self.stop()
if self.running:
log.error("Ping is still running")
return
self.running = True
log.debug("Pinging %s..." % (" ".join(self.devices.keys())[:100]))
start = time.time()
revMap = dict([(ip, d) for d, ip in self.devices.items()])
fd, fname = mkstemp()
fp = os.fdopen(fd, "w")
log.debug("Writing devices to tempfile %s." % fname)
fp.write('\n'.join(revMap.keys()) + '\n')
fp.close()
from twisted.internet.utils import getProcessOutput
fping = os.path.join(os.path.dirname(__file__), "fping.sh")
log.debug("starting %s" % fping)
yield getProcessOutput(fping, (fname,))
log.debug("fping returned: %s" % driver.next())
for line in driver.next().split('\n'):
if not line: continue
match = parseLine.match(line)
if not match:
log.debug("%s does not match expected output" % line)
continue
ip = match.group(IP)
ms = float(match.group(MS))
if not revMap.has_key(ip):
continue
device = revMap.pop(ip)
path = 'Devices/%s/ping_time' % device
ms = self.rrd.save(path, ms, 'GAUGE')
for ev in self.thresholds.check(path, time.time(), ms):
self.sendThresholdEvent(**ev)
os.unlink(fname)
self.heartbeat()
cycle = self.pingCycleInterval
self.rrdStats.gauge('devices', cycle, len(self.devices))
self.rrdStats.gauge('down', cycle, len(revMap))
self.rrdStats.gauge('cycleTime', cycle, time.time() - start)
d = drive(inner)
def clearRunning(arg):
Add a Performance Daemon
72
self.running = False
if isinstance(arg, Failure):
log.error("Error pinging devices: %s" % (arg,))
return arg
d.addBoth(clearRunning)
return d
This is a long method, so let's take it in parts. Let's take everything outside of the inner() function:
def inner():
# ....
d = drive(inner)
def clearRunning(arg):
self.running = False
if isinstance(arg, Failure):
msg = "Error occurred in pingperf collection: %s" % (arg.value,)
self.sendEvent(WARNING_EVENT, summary=msg)
return arg
self.running = True
d.addBoth(clearRunning)
return d
Again we are using the same drive/inner approach to serialize asynchronous calls. We also want to track the fact
that we are running the inner method so that we can detect cases where our collection cycle is taking too long. The
clearRunning() function is added to the callback chain to ensure that the running flag is reset however the inner
function completes. It was also a convenient place to report on any errors. Here's the definition of WARNING_EVENT
to remove any mystery about its value:
The following is a constant definition used to send an event if the collector has an error:
WARNING_EVENT = dict(eventClass=Status_Ping,
component="ping",
device=socket.getfqdn(),
severity=Warning)
The inner function does all the work:
def inner(driver):
reactor.callLater(self.configCycleInterval, self.pingDevices)
if not self.options.cycle:
self.stop()
if self.running:
log.error("Ping is still running")
return
This bit of code controls the ping cycle. By starting the timer call chain immediately we are ensured to repeat the
call in the future even if an error occurs or the collection takes too long.
log.debug("Pinging %s..." % (" ".join(self.devices.keys())[:100]))
start = time.time()
revMap = dict([(ip, d) for d, ip in self.devices.items()])
fd, fname = mkstemp()
fp = os.fdopen(fd, "w")
log.debug("Writing devices to tempfile %s." % fname)
fp.write('\n'.join(revMap.keys()) + '\n')
fp.close()
Our implementation for pinging all the devices is farmed out to an external process (fping). So we write a config file
for fping (a list of IP addresses) into a temporary file. Next, we run fping and collect the results:
from twisted.internet.utils import getProcessOutput
fping = os.path.join(os.path.dirname(__file__), "fping.sh")
Add a Performance Daemon
73
log.debug("starting %s" % fping)
yield getProcessOutput(fping, (fname,))
log.debug("fping returned: %s" % driver.next())
The next loop parses each line of output using a regular expression:
log.debug("fping returned: %s" % driver.next())
for line in driver.next().split('\n'):
if not line: continue
match = parseLine.match(line)
if not match:
log.debug("%s does not match expected output" % line)
continue
ip = match.group(IP)
ms = float(match.group(MS))
if not revMap.has_key(ip):
continue
When a match is found, we determine the device from the IP address and post the value to an RRD file:
device = revMap.pop(ip)
path = 'Devices/%s/ping_time' % device
ms = self.rrd.save(path, ms, 'GAUGE')
We use the resulting value (which may have been averaged in with other data from the RRD file) to check thresholds:
for ev in self.thresholds.check(path, time.time(), ms):
self.sendThresholdEvent(**ev)
Finally, we remove the temporary file, send a heartbeat, and report statistics on the total number of devices, the
devices that did not report, and the total time to process the device list.
os.unlink(fname)
self.heartbeat()
cycle = self.pingCycleInterval
self.rrdStats.gauge('devices', cycle, len(self.devices))
self.rrdStats.gauge('down', cycle, len(revMap))
self.rrdStats.gauge('cycleTime', cycle, time.time() - start)
74
Chapter 12. Adding a Device Type
In this example we'll add platform support for AIX, which uses vendor extensions to store MIB data which Zenoss
does not understand. To simplify things a little, we'll say that our Zenoss server name is zenoss1.
12.1. Overview
Adding support for a new platform can be broken down into a number of easily-defined steps:
• Add the platform-specific MIB to make it easier to find items to collect SNMP information and map numeric
OIDs to names.
• Add a device organizer for the platform to create a tidy place to store platform-specific information.
• Create modelers to gather information that does not change often (such as network cards or file system names)
• Create performance data collectors which will be used to gather current usage statistics (how full the file system
is now).
• Create templates which will be used to store the results from the data collectors and use the data for graphing.
This also allows us to set thresholds so that we can generate events when certain conditions are met (such as
when the file system is 95% full).
• Create event mappings to create reasonable responses to events coming from the devices. Additionally, if the
new device warrants it, create a new event organizer to manage new events.
If the data is collected through an API or network protocol that Zenoss doesn't natively support, it may be necessary
to create a daemon that understands that protocol. This daemon might allow Zenoss to model, collect performance
data and event information, and then store that information.
12.2. Add the MIB
MIBs are used by Zenoss as a way to convert trap output from numeric OIDs to named OIDs. Once you add the
MIB it should be easy to point your device's trapsink to the Zenoss server and from the Zenoss server convert the
traps into Zenoss events.
The AIX MIB which is stored in the /usr/lib/samples/snmp/aix.my MIB file on any AIX server. Copy the MIB file
to your Zenoss server and add it with the command:
zenmib run $ZENHOME/share/mibs/site/aix.my
Verify that the MIB is in the management page at:
http://zenoss1:8080/zport/dmd/Mibs
To add a Python-ized MIB (saved from a previous run of zenmib) to the DMD, use the --evalSavedPython flag.
For example:
zenmib run -v10 --evalSavedPython=/tmp/isis.py
12.3. Add a Device Organizer
If you want to create a device organizer so that it is easy to differentiate between other types of devices and the
type that you're adding, feel free to do so. In the case of AIX, there are a couple of types of setups:
Generic AIX Definitions
Standalone This describes the case where the entire pSeries server is dedicated to running
one instance of AIX.
Adding a Device Type
75
Logical PARtition (LPAR) Some AIX pSeries servers are capable of running multiple instances of AIX. An
AIX instance (LPAR in IBM speak) is equivalent to a VMware image.
Frames AIX LPARs are hosted on physical hardware (ie a pSeries server), which is re-
ferred to as a frame. These frames are capable of being run as either a stan-
dalone server or as a bunch of LPARs. The frame is like a VMware host.
Virtual IO (VIO) Server A VIO server is a special LPAR that allows you to consolidate IO hardware (eg
Ethernet, Fibre Channel cards) and share virtualized hardware with other LPARs.
This is one of the key technologies required in order to perform VMotion-style
activities for AIX LPARs.
A separate server (called a Hardware Management Console (HMC)) is used to manage standalone devices, frames
and LPARs (including VIO servers). The HMC is actually a Linux server with a custom configuration to support AIX.
In this example, we'll just add the AIX parts and ignore the HMC.
Add a device class for AIX in the /Devices/Server/AIX class:
1. From Infrastructure > Devices, select Server in the tree view.
2. click (below the tree view) to add a device class.
The Add Device Class dialog appears.
3. Enter a name and description, and then click Submit.
Under the newly created /Server/AIX organizer, repeat the previous steps to create the LPAR class. Under that
class, create a VIO class.
In this newly created scheme, we're intending on putting standalone servers and frames in the /Server/AIX class,
any LPARs in the /Server/AIX/LPAR class, and any VIO servers (which are a special type of LPAR) under the /
Server/AIX/LPAR/VIO class. If we wanted to have each frame contain its own tab showing the LPARs that it hosts,
we would need to create new ZenModel objects (complete with relations), instantiate them at the base of /Server/AIX
and then write more ZPTs to handle our custom behaviors.
Another situation where we might be forced to write our own device class Python code is where we want to add
properties that don't exist in other devices. For instance, we may want to record whether or not a Fibre Channel
device supports N-Port ID Virtualization (NPIV). This extra property would need to be subclassed from the ZenModel
class and the object initialized from within our ZenPack's __init__.py file.
12.4. Create a Modeler
When you navigate to a particular host, select Modeler Plugins from the left panel, and then select Model Device
from the Action menu, the system runs all of the associated modelers. What we need to do is copy and customize
an existing modeler plugin from $ZENHOME/Products/DataCollector/plugins/zenoss/snmp and then add that plugin
to our list of plugins that our platform's device class will use.
We'll start with creating a Filesystem modeler plugin. We'll copy the HRFileSystemMap plugin and call our plugin
AIXFileSystemMap.py. Using the information in the MIB, we can find the place where it stores the list of file systems.
Name Required? Description
condition() N Returns True or False to indicate whether or not to run the other functions
preprocess() N This will get called before the process() function
process() Y This is the actual function that processes any information retrieved from a
query and converts it into a format suitable for updating the device model.
Table 12.1. Modeler Functions
Adding a Device Type
76
12.4.1. Verify the SNMP connectivity and OIDs
First, verify that your server's SNMP daemon is functional and that you have the correct SNMP version and creden-
tials. We'll assume that we're using SNMP version 1 and are using the public community, and that your new host
will allow connections from our Zenoss server.
Run the snmpwalk command from the Zenoss monitoring server:
snmpwalk -v1 -c public myaixbox.example.com 1.3.6.1.4.1.2.6.191.1 | head
This produces a lot of output that we've truncated to save patience and space.
SNMPv2-SMI::enterprises.2.6.191.1.1.1.0 = INTEGER: 5
SNMPv2-SMI::enterprises.2.6.191.1.1.2.0 = ""
SNMPv2-SMI::enterprises.2.6.191.1.1.3.0 = INTEGER: 2
SNMPv2-SMI::enterprises.2.6.191.1.1.4.0 = Gauge32: 0
SNMPv2-SMI::enterprises.2.6.191.1.1.5.0 = INTEGER: 0
SNMPv2-SMI::enterprises.2.6.191.1.1.6.0 = INTEGER: 2
SNMPv2-SMI::enterprises.2.6.191.1.1.7.0 = STRING:
"The current used percentage 93 of the file system /mnt has gon"
SNMPv2-SMI::enterprises.2.6.191.1.1.9.0 = INTEGER: 0
SNMPv2-SMI::enterprises.2.6.191.1.1.10.0 = INTEGER: 0
SNMPv2-SMI::enterprises.2.6.191.1.1.11.0 = INTEGER: 0
If you do not see output like that above, nothing else will work. Find the issue and fix it.
The Zenoss community Web site has a ZenPack with a graphical MIB browser that might help for these steps.
12.4.2. Common SNMP Issues
Following is a list of some common reasons why snmpwalk may not return any data:
• SNMP daemon on the remote system is not running.
• SNMP daemon on the remote system has different security credentials than what you are using (for example,
version 1 instead of version 2c, wrong community name).
• SNMP daemon on the remote system allows connections only from certain IP addresses or IP address ranges,
and the Zenoss server does not meet that criteria.
• SNMP daemon on the remote allows queries only to certain portions of certain MIBs, and you have specified
something not allowed by that policy.
• Firewall or firewalls between the Zenoss server and the remote system to not allow UDP or SNMP traffic.
• Firewall on the Zenoss server does not allow UDP or SNMP traffic outbound or inbound.
• Firewall on the remote system does not allow UDP or SNMP traffic outbound or inbound.
As a first sanity check, try the snmpwalk command on the remote host. For example:
snmpwalk -v1 -c public localhost 1.3.6.1.4.1.2.6.191.1 | head
12.4.3. Modeler Code
Multiple modelers for different components of a system can be created, or one huge modeler for everything can
be created. Smaller modelers are preferred for maintenance reasons. The following modeler is for the file systems,
and would live in the modeler/plugins/ directory of your ZenPack.
Python requires that __init__.py files be in both the modeler/ and the modeler/plugins/ directories. If they are
missing your modeler will not load.
__doc__ = """AIXFileSystemMap
Adding a Device Type
77
This modeler determines the filesystems on the device and updates
appropriately. It is up to the monitoring template that must be
named 'Filesystems' to collect the actual performance data
(eg free/available blocks).
"""
import re
from Products.ZenUtils.Utils import unsigned
from Products.DataCollector.plugins.CollectorPlugin import SnmpPlugin, \
GetTableMap
from Products.DataCollector.plugins.DataMaps import ObjectMap
class AIXFileSystemMap(SnmpPlugin):
maptype = "FileSystemMap&