Scott's Batch Loader

My story

So, how did I get here? I initially spent several months getting things loaded into Zenoss v. 2.2.0 the first time. This wasn't zenoss's fault, I was doing a whole bunch of "make the servers more managable" stuff. It was running semi-well for a few months, then Zenoss pretty well refused to allow me to add new servers. So, I dumped stuff out, in several formats, backed things up, and uninstalled version 2.2.0, and installed 2.2.3. I didn't want to attempt to "upgrade".

I then read up, a lot, on how to import stuff into zenoss, and was utterly unimpressed at both the number of ways to do it, and at the effort put forth into making this type of thing ready for the real world to use. Kudos for having more than one API, minus several million for not expounding how to use any of them for more than simple "here's a server name, add it to zenoss" examples.

So, I learned a bit of python, learned to both love and hate zendmd, and after about a week of futzing, I finally have something that I'm able to use to reload my servers.

Update 9/10/2008: Fabio Paracchini reported the posted script had errors; indeed the wiki ate everything in <angle brackets>. This has now been fixed.

The Script

#!/usr/bin/env python
import Globals, re, string
from Products.ZenUtils.ZenScriptBase import ZenScriptBase
from transaction import commit

dmd = ZenScriptBase(connect=True).dmd
rexDevInfo = re.compile(r'^\s*(?P<devname>[^,]*),\s*(?P<devType>[^,]*),\s*(?P<ipAddr>[^,]*),\s*\"(?P<user>[^\"]*)\",\s*\"(?P<pass>[^\"]*)\",\s*\[(?P<groups>[^\]]*)\]$')
rexGroups = re.compile(r'^\s*\'(?P<group>[^\']*)\',*\s*(?P<rest>.*)$')

for line in file('devices-to-add.txt'):
    m = rexDevInfo.match(line)
    devname=m.group('devname')

    if(m.group('devType') == 'Linux'):
       d = dmd.Devices.Server.Linux.createInstance(devname)

    if(m.group('devType') == 'Windows'):
       d = dmd.Devices.Server.Windows.createInstance(devname)

    d.setManageIp(m.group('ipAddr'))
    d.setSystems(m.group('devType'))
    d.zWinUser = m.group('user')
    d.zWinPassword = m.group('pass')

    grplist = []
    if(bool(m.group('groups'))):
       groups = rexGroups.match(m.group('groups'))
       while(bool(groups.group('rest'))):
          grplist.append(groups.group('group'))
          groups = rexGroups.match(groups.group('rest'))

       if(bool(groups.group('group'))):
          grplist.append(groups.group('group'))

    d.setGroups(grplist)

    commit()
    dmd.DeviceLoader.loadDevice(devname)
    commit()
    d=dmd.Devices.findDevice(devname)
    d.collectDevice()
    commit()

The data file format

server name, system type, ip address, zWinUser username, zWinPassword password, [list of groups this server should belong to]

here's some semi-real world data:

axis-dev1, Linux, 10.9.8.56, "", "", ['/Dev', '/Axis', '/Physical Servers']
axis-dev2, Linux, 10.9.8.57, "", "", ['/Dev', '/Axis', '/Physical Servers']
axis-dev3, Linux, 10.9.8.58, "", "", ['/Dev', '/Axis', '/Physical Servers']
axis-qa1, Linux, 10.9.8.46, "", "", ['/QA', '/Axis', '/Physical Servers']
ks-qa-ctx1-st, Windows, 10.9.8.121, "qa2003\zenoss", "myZenossPassword", ['KS', '/QA', '/VMwareGuests']
qa-xp1, Windows, 10.9.8.82, "qa2003\zenoss", "myZenossPassword", ['/QA', '/VMwareGuests']
al-deploy-win1, Windows, 10.9,8.153, "qa2003\zenoss", "myZenossPassword", ['/AL', '/QA', '/VMwareGuests']
al-qa-ctx1-st, Windows, 10.9.8.174, "qa2003\zenoss", "myZenossPassword", ['/AL', '/QA', '/VMwareGuests']
al-qa-jp1-st, Windows, 10.9.8.175, "qa2003\zenoss", "myZenossPassword", ['/AL', '/QA', '/VMwareGuests']

How to modify this to your own purposes

So, this is semi-easy for those hard core coders out there to pick up from here and do what ever they want with it. For the mere mortals out there, I'll attempt to make it a bit easier. First you need to understand how this script works.

The Regular Expressions

The most difficult thing to understand about this script is probably the regular expressions. If you're new to regular expressions, go somewhere like this to learn about them. I'll try to explain what's happening semi-quickly.

First, the constructs that look like this:

(?P<name>[^,]*)

The expression "[^,]*" means to collect as many characters that are not commas as it can. A better way of thinking about it is to say "collect everything until you see a comma".

The expression ",\s*" says to match the comma we stopped at, and then any and all white space characters.

The initial "^" anchors the expression at the beginning of the line, so it can't start matching somewhere in the middle

Putting this stuff together, and the regular expression rexDevInfo says this:

• ^\s*(?P<devname>[^,]*)
  At the beginning of the line, ignore whitespace, grab the device name, end as soon as you see a comma
• ,\s*(?P<devType>[^,]*)
  ignore the comma and any whitespace that may or may not be there, collect the device type, end as soon as you see a comma
• ,\s*(?P<ipAddr>[^,]*)
  ignore the comma and whitespace, collect the ip address, end at a comma
• ,\s*\"(?P<user>[^\"]*)
  ignore the comma and whitespace, there must be a double quote, collect the username, end at the next double quote
• \",\s*\"(?P<pass>[^\"]*)
  ignore the double quote, the comma, and whitespace, there must be a double quote, collect the password, end at the next double quote
• \",\s*\[(?P<groups>[^\]]*)
  ignore the double quote, the comma, and whitespace, there must be a left square bracket, collect the groups as one long string, ending at the first right square bracket found
• \]$
  The right square bracket must be the last character on the line

The regular expression rexGroups says this:

• ^\s*\'(?P<group>[^\']*)
  At the beginning of the group string, ignore whitespace, find a single quote, collect a group name, end at the next single quote
• \',*\s*(?P<rest>.*)$
  ignore the single quote, a comma, and any whitespace, then collect everything else (the rest).

The logic of the script

So, the rest of the script uses the regular expressions to split each line up into the appropriate parts, and uses those parts to create a device, and populate the device's information. The only semi-tricky part is the groups area, since the initial regular expression just grabs all the groups together, and we want to be able to parse an arbitrarily long list of groups, we need a loop to grab each individual group string from the large group list.

The boolean checks are there to make sure we aren't dealing with edge cases where, maybe we have no groups, or maybe we only have one group. Once all the groups have been collected the group property is set.

Then all the information is committed to the database, the device is initially loaded, then collected.

I do not know whether all the commits are needed, nor whether it's necessary to re-find the device after loading it, but I know it works. I wasn't that interested in finding the minimal amount of work needed to get it to work correctly. If you know, or want to find the minimal amount needed, shoot me an email once you figure it out, and I'll fix the script above.

Modifying it

• you first need to know what your text file data format will be
• then you'll need to modify the regular expressions to break things apart
• then you'll need to modify the logic to do what you want with the data you've read from the file

Suggested improvements

Here's a list of things that I've thought of that one might want to do:

• Allow the datafile name to be specified on the command line.
  I'm just a beginner in python, I haven't even looked to figure this out yet
• Allow a list of "systems" like I allow a list of groups
• Set the snmp strings
• Set the zIcon
• Set zFileSystemMapIgnoreNames
• Set Hardware and/or OS information

There are LOTS of options...

I hope this helps those who need it get started on writing a script that does what they want it to do. Please feel free to post improvements, (email me if you want me to do it for you), and let me know if you find this useful/helpful.

-Scott L. Miller