Adventures in Puppetry!

My first attempt at using Puppet to automate a server build was not as easy as Id hoped. The goal seemed simple enough: Build a RedHat server running Solr 4. What I didnt know was that this particular path is not as well-traveled as, say, running WordPress on Ubuntu. Youll notice I wrote “running Solr 4″ rather than Tomcat or Jetty. When I began this quest I didnt care which container held Solr. After a quick discussion with Eric Pugh I found out that, unofficially at least, the core Lucene and Solr developers recommend Jetty 6 for running SolrCloud. The reasoning was easy enough to understand: Jetty 6 is packaged with Solr and thats what gets tested the most. Since SolrCloud is a very new feature I was inclined to go with whats tested. So from that point on I focused on installing Jetty 6. Getting Puppet installed was easy. We use Amazon Web Services a lot, so the first thing I did was log into AWS and instantiate a RedHat Enterprise Linux 6 (RHEL6) virtual machine. Thats the RHEL version that shows up in the quick launch screen, so thats what I picked. Latest and greatest, right? Heres the Puppet install distilled:

sudo rpm -ivh vi /etc/yum.repos.d/puppetlabs.reposudo yum updatesudo yum install puppetsudo puppet resource cron puppet-apply ensure=present user=root minute=1 command='/usr/bin/puppet apply $(puppet --configprint manifest)'

Well that was easy! In fact, I could have gotten that far by using one of the sample Cloud Front templates AWS provides. My next step was to see if PuppetForge had a Jetty 6 module: Nope! But I did see that there was a Solr module, but it was only working on Ubuntu. To see why, I went to the projects GitHub repository and browsed the init.pp to see if I could find anything Ubuntu-specific. The only thing I found was a reference to a package called “solr-jetty”. That meant that there was an Ubuntu package called solr-jetty available, but not one for RHEL6. To make some forward progress I decided to try this out on Ubuntu so at least our development environments could be automated. What I found out was that the solr-jetty package uses jsvc to launch Jetty, and jsvc is a binary executable thats dynamically linked to a shared library called “”. My Ubuntu VM only had something called “”, and no, it wasnt compatible. Fine. I didnt want to run on Ubuntu anyway. From there I spent some time searching for and checking out the various Jetty RPMs available for RHEL6, but they only included the APIs. What I really wanted was an RPM that did operational things like create a Jetty user and init.d service. The Red Hat Yum facilities dont have it, but I found a package on JPackage that fit the bill: Plain ol jetty6 RPM. But JPackage doesnt work on RHEL6! This might have been a temporary bug in their site mirrors, but it kept me from getting what I wanted. Fine. Ill run RHEL5. And then something magical happened: Things started working! RHEL5 doesnt include wget, so I tried installing it through puppet:

puppet module search wgetpuppet module install maestrodev-wgetpuppet apply -e 'include wget' 

And it worked! Next I hooked up JPackage:

wget jpackage50.repo

I had to enable the distro-specific repository, but when I automate all this Ill just install a pre-edited jpackage50.repo as part of the manifest. The Jetty Puppet modules I found were only available on GitHub and not PuppetForge. I briefly tried using one of the git modules available on PuppetForge. I chose one that seemed the simplest and installed it to my locally available modules:

puppet module search gitpuppet module install puppetlabs-gitpuppet apply -e 'include git'

This didnt work, but that was probably due to my newbie approach in the last line rather than putting something in site.pp. I decided to uninstall that module and just pull down the zip file I needed manually:

puppet module uninstall puppetlabs-gitwget master master.zipunzip master.zipmv puppet-jetty-master /etc/puppet/modules/jettypuppet apply -e 'include jetty'

This spat out a bunch of errors related to the particular Java package RHEL5 came with: 1.4.2-gcj. Wow: I didnt realize RHEL5 was that old! Now, the GCJ Java package is a dependency for a few other packages so Puppet couldnt uninstall it without some help. It didnt look like I needed those other packages, so rather than risk running into dependency hell by having Puppet uninstall them and then repeating for every other dependency, I decided to use Yum to uninstall:

yum erase java-1.4.2-gcj-compatpuppet resource package java-1.4.2-gcj-compat ensure="absent"

Im not sure if the last line did anything to change the state of that Java package in Puppets internals, but it did show me that Puppet believed Java was absent. Finally:

puppet apply -e 'include jetty'

Puppies and rainbows! It worked! Then the icing on the cake:

puppet resource service jetty6 enable="true" ensure="running"

I poked around the running processes, etc and var/log folders, just to see if it looked like Jetty was installed properly. Then I hit localhost:8080 with curl and got a nice 404 error to stdout and a log entry in the request log. So the road was rocky and filled with dead-ends, but it worked. I saw a few comments along the way from people who were hosting their own Yum repository for deployments. At the time I didnt want to add that complexity to our environment, but now it looks like the best way to proceed if we want to run Jetty on RHEL6. As for running it on Ubuntu, well, thats a challenge I can safely pass on for now.

1/25/2013 Edit: It looks like theres an even easier way to install JPackage with Puppet:

puppet module install yguenane-jpackagepuppet apply -e "include jpackage"puppet resource yumrepo jpackage-rhel enabled="1"