Playing with Hawt
We’ve been using Apache Camel on a number of projects over the past two years, and it’s worked very well for the fairly simplistic use cases that we’ve needed, primarily taking incoming data files and reprocessing them into a format suitable for searching, and then indexing them into Solr.
However, one of the things we’ve noticed is the lack of visibility into what Camel is doing. We can monitor things by looking at the incoming/processing/done queues, but getting hard numbers on performance etc is hard. And looking at JMX output is not very fun!
So I’ve been taking a spin around Hawt.io. This GUI console seems pretty targeted at the enterprise integration market, despite it being positioned as a general purpose management console for your JVM based applications.
My first test
The first test was to see if Hawt would work fine with an already existing Camel process. In this case, we monitor S3 bucket, and grab the files that land in it, convert them from XML to Java POJO’s, send them to Solr, and then move the completed file back to another S3 “Done” bucket. I took the war file that we had, and dropped it into the same Tomcat hosting Hawt. Wow! It worked! You can see the four different routes that we require loaded in the web app, along with some of the end points:
Here is a visualisation of the steps for the route that populates Solr from a file made up of many XML documents:
But my world is larger then 1 server!
Ever since the cloud came along, I can’t imagine only having one copy of an application running. Most of our ingestion processes consist of a pool of servers processing workloads. So, what happens when I have a couple of Camel processes running? I started up a separate Tomcat on port 8081, and dropped in an “agent” called Jolokia into that Tomcat. I started both Tomcats, and then clicked the “Connect” button. You then pass in the credentials to connect to the other tomcat, called server8081 in the screenshot below:
After connecting I see the routes for a sample Camel webapp I had deployed on server8081:
So this is okay, because if I have a cluster of 2 or 3 servers running Camel deployed, I can swap between them. However, if I need to scale out, then how can I monitor my Camel cluster across all the servers? I have to go to each one to check on it, and aggregate metrics manually! Now, I think folks might be saying “This is what Fabric8 and Kubernetes are all about”, to which I answer… “sigh… Yet another layer.”.
Is my Camel process dead?
In one of the projects, we are shipping 250 MB csv files around (and yes, I know it should be zipped!). The download and upload can take a while, and while that is happening “nothing” is going on… Additionally I was seeing some failures in the upload process, but didn’t have a good sense of what was going on. However, the “inflight exchanges”, which in English means “messages moving around” gives some great insight to what is going on:
You can see that the upload process is frequently failing, 24 times so far, but that it does periodically complete, 4 times. This also gives me a nice sanity check, I expect the ingestFile value to match the uploadDone value, at least, in the success Completed #. And that despite the number of failures that uploadDone is incurring, the overall systems is limping along.
The Tomcat plugin gives some nice metrics and server load utilization, but nothing that other tools don’t provide as well.
There is some sort of general purpose dashboarding capability that lets you tailor what you see, but I couldn’t quite grok how to get it working.
So Hawt.io, in a trend that I’ve seen in other open source projects, is clearly meant to be just one cog in a much bigger stack that the sponsors of the project, JBoss in this case, want you to adopt.. It so clearly is meant to be a gateway into Fabric8, as much of the capabilities, like the dashboard, I suspect come to life when you use those other projects.
It’s also rather limited in what it supports. They have clearly gone deep in Camel, ActiveMQ, and some other closely related projects, but there doesn’t seem to be any plugins being developed by folks outside of the immediate Hawt team. Hawt is also going through a big v1 to v2 upgrade, which may also be delaying broader adoption.
Don’t count on Hawt.io being the broad based administrative console you were hoping for. Today at least it isn’t quite:
a modular web console for managing your Java stuff
However, since Camel’s existing Web Console isn’t very good, and is now deprecated, then that iself is a huge argument in favor of Hawt.io for monitoring your Camel setup.