Disclaimer: Sorry, another nerd post.

One of my personal projects recently has been replacing my Cacti setup. I realized I had been moving computer systems around and hadn’t updated the graphing system that monitored them over time, so I figured it was about time to overhaul.

The actual replacement system was pretty easy. I’ve rolled out Cacti several times, and this wasn’t much different. Learning from the past mistakes helped me utilize templates much more efficiently, the only thing I’m really lacking at this point is the ability to monitor a handful of devices on my network.

The other thing I’ve been eyeing for a while is a tool called “Network Weathermap” – I’ve been trying to push for us to build weathermaps for at least the larger market presences we have at work, either as a marketing tool or just to help diagnose high network traffic issues quickly. I showed it to my boss and he was impressed, but I wanted to actually make it work so that I had some idea of what kind of work would be involved. This is my network:

You see the green section, those are virtual servers represented individually. My home network is in the orange. Physical links (or virtual physical links) are shown, unfortunately I don’t have the ability to monitor any of my switches. The one I might be able to is the Guest Wireless switch but it’s locked down too tightly. But that’s a problem I can solve later. Guy Smiley is my VMWare server, the other machines are all guests on that server.

I’m actually slightly disappointed at how boring it is!

That said, as a brief review of Network Weathermap, it’s a pretty awesome tool. It’s fairly picky about how it is set up, and the WYSIWIG editor that comes with it needs a lot of work (which it freely admits). That said, getting all the nodes placed was simple. Getting the links configured was also pretty easy (it’s just a case of linking nodes the right way round for the Cacti graphs, and if you get it wrong, editing the config file is simple). The thing that took the longest time was getting everything aligned to my liking. The WYSIWIG editor wouldn’t let me select more than one node and move groups of nodes together.

If you’re a nerd with a monitorable network like me, give it a try!

As an administrator of several servers hosting a few websites, small as they may be, it is important to me that I can do my best to ensure that the services I am providing are online every second of the day and that I am aware of any issues when they happen. It is also important to me that I can monitor certain things over time and have a visual reference to them. I have four separate tools in my belt for keeping track of my systems and services: UptimeRobot to monitor from the outside; Nagios to monitor from the inside; OpenStatus to provide a quick reference; and Cacti to provide historical data.

Let’s run down each of them, starting with external monitoring. For the last year or more I have used a monitoring service called UptimeRobot. They provide free monitoring for up to 50 services and utilize both email and SMS (via email to SMS gateways) to alert me when something is down. I have monitors set up with them for each of my servers for website, ping, and for a couple of specific services as well.

For what I need, UptimeRobot do a great job. It’s enough to know that my website can or can’t be reached, but that doesn’t give me a lot of detail. Several months ago I implemented a system called OpenStatus. I use this as a public monitoring tool for my services, you can find it here. It gives a regular update of CPU load, memory usage, drive space availability and network usage. I use it regularly for a quick visual reference as to what is happening right now.

I’m not always at a computer watching OpenStatus, however. In order to get more detail with regard to “right now” and get alerts when things are going wrong, I use Nagios:

Anyone familiar with Linux servers has at least heard of Nagios. It is a very powerful tool for monitoring: it schedules regular checks as defined in the configuration and the checks return “OK,” “Warning” or “Critical.” Again, it’s adjustable by configuration but the default is to check each service every 5 minutes. If the service returns a new status (goes from “OK” to “Warning,” for example) it will recheck a minute later, and if the same status is seen it will send an alert. I use Nagios to monitor more specific details that UptimeRobot can’t, such as disk usage and CPU load. If memory usage gets too high, I get an email. If CPU usage gets to high, I get an email. When they return to normal I also get alerts by email.

These alerts are particularly useful, as they can help me with the warning signs of issues about to happen before I get the terrible text message telling me my site is unavailable.

This isn’t the end of my monitoring strategy, however. So far I’ve outlined basic external monitoring for here-and-now, as well as internal monitoring on a finer level. Another aspect to monitoring that is often overlooked (and I consider to be quite important) is looking at trends over time. This is where Cacti comes in:

Cacti focuses on SNMP but it can use anything that returns a numerical value over time and converts those values into graphs. Here you can see the CPU usage is climbing steadily, and it is an issue I will need to look into before it causes problems. You might also notice that over the last couple of days my load averages have spiked a little more, which may be related to the CPU usage problem.

This shows the importance of historical data. I might check CPU usage and notice it’s a little higher than yesterday, but unless I’m noting that down I may not realize that it’s significantly higher today than it was a week or a month ago.

If you are an administrator of anything you consider important, you should be monitoring it somehow. If it goes down, you should at least be aware of it. I am typically made aware of an outage when I get an alert from UptimeRobot to my phone (occasionally I’m monitoring my email and I get Nagios alerts in time for them to help). From there I can check my email to see if there were any warning signs – I have common issues where my web-server will reach it’s RAM limit and start swapping more than my virtual server can handle and Nagios alerts will indicate this. That let’s me know what my next steps should be. Can I log in with SSH and fix the problem? Can I log in to the control panel and reboot the server? Is this a network outage that I just have to wait for? All of these are critical details if I want to maintain an up-time above 99%.

Over the last month I’ve been going through the job application process with CustomInk, a T-shirt printing company based in McLean, VA. The job was perfect! It was a blend of Helpdesk, Desktop Support, and a small amount of Systems Administration. It was a small package of Active Directory support and troubleshooting which I knew, and some commercial printer support and other things which I didn’t. It seemed a great opportunity to learn and grow, but it wasn’t to be. After a month of emails and interviews they finally rejected me yesterday.

And so the search continues. I’m looking in the Harrisonburg and Charlottesville areas (one to apply for in Crozet, and I keep looking all over) for Helpdesk and other entry-level IT jobs. For the right job and money I might be able to persuade Kelly to try Northern Virginia or Richmond but it’ll be a stretch. If anyone is looking or interested, my resume is up here. I have a few plans for self-improvement, with plans to learn more about Active Directory, VMWare (ESXi), and a few other Linux distributions that I haven’t spent a lot of time on. All, for me, is not lost.

That said, Kelly is also looking for a new job. While I’m unhappy with the pay and the career options that my job provides, she seems genuinely unhappy with both of those as well as her job itself. She’s also much less specific in what she’s looking for, having put in applications to be a children’s specialist at a library, an editor for a college department, and all kinds of things. If you see something that you think she might have an interest in, or you want to know what you should be looking out for, hit her up on Facebook (I’m assuming you know her already) and let her know.

Please don’t hate on CustomInk too badly! They seem like a great company with a lot going for them, and I’ll be more than happy to apply with them again in the future. I asked them if they could provide feedback on my application to see if there is anything I could improve on (things to learn, skills to improve, etc). I’m not expecting a reply, but I’m hoping.