Compound conditionals in CFEngine

I have been working with CFEngine for about half a year no in my new role. CFEngine is known to have a steep learning curve. I can’t disagree with that sentiment, I can however say that it is intuitive.

This tip though covers compound conditionals and classes. CFEngine uses classes to define a desired end state.

In cfengine this can difficult because the syntax is very restrictive.

Job Titles Galore


I have been out of school for 8 years now and still trip myself up when people ask me what i do. It usually is some variation of “I run computers for an investment firm.” Which is true, but suffers from a massive lie of omission.

Computers is an understatement. I imagine most people think of computers as desktops. I used to do desktop management, transitioned to servers management, then spent a couple years as the sole HPC administrator before transitioning to my new role as CFEngine ninja.

Investment firm is also an understatement. I work for Two Sigma Investments and we do a lot more than manage investments.

This post is not about either of those understatements though. It was meant to be a short meandering through the various job titles I have held. So join me for a brief nostalgic sojourn.

Junior Systems Administrator
mostly perl scripting and hardware builds

Jack of all Trades Systems Administrator
desktop/phone/server support, window admin, linux admin, networking, data center management, pretty much EVERYTHING

Senior Systems Administrator
This role focused more on the linux admin and a lot of debugging

Global Systems Infrastructure Manager
I managed systems infrastructures in 4+ countries.

High Performance Computing Systems Administrator
I grew a 100 server compute farm into a 700+ server farm, managing multiple generations of hardware. I even brought in our first GPU servers. I managed a multimillion dollar budget.

Senior Linux Architect/Engineer
I worked with internal customers delivering a solution to meet their needs. I did a lot of hardware evaluation.

These days I am not sure what to call myself. All the above still applies but I spend the bulk of my time knee deep in CFEngine and managing the internal Sysadmin infrastructure (repository servers, CFEngine hubs, etc)

Here are some options for my current title:

Senior CFEngine Ninja
Grand CFEngine Poobah

Das Keyboard

_20140922_131900Back in the beginnings of the computer era, the keyboard you received with your computer was a mechanical keyboard. Now when you buy a computer, you get a cheap membrane keyboard. Educate yourself on the difference.

I recently purchased a Das Keyboard 4 Professional (Soft Tactile) that uses Cherry MX Brown switches. Typing on it makes me nostalgic. 20 years ago I had a Tandy 1000 with a mechanical keyboard and remember the sound and touch of the keypresses.

I spend my entire working day using a keyboard and I am very happy with my purchase.

I’ve had my Logitech Revolution MX mouse for 8 years now. It has required a few cleanings and one major overhaul. Hopefully this keyboard last just as long.

New setup at home

I just discovered, self described as “for those who have a computer lab at home.” The first computer homelab I had was about a decade ago when I built my first computer to run linux and serve webpages. I started with Fedora, then Debian, and finally Ubuntu.

I recently retired my DD-WRT router in favor of pfSense and a Unifi wireless AP. I have run DD-WRT on a router for about 7 years. First on a Linksys WRT54G (v2) for 6 of those years and then on a shoddy Netgear for the last 8 months. Until now I had never bought a new router. The WRT54G I bought off craigslist for about $30 back in 2007. The netgear was my parents old router. Now that we rely on wifi for more than the occasional laptop, it was time to beef up the setup. We use wifi for 2 Nexus phones, an old ASUS netbook, and a Chromecast.

I’m running pfSense on an old Dell desktop with an added Intel NIC. I have the Ubnt UniFi controller software running on pfSense as well. I’ve prefer to have a new appliance to run pfSense but its cheaper to run the more power hungry desktop (which cost nothing to buy) than buy a $100 or $200 router. Right now I’ve sunk $68 (the cost of the UniFI AP) into the new setup.

I should run the power number of the desktop and see what my return on investment would be if I bought a Ubnt EdgeRouter-Lite. There are rumors that the EdgeRouter-Lite will be able to run pfSense in the future.  Ubnt’s EdgeOS might be sufficient too.

LOPSA Recognized Professional Program

ONce4itI was recently recognized by the LOPSA Professional Program (LPR). This is the inaugural class. You can read up on it here but the TL;DR of it is: the LPR aims to benefit both IT Professionals and Employers by creating a way to recognize those who strive to make Systems Administration a Profession and not just a $job.


What it takes:

  1. Worked as a SysAdmin
  2. Abide by the Systems Administrators’ Code of Ethics 
  3. Be a member of the League of Professional Administrators (LOPSA)
  4. Met the educational requirements.

For #4, last year I attend training at LOPSA-East, took An Introduction to Operations Management MOOC on Coursera, and read books focusing on Systems Administration.

Why did I apply? I love my profession. I didn’t know what I was looking for when I was applying for jobs my senior year at the Fu Foundation School for Engineering and Applied Sciences. With a Bachelor of Science in Computer Science I could have been a developer, I could have gone on to graduate work, but when I discovered that people pay you to do what I had been doing for free, I knew systems administration was my calling.

Since then I have learned everything I could. I have been mentored by some smart and great people. I have branched out and tried out new forms of learning (MOOCs, certifications, study groups). I started a LOPSA Local Chapter and eventually was elected as a Director of LOPSA.

How to hire a Systems Administrator

The job market is bullish for System Administrators and their ilk. I see new job postings daily and recruiters have been contacting me through LinkedIn and my blog for awhile. Unfortunately most of the job postings are horrible. Just flat out bad.

How to write a BAD sysadmin job posting:

1. Include every acronym known to IT

I have seen posting that include every acronym possible: DNS, DHCP, FTP, EGREP, OSPF, LMNOP. This is not helpful. If the position requires a more capable sysadmin, just say you are looking for a jack-of-all-trades. We are out there. I started out doing everything. Some people prefer variety from their work demands.

This is also exacerbated by sysadmin resumes. In an effort to land every possible interview resumes include whatever can catch HR or a recruiter’s eye.

2. Unrealistic expectations

You are not going to find someone who is an expert web designer and apache tomcat ace. Sysadmins are great dabblers, we love to try out new technologies. So there is a good chance that if an applicant has some experience it is cursory, and if they don’t they can probably teach themselves in short order.

Don’t expect a 10 year veteran to do helpdesk duty. Completely unrealistic.

3. No salary range or ridiculous salary ranges

Unless you are Google, Facebook, or a NYC Investment firm, I need to know you are going to pay me commensurate with my abilities. The aforementioned get a pass on street cred. I am not going to apply for your job unless I know the pay will match or exceed my current position.

You get what you pay for. If you think you are going to hire the next Ninja Sysadmin paying hourly, you need to reset your expectations. This is an issue that arises because “everyone’s nephew knows computers.” The devaluation of our profession due to amateurs is real. No one goes to amateur doctors, don’t let a hack touch your production network.

There are more, but that is all I have time for today boys and girls. I will try to dig up some examples from craigslist or listservs.

ITIL Foundation Training

Last week I took 2 days to take ITIL Foundation training through Simplilearn. I am going to review the training company and the subject matter separately.


There a lot of companies willing to provide for any certificate acronym you can think up. I could find no useful reviews of whose training was better. I knew I wanted to take a course in person. I have done a number of MOOCs over the past 2 years and am burnt out on self-directed learning. I tried to study for the LPIC certification by myself using an online training course but it was horrible and I never felt confident enough to take the certification exam. I signed up for training in September but received a call from an Indian gentleman telling me he had to cancel the in-person training in Pittsburgh for September due to lack of students. He offered to refund me a portion of my fee to travel to Philadelphia to receive the training. That wasn’t going to work for numerous reasons. He offered to refund the difference if I could take the online training, again no. He assured me that the training would happen in October.

The course would be 8am-5pm Thursday and Friday. I was going to take the exam at the end.  I received confirmation that the training was happening the preceding Monday. The class was myself and one other student. The trainer had traveled in from Chicago. The course was supposed to take 18 hours over two days. I think we really spent 10 hours working on material. We finished early both days, and had frequent breaks and started late both days.

The training is very singularly focused on preparing you to pass the exam. It is aimed at certification, NOT education. I am not a ITIL Foundation expert by any means. I know the process and vocabulary, but we didn’t do anything that goes on in most classrooms. We did no case studies, had no discussion, and didn’t apply anything we were learning.

Subject Matter / Course Content

ITIL is a lot of things. It is a framework, it is a list of best practices for IT service management. It is also confusing. The pot of gold at the end of the rainbow is that it is useful. Its flexibility and therefore almost universal applicability to our profession means it can help out just about everyone and anyone. System Administration is a relatively new profession but its not so new that the wheel hasn’t been invented and re-invented, and re-re-invented. Some really smart and focused people have thought hard and long about how to deliver value through IT. I am smart enough to know when to defer to the experts. I would recommend at least the Foundation training for every sysadmin regardless of experience. The key though, like all learning, is to have an open mind.

Other Thoughts

As soon as I receive my exam results I am going to pursue the next steps in ITIL certification. Thankfully there is no shortage of content or training. I am thinking of focusing on the Operations aspect but almost every stage appeals somewhat to me and how it could better my work and help my company.

Downtime & DNS Registrars

I wanted to take a second to explain why has been down for about a week. I was forced to change my IP address on the hosting server. I also host my own DNS and backup DNS. I have been a customer of 1&1 Internet for about a decade now. They recently changed their UI and it was not letting me change the name of my nameservers. Every registrar requires you to create NS records if you want to host your own DNS, instead of using theirs. Well 1&1 makes you create subdomains for each nameserver instead of just A records. Their horrible UI was not letting me accomplish what I wanted.


Goodbye 1&1, hello Namecheap. I initiated the DNS transfer last week after just a day of fighting 1&1’s UI. Most DNS registrars will transfer your domain quickly. Not 1&1. They decided to make me wait the full 5 day period.

TL;DR I stopped using 1&1 for DNS registration because they replaced something that worked with broken-ness.

Total Cost of Ownership in the System Administration World

In the System Administrations world the total cost of ownership (TCO) is the price you pay for equipment from before its purchase to after its demise. There are always hidden costs that you can not anticipate, but through diligent evaluation and selection you can limit these unforeseen costs and minimize your TCO.

My main focus for this post is going to be vendor selection, specifically vendor plurality.

Unlike Google of Facebook, most Systems Administrators buy their server equipment from the following vendors: HP, IBM, Dell, or through a re-seller of Supermicro. Each vendor offers different product lines and while one product line may be more reliable than another, as a whole let’s assume they are all as reliable within some standard deviation. Why then would you ever buy equipment from more than one vendor? Price. IT and System Administration are sunk costs, they are the cost of doing business (ignoring IaaS,PaaS, & SaaS). A cost conscious company will want to get the best price on hardware.

Why You Should Support More Than One Vendor

Depending on your computing needs you could be spending millions on hardware, one companies margin could be tens of thousands of dollars. It is a good practice to always get competing quotes from different vendors (not just different re-sellers). That way you keep the vendors honest (or at least more honest).

You can save a significant amount of money this way, which you should obviously make sure your bosses know.

But what hidden costs arise?

Why You Should Support the Least Amount of Vendors as Possible

For every vendor you support it requires your time as a system administrator. To name a few considerations:

  • different RAID controllers with different syntaxes
  • different OOBM (imm,idrac,ilo) with various nuances
  • support from vendors can differ widely in process and competence
  • automated hardware monitoring is different. While most vendors support IPMI, there are always little differences
  • all new vendor equipment needs evaluation and a familiarization

What am I to do?

There is no optimal solution, only what works for you. I recommend limiting the number of supported vendors to the absolute minimum while still keeping a 2nd around to make sure the other vendor’s pricing is honest and competitive.

Each vendor adds to a server’s TCO. Therefore limiting supported vendors, minimizes every servers’ TCO.