I have been administering puppet installations for almost five years; FIVE years! I have always known that at some point, node management would become an issue. Unfortunately, I have never worked with an infrastructure that had massive numbers of exactly identical servers. I started by configuring nodes to inherit group definitions that were stored in another manifest. Then I saw the environments feature but never implemented it mostly due to overhead and the numerous and well documented bugs that will be solved in Puppet 4.
I tried to implement an external node classifier. The main problem with this was with the concept of a directory of YAML files to describe nodes. I have never been fond of a directory full of very small files to describe anything. I know what your thinking, “Your a Linux engineer! What about /etc?” Well, thats easy, each software package manages its own files, unlike me managing 600 odd files describing 300 odd servers.
Two years ago, I converted all my modules to the “Roles/Profiles” pattern and broke all the variables into a parameter file. I setup a instance of the forman hoping this would solve my problem. I subscribe to the unix rule of “Do one thing and do it well”. While forman is an excellent piece of software, I feel like it fails at this.
When I read the puppetlabs press release about Hiera, my heart skipped a beat. Was this the solution to my problem? Sadly no, because once again we should manage the slew of hosts we have with a directory of YAML files. According to this blog post, I like to stalk my problems to make sure I have a good understanding of the issue and can create a solution that minimizes the edge cases. If I have learned anything during my time as a systems engineer, it is to reduce edge cases in every aspect of your job (and life). With this in mind, my ideal solution was to find or write a hiera backend to query a ITIL service catalog or change management solution. First I had to find a change management solution. FYI, I am still looking. I was perusing github an few months ago and stumbled upon hiera-etcd, I had found my solution! I finally got a chance to setup hiera in the test lab this week. A little back story, I find the concept of key:value stores very intriguing. I have used zookeeper and eureka, but finally settled on etcd for my needs.
The first thing I did was write a puppet module to deploy etcd. Your probably wondering why I didn’t download an existing module from the puppet forge, and that because I already built an rpm for etcd with fpm. My etcd module only has three stanzas: a package stanza, a user/group stanza, and a systemd unit file that is derived from an erb template. I think the whole process took me an hour to write including building the rpm.
Using hiera and etcd, I was able to convert 10 lines of module code into a singe hiera lookup.
$ntp_hosts = split(hiera($ntp_hosts), ",") |