Nearly eight years ago, I complained about the lack of an adequate configuration management tool. So, I wrote one.
Eight years ago, in my Agent vs. Agentless post I wrote about how I wanted something new and/or different for a configuration management tool. And after eight years, I had an epiphany. Truth be told, I had the epiphany over a year ago, I am only writing about it now.
In the “Agent vs. Agentless” post, I already had a name picked out: govern.
It’s a verb.
It is written in Go.
It starts with “g-o”.
nice.gif
Without further delay, I would like to announce Govern.
This post is mostly about how Govern came to be. If you do not care about that, and just want to read the technical documentation, go here.
Most configuration management tools provides swathes of built-in plugins that will allow the user to manage packages, manage the contents and properties of files, and do so much more.
Govern, on the other hand, keeps things as simple as possible. Govern is a configuration management tool that tries to do as little as possible and relies on external executables to accomplish the desired tasks.
When I initially started trying to write Govern, I ran into an issue where I needed it to actually do something. Sure, I could just start slogging away at writing individual actions like “install a package”, “make sure a service is running”, “template this file”, etc. I quickly lost all motivation, because at that point, I was repeating work that has already been done by dozens (if not hundreds) of people who have contributed to Ansible, SaltStack, Chef, and Puppet.
It stopped being fun. I had not done anything differently. I was left with something that was completely useless on its own.
And after some years of stewing on it, I had the epiphany that lead me to the initial implementation.
I don’t need to implement all of the actions myself. I don’t even need to provide them. All I needed was a way for people to be able to write them and their own.
To provide a brief detour of insight into how I happened upon this realization: at work, we were doing some work with Ansible. My coworker had just written a really, truly, very basic action that rendered a set of templated configuration file based on some predetermined paths. The argument could be made that it could have been done differently, but there were two things that really stuck out to me:
My coworker did a really good job of only writing enough code to accomplish the task. It wasn’t a matter of programming taste. The code was fine, and it did the job in as few lines as required. What stuck out to me was how much boilerplate there was, and how impenetrable and opaque the boilerplate was. What really bothered me was how befuddling the code was. Most of the file was nothing but boilerplate. The code that was actually performing the intended function was maybe 10% of what was written.
That irritation, at how obtuse and cludgy the plugin system was, got me thinking about how this could be made easier.
There are several established patterns for configuration management tools:
ohai. Puppet has facter.NOTE: Yes, I know, Windows folks now have SCCM, or MECM as it is now known; and yes, I know Ansible and folks have somewhat supported Windows for a while. It does not change the fact that most Windows administrators I have met, typically did not want to shoehorn a UNIX-style configuration management tool into their GUI-driven lives. It is not a dig at them. It is an indication that “the UNIX way” is not universally accepted as the better approach. Windows distinguished itself by being “not UNIX-like”. It doesn’t mean I think they are right or wrong.
Instead of writing all of the plugins, or “things that do things” for Govern, I realized all it had to do was be able to execute something. This little realization opened up the world of possibilities for me.
As a brief aside, I do not manage Windows systems. I never have, and $DEITY willing, I never will have to. All of my work, for as long as I have been doing it, targets Linux systems. Once in a while, a snazzy little BSD box shows up, but for the most part, it is Linux day-in, day-out.
When Govern needs to check for facts, sure, it can probe the underlying system, but how scalable would it be if I needed to write all of those facts into the underlying system from the get-go? The answer is: not very. So, to jump to the conclusion, I thought, “all I have to do is walk a directory tree, and look for files with the executable bit set on their modes.” With that, I wrote some really simple code that crawled one or more directories, looking for executable files. In keeping with “the UNIX way”, each of these executables writes their output to STDOUT.
When you look at system facts gathered by other tools, there is some scrapping about with the names and hierarchies of those facts. In Govern’s case, the fact’s “name” is its position in the directory hierarchy.
For example, say our “facts” directory (holding our executables) is at
/usr/share/govern/facts.
Within that directory, we can create other directories holding
executable files.
name is a pretty generic fact name, and it does not tell us anything
about what name that fact is providing.
But, if we put it under the os directory, its name becomes
os/name.
This makes it pretty clear that this fact is going to return us the
name of the operating system.
Now, we are getting somewhere.
I followed the same approach for “runners”.
Runners, in Govern’s taxonomy, are the actions plugins, or the “things
that do things”.
If you are using SaltStack, you might invoke the pkg execution
module to install a package.
The Govern equivalent, is the pkg runner.
Runners are discovered the exact same way: when Govern starts, it looks at all of the runner directories it was configured with, and walks each of those directories looking for executable files.
Unlike facts, which are always expected to return output, runners only need to return output when something goes wrong. In this case, the error is expected to be written to STDERR. Information can be written to STDOUT, but this is expected to be purely diagnostic information in successful cases, and is ignored by Govern unless you explicitly ask for it.
The one different between the discovery of facts and runners, is that runner names are not hierarchical. Govern will start probing for executables in a given base directory, but once all executables under that directory have been found, the hierarchy is flattened, and all runners exist within the same “namespace”.
As I mentioned, you can configure Govern with multiple facts and
runner directories.
What happens if there are multiple facts or runners with the same
name?
In keeping with simplicity and prior art, I followed the example of
the UNIX $PATH variable: if multiple executables exist with all
searched directories, the last one wins.
For example, if you have two facts directories defined in this order:
/usr/share/govern/facts/usr/local/share/govern/factsand both of them contain an os/name fact, the last one will win.
In this case, it means /usr/local/share/govern/facts/os/name will be
the fact that is executed.
Again, the necessity of a templating engine is arguable. It is nice to have one, especially if it is documented. Ansible and SaltStack both use Jinja2, and it is really nice.
Fun fact: Python used to be my primary programming language. During the whole Python 2 → 3 changeover, I lost my patience with the whole PyPI ecosystem as some packages quickly updated to Python 3, and some dragged their heels. One day, I experienced my last
UnicodeDecodeError, and said enough is enough. I heard about this nifty little language called “Go”, written by some UNIX juggernauts, and from that point on, Go has been my primary programming language.
Go has a built-in templating language: text/template. It has every feature you would want from Jinja2, even if it is juuuuust ever-so-slightly different in use. It is a perfectly functional substitute, in my opinion.
With that, Govern had a templating language.
As I mentioned earlier, my one, big irritation with the existing systems was how opaque it was to extend it. You had to read its documentation, and understand exactly how to initialize your plugin and get it to register with the core system. This is partly fixed by allowing users to place an executable file in a directory (that they then configure Govern to look in).
If you are trying to extend Ansible or SaltStack, you have to write your plugins in Python. If you are using Chef or Puppet, you have to use Ruby. But if that weren’t enough, you also have to import packages or modules, and extend some base classes to have your new plugin be automagically recognized by the core system.
With Govern, you can write your facts or runners in any programming language. The only thing Govern cares about, is that its executable bit is set.
At the end of the day, this means that the Govern core can stay exceptionally small, and be infinitely flexible to the needs of its users. In the existing systems, if you found a bug in any of the existing plugins, you would have to wait for the bug report to be triaged, the bug to be reproduced, and for the maintainers to cut a new release. Depending on the project’s overall cadence, you may likely be waiting a while. With Govern, you have the option to add, replace, or completely remove any facts or runners!
One thing I really liked from SaltStack, was how the .sls files
were passed through the templating engine, before being applied.
In Ansible, you have to carve out completely-separate tasks files, or
conditionally override variables if, say, a pacakge name is different
between two Linux distributions.
In Salt, it kept a lot of that in one, very convenient place.
The templating engine in Govern, Go’s text/template package, is
bundled into and provided by the core govern executable.
With templating being something bundled in, this allowed me to bring
along the “templated state files” approach I enjoyed so much, from
SaltStack.
As I mentioned earlier, declarative state is where it’s at. Imperative configuration management, in professional practice, is for masochists.
Govern needed a file format to allow its users to declaratively define their desired state of the system Govern is run on. After fiddling around with different formats, and eventually settled on HashiCorp’s configuration language, HCL.
While the hclsimple package lets you quickly unmarshal
HCL from an io.Reader into a structure, you can also use other
foundational subpackages to dynamically parse the read state files.
This is important because Govern re-discovers all executable facts and
runners each time it is started.
In practice, there is no real concern of this being a slow operation,
since a lot of filesystem operations are cached by the operating
system, and doing something like walking a directory is just walking a
bunch of direntry structs in a filesystem journal.
It’s the “dynamic” part that is important.
To help frame the rest of this section, here is an example of a resource declaration within a state file:
pkg "emacs" {
state = "installed"
}
This resource defines a pkg resource called emacs, and says it
should have the installed state.
A resource is structured like so:
runner-name "resource-name" {
arg = val
...
}
So, with our pkg "emacs" resource, it tells Govern the emacs
resource should be handled by the pkg runner.
Since runners and facts can come into, or be snuffed from, existence
between any two executions of govern, it would not make sense to
store any state about Govern’s runners or facts across executions.
When Govern starts up, it first scans for facts and runners.
Next, it will read in any state files from — you guessed
it — one or more “state directories”.
As each resource defined in a state file is parsed, Govern will attempt
to match the resource’s runner-name to a discovered runner.
Govern does not know the name of runners before it executes; it
can’t.
If Govern cannot find a runner to hand the resource off to, it will print an error message to the screen, and exit before applying any state.
Similarly for facts, if a fact is explicity
requested within a templated state file (and
remember, all state files are technically templates), Govern will exit
before doing anything else.
Unfortunately, this currently does not hold true for templated files.
If you attempt to use the template runner (which simply calls
govern template) to render a templated file, and it cannot find a
fact using the added fact template function, the resource itself
will fail, but the rest of the state will still be applied.
What’s more, is that the key = value pairs within the body of a
resource block are completely free-form.
There are only two, reserved
keys in a resource
block: before, and after.
These currently-unimplemented features are intended to allow Govern to create a directed-acyclic graph (DAG) of the resources to apply, so that users can order the application of resource state across different state files.
Since the runner names are dynamic, and the key-value pairs within a
resource name are relatively free-form, in combination with Go being a
statically-typed language, the HCL libraries provide a nice set of
tools to be able to flexibly parse structured data, without having to
force them into a pre-declared struct.
There are a bunch of features in other configuration management tools that I know I did not want to replicate in Govern.
Configuration management processes like SaltStack’s minions, like to
run as daemonized processes.
Since a lot of these systems want to ensure the state declared by
the user as often as possible, these processes like to periodically
execute their declared state files on some regular cadence, and this
means that these processes (since they are running as daemons), need
to have some sort of timer to run on.
Linux and other UNIX or UNIX-like systems have had periodic
execution scheduling for a very long time.
It’s called “cron”.
Nowadays, you have systemd.timer(5) units.
This does not need to be a thing in Govern, since Govern does not
run as a daemonized process.
This was one feature that always stuck out like a sore thumb to me,
from SaltStack.
Everyone wanted to use it, probably because “it was there”, but it
was always such a chore to set up.
The fact of the matter is, if you want to make sure the
configuration management system applies the latest declarative
state revision, you can pull that down yourself.
Be smart about it.
You know you already have a task scheduler like cron, or systemd, so
write a script that runs git pull in the directory where you have
Govern configured to look for state files, and make sure it executes
before Govern does.
Alternatively, write a wrapper script that pulls the latest
revisions, then calls govern apply.
As I have already pointed out, ad nauseum, the downside of bundling a lot of things into the core product just leads to a burden for the users when you do not get everything perfect out of the gate. At some point, as the developer and maintainer, I want to keep my job as easy and frustration-free as possible, and I do not want to have to deal with long-running email threads about “oh hey, people don’t use yum(8) any more to install packages on RHEL; they use dnf(8) now,” or suffer through conversations about what someone else thinks the best set of flags to pass to some command are. All that does is burden me with the task of keeping the product supportive of systems I probably do not use on a regular basis, and it makes users far less likely to want to adopt it if it does not support their preferred Linux distribution, out of the box.
The most important goal of Govern, for me, is that it stays as a
simple and flexible core executable, without being mired in too many
features that maybe some people will use.
I especially do not want to add features that are already provided by
the underlying system.
That is why Govern’s plugin system is not some complex internal
machinery; it mimics a UNIX (or UNIX-like) system’s shell, where it
essentially looks in directories for sets of executables to run, like
$PATH.
This is why Govern will likely never run as a long-running process.
It does not need to.
It does not need to replicate existing functionality, nor does it need
to provide parity for all of the little, seemingly-convenient features
that other systems have.
At the end of the day, Govern is intended to be a very basic, and very functional configuration management tool. It will reuse whatever the underlying system provides.
Govern has been my little hobby on the side, for the past year or so. I have experimented with writing runners and facts in various programming languages.
For facts, mostly, I will use good ole, “standard” POSIX sh(1). This has been fine for the most part.
For runners, I started writing them in POSIX sh(1), but it got really cumbersome as I started to add support for other Linux distributions and BSDs I use on occasion. Depending on the task at hand, if it is simple enough, I have really taken a shine to Lua. It is simple, and embeddable within Go, thanks to go-lua. Remember that one of my desires, in Agent vs. Agentless, was that I did not want to have to depend on an interpreter being installed on the managed host. With being able to embed a Lua runtime into a Go program, Govern becomes the interpreter.
For anything more complicated than feels comfortable writing in Lua, I have started putting together a package that handles all of the boilerplate for writing runners in Go. The upside to this, is that you can easily write runners in Go. The downside, is that it makes these runners impenetrable to a non-programmer (or a programmer or sysadmin who is not familiar with Go) that finds a bug. And despite disk space being cheap these days, it also results in runners being incredibly large, statically-compiled binaries.
The Govern core is very functional. Any time I have put into the project lately, has been mostly around adding runners as I need them. I hope you find Govern interesting and useful.