Note: this blog post was updated on September 21, 2017.
Jails are FreeBSD’s native solution to contain and isolate server processes. They are an alternative to (and predate) Linux cgroups, Solaris zones, and other OS-level process isolation technologies (the technologies that underlie Docker, CoreOS and a few others) .
This blog post will explain how to natively run CockroachDB in a FreeBSD jail. This is lighter weight and as secure as running Docker on FreeBSD.
Manually encapsulating CockroachDB using Linux cgroups is no easy task, which is why tools like Docker exist in the first place. By comparison, running server processes natively in FreeBSD jails is straightforward and robust.
Call for contributions: We welcome patches to increase the portability of CockroachDB, as well as external support to create and maintain installable CockroachDB packages for various Unix flavors and distributions, including a BSD source port.
Disclaimer: We do not yet claim to support FreeBSD compatibility. The solution below has been tested on FreeBSD 10.3-STABLE with today’s version of CockroachDB. Tomorrow, YMMV.
Prerequisites to build CockroachDB
Pending an official package for CockroachDB on FreeBSD, you will need to build from source.
For now, CockroachDB requires a 64-bit machine.
Your system should provide the C and C++ development tools. FreeBSD’s native C and C++ compilers are appropriate. You also need the additional packages for Bash, Git, GNU make, and Go. These can be installed with:
pkg install bash go gmake git
Ensure that go version 1.8 (for CockroachDB 1.0 or 1.1) or 1.9 (for CockroachDB 1.2) or higher is installed as a result: go version.
Creating a CockroachDB binary
The official instructions (select Linux and then Build from Source) apply with some tweaks:
Ensure that
/tmpis mounted as tmpfs or from a ramdisk prior to the build. We’ve seen that builds can be very slow otherwise.You’ll need to run the build with
gmakeinstead ofmake.
Testing your build
Ensure that $GOPATH/bin is in your PATH and the cockroach binary is there.
Ensure that the Linux “/proc emulation” is mounted in /compat/linux/proc. If not done yet, do this with:
sudo mkdir -p /compat/linux/proc
sudo mount -t linprocfs /dev/null /compat/linux/proc
(Note: Linux compatibility files / packages / libraries are not needed further. CockroachDB uses Linux’s procfs to inspect system properties via gosigar**. If/when gosigar evolves to read FreeBSD properties natively, CockroachDB will not need linprocfs any more.)**
Then as per the official CockroachDB docs:
cockroach start --insecure --host=localhost
Then in another terminal:
cockroach sql --insecure
There, play around with SQL, for example:
CREATE DATABASE test;
CREATE TABLE test.t (x INT);
INSERT INTO test.t(x) VALUES (42);
SELECT * FROM test.t;
If the program works as expected, the last query should return the inserted row from the table.
Starting/stopping CockroachDB as a system daemon
FreeBSD uses scripts in /etc/rc.d and /usr/local/etc/rc.d for starting and stopping system daemons. Which daemons are actually enabled to run during startup is specified using definitions in the common file /etc/rc.conf.
To integrate CockroachDB into a standard FreeBSD startup, we need to create a suitable startup script in /usr/local/etc/rc.d or /etc/rc.d. One such script is the following:
#!/bin/sh
#
# PROVIDE: cockroach
# REQUIRE: LOGIN
# KEYWORD: shutdown
# Add the following lines to /etc/rc.conf to enable cockroachdb:
# cockroach_enable="YES"
# cockroach_flags="<set as needed>"
. /etc/rc.subr
name="cockroach"
rcvar=cockroach_enable
pidfile=/var/run/cockroach.pid
load_rc_config $name
: ${cockroach_enable="NO"}
: ${cockroach_user="cockroach"}
: ${cockroach_flags="--insecure --host=localhost --store=path=/home/cockroach"}
start_cmd=${name}_start
stop_cmd=${name}_stop
cockroach_start() {
daemon -u $cockroach_user -p $pidfile /usr/local/bin/cockroach start $cockroach_flags
}
cockroach_stop() {
[ -f $pidfile ] \
&& /usr/local/bin/cockroach quit --insecure \
|| echo ${name} not running? \(check ${pidfile}\)
}
run_rc_command "$1"
This script runs a CockroachDB node using account cockroach. The default settings for the configurable variable cockroach_flags instructs it to store database files (--store) in /home/cockroach and only listen to a TCP port on the localhost interface. The latter is only necessary until you follow the instructions to secure your cluster and remove --insecure. The flags specified via cockroach_flags in /etc/rc.conf are passed as-is to the database start command, cockroach start, so we can also use this later to set certificates, addresses and the like.
It uses daemon(8) because the CockroachDB server does not daemonize itself.
If you want to use this script, be sure to name it cockroach (without ".sh"), ensure it is executable (chmod +x) and add the appropriate enabling flags to /etc/rc.conf (see comments at start of script).
Running CockroachDB in a FreeBSD Jail
Jails are FreeBSD’s solution for process encapsulation, more commonly called “containers.” Jails are both older than most similar technologies and comparatively more feature-complete.
In a nutshell, running CockroachDB in a FreeBSD jail amounts to the following steps:
Ensure that the virtual filesystem
linprocfsis mounted in the jail’s/compat/linux/procCreate a user account
cockroachfor CockroachDB in the jail.Copy the above
cockroachbinary in the jail’s/usr/local/binCopy the above startup script in the jail’s as
/usr/local/etc/rc.d/cockroachAdd
cockroach_enable=YESto the jail’s /etc/rc.conf, and tunecockroach_flagsandcockroach_db_directoryas desired. In particular you probably want to change--host=localhostto--host=127.0.1.1to match the Jail internal address. Note that by default CockroachDB runs in “insecure” mode, i.e. with SSL to clients and between nodes disabled. To change this, you need to set up certificates and configurecockroach_flagsas per the CockroachDB documentation.Start the jail.
While these instructions should be sufficient to seasoned jails users, here is a step-by-step tutorial for jails beginners. This uses ezjail-admin from the ezjail package.
1. Create an alias to the network loopback interface:
Add
cloned_interfaces="lo1"to/etc/rc.confActivate the interface with: service netif cloneup
2. If not installed yet, install ezjail:
pkg install ezjail
3. If not created yet, create the jail template; note that we do not need FreeBSD ports to run CockroachDB so we do not need to pass the parameter-p:
ezjail-admin install
4. Create a new jail, using the aliased loopback interface to attach the jail:
ezjail-admin create crjail “lo1:127.0.1.1”
This creates the new jail in /usr/jails/crjail. From now on we use “$J” in the examples to refer to this directory.
5. Start the jail manually for initial configuration:
ezjail-admin onestart crjail
6. Edit $J/etc/hosts and change the localhost address to 127.0.1.1.
7. Edit $J/etc/resolv.conf to add a DNS resolver, for example Google’s:
echo nameserver 8.8.8.8 >$J/etc/resolv.conf
8. Mount the linprocfs filesystem onto the jail:
mkdir -p $J/compat/linux/proc
mount -t linprocfs linproc $J/compat/linux/proc
9. Create a new cockroach user in the jail:
chroot $J adduser
You can choose a random password for the account. It will not be used by CockroachDB. We assume below that you keep the default home directory setting /home/cockroach here.
10. Copy the cockroach binary generated above to $J/usr/local/bin. You may need to create this directory yourself. (use mkdir -p)
11. Copy the cockroach startup script above to $J/usr/local/etc/rc.d. Again, you may need to create this directory yourself.
12. Add cockroach_enable=YES to $J/etc/rc.conf:
echo cockroach_enable=YES >>$J/etc/rc.conf
13. Restart your jail to start CockroachDB:
ezjail-admin onestop crjail
ezjail-admin onestart crjail
14. Ensure that CockroachDB has started and is reachable via 127.0.1.1:
curl http://127.0.1.1:8080
If CockroachDB does not appear to be running, you can investigate whether it has started by logging into the jail console with ezjail-admin console crjail and looking into /home/cockroach from there.
15. If this works, congratulations! You successfully jailed the critter.
Where to go from here?
The steps above isolate the CockroachDB server process from other processes in the same system. The server becomes reachable only from the local address 127.0.1.1.
From there you can move forward in various directions:
Automate starting up your jail(s) when the host system comes up (e.g. add
ezjail_enable=YESto the host system’s/etc/rc.conf).Adjust
cockroach_flagsin the jail’s/etc/rc.confto spread the data store over multiple disks and/or ramdisks using CockroachDB’s--storeflag.Set up resource limits on the jail, for example memory or disk usage, see jail(8).
Add an alias to the LAN network interface to the jail, so that the CockroachDB server can connect to other nodes in the cluster. (be sure to configure security certificates!)
Set up Mandatory Access Control (MAC) to further restrict CockroachDB’s security attack surface.
Monitor the database server using Nagios (or your own favorite monitor tool) to set up administrator notifications and automatic restarts whenever the database process fails.
Really from this point CockroachDB becomes one among your other contained server processes. All standard FreeBSD best practices and literature apply.
