Reproducible RoR development environment setup with Nix
But it works on my machine!
Back in the mid-2000s, my first encounter to get a pre-ordained re-producible J2EE development environment for a classic enterprise J2EE application with the necessary toolchain (e.g. IBM RAD, Informatica ETL tool, etc) with credentials setup to our central development mainframe IBM DB2 database, was the all too familiar technique of being dished a VMWare image. I had to boot-up the development image VM in our measly corporate IBM Thinkpad laptops (with Microsoft Windows XP), with a measly 1GB volatile memory with a single-core Intel Pentium processor. It solved most of the typical "...but it works on my machine..." frenzied setup phase for a new recruit to the software development organisation. Anyone who has gone through that experience of running IBM RAD, in your Windows XP machine, knows how resource-intensive the IDE was; Now imagine doing that in a VM!
It kind of work, but whoever came up with the idea, probably did not do coding day-in-day-out. It was impossibly slow for typical development work. To conflate the issue, working in a Microsoft OS to automate the configuration of your local development is already not a good place to start with.
Suffice to say, it was a short-lived endeavour. I soon find myself again in similar situations, at subsequent employments, where I had to follow a not so up-to-date README.md and a bunch of well-kept tribal knowledge around how to set up and configure the local development environment before I can become productive and contribute my first line of source code.
The age of tool version managers and docker containers
Life was better, with homebrew , sdkman , rvm / chruby , nvm and my current goto - asdf (a much-improved cross-development tooling manager) in OSX. Still, it was far from perfect. When things works, you do it once and forget about it, until someone new joins, and you repeat the steps religiously as documented in your README.md, while digging into your memory to resolve issues with that one or two particular lib/s that fails to compile (e.g. mysql2, nokogiri, libv8, etc). Stackoverflow was a close ally. The README.md is littered with pointers like, "...if you see insert arcane error message, then do this...", and it soon warps into a maze of documentation to be navigated by the uninitiated. There is no guarantee it will still work in a few months, and the hardworking folks will try to update the documentation to reflect the current reality.
Tools such as chruby and asdf can install exactly the version you want for each language it handles (chruby being specific to ruby, and asdf being more generic, supporting a spectrum of languages and tooling through a plugin system). Yet tiny nuances across Debian vs CentOS vs macOS problems, never fail to creep into your journey.
Example of using asdf-vm :
brew update && brew upgrade
brew install coreutils curl git
brew install ruby-build
brew install asdf
asdf plugin-add ruby https://github.com/asdf-vm/asdf-ruby.git
asdf install ruby 2.7.1
cat > $HOME/.asdfrc << EOF
legacy_version_file = yes
EOF
cd ~/src_repos/my_ror_project
asdf local ruby 2.7.1
When docker containers came along around late 2013 and early 2014, it solved a huge bulk of issues, like preventing my host OS development workhorse from being polluted by various versions of the same binaries and additional libs and compiler configurations to get transitive dependencies to compile correctly. It provided a nice way to sandbox different projects with broadly varying toolchains, frameworks and programming languages installation and configuration. But running docker in macOS, while lighter, can still 'spin the fan' of your macOS a bit. Building the images is not considered the speediest way to iterate through changes in your build environment, like introducing new gems or hex packages, runtime environment changes, etc.
A new motivation, inspired by ihp
While asdf and docker containers worked well for me, it was not until I recently came across what the brilliant people at digitallyinduced were doing with the Integrated Haskel Platform. "IHP is using the Nix Package Manager for managing its dependencies". Coincidentally, we had a new team member join, working on our Ruby on Rails product. I spent some time navigating him "through the woods" to resolve the development environment setup peculiarities he was facing. The Nix package manager solution sort of resonated with me, and seems like a good candidate for me to dive down the rabbit hole to solve a nagging itch I wanted to resolve for our product's development environment setup.
Nix is described as "...a powerful package manager for Linux and other Unix systems that make package management reliable and reproducible. Share your development and build environments across different machines.". It touts to produce a Reproducible and Reliable language and tool to solve package and configuration management, Declaratively. A shell.nix file contains the syntax to describe the end desired state of your environment. The inputs from shell.nix defines the packages that are needed to get your development environment into a correct state, and these may include, your programming language toolchain, version control manager, database server, message queue tool, etc. The combined state of the whole of the environment is named a derivation in Nix parlance. A derivation is made up of functions that take other derivations being transitive dependencies as input and produce a derived reproducible state. Each derivation is sandboxed and built-in isolation, as a file system hashed directory location. A tiny change in the content of shell.nix results in an altogether different derivation. This ensures reproducibility. Multiple versions of the same binary/library can co-exist across derivations, without fear of interfering or polluting each other as each nix-shell is constructed from the derivation described by shell.nix. This produces an immutable and predictable environment outcome.
While all is good, to get Nix to work in macOS Catalina, requires a different setup, which until recently (Mar/Apr 2020), was still an unattainable situation (unless you are not on the latest macOS version).
A quick setup of Nix in macOS Catalina:
sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume
source ~/.nix-profile/etc/profile.d/nix.sh
To build up my nix-shell environment, I created a shell.nix file at the root of our project directory, where we manage the source tree artifacts using git. The shell.nix describes the entirety of the binaries and libraries required to spin up our local development environment to compile and install third party and custom libraries (in our case ruby gems), along with database server and whatever else is needed to run our development server and test suites. Here's a sample of our Ruby on Rails development environment shell.nix file (ruby 2.7, nodejs 14.7, maraidb 10.3, redis 6.0, python 3.8, ansible 2.9 and docker 19.03):
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
with stdenv;
let
apple_sdk = darwin.apple_sdk.frameworks;
ruby = ruby_2_7;
nodejs = nodejs-14_x;
python = python38;
ansible = ansible_2_9;
rubyPackages = rubyPackages_2_7;
pythonPackages = python38Packages;
in mkDerivation {
name = "less_site";
buildInputs = [
ruby bundler rake nodejs yarn python
autoconf cmake coreutils-full gcc10 gnumake gnupg
git git-lfs git-secrets gitAndTools.delta
bat bench binutils-unwrapped curl glances pkg-config
lzma time vagrant vim wget which
rubyPackages.mysql2 rubyPackages.sassc
rubyPackages.pry-byebug rubyPackages.pry rubyPackages.rubocop-performance
aws imagemagick7 jq libpcap libressl libsass libxml2 libxslt v8
libmysqlclient mariadb redis sqlite
chromedriver phantomjs2
pythonPackages.jq pythonPackages.pytest
ansible docker
] ++ lib.optionals isDarwin [
apple_sdk.CoreFoundation apple_sdk.CoreServices apple_sdk.Security
libiconv
];
}
With the above, I am ready to kick off the development environment auto build and configuration via nix-shell --pure command. This triggers a bunch of derivation builds of each dependencies described in the shell.nix file:
$ nix-shell --pure
these derivations will be built:
/nix/store/0rra8q6ywdkaavbjqy3zd4ixpj7j68ph-ruby2.7.1-unicode-display_width-1.6.0.drv
/nix/store/1lrzf9k16pgrkcky796f27hil1c6i3wq-builder.pl.drv
/nix/store/5xlgxm2hzx6hq5nwmcsccfd98gpw558w-ruby2.7.1-byebug-11.0.1.drv
/nix/store/5xmyyxnns9a04i02n6dngsxbb14qprgl-ruby2.7.1-jaro_winkler-1.5.4.drv
/nix/store/ak3y0rmxgfbs3v97ssdk4k26xf9m2l6h-ruby2.7.1-coderay-1.1.2.drv
/nix/store/f7izd845bgcfmv4881pskjghv79ws3yf-ruby2.7.1-method_source-0.9.2.drv
/nix/store/njwinj44kwwsc5bc757i379ha7axzmqf-rake-12.3.2.drv
.
.
Successfully installed rubocop-performance-1.5.2
1 gem installed
/nix/store/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2/lib/ruby/gems/2.7.0 /private/var/folders/11/btv6w5410yz8bc
yrx0nn6gsw0000gn/T/nix-build-ruby2.7.1-rubocop-performance-1.5.2.drv-0
removed 'cache/rubocop-performance-1.5.2.gem'
removed directory 'cache'
/private/var/folders/11/btv6w5410yz8bcyrx0nn6gsw0000gn/T/nix-build-ruby2.7.1-rubocop-performance-1.5.2.drv-0
post-installation fixup
rewriting symlink /nix/store/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2/nix-support/gem-meta/spec to be relative
to /nix/store/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2
strip is /nix/store/5vwyr9w0cy0g7hlzfrpnqas495av3xhj-cctools-binutils-darwin-927.0.2/bin/strip
stripping (with command strip and flags -S) in /nix/store/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2/lib /nix/s
tore/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2/bin
patching script interpreter paths in /nix/store/g4v67shwdfh554s19n0mm9403b6kxnsh-ruby2.7.1-rubocop-performance-1.5.2
[nix-shell:~/odd-e_code/less_site]$
If all goes well, I am dropped into a nix-shell prompt, which signifies the final state of my development environment. From here, I can proceed to run bundle, yarn tasks to install the project relevant ruby gems and yarn packages. I could perform database migration tasks via RAILS_ENV=test bundle exec rails db:migrate, and kick off our test suites via bundle exec rake, which runs both our unit test suite (RSpec) and our acceptance test suite (cucumber). Javascript unit test suite would run with yarn test. I am able to verify that all my tests are in good shape, by which time, I may decide to spin up my local Rails server and perform eyeball verification of the website from a real browser (firefox or chrome). All this while, noticing I am in good hands wherein no native binaries or libraries compilation would fail. If any such situations arrive, I will dig through OSS documentation or StackOverflow and repeat the changes in shell.nix contents to fill in the gaps and iterate till I get a working environment (each iteration I would repeat by nix-collect-garbage -d and destroying everything in vendor/bundle and node_modules). The final state is the known working state of a well working local Ruby on Rails development environment specific to our product's needs.
There is a catch here though. Due to the way, macOS implements the Security Framework, running nix-shell --pure and running your rails server web app with HTTPS, you will end up with an error (see nixpkgs GitHub issue ):
#<OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)>
I would ideally want to run my mint development environment unadulterated by what is there in my host OS (macOS Catalina) using nix-shell --pure. The above unfortunate roadblock is summarised in the Github issue discussion - "NixOS has the security.pki module that can be used to add custom certificates. Unfortunately, macOS does not come with an equivalent .crt bundle for us to use.". With this reality, I am currently working on the compromise of just running nix-shell, which circumvents the problem of missing OpenSSL certificates in my virgin development environment, while having the convenience to fallback on utilities and tooling supplemented by macOS. Not ideal, but it works.
UPDATE: The OpenSSL problem in macOS can be resolved by telling nix-shell where my ssl certs are located. I do that by export NIX_SSL_CERT_FILE=/etc/ssl/cert.pem. This finally gives me the ability to run test suite and manually browse to my locally started Rails web-app from a browser that hits it over HTTPS.
Iterating the shell.nix to arrive at my desired development environment
To iterate and incrementally test out changes in my shell.nix description of the target environment, I would nix-shell to build up the environment, perform tasks in the spawned nix-shell, then exit from the nix-shell and alter my shell.nix contents, then run nix-collect-garbage -d to clean up the previously cached state of the derivations, and repeat. I kept doing that and running each task needed in our development workflow until I could successfully complete all those tasks (e.g. static content generation, assets compilation, database reset, database migrations, javascript tests, rails test suite runs, etc).
Once I am satisfied with the outcome, it is time to commit the shell.nix into my codebase and push it upstream to share amongst my fellow teammates. All they have to do now is to pull the latest changeset from Github, install nix, and fire off nix-shell from the root of our product source tree.
Where it began and getting pushed off the cliff
While I had been toying with Nixpkgs and Nix for some time now in the last 2 years, pretty much as a go-between to slowly replace asdf and homebrew, it was honesty not until I got triggered from seeing how the guys at Digitally Induced - Integrated Haskel Platform had solved a typical problem, that made me take the dive to explore further. We had a valid need, and I had an itch. Took me about 3 days to get accustomed to the syntax of nix for shell.nix and understanding the iterative development workflow to come up with what I have now.
I hope this write up will give you the motivation to try it out and solve some real pressing problems in your quest for a reproducible development environment setup.
A write-up on nix and nixpkgs by the kind folks at Shopify that does the topic better justice: "What is Nix"