Using overmind for development

What is overmind?

overmind is process manager for Procfile-based applications. Since it uses tmux under the hood, you can easily connect to processes to interact with them, view and search logs, and it forks to the background, allowing it survive shell resets.

I’ve heard of tools that can run Procfile applications locally (e.g. foreman) but never bothered to use them. I used tmux/tmuxinator since it was good enough, but those tools alone don’t give you the ability to manage the processes automatically.

After using overmind for a bit, it reminded of me Docker. While it doesn’t run the process in a container, it gives you similar controls (e.g. view logs, restart process, etc.). So why bother using overmind/foreman/tmux/etc. when you can just use Docker to develop apps?

When not to use Docker for local development

It’s common for developers to run one-off commands when developing apps. When using Docker, running commands like rails console or rails db:migrate means having to run docker exec $container bash to get a shell inside the container. This means:

No shell command history across container resets
You can’t customize the shell unless you plan on provisioning a ~/.{bash,zsh}rc inside the container along with all the addons and plugins, which limits how much team members can personalize their experience

Additionally, volume mounting means directories are created with root owner/group, so interacting with files that are created by the app can be tiresome. Its common for developers to want to delete or edit files generated by the app (e.g. uploads). Since docker containers run as root, accessing those files can be inconvenient. You could change the file permissions on a per-file basis, but you’d have to docker exec in, and run chown, or run sudo from the host and change permissions there. Overall, a bad experience, and adds a lot of overhead to development.

When to use Docker for local development

Docker is great for running backing services as it helps maintain dev/prod parity. This allows you to build your app against the same version of the backing services as you run on production. Also, if you need to run multiple versions of the same service (e.g. a database), doing so with the system package manager can be hectic, but with Docker, it’s trivial.

An ideal setup

With all that mind, the current setup looks something like this

Run main apps on the host, not inside containers
- ruby/python/node installed on the host and managed by rbenv/pyenv/nvm
- direnv
- overmind
Run backing services inside containers
- Docker & Docker Compose
Inside each project directory
- .envrc containing overmind_* vars among other vars
- Procfile.dev containing the application processeses
- docker-compose.yml file containing the required backing services
- .ruby-version/.python-version/.nvmrc files so versions can automatically be set when navigating to those directories

Where overmind can improve

Having used overmind for a while, I’ve already a created nice-to-have feature requests (1, 2). However, it can be improved further.

Updating environment variables without restarting overmind

One of overmind’s strengths is its ability to restart individual processes without restarting the whole stack. However, if you want to update an environment variable, you’ll have to restart overmind altogether. While overmind does support reading from environment files (.overmind.env and .env), it doesn’t re-read them when restarting a process.

The only workaround I can find is to manually source the env file

Procfile.dev

web: source `pwd`/.envrc && rails s

Built-in reverse proxy

Overmind allows you to scale processes e.g. overmind s -m web=2 will run 2 web processes and will set $PORT to a different value for each one. While this is neat, it’d be great if we could take advantage of that. To do so, overmind could create its own reverse proxy that can automatically forward traffic to web processes dynamically as they scale up or down. I imagine it could work similarly to Heroku’s router.

If this is beyond the scope of the project, an alternative would be to create another project that serves as a special reverse proxy that can communicate with overmind via the socket to dynamically route traffic to the desired processes. This could work similarly to Traefik, and how it dynamically routes traffic by communicating with the Docker daemon to monitor for new containers.

We could call it overmind-proxy and declare it in the Procfile. Its logic won’t have to be too complex. It should only route traffic to processes specified by the arguments.