Core BeLeaf: Simplification @ALabsEngineering
Leaf was built in part with a focus on solving workflows that impede a grower’s operation but are in fact important. For example, having to worry about “starting” a task while operating a $500,000 piece of heavy equipment. While our solutions are unique to the ag industry, the problem is universal. The reality is workflow and process irritations affect every industry, and we faced them first hand bootstrapping our MVP. As a result internalizing core beliefs about simple, clean workflows became really important to us. We needed to practice what we preached.
In our engineering team, most of the work is done in a local development environment, so that became the focus of our attention.
A properly designed environment can remove process impediments for us the same way that Leaf removes the need for creating hundreds of tasks for a grower. Examples for us included the pain of setting up and starting the environment (more generally: actually getting the real work done) and software behaving differently in different environments (quickly diagnosing and fixing issues). The kinds of tasks can often add weeks of extra time to an engineering project.
The primary focus of this post is how Agrarian Labs invested in a development environment that our engineers loves working with and why that experience reinforced an important piece of our company's culture.
Getting started designing a simple, automated dev environment is straightforward: ask the team what obstacles need to be removed.
Characteristics we rallied around for our development environment:
The mechanics of installing, running, and deploying the environment are easy and user friendly.
The local dev environment is identical & replicable on every engineer’s machine.
The local dev environment is a scaled-down, mirrored version of production.
At a high level, we believed implementing these ideas would let us focus on getting the real work done. Let’s explore in depth what the implementation looks like.
The mechanics of installing, running, and deploying the environment are easy and user-friendly.
Setting up our local environment can be done from scratch in under an hour without prior experience on the team. This is despite the environment carrying a lot of complexity - we run a distributed microservices deployment. Essentially we’ve tried to mirror new environment setup from Leaf, where an operator can just hop in a tractor and start farming without touching a single button. Internally this tremendously accelerates our team’s on-boarding process and makes machine upgrades a non-issue. The entire process to start and run your first service at Agrarian Labs is:
Clone the desired repository.
Enter the directory and run $ make start
One of the core technologies that enables this workflow is Docker. Docker’s motto is “Build, Ship, and Run Any App, Anywhere”. Docker ensures that our environment is identical from one machine to the next by running our programs in containers that isolate them from outside influences.
The next crucial technology is Make, which we use in our build and deployment instructions. Our Makefile set automates instructions that speed up development and make code releases easy. One of the original build tools, Make encourages developers to record each step in the process. With make being machine-readable, you know exactly the commands you are making in the operating system. It “makes” explicit exactly what’s happening in your machine, and allows the team to easily find and fix bugs in the build or deployment process.
Go is the language of choice for our backend team, and Go contains a set of characteristics that enable a straightforward and automated workflow that are worth mentioning in this post. We like go for the simplicity, performance and speed of development it provides. Developers with no prior experience with Go find that learning the basics is fast and they can be productive in a short amount of time.
The final steps are cloning the git repository which has the service you want to work on and running the appropriate make target. At ALabs each service is represented by a Github repository. Consequently, each repository has a Makefile with instructions on how to get the service, and related services, started. An example would be starting our resources service. This is the service which exposes an API to perform CRUD operations on the system models. The workflow to do this is extraordinarily straightforward:
$ cd agrarianlabs/resources
$ make start
You will end up with each of the necessary micro-services in your local machine running and connected to each other. If your service exposes an API, then they are reachable via the $DOCKER_IP environment variable. For example, each of our services has a “/health” endpoint for general diagnostics. A simple command to test if things are working is:
$ curl http://$DOCKER_IP:$(cat .router_port)/resources/v1/health
Verifying the health of the resource service (after just two commands):
At this point, the resources service is up and running. /health gives me configuration values for my service. Note that I prepended “/health” with “/resources/v1”. That indicates that I want the health status on the resources service itself.
Our work to automate installing and running our environment results in a process of ~5 steps, concretely removing workflow obstacles for the team.
The local dev environment is identical & replicable on every engineer’s machine.
Those commands did much more than just kick off the resources service. It actually created an minimal environment to operate resources that will be identical on each developer's machine and inside different environments. Let’s take a look at everything that happened by calling the router’s health endpoint, rather than the specific resource service.
The routes indicates which services the router sees. Our minimal environment includes an authentication service, heartbeats service (to send geolocation data to the DB), the router service itself and our expected resources service.
We can run $ docker ps to get a more in depth look at the setup and workflow we’ve abstracted away.
In addition to a subset of our microservices, our environment includes the mdillon/postgis image for PostgeSQL, nsqio/nsq for NSQ and jplock/zookeeper for Zookeeper, among others. These images are essential to making the resource service work just like in production.
It’s worth mentioning a handful of other tools enabling our productive, simple and fun environment.
Docker images are uploaded in Docker Hub.
Service discovery started to connect all the services.
Seed data initializing state for the local postgres.
Below is a diagram of the type of architecture produced locally just by running $ make start in one of our repositories:
Some services communicate over HTTP API requests, some listen to NSQ messages and some do both. Some services expect authentication, while some don’t. Some need access to a datastore, while others need only memory. Each service has to function after make start is done. There is a lot of complexity in the background to simplify the experience for the developer.
Producing identical environments is powerful beyond just developer to developer communication. We have to trust various 3rd parties to run the same exact system we do. For example, let’s explore the chain of events when we update our software with a pull request created in Github.
A classic workflow to adjust behavior of a particular service looks like:
Make the fix.
Run $ make test locally.
If it passes, $ git commit then $ git push.
Since my environment is the same environment that the CI server uses, to run the test, then I know that my results will be the same in the CI server.
At this point, our team's workflow for starting, running, and deploying the Leaf platform is streamlined. We remove barriers to working with identical and replicable environments. But there’s one last goal we need to achieve to deal with real-world issues.
The local dev environment is a scaled-down, mirrored version of production.
To mirror production locally we need to replicate the internal state of the production application, and simulate incoming events from the real world that can change our application state. To achieve this we store a subset of our production data as seed files. These are SQL files which we then copy to the local postgres during startup.
In each repository we have a folder called “_fixtures” and a subfolder called “testdata”. In the testdata folder we have a zipped files containing production which we then unzip and copy to the local postgres instance.
Examples of what those files could look like are the following:
companies.sql.gz <- a list of companies in our system.
users.sql.gz <- a list of users in our system.
The companies.sql.gz file contains information such as this:
COPY companies (company_id, company_name, created_at, updated_at) FROM stdin;
1020 Austin Farms 2016-03-28 20:48:50.02006+00 2016-08-23 16:56:33.919459+00
1021 Stotz 2016-03-29 20:10:02.493055+00 2016-08-2316:57:18.078068+00
1022 Avondale Farms 2016-06-20 19:10:00.126764+00 2016-08-23 16:57:18.078068+00
From our seed files we can simulate what the application looks like for a production farm using Leaf in real life.
The next step in replicating production is to mirror the behaviors of our farms and users that can change our applications state.
A primary example of mimicking changes to the application would be feeding it geospatial data as though they came from real devices out in the field, and turn them into the different data artifacts & events that impact our system.. For example, we use geospatial data representing machine movement - known as heartbeats - from the field and convert them to farm operations such as tilling or harvesting. We currently receive millions of heartbeats per day, so it’s unreasonable to keep all of them in the local environment. But, we have to have some in order to run unit tests and to help debug production issues. We do this in two ways:
The first way is to store heartbeats, used by unit tests, as seed data files. Much like the profile information we have zip files representing this data.
The second way is to have a suite of tests ingesting the heartbeats in a similar manner to how they come in from iOS devices.
This example test takes the heartbeats from the file above and generates operations from them. The test expects the seed data to generate 5 operations.
// TestOperationCount tests the accuracy of the operation detection algorithm on
// company BP. The data set used can be found in: bp_hbs_20161101.sql.gzip
func TestOperationCount(t *testing.T) { ops := processDay(t, BPID, "2016-11-01T00:00:00-07:00", "2016-11-02T00:00:00-07:00") if expected, got := 5, len(ops); expected != got { t.Errorf("expected=%d operations, got=%d", expected, got) } }
An engineer who didn’t write this test can quickly know what data set is being used and how it’s being used in the test.
Tying It All Together
On the ALabs Engineering team, we’ve worked to abstract a complex environment (a distributed microservice deployment) into a simple user interface with several automated workflows. It allows us to focus on the creative and challenging parts of our job, and use computation as the workhorse.
In many ways it is an internal manifestation of what our product is focused on: turning a distributed environment on the farm into meaningful farm and business events without creating work for the user in the process.
We’ve combined three principles to deliver that workflow:
1. The mechanics of installing, running, and deploying the environment are easy and user friendly.
2. The work environment is identical & replicable on every engineer’s machine.
3. The work environment is a scaled-down, mirrored version of production.
In combination our engineering team has a simple process backed by complex infrastructure which results in us working smarter and enjoying the journey together. The end result is a simple process for our team backed by a complex architecture. Together they enable us to work smarter and appreciate our journey at Agrarian Labs.