How Wagon Rolls: Tooling
March 06, 2015 | Patrick Flor

#engineering

In a recent engineering blog post, our CTO Mike walked through our system architecture. As a complement, I’d like to describe the tools and processes we use to build Wagon.

We’re always learning new things from our customers that impact the product and our technologies. Fast iteration based on feedback drives our development, so we have invested significant time in testing, integration, deployment, and monitoring frameworks. We want to minimize the friction needed to turn new ideas into features.

Here is an overview of our tools and engineering development flow:

Docker + CoreOS

Docker and CoreOS gives us a common interface to our backend systems across development, staging, and production environments. If it works on my machine, it works in staging; if it misbehaves in staging or production, I can debug it on my machine. CoreOS’ clustering and service discovery features also give us a path forward as we scale.

Github + CircleCI

CircleCI’s enthusiasm for experimentation and great customer service have helped us quickly evolve our technical architecture. Their highly configurable Linux and Mac build environments let us consistently test and deploy pieces of Wagon while we transitioned from a server/browser web app to a hybrid native/JS desktop tool. CircleCI’s inferred build processes and Github integration help us to get traction on quick spikes, and their configuration and documentation let us customize CI and get back to building our product.

Here are some details of how we use them to minimize manual testing and deployment:

  • CircleCI builds and runs the full test suite on updates to every GitHub branch, so we know the state of any mainline or feature branch.
  • We dial up the counts on our generative tests in CI to ferret out uncommon and transient bugs in the build farm.
  • All changes are reviewed by at least one other team member before they are merged to a mainline branch. We use GitHub’s Pull Requests for code review and to accept changes.
  • Staging and production deployments are a direct function of successful CI builds on mainline branches. Any pull request accepted into master goes to staging and anything that makes our production branch goes live to users, with no additional tagging or manual deploy steps.
  • We use CircleCI’s API to kick off transitive builds for more complicated pipelines: this helps us keep the component projects small and single-purpose.
  • Each deployment pushes a notification to Slack, so everyone knows there is a new Wagon to enjoy.

Structured logs + Wagon

We log all application events–native, web, and server-side–as structured JSON. We use this data to answer questions about our systems post-hoc. It also gives us a structured stream to pass along to monitoring and alerting tools.

After some light ETL, we use Postgres’ awesome JSON support to query our logs with Wagon itself. This combines dogfooding and product analytics into a workflow that’s so fun and beneficial we dedicate a Slack channel to it. We use Wagon to investigate our application performance and user engagement the same way our users analyze their own data: by writing queries, visualizing results, and sharing charts.

Here’s an example of how we use these tools together in practice.

We find a bug! If we’re unsure how to reproduce it, we use Wagon to query our logs to confirm the bug and find more test cases. Since our development setup works just like production, we can quickly spin up a local instance of the specific environment we’re debugging–this makes root cause isolation much easier. When the fix is reviewed (any emoji on the Github PR is our stamp of approval) and merged into master, it automatically kicks off a staging CI build, whose success transitively launches a new build of our hybrid native/JS Mac app. CircleCI gives us a native OS X build environment with scaled-up generative testing, so we have an additional level of verification that our fix performs as expected. Within 5 minutes, the new app build is available in staging and we are alerted in Slack to download and give it a spin. When we’ve queued up a cohesive set of features and fixes, a pull request from master to the production branch automatically cuts a new release and deploys it to users. We then explore and monitor the effects of those changes by querying Wagon’s logs with Wagon.

If this sounds like fun to you, say hi at our sunny office on Valencia St. in San Francisco and check out our open positions: we’d love to talk to you!