Deploying Electron
February 23, 2016 | Matt DeLand

Electron makes it easy to write cross-platform desktop apps using web technology— but how do you build and deploy these hybrid apps in production?

At January’s Bay Area Electron Meetup, we presented how Wagon builds and delivers new updates to our users. This talk covers both standard and custom Electron usage for background updating, uninterrupted usage, helper processes, cross-platform builds, OS notifications, and the occasional hotfix.

---

Thanks to Ben Gotow at Nylas for organizing!

Haskell for commercial software development
February 02, 2016 | Mike Craig

Inquiring minds on Quora want to know, is Haskell suitable for commercial software development?

I’m looking for an alternative to Python mainly for reasons of Python having so much trouble with workable concurrency, and Go seemed great at first look (it’s still not bad, but indeed error handling and lack of generics are a few features cut away too far).

This pretty much leaves Haskell as workable alternative. But is it really a practical alternative?

My biggest fear that while Haskell seems very powerful, it is also difficult and so it may be too difficult for person A to read code written by person B (what seems to have killed Lisp in my view is that problem and person-specific DSLs written in Lisp were simply unreadable for too many other people).

PS, in other words: I do not ask about strong sides of Haskell (or language X), those are well known, but rather about a lack of show-stoppers.

Absolutely, yes! At Wagon, we’ve built server- and client-side systems in Haskell, running in AWS and distributed to our users in a cross-platform desktop app.

Possible show-stoppers to using Haskell commercially:

Community support
A good open-source community is a big deal. It’s the difference between fixing other people’s code and finding an open GitHub issue with a workaround and an in-progress fix. With Haskell, we find the latter much more frequently than the former. The community is positive, active, and generally helpful when problems come up. Lively discussion takes place on Reddit, IRC, a public Slack channel, and several mailing lists.

Hiring
There are many factors involved in hiring: team size, location, remove vs on-site, other expertise required, etc. At Wagon we’re an on-site team of ~10 in San Francisco, with lots of web- and database-related problems to work on. Finding qualified Haskellers has not been an issue.

Haskell is different from the languages lots of developers are used to. ML-style syntax, strong types, and lazy evaluation make for a steep initial learning curve. Anecdotally, we’ve found Haskell’s secondary learning curve smooth and productive. Intermediate Haskellers quickly pick up advanced concepts on the job: parsers, monad transformers, higher-order types, profiling and optimization, etc.

Available industrial-strength libraries
Haskell lends itself to lightweight-but-powerful libraries that do one thing very well. We get a lot done by composing these small libraries rather than relying on big feature-complete frameworks. Some libraries have become de-facto standards that are stable and performant, and Stackage gives us a reliable way to find additional packages that usually “just work”.

We do sometimes find gaps in Haskell’s available libraries. For example, Haskell can’t yet match Python’s tools for numerical computing. And like every open-source ecosystem, there is some bad code out there. We rely on community recommendations and a sense of dependency hygiene to stay productive.

Performance
A quick Google search turns up benchmarks comparing Haskell to anything else you can imagine. In practice, we’re interested in tools with two properties:

1. Fast enough under most circumstances. “Fast enough” means performance is never an issue for a given piece of code.
2. Easy to optimize when needed. When we do need more speed, we’d like to get it incrementally rather than immediately rewrite that code in C.

Haskell has both of these properties. We almost never have to address performance in our Haskell code. When we do, we turn to GHC’s profiler and its many performance knobs: optimization flags, inlining, strictness and memory layout controls, etc. Haskell also has good interopability with C for when that’s appropriate.

Memory leaks
There’s a lot of talk about memory leaks caused by Haskell’s lazy evaluation model. We run into this very infrequently in practice, and in those rare situations GHC’s profiler has led us to solutions quickly. The real problem is not lazy evaluation but lazy IO, which we avoid with tools like conduit and pipes.

Debuggability
Haskell doesn’t have a debugger in the same sense as Python or Java. This is mostly a non-issue, because exceptions are rare and GHCi gives us a flexible way to run code interactively. Nonetheless, hunting down problems in a big application can be difficult. This has improved in recent versions of GHC with support for proper stack traces.

Haskellers reading other Haskellers’ code
This hasn’t been a problem for us. Haskell is a flexible language both in syntax and semantics, but this leads to better solutions more than it leads to opaque or unreadable code. Heavy indirection—via embedded DSLs or deep typeclass hierarchies—is unusual in practice. We reach for those tools as a last resort, knowing that they come with a serious maintenance cost. Agreeing on a basic style guide helps smooth out minor syntax differences.

Read our other Haskell engineering blog posts, come to our community events, or better yet, join our team!

FUNctional Programming with Haskell - Twitter Tech Talk
December 21, 2015 | Jeff Weinstein

Haskell is our primary backend language at Wagon— it helps us (safely) iterate faster, build for multiple environments, and attracts great engineering talent. It is fun to meet engineers from startups and large companies at Haskell meetups, BayHac, and in the functional programming Slack channel. Many engineers are curious how we use Haskell in production and have invited us to speak at their companies.

Wagon’s CTO Mike Craig spoke last month at Twitter (and earlier this year at Square) on the pros and cons of using Haskell in production. Thanks to Twitter’s Peter Seibel for inviting us and to their team for posting the video.

Here’s the talk:

If you’re using Haskell at your company, let us know.

Electron meetup at Microsoft
November 06, 2015 | Jeff Weinstein

Electron has momentum. The open source project for building cross platform desktop apps with web technologies now has 6000+ commits from 200+ contributors, a 3500+ person Slack room, and now it’s 4th meetup with 90+ RSVPs (the 1st was a few of us at a bar).

On Monday, the Bay Area Electron group met at Microsoft Reactor, an event space in San Francisco. A few members of Microsoft’s open source team were in town to hear how people are using Electron to build Windows apps. The team is dedicated to helping projects run well with Microsoft platforms. It’s also exciting to see that Visual Studio Code is using Electron. Two of the five talks were Windows related:

---

Kevin Sawicki from GitHub spoke about how to test Electron apps using ChromeDriver and his project Spectron. Try this in your next project.

Felix Rieseberg from Microsoft’s open source team gave us walkthrough of WinJS. It seems like a great way to make Electron apps look native on Windows 10. Here are the slides.

John Haley from Axosoft’s GitKraken team talked about how they handle task scheduling in their Electron apps. This strategy seems like a new standard way to handle both UI and processing intensive applications.

Wagon’s Matt DeLand presented a few tips and tricks for developing, building, and deploying on Windows along with a demo of our latest version. Sign up for early access to try Wagon on Windows.

Edgar Aroutiounian gave a quick demo of how he used OCaml to build an Electron app. Checkout the example project here.

---

Special thanks to Microsoft for hosting. We’re excited the Electron community is growing and that large companies are adopting and supporting this platform. See you at the next event!

Haskell in Production - Square Tech Talk
October 28, 2015 | Mike Craig

The Square engineering team invited Wagon to give a tech talk on how we use Haskell in production. Their teams are interested in functional programming and we were honored to walkthrough our experience building a modern analytics tool using Haskell, React, and Electron. Thanks Square for welcoming us!

If you’re using Haskell at your company, let us know. We’d love to trade notes.

React at Wagon
October 01, 2015 | Mike Craig

We’re building a hybrid web/native application that runs both in the browser and as a downloadable desktop app. Analysts use Wagon to query, analyze, visualize, and share data: the app is highly interactive and data-heavy. It has to be fast, furious, and stable even when used for hours.

It ain’t all gravy: it’s difficult to maintain a UI with asynchronous updates, large scale data manipulation, and many other cross-cutting concerns. How can we build a sane frontend codebase, without losing our ability to iterate and ship quickly? The answer is to separate concerns. We break our UI into small self-contained components, and we isolate state and manage it separately from the UI. Facebook’s React and Flux libraries make this practical.

The big idea behind React is this: a UI component is just a function from its inputs to its content. All a component needs is a render() method that returns the elements we want the user to see. As an example, here’s a component that takes a size and color and renders a div displaying a filled-in square. Notice that users of this component don’t need to know about how it is implemented.

React components are simple to reuse because they nest like HTML elements. It’s easy to wrap an existing component to add additional styles or behavior—React favors composition over inheritance. We can forget about carefully maintaining the DOM to avoid excessive redraws and flicker: we declare what our components should look like, and React makes it so.

React is great for organizing view elements, but an application is more than static UI. Users generate events and we need to capture them, update state, and direct how the app should respond. Flux manages this flow by clearly seperating user action events from application responses.

Actions encapsulate events. They’re the application logic that runs in response to users doing stuff. In our example, when a user clicks a colored square, we update the server and dispatch to let the rest of app know what happened:

Stores encapsulate state. They listen to state changes dispatched from actions, and they update themselves to record the changes.

UI components listen to stores and re-render when relevant state changes.

Building a solid, maintainable frontend is still difficult despite these great libraries. Here are a few other strategies we’re using:

• Build pure React components whenever possible. A pure component doesn’t make any external calls from render()—it’s a pure function of the component’s properties and state. Components like this are much easier to test, debug, and reuse. It’s such a good idea it’s included in React itself!

• Separate React components that listen to Flux stores from those that render the UI. Wrapping UI components in container components is another win for reuse. Jason Bonta mentioned this in his great talk at React.js Conf 2015.

• Take care when integrating non-React UI components. Mixing React’s declarative API with another library’s imperative API can be painful. Build wrapper components around external libraries, and use React’s lifecycle methods to handle setup and teardown. When possible, avoid exposing direct-update methods like drawChart() or setCursorPosition()—manage state through component properties or Flux stores.

• Split Flux actions into modules by UX concern. We separate navigation and authentication from running queries and making charts. Carve out submodules for cross-cutting concerns, like AJAX requests or logging.

• Split Flux stores by domain. It’s helpful to separate persisted server-side state from ephemeral page state, for example. We hide the state of the URL bar behind a store, too!

We’re tackling fun engineering challenges at Wagon. If you want to learn more or work everyday on these technologies, check out our jobs page and get in touch!

Electron's first meetup
August 28, 2015 | Jeff Weinstein

The Electron community is positively charged! The project, formerly known as Atom-Shell, is a great way to build cross-platform desktop apps using web technologies. Given the 2200 people in the Electron Slack channel, the nearly five thousand commits, and the active discussion board, we figured it’s about time the project had a meetup in real life.

The community seems to agree: we hosted more than 50 people for the inaugural Bay Area Electron Meetup for four great talks and a big pile of pizza. Engineers from GitHub, Slack, Wagon, and Nylas spoke about building polished apps across Mac, Windows, and Linux using the Electron framework.

1. The History of Electron by Kevin Sawicki (Github)
2. Integrating with Native Code by Paul Betts (Slack)
3. Electron, React, and Haskell. Oh my! by Mike Craig (Wagon)
4. Making a web app feel native by Ben Gotow (Nylas)

---

The History of Electron by Kevin Sawicki (Engineer at GitHub)

Why Electron was built, what makes it special, how it differs from similar frameworks, and where it is going.

The History of electron

Integrating with Native Code by Paul Betts (Engineer at Slack)

How Slack’s Desktop app calls native operating system and library methods on OS X, Windows, and Linux, via Node Native Modules, node-ffi, and edge.js.

Native Modules in Electron

Electron, React, and Haskell. Oh my! by Mike Craig (CTO at Wagon)

How Wagon is building a hybrid desktop/web data analytics app using Electron, React, and Haskell to solve engineering challenges that aren’t possible with the browser alone.

Electron, React, and Haskell Oh My!

Making a web app feel native by Ben Gotow (Engineer at Nylas)

A dive into some JavaScript and CSS tricks the Nylas team uses to make their Electron-based mail app feel native across Mac, Windows, and Linux.

Building Native Experiences with Electron

---

If you’d like to come to the next one, join the Electron meetup group and we’ll see you soon.

Building an Analytics Pipeline in 2015
August 06, 2015 | Andy Granowitz

Every company needs an analytics data platform queryable by SQL.

Using a single analytics tool or relying on logs from a single source is a fast way to get started but is rarely sufficient. You’ll realize you need a better data strategy when attempting more detailed analytics tasks: cohorting customers based on segments available in multiple data sources, analyzing long time scale trends, or making data available to other applications. Unfortunately, you’ll quickly reach the limit of your off-the-shelf tool.

There has been a dramatic increase in data being created and fascination with Big Data, but less of a corresponding uptick in how to capture its value. Engineering a system to ingest, transform, and process data from many (changing, flaky) sources has been a long time, Very Hard Problem™. Doing this well requires hard work – the dreaded ETL.

We see more and more companies choosing to invest in SQL warehouses and the requisite engineering well before they become large businesses. How do you effectively build one of these things? And what makes building robust analytics infrastructure difficult?

See full Google Trends report

Here’s an example illustrating the core problems: You implemented a new purchase flow in your app and you’d like to understand conversion rates (tracked from logs) broken down by your email marketing A/B test (tracked from a 3rd party). The log lines you’re generating have new structure and may need to be re-parsed to fit into your existing schema. The A/B testing info may live in a different place than user data. Boiler plate reporting tools and drag and drop analytics UIs are great, but they require structuring ahead of time and the new checkout flow change is already live in production. Manually doing this analysis one time is annoying, but turning it into a reliable, repeatable practice is nearly impossible without dedicated engineering effort.

Your goal should be to get your data into a data warehouse that can be queried directly by people and programs. While it’s not straightforward, it’s important to understand the pieces. We see companies addressing this problem by focusing on the following steps:

1. For each data source: generate, collect, and store your data
2. Transform data into usable, queryable form
3. Copy multiple sources into a single place
4. Enjoy the data fruits of your data labor

The first step is collecting the data with as much structure as possible. You need to generate the data, transmit it from apps, browsers, or services for collection, and then safely store it for later. Many mobile and web analytics providers offer these three steps, others focus on a subset. For example, Heap and Mixpanel generate many app usage events automatically. Others focus on receiving data and making it available to read later (Keen and Splunk as different examples). Segment takes advantage of the difficulty of logging to many places by transmitting data to many of the above services with one API call.

Another large source of data is logs (usually messy and unstructured). Just having logs is not enough - it must be massaged into usable rows and columns. Some log lines help engineers analyze technology performance or debug errors, some log lines must be interpreted to signal “human” events, and some log lines have been around for so long that no one remembers why they’re there. Logs are rarely generated with their end purpose or a fixed type system in mind. Transformation of these raw strings is necessary to make them usable rather than just searchable.

For example, you may need to combine three separate log lines in order to signal a successful-user-flow, or to compare log lines against prior data to understand if a user is new, active or re-activated. Or maybe you need to remove those extra pesky spaces around your beautiful integers or standardize timestamps across timezones. Trifacta, Paxata, and Tamr offer technical tools for transforming ugly log forms to structured rows and columns. Or you’ll roll your own.

Once data collection systems are in place, you want to get this data flowing into a data warehouse. While some of the aforementioned tools provide their own interface for accessing collected and processed data, joining across multiple sources is difficult if not impossible, and their interfaces are often inflexible and cumbersome. Luckily, many of these services recognize this, and offer easy exports to data warehouses. Amplitude and Segment do this well and offer straightforward exports to Redshift. Google Analytics offers export to BigQuery for Premium customers (~$150k / year). Others make it possible, but require a bit of work (for example, Keen). New startups like Textur and Alooma are working on plumbing data into hosted RDBMS systems.

Outside of dedicated analytics solutions, you often have data from third party sources you’d like to join and analyze (e.g. Salesforce, ZenDesk, MailChimp, etc.). Most of these tools offer APIs to extract data. Building and maintaining these connections from 3rd parties to your data warehouse on your own is doable, but this is often where data integration tools are helpful. Services like Snowplow and Fivetran help.

At this point, data is flowing in a structured way. But where is it going? When assessing a data warehouse, look for:

1. Large scale ingestion and storage
2. Fast computation and multi-user concurrency
3. Easy management and scaling with data
4. Security and permissioning

There are many that meet these criteria: Amazon Redshift, Google BigQuery, Microsoft Azure SQL Data Warehouse, Hive, Spark, Greenplum, Vertica, Impala (the list goes on!). The largest technology companies (Amazon, Google, Microsoft) are investing in, and subsidizing, these data warehousing solutions. It’s a crucial component to nearly every business, which naturally draws the attention of the tech titans.

It’s a data jungle out there. Diagram from 2014 by Matt Turck at FirstMark

---

Phew, now you can enjoy the freedom of querying structured data, and the work (and fun!) begins. We’ll have more on the data analysis step soon!

We’d love to hear how you’re tackling these problems. What are your favorite tools? Where are your most painful pains? Tweet at @WagonHQ or send us a note at hello@wagonhq.com!

Electron at Wagon
July 15, 2015 | Mark Daly

While building Wagon, we’ve encountered a few engineering challenges that aren’t easily solved in the browser. Our users want Wagon to connect securely to their database and analyze large amounts of data, while being easy to setup and always up to date. Unfortunately, browsers can’t connect directly to databases and aren’t optimized for processing millions of data results. The standard web browser won’t suffice, so what should we do?

As many companies are discovering, mixing web and native technologies is very compelling. Early adopters including Slack, Atom, Quip, Visual Studio, Spotify, MapBox, Front, and Nylas have found ways to weave these once unrelated approaches.

At Wagon, we’re using Github’s Electron (previously “Atom-Shell”) as our underlying app framework. Electron was carved out of the Atom editor project and lets us deploy web UIs to the desktop. Our CTO Mike described Wagon’s technical architecture: a Javascript application for user experience along with a native process for database connections and streaming data computation. We want a capable desktop application with the ease of developing for the web.

The earliest alpha of Wagon was a command line program that powered a browser app available at localhost. We soon shipped our first Mac app: a download-able, double-click-able version using MacGap. As more people used Wagon, we wanted to update the code silently in the background, deploy to Windows and Linux, and move away from WebKit (we <3 Chromium’s dev tools!). We frequently ship new versions of both the JS and compiled Haskell, and it was painful for our users to manually download and reinstall the application. We briefly considered Mac-specific update mechanisms but it became clear we needed to replace MacGap. In searching for a full-featured, cross-platform web-view container, we found Electron.

Electron is based on Chromium, runs on multiple platforms, and comes packaged with useful features like desktop notifications, custom keyboard shortcuts, and native menus. Auto-update, which originally motivated us to try Electron, is easy to set up and simplifies shipping new versions. Electron is evolving quickly and openly, supported by its vibrant community, active development, public Slack room, and commercial backing.

Migrating from MacGap to Electron was straightforward, as Electron’s excellent documentation has instructions on rebranding, packaging, and distribution–we got a prototype up and running on a Friday afternoon. Electron differs from other web-view containers by using Node.js (via io.js) as its entry point: when an Electron app starts, execution begins in a JS program included in the app bundle, which can open windows and interact with the host OS. Node’s and Electron’s rich JS APIs made porting from MacGap easy, and we’ve added features that would otherwise only be possible in a native application (like custom menus and dialogs).

Our static assets are hosted on CloudFront, so we can update the UI without requiring users to redownload the whole application. However, it can take a few seconds for these assets to load and we want an immediate cue that Wagon is working. Here’s how Wagon ensures a smooth app launch experience:

1. On app start, the main Node process runs our JS App Loader. It loads configuration files, starts background tasks, and opens a renderer to load the latest UI.
2. The renderer immediately displays a splash.html page that is shipped with the Wagon.app bundle.
3. The splash page uses Electron’s <webview> tag to load our remote assets and start the JS app in the background while the Wagon logo rolls (some say dances!) on screen.
4. Once the UI is ready, it notifies the splash page via Electron’s IPC API.
5. The splash page swaps its content for the UI, creating a seamless transition into the full application.

When we’re ready to roll out a new version of Wagon.app, we use an automated deployment approach built on GitHub and CircleCI. When we merge a pull request into the master or production branches of our repos, CircleCI automatically builds the application bundle. The components are dropped into the Electron app structure, code-signed, and uploaded to S3. CircleCI also updates a configuration file that our auto-updater API endpoint reads, which lets us notify running instances of Wagon that a new version is available. The update is automatically downloaded in the background, installed, and triggers a desktop notification for the user.

We believe that great software should run in the browser, on phones, and on the desktop. If building across these platforms sounds exciting, check out our open positions, and email us or tweet @WagonHQ. Gogogo!

Bayhac 2015
June 30, 2015 | Joe Nelson

Bayhac is back. The annual Bay Area Haskell conference and hackathon met last weekend for its fifth consecutive year, bringing us fascinating talks and strongly-typed bonhomie. It drew attendees from all over the country, even from faraway Oakland (I’m told to say, “Go Warriors!”).

Each year at Bayhac I am reminded how the Haskell community is thriving: more adoption, new libraries, robust tooling. It seems that the language is destined to fail at its goal of “avoiding success at all costs.” What struck me this year is the number of funded startups developing critical parts of their products in Haskell. They are springing up in San Francisco, some blocks away from the Wagon office, companies like Mirror, Front Row Education, Projector, IMVU, Alpha Heavy Industries, and Pingwell.

The conference started out strong with two talks which interested me personally. Tikhon Jelvis demonstrated how lazy evaluation is fundamental to designing modular code, not just an incidental curiosity. He gave numerous examples of laziness solving problems (the video for his talk is available here).

Then Dan Burton took the stage and gave us a tantalizing vision of the world post cabal-install, a glimpse of Stack. It is the spiritual successor of stackage-cli. In addition to locking project dependencies at mutually compatible versions like its predecessor, Stack can install necessary binaries and fetch packages right from git. Building projects, arguably the biggest pain in Haskelland, is getting better.

That was just Friday night. The rest of the weekend had plenty in store including our very own Mike Craig sharing Wagon’s experience of using Haskell. This technology choice has served our small team well and his talk dives in to its strengths and weaknesses. The presentation showcases our product space, technical architecture, the libraries we lean on, and our deployment strategy. He also discusses Haskell’s learning curve (pros and cons) as well as its positive impact on recruiting. It’s one of the core reasons we’re able to frequently ship new features with confidence.

If a picture is worth a thousand words then a thirty minute video at thirty frames per second is worth fifty-four million words. Here they are:

Big thanks to Bayhac organizers, notably Maxwell Swadling who stepped up to bring everyone together, handling everything from food to soliciting talks to the conference web presence. We’re looking forward to next year’s conference. In the meantime, keep a look out Wagon and community hosted Haskell events, or better yet, join our team!

Weekly Roundup: React
May 27, 2015 | Mike Craig

Our hybrid native/web application’s frontend layer is built with Javascript, React, and Flux. In 2014, our friends at Facebook advised us to try React and it has since been a core part of our stack.

The React community is bustling and there are some great posts to help people start, organize, and scale frontend applications. These articles have helped us:

1. Alex Lopatin details Sift Science’s migration from Backbone to React in Best practices for building large React applications.
2. Alexander Early from Fluid maintains an in-depth set of Tips and Best Practices. The discussion in the comments is good too.
3. Facebook’s Christopher Chedeau has many talks on React architectures but this review of how to scale CSS in a React stack guided our development of Wagon’s custom UI. Merci!
4. Alex Schepanovski describes React from a jQuery perspective in Boiling React Down to a Few Lines in jQuery.
5. Ryan Clark has a technical but still approachable Getting Started With React walkthrough. If you like that, read the sibling article Getting Started with Flux too.

We’ll be doing a deeper dive into Wagon’s React architecture, tooling, and how we see it changing in our next engineering blog post. If you like to hack on React, Flux, Javascript, and Electron, we’re hiring a frontend engineer in San Francisco. Say hi!

Weekly Roundup: Cloud Data Posts
March 12, 2015 | Jeff Weinstein

Over burritos, our team enjoys discussing what’s new in cloud data infrastructure. Here are a few recent articles and announcements that caught our eye:

1. Three Myths about Hybrid Architectures Using the Cloud

Stephen Orban, Head of AWS Enterprise Strategy, unpacks a few commonly held beliefs about hybrid architectures. A must read for larger companies that are attempting to migrate from legacy systems.

2. Video of Mesosphere's DCOS

Check out this awesome demonstration and rad visualization of Mesosphere’s DCOS by Matt Trifiro.

3. Observations on the Importance of Cloud-based Analytics

Werner Vogels, CTO at Amazon, details how cloud based analytics can improve different sectors including healthcare, cities, internet of things as well as business intelligence.

4. Introducing Google Cloud Storage Nearline: (near)online data at an offline price

Google recently announced $0.01 per GB per month storage, available with ~3 second response time. Amazing!

5. Cloud-Client Computing

A few months ago, Andreessen Horowitz published 16 ideas they’re watching and the one that spoke to us the most is cloud-client computing. It’s how we’ve architected Wagon and is worth a repost!

Also, shoutout to our friend and Quizlet infrastructure lead Peter Bakkum on his open source tool for capturing and replaying HTTP traffic. We love seeing the old Hyperpublic crew doing big things.

How Wagon Rolls: Tooling
March 06, 2015 | Patrick Flor

In a recent engineering blog post, our CTO Mike walked through our system architecture. As a complement, I’d like to describe the tools and processes we use to build Wagon.

We’re always learning new things from our customers that impact the product and our technologies. Fast iteration based on feedback drives our development, so we have invested significant time in testing, integration, deployment, and monitoring frameworks. We want to minimize the friction needed to turn new ideas into features.

Here is an overview of our tools and engineering development flow:

Docker + CoreOS

Docker and CoreOS gives us a common interface to our backend systems across development, staging, and production environments. If it works on my machine, it works in staging; if it misbehaves in staging or production, I can debug it on my machine. CoreOS’ clustering and service discovery features also give us a path forward as we scale.

Github + CircleCI

CircleCI’s enthusiasm for experimentation and great customer service have helped us quickly evolve our technical architecture. Their highly configurable Linux and Mac build environments let us consistently test and deploy pieces of Wagon while we transitioned from a server/browser web app to a hybrid native/JS desktop tool. CircleCI’s inferred build processes and Github integration help us to get traction on quick spikes, and their configuration and documentation let us customize CI and get back to building our product.

Here are some details of how we use them to minimize manual testing and deployment:

• CircleCI builds and runs the full test suite on updates to every GitHub branch, so we know the state of any mainline or feature branch.
• We dial up the counts on our generative tests in CI to ferret out uncommon and transient bugs in the build farm.
• All changes are reviewed by at least one other team member before they are merged to a mainline branch. We use GitHub’s Pull Requests for code review and to accept changes.
• Staging and production deployments are a direct function of successful CI builds on mainline branches. Any pull request accepted into master goes to staging and anything that makes our production branch goes live to users, with no additional tagging or manual deploy steps.
• We use CircleCI’s API to kick off transitive builds for more complicated pipelines: this helps us keep the component projects small and single-purpose.
• Each deployment pushes a notification to Slack, so everyone knows there is a new Wagon to enjoy.

Structured logs + Wagon

We log all application events–native, web, and server-side–as structured JSON. We use this data to answer questions about our systems post-hoc. It also gives us a structured stream to pass along to monitoring and alerting tools.

After some light ETL, we use Postgres’ awesome JSON support to query our logs with Wagon itself. This combines dogfooding and product analytics into a workflow that’s so fun and beneficial we dedicate a Slack channel to it. We use Wagon to investigate our application performance and user engagement the same way our users analyze their own data: by writing queries, visualizing results, and sharing charts.

---

Here’s an example of how we use these tools together in practice.

We find a bug! If we’re unsure how to reproduce it, we use Wagon to query our logs to confirm the bug and find more test cases. Since our development setup works just like production, we can quickly spin up a local instance of the specific environment we’re debugging–this makes root cause isolation much easier. When the fix is reviewed (any emoji on the Github PR is our stamp of approval) and merged into master, it automatically kicks off a staging CI build, whose success transitively launches a new build of our hybrid native/JS Mac app. CircleCI gives us a native OS X build environment with scaled-up generative testing, so we have an additional level of verification that our fix performs as expected. Within 5 minutes, the new app build is available in staging and we are alerted in Slack to download and give it a spin. When we’ve queued up a cohesive set of features and fixes, a pull request from master to the production branch automatically cuts a new release and deploys it to users. We then explore and monitor the effects of those changes by querying Wagon’s logs with Wagon.

If this sounds like fun to you, say hi at our sunny office on Valencia St. in San Francisco and check out our open positions: we’d love to talk to you!

Engineering at Wagon
February 13, 2015 | Mike Craig

Over the last few months, we’ve been user-testing a preview version of Wagon—a modern SQL editor that powers a Google Docs-style experience for analysts, engineers, and their teams. We’ve iterated on both our design process and technology choices to build a tool that we and our users rely on every day.

Wagon takes advantage of cloud-local computing, combining the strengths of remote servers like syncing, sharing, and storage with local client advantages like data locality and device-specific user interfaces. Our architecture has three components: a frontend app, a native client, and backend services.

Wagon architecture

We make it easy to get started. You can download Wagon—a hybrid web application and native client, like Slack—and connect it to an existing local, private, or public database. The frontend is a slick interface for writing and running SQL, interacting with large results, and generating statistics and pivot tables. The native client locally handles connections, manages query runs, and processes results as they return from the database. The results, statistics, and pivots are cached locally and can be used to create sharable charts. When shared, Wagon securely transfers data from the local machine to the backend, making it available to permissioned teammates.

Frontend App

We use HTML, CSS, and JavaScript to maintain a consistent user experience across platforms and environments. The frontend is a single-page application built with React and Flux. We bundle it with the native client as a desktop application, and we have an in-browser experience for sharing and collaboration.

Technology: Javascript, React, Flux, Ace

• React gives us declarative, reusable UI elements. React components compose like HTML elements with rich state handling, so we manage fewer event listeners and nested callbacks.
• Flux complements React and helps structure our frontend code. It separates logic, state, and UI into loosely coupled components, and has a straightforward data flow.
• Ace is an extensible, embeddable code editor. We use its syntax highlighting and have extended its autocomplete.

Native Client

Our users want to query their databases without any new IT or admin setup, so users connect via their local machine with their existing credentials. Wagon handles connections through the native client, since most databases require socket-level network access that isn’t possible directly from browser.

Exploration and analysis can be a time-consuming and tedious job with existing SQL tools. Each step is an additional database query subject to queuing, slow execution, and changing data. Our users want to quickly explore results and evaluate summary statistics without executing subsequent queries. To power that experience, the client also computes statistics and caches query results on the local file system.

We built the native client in Haskell. It’s a high level language with great libraries, static compilation, a high performance ceiling, and an active community. We use MacGap to bundle the client and the frontend into a downloadable OS X application (Windows soon!).

Technology: Haskell, MacGap

• Prototyping and refactoring is safe with Haskell’s type-driven development. The compiler helps us avoid bugs and edge cases as we iterate.
• Haskell’s opt-in control over compiler optimizations and strictness allows for incremental optimization. GHC includes a capable profiler, and the community maintains libraries for testing and performance measurement.
• MacGap is a lightweight (~1MB) web-to-native wrapper. It provides a JS API to integrate with the file system, subprocesses, and OS-specific features like notifications and menus.

*Note: Wagon is now powered by Electron rather than MacGap. Read our Electron blog post to learn more.

Backend Services

Our backend services persists state and shared data to allow users to securely collaborate on queries, results, and charts. The backend is also written in Haskell: using the same language for the client and the server helps us stay flexible and reduce complexity. We store application state in Postgres while larger immutable data, like shared query results, live in Amazon S3.

Technology: Haskell, Postgres, Amazon S3

• Haskell’s lightweight threads and the GHC runtime scale well on multi-core machines.
• Postgres is powerful, flexible, and straightforward to host. We use features like TOAST and JSON support to test new features.

There are exciting upcoming technology challenges including optimizing compute across local and remote environments, designing efficient streaming statistics algorithms, building features that learn and improve with usage.

We’d love to hear from you! Follow @WagonHQ, check out our stack on Stackshare, and learn more about our engineering and design team. Stay tuned for future engineering deep dives!

Thanks to Steve Pike and Doug Petkanics for reviewing this post. Image credit Adam Wilson from The Noun Project.

Design at Wagon
January 22, 2015 | Jeff Weinstein

We have a challenging design goal: build an easy-to-use product for analysts and their teams to collaborate on understanding data. Our product is (1) a database query editor for analysts and (2) a Google Docs-style experience for non-technical people to explore results together.

We’re looking for someone to be our first UX designer – come craft our user experience, our voice, and our pixels. You’ll grow a team of designers and engineers that love data analysis and collaboration software.

As a team of engineers and data scientists, we have more experience building systems than storyboarding. Over the past few months, we’ve developed a structured design workflow. It starts with stories, drawing, and getting feedback from customers. We then specify the application’s states and object models and finally code mockups. Here’s how it all works:

Mark and Mike sketch application states.

Story

Many friends and former colleagues have the analysis and collaboration problem that Wagon solves and they’ve been very helpful testing the product. We watch them work, document their pain points, and listen to their feature requests. We winnow our notes into specific user stories: “As an X person, I need to do Y, in order to get Z.” To avoid groupthink, we independently draft and prioritize these stories, then collectively compare and merge them.

Draw

We sketch on the whiteboard, use pen and paper, and drag and drop in Google Draw to visualize our ideas. We review screenshots and pictures in Github Issues and Slack and share paper prototypes using POP. One surprisingly successful mockup was built with ASCII in Sublime Text. As our friends at Wildcard detailed, there are plenty of other design tools that we’d be excited to learn.

Feedback

Early and frequent user testing is essential. Our beta testers’ eyes surface bugs and corner cases at a humbling rate. They contribute directly via our public Hipchat room and shared Google Docs. Fortunately, our users have been patient, thoughtful, and enjoy participating.

Specify

We define application states and user flows before writing any code. These detailed state diagrams are critical for building quickly and reducing the confusion and blockers that can arise during implementation.

Wireframe

During implementation, while APIs and data are developed and tested, we lay out the user experience with HTML/CSS. This helps us uncover usability problems and lets us tweak the design in conjunction with other tasks.

To stay focused as a team, our five core product values set expectations and help us measure progress.

Intuitive

Clean design and approachable voice

Aware

Communicate context and insight

Interoperable

Connect to the outside world

Responsive

Iterate at human speed, always in control

Collective

Shared knowledge improves individuals and unites teams

Design at Wagon has been a team effort. We’ve been lucky enough to be advised and supported by a set of design advisors. Curious about data and design at Wagon? Say hi.

Thank you to Hillary Kelly for reviewing this post.