Haskell for commercial software development
February 02, 2016 | Mike Craig

Inquiring minds on Quora want to know, is Haskell suitable for commercial software development?

I’m looking for an alternative to Python mainly for reasons of Python having so much trouble with workable concurrency, and Go seemed great at first look (it’s still not bad, but indeed error handling and lack of generics are a few features cut away too far).

This pretty much leaves Haskell as workable alternative. But is it really a practical alternative?

My biggest fear that while Haskell seems very powerful, it is also difficult and so it may be too difficult for person A to read code written by person B (what seems to have killed Lisp in my view is that problem and person-specific DSLs written in Lisp were simply unreadable for too many other people).

PS, in other words: I do not ask about strong sides of Haskell (or language X), those are well known, but rather about a lack of show-stoppers.

Absolutely, yes! At Wagon, we’ve built server- and client-side systems in Haskell, running in AWS and distributed to our users in a cross-platform desktop app.

Possible show-stoppers to using Haskell commercially:

Community support
A good open-source community is a big deal. It’s the difference between fixing other people’s code and finding an open GitHub issue with a workaround and an in-progress fix. With Haskell, we find the latter much more frequently than the former. The community is positive, active, and generally helpful when problems come up. Lively discussion takes place on Reddit, IRC, a public Slack channel, and several mailing lists.

Hiring
There are many factors involved in hiring: team size, location, remove vs on-site, other expertise required, etc. At Wagon we’re an on-site team of ~10 in San Francisco, with lots of web- and database-related problems to work on. Finding qualified Haskellers has not been an issue.

Haskell is different from the languages lots of developers are used to. ML-style syntax, strong types, and lazy evaluation make for a steep initial learning curve. Anecdotally, we’ve found Haskell’s secondary learning curve smooth and productive. Intermediate Haskellers quickly pick up advanced concepts on the job: parsers, monad transformers, higher-order types, profiling and optimization, etc.

Available industrial-strength libraries
Haskell lends itself to lightweight-but-powerful libraries that do one thing very well. We get a lot done by composing these small libraries rather than relying on big feature-complete frameworks. Some libraries have become de-facto standards that are stable and performant, and Stackage gives us a reliable way to find additional packages that usually “just work”.

We do sometimes find gaps in Haskell’s available libraries. For example, Haskell can’t yet match Python’s tools for numerical computing. And like every open-source ecosystem, there is some bad code out there. We rely on community recommendations and a sense of dependency hygiene to stay productive.

Performance
A quick Google search turns up benchmarks comparing Haskell to anything else you can imagine. In practice, we’re interested in tools with two properties:

1. Fast enough under most circumstances. “Fast enough” means performance is never an issue for a given piece of code.
2. Easy to optimize when needed. When we do need more speed, we’d like to get it incrementally rather than immediately rewrite that code in C.

Haskell has both of these properties. We almost never have to address performance in our Haskell code. When we do, we turn to GHC’s profiler and its many performance knobs: optimization flags, inlining, strictness and memory layout controls, etc. Haskell also has good interopability with C for when that’s appropriate.

Memory leaks
There’s a lot of talk about memory leaks caused by Haskell’s lazy evaluation model. We run into this very infrequently in practice, and in those rare situations GHC’s profiler has led us to solutions quickly. The real problem is not lazy evaluation but lazy IO, which we avoid with tools like conduit and pipes.

Debuggability
Haskell doesn’t have a debugger in the same sense as Python or Java. This is mostly a non-issue, because exceptions are rare and GHCi gives us a flexible way to run code interactively. Nonetheless, hunting down problems in a big application can be difficult. This has improved in recent versions of GHC with support for proper stack traces.

Haskellers reading other Haskellers’ code
This hasn’t been a problem for us. Haskell is a flexible language both in syntax and semantics, but this leads to better solutions more than it leads to opaque or unreadable code. Heavy indirection—via embedded DSLs or deep typeclass hierarchies—is unusual in practice. We reach for those tools as a last resort, knowing that they come with a serious maintenance cost. Agreeing on a basic style guide helps smooth out minor syntax differences.

Read our other Haskell engineering blog posts, come to our community events, or better yet, join our team!

FUNctional Programming with Haskell - Twitter Tech Talk
December 21, 2015 | Jeff Weinstein

Haskell is our primary backend language at Wagon— it helps us (safely) iterate faster, build for multiple environments, and attracts great engineering talent. It is fun to meet engineers from startups and large companies at Haskell meetups, BayHac, and in the functional programming Slack channel. Many engineers are curious how we use Haskell in production and have invited us to speak at their companies.

Wagon’s CTO Mike Craig spoke last month at Twitter (and earlier this year at Square) on the pros and cons of using Haskell in production. Thanks to Twitter’s Peter Seibel for inviting us and to their team for posting the video.

Here’s the talk:

If you’re using Haskell at your company, let us know.

Haskell in Production - Square Tech Talk
October 28, 2015 | Mike Craig

The Square engineering team invited Wagon to give a tech talk on how we use Haskell in production. Their teams are interested in functional programming and we were honored to walkthrough our experience building a modern analytics tool using Haskell, React, and Electron. Thanks Square for welcoming us!

If you’re using Haskell at your company, let us know. We’d love to trade notes.

Bayhac 2015
June 30, 2015 | Joe Nelson

Bayhac is back. The annual Bay Area Haskell conference and hackathon met last weekend for its fifth consecutive year, bringing us fascinating talks and strongly-typed bonhomie. It drew attendees from all over the country, even from faraway Oakland (I’m told to say, “Go Warriors!”).

Each year at Bayhac I am reminded how the Haskell community is thriving: more adoption, new libraries, robust tooling. It seems that the language is destined to fail at its goal of “avoiding success at all costs.” What struck me this year is the number of funded startups developing critical parts of their products in Haskell. They are springing up in San Francisco, some blocks away from the Wagon office, companies like Mirror, Front Row Education, Projector, IMVU, Alpha Heavy Industries, and Pingwell.

The conference started out strong with two talks which interested me personally. Tikhon Jelvis demonstrated how lazy evaluation is fundamental to designing modular code, not just an incidental curiosity. He gave numerous examples of laziness solving problems (the video for his talk is available here).

Then Dan Burton took the stage and gave us a tantalizing vision of the world post cabal-install, a glimpse of Stack. It is the spiritual successor of stackage-cli. In addition to locking project dependencies at mutually compatible versions like its predecessor, Stack can install necessary binaries and fetch packages right from git. Building projects, arguably the biggest pain in Haskelland, is getting better.

That was just Friday night. The rest of the weekend had plenty in store including our very own Mike Craig sharing Wagon’s experience of using Haskell. This technology choice has served our small team well and his talk dives in to its strengths and weaknesses. The presentation showcases our product space, technical architecture, the libraries we lean on, and our deployment strategy. He also discusses Haskell’s learning curve (pros and cons) as well as its positive impact on recruiting. It’s one of the core reasons we’re able to frequently ship new features with confidence.

If a picture is worth a thousand words then a thirty minute video at thirty frames per second is worth fifty-four million words. Here they are:

Big thanks to Bayhac organizers, notably Maxwell Swadling who stepped up to bring everyone together, handling everything from food to soliciting talks to the conference web presence. We’re looking forward to next year’s conference. In the meantime, keep a look out Wagon and community hosted Haskell events, or better yet, join our team!

My First Two Weeks of Haskell at Wagon
June 11, 2015 | Joe Nelson

It’s no secret I’m into Haskell. Two years ago a friend introduced me to what I then affectionately called “moon language,” and it’s been a continuous experience of learning, cursing, and joy ever since.

Loving the language is easy; finding a team who shares this functional programming passion and uses it to build their core product is more rare. That’s what I found at Wagon and I’d like to share my experience of the exciting first two weeks as a new Wagoner.

I’ve written Haskell professionally before, so that part wasn’t a change. What’s different is that my prior use of the language felt almost surreptitious. Building an API server here, writing a script there, whatever could fit comfortably within a more conventional consulting project. Now it’s different, everyone is onboard: we own our product destiny and technology choices.

For me it means being exposed to practices across the whole tech stack that were designed with Haskell in mind, not as an afterthought. From dockerizing Cabal to sharing and versioning packages between client and server code, to streaming statistical calculations over datasets with Conduit. There is a lot that goes into the company goal of creating a modern data collaboration tool.

Although we work together in person (in the Mission district of San Francisco where the walls are muralier, the coffee pour-overier and the bikes fixier) we do much of our code communication via pull requests. Everyone tries to have at least one other person review their work, which keeps us all in the know and leads to a lot of teaching. For instance, here is my teammate Mark suggesting an improvement to my coding style:

(bid, now) <- liftIO $ do
    b <- randomIO
    n <- getCurrentTime
    return (b, n)

Challenges go beyond mere coding style of course, and the team excels at debugging gnarly issues, including a memory leak caused by a useful yet tricky language feature called lazy evaluation. Our architecture is split between a shared server (Haskell), a client interface (Electron + React) and a client server (Haskell) for local data access. Nowadays it’s common for the browser component on the client to use hundreds of megabytes of memory while the Haskell component races along at twenty megabytes even when doing intensive streaming operations. This was not always the case however, and the team used the GHC profiler to identify and correct a memory leak in the streaming subsystem.

During the past two weeks I’ve been getting acquainted with more type-safe ways of doing things than I had previously tried. For instance I’ve previously worked with Hasql for database access (a good choice for low-level and performant PostgreSQL work) and now I’m learning Persistent, which uses Template Haskell to generate structured ways to access and migrate databases. The Wagon server includes a custom routing system with some handy combinators which has helped me think about patterns of improving code with types.

I’m excited to share more of what I learn in the coming weeks, both the coding and the cultural practices that help us do our best work, as well as things we’re learning to improve. (See also Engineering at Wagon post)

We’ll be at BayHac this weekend so make sure to check out our CTO Mike Craig’s talk Sunday, June 14 at 11am or find me. If you love this stuff the way we do, you should come spend a day with our team.

Engineering at Wagon
February 13, 2015 | Mike Craig

Over the last few months, we’ve been user-testing a preview version of Wagon—a modern SQL editor that powers a Google Docs-style experience for analysts, engineers, and their teams. We’ve iterated on both our design process and technology choices to build a tool that we and our users rely on every day.

Wagon takes advantage of cloud-local computing, combining the strengths of remote servers like syncing, sharing, and storage with local client advantages like data locality and device-specific user interfaces. Our architecture has three components: a frontend app, a native client, and backend services.

Wagon architecture

We make it easy to get started. You can download Wagon—a hybrid web application and native client, like Slack—and connect it to an existing local, private, or public database. The frontend is a slick interface for writing and running SQL, interacting with large results, and generating statistics and pivot tables. The native client locally handles connections, manages query runs, and processes results as they return from the database. The results, statistics, and pivots are cached locally and can be used to create sharable charts. When shared, Wagon securely transfers data from the local machine to the backend, making it available to permissioned teammates.

Frontend App

We use HTML, CSS, and JavaScript to maintain a consistent user experience across platforms and environments. The frontend is a single-page application built with React and Flux. We bundle it with the native client as a desktop application, and we have an in-browser experience for sharing and collaboration.

Technology: Javascript, React, Flux, Ace

• React gives us declarative, reusable UI elements. React components compose like HTML elements with rich state handling, so we manage fewer event listeners and nested callbacks.
• Flux complements React and helps structure our frontend code. It separates logic, state, and UI into loosely coupled components, and has a straightforward data flow.
• Ace is an extensible, embeddable code editor. We use its syntax highlighting and have extended its autocomplete.

Native Client

Our users want to query their databases without any new IT or admin setup, so users connect via their local machine with their existing credentials. Wagon handles connections through the native client, since most databases require socket-level network access that isn’t possible directly from browser.

Exploration and analysis can be a time-consuming and tedious job with existing SQL tools. Each step is an additional database query subject to queuing, slow execution, and changing data. Our users want to quickly explore results and evaluate summary statistics without executing subsequent queries. To power that experience, the client also computes statistics and caches query results on the local file system.

We built the native client in Haskell. It’s a high level language with great libraries, static compilation, a high performance ceiling, and an active community. We use MacGap to bundle the client and the frontend into a downloadable OS X application (Windows soon!).

Technology: Haskell, MacGap

• Prototyping and refactoring is safe with Haskell’s type-driven development. The compiler helps us avoid bugs and edge cases as we iterate.
• Haskell’s opt-in control over compiler optimizations and strictness allows for incremental optimization. GHC includes a capable profiler, and the community maintains libraries for testing and performance measurement.
• MacGap is a lightweight (~1MB) web-to-native wrapper. It provides a JS API to integrate with the file system, subprocesses, and OS-specific features like notifications and menus.

*Note: Wagon is now powered by Electron rather than MacGap. Read our Electron blog post to learn more.

Backend Services

Our backend services persists state and shared data to allow users to securely collaborate on queries, results, and charts. The backend is also written in Haskell: using the same language for the client and the server helps us stay flexible and reduce complexity. We store application state in Postgres while larger immutable data, like shared query results, live in Amazon S3.

Technology: Haskell, Postgres, Amazon S3

• Haskell’s lightweight threads and the GHC runtime scale well on multi-core machines.
• Postgres is powerful, flexible, and straightforward to host. We use features like TOAST and JSON support to test new features.

There are exciting upcoming technology challenges including optimizing compute across local and remote environments, designing efficient streaming statistics algorithms, building features that learn and improve with usage.

We’d love to hear from you! Follow @WagonHQ, check out our stack on Stackshare, and learn more about our engineering and design team. Stay tuned for future engineering deep dives!

Thanks to Steve Pike and Doug Petkanics for reviewing this post. Image credit Adam Wilson from The Noun Project.