The Serendipity Machine
August 11, 2015 | Jeff Weinstein

We enjoy wonderful little surprises every day at Wagon: serendipitous run-ins, small-world moments, and hilarious coincidences. Whether it’s the cadre of visitors we host for lunch, the tipsy bar patron who is secretly a SQL analyst, or the friend-of-a-friend lurking within our teammate’s apartment asking: “Do you work at that Haskell data startup?” - we’re amazed by where our fellow thinkers are hiding. The real secret is that these things, which might appear random, happen for a reason: we constantly seek to talk with people and bring them together. Serendipity doesn’t happen; we make it.

Secret Worlds by xkcd

We’re fortunate to have a wide network of people: users, open-source contributors, recruits, neighbors, partners, friends of friends, out-of-town visitors, investors, etc… It’s invigorating to hear stories from these different groups; their experiences motivate us and give context to the problems we’re solving. We talk with hundreds of people each day through Intercom chat, our public #bandwagon Slack channel, and on Twitter.

A happy surprise about early-stage startups is that you’ll meet more people than at a large company. Your network isn’t just your immediate team and your chain of managers, it’s anyone, anywhere.

We create the space for serendipitous encounters through tech meetups, our #bandwagon happy hours, open invites to lunch, our high touch user communication, and Wagon’s faux-EIR program. Hosting events at our space in San Francisco is a great way to bring people together. Almost every week, we have 40+ people at our space for an after-work event. To make sure things go smoothly, events even have their own checklists: “In Event of Meetup” and “In Case of Party”. We maintain a mini-CRM so we can throw impromptu events in other cities - we had a lot of fun hosting a happy hour in New York with Wildcard. We have an open space facing Valencia Street with views of Sutro Tower. Our Wagon office is configured for flexibility rather than posh — we hope you’ll find it welcoming.

Upcoming Wagon Events

Bay Area Haskell Meetup - Wed, Aug 12 at 6pm

Join us for three half-hour Haskell talks about distributed code, graphs, and more. Matt DeLand (Wagon), Tikhon Jelvis (Esper), and John Chee (Twitter) will be speaking at our next Haskell event.

Wharton Customer Analytics Meetup - Tue, Aug 18 at 6:30pm

With two Penn grads on the team, we’re excited to host the next Bay Area Wharton and Penn analytics event! There will be a few 5 minute “lightning talks” from alumni working on applications of customer analytics.

Electron Meetup - Tue, Aug 25 at 6pm

Learn from Kevin Sawicki (GitHub), Paul Betts (Slack), Mike Craig (Wagon), and Ben Gotow (Nylas) about how to make polished apps across Mac, Windows, and Linux using the open source project Electron (formerly Atom-Shell). We’ll have four 20-minute tech talks. Come celebrate the community’s first official meetup in San Francisco! There will be pizza.

#bandwagon Happy Hour - Wed, Sept 2 at 6pm

Come to our monthly happy hour at our office in San Francisco. It’s a great way to meet other data folks and friends of Wagon. It’s very relaxed so feel free to just pop in between 6 - 9pm, Wed Sept 2nd.

Joe Nelson’s fireside Wagon Haskell chat and a recent #bandwagon happy hour at our office in San Francisco.

Thanks to Mission Bicycle Company (our neighbors), Mom & Daughters Chair and Table Rentals (just-in-time chair rental), and Instacart (official Tecate supplier) for making these events possible.

We hope to see you soon! If you’d like to be our next Wagon faux-EIR or want to stop by to say hi, just email us at hello@wagonhq.com or give us a tweet at @WagonHQ.

Field trip to Arion Press
August 03, 2015 | Jeff Weinstein

Software can feel ephemeral: changing technologies, new platforms, and shifting demands. So, for our recent team outing, we wanted to learn about work that is timeless. We found Arion Press, a handmade book publishing company tucked away in the Presidio, situated between our other planned Outer Richmond activities: slurping soup dumplings and exploring the beaches south of the Golden Gate Bridge.

Arion Press, along with M & H Type, is the oldest and largest type foundry the country: Casting hot type daily since 1915! They’re true craftspeople: printers, typesetters, bookbinders, and publishers continuing the tradition of making books from scratch. They combine fine art, typography and prose; frequently pairing commissioned artwork from contemporary artists with great works of literature. They publish two or three high quality books each year.

Much like the puppet show that is software, producing books is (one) part technical mastery and (ten) parts hard work, checklists, patience, and teamwork. They have their own foundry where they manufacture individual types: think Times New Roman, 10 point, lowercase, italic, letter ‘a’ as one specific thing to be made by hand. They compose the prose by combining these individually made letters, molding them into larger lines and eventually pages. They then pass handmade paper through a machine about the size of two cars that requires three people to operate. At high speed, the machine dips the large metal page stamps into ink, presses it on to the paper, and flings it into a pile. It doesn’t seem like it should work but seems to have better performance and reliability than any office printer we’ve jammed.

After the pages are printed, dried, and have passed the close eye of inspection (they have to occasionally remove ink splotches with a knife), the pages are ready to be bound. The bookbinders taught us how everything comes together to go from the raw materials to a beautiful book.

Thanks to their whole team for having us. Read more about the history of Arion Press and maybe give them a visit during their Thursday tours (book ahead). Feeling inspired by the aesthetic of hand set type, here’s a quick version of our beloved Wagon that we hope to see inked one day:

We’re on the look out for other great San Francisco afternoon trips. If you’ve been to any other off the beaten path spots, let us know!

Wagon’s recent field trip. Not pictured: lots of Chinese food!

My First Two Weeks of Haskell at Wagon
June 11, 2015 | Joe Nelson

It’s no secret I’m into Haskell. Two years ago a friend introduced me to what I then affectionately called “moon language,” and it’s been a continuous experience of learning, cursing, and joy ever since.

Loving the language is easy; finding a team who shares this functional programming passion and uses it to build their core product is more rare. That’s what I found at Wagon and I’d like to share my experience of the exciting first two weeks as a new Wagoner.

I’ve written Haskell professionally before, so that part wasn’t a change. What’s different is that my prior use of the language felt almost surreptitious. Building an API server here, writing a script there, whatever could fit comfortably within a more conventional consulting project. Now it’s different, everyone is onboard: we own our product destiny and technology choices.

For me it means being exposed to practices across the whole tech stack that were designed with Haskell in mind, not as an afterthought. From dockerizing Cabal to sharing and versioning packages between client and server code, to streaming statistical calculations over datasets with Conduit. There is a lot that goes into the company goal of creating a modern data collaboration tool.

Although we work together in person (in the Mission district of San Francisco where the walls are muralier, the coffee pour-overier and the bikes fixier) we do much of our code communication via pull requests. Everyone tries to have at least one other person review their work, which keeps us all in the know and leads to a lot of teaching. For instance, here is my teammate Mark suggesting an improvement to my coding style:

(bid, now) <- liftIO $ do
    b <- randomIO
    n <- getCurrentTime
    return (b, n)

Challenges go beyond mere coding style of course, and the team excels at debugging gnarly issues, including a memory leak caused by a useful yet tricky language feature called lazy evaluation. Our architecture is split between a shared server (Haskell), a client interface (Electron + React) and a client server (Haskell) for local data access. Nowadays it’s common for the browser component on the client to use hundreds of megabytes of memory while the Haskell component races along at twenty megabytes even when doing intensive streaming operations. This was not always the case however, and the team used the GHC profiler to identify and correct a memory leak in the streaming subsystem.

During the past two weeks I’ve been getting acquainted with more type-safe ways of doing things than I had previously tried. For instance I’ve previously worked with Hasql for database access (a good choice for low-level and performant PostgreSQL work) and now I’m learning Persistent, which uses Template Haskell to generate structured ways to access and migrate databases. The Wagon server includes a custom routing system with some handy combinators which has helped me think about patterns of improving code with types.

I’m excited to share more of what I learn in the coming weeks, both the coding and the cultural practices that help us do our best work, as well as things we’re learning to improve. (See also Engineering at Wagon post)

We’ll be at BayHac this weekend so make sure to check out our CTO Mike Craig’s talk Sunday, June 14 at 11am or find me. If you love this stuff the way we do, you should come spend a day with our team.

Bandwagon
May 23, 2015 | Jeff Weinstein

Delighting our users drives the Wagon product and motivates our team. We see startups as a community of people coming together around a goal: ours is to make data analysis easier for everyone. Building something great is the collective work of our team, along with our users, friends, family, open-source contributors, advisors, and investors. Sharing the journey is such a positive force for us and it also happens to be a really fun way to run a company.

I'm having one of those moments where @WagonHQ feels like collective work of a community, not just our team. Wild.

— Jeff Weinstein (@jeff_weinstein) March 19, 2015

We want people to adore Wagon. Iterating on feedback and shipping frequently is our tact to building the best product (see our design post). While people tend to inherently enjoy giving their opinion, we know detailing constructive criticism and bug reports can be a burden on our users especially when the product is changing quickly. We focus on a few ways to encourage our community to come together: be welcoming, actively listen, communicate directly, encourage others, and have fun.

Our main community space is a shared Slack channel: appropriately named #bandwagon. We have 50+ people along with our whole team. It’s a safe place to ask questions about data, engineering, analytics, SQL, and give feedback on Wagon. The tone is helpful, collaborative, and positive. We like to make a custom emoji for each new bandwagoner when they sign up!

The conversation is often led by others and many people contribute to answering questions— it’s really fabulous. One early user described it as: “#bandwagon is an awesome Slack channel for Wagon users that makes the app feel bespoke; some of my feedback has been implemented!” We’ve even hired someone from within #bandwagon (welcome @agrano!).

It’s hard work to be active and available for our growing userbase but it’s worth the energy. Here’s a look at our team’s Slack activity over time…

Wagon's Slack messages history.

We love to interact with people on Twitter and Instagram , host monthly happy hours at our office in San Francisco, and welcome anyone to hang out for a day (we have both coffee and internet). We have t-shirts for bandwagoners and enjoy sending handwritten cards. On the silly side, kids love our adorable stickers! And while we don’t encourage vandalism, it’s fun to see that someone tagged our favorite watering hole.

Wagon’s monthly happy hours!

Without our community, we’d be just a small team with a long list of TODOs. But together, we feel like a group marching toward a common goal.

Want to hop on? Send us an email or tweet at @WagonHQ.

Engineering at Wagon
February 13, 2015 | Mike Craig

Over the last few months, we’ve been user-testing a preview version of Wagon—a modern SQL editor that powers a Google Docs-style experience for analysts, engineers, and their teams. We’ve iterated on both our design process and technology choices to build a tool that we and our users rely on every day.

Wagon takes advantage of cloud-local computing, combining the strengths of remote servers like syncing, sharing, and storage with local client advantages like data locality and device-specific user interfaces. Our architecture has three components: a frontend app, a native client, and backend services.

Wagon architecture

We make it easy to get started. You can download Wagon—a hybrid web application and native client, like Slack—and connect it to an existing local, private, or public database. The frontend is a slick interface for writing and running SQL, interacting with large results, and generating statistics and pivot tables. The native client locally handles connections, manages query runs, and processes results as they return from the database. The results, statistics, and pivots are cached locally and can be used to create sharable charts. When shared, Wagon securely transfers data from the local machine to the backend, making it available to permissioned teammates.

Frontend App

We use HTML, CSS, and JavaScript to maintain a consistent user experience across platforms and environments. The frontend is a single-page application built with React and Flux. We bundle it with the native client as a desktop application, and we have an in-browser experience for sharing and collaboration.

Technology: Javascript, React, Flux, Ace

• React gives us declarative, reusable UI elements. React components compose like HTML elements with rich state handling, so we manage fewer event listeners and nested callbacks.
• Flux complements React and helps structure our frontend code. It separates logic, state, and UI into loosely coupled components, and has a straightforward data flow.
• Ace is an extensible, embeddable code editor. We use its syntax highlighting and have extended its autocomplete.

Native Client

Our users want to query their databases without any new IT or admin setup, so users connect via their local machine with their existing credentials. Wagon handles connections through the native client, since most databases require socket-level network access that isn’t possible directly from browser.

Exploration and analysis can be a time-consuming and tedious job with existing SQL tools. Each step is an additional database query subject to queuing, slow execution, and changing data. Our users want to quickly explore results and evaluate summary statistics without executing subsequent queries. To power that experience, the client also computes statistics and caches query results on the local file system.

We built the native client in Haskell. It’s a high level language with great libraries, static compilation, a high performance ceiling, and an active community. We use MacGap to bundle the client and the frontend into a downloadable OS X application (Windows soon!).

Technology: Haskell, MacGap

• Prototyping and refactoring is safe with Haskell’s type-driven development. The compiler helps us avoid bugs and edge cases as we iterate.
• Haskell’s opt-in control over compiler optimizations and strictness allows for incremental optimization. GHC includes a capable profiler, and the community maintains libraries for testing and performance measurement.
• MacGap is a lightweight (~1MB) web-to-native wrapper. It provides a JS API to integrate with the file system, subprocesses, and OS-specific features like notifications and menus.

*Note: Wagon is now powered by Electron rather than MacGap. Read our Electron blog post to learn more.

Backend Services

Our backend services persists state and shared data to allow users to securely collaborate on queries, results, and charts. The backend is also written in Haskell: using the same language for the client and the server helps us stay flexible and reduce complexity. We store application state in Postgres while larger immutable data, like shared query results, live in Amazon S3.

Technology: Haskell, Postgres, Amazon S3

• Haskell’s lightweight threads and the GHC runtime scale well on multi-core machines.
• Postgres is powerful, flexible, and straightforward to host. We use features like TOAST and JSON support to test new features.

There are exciting upcoming technology challenges including optimizing compute across local and remote environments, designing efficient streaming statistics algorithms, building features that learn and improve with usage.

We’d love to hear from you! Follow @WagonHQ, check out our stack on Stackshare, and learn more about our engineering and design team. Stay tuned for future engineering deep dives!

Thanks to Steve Pike and Doug Petkanics for reviewing this post. Image credit Adam Wilson from The Noun Project.

Design at Wagon
January 22, 2015 | Jeff Weinstein

We have a challenging design goal: build an easy-to-use product for analysts and their teams to collaborate on understanding data. Our product is (1) a database query editor for analysts and (2) a Google Docs-style experience for non-technical people to explore results together.

We’re looking for someone to be our first UX designer – come craft our user experience, our voice, and our pixels. You’ll grow a team of designers and engineers that love data analysis and collaboration software.

As a team of engineers and data scientists, we have more experience building systems than storyboarding. Over the past few months, we’ve developed a structured design workflow. It starts with stories, drawing, and getting feedback from customers. We then specify the application’s states and object models and finally code mockups. Here’s how it all works:

Mark and Mike sketch application states.

Story

Many friends and former colleagues have the analysis and collaboration problem that Wagon solves and they’ve been very helpful testing the product. We watch them work, document their pain points, and listen to their feature requests. We winnow our notes into specific user stories: “As an X person, I need to do Y, in order to get Z.” To avoid groupthink, we independently draft and prioritize these stories, then collectively compare and merge them.

Draw

We sketch on the whiteboard, use pen and paper, and drag and drop in Google Draw to visualize our ideas. We review screenshots and pictures in Github Issues and Slack and share paper prototypes using POP. One surprisingly successful mockup was built with ASCII in Sublime Text. As our friends at Wildcard detailed, there are plenty of other design tools that we’d be excited to learn.

Feedback

Early and frequent user testing is essential. Our beta testers’ eyes surface bugs and corner cases at a humbling rate. They contribute directly via our public Hipchat room and shared Google Docs. Fortunately, our users have been patient, thoughtful, and enjoy participating.

Specify

We define application states and user flows before writing any code. These detailed state diagrams are critical for building quickly and reducing the confusion and blockers that can arise during implementation.

Wireframe

During implementation, while APIs and data are developed and tested, we lay out the user experience with HTML/CSS. This helps us uncover usability problems and lets us tweak the design in conjunction with other tasks.

To stay focused as a team, our five core product values set expectations and help us measure progress.

Intuitive

Clean design and approachable voice

Aware

Communicate context and insight

Interoperable

Connect to the outside world

Responsive

Iterate at human speed, always in control

Collective

Shared knowledge improves individuals and unites teams

Design at Wagon has been a team effort. We’ve been lucky enough to be advised and supported by a set of design advisors. Curious about data and design at Wagon? Say hi.

Thank you to Hillary Kelly for reviewing this post.

Hello World
January 14, 2015 | Jeff Weinstein

Welcome to the Wagon blog. We’re excited to show you what we’ve been building, the technology behind it, and stories from the wonderful world of data.

At Wagon, our mission is to make data analysis easier for everyone.

Our first step on this path is a modern, easy to use, web-connected SQL editor. We’ve built a simple way to write queries, visualize statistics, pivot results, and securely share charts with your team. You can download Wagon and connect to a local, private, or public database without involving IT. Sign up for our private beta to get started.

We’re also hiring! If you love data and design, check out our jobs page. We’re looking for a user experience designer, a frontend engineer, a data visualization engineer, and a backend engineer.

We’d love to hear from you. Tweet us @wagonhq, email the team, or visit our San Francisco office.

Thank you to our early beta testers, investors, advisors, family, and friends. As we say at the office, gogogo!