Warp: a query-by-example analysis tool for big data

The spreadsheet as we know it is in dire need of a big data-ready replacement. The spiritual successor of the spreadsheet needs to be a tool for analyzing big data that combines powerful features with a mental model as simple as the traditional spreadsheet.

So what are the options? A traditional spreadsheet like Excel is flexible and intuitive, but not so powerful in terms of big data analytics. There’s the traditional relational database, which can be quite powerful and flexible, but is nowhere near as easy to use (except, maybe, Access). There are various BI tools such as Qlikview and Tableau, usually lacking in flexibility — and of course there are big data frameworks such as Hadoop which, well, really only programmers can use. Flexible, powerful, intuitive: pick two.

Warp is designed around the premise that non-programmers have difficulty expressing their analysis ideas in code, or any of the typical database ‘concepts’ such as tables, rows and expressions.

With Warp, you teach the computer how to analyze your data the way you would teach another human: just show how it’s done.

Warp allows you to describe the desired operations by, well, performing them by hand. Like you would in Excel, but easier. Say you want to extract the year from a date in some column. No need to type a formula: just type the year for one of the rows in the desired column, and Warp will figure out a formula to extract it for the other rows.

Because big data sets are typically too large to be handled by a single user, Warp lets users work on a small (randomly selected) subset of the data. When the analysis is designed, Warp can run it (efficiently) on the full data set. Warp supports MySQL and SQLite directly (for ‘medium data’) and can interface with Hive/Hadoop clusters, PostgreSQL and others through Facebook Presto.

Warp design principles

Big data analysis by-example

Warp watches as you work with your data and translates it to formulas. If you have a cell that contains an address and you want to extract the zip code from it, simply make the change yourself. Warp will suggest to extract the zip code automatically from the other cells based on what you did. Of course, you can always edit the underlying formulas by hand.

 

Run small, think big

In Warp, you work with a small selection of the data. If your analysis is done, you run it on the full data set with the click of a button. Warp ensures that each step is performed where it can be executed most efficiently. If you are selecting items from a database, Warp is smart enough to let the selection happen in the database instead of pulling all records from the database and selecting later. Warp can even support even larger data sets by delegating work to Hadoop-like systems. But at design-time, you work with a much smaller set that is much easier to mentally grasp.

Immediately familiar to Excel users

Warp users coming from Excel will immediately feel at home. We have included support for the most popular Excel formulas. We also provide pivot table functionality. With Warp, you can effectively query databases using Excel formulas and tools you already know how to use.

Where to get?

Warp is distributed from the Mac App Store, which entitles you to free updates and, importantly, supports development. As features are added, the price will increase — being an early adopter is rewarded.  A beta version is available right here .

Get the app at warp.one