When I saw ClearStory’s approach to simplifying disparate data analysis, I knew right away that they were onto something big and I just had to be a part of it. I don’t just love the vision; I love what I see today.

It’s dead on with where things are going – working with data just has to be easier, and these guys are absolutely nailing it. They are fanatical about making it possible for more people across organizations to work with data.

The Engineering team here is amazing. Each member of the team draws from a unique set of past experiences that are perfect for this challenge. They are super passionate about solving this problem and building a product the world loves to use, and they have been building some very cool technology to do it. The end result is a beautiful and intuitive application that blew me away. Crunching big data with speed and scale is one thing, but doing it so a business user can actually use it is the future.

I’ve built teams from the ground up in startups and managed large development teams at Salesforce.com and I know this team has what it takes to be successful. Great teams make great products and I’m here to continue to help make that happen. Add this engineering team to the experience of the founders, our investors, and the interest of the market, and you get something huge. I’m thrilled to be part of this; from the product, the technology, the team, the culture and the trajectory.

If this excites you like it excites me, and you want to put a huge dent in the new data universe, we want to hear from you. Join the team.

In this continuation of Sunday’s blog post, I’d like to share two more critical principles that are helping us achieve the goal of great product design at ClearStory Data.

3. Everyone is a Designer

There is a common misunderstanding that a designer’s job is to mysteriously come up with the perfect design without any collaboration with other people across the organization. The fact is, most of the best designed experiences come from different groups of people working together. Sometimes, the job of a designer is to get people from across the organization together and encourage them to work together and think about possible solutions to a problem. Including diverse opinions and different perspectives in the design process will likely yield better results than if one or two people who work in isolation try to figure it out themselves. This is especially true for big data products. Technology in this area is moving at an incredible pace and to truly utilize these advances and create innovative products, teams of people from many different aspects of organizations need to work together to craft the best possible user experience that fully leverages this new technology.

I saw a good example of this at Google where I was the design lead for Google Analytics. Our vision was to create an innovative, yet easy to use advanced segmentation solution. Since the technical components were new, it was critical that my design team and the engineers worked together to come up with the best solution. As designers, we needed to know what the technical limits were. For the engineers on the team, it was important that they understood how their technical contributions affected the usefulness of the end product. The result of this close collaboration was a much easier and more elegant user experience for advanced segmentation that leveraged an innovative technical foundation.

At ClearStory, design innovation means involving many different people during the process of feature development. As a team, we work across engineering, product management and design functions collaboratively, to define what we are trying to solve. Our assumptions are testing against direct feedback and input from early access customers. Then, with the problem well-defined, we iteratively review different options and debate different approaches to delivering on the product vision. Features are refined, designed and coded, often in iterative cycles until the product is fully developed. The role of a designer at ClearStory Data is to lead the team through this process, represent the voice of the customer experience and orchestrate the best ideas discussed into a complete and expertly designed product.

4. Make Beauty a Priority

There are markets where beauty is treated as an afterthought. Often, the more technical a product, the easier it is to overlook the need for beauty. However, companies that embrace beauty, particularly in technical markets, often build a connection with customers that is visceral. Customers crave their products. Dieter Rams, one of the world’s most iconic designers, provided us with a great example of this at Braun. One of Rams’ lasting design principles is: “Good Design is Aesthetic.” Just look at his portfolio of products and you’ll see what he described in his own words as, “the aesthetic quality of a product is integral to its usefulness because products we use every day affect our person and our well-being.”

If mass adoption of big data applications is important, then these products have to have a certain level of aesthetics or beauty blended into their function. So what are the elements of beauty? In our space, it could be using a very elegant color palette, or using typography in a meaningful way. It could be an application that actually embraces open space rather than simply shoving in more buttons and controls. It could even be a highly intuitive interaction, or a particular way a problem gets solved. It certainly will be a combination of some of these elements and many more.

There are many opportunities to make technologically advanced products beautiful but it has to be a priority for organizations. Beauty becomes a priority when organizations put a premium on ensuring that what people experience in a product is something they will not only get value out of but also enjoy.

At ClearStory Data, our approach to big data is to make solving complex big data problems accessible, while ensuring our product’s overall performance and function are never sacrificed. Beauty is a huge part of that. The more addictive the experience, the more organizations will naturally embrace a data-driven culture. Embracing data will yield a constant flow of intriguing insights that businesses are craving.

Utilizing these four principles, we are excited to share our thoughts on how ClearStory Data’s solution helps organizations evolve to a data-driven culture. Join us at Strata today to discuss how design and technology embrace one another to deliver the big data products of the new data landscape.

At ClearStory, we are enabling organizations to move to a data-driven business culture by delivering products that are useful, easy to use, and accelerate insight. We made a conscious decision from the start to build design innovation into our development process. Over the next few days leading up to my session at the Strata conference, I’ll share a few of the principles and best practices that are driving our design-driven product development process.

Today’s principles speak to the mindset that we’ve adopted to balance our focus on design with the practical reality of delivering great products in a fast-moving market.

1. Embrace Empathy

Empathy is defined as “the ability to understand and share the feelings of another.” Developing deep empathy for people who use your product is one of the most important traits a design, product and engineering team can have. It enables teams to make choices about the user experience not based on what is technically possible, but on what is most useful for people. Empathy is a mindset that is carried through everything a team does.

For big data products, an empathetic mindset is often challenging because it is all too tempting to focus on technology platforms and forget about providing a useful experience. Overcoming this temptation and developing true empathy has the power to transform products from a random collection of knobs and buttons into a delightful experience that enables real people to solve important real-world problems.

At ClearStory, we fully embrace an empathetic mindset. We do this through constant customer interaction and spending time “walking in the shoes” of our users. We’re not simply talking to customers to gather usability feedback and prioritize features. We are working with them to understand how they want to reach a specific data-driven outcome through the use of our product. We iterate not just until the feature technically works, but until we feel that the experience is addictive and incredibly useful. With this mindset, we are able to invent innovative experiences.

This nuance of empathy is what makes design-driven products so great. Take a product like Nest’s Learning Thermostat that is transforming the way we interact with our cooling and heating systems. Behind everything complex, they believe a simple, beautiful experience matters. A home thermostat should not just functionally control the temperature of your home, but it should be a delightful experience. You should even be able to install one yourself without needing to be a certified electrician. Similarly, here at ClearStory Data, we believe that you don’t need to be a data scientist to work with diverse, complex data.

2. Always Improving, Never Perfect

One of the most paralyzing attitudes designers and teams can have is to strive for the “perfect” solution. There is no such thing. The danger of striving for perfection is that during that process, no one agrees on what is perfect. The goal should be simply to create something better than what currently exists. This mindset liberates designers and teams from the shackles of perfection and allows them to focus on what is truly useful to customers, not what is perfect.

At ClearStory, this mindset allows us to engage with customers in a healthy feedback loop that constantly delivers customer value while at the same time providing hands-on feedback for quick iteration. As an example, take our process of developing our intuitive user controls. As a designer, I often prototype with the team in real-time, testing page layouts, interactions and labels to produce a “good enough” prototype to share with a customer or prospect. We then take that prototype out in direct customer interactions.

Over the last few months, we’ve iterated through a number of revisions to our intuitive user controls. Each design cycle involved feedback from a number of different customer teams. Our cross-functional team of product managers, engineers, and designers focused on actively listening to the customer experience and incorporated their learnings into the next product revision. The goal was to make the customer experience as intuitive and easy as possible. It is our view that design innovation comes from smart iterations rather than dreaming up that perfect design from the start.

Hopefully, I’ve given you something to think about when it comes to the role of design in big data product development. On a daily basis, we’re working to incorporate great design into the culture of our company. We’d love to hear from those of you who share this passion. Keep watching this space over the next few days for my next post and join me at Strata for a continued dialog on big data and design.

Upcoming Principle: “Everyone is a Designer”…

But MapReduce’s batch-oriented processing can only go so far. Problems with high latency of execution, real-time and streaming data management, an API that is too low-level and involved for mass adoption, and other issues have resulted in a dramatic evolution of the platform itself. This evolution has forced the addition of support for higher level languages (Pig & Hive), new real-time storage engines (HBase), extensions for streaming data (Hadoop Streaming), and the most recent addition, Impala, from Cloudera.

(As a digression, it is interesting how many of these transformations and alternatives mirror Google’s own evolution beyond the MapReduce paradigm, with systems like Dremel, Tenzing, PowerDrill and F1.)

Today Hadoop can be evaluated in a new light. Hadoop is moving beyond MapReduce’s initial success as a batch-oriented, better way to support ETL. In an enterprise architecture rife with a sea of Hadoop-to-database connectors, technologists realize that a new approach is necessary. New Apache™ projects are focused on evolving Hadoop beyond MapReduce and into a formidable Big Data stack.

There’s also a flurry of alternatives introduced to solve one or more weaknesses in Hadoop MapReduce’s initial footprint. These include projects like the Dremel-inspired Drill project (or Google BigQuery), Spark & Shark from the Berkeley AMPLab BDAS stack, and Storm from Twitter. I’ll get deeper into my experience with these, and others, in upcoming blog posts.

There are many other promising projects outside of the ones mentioned above, each with their strengths, which is why Hadoop MapReduce is only the beginning of Big Data analytics. But the other alternatives don’t quite deliver when it comes to delivering a big data solution for the way business users want to work. Business users want a solution to derive actionable, trustworthy insights from diverse data sources, rapidly, and in a repeatable fashion.

At the core of such a solution is a data processing engine which has a flexible data model that can work with diverse data sources at scale; an expressive execution engine that subsumes both batch processing and real-time, interactive processing; and one that supports advanced operators without overwhelming users with too much complexity. At ClearStory Data, we are building such a solution, and are excited to start sharing the details. Watch this space as we continue to reveal more about what we do and how we do it.

that’s why I’m excited about kleiner perkins’ investment in clearstory data. data is everywhere, impacting everyone. however, the power to ask questions and get answers from data remains in the hands of data scientists. 2013 will change that.

as i wrote on uncapitalized after we announced our investment in clearstory, “i’ve seen over the years, from my time at composite software to palm to twitter, and even at kp, that companies desperately want to leverage their data yesterday, but there aren’t many people with the requisite data science and computer science skills to help them.”

i look forward to working with clearstory to disrupt this status quo.

Read more posts on the uncapitalized blog by Mike Abbott, General Partner at Kleiner Perkins Caufield & Byers and former Vice President of Engineering at Twitter.

We get data every day, quickly and easily. We take these data-intensive actions for granted.

Why, then, is everything so much more difficult when businesses try to use data? At a time when data is increasingly recognized as the business currency of the 21st Century, companies are stuck with 20th Century tools when they try to work with it.

Data is exploding. Data is dispersed.
Vital business information now resides not just on your desktop computer and in your company’s databases or data warehouse. It’s also in new “big data” platforms like Hadoop. It’s outside the walls of your company, too. Specialized data vendors maintain petabytes of information about every conceivable product, place, trend, spend, market segment, behavior and customer. And the popularity of clickstreams, reviews, tweets, ‘likes’ and other social data presents a trove of new business insights for companies smart enough to be able to separate their signal from their noise.

However there’s no single easy way of accessing this information. Instead, managers and analysts are presented with complex interfaces to data platforms, a Babel of third-party APIs, high-priced analytic frameworks and silos of point products that each require a small army of tech-savvy data specialists, all of them deeply entrenched in their individual worlds. Simply retrieving a sliver of data from a single database or web site can be a chore; harder still is combining data from multiple internal and external sources. Even with the data in hand, you’re still faced with the challenge of force-fitting it into one of the cumbersome analytics or “visualization” products now on the market. Throw data at massive scale into the picture, and most of these products abruptly grind to a halt.

Data-driven companies have many distributed data teams.
Today’s most successful businesses are agile enterprises that shun staid hierarchies and rigid bureaucracies in favor of constantly-evolving teams that come together, solve a problem, and then disperse to find new challenges. But these ad hoc teams are badly-served by the traditional corporate data retrieval model, in which tools and expertise are tightly controlled by an IT department. These distributed teams need data solutions that are as flexible as they are.

Big, dispersed data mandates a new way of looking at things.
Today’s most popular data viewing approach — the dashboard — is in fact one of the least useful. It’s a rear-view mirror look at the business, based on events of the past. Because they are built on rigid data models, dashboards are not a reflection of how the business is doing right now. Business users have relied on them only because there have been few alternatives. As data-driven businesses evolve, traditional dashboards will be relevant to fewer and fewer tasks.

At ClearStory, we think it’s time to modernize data analysis – all the way from the user model to the data pipeline. For starters, this means recognizing that not all data is born and lives inside your company. There are great stocks of it emerging from the web and from specialized data providers. When intelligently combined with your own data, this information can lead to valuable new insights about your business.

We at ClearStory have rethought everything about how companies should do data. How they find it, both inside and outside their organizations. How they collect, combine, refine, picture and analyze it. How they swap their data-driven insights with others. And how they can quickly repeat the process, using speedy iterations to reach even fresher insights in a business environment that never stops changing.

ClearStory is accomplishing all this by re-imagining not only the end user experience of data analysis, but also by reworking the behind-the-scenes connections to disparate data sources and how it’s all processed. We make finding, querying and interacting with data — even when it requires numerous operations across multiple internal and external data sets — as easy as getting directions from your mobile driving app.

You’ll be hearing a lot more about ClearStory’s new approach to diverse data analysis in the coming months. Our goal is to fundamentally change the way businesses work with data, turning the whole business of big data on its head. We are re-thinking everything about data collection and analysis from the point of view of a business user eager to get answers to time-sensitive business questions. No business user or analyst today has the time to train as a data source expert or an analytics genius. Nor can they afford to wait for an overtaxed IT department.

Today’s business users need rigorous and speedy data analysis to always be at their fingertips, much like the Internet itself. And just like your favorite website or smart phone app, a modern data tool should be not only useful, but also pleasant — even enjoyable — to spend time with. There is certainly nothing pleasant or enjoyable about doing battle with the volumes of diverse data that confront business users every single day. Changing that is ClearStory’s mission, and we welcome you to join us.