Uncanny Data

An article in the New York Times this last Tuesday cited new research that suggests we are more alert to patterns when we have seen, or experienced, something odd or uncanny.  These disorienting experiences “may prime the brain to sense patterns it would otherwise miss,” writes Benedict Carey, the article’s author.

This reminded me of a phenomenon I have struggled to describe, but which I have been interested in for some time. There is a state of heightened awareness we are all familiar with.  It happens after you see something interesting or intriguing that you can’t quite figure out.  For example, as you walk by a building, you happen to look through a glass door and see a car idling in what looks like the lobby of the building.  Wait a minute, you think after taking another step or two, what’s a car doing in the lobby of that building?  You stop and step back for another look and realize that what you thought was a lobby is in fact the courtyard of the building.  These misapprehensions occur every day and we probably for the most part easily forget them.  As you were walking by the building, maybe you had been thinking about some emails you have to answer, people you need to call, things to get at the store on your way home, whatever.  As soon as that odd visual presented itself to you, however, nothing was as interesting as figuring out what was going on.  During the moment when you were trying to resolve that confusion, everything else flew out of your mind.

What was going on in that example?  The context– the way the building presented itself to the street, the style and size of the glass door, etc.– all suggested to you that what you would see when you looked through the door is a typical building lobby.  What you did see, however, violated that expectation.  Or perhaps more precisely, your brain thought it had enough information to make a decent assumption, but you suddenly learned that it did not have sufficient information to make the particular assumption that a lobby stood behind the glass door.  If reality had aligned with our expectation– if you had seen an ordinary lobby behind the door– you wouldn’t have given it a second thought and may not even have noticed that you had seen the door or lobby at all.  On the other hand, if the door hadn’t looked like such a lobby door, or if the building hadn’t looked so much like other buildings on that street– all of which have lobbies– your brain might have reckoned that it didn’t have enough information to make any assumption at all about what was behind the door.  You wouldn’t have had any expectation that the space was anything in particular.  So seeing a car there probably wouldn’t have surprised you.  It was the disconnect between your assumption of what you would see, and what you did see, that created that state of heightened awareness.

Why spend so much time on an example like this?  Three reasons:

1. A state of heightened awareness is valuable, so it’s helpful to understand what may produce it

2. I have an unconfirmed suspicion that this example is related to effective reporting and visualization of data

3. I like odd and uncanny stuff like this

For now, I’ll focus on #2.

I think the link between my example above and data visualization has to do with pattern recognition.  Consider these propositions:

Case 1: Too much data. This  is chaotic and overwhelming.  We are unable to discern a recognizable pattern, so we do not make any assumptions about what the data tells us:

Case 2: Too little data. This is boring.  We barely notice that there is useful information there at all, and don’t even bother to discern a pattern or make assumptions.

Somewhere in the middle– where the amount of data is about right– I suggest there are two possibilities:

Case 3: The amount of data is about right. The pattern is clear and easy to recognize. We instantly understand the situation:

Case 4: The amount of data is about right. There seems to be a pattern. Maybe the pattern is not what we expected to see.  Maybe it’s difficult to make out or understand the pattern.  Either way, we want to learn more to figure out what’s going on:

How does this relate to reporting and visualization?  Here are my thoughts:

Case 1 & Case 2 = Not useful

Case 3 = Useful reporting.  This gives you an answer to a question you have asked.

Case 4 = Useful data visualization/exploration.  This gives you insight, allowing you to ask more questions.

Do interest groups agree on which lawmakers they like?

I was interested to see if there was a visual way to get a feel for how members of congress arrayed across the political spectrum.

To do this, I thought it would be useful to mash up ratings of sitting US House and Senate members, by various interest groups. Looking for data, I found a site called votesmart.org, a tremendous resource– they keep track of voting records, have a searchable position papers database, and (most relevant to what I was looking for) ratings data from dozens of different advocacy and interest organizations. I loaded the data into Visual i|o and found patterns you would expect, but also some interesting insights and a few surprises. (If you’d like to see the application yourself, it’s here.  To see it, you’ll need a visual i|o login, which you can request at beta.visual-io.com by clicking the top “Try it” button.)

Looking at the histogram filters can give you a good idea of how polarized an organization’s ratings are.  Look at the distribution of ratings for the League of  Conservation Voters (an environmental advocacy organization), for example:

Distribution of lawmaker ratings by League of Conservation Voters, 2007-08

The histogram is shaped like a bowl– with lots of lawmakers rated at the top and bottom of the range, not too many in the middle.

Contrast the distribution of ratings from the US Chamber of Commerce:

Distribution of lawmaker ratings by the US Chamber of Commerce, 2008

I was interested to see how different organizations’ ratings would correlate.  Does the American Conservative Union align with the Christian Coalition in their ratings?  How inversely correlated are the ratings of, say, the NRA and the League of Conservation Voters?  Here are lawmakers (Senate + House members)  plotted on American Conservative Union vs. Christian Coalition ratings:

Lots of red dots (Republicans) in upper right– rated highly by both ACU and the CC.  Not surprising.  Mostly blue dots (Dems) in the bottom left– low on both organizations’ scales.  Also not too surprising.  But the alignment between the two organizations is not as strong as one might have thought.  Some dots in the bottom right represent a few lawmakers– mostly Democrats– that get high ratings from the Christian coalition (80+) but low ratings from the Conservative Union.  A few Republicans, like Rep. Michael Castle of Delaware, look more like Democrats– a red dot in a sea of blue:

One thing that gets hidden in a view like this is how many dots are stacking up directly on top of each other.  You can see this to some degree in the darkness of the dots, since they are somewhat transparent.  It’s hard to know just from looking at the chart, though, that 27 people have perfect 100 ratings from both the ACU and the Christian Coalition.  Conversely, 39 people get zeros from both organizations.  This blind spot can be avoided in a view that sorts items into columns and rows so that no items overlap:

Lawmakers are plotted by American Conservative Union ratings on the vertical axis, and grouped into columns by office– House or Senate.  Judging by this, the House appears more divided than the Senate.  Why do I think that?  A lot more dots at both the top of the chart (high ACU ratings) and the bottom (low ACU ratings). In the Senate there are fewer 0’s and 100’s, relatively-speaking.   Again, a few red dots seem out of place:

Olympia Snowe and her fellow Senator from Maine, Susan Collins, both seem more like Democrats than Republicans in the eyes of the Conservative Union.

The NRA, unlike most organizations, gives letter grades rather than numbers.  I mentioned before that I wondered how differently the NRA and the League of Conservation Voters viewed lawmakers.  I plotted League of Conservation Voters on the vertical axis:

…then changed to color by each lawmaker’s NRA rating (dark green highest, dark red lowest):

Among all the red dots–congresspeople getting F’s from the NRA– atop the Conservation Voters chart are two green dots with perfect 100 scores from the LCV and bright green A/A-minuses from the NRA:

If you don’t think it’s possible to please both the environmentalists and the NRA, just ask Albert Chandler and Paul Hodes.  They figured out how.

Better charting and visuals can create new problems for organizations

There are now loads of ways to create charts and graphs and publish them for consumption by others. But a big problem though is the skill set that is required to use these tools and technologies. You need a very specialized combination of business/domain knowledge, data and technical skills and graphical design abilities. If you don’t get all these bases covered, you don’t just get bad reports, you can produce dangerously misleading information [read about bad visualization by Stephen Few http://www.perceptualedge.com/blog/?p=266 and Howard Spielman http://www.information-management.com/bissues/20061201/2600277-1.html]. As the CIO of a large consumer products company told me, “we don’t have those skill sets and I don’t want to have to build teams with those skills.”

This is a big gap and an important factor that has been overlooked in vendors’ eagerness to get eye candy into their products. Who is going to use these tools and how do they dovetail into the workflow– current and future?  So, some of the issues/gaps that need to be addressed in getting better information to decision makers:

1.       Skills sets:  As mentioned, those needed to create accurate and meaningful charts and dashboards are hard to hire for, so tools need to get “smarter”. That way, users don’t have to get skilled up on esoteric competencies and management isn’t confused. 

2.       Workflow: Exploration needs to come before the output of explanation—this is the problem with tools that just create diagrams and slides and status views on a portal page: it creates an inefficient process when the analysis has to be done using a completely different set of tools (and often by a different set of people) and then ported into reports. Analysis comes first, then publishing of the analysis. It just makes sense then that analysis and reporting need come from a single tool to efficiently fit into the day-to-day workflow.

3.       Analyst bottleneck: In the future organization, doesn’t everyone want more speed and efficiency? That won’t happen if we’re still bottlenecked getting every question answered through an analyst or report writer and then forcing those answers up through MS Office documents.  As that same CIO also said, “I’ve got 500 people here and 350 of them are writing SQL queries responding to ad-hoc report requests. It’s slow and inefficient, and we can’t continue to scale like this.” The business side needs to be able to iteratively ask and get answers and satisfy themselves.

What does easy-to-use really mean?

Everyone agrees that BI (business intelligence) needs to be “easy”. But what really defines easy? And easy for whom (technical implementation and support teams, analysts and report writers or business management and decision makers?)

With all due respect, the bar is set pretty low to make things easier for technical teams—traditional BI tools are so complex to implement and maintain that there are whole industries of consultants who specialize in tool-specific services. And the same can be said of the analyst and reporting staff, whose roles and job security, as I’ve said before, are defined by the gap between data and decision making. The real challenge is making BI tools easy enough but still meaningful enough for business people to use.

What decision makers require is:

·         Access to live data in a form that exposes pertinent information, patterns and anomalies at-a-glance

·         Capability to then ask follow-on questions based on what is identified (i.e. is that still an outlier if we look at historical performance or just in this category over this last period?)

·         Capability to document and share findings and insights

·         Reduced reliance on analyst and IT grunt work

Conversational analytics—Dialoguing with your data

There’s a lot of talk about “Easy BI” and “business intelligence for everyone”.  And for obvious reasons: the current state of the art is hard to use and has required companies to scale up on report writers and other personnel with technical-to-business “translation” skills. This swath of infrastructure is responsible for getting still-pretty-raw data from enterprise systems into the hands (and more preferably the brains) of managers and decision makers. But the standard tools of the trade—queries, spreadsheets, slides, printed reports and dashboards—are bottlenecking the process as more data and more complexity needs to be communicated. [link to “Current State” slide]

Someone said to me recently “We’ll prepare 300 slides for management—they only end up looking at 10, and they ask for things we didn’t do. We can never anticipate how the conversation will go so we are always going back to the drawing board.”

That’s a lot of inefficiency and a lot of cycles at a time when all businesses are trying to do more with less. This loop between management questions and analyst responses has created a culture of ‘analysis interruptus’. Another senior business analyst told me “I have seen over the years that senior managers have trained themselves NOT to ask too many questions because they’ve seen how off-hand remarks can send analyst teams down a rat hole and burn weeks of time.”

There are two lines of attack—one is to enable management and business users with easier to use tools for analysis, reducing report requests and decision cycle times.  The other is to better equip analysts and report writers to have more answers at their finger tips to reduce the manual efforts and lead times of one-off reporting projects—less need to go back to that drawing board [link to Future State slide]. Both of these strategies rely on visualization (to convey complexity and spot patterns quickly) and interactivity (to ask and answer questions and iterate analysis on-the-fly). When these two capabilities are brought together in a fluid and intuitive way, analytics can transcend the technology of cubes and queries and algorithms and become a dialogue or conversation with data. This is what I think of as conversational analytics.

[Click here to see an example of Conversational Analytics]

Designing for decision making—not the same as workflow

One of the challenges of the visualization business is that ‘visualization’ means different things to different people. And naturally, people go with what they know. One of the related areas –and one more in the comfort zone of a lot of buyers—is user interface (UI) design. Good web designers should be able to create meaningful visualizations, right?

Well, not exactly. Visualization actually requires a whole different set of skills than UI and web design. Fundamentally, visualization is about decision making—understanding the information and its context better so that you can ask better questions, get better answers, and make better choices. UI design is workflow—like a data entry form, a website shopping cart, or trying to figure out how to reset the bullet formatting in Powerpoint.

Both have an important element of navigation, which may be why the two seem interchangeable. But one is navigation based upon the way the data is interpreted via the visualization (you can’t say where you are going but you’ll know it when you get there). The other is navigation to accomplish a task—fairly straight-forward to map with predictable parameters.

And visualization is data-driven. This, it turns out, also demands a different approach and skills than UI design. That’s because UI designers are used to being able to control the way something looks, but visualization renders based on the data driving it. It’s hard to get away from the legacy of the graphic design thinking that has dominated screen-based design—but a fundamentally print-based approach demands control over almost every pixel and the way any page or view will render. Visualization is not about designing information graphically, it’s really about designing graphical systems that will render unknown and unpredictable data into consistent, intuitive, visual metaphors.