The study is comparing the mosquitos-killing efficacy of sledgehammers v. flyswatters. For a graph with six data points, nearly any approach would work (even a pie!). For the presented cases, graphs are altogether unnecessary; a table or sentence would suffice: “Total House and Senate campaign expenditures increased from ~$50 to ~$300 million from 1972 to 1982!”

A more suitable visualization for any of these tiny data sets would be a sparkline, i.e. a word-sized graph. Effective design does not involve constructing vacuous and sprawling graphs (even well-formatted ones that make good use of color) to present virtually no data. Certain design principles are valued because they facilitate pattern-recognition and information processing among large quantities of data. Using Holmes-like embellishments to represent multiple data series with hundreds or thousands of data points would fail. This would be evident if the study took the opposite approach, namely selected exemplary data-rich graphs and converted them to Holmes-style graphs.

The article demonstrates that visual embellishments are more effective at marketing, i.e. having folks take notice and remember your message. (However, the study is based on 20 subjects, so I would be reluctant to draw even that conclusion.) If marketing is your goal, it’s no surprise that minimalist charts are not the best medium. Most of my work involves trying to makes sense of large quantities of complex, multidimensional data. And for that, I prefer not to use flyswatters.

(Full-size pic)

Using actual charts results in file sizes that are far larger and sort times that are slower than the alternatives. Interestingly, the file size when using formula v sparklines is essentially the same, although sparklines take roughly twice the time to sort. Clearly at some point the chart/ layering method becomes unwieldy, but it surprising how many charts can be included in a workbook while keeping the size and performance reasonable. Consider test 2: 1,600 graphs and 200 rows, yet the file size is only 2 MB and sorting takes only ~1/4  a second. For many purposes this is reasonable. The same could be said for 400 rows and 3,200 graphs; which results in a file 4 MB in size with a ~1/2 second sort time.

In sum, if the graph you need is available as a sparkline or formula, choosing one of those approaches will result in a smaller and quicker workbook. However, there are plenty of case where it is simpler or perhaps no alternative exists to using actual charts. If that is the cases, it is still possible to pack hundreds or thousands of charts in a single workbook.

The following is example of a tool that facilitates both comparison of individual components as well as cumulative values. The tool also illustrates an effective approach to analyzing data with multiple categorical variables, which is also common among surveys. By representing the categorical variables within columns rather than across a row, we maintain the ability to filter and sort the category’s values. The tool also uses layering as a means of creating reference lines. In sum, with this approach a tremendous amount of comparisons are available within a few seconds. In previous posts I have presented a similar tool, VBA to assist in small multiple creation, and layering. The actual graphs in this example are a bit odd for two reasons: (1) I neglected to fix my axis, and (2) the random data I generated does not add up to 1.

(full-sized pic)

Note: workbook provided upon request.

In contrast a tool-oriented state of mind would take a step back beyond the singular problem and seek to create a basis for addressing a broader set of questions of or area of analysis. More often than not, a tool-oriented approach yields better medium of long term results. Moreover, a well constructed tool can serve the role of a report; whereas the opposite is not possible.

In November Chandoo presenting a visualization challenge with the following objective: “You have to make one chart (or dashboards) using this data that would help a senior manager understand how the sales people have done in the 24 months.” Approaching this from a very report-oriented approach I presented the following:

(Full size pic. Also, complete workbook at: http://cid-4e546fec4f1c6d0d.skydrive.live.com/self.aspx/.Public/dashboard%20submission.xlsx)

Given the question, I think it presents an extremely pertinent view of the data. Namely, it is a trellis plot visualizing how each salesman perform acorss different variables (e.g. region, product, and customer type). If the goal is simply to address the original question of understanding salesman performance, the report hits the nail on the head. However, in reality, when you present this to your manager immediately he or she will ask a half dozen more questions: how do the regions compare? which product made the greatest percentage increase? how do product sales differ by region? The salesperson report does not address these question; after all we were originally interested in sales performance. So at this point, we are largely starting from scratch in addressing these questions.

Posed with the same request now, I would look to create something more tool-like such as the following:

(Full size pic)

Here, by leveraging Excel’s powerful filtering and sorting capabilities in conjunction with small-multiples or custom-made sparklines we are in a position to answer any of the questions thrown at us in short order, perhaps even within seconds. Moreover, if a report is needed to a address a particular question it can be created on the fly as well. In this case, it is requires nothing more than creating a pdf of the relevant filtered and sorted rows.

In subsequent posts, I intend to elaborate on the distinction between report and tool and go into the details of constructing this type of tool, which takes remarkably little time to construct given its capabilities as well as present other examples that illustrate the power of this sweet filter / sorting / in-cell diagrams combo.

However, I think the case can be made that the square pie charts, by representing the status of 100 individual units, more forcefully convey an image of individual students dropping out. And in contrast to a round pie chart, the exact values can be determined without too much trouble. I am definitely not making the case that square pies are generally great visualization tools, but in rare cases and contexts they may have their place. For folks that are not very statistically literate perhaps, perhaps they may even be a more intuitive and clearer way of conveying the data. (I am not stating that this is the case, but merely raising the possibility. I have not read encountered any studies of which chart types are most easily understood with audiences of different level of statistical of graph literacy, although I would be very interested in such studies . Please drop me a line if you know of any.)

In an upcoming post, I intend to explore the concept of deliberely representing individual data points over aggregate statistics to underscore messages.

If you have any interest in creating square pies, especially many of them, e.g. for report production, the following macro may be of value. It pull parameters from a worksheet such as the one presented. The code is not very robust, i.e. I think it only works for pies of limited dimensions and has no error handling, but it worked for my purposes.

What is required from the end-user is that they create the lines they want on the template, give them a name, and add the relevant count to the data tab. I tested this versions by creating a piece of an energy flow diagram presented at Excel Hero. I didn’t seek to replicate this with any accuracy, but just used the concept as of something different than the original grades changes example. In my first attempt, I created the curved lines by grouping a few straight lines. This works in the sense that the grouped line can be referenced and the width adjusted as a single object. It doesn’t work in the sense that the lines are not actually seamlessly connected. I actually like how this looks.

So I gave it another try using a freeform line and that worked fine.

But that did not entirely represent a true Sankey as the lines are essentially separate and are not breaking off a single line.  To address this, I broke up the main line into a few segments and decreased the width of each segment as other lines forked off it. These calculations are simple to add to the data worksheet. Still this is a bit choppy. The lines connect in odd places. In this case, a few minutes of manual movement of the lines cleaned things up nicely. It would be nice if the lines connected with each other correctly, but I don’t see myself creating a whole lot of these diagrams on the fly. Any time these are created there will need to be some manual decisions regarding where to place the nodes and how they should be connected. An algorithm could be developed (although maybe not by me), but I think for many applications there will be a need for manual grouping depending upon the desired message and to improve clarity. At this point I have automated basically everything I would want to automate in regards to Sankey creation, which may not be saying much, given that I have rather limited use of Sankey diagrams.

Template

Data

Green requires data entry, the rest is calculated.

Diagram pre-processing

Diagram couple minutes of manual modification

Coincidently, I ran across a diagram that looks remarkably similar to my circular Sankey two days after posting it. The purpose of Martin Wattenber’s diagram is entirely different, not to mention far more sophisticated and apparently was created 9 years. If you are looking for lame Excel visualizations decades after they could be considered novel, this is the place.

Note: As usual, if you want the workbook, request it.

Caveat: There is a decent chance that the actual figures I present are erroneously labeled or calculated since I did not bother to review. If the data is of interest, I would work from the raw data. My aim is to present an approach to data visualization, not to present an actual analysis.

Note: If anyone is interested in the workbook used to create these diagrams, let me know and I will provide it.

Value of Sankey v crosstab or pivot table

Crosstab

Sankey, overall

Sankey, environment

Sankey, performance

Sankey, progress

I think the Sankey offers improved pattern recognition over a crosstabulation, even when formatted (although a more nuanced heatmap may have made the table more revealing). For simple and small tables there may be relatively little to gain from visualizing data; but as is the case with visualization in general, the greater the quantity and complexity of the data, the more advantageous it is to visualize. The Sankey I have created has increases in grade represented on the top (green), static grades in the middle (brown), and decreases on the bottom (blue). The height of the path or arc represents the magnitude of the increase or decrease (i.e. big or small jump), and the width of the path represents the portion of schools it represents. In this case, you can learn a lot with a quick glance. For example. there is relatively less movement among overall grades than category grades, environment had relatively little increases compared to other categories, and performance has more varied movement than performance. When I look at the tables, they all look more or less the same.

Typically visualizations improve pattern recognition at the loss of look-up functionality, i.e. exact figures are not available. In this case, the exact number and percent of schools label each line, so it contains all the detail available in the table. The coloring may be more effective if red colored the decreases, but I was inspired by the riverbank metaphor used in this music history visualization (the metaphor does not really apply in my case, but I like how it looks). The line are made semi-transparent and use relatively light colors to improve clarity in cases of overlapping lines. One limitation of this approach is that unless you have a good printer, the diagram looks like junk.

This circular Sankey is generalizeable to any crosstab of categorical data that is ordinal and symmetrical. Sankeys commonly depict  asymmetrical or geographic data, but in my opinion Sankeys offer a good alternative to more run-of-the mill crosstabs as well.

I have seen other Sankey or network visualizations created in Processing, and in most cases my Excel creation looks crude in comparison. Here is an example that came to mind, but upon close review it is not a Sankey since line widths do not convey information. Nonetheless, it is an interesting and beautiful visualization.

Creation
The workbook consists of three worksheets:

*Worksheet with the original actual crosstab table. This provides an easy location to populate new raw data.

Worksheet, raw data

*An intermediate data worksheet. This references and restructures the crosstab into one value per row, and each value is named; this provides and easier way of referencing the values in VBA. The worksheet also performs calculations, e.g. each values percent of total and width, subtotals.

Worksheet, intermediate data

*A worksheet containing a template of the diagram, where all of the lines are uniform in width. I created the initial lines and labels manually, and named each line and width with the connection is represents, e.g. a_b.

Worksheet, template

A macro uses the values from the intermediate data worksheet to set the widths of all the lines. It enters data from the intermediate data worksheet into the labels. So the crux of the program is:

One annoyance when creating this was figuring out which shapes are actually accessible to VBA. I think something like smart shape objects cannot be referenced, but lines can. So for example, I originally attempted to have the brown lines be actual box shape objects, but this did not work, so I had to make them lines. And the labels on the brown lines did not seem to be positionable where I wanted them, so I had to use text boxes. Dealing with small stuff like this accounted for about half the time it took to put this together.

When all is said and done, the end-user just needs to populate the crosstab tables and provide a single overall width multiplier for all lines (this takes a few attempts before a width that is aesthetically pleasing and results a clear digram is arrived at) and hit run to generate a new Sankey.

“small high-resolution graphics usually embedded in a full context of words, numbers, images. Sparklines as datawords: data-intense, design-simple, word-sized graphics.”

The most common application of sparklines within business intelligence departs from this idea a bit, as the images are rarely embedded within text. Their use in the context of dashboards and reports could better be conceptualized as an extreme form of the more general concept of small multiples.

If one adopts this perspective, the introduction of sparklines in Excel 2010 without improved support of small multiples in general is a disappointing and rather narrow improvement. For example, sparklines have the following formatting options:

• highlight the highest and/ or lowest data point
• highlight the first and last points in the chart
• color negative points using a different color.

As far as I know, these options do not exist for normal charts, though they could clearly be done manually. Conversely, sparklines do not offer anywhere near the level of customization of normal charts.  For this reason, I intend to continue creating sparklines the same way I did in earlier versions of Excel, namely by shrinking down a graph to a few cells in size and deleting the axes. Using this method I would not be able to replicate the win loss sparkline easily, but I do not have much use for that anyway.

The following explains how to create small multiples and the advantages of this approach over the sparklines feature or a single large graph. In this example, I created dotplots for a series of 9 values. The first set of plots is created using a single graph, the second set of plots is created using a graph for each value, i.e. 9 graphs.

The advantage of the small multiples in this case is that they remain synchronized with the text in accompanying cells when the values are sorted or filtered. Filtering and sorting are important features that most business intelligence tools should support. The following illustrates problem with sorting when a single graph is used.

The small multiples are created by plotting a dummy value, unhidden in this version. More specifically, the rows are not ‘hidden’, as this would remove the data points from the graph. Instead, the column widths are set to a fraction of an inch.

To create the individual small multiples, the vertical axis is set so the data point is centered.

The axes are deleted. Then the plot area is stretched to match that of the chart area and both the plot area and the chart area are set to have no fill. The small multiple is then copied 8 times, each one set to refer to a different data point. The small multiples are snapped to the grid to assure proper alignment and size.

Sparklines do not offer dot plots, so they are of no use for this type of application. Moreover, two of the graph types supported by sparklines can be created with far greater customization using this small multiples method. For example, these line graphs highlight a specific value which is not the highest, lowest, first, or last data point. The highlighting is done with a cross as a symbol. And tickmarks indicate precise data points. None of this can be done with sparklines.

Clearly, creating custom sparklines from small multiples takes a bit longer, but the difference is a matter of minutes. When creating dashboards or reports that have a wide audience or will be re-used, I prefer to spend the few extra minutes to create the visualization that best represents the data. Another advantage to creating sparklines the old-fashioned is better backward compatibility; I do not expect widespread adoption of Excel 2010 among my users for a few more years.

In sum, in many cases I do not anticipate adopting the sparklines feature and will instead rely on a small multiples approach to achieving a similar, yet far more flexible visual display. However, for something like exploratory data analysis or prototyping, sparklines may be useful.

Layering is the placing of multiple charts and/or shapes over the underlying spreadsheet grid; the objects are made transparent, so multiple layers can be viewed at once. For charts the transaprency is achieved by setting ‘no fill’ for both the plot area and chart area formats. There are several reasons for layering:

• Integrating multiple charts
• Integrating charts with non-chart report elements
• Formatting and customizing charts
• Creating compound charts

I will cover each of these possible uses of layering. If you have any other uses or examples of layering please indicate in the comments. I assume I am only scratching the surface of this technique.

Integrating multiple charts

In a previous post Integrating report elements in Excel I demonstrate use of a common spreadsheet layer providing unifying zebra stripes and axes to integrate completely different chart types (e.g. stem-and-leaf-plot and scatter plots) or small multiples of the same chart type (e.g. dot plots). Alternatives to integrating multiple charts exist, but layering seems simpler, especially if different chart types are involved.

Integrating charts with non-chart report elements

Chandoo’s post provides a good example of this type of application.

Formatting and customizing charts

One example is creating a colored background. Zebra stripes is one example already noted. Another would be to highlight different time periods or different quadrants or sections of a graph. Jon Peltier provides some examples here and here of how to set background color for charts by using a stacked bar within the chart. However, layering seems to be a far simpler and flexible solution. Consider the following scatterplot. To color the four quadrants, the plot area and chart area backgrounds are set to ‘no fill’ and the plot area is snapped to a 10 x 10 grid of cells, which are colored accordingly. This is effectively the same effect that was achieved with the stacked bar chart method.

However, we can add many different types of chart customization by similarly adding text of formatting the underlying cells. Here sub-quadrants are created with different colors, borders, and labeling.

Another application is highlighting of groups such as quarters and months. Again, only the cells underneath the transparent graph are colored. The following chart was created using layering, which is relatively simple compared to a graph within a graph approach explained by Chandoo .

Chandoo’s example is dynamic. Likewise, the cell coloring can be made dynamic with some simple conditional logic within a macro. The benefit here is flexibility. If you wanted a more complicated pattern, it would only be a matter of specifying the colors of ranges of cells. The full code used for this example is as follows (The code is a bit redundant and could be made more concise, but it works for now):

Creating compound charts

One approach to creating combination charts is to select individual data series and change the chart type just for that data series. This method is explained by John Walkenbach, Jon Peltier, and Chandoo . Layering can also be used as an alternative to this approach. Here is a Pareto chart, which consists of a transparent line graph over a transparent bar graph. The plot sizes are snapped to the same cells so they overlap correctly.

Compared to the aforementioned method, the time to create the graph is roughly the same. However, I think the layering method offers advantages when dealing with more complex types of combinations. Here we have three layers of graphs: two line graphs and a bar chart. The two line charts use different axes.

Conclusion

In sum, layering is a method that is conceptually simple and intuitive, yet is both flexible and powerful in its application. In some cases, alternative methods exist to achieve the same effect as layering, but in nearly every situation layering offers greater flexibility. Moreover, in the absence of layering one would need to learn many different excel tricks, some of which are hard to remember, to perform the same tasks. In contrast, layering is one simple method amounting to little more than removing the chart and plot areas of graphs and using the underlying excel grid or other objects as background.

In a future post I will show how layering can be used to create create rugs and more informative axes.