Part 1: Background

“In the beginning was the deed” – Goethe (cited by Vygotsky and Luria 1994)

Chapter 1

Exploiting Interactivity

“The ability to draw and then move linkages opens up a new field of graphical manipulation that has never before been available” Ivan Sutherland (1963)

1.0 Interactivity: A paradigm shift

As computers become faster and more powerful, they are providing us with many new opportunities. One of these is the advent of visualisation applications which allow users to view large data sets in the form of abstract representations. Interactive tools can then be used on parts of that representation to identify the useful data and assess the alternatives available. Such Visualisation applications can be applied to many different scenarios. One of their values is that they can enable the user to summarise vast quantities of information very quickly. Another is the ability to examine trade offs and correlations.

Adding interactivity may seem a very simple modification but the insight that results can be very valuable. Instead of examining a static piece of information the user is now in dialogue with their data. Bill Buxton (1994) makes the point that the difference between having interactivity and not, is the difference between a two way dialogue and one way information provision. Adding such a two way interaction produces a whole paradigm shift in the ways a piece of technology can be used. He illustrates this by saying: “Interactive television is to broadcast television what the telephone is to radio (since the telephone is interactive audio, whilst the radio is broadcast audio). Any model of interactive television that does not acknowledge a change of paradigm of that magnitude is missing a large part of the potential of that technology“.

Another aspect of interactivity is that it encourages a user to ask many different questions of a set of data. When describing static representations of data Casner (1991) established that “Different presentations of the same information best support different tasks”. In other words, each question that a user wants to ask of a data-set requires an entirely different presentation. Whilst this limitation applies to static presentations, it is not relevant when interactivity allows different features of the data to be made salient as and when required. Once such interactivity is provided, the underlying representation becomes a medium through which different features of the data are made explicit and so a single representation cn be used to answer many different questions.

This shift in paradigm from static representations (graphs) to interactive ones (visualisations) has not yet been fully conceived or utilised in everyday applications. For example, the standard displays in the Stock Exchange still use tables of data with no interactivity. In order to get a graph of the trends present in the data many traders will cut and paste data to a spreadsheet. However, none of the spreadsheets currently allow even simple interactivity of the sort that will be described in chapter four. Considering the trade-off tasks that these people are working with everyday, simple interactive graphs would be very useful tools.

Another application area where interactivity could usefully be applied is the visualisation of mathematical models. In many situations, people within science and business are developing mathematical models of problems. Such models can be very complex and are often difficult to think about qualitatively. This thesis demonstrates a number of ways that these models can be visualised and explored. Such exploration can help users understand their problems, find useful solutions an communicate aspects of their models.

1.1 Visualisations as external representations

Another fundamental concept behind this thesis is that by combining interactive tools with abstract representations it is possible to build external representations of the user’s problem. Such external tools allow users to transfer much of the procedural work involved in problem-solving to the computer, enabling them to concentrate on interpreting their problem and searching for a solution. Nardi and Zarmer (1993) suggested a similar idea when they described a system in which users could build up problem-solving representations from simple building blocks that they called “Visual Formalisms”. As they identified, such tools can form an integral part of the problem-solving process and “can stimulate and initiate cognitive activity”.

Another value of such a representation si that because it is external it can be shared. Thus visualisations can become valuable problem representations with which to communicate with other people and educate them about the constraints one is working under. Such communication can also enable a group to negotiate  between alternatives in a reasoned manner.

1.2 The dangers of automation

One of the problems with having such powerful computers is knowing how and when to use that power. In many situations it is possible to fully automate a representation. For instance Casner’s solution to the problem of providing a representation for each of a user’s different questions was to build an automatic graph builder (BOZ) based on a task analysis of each question a user wanted to ask. For each question BOZ analyzes the task, works out an appropriate representation and then presents the user with a single static solution to interpret.

However a contradictory idea to such automatic graph building is that part of the value of building a graph is in how one constructs it. As Cleveland (1985) puts it: “graphing data needs to be iterative because we often do not know what to expect of the data; a graph can help discover unknown aspects of the data, and once the unknown is known, we frequently find ourselves formulating new questions about the data“. Similarly, in describing the value of interactive graphics, Bertin (1977) says: ” A graphic is no longer “drawn” once and for all; it is “constructed” and reconstructed(manipulated) until all the relationships which lie within it have been perceived… a graphic is never an end in itself; it is a moment in the process of decision making“. By automatically generating graphs much of the valuable constructive process is lost. A better alternative might be to present the data in the form of an interactive external representation (or externalisation). In this way the user can gradually build up an understanding of their problem.

The issue here is partly one of context. BOZ would be a useful tool for a situation where a user wanted to generate the best static graphic to communicate a particular point (e.g. for publication in a magazine). However, if this tool were to be used for graphical problem solving, then automation removes a lot of the externalisation’s value by taking away control from the user. The opportunistic and constructive support that could be added to the externalisation through the use of a computer is absent. Added to this the user would lose the continuity and flow of their problem solving process when they describe each new question in terms of a formal task analysis.

Much over-use of automation can be found in engineering software. A lot of these tools apply algorithms to mathematical models and come up with a “solution”. However the engineer who is using it may not have a good understanding of why that particular solution has been suggested. As Julie Bort (1996) points out “Data is fed in and results come out, but the tool doesn’t report how it comes to its conclusions. Sometimes the how is as revealing as the what.” The visualisations described in Chapters 5 to 10 allow the user to stay in control of their problem by actively constructing their solution using an external representation. In this way the user should be able to gain an understanding of how they achieved their solution and the alternatives available.

The problem with automation is that it tends to be viewed as a “cure-all” As Newell and Card (1985) warn, some researchers seem to act as though “if the interface is intelligent then it is not necessary to know anything about the user because the interface will be able to interact intelligently”. This thesis is not arguing against automation per se, but rather that we must focus our attention on exploiting the different strengths of both humans and computers.

1.3 Problem Solving as Exploration

One of the reasons that automation is not always an ideal solution is that a user often does not really have a good understanding of the available information before they start to solve a problem. For instance if I were to start to look for a house to buy I might have a fairly good idea of what my ideal house would be but I don’t have a good feeling for the current market.. Many of today’s database tools would require that at this early stage in my decision making I should define a specific query for the computer to search. For example: Find all the houses, with three bedrooms and a garage, that cost less than £80,000. The database would then return a resulting subset of houses that satisfy this query, however I would not gain an understanding about how this subset relates to the whole set of available houses.

Ideally I would like to start my problem-solving by taking a rough look at the number of houses in a variety of areas to get a feel for what is “out there”. Once I have explored the problem a little I can come up with a clearer specification of what I am looking for. Then I would be interested in seeing a variety of options that are close to satisfying my specification. Finally I would make a choice usually based on some form of trade-off, since it is unlikely that the “perfect” house exists. Lunzer (1996) has called this process reconnaissance. It appears to be a crucial part of many problems solving situations.

Interactive external representations are ideally suited to solve this sort of problem. The  underlying representation can provide contextual information about the whole data space and the user can make use of interactive tools to move through the space, iteratively adjusting their query. Responsive feedback from each interaction provides the information needed to perform each iteration of problem solving.

The visualisations presented in this thesis will support two forms of exploration. Firstly exploration where one is looking to select a subset from a large data set (The Attribute Explorer – chapter 4). This sort of interface can be valuable for any sort of database. Secondly the exploration of a mathematical model where one has a more or less continuous representation of the whole space. In this case the solution sought is often the definition of a useful region that satisfies a number of specific constraints. There are a huge range of situations in which such models are used. Both of these forms of exploration have many potential applications.

1.4 Thesis Structure

 The Thesis is made up of four parts: background, invention, abstraction and epilogue.

This first chapter of Part I has introduced the importance of exploiting interactivity in external represenations, the main topic of this thesis. Chapter 2 examines the value of external representations in a little more detail. Chapter 3 outslines the 3 research strategies adopted during the research work: exploring the problem space by designing novel artifacts and providing designers tools for thoughts by developing abstractions. these first three chapters together (Part 1) provide the background to this thesis.

Part 2 comprises five chapters describing the novel artifacts designed during the course of research fo different forms of grapical probelem solving. Chapter 4 introduces the Attribute Explorer, a tool for exploring multi-attribute data (a set of objects each described by values to a set of attributes). Chapter 5 explains Pre-Calculation, a method which enables us to create data from mathematical models which we can then visualise. Chapters 6 and 7 describe the Influence Explorer and Prosection Matrix. These tools have been applied to financial, electronic, structural and mechatronic problems. Chapter 8 outlines a formative design evaluation that was carried to identify weaknesses in the visualisation as they progressed.

Part 3 introduces a number of different abstractons that can be used to characterize these tools. Chapter 9 describes interactive external representation in terms of the different forms of representation and interactivity that they use. A number of different visualisation techniques are described in terms of this characterisation and some simple guidelines are formulated. Chapter 10 examines a few of these visualisations in more detail using a data centric notation. The notation is used to describe the specific data selections possible with each visualisation, the consequence of this data selection and the perceptual comparisons that these consequences then facilitate. Such descriptions can be valuable for comparision that these consequences then facilitater. Such descriptions can be valuable for the comparision of different visualisation tools and the identification of weaknesses.

Finally Part 4 summarizes the findings of this thesis and suggests how we should move forward from here. In particular it identifies opportunities for further development and reasons why these might prove valuable.

1.5 Contributions

This thesis makes a number of novel contributions

• it brings together an innovataive research philosophy
• It presents pre-calculation as a novel method to create data to be visualized from the mathematical models (Chapter 5)
•  It presents a number of novel artifacts (The Attribute Explorer in Chapter 4, The Influence Explorer in Chapter 6 and the Prosection Matrix in Chapter 7)
• It characterises the space of interactive externalisaations (Chapter 9)
• It presents DIVA – a new notation to describe the perceptual information that interactive external representations afford to users (Chapter 10).

1.6 Research Context

Having introduced the concepts that structure the thesis I would now like to briefly explain he context within which it was written, both physically and academicially.

My own undergraduate bacground is in psychology (although one upon a time I did do the first year of an engineering product design degree). I then worked for a while as a research assistant doing empirical HCI studies on icon search(May et al 1993). The work reported in this thesis has been carried out within the Electrical Engineering Department at Impeerial College. The visualisation tools arose out of direct collaborations with engineers in the labs at Imperial and a number of different industrial partners (Philips, Infolytica, Reuters & First Derivative Associates). The context that grounds the designs came from meetings and conversations between: industrialists, scientists, domain specialists (in financial modelling, marketing and many varieteis of engineering), statisticians, engineers, designers, computer scientists and psychologists.

In a sense for me this work has actually been “trans-disciplinary”: I have spent much of my time over the last four years learning a lot about engineering, statistics and programming. These are all important skills. In a sense they have helped me learn about the context of my problem domain in an almost ethnographic way. I have in effect worked next to my “problem holders” for the last four years. For a psychologist, even one doing HCI, that a pretty priviliged position.

However the other side of thsi coin is that the academic field of HCI has also provided an important context for this work. Although Rasmussen (1992) argues that “HCI or, better, cognitive engineering is not a new academic discipline but a conceptual market place for exchange of cross-disciplinary work”, I would disagree; it is important that we develop an academic discipline, sharing a common ground, within which we can foster the cumalative growth of ideas.

This thesis is based on many concepts and arguments that have emerged from within HCI. Without the existence of that community I don’t believe it could have been written.