Iconic Communication

Colin Beardon

Rediffusion Simulation Research Centre, University of Brighton, Grand Parade, Brighton BN2 2JY, UK

The popularity of the WIMP interface opens up real possibilities for developing new forms of human communication based around images rather than text. The success of such systems will depend upon our ability to embed them in existing social practices at the same time as identifying and exploiting what is novel and useful in the computer. The use of pictures and gestures to convey our ideas is a basic form of communication that two people frequently resort to when they find they share no common language. This is illustrated equally well by the signs I make to a person when I do not know the local language and want to find somewhere to eat, as by the pictorial signs that we find in international airports. Locate this basic ability within the novel features offered by the WIMP interface and it is possible to imagine a new medium in which computer processes are more easily accessible and communication between human beings is simplified.

The idea of iconic communication is, of course, not new. During the 1930s Otto Neurath developed Isotype, a system for communication which uses stylised graphics within a two-dimensional syntax. Examples of Neurath's work ranged from very specific examples of how a complex idea can be conveyed graphically, to a proposal for an international set of iconic images (Neurath, 1978).

In the 1950s, Charles Bliss developed a set of atomic icons that represent basic objects in the world and their features (Bliss, 1965). These can be combined to form complex icons that map on to the set of words found in natural languages (Figure 1) including such linguistic entities as pronouns. Blissymbolics has been used as the basis of several computer based systems of communication.

Mouth ---- ear -------- language -------- electricity --- -telephone

Figure 1: The construction of the Bliss symbol for "telephone"

Minspeak is a system that can generate natural language words or phrases as a result of inputting a sequence of icons. In a typical application there would be a set of about fifty icons and each sequence of three icons is mapped onto one of about two thousand words or phrases. Compositional rules, represented as an Icon Algebra, indicate how strings of icons are combined into a single concept. In conjunction with a modified keyboard this means that individual entries in an extensive vocabulary can be accessed within three keystrokes by a user (Barrow and Baker, 1982).

These iconic languages were not all as successful as their developers hoped, but they do show that there are distinct advantages in a method of communication based upon graphical icons. Our ability to learn or recall the meaning of a sign seems to be greatly enhanced to the point where we may not need to be told what it represents or to explicitly learn its meaning. There can also be a certain efficiency of expression in that only a relatively small number of icons may be needed to express a quite complex idea whereas many more letters or words are needed to represent the same idea in textual form. Furthermore, our ability to recognise icons does not seem to depend upon the natural languages we know, suggesting that iconic systems may overcome the problem of linguistic diversity. There is, however, recognition that iconic images are often culturally dependent.

Before we get too euphoric about iconic communication, a note of caution is advisable. Both Neurath and Bliss saw iconic languages as providing world-wide systems of communication with the morally laudable aim of promoting world understanding and thereby world peace (see William Edmondson in this volume for some further examples of this point of view). These typically modernist sentiments would nowadays be seen as simplistic, assuming a universality that has not been proven and probably does not exist. Diversity, in our postmodern age, is endemic. We prefer to see iconic systems as potentially enabling some level of communication between people with different viewpoints and we have learned to be wary of any attempt to claim universality or objectivity for what we are creating.

A further cautionary note comes from Alan Kay, who distinguishes "iconic" from "enactive" and "symbolic" mentality (Kay, 1990, p.196). Iconic mentality, he argues, is particularly good for the tasks of recognising, comparing and configuring and for representing concrete things. It is not particularly good when the task is locating oneself, manipulating, reasoning and representing abstract things. If this is true, and it certainly is the experience of several people I know working in this area, then it not only provides a further reason for questioning the universality of iconic systems, but also suggests that we should be looking at the communicative tasks that it might be appropriate to undertake iconically. It also raises the need to integrate iconic systems with other systems, the most obvious being symbolic languages.

It was with such a mixture of enthusiasm and caution that a group of interested people, from various academic disciplines and from the commercial world, met in Brighton in December 1992 to discuss the topic of "lconic communication". In the call for participation iconic communication was described as the attempt to build cross-language communication systems that completely avoid the use of words and rely solely upon pictorial symbols. While the main focus was intended to be general purpose person-to person communication systems there was also interest in purely iconic interfaces to existing software.

Seven presentations were made at the meeting and these covered a wide range of topics including: paperbased iconic systems; abstract vs. concrete icons; icon recognition for the blind and partially sighted; the use of metaphors; semiotics; communication by animation; self-explaining icons; and iconic languages. There were also visual presentations by graphic designers, illustrations of icon use in commercial products and demonstrations of experimental systems. All the presentations were of high quality and three of the presenters were asked to produce papers for this volume mainly because their contributions, taken together, were representative of most of the major issues that evolved during the course of the meeting.

William Edmondson's paper not only seeks to clarify the terminology used in the debates, and particularly the concept of an 'icon', but also presents some arguments against the feasibility of general purpose iconic languages. By focussing on terminology we are forced to make explicit our assumptions about what icons are and why they are special. The naive answer is that an icon refers to an object by simply representing it. Thus by recognising the image in Figure 2 as a man we know that it refers to a man. This is quite different from the word "man" whose relationship to the object it refers to is seen as arbitrary, conventional and simply has to be learned.

 

Figure 2: Icon for 'man'

Even in this simple case the naive view does not stand up to close examination. The image in Figure 1 does not look like any man that I know: it is an idealised image of a man. It also represents a single man whereas it may be used to refer to the class of men. Furthermore, I may recognise it straight away as a man but someone else may see it initially as a special kind of paper fastener or a clothes peg. I may point out my preferred interpretation and they may agree but then they will have learned my conventional interpretation. Finally, the meaning of the icon may not simply be its denotation but rather its pragmatic effect. The symbol is likely to be found on a washroom door and has the effect of "Here is the men's washroom".

Edmondson first dispels the common myth that icons are isolated, stable., non-arbitrary, non conventional signs which refer objectively and obviously. The difference between icons and symbols, he maintains, is more a matter of degree than of kind and neither form can be seen as employing a pure means of denotation. Iconicity, as several speakers agreed, is really a matter of degree.

Given that some signs (that we will continue to call 'icons') are intended to rely to a greater degree on their iconicity then how, it may be asked, are we to judge their success? Again, the most common assumption is that they are successful if we can recognise their referent when we first see them. If a large proportion of people recognise what a particular icon stands for, it is argued, then it is a successful icon.

This test is entirely semantic whereas some would argue that we require a pragmatic test. A road sign, for example, that indicates a major road ahead is not properly interpreted if a driver recognises the particular configuration of roads it depicts but does not relate it to their current situation and carries on regardless. We would say that it has been properly interpreted if the driver applies it to their position on the road and at least tries to modify their speed accordingly. Recognition, it would seem, should involve understanding the intention behind the message.

Levels of abstraction and indirection can also be a problem with too simple a criterion for success. The road sign for a bridge does not try to represent the view that we normally have of a bridge, but owes more to the way that bridges are represented on maps (Figure 3). We first have to recognise this reference and then transfer it to our immediate environment. Some signs employ more complex rhetorical techniques, such as using an object that is associated with the object to be represented. Examples of this are using the image of a top hat to represent "man" or the image of a goat's horns to stand for "goats".

 

Figure 3: Icon for 'bridge'

The significant task of recognition is usually assumed to occur when the icon is first encountered, but there is a strong case to be made that it is more important that the meaning of the icon is remembered. Often, when confronted with a novel set of icons in some new application, users will discover the meaning of each icon by experimentation with occasional reference to a tutorial guide or reference manual. If initial recognition is the important criterion, then one could argue that use of tutorials or reference material indicates the ineffectiveness of an icon but this would seem a harsh judgment. An initial struggle to determine the intended meaning of an icon may be a small price to pay if it is then remembered accurately.

Icons are also rarely used in isolation. Whether we consider icons as road traffic signs, for directions in zoos, for identifying sports at the Olympic Games, or as tools within a computer program, the most common applications of icons is within sets (Dreyfuss,1972 ). Here the criteria for success will not involve recognition alone but will also include differentiation from other icons in the set. Some keyboards intended for small children have taken this to the extreme where there are four main function keys coloured read, yellow, green and blue. Each application maps functions onto these four keys in its own way. Whilst there may be some association between the colour chosen and the underlying functionality, the mappings are usually purely conventional and always have to be learned. This does not mean they are ineffective, rather they succeed simply because the differentiation is high.

Edmondson stresses the importance of considering icons within systems of icons. While generally pessimistic about the possibility of general purpose human-to-human iconic communication systems, he is more hopeful about the prospect of iconic systems for interfacing with computer systems. These two projects clearly pose different sets of problems for designers. If the number of objects to be represented is known in advance, then the designer of an icon set can have clearly defined objectives such as recognition, recall and differentiation. If we are designing a more open iconic language system, then it is important that the language is able to grow and evolve in the same way that natural languages do.

In either case, one of the reasons why icons should be considered in the context of sets is that they can then exploit metaphors. Philip Barker and his colleagues concentrate upon this approach to iconic systems in computer interfaces. We know that word recognition within meaningful text is a completely different process from word recognition for isolated words, to such an extent that in-text word recognition seems to be largely based upon expectation. Their argument that a well-chosen metaphor can assist in the recognition of iconic symbolism is therefore particularly well-taken.

They point out that metaphors do more than just assist in the recognition of the object that the icon represents, they also help identify its underlying functionality. This is an important point. If I am confronted with a set of icons in a tool pane - say, a pencil, a line, a brush, a paint pot and an aerosol spray - then it is one task to be able to recognise that each icon represents one of these objects. What is more useful is that the metaphor helps me select the appropriate tool for the task I want to perform. If I want to produce a small detailed drawing I know from my real world experience that a pencil, or perhaps a very fine brush, would be appropriate. If I want to produce a large, bold, free curve I would select a large brush. If I want to cover a large area then I would need a paint pot or an aerosol. The significant power of the painting metaphor is that my real world experience of using painting and drawing tools enables me to select the appropriate tool for the task at hand. This would not happen, for example, if the pencil tool drew a line that was 8 pixels wide, or the aerosol only worked in a straight line.

The use of metaphors within icon systems has its own problems and these are also described by Barker. Creating icons for every function can prove problematic. Many applications have a large number of functions (it is not unusual in CAD systems to have well over one hundred) and if every one has to be represented by a different icon this can be difficult. There are problems both in creating so many different images within what is traditionally a very limited medium (a 32 x 32 bit square), and also in displaying so many icons on the screen at the same time. One solution to this latter problems is that of grouping the icons into subsets, each of which is represented by another icon. Clicking on this 'group' icon will cause the subset to be displayed.

The power of the metaphor in people's understanding was illustrated in a recent experiment at the University of Brighton where a number of students were each asked to produce an animation to explain the functionality of the "paint pot" tool as found in many Apple Macintosh packages. The students were familiar with the tool which is used to instantaneously fill a bounded area with the current pattern and colours.

The resulting animations varied from a sequence of quite literal screen copies, through the use of alternative real-world painting tools such as paint rollers, to some quite unexpected metaphors such as wild animals. In a large proportion of cases, and especially in the more literal ones, the filling process was shown as being gradual rather than instantaneous. This would seem to indicate that the students did not have a cognitive model of the tool in terms of its formal or geometric meaning (which has no meaningful time coordinate) but one in which painting or filling occurs over time. This limited experiment tends to reinforce Barker's conclusion that metaphors in an end-user interface are significant ways of setting a context within which the user develops cognitive and conceptual models.

Based on a number of case studies in which metaphors are used in commercial products Barker argues that metaphors are not only unavoidable but to be positively welcomed as they greatly assist the user. Further, the metaphor design should become an integral part of the design process. One advantage of such an integrated approach is that extensions to the proposed system may be suggested by properties of the metaphor which are appropriate to the system being designed.

The question of the international acceptability of pictorial images is also raised. It is worth mentioning in this regard the recent attempt by ISO and IEC to develop international standards for icons (ISO/IEC, 1992). The first tasks in this process were to decide what types of image should be covered by a standard and how such a standard should be written. Then a number of particular standards have been proposed so that if a certain functionality is represented iconically in an application then it should adopt one of the recommended images in order to claim compliance with the standard. A typical example would be an icon to represent a printer and that particular standard would say that, if an application refers to a printer, it must use one of the specified printer images.

Such a process of standardisation seems appropriate as it reflects emerging consensus practice but it will necessarily be quite slow as common functions take some time to emerge. Over time, however, we could see a process whereby a sign)ficant number of icons develop conventional meanings that are accepted as

having a certain legitimacy through standardisation. This does not preclude their use in other contexts, within more experimental systems for example.

William Edmondson raises doubts about the universal ability of icons to signify, arguing that their interpretation is necessarily subjective and conventional. In her paper Claire Dormann suggests a way of addressing these perceived weaknesses by exploiting some of the specific advantages that the computer medium provides, namely the ability to respond interactively to the user and the ability to provide animations. Following from an idea originally developed by Stuart Mealing, she discusses the possibilities of providing animated explanations of the meaning of icons (Mealing, 1992).

Dormann's paper suggests a significant new feature which could one day become a standard part of applications software - on-line animated help. With such a facility the user of a system can interrogate any icon and receive an animated explanation. Of course, there are many different types of explanation that are appropriate under different circumstances and other work has looked at the provision of explanations appropriate to the user's needs (Baecker, 1987). The advantages of animated help in general are great. It would provide assistance that is free of any particular natural language so that only one version needs to be provided by the manufacturer. It can employ various rhetorical techniques to provide explanations that seem appropriate to the user's experience (e.g. exaggeration, personification, etc.). It can slow processes down and can show the reasons for common mistakes.

The success of such a method of bundling further semantic and pragmatic information with an icon will depend upon the development of appropriate narrative techniques for such animations and here we have much to learn from documentary film makers and illustrators. Film animators are surprisingly successful at communicating across language, and sometimes even cultural, barriers. They frequently use conventions but also sometimes create the very conventions they later exploit. There is much research to be done on the structure of explanations in general and, particularly, whether the same basic structure can support both textual and visual explanations.

Both Edmondson and Dormann suggest that the use of icons should be seen in terms of syntax, semantics and pragmatics, which suggests a strong connection with linguistics. This will no doubt be another fruitful area of research but we have, I hope, learned from weaknesses in the prescriptive and compositional semantics of previous iconic and linguistic systems. Any communication system must allow for change if it is to survive and be continually useful within its domain. It must be designed to change or it will become irrelevant.

In practice this means that iconic communication systems, if they are to be at all general, must allow for meaning to be negotiated between author and reader and for conventions to emerge from use. In this respect the really challenging area to face us in the future is the development of iconic dialogue systems in which statements can be made, questions asked, explanations given and agreements arrived at. Stuart Mealing's work on a hotel booking system is an important first step in this direction (Mealing, 1992). Such systems should also understand the need to integrate an iconic system with non-iconic systems, as suggested by Jennifer Lennon and Herman Maurer in their MUSLI system (Lennon & Maurer, 1994).

The field of iconic communication is in its infancy and we consider it today in the context of the technology we know about today.The one thing we have seen is that today's computers provide a context within which many of the limitations of traditional pen and paper iconic systems can be addressed. How far this will take us is a matter of both theoretical research and the development of systems based in social practice.

References

Baecker R (1987) Animation at the interface. In: R Baecker and W Buxton (eds.) Reading in HCI, a multidisciplinary approach, Morgan Kaufman, California, 251-269.

Barrow H & Baker B (1992) Minspeak: a semantic compaction system that makes self-expression easier for communicatively disabled individuals. Byte, 7, 9, 186-202.

Bliss C K (1965) Semantography (Blissymbolics). Semantography publications, Sydney.

Dreyfuss H (1972) Symbol Sourcebook. McGraw-Hill, London.

ISO/IEC (1992)Information Technology -Text and office systems -graphical symbols used on screens: interactive icons. Report No JTC1/SC 18. ANSI, USA.

Kay,A. (1990) User Interface: a PersonalView. In B. Laurel (ed.) The Art of Human- Computer Interface Design. Addison-Wesley, Reading, Mass. pp. 191207.

Lennon J & Maurer H (1994) MUSLI: a MUltiSensory Language Interface. In: T Ottmann & I Tomek (eds) Educational Multimedia and Hypermedia, 1994. AACE, Charlottesville, Va.

Mealing S (1992) Talking pictures. Intelligent Tutoring Media, 2 (2), 63-69.

Neurath O. (1978) International picture language. University of Reading, Reading.