A Computer-based Iconic Language
Stuart Mealing and Masoud Yazdani
Department of Computer Science, University of Exeter - England
Introduction
In "The Hitch-hiker's Guide to the Galaxy" [Adams 1979] it becomes possible to understand communication in any language by plugging a Babel fish into your ear. The fish automatically translates from any language to any other language, and exemplifies the long-term dream of mankind of being able to communicate across all language barriers. Computer assisted language translation [Yazdani 1989] is a tool of Artificial Intelligence (AI) which will eventually help realise this dream, and the vision of Koji Kobayashi [Kobayashi 1986] - that it will be possible for an English speaker to pick up a telephone and speak with a Japanese in English - is only ahead of its time in the sophistication required of the system. An alternative approach is to use a single, international language, but attempts to popularise Esperanto have met with little success, largely due to the initial effort required to learn it (by a worthwhile number of people). There are, however, existing signs, symbols and icons which are understood internationally and it is the aim of this research to discover the level of subtlety of communication which icons can achieve, and to develop a computer-based iconic language. This will be designed, in the first instance, to operate within a limited domain such as `hotel booking', where it is common for language difficulties to arise but where a limited and predefined range of ideas is likely to be expressed. A scenario such as this could be treated as having a `situational script' [Schank 1977].
Developing the language for use on a computer will significantly impact its design, offering a flexibility and `intelligence' in presentation and interaction which is not available in other media. Whilst the ideal iconic language would require no learning and would be intuitively obvious to use, it is more realistic to expect some explanation to the user to be necessary, and the system employed will seek to minimise this need. A `friendly' system will encourage discovery through use and the computer will allow, for instance, for the user to interrogate icons to learn (or confirm) their meaning, this explanation coming in a lower level iconic form. The user interface can also be supported at any point by labelling in written or spoken natural language.
Icons
An icon is a symbol representing, or analogous to, the thing it represents [Collins 1986], it is something which looks as though it could act as a proxy. Huggins [Huggins 1974] asserts that: "Iconic communication deals mainly with non-verbal communication between human beings by the use of visual signs and representations (such as pictures) that stand for an idea by virtue of resemblance or analogy to it in contrast to symbolic communications where the meaning of a symbol is entirely nominal (such as English text describing a picture)". Pure icons, therefore, rely initially on recall of a previous visual experience on the part of the user (either first or second hand) with sufficient particularity to make their use in a particular context clear to him. They may, however, take on the role of symbols in subsequent use, and can be used in conjunction with symbols i.e. a diagonal bar through an icon can indicate negation. In the context of this research the language will be designed for display on a computer screen, and prototyped initially in HyperCard TM.
Criteria for the icons are that they should be:
As it is desirable that the system should not rely at any stage on any natural language, an icon whose meaning is not immediately obvious to the user should be able to explain its meaning in terms of more fundamental icons or through diagramming its evolution from its source imagery. It would, for instance, be possible to `click' on a compound icon (standing for an event or concept having several elements) and have its meaning expressed by several static base icons (standing for those individual elements). Alternatively, a simple animation of the `actors' in the base icons could explain the meaning of the compound icon, by performing its message or by diagramming its construction. `Clicking' on those base icons might initiate a simple, but revealing, transformation from a more comprehensive (possibly photographic) image showing the source of the icon, to the icon itself. The use of relevant photographic images as backgrounds to messages could establish the current domain, and the use of colour will be a helpful adjunct but cannot be used to carry essential information that might become lost on a monochrome system.
The icons will therefore comprise several levels. The top level will present the message and will incorporate compound icons and symbols where needed. Querying any icon will initiate a move to the second level, at which base icons will seek to explain the meaning of the top level icon, acting out the meaning if appropriate. A further level could trace the icon's development from suitable photographic references to its source, and at any level natural language could act in support. The natural language which is active would have been chosen by the user on entering the system. In keeping with the intention of pictorialising the complete interface, this selection would be made through identifying the user's country of origin on a map.
Additional weapons in the design armoury are therefore:
Visual language
If a picture is worth a thousand words, an icon must be worth at least a sentence! Pictures can communicate more than just their visual contents because a wide range of prior knowledge and experience is brought to bear on the imagery by the viewer, and this can be exploited to enrich the meaning of an icon. Pictorial images are specifically used for communication in many situations, and their interpretation is often subject to the context in which they are found. For example, images on a can of food often illustrate the contents, but a label showing a picture of a smiling child would not be assumed to be an indication of the contents of a can, since eating people is not part of our cultural heritage [Yazdani 1990]. Similarly, the interpretation of gestures that accompany spoken language can be subject to context, but can often significantly improve understanding, notably in a situation where the spoken word is difficult to hear or to comprehend. Many people speak little of any foreign language but are able to understand and communicate a surprising amount when abroad with intuitive sign language. As well as these multilingual gestures there are many internationally understood symbols (such as arrows to indicate direction and overlaid diagonals to indicate negation), a study of which can usefully be brought to the design of our icons.
Written sign language is pictographic, that of the North American Indian being described as "elemental, basic, logical and largely idiomatic" [Yazdani 1990]. Since it is likely to be written/drawn on a single surface, the whole message is visible at any moment in time and therefore any part of it is open to scrutiny at leisure in the context of the rest. This is very different from gestural language which is serial in nature, only a small part of the message being communicated at any one moment, and attempts to look for context relying on memory. Whilst it is envisaged that the proposed iconic language will employ the advantages of viewing a complete message (or `page' of a long message), the alternative of having a message unfolding over time could be employed in part. An animated icon can `act out' simple time-based concepts (such as walking or shrugging) with the advantage that it can be repetitive and incorporated as an element in a continuously visible message.
It is practical to categorise icons (in the manner of Marshall McLuhan) in terms of the `temperature' of information which they can convey, according to their position on an axis stretching from pictures to symbols, and on their level of animation. (In other contexts, icons made of pictures have been called `picons' and moving icons have been called `micons'.) It is not necessarily the case, however, that the hottest presentation is the best, as it may become too specific to illustrate a general case.
Design considerations
The `dustbin' icon is now used regularly in GUIs (graphical user interfaces) and provides a suitable example for discussion of some of the considerations necessary in design of an icon. Already we have referred to it as a `dustbin' icon, whereas in America it would be a `trashcan' and in France a `boite å ordures' so it is fulfilling its multilingual function, but does it represent a globally used waste disposal method? Even in a Western society, people living in blocks of flats are more likely to use rubbish chutes than dustbins. The icon is also derived from what used to be a typical dustbin, but which is now superseded in many instances by more modern designs. If dustbins cease to be used in the future, subsequent generations will require either a new icon or a short history lesson. Providing, however, that everyone who uses a computer is familiar with that particular form of waste disposal, then it is clear what object it stands for, but otherwise the image will need to be explained. Having understood what the object is, do we now understand what function the icon serves in the application? It is not a big conceptual jump from disposing of unwanted material in a dustbin to disposing of unwanted files in an application, but it is one that needs to be explained on first encounter with a GUI, and it is not obvious that the same icon may be used to eject discs in some machines. Typically of icons, the dustbin image is standing for a noun, but is intended to implicitly conjure up a didactic phrase associated with the noun. (Since the other icons on the computer screen at the same time as the dustbin are usually standing for items in an office environment e.g. desktop, files etc., it might be thought that a waste-bin would provide a more appropriate disposal metaphor, and this is sometimes used). Other, more complex, icons such as a figure ascending a staircase (to indicate an exit route, perhaps), are far more explicit in their description, and have a sentence-level structure.
The style of existing icons falls into three main categories:
The icons are normally diagrammatic and may be either combined with, or interspersed with, symbols (arrows, numerals etc.). It is unfortunately apparent that their style is as subject to fashion as any other design product, even typefaces, and that the ideal, timeless range of icons is unlikely to be found. It is, however, possible to avoid the excesses of fashion, and to produce icons which have a long life span and are amenable to future stylistic updating without compromising continuity of recognition. Of the styles described, the three-quarter top view is potentially the most informative but requires a slightly more sophisticated visual understanding, the solid silhouette is the most arresting but somewhat limited in the range of things it can deal with easily, and the `realistic' is easy to recognise but risks moving too far away from the generalisation which an icon often requires. Pictographic sign language is more calligraphic in its visual character.
It is normally confusing to mix styles within a message, but the possibility of doing so for the sake of emphasis can be explored. The reflex recognition required of a road sign, however, is not the same as that required of icons to be used in the composition of a message. The different styles lend themselves to different degrees of abstraction, and the likely criteria of adopting a set matrix size for the icon will condition the level of detail and stylization.
Animation of icons has potential for extending the range of concepts that can be dealt with, but should only be used once static icons have proved inadequate, or as an aid in explaining the meaning of static icons. In the above example, if the user interogates the `tick' or `cross' icon, a head either nods or shakes to explain the symbol (though it should be recognised that these gestures are not universally recognised). It should also be used conservatively to avoid an iconic message deteriorating into a fairground of visual activity. Given those reservations, the possibility of limited animation being used to make connections between static icons, to describe concepts involving verbs, and to be used in the explanation the evolution of any unfamiliar icon, is very promising. Whilst the implementation of the language on computers makes animation viable, storage and display considerations for animated icons must be considered, but current technology and data compression techniques should cope with the relatively limited requirements. The envisaged icons will also lend themselves to vectorisation (now commonly used for fonts in DTP applications) which implies very limited storage, easy transformation and scaling, and a high level of portability. At some future date it might prove relevant to consider the possibility of 3-D icons or of 2-D icons inhabiting 3-dimensional space.
The judicious use of colour can be employed to good effect, most obviously for coding, but the system must be able to communicate fully in monochrome. In many situations, such as at an airport, the location itself acts as a cue to the domain of the icon, but this is not necessarily the case in a computer-based language capable of dealing with many domains. Background images, however, could help to establish context for a message and multi-media developments are bringing closer the option of using video in that role ! In fact, since much multi-media production is visual, and hence alingual, it would be a very appropriate medium in which to use an iconic language.
It is worth noting that the potential standardisation of icons internationally across a range of applications is currently inhibited by legal restraints regarding copyright, and it is useful to consider the implications of an iconic language bridging that barrier.
Syntax
In most everyday situations icons do not interact with their neighbours. They often appear singly and, when grouped, usually constitute a list of features or properties (such as listing the facilities of an hotel). In the earlier example of the `dustbin' icon, the icon is a passive representation of a word around which the user might build an understanding of its intention. A more dynamic icon would offer a far more explicit instruction about its function, and as such would embody a unit of meaning equivalent to a phrase, sentence, or perhaps even a paragraph. To comprise an efficient language, icons need this flexibility and the ability to deal, for instance, with concepts incorporating the equivalent of natural language verbs. It is unlikely that constructing a message with a one to one matching of icons to words would be appropriate, and the size of the unit of meaning which an icon could encompass is likely to grow in proportion to familiarity with the system. The ability to interrogate icons therefore offers the advantages of efficient, high level, compound icons, with integral explanatory support, the need for which decreases with familiarity. In this context, communicating meaning clearly need not rely on an explicit grammar or syntax and it does not become necessary to think of an icon being translated into natural language as a stage in its comprehension.
Basic icons can be combined into compound icons. For instance, we are familiar with icons for man and stairs being combined to show a man on stairs. An arrow added to indicate the direction upwards produces an icon telling us to use the stairs to go up. This is often combined with a word or an image to explain that the stairs take us to the exit, or perhaps provide our exit route in the case of fire. A diagonal bar through the same icon (often in red) would tell us not to use the stairs. The computer would allow us to produce compound icons with some ease, to modify scale or weight to suggest priority or emphasis, and to create links between icons. Some punctuation marks, such as full-stops and brackets, might also be used.
The establishment of an iconic dialogue is more challenging than the preparation of an iconic message. It could take the form of alternating messages, or the receiver might be empowered to interactively modify, or comment on, the sender's message (perhaps overlaying a request for a hotel room with signs to indicate the availability of the required accommodation on the required dates). Pressure on an icon library would increase with the size of the domain of the dialogue and the situation encourages the ability to develop fresh icons, according to need, from base icons. Comparisons with morphemes and lexemes are possible, but the scale of meaning of a base icon is likely to be greater and open to inflection by modification of its visual detail. Once a dialogue is established, its own dynamic will suggest context and assist with understanding of the icons, but its initiation may require more particular handling.
Conclusion
Icons offer a rich potential for communication across natural language barriers. If confined to the European arena, the many shared conventions (such as the use of Arabic numerals and question marks) make their design much simpler and their sure interpretation more certain. The computer provides an ideal device for the implementation of a flexible iconic communication system.
The need for such a system is made more urgent by the increasingly international nature of commercial, educational and social communication. Examples such as booking a hotel room abroad, ordering machine parts from a foreign subsidiary, or accessing EPOS (the European PTT Open Learning Service) all provide occasions for such a system to prove its worth.
References
Adams D. (1979) The Hitch-hikers' guide to the Galaxy - Pan Books Ltd.
Yazdani M. (1989) Le-Mail: A bridge to International Communication - Dept. of Computer Science, Exeter University
Kobayashi K. (1986) Computers and Communications - MIT Press
Schank R. & Abelson R. (1977) Scripts, Plans, Goals and Understanding - Lawrence Erlbaum Assoc.
The Collins English Dictionary (1986)
Huggins W. H. & Entwisle D. R. (1974) Iconic Communication: an annotated biography - The John Hopkins University Press
Yazdani M. & Goring D.(1990) Iconic Communication - Dept. of Computer Science, Exeter University
Selected bibliography of icons
Kepes Gyorgy Ed. (1966) Sign, Image & Symbol - Studio Vista, London
Dreyfuss Henry (1972) Symbol Sourcebook - McGraw-Hill Book Co.
Diehtelm Walter (1974) Form + Communication - Editions ABC, Zurich
McLendon C. & Blackstone M. (1982) Signage - McGraw-Hill Book Co.