From: [log in to unmask] [mailto:[log in to unmask]]
Sent: 09 September 2003 22:02
To: [log in to unmask]
Subject: [CTHEORY] Article 131 - Speaking in Djinni
_____________________________________________________________________
CTHEORY THEORY, TECHNOLOGY AND CULTURE VOL 26, NO 3
*** Visit CTHEORY Online: http://www.ctheory.net ***
Article 131 03/09/09 Editors: Arthur and Marilouise Kroker
_____________________________________________________________________
Speaking in Djinni:
Media Arts and the Computational Language of Expression
==========================================================
~D. Fox Harrell~
I. Introduction -- Speaking in Code
In the Djinni's lair they speak in code. Their Djinnish language
does not reflect a subservient nature as they announce "your wish is
my command." They have mastery over elements of our mortal realm,
their words cause events to occur. What people hear in their words
is their imperative language translated into our human tongues. They
speak in commands, power words with concrete effects. "Open sesame"
opens doors, a wish becomes tangible possibility. Humans translate
thoughts into operations using computational media. The languages
humans use to express these commands affect what they produce using
these media. The process of translating from ideas into imperatives
has profound consequences. Aladdin's tale narrates one example of
this process, the tale here tells another.
Computational media reflect the previous generation of media. They
also offer new characteristics unseen in earlier technological media.
What are these new characteristics and how can they be examined?
There are many varieties of computational media and it is quite
difficult to isolate a particular subset to begin investigation.
However, the tools used to present and create media art lie behind
every media artwork. The theory of programming languages is a useful
means by which to characterize these media. Formal languages offer
broad insight into the nature of computational manipulation and
specific organizational structures of imperative languages reveal
reflections of these structures in media software. This is a natural
reflection because the theory of languages expresses organized models
for executing algorithms and structuring data, which are the types of
manipulations human creators perform on media when treating it as
computational data. Finally, this is a means to characterize only
formal aspects of media manipulation, not the semantic. The ideas
here are not presented in a technological determinist vein -- stating
that it is programming language theory that caused this restructuring
of media. The influences of media and culture upon each other are
mutually dependent. This will be shown by reviewing examples of
these concepts of formal manipulation from both art and computing.
The reflection of programming languages in computational media should
be no surprise, and using the tools of programming language theory to
consider such media provides illuminating insight.
II. Interfaces Influence Art
I sit now in an accursed palace, corrupted by the literal and
malicious nature of a Djinn. I am a humble man, made from earth as
we all are. Still, that the language of the Djinni, of those beings
composed of fire, should be so malevolent, is a fact that I shall
always lament. When he emerged from the lamp and acknowledged my
three wishes, when I wished for the fulfillment of my artistic vision
as a lavish palace of lacquered coral and gold, alas I did not
realize the venom dripping from fate's bite. Had I been a stonemason
and crafted this castle with my own two hands the ceilings would not
be so low that my back scrapes them as I shuffle, bent over, to my
throne. Had I been a carpenter my satin canopied bed would not be so
splintery that it takes thirty minutes for me to slip safely from it
at dawn. Had I been a master of the olfactory arts of oil crafting
and perfumery I would have been able to banish the aroma of sulfur
and replace it with myrrh laden zephyrs wafting through every
corridor. Alas, the tool I used was a contractually obligated, but
ungrateful and rancorous, Djinn. The words "make for me a lavish
palace" were made reality by the Djinn's language and all that I had
left unspecified was decorated by his cruel tastes. My consolation
is that my request was not spoken: "make me a lavish palace" (as the
colloquial version would have been worded) lest instead I would now
be the inhabited abomination rather than be inside of it!
Artistic works are related to the tools used to create them. The
mark of the tool is apparent, whether the work has an explicit
relationship to the tools as in tromp l'oeil painting which reveals
it effect only when the mark of the tool becomes recognized and the
illusion is broken, photography which in the past has been presented
by news media as a transparent tool that represented a window to
reality--an aura which continues now despite its double nature now as
a purveyor of false images, or Nam Jun Paik's cyborg materials to
evoke his "cybernated" art concept.[1] This paper considers
primarily screen based media artwork generated on a personal
computer. Typically these works are created using commercial
software such as Macromedia Director, Adobe Photoshop, Adobe
Premiere, etc. Of course some minority of artists also create their
own computational tools. Regardless, even these artists are
influenced by the medium of their art and it is important to
understand what this influence is when engaging their work.
A starting point in this examination is the recognition of the role
of the interface as being intimately tied to content. Douglas
Englebart was one of the early innovators of the contemporary
computer interface.[2] He designed a networked environment designed
to support collaborative interaction between people using
computers.[3] Englebart is known as the inventor of the mouse,
windows, email, and the word processor. In his times even the idea
of a computer as a monitor connected to a console with an input
device such as a keyboard was novel. It transformed the relationship
between people and computers in the fifties and sixties. Englebart
proposed "a way of life in an integrated domain where hunches,
cut-and-try, intangibles, and the human "feel for a situation"
usefully co-exist with powerful concepts, streamlined terminology
and notation, sophisticated methods, and high-powered electronic
aids." Echoes of the contemporary computer interface reverberate
from his speculative way of life. When describing a tool that an
architect could use to describe a working list of specifications and
considerations he wrote: "the lists grow into an ever-more detailed,
interlinked structure, which represents the maturing thought behind
the actual design."
Englebart describes the language of his augmentation medium as "the
way in which the individual parcels out the picture of his world into
concepts that his mind uses to model the world, and the symbols that
he attaches to those concepts and uses in consciously manipulating
the concepts ("thinking")." Process hierarchies represent the
hierarchical approach that humans use to solve problem. He viewed
symbol structuring as a crucial part of language, a new means by
which people could "begin experimenting with compatible sets of
structure forms and processes for human concepts, human symbols, and
machine symbols." The cumbersome acronym H-LAM/T (Human using
Language, Artifacts, Methodology, in which he is Trained) reveals the
connection between human work and machine symbols underlying
Englebart's work. The primacy of language as a structuring device is
apparent and the key idea of associating concepts and names within a
hierarchical structure is a main theme.
Another example of the connection between the conception of media
production and the user interface is apparent in the work of Alan
Kay.[4] He invented the object-oriented language "Smalltalk" and
many innovative features used in contemporary interfaces. Kay's
slogan is "Doing with Images makes Symbols." In this case, he
associates the symbolic with the imperative object oriented
programming language Smalltalk (the mouse is associated with "doing,"
and "images" are associated with icons and windows). Object oriented
programming means that objects know what they can do. In this model
it is possible to select an object and send it a message asking it to
fulfill the user's desires. The paintbrush knows what colors it is
allowed to paint with, all the user has to do is tell it to use one.
The concept of the symbolic is what allows manipulation of objects in
this way. Kay describes the symbolic as that which enables humans to
tie together long chains of reasoning. In this regard, the reasoning
process of the user of the interface is expected to correspond to
this symbolic model embodied in the programming language paradigm.
Working in a realm defined by what the program and its programmer see
as the user's mode of working influences the method by which the user
engages those using the tools. The user of a computational interface
can then ask "where lie romantic notions of art process?" "What of
inspiration and random or spontaneous methods?" "What of
non-hierarchical thinking and procedure?" Certainly conventions of
artistic production have not disappeared in media production
software, in fact artistic paradigms have been as hard-coded into
these tools as programming conventions. Adobe Premiere does not
force users to think in terms of frames and transitions that are
inherited from conventions of cinema or constraints imposed by
hardware in older media technology (such as frames and linear time
progression). Despite the fact that the influence of artistic
production and technology is a two-way street it is useful to examine
the direct influence of software upon artistic production. To
understand the assumptions made by user interface designers it is
informative to look at programming language theory. Then it is
possible to step back and look at how software such as Photoshop or
QuickTime leave their marks upon media production.[5]
III. Formal and Programming Languages
I do not wish to credit myself too greatly. I was a humble
fisherman, and a dabbler in algebra and alchemy, before my discovery
of the lamp. My knowledge of fish far exceeds my knowledge of
unearthly creatures. Still I cautiously praise myself that, in my
elaborate, if cramped, gardens I planned my next wish for months. I
crafted my language perfectly. I would be completely literal and
unambiguous in my next request for the Djinn. I invented a language.
It would specify first the number of requests and subservient wishes
embedded within my one wish. It was not an attempt to greedily
exceed the bound of two more wishes. My desire was only enough
precision so that I would not be thwarted and cursed once again
rather than finally granted boon. This language would translate my
clay human tongue to the language of spirits. My plan had grace and
elegance, it put my knowledge of the algebra to its greatest test.
In the end, I succeeded at that! Why then, reader, do you detect the
disconsolate tone in these words. Because as I prepared to say my
formal, refined, and completely specified wish for a bride that would
inspire clouds to gather and disperse, waves to roll more quickly,
and my heart to infinite joy -- I realized that in uttered word my
wish would take me four thousand and ninety six days to pronounce.
A formal language is defined as a set of finite strings, each made up
of atomic symbols.[6] Formal languages can be denoted by a
mathematical structure called a grammar. The main form in a grammar
is a "production" which describes a rule by which one string in a
language can produce another. Languages are classified according to
the types of productions allowed in them, and this is one way of
characterizing the power of a language. A very simple type of
grammar is defined by regular expressions. Regular expressions allow
for composition of strings in a few very particular ways. These are:
alternation, concatenation, and closure. To make an analogy with
natural language: alternation means choosing one expression from two
lists of expressions in English, composition means joining two
expressions together to form one, and closure means describing
expressions in part (e.g. looking at only expressions that contain
the phrase "media art.")
More complicated formal language systems allow naming of separate
parts of language and specifying the ways in which they are put
together. This is analogous to describing the English language in
terms of paragraphs, words, or letters. It is possible to describe
rules for how these pieces fit together. More powerful formal
language systems allow use of these rules to describe the function
of the language and actually make deductions using the language.
These types of languages are an abstract way to categorize
programming languages.[7] Taking a step back reveals that the
methods of composition used in formal languages are mathematical
descriptions of some processes humans use to think and create
artifacts. Making film consists of concatenation of frames or clips
onto one another. Editing music uses the principle of alternation as
clips are spliced together from different sources. Compositional
elements are often labeled to aid in the process of recombination.
Rules are defined for how these elements should be recombined --
personal rules, rules of convention, and physical rules limited by
the physical means by which the media may be assembled.
In computational media these rules can be automated and implemented
algorithmically. Moreover, it is possible to force compliance to a
particular set of rules to assure that a particular organizational
strategy is followed. It is possible to concatenate two film clips
using any of a variety of dissolves or wipes. One may pick images to
insert in word processing documents as easily as one can cut and
paste words from a variety of different sources. One can label these
individual segments, paste them in layers over one another, perform
searches through them, or substitute one element for another using
convenient thesaurus features. Formal languages provide a concrete
way to talk about these processes that people engage in with texts.
One can go further, however, and look at some of the constructs of
modern imperative programming languages. Nearly all commercial
software is constructed with these languages and the structure of the
languages is directly reflected in the structuring of the software.
Programming languages are designed to fit very particular criteria.
A good language is designed for: abstraction, orthogonality (features
should be free from unexpected interaction), simplicity, regularity,
consistency, and ease of translation.[8] Not every language fits
these criteria, but as abstract goals it can be seen that these
features also carry through to software interfaces. The goals of
regularity and consistency are evident in the means for cutting and
pasting provided in many user-interfaces. The method of selecting
"cut" from a menu is the same as the means for selecting "paste."
This may seem obvious, but another choice could have been to make the
means of cutting analogous to the real world of cutting and the means
of pasting analogous to the real world experience of pasting
resulting in different modes of interaction. Instead, however, the
mindset associated with the design of traditional interfaces has been
influenced by the design of programming languages since the early
days of Douglas Englebart's research in the fifties. Hierarchical
organization and categorization within lists of information are often
the underlying means by which media production software is forced to
structure artistic information. Despite the presence of a "cut"
command, there is no easy analogy using commercial software to the
cutting of a text into pieces, tossing those pieces into a shoebox,
and recombining them at random. There is no organizational feature
in Adobe Photoshop analogous to dumping a pile of photos onto the
floor and running ones hands through them until s/he finds the one
that "calls out to her/him." There are no commands to force a font
to forgo its natural irregularity and take on an expressionist
texturing or to become blurred by teardrops such as is possible when
writing a handwritten letter.
It seems unreasonable to expect such individualized features to be
available when working with computational media, yet it is far from
unreasonable to acknowledge that artists, scientists, and casual
creators of media work in extremely diverse and personalized ways.
It is far easier to achieve a lack of orthogonality when working in
paint (mixing in strange fluids to create unexpected interactions)
than it is in commercial software. It takes an extraordinary amount
of effort to create one's own software production tools that do not
enforce the features of a programming language. The argument being
made here is that these features of formal and programming languages
are not unique to technological media, but their enforcement and
implementation are. The restrictions imposed by these features and
the power of structuring information and media in complex patterns
and manipulating it algorithmically are not easily separable from the
creation of computational media or the experience of being an
audience of it.
IV. Characteristics of Programming Languages
I was happy for a time, I had forsaken my wish for a bride and was
married instead by love, providence, and whatever fortunate design
allowed my simple person to appeal to my mate. My second wish was
for a ring fit for her desires, and somehow divine predestination
allowed this simple request to be fulfilled. My final wish, I
concluded, would be grand. A wish for love, a wish for our future, a
wish for forever, a wish that would make each day as fresh as sipping
from a chalice brimmed with morning dew, fresh as new moments after a
rainshower. So I toil still, to make my perfect language more
concise, to let it be spoken naturally. I must let it be composed as
neatly as a fern's leaves, but it must be as infinitely expressive as
the conch's spiral. It must signify wonderfully like Mowlana
Jalaluddin Rumi's poems.[9] It must embody the clear thought of Abu
Ja'far Muhammad ibn Musa al-Khwarizmi's mathematics.[10] In these
words I realize that it is not the task of one man to create this
language. I do not pretend to be the superior of these thinkers and
feelers that loom across time. I am in their shadows. With the
diligence of women and men across many times, throughout many lands,
and with the proper blessings, I hope one day my dreamed language
will be expressed. Until then, I reserve my love-inspired wish and
ponder more the structure underlying the speech of the beings made of
fire whose words cause action and reshape our earthbound experience
of this world.
Following this account of programming languages and their appearance
within media creation tools it is possible isolate other relevant
characteristics of programming languages. Important characteristics
include: reference, control, abstract data structures, block
structure, and polymorphism. There are direct connections between
each of these concepts and computational manipulation of media.
Reference is the idea that informational elements and their names are
separable within a computer. Naming systems can be introduced to
describe a text in some media and fluidly change names and content
independently. In a trivial example, sections of a story can be
labeled as: "beginning", "middle", "climax", and "ending. " Then it
is possible to swap out the information labeled "climax" with revised
information, thus retaining the story structure. This example is
simplified but not far-fetched. Professional graphic designers use
such features in software regularly, as do professional sound
technicians.
Control means that it is possible to structure the flow of
information in a system. An example of a control structure in a
programming language is a loop. In both music and video processing
software artists can create explicit media loops. In a great deal of
modern popularized music computational media are used to create drum
loops as a basic unit of music composition. The video and
installation artist Bill Viola is vitally aware of this as he states
that "viewing becomes exploring a territory, traveling through data
space."[11] Control structures represent the means used to shape
these paths through media data space. Bill Viola continues his
description of computational media manipulation asserting that
"editing will become the writing of a software program that will
tell the computer how to arrange (i.e., shot order, cuts, dissolves,
wipes, etc.)"
Abstract data structures are the means by which data spaces are
organized. Lists and arrays of information are basic compound data
types, but one can combine base data types in any variety of abstract
ways to represent current needs. Circles become numbers: integer
coordinates for the center point and a floating point radius, dates
become several fields of integers. Basic programming structures
became revolutionary structures when introduced explicitly to media
art. This can be seen in work such as Luc Courchesne's "Family
Portrait" which uses laserdiscs indexed by HyperCard stacks to allow
users to respond to questions asked by members of a family to create
startling sensitive interactions. When describing ways to construct
video artwork, Bill Viola refers to "visual diagrams of data
structures already being used to describe the patterns of information
on the computer video disc." The medium of a videodisc can be seen
as merely a digital update of former videotape media, but when looked
at for the actual way in which the information is structured a quite
distinct nature is revealed. The structure of the computer
programming language has been used to restructure the information of
video data. Data types such as matrices, linked lists, and records
have become the status quo for the organization of media in digital
formats.
Block structuring allows for hierarchical organization of computer
programs. It also allows each block level to be manipulated
separately. This structuring is pervasive throughout media
production software. Text can change font or color at the letter
level, word level, sentence level, paragraph level, page level, or
document level with precision and ease. Photographs can change
onionskin layer at a time, affecting lower levels but not higher
levels. These levels are nested into hierarchies with ease and can
be imported into other such hierarchies at will. Whereas the nature
of a photographic manipulation could be seen as based in continuous
fluid and light when done using traditional darkroom techniques,
professionals using media software often work using block structures.
The software is structured in such a way as to discourage and
inhibit other methods of use.
Polymorphism is the idea that data or functionality can change
depending upon its context. The extension of this idea is that
structures for manipulating one data type can manipulate data of
another type. It is possible to use a jpeg image just as easily
within a word processing document as it is to composite the image in
a photographic document. The data here is not actually polymorphic,
but the concept of using the data in such a variety of settings is.
Cutting and pasting a graph is done using the same mechanism as
cutting and pasting a paragraph. The same "play" button on a
Quicktime player starts playing an audio file the same way as it
starts a video file. This software exhibits the traits of
polymorphism. This type of media fluidity, unknown in previous
media, is becoming an expected trait of the experience of
computational media.
It is certainly possible to look at various models of programming
languages in order to elicit more parallels in computational media.
Those listed here represent some of the most important
characteristics of computational media that are often enforced by
software. The idea behind such programming languages extended far
beyond computer science and in some sense are general concepts to
organize information. The concentration of all of these modes of
operation and ways of thinking, their rigorous enforcement by a
machine, and the compliance of media artists to their tough
strictures (by default of the tools they work with) is something new.
IV. Conclusion
The havoc unleashed by Djinni's granting ill-considered wishes by
humans reveals the Djinni's lack of concern for the environment in
which s/he is operating. The considerate Djinn probably could
understand that the human languages cannot properly express wish
fulfilling words. Likewise, humans should realize that computational
languages have effects upon what we create using them. This essay is
meant to use accounts from theory programming languages to reveal a
glimpse into the computational medium and the ways in which a
meta-medium is not a consolidation of previous media but has its own
recognizable traits and languages. No creation by any artist can
escape this chain and transcend the nature of its medium in a
material sense. The expressive, analytical, evocative, or otherwise
subjective interpretation or intent of the work can certainly
transcend the computer, but in a concrete material sense the mark of
the programming language as a primary characteristic of computational
media is always evident.
Notes:
------
[1] Nam June Paik. "Cybernated Art," Manifestos, originally published
in _Great Bear Pamphlets_ (New York: Something Else Press, 1966).
Reprinted in _Theories and Documents of Contemporary Art: A
Sourcebook of Artist's Writings_, Berkeley, CA: University of
California Press, 1996) and _Multimedia: from Wagner to virtual
reality_, edited by Randall Packer & Ken Jordan, New York: W.W.
Norton and & Company, Inc., 2001. p. 39-43.
[2] Douglas Englebart. "Augmenting Human Intellect," originally
published in "The Augmentation Papers", Bootstrap Institute, 1962.
Reprinted in _Multimedia: from Wagner to virtual reality_, edited by
Randall Packer & Ken Jordan, New York: W.W. Norton and & Company,
Inc., 2001. p. 64-90.
[3] ONLine System (NLS). developed at the Augmentation Research
Center of the Stanford Research Institute and unveiled in 1968.
[4] Alan Kay. "User Interface: A Personal View," _The Art of
Human-Computer Interface Design_, edited by Brenda Laurel (Reading,
MA: Addison-Wesley Publishing Company, 1989. Reprinted in
_Multimedia: from Wagner to virtual reality_, edited by Randall
Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc.,
2001. p.121-31.
[5] Lev Manovich. _The Language of New Media_, Cambridge, MA: The MIT
Press, 2001. p. 140-41.
[6] Noam Chomsky. _Syntactic Structures_, The Hague: Mouton, 1957.
[7] Ryan Stansifer. _The Study of Programming Languages_. Upper
Saddle River, NJ: Prentice Hall, Inc. 1995. Chapter 2.
[8] Ibid. p. 5.
[9] Mowlana Jalaluddin Rumi is recognized internationally as one of
the world's great literary figures.
[10] Abu Ja'far Muhammad ibn Musa al-Khwarizmi is generally
recognized as the "father of algebra."
[11] Bill Viola. "Will There Be Condominiums in Data Space,"
originally published in _Video 80_, no. 5 (Fall 1982). Reprinted in
_Multimedia: from Wagner to virtual reality_, edited by Randall
Packer & Ken Jordan, New York: W.W. Norton and & Company, Inc.,
2001. p. 287-98.
--------------------
Fox Harrell is currently pursuing a Ph.D. in Computer Science at the
University of California, San Diego. His research focuses on
developing new improvisational narrative forms. He earned an M.P.S.
in Interactive Telecommunications at New York University's Tisch
School of the Arts, and both a B.F.A. in Art and a B.S. in Logic and
Computation at Carnegie Mellon University. He has worked as a game
designer and animation producer in New York City. He recently
completed his first novel, "Milk Pudding Flavored with Rose Water,
Blood Pudding Flavored by the Sea."
_____________________________________________________________________
* CTHEORY is an international journal of theory, technology and
* culture. Articles, interviews, and key book reviews in
* contemporary discourse are published weekly as well as
* theorisations of major "event-scenes" in the mediascape.
*
* Editors: Arthur and Marilouise Kroker
*
* Editorial Board: Jean Baudrillard (Paris), Paul Virilio (Paris),
* Bruce Sterling (Austin), R.U. Sirius (San Francisco), Siegfried
* Zielinski (Koeln), Stelarc (Melbourne), Richard Kadrey (San
* Francisco), DJ Spooky [Paul D. Miller] (NYC), Timothy Murray
* (Ithaca/Cornell), Lynn Hershman Leeson (San Francisco), Stephen
* Pfohl (Boston), Andrew Ross (NYC), David Cook (Toronto), Ralph
* Melcher (Sante Fe), Shannon Bell (Toronto), Gad Horowitz
* (Toronto), Deena Weinstein (Chicago), Michael Weinstein
* (Chicago), Andrew Wernick (Peterborough).
*
* In Memory: Kathy Acker
*
* Editorial Correspondents: Ken Hollings (UK),
* Maurice Charland (Canada) Steve Gibson (Canada/Sweden).
*
* Editorial Assistant: Ted Hiebert
* WWW Design & Technical Advisor: Spencer Saunders (CTHEORY.NET)
* WWW Engineer Emeritus: Carl Steadman
_____________________________________________________________________
To view CTHEORY online please visit:
http://www.ctheory.net/
To view CTHEORY MULTIMEDIA online please visit:
http://ctheorymultimedia.cornell.edu/
_____________________________________________________________________
* CTHEORY includes:
*
* 1. Electronic reviews of key books in contemporary theory.
*
* 2. Electronic articles on theory, technology and culture.
*
* 3. Event-scenes in politics, culture and the mediascape.
*
* 4. Interviews with significant theorists, artists, and writers.
*
* 5. Multimedia theme issues and projects.
*
*
* The Editors would like the thank the University of Victoria for
* financial and intellectual support of CTheory. In particular,
* the Editors would like to thank the Dean of Social Sciences,
* Dr. John Schofield and the Dean of Engineering, Dr. D. Michael
* Miller.
*
* No commercial use of CTHEORY articles without permission.
*
* Mailing address: CTHEORY, University of Victoria, PO Box 3050,
* Victoria, BC, Canada, V8W 3P5.
*
* Full text and microform versions are available from UMI, Ann Arbor,
* Michigan; and Canadian Periodical Index/Gale Canada, Toronto.
*
* Indexed in: International Political Science Abstracts/
* Documentation politique international; Sociological Abstract
* Inc.; Advance Bibliography of Contents: Political Science and
* Government; Canadian Periodical Index; Film and Literature Index.
_____________________________________________________________________
_______________________________________________
ctheory mailing list
[log in to unmask]
http://lists.uvic.ca/mailman/listinfo/ctheory
************************************************************************************
Distributed through Cyber-Society-Live [CSL]: CSL is a moderated discussion
list made up of people who are interested in the interdisciplinary academic
study of Cyber Society in all its manifestations.To join the list please visit:
http://www.jiscmail.ac.uk/lists/cyber-society-live.html
*************************************************************************************
|
