computer interface as theater


We are at the mercy of our senses; what we perceive we call the world ("nature") but it's all a construct, and often outright fiction. Of course all of us having more or less the same senses the worldview each of us construct is similar, assumptions we make about the world therefore overlap, and we call this consensus reality. Occasionally we trip over cracks in the world, but most often we pick ourselves up, proceed as nothing unusual has happened, and quickly forget the anomalies.

In art, we sometimes exploit or explore these cracks in the world. In a computer, the world we experience does not exist; the only access a computer has is through sensors we attach to it. Within a computer, the world can ve viewed only through tiny cracks in the seamlessness of mathematical logic that our stored-program digital computers are made of.

Computers cannot even achieve "stupid". They are procedural machineries only. They cannot understand anything, but they are very very good at following complex rules of logic at our command. Unfortunately people are not good at being rigorous writing (or following) complex rules of logic, hence we have to use computer "languages" to manage the complexity and write programs that do not work first time, etc.

Sensors are physical devices that translate some phenomena or energy or something into a signal that can be "read" by a computer; a state (on or off) or a quantity (light from dark to bright). As humans we can understand that "one side" of a sensor responds to light (photons strike a small bit of cadmium sulphide and free up electrons) and the "other side" produces a change in electrical behavior that when attached to a computer through appropriate junk (aka an interface) the computer sees the electrical behavior -- only.

(There are also things you can call actuators, that do something; motors move, speakers make sounds, but those are much easier to understand once you grasp sensing. Actuators are ignored, for now.)

Take for example a simple light sensor. In essence, from the computer's point of view, it's output is a quantity, in our (common) example a number from 0 to 1023, where 0 may mean very little/no light, and 1023 means "bright" light.

Here's the thing: you and i as humans know that this relation is "amount of light". The meaning is in our heads, only, and in our software, when we arrange for our software to "assume" that this quantity "means" light. There is no truth here, only logic, and meaning is a human thing mapped by us intentionally into our machines, if at all.

If instead of a light sensor, you attached a potentiometer (the "VOLUME" knob on a radio is a potentiometer, aka "pot"), any running program looking at that sensor would be unable to tell, since it's entire knowledge of that particular tiny world is a quantity that varies from 0 to 1023.

I'm emphasizing this apparent lack of meaning because it's the key to thinking about what "sensing the world with a computer" means.

Sensors as simple as a dark-to-bright are obviously easily fooled (substitute a pot, or shine a flashlight on a "daylight" sensor). But even complex sensors cannot "see" reality; full-color high resolution cameras with sophisticated software that can identify individual faces is trivially fooled by holding up a photo to a camera. To people observing this prank the subterfuge is obvious immediately; why? Because we have vastly more sense-ability that provide reduncancy in sensing (understanding that a hand is holding a photograph, etc) but mainly we can *change contexts* and look at the situation not just as the narrow view of camera-seeing-human-face but stand back, literally and figuratively, at the larger situation of camera, person, room, photograph, software, etc.


The paragraphs above are trying to get across to you the idea that sensing does not convey meaning, nor really tell you anything about what you might *do* with the information handed to you.

The closest practical analogy to interfacing a computer to "the world" (sic) that i can think of is theater. When you attend a play, you arrive with a general understanding of the situation and context. There is literally a boundary between you, the audience, and the stage containing actors and props. You realize there is a script, but it's entirety is hidden from view; you experience it line by line as the play unfolds. There is (usually) a narrative which unfolds in sequence; the actors scripted moves and utterances, the props, creating an experience, each step of which provides a context for the next. Most importantly, you see a fictional narrative world created by the actors, props, and hidden script, within the constraint of the stage.

Behind the scenes -- literally -- everything is utterly, completely different. The beautiful mountain background prop is painted on a board and held up with sticks and rope; the handsome main character has a bright light in their face, and looking around them to stay synchronized with other actors and possibly moving props. There are back-stage people manipulating lights and objects to create the fictional reality so seamlessly viewed by the audience. And finally, scripts are substantial works, produced elsewhere and elsewhen at great effort, choreographing all of the on-stage actors and props. And for the audience's part, not to be overlooked, is, except for the best illusions that actually fool human senses, a willing suspension of disbelief at what they see; a magic spell easily broken by bad acting or bad scripts as we all know.

Theater people must easily see both sides of the play; from the audience and from the view back-stage. And backstage, there are of course multiple viewpoints; actors each with their lines, lighting, props, all contribute and each needs individual design and coordination.

All interactive computer software, art or otherwise, is essentially a theatrical production, real-time interactive works more so. In my experience the hardest part to learn when developing art projects involving any sort of computer -- laptop, desktop, Arduino controller -- is this flipping of "ground" from audience to script/backstage.

First and foremost, you must define the audience -- "user" -- experience. You cannot script a performance if you cannot state or demonstrate what the narrative is! And this part of the design truly has nothing to do with computers, sensors or anything technological.

A computer-reactive sculpture's (re)actions to human actions can be played out with cardboard props on a table. A reactive room installation can be walked through -- literally -- with tape or passive objects marking significant locations, and notes or a third party making sounds/shining lights/playing music/whatever at appropriate times and places.

Even a low-resolution, sloppy approximation of your aesthetic goal will often reveal fatal flaws in thinking -- contradictory requirements, impossible time scales... how long does it take to walk across the room? What happens when one, two, three, many? people crowd the room? Does a stranger unfamiliar with your project "get it" when they walk through?


Back to the light sensor attached to your computer. As stated, the sensor is stupid, light and dark as a quantity, devoid of any specific meaning. What can it mean?

You create the meaning! Aimed out the window, the sensor tells you maybe daytime versus night time. Placed in a room, about 36" off the floor, in the left side of a door way aimed at the right side, it might tell you when a person walks through that door way. How? If all of the walls are painted white, as is often the case, left to itself the sensor "sees" "bright". When a person walks through the doorway, their probably-darker clothing occude the light, the sensor sees "darker". Now if a script in the computer is arranged to "look" at the sensor 10 times a second, and there is no one in the room, it sees:

light, light, light, light, .... light, light, ...

ad nauseum. "Light" is not interesting, here. It's a computer, it's stupid, it has no problem doing this 10 times a second, for ever. Now imagine a person walks through the door way, momentarily blocking the sensor's view of the wall:

light, light, light, light, dark, dark, dark, .... dark, light, light, light, ....

Dark isn't interesting either; what is interesting is change! Light-dark-light means something happened, because we arranged things to be so.

Meaning is created by context. The sensor, and the computer, by themselves know nor mean anything. They are just active tools. You still need to learn computer programming and such, but this technically-mediated sensing, scripting, and subsequent action from the result is key.