Group members:
Michelle Hua
Karen Cochrane
Kelvin Hu
Benson Wong


Our goal is to create a piece that brings people closer together socially. It is clear that the world’s population is increasing drastically and with that, cultures become much more dense. This usually means that language barriers will appear and create problems in communication between different cultures. We hope to create a piece that will teach people a global language in a fun way by incorporating the Microsoft Kinect. We are trying to create a communication network by using an original language, processing and the kinect. The piece will include participants who will try and communicate with each other by using the language we provide. This piece will require two participants in different locations with the program open. By using the network program, the individuals will be able to communicate with our gestural language that breaks all cultural barriers. Gestures will be performed by the users in front of the kinect camera and the program will analyze the gestures, determining if it is readable.

Project Description:

Processing is used to code the the network and the functions of the kinect. Situating two participants in two different locations emphasizes how this is a networked communication. At this stage, the program can only read the movements of one object (In this case, one hand). There will be five markings on the screen that shows where the participants hand should move through when making gestures. The outcome of the participant is determined by a switch/case in the processing program.What will be printed on the screen depends on where and in what order the hand passes through. The gestures created for this piece are relatively simple only because of the restrictions of the six markings. We hope to offer much more freedom in terms of movement in the future by expanding on the Processing program.


As students working in the field of new media and innovative technology, we wanted to create a communicative program that allows for two users to speak with one another via a unique gestural language. Understanding the way that humans communicated with one another for these past centuries is a mysterious, yet crucial aspect of our culture that explains our past origins. Assuming that human beings have evolved from ape-like creatures, studying their gestural ways of communications can help us trace back to where languages could have evolved from. Even further back in history, cave drawings and paintings can be analyzed by looking at how symbols and patterns are used to depict images and concepts. By researching how these other languages helped their users communicate, we can develop a simple and useful gestural language.

Essentially we are coding our own language to track movements with the Microsoft Kinect utilizing infrared sensors. We are creating an environment for two people to communicate, even if there are language barriers due to being situated in two different, distant locations. The only aspects needed to successfully communicate through the network are, 2 Kinects, computer/mobile screens and 2 participants (1 per Kinect). This mode of communication engages the user as they need to use their limbs in a playful manner, hopefully even educating them with current technology and its possible uses.


We can only track one hand from the program so we need to change the gestures a bit. We are going to have markings(the circles) on the screen that shows where the participant’s hand should move through when making gestures. We are going to make switch/case in the program. What will be printed on the screen depends on where and in what order the hand passes through. Let’s say the hello gesture is a clockwise motion, the hand has to move from the bottom left circle, up to the top and down to the right. Through the use of switch/case, if the circles are touched in that order, it will print out “Hello”.

(Our final working piece only has 5 circles)

– Amount of circle depends on our gestures.

New set of gestures:

Hello: Starting at left center circle, move hand clockwise to the top center circle

Goodbye: Starting at left right center circle, move hand counter clockwise to the top center circle.

Please: Starting at upper left circle then top right circle then upper left circle

Thank you: start left center circle then right center then shake hand gesture.

How are you? upper left circle then upper right circle

(when read from top to bottom, shows how our ideas have evolved and changed)

Kinect – Karen has one
(make sure it can detect with the library for Processing)
Will need a legend/dictionary

https://github.com/Sensebloom/OSCeleton – program we can use to track a skeleton in Kinetic then we can use Processing to read the information

– Create own unique language for communication
– Using Kinect or webcam
– Colour detection
– Need the library

– Morse code – lights
– Contemporary dance – exaggeration
– Flags – semaphore

– Gmail Motion – Google’s April Fool’s Day

– Sign language
– Semiotics


So You Think You Can Dance Woodpecker Dance

Thoughts: Over the top motions, very interesting – kind of want to use over the top motions to create our piece

Interesting video on body gestures throughout different cultures.

Nonverbal Communication Web Project – Cultural Gestures\


This is just a summary of various gestures from each respective country that is listed. I remember we were talking about how we wanted to create a language that would break all barriers, so this may be helpful in understanding the different cultural gestures.

“…some visual communication remains to be understood amongst the diverse populations around the globe. The latter is defined by anything consisting of eye contact, facial expressions, patterns of touch, gestures, spatial arrangements, tones of voice, expressive movement, and other cultural differences.”

(Kelvin and Michelle)

Notes on creating our language…

– Keep in mind that there are gestures that have various meanings across the different cultures

Ex. The “OK” gesture made with the hand means “A-OK” to us, while it means “F*** you” in Brazilian culture

The peace sign forwards means “Peace” and backwards iit means “Vagina” in European culture

– Create gestures that don’t already have a meaning – make them a combination of gestures
– Be specific and exaggerate them so they are not to be confused with any other gestures of our language
– Using points of the body and limbs

What kind of communication:
– program needs to see each separate step to distinguish which gesture is being acted out
(some sayings use similar gestures as part of communicating them, but can always change it up)

Basics (how to get by)
Hello – Two hands (palms out) starting in front of face, make large outward circles

Goodbye – Two hands (palms out )starting in front of face, make large inward circles

Washroom – Squat with hands on thighs, and use right hand to do a wiping gesture to the right side of body (front to back)

Food – Rub belly and feed face (will need to specify which hands)

Please – One hand (right) on head with elbow out to the side, bring other arm up and place hand on top of other hand (left) also with elbow out (can we make him squat once? it’s funnier that way)

Thank you – Arms stretched out the side, bring arms together to point at the person being thanked

Going – Swing arms at your side very drastically, front to back, make sure elbows are bent

Where is? – Hands horizontally on forehead/above eyebrows, elbows out, look side-to-side

Advanced (concepts)
How are you? -right arm bicep curl then punch motion toward the person then stick out the pinky.
I don’t know – rotate your head one full circle clockwise(<–lol I don’t think you’ll live from that) then put both hands on top of head
Would you like to go out? – First, Spread arms apart to T shape then slowly bringing them closer together to hold hands position then raise and lower joined hands two times then drop left hand right hand points at the person you are asking
The universal swear word- Do a squat then while lifting one leg,  put out Left hand middle finger while rotating your entire right arm in circles like a fool.

Side notes

-To create a project that uses body movement to communicate through a network  (processing + Kinect)
-To develop original, creative body gestures for communication (essentially creating our own language) -keeping it simple for now
-We want to create a universal language that would not offend cultures- after some research we encountered many gestures that have completely opposite meanings in different countries
For example: The “okay” hand gesture in America means “Go Fuck Yourself” in Brazil.
– Exploring the concept of unifying loose ends of various cultures by utilizing the advanced technology available to us. Possibilities are endless as we have access to tech. like Kinect and processing – advantage.
-This can be beneficial to society as it can encourage communication with those suffering from permanent disabilities, language barriers etc


LANGUAGE: Cavemen, Apes, Sign Language, touch on other forms of language we have looked into and used for inspiration (up top)

Some things to look at..

Title: Ape gestures and language evolution

  1. Amy S. Pollick and
  2. Frans B. M. de Waal *

An article reminding us that primates were communicating before humans. It is a case study of observing gestures from bonobos and chimps. We can look at what signals and gestures these monkeys use and incorporate them into our project. Monkeys usually communicate with their limbs anyway so it will make sense since kinect can read limbs much clearer than detailed bits like fingers etc.

Sign language in the wild

A study by the researchers from the University of Stirling found 30 different manual gesture types in wild chimpanzees at Budongo Conservation Field Station in Uganda. Many of these gesture types appear similar to human manual gestures such as arm beckon, point, clap and flail. A parallel study by researchers from the University of St Andrews, focused on all gesture types including bodily movements and locomotory gaits and concluded that this wild group make use of at least 66 distinct gestures to facilitate communication.[19]

– Scientists thought the origins of human language stemmed from the vocal cords, but latest evidence shows that our hands are actually the source
– Many institutes and universities conduct research on our four closest evolutionary ancestors: chimps, bonobos, gorillas and orangutans

– These 4 in particular have a complex system of waving and gestures “in the first twenty months of life” – ranging from poking to get other apes attention or shaking their heads and extending their arms outward

– Different than monkeys because they only develop gestures as adults

University of Auckland researcher Michael Corballis adds:

“In monkeys, intentional arm movements are dedicated mainly to grasping. Communicative gestures probably emerged in apes, and began to assume grammatical forms in hominins.”

– Though there are crucial differences to how humans and apes communicate…

– Children at 14 mths old from various cultures all pointed with their index finger – all for the same basic reasons

– While apes didn’t show any common meanings for similar gestures

– Only commonality was that the apes used hand gestures from a very young age

“Yo dawg I heard you stole my banana”

Ape Gestures and Language Evolution
Gestures – communication by means of feet, hands or limbs
– Unique from other bodily communication because its neurologically distinct in both production and perception by other communicating with
– All primates use vocalizations, facial movements, body gestures, and free brachiomanual gestures (communication without touching another individual)
– Typical of humans and apes
– To figure out the meaning of a gesture, need to know which context the gesture is being used in
– Same gesture can mean a various meanings depending on the context
– Failure to teach the apes vocal modifications as to speak – but had no trouble learning the American Sign Language
– Humans evolved with a differential growth of the brain and vocal apparatuses

“Gestures are rarely produced in the absence of other communicative signals, such as facial expressions and vocalizations.”

(ours would be with the aid of words)

bringing relationships closer with wildlife? – Kanzi THE Ape !

Did Stone Age cavemen talk to each other in symbols?
– Paleolithic drawings from the caves of Pech Merle, Font-de-Gaume and Rouffignac in southern France
– Images of animals, as well as shapes and forms
– Rouffignac dates back to at least 13,000 years ago, Chauvet and Lascaux – 30,000 years plus
– Created by using ochre (earth derived pigments – clay, ash, etc) and charcoal mixed with fat and “spittle” – saliva
– People have always needed a way to communicate and depict the world around themselves
– Photographers have tried to decipher the symbols by looking at patterns of the symbols
– Brings upon questions such as; Who developed the language and first used them?
– Was it developed after Europeans arrived or even older lineage?
– Some anthropologists argue that a “Great Leap Forward” occurred – an abrupt change in intellect and brain structures
– Others have argued that the latter is just “Eurocentric nonsense” and that clues such as flint points, and bright ochre colours found in jewelry and art can be traced back even further to 70,000 – 90, 000 years
– From African homelands
– Very intelligent to have made drawings within a cave the preserve the paint

A website showing many cultural gestures. Interesting read but more importantly, it’s useful for what we can and not use for our language.

Helpful too

Shows the contrast of similar gestures between cultures. Even the comments from the website has some good info

Evolution of sign language

A short article on the pros and cons of sign language


– To create a language to break down language barriers.
– Different cultures have different languages so it makes it hard for different groups of people to communicate.
– Sign language is a generic, global, already existing type of communication. We want to use this concept of using gestures to communicate so people all ages (even toddlers) can easily use to converse.
– We also want to incorporate the gifts of modern technology by using the revolutionary microsoft kinect as it allows us to create such a system.
– The kinect is able to track and map specific parts of the human body so it makes deciphering gestures possible.
– This can also be a learning tool as the kinect is widely available to the general public. Could be fun and educational at the same time.
– The power of the kinect shows great potential but we simplified our gestures using the primary, bigger limbs and joints as we think the kinect will come across some trouble if it tries to track the joints of our fingers for example.
– The kinect also allows the possibility detecting depth and height so it actually provides more choice and possibilities to create a language other than just using your hands, like sign language.

– We mapped the kinect so that it will track the user according to a general height we determined.
– People with different heights will vary but most people in our class are similar in height.
– We hope to overcome this issue if we continue to pursue with this project.

– To quote marshal McLuhan, the medium is the message:
– Our message in terms of using this medium is to unify the communication aspect of society.

– This project offers a playful aspect  (people of all ages)
– Also beneficial for people with disabilities

– Using gestures to represent meanings rather than actually recreating an entire alphabet makes it much easier to learn.
– Keeping it simple and clean
– Mixed meanings in gestures will be obsolete

– Incorporating the word on screen feature gives the user a translation. (clearer and used to not confuse the users)
– Must perform the gesture properly in order for the word to appear on screen.
– Practice makes perfect dawgs

– Device is mobile and can be used to communicate to other computers in other locations
– Uses infrared technology which doesn’t have limitations in terms of what colors the users are wearing
– This project is created for one user at a time and is not meant for 2 or more.
– Doing that will cause confusion with the sensors in the program and will not allow the kinect to track the preset gestures

– Can imagine using this device/program in locations where sound is not permitted I.E Libraries etc
– Imagine the possibilities! Military uses etc

– For our presentation we should perform once and ask volunteers to experience this project.
– Also, before we present it, we should ask some of our friends to try it to see if its too hard.
– If it is than we should work on it to simplify it. (utilize the Trial and Error method + experimentation).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s