Research
Proposal

Main menu

Design

Project: Four stages of complexity, each of which have systems that are relatively simple to understand

User Interaction

The only interface is a tablet. User has access to mouseX mouseY, clicking, pressure sensitivity, and tilt. These are used only to create agents that exist on-screen.

Agents

These agents should be abstract, look and behave playfully. They exist because the user caused them to exist. Their form is based on the strokes made by the user, and their behavior is also dictated by that stroke. Once alive, they will actively perform the construction of neural network nodes, maintaining them, modifying them, and cultivating them like a farm. This is important to the musical aspect, because it allows for automatic modification of the music over time.

The agents might also interact with each other.

Structure

The structures created and maintained by the agents behave similar to a neural network. Each node in the network has a fixed time before it can fire. The nodes fire signals to each other. This event causes sounds to be played. The sound played is determined by the information contained in the node, which is set by the agent. Loops are allowed to form between the nodes, causing continuous sounds.

The information passed is also dynamic. Information can also be passed by the nodes to signal volume change, tempo change, etc.

Sounds (Music?)

Several important things to note about how to make music sound good:

• all pitch shifts should agree on one or two musical modes, such as minor, major, D-Dorian, etc
• rhythm should be aligned to a certain beat (in bpm, or a musical tempo such as 4/4)
• layering should be emphasized, as opposed to "overlapping sounds"
• style/behavior of song? how can these be created in this paradigm

Feedback

The resulting sounds can then be read back as data, which can alter the behavior of the agents. Perhaps they will dance to the music, causing them to do different things to the structure.

The user might also be surprised by the music created, and would create more agents based on what they hear as a level of feedback.

Explanation of Variable Map (large version)

This map represents what inputs can be mapped, what internal variables will be manipulated, and what outputs are to be expected. Note that there is no linear one-to-one mapping, and that many variables can influence many other variables down the line.

The purpose is to outline how the user has an indirect control over what music is being created, and that the music will also influence itself. The user's input only causes the first layer of things to emerge, which causes a chain-reaction of events to occur, causing music to be produced.

Drawing Organisms

Organisms (agents) are created by user interaction with the tablet. Each stroke, pressure, length are recorded. If a certain length of time has passed, the strokes will produce one organism based on the information saved. The physical traits and behavior of this organism will be derived from that drawing information.

Visually, the organisms have several possible ways they can look. Here are some ideas to explore

• robots
• organic slugs
• dragons/fantasty creatures
• abstracted forms
• CA cells (boring..)

The way they move are also based on the drawing information, primarily the speed/tempo/curvature in which it is drawn. They will give the organism characteristics that define them from others. Here are some examples of how they could possibly move:

watch video

The purpose of these organisms are to create network structures.

Crystal Networks

These structures, constructed and maintained by the organisms, make sounds. The sound they make is dependent on what organism made it, and how they modified/manipulated it. An organism is able to change the pitch, volume, attack, and other qualities of a crystal node. They can also activate it.

By activating a node, it will send a pulse to all connected nodes. This acts like a neural network, causing other nodes to fire, perhaps even making loops of firing nodes. Everytime a node is activated (fired), it produces a sound that's defined by the crystal (which is previously established by the organism).

This sound loop reaction is much like the music-toy example, built in 2002.

Each node also has qualities that behave like a neural network. For example, each node has polarity, as well as a set signal strength, and a refractory period. These allow the nodes to interact like a parallel computer, having AND and XOR operators.

Visually, the nodes can look like any number of things, but must hint at their connectiveness:

• crystals
• abstract ball/stick diagrams (boring!)
• neurons
• ??

Sounds

Each sound played by a crystal node can be either generated (sine, saw, and other operators), or by using an mp3 or wav sample. Audio will be using the new Ess Sound Library by Krister Olsson. The sounds can be manipulated by

• pitch shift
• volume/cut/resonance
• attack
• holding a note
• VSTi filters

For technical purposes, each sound might have to be registered onto a timeline, and played in sequence to sound "on tempo". This may or may not be implemented.

Melody

Since I don't want my software to simply be a beat-box device, melodic elements are vital to making this project successful. Worst comes to worst, this project becomes a beat-box device. Let's try and avoid this!

Melody is described as pitch change over time, or a "successive arrangement of sounds".

• A rhythmically organized sequence of single tones so related to one another as to make up a particular phrase or idea.
• Structure with respect to the arrangement of single notes in succession.
• The leading part or the air in a composition with accompaniment.

Melody will probably be established by a few rules. First and foremost, a key must be chosen and followed. Second, a tempo should be set, and subdivided accordingly.

A few rules will be written to prevent "unpleasant" exchanges and or combinations notes to be played.

In essence, there will be a selection of unwanted pitches to be removed. Organisms will probably be employed to remove them, or modify them.

Keys

These will be based on a few rules to disallow certain pitches to be used due to the existence of current pitches already playing. What might occur is that known keys will naturally emerge, like C, or Dorian.

Tempo

The tempo should emerge from the network causing the sounds to be played. However, they should happen on a chosen, logical tempo like 4/4.

Article on Microstructures, Motifs, Themes, and Generative Melody

(this part obviously needs additional work and research as project progresses)

Orchestration

When I mean orchestration, I mean the interaction between the multiple melodic and rhythmic structures. As multiple crystal networks are created, and sometimes seperated by distances, a call and response effect could happen. This also allows for multiple melodies and rhythms running alongside each other.