Introduction to MIDI and Computer Music: Assignment 4

Assignment 4: Combining MIDI and Audio

What we're trying to do...

Incorporate sound files into a Digital Performer sequence.
Understand the difference between destructive and non-destructive audio editing.
Learn some basic audio-editing techniques in Digital Performer.
Learn how to time-scale and pitch-shift audio.
Add real-time effects to audio tracks.
Combine MIDI tracks with audio tracks.
Make a creative, musical collage of spoken sounds, accompanied by MIDI tracks — at least 90 seconds in length.

Where to get the source sound files...

The source sound files — poetry readings — from this web page.

What to turn in...

One project folder containing
1. a Digital Performer sequence file, and
2. an "Audio Files" folder that contains the sound files used in the sequence.
(Your project folder will include other folders created by Digital Performer, such as "Fades" and "Analysis Files." Be sure to keep these folders; don't move them.)
Name the project folder "assign 4," and copy it into your own folder on the Music Server.
Verify that you've satisfied all the requirements by consulting the Assignment 4 checklist.

Some Background

The sound files you use in a Digital Performer sequence can come from anywhere. For this assignment, you must use a sound file I provide for you. As long as you've made use of that file, you may include other sound files.

Sound files can have different formats. Files from the Windows world tend to be in Wave (.wav) format. The standard format in the Mac world is AIFF. But either format can work on either Mac or Windows computers. It's easy to convert files from one format to another, though, using a sound editor program.

Sound files, of course, store data in digital form, using ones and zeros. For some background on the process that turns a sound wave into a stream of numbers, see Digital Audio Concepts.

Digital Performer specializes in non-destructive editing of audio. This means that in most cases, it will never alter your original sound file. Instead, it either creates new sound files based on the original, or it lets you work with soundbites, which are references to segments of the original sound file. Other programs, such as Peak, offer destructive editing of audio. Operations you perform in such programs can alter the original sound file. Both methods of editing have their place, and many programs, including Digital Performer, let you use both. When you have a choice, non-destructive editing is usually preferable.

Musical Considerations

Your sequence must last at least 90 seconds. The sound file I give you to work with contains a reading of a poem. You must perform extensive edits of this sound, cutting and pasting it to create a collage of spoken words. Try to make a vivid musical "statement." You need not preserve the intelligibility of the text, if you don't want to. Consider that the reading of the poem is already very musical, in the sense that the speech has strong rhythmic and pitch features.

I expect you to tell a musical "story" or convey a definite mood using sound. This assignment is not designed merely to exercise your technical knowledge of the software, though the descriptions below may leave you with that impression. It's an opportunity to be creative.

Placing a Sound File into a Track

The steps below tell you to make a new Digital Performer project, obtain the required sound file and import it into the project. You can use these instructions to import other sound files later. The sound files must be in Wave, AIFF or Sound Designer II format. If one is in some other format, such as MP3, you can open it in Audacity and save it as a Wave file using File > Export as WAV.

After you import a sound file into Digital Performer, it becomes a soundbite. (For more on soundbites, see below.) You drag soundbites into audio tracks to build a sequence. You can drop a soundbite into your sequence any number of times. Each instance of a soundbite behaves somewhat like a MIDI note.

Before working with Digital Performer audio, you should turn the Mac volume up to the maximum, as you did for Reason. Use the speaker keys on the Mac keyboard or the speaker icon near the clock in the menu bar.
Create a new Digital Performer project, saving it on the Desktop. Please name this project "assign 4."
A new sequence file already has some empty audio tracks. The tracks are either stereo (for 2-channel audio) or mono. The number of squiggles next to the track name tells you which (1 for mono, 2 for stereo).

You can use stereo soundbites only in stereo audio tracks, and mono soundbites only in mono audio tracks. If you need more tracks, create them by choosing Mono Audio Track or Stereo Audio Track from the Project > Add Track menu.
The easiest way to get a sound file into your project is to drag it from the Desktop into an appropriate audio track.

When you see the light blue rectange, release the mouse button to drop the file into that track. Digital Performer won't let you drag a stereo sound file into a mono track, or vice versa, and it won't let you drag a sound file into a MIDI track.
When you import sound files, Digital Performer automatically copies them into the Audio Files folder inside of your project folder.
Now you can drag the soundbite around in the track, as if it were a MIDI note. You can create other instances of the same soundbite by option-dragging it (i.e., holding down the option key while dragging).
Once in your sequence, the soundbite appears in the Soundbites view, which you can open by typing shift-B. As you edit soundbites, Digital Performer will create more entries in the soundbites list. You can place an instance of a soundbite into a track by dragging its move handle (the squiggly shape in the MVE column of the Soundbites window) into a track.

In the Soundbites view, you can click on a soundbite name to hear the soundbite. (If this doesn't work, click on the speaker icon next to the metronome in the Control Panel.)
CAUTION: If you double-click on the soundbite name, instead of single-click, then you'll open a destructive audio-editing window. You don't want to do anything destructive now, do you? Close that window before it's too late.

DO THIS: Create a new Digital Performer Project, and import one or more of the sound files I provide into your project by dragging it into a track.

Project Folder Organization

It's important to keep your project folder organized, because otherwise you will have problems with lost sound files later.

A project folder contains a sequence file, as well as folders for sound files and other things. Sound files are much larger than MIDI files. Because a project contains multiple files and folders, you need to be especially careful about backing up. My recommendation is to keep everything relevant to the project inside your project folder, and copy this folder to and from the Music Server. Keep an extra copy on a USB flash drive.

You should NOT run your project directly from the Music Server. Even though this may work some of the time, it's usually too slow to be reliable. You might hear audio drop-outs, or Digital Performer might complain about not being able to run all the audio effects you want. Never try to run a project directly from a CD or USB flash drive, which are too slow.

So when you come in to work on an existing project, you should copy the entire project folder from the Music Server onto the Desktop (i.e., the Mac's internal hard disk).

Notice that it's not sufficient to copy just the sequence file. If the sound files are still on the server, DP may be cranky when you play back the sequence, because accessing the sound files on other disks is too slow.

To reduce the chance that Digital Performer will use a sound file on the Music Server or your CD or USB drive instead of one on the hard disk, eject all servers and other disks before opening your sequence file.

WARNING: At least half of the problems people have working with audio in Digital Performer are due to carelessness about the issues discussed above. If you turn in a project that doesn't work, because your sound files are living somewhere else and Digital Performer can't find them, then that will affect your grade.

Soundbites — What are they?

In Digital Performer, you work with the audio in a sound file using soundbites. A soundbite is a reference to a portion of a sound file on the hard disk. For example, say you have a sound file called "locomotive.aif." It's a 30-second recording of a steam engine, which blasts its whistle for 10 seconds during the middle of the recording. You could make a soundbite, called "whistle," that refers just to the portion of "locomotive.aif" during which you hear the whistle.

The soundbite stores the start time of the whistle, relative to the beginning of the sound file, and the duration of the whistle. (You can see the duration in the Soundbites view.) When Digital Performer plays this soundbite, it looks up the timing information in the soundbite, and then uses it to read just the specified portion of the sound file.

Here's the important part: the soundbite does not contain a copy of the portion of the sound file. In other words, the soundbite does not contain audio samples copied from the sound file. It just contains two references — start time and end time — to the sample data in that file. This means that the soundbite doesn't take up very much memory or disk space — nowhere near the amount used by the audio data. It also means that editing soundbites is very fast, because only the start-time and end-time references must change, not the actual audio data in the sound file. Soundbites are the cornerstone of Digital Performer's non-destructive editing environment: they make it possible for you to cut and paste bits of audio without ever altering the original sound file.

NOTE: Soundbite is a Digital Performer term. The same thing is called a region in Pro Tools and some other software.

Editing with Soundbites

Open the Sequence Editor view by double-clicking an audio track in the Tracks window. The Sequence Editor shows all tracks, including MIDI tracks, in one window. This is where you'll do most of your soundbite editing.
You can hide and show audio tracks in the Sequence Editor view in the same way that you do this in the Mixing Board window: by clicking on track names in the Track Selector (Studio > Track Selector).
There are many ways to edit with soundbites, copying and moving them around, as well as creating new soundbites that have different dimensions. (Remember the example of the locomotive whistle above.)
Here are some of the more common editing techniques.
- Drag a soundbite to move it to another place in the same track or to move it to a different track, just as if it were a MIDI note. Pay attention to the Snap To Grid controls in the upper right part of the Sequence Editor view.
- Option-drag a soundbite to place a copy of it in another track or at another time in the same track.
- Control-drag a soundbite to make it snap to the edge of another soundbite.
- Option-control-drag a soundbite to copy it and snap the copy to the edge of another soundbite. This is a way to make repetitive patterns like drum loops.
- To extract audio from one soundbite into new, shorter ones...
  - Open the soundbite for editing by double-clicking it. The cursor becomes an I-beam.
  - Make a selection by dragging in the soundbite with the I-beam tool. The segment you select will play immediately. Hold the shift key down while clicking on another part of the soundbite to extend or shrink the selection.
  - Edit > Copy.
  - Set an insertion point at an empty place in a track by clicking there with the crosshair (+) tool.
  - Edit > Paste.
- Use the Split command (Edit menu) to break one soundbite into three separate soundbites. First double-click a soundbite and make a selection, then use the Split command.
- You can change the dimensions of a soundbite by edge-editing. This means dragging the left or right boundary of a soundbite to reveal or exclude some of the underlying audio file. Move your mouse over the left (or right) edge of a soundbite — over the bottom 3/4 of the soundbite. Your cursor changes to the edge-edit cursor.
  
  Now click and drag. This changes the start (or end) time of the soundbite, relative to the start of its sound file.
  NOTE: When edge-editing, don't drag the edge near the top of the soundbite, since that time-scales the soundbite instead.
  So if you have a soundbite that refers to the entire duration of its sound file, dragging the right edge of the soundbite to the left will shorten the soundbite. (Don't worry: this doesn't delete any samples in the sound file!) Dragging the left edge of the soundbite to the right also shortens the soundbite, but it suppresses the beginning, instead of the end, of the sound file, and it changes the start time of the soundbite in the sequence. (If it started at the beginning of a measure, now it might start on the second beat of the measure, for example.)
  CAUTION: By default, edge-editing affects the current soundbite and all instances of it in any track, as well as future instances. If you don't want the soundbite you edge-edit to affect other instances, first click on it to select it, then issue the Audio > Duplicate command. This makes the soundbite you select unique — independent from other instances of it.

The soundbites Digital Performer creates when you import a sound file refer to the entire file, whereas the soundbites you make when editing usually refer to just part of the file. When you edit soundbites, Digital Performer often creates new soundbites without you knowing about it. This is convenient, but sooner or later you'll want to manage your huge list of automatically created soundbites. In the Soundbites view (Project > Soundbites), look for the Mini Menu triangle, and click to get a pop-up menu.

Choose Select Unused Soundbites from this menu, followed by Remove From List or Delete Soundbite (careful with that one). Try using the View By pop-up menu. Rename soundbites and sound files by option-clicking their names in the Soundbites cell.

DO THIS: Using the techniques listed above, chop up the sound file I gave you into smaller segments — short phrases, words, even syllables. Each segment is a soundbite. Then arrange the segments on multiple tracks to make an interesting collage of spoken sounds. Use at least three tracks.

Volume and Pan

You can change the volume and panning of a soundbite while it plays. You do this either with the Mixing Board, just as you did for MIDI tracks, or by manipulating "rubber band" lines in the Sequence Editor. When using the Mixing Board, record-enable automation, play the sequence, and adjust the fader and pan knob. Here's how to change volume using the "rubber band" lines.

Open the Sequence Editor. At the left of each track is a control panel. A pop-up menu lets you select different editing modes. Normally you'll see Soundbites here. This mode lets you drag your soundbites around and edit them.
Choose Volume from this pop-up menu. Now any soundbites in the track are dimmed, and you can use the pencil tool to draw a volume line. (If you don't see the Tool palette, choose Tools from the Studio menu.) The volume line works by "connecting the dots." You supply the dots, and the program draws lines between them to ramp the volume from one setting to another. Digital Performer calls one of these dots an audio volume event, which you can edit very precisely, as if it were a MIDI volume controller event.
The volume lines you've drawn may display as dotted lines. This means that automation is not play-enabled for that track. If you play the track, you won't hear any volume changes. You can play-enable automation using the Mixing Board, as we did for MIDI tracks. Or you can use the handy Auto pop-up menu in the Sequence Editor control panel.
You can drag the dots around to change the shape of the volume line. Click on a dot and press the delete key to delete it.

The audio volume event values are expressed in decibels (dB). A value of 0 dB means the audio plays from the disk with no amplitude scaling. If you shape the line so that it goes above 0 dB, the audio samples will be multiplied by a factor greater than one, and you could easily cause clipping (digital distortion). If this happens, the track's meter in the Mixing Board will show it.

Panning works similarly to volume:

Choose "Pan" from the same editing mode pop-up menu from which you chose "Volume."
Shape the line just as you did for the volume line. To pan to the left channel, drag a "dot" to the top of the graph. To pan to the right channel, drag to the bottom. The vertical center of the graph corresponds to the center of the stereo field.

Choose "Soundbites" from the editing mode pop-up in order to edit soundbites again.

Shortcut: Hold the option key down while selecting an editing mode (e.g., Soundbites, Volume, Pan) from any audio track's popup menu: all tracks will change to that mode.

DO THIS: Include some volume and pan events in your audio tracks. Use any method you like to create these — recording automation in the Mixing Board, or inserting them in the Sequence Editor. Use some continuous panning and volume changes in at least two places. Play-enable automation for any tracks that have changing volume or pan.

Time-Scaling and Pitch-shifting Audio

Digital Performer lets you change the duration of a soundbite without changing its pitch, and it lets you change the pitch of a soundbite without changing its duration. This is different from the way samplers in Reason work, where transposing a sample also changes its duration.

To time-scale a soundbite, open the Sequence Editor, and move the mouse over either edge of the soundbite — over the top 1/4 of the soundbite. The cursor changes to a hand.

Click and drag to stretch or shrink the soundbite horizontally. When you release the mouse, Digital Performer creates a new time-scaled sound file, and replaces the soundbite you dragged with one referring to the new sound file. This process takes a moment, during which time the waveform appears hollow, and the soundbite will be silent if you play the sequence.

To pitch-shift a soundbite, select it and choose Transpose from the Region menu. You then see the same window that you would use to transpose MIDI notes. Be sure that Transpose audio is checked, and choose Transpose audio by creating new soundbites. The specification of transposition interval works the same as it does for MIDI notes.

Don't expect time-scaling by large percentages or transposing by large intervals to sound natural. But sometimes, unnatural is good.

The Spectral Effects menu command combines time-scaling and pitch-shifting with formant-shifting. (See the last few slides in the Acoustics PowerPoint file.) Go ahead and play with it. But please don't use this to fulfill your audio effect requirement; do that with the audio effects you choose in the Mixing Board (see below).

Some extra info about transposition...

Digital Performer has two methods of transposition. The default "PureDSP" method works well for cleanly-recorded, non-reverberant sounds containing a single pitch or melody. (In music theory terms, it works for monophonic, not polyphonic, sounds.) This method lets you shift formants independently of pitch, using the Spectral Effects command. The other method does not let you do formant-shifting, but it works better with sound files that are polyphonic or contain reverberation. If the result of the default "PureDSP" transposition sounds garbled, you can arrange for a soundbite to use this alternative method of transposition. This setting, "Use Standard Pitch Shift," is in a pop-up menu in the Sound File Information view (Studio menu).

DO THIS: Time-scale or pitch-shift at least one soundbite, and use that somewhere in your sequence.

Audio Effects

One of the great things about using audio in Digital Performer (or routing Reason audio into Digital Performer) is that you can use audio effects. There are many of these, and we'll discuss some of them in class. For this assignment, though, feel free to play around with any of them, even if you don't understand what they're doing.

You access the audio effects just like you did the MIDI ones used in Assignment 2: choose them from the insert pop-up menus at the top of each audio track in the Mixing Board. Many effects have a wet/dry mix control that governs the amount of effect you hear. All effects have a bypass button; toggle this to compare the dry track with the wet (or "effected") track.

The best way to get started with effects is to try the presets that most of them include. Select these from the effect window Presets pop-up menu (which normally reads "None").

A real-time effect assigned to a track affects the track for the entire duration of a sequence. You can apply an effect to a single soundbite by selecting the soundbite and choosing the effect from the Audio > Audio Plug-ins menu. This creates a new soundbite with the effect applied; it leaves the original soundbite alone. This is sometimes a good thing to do, if you want just one soundbite to have an effect but don't want anything else in the track to have the effect. However, you lose the benefits of real-time audio effects: effect-setting automation, easy changing of settings, and chaining of effects; and you'll have trouble with effects, such as delay and reverb, that create sound that continues to play after the soundbite stops.

Automation works for most real-time effect settings. Use it just like the automation of volume and pan: record-enable automation, then change effect settings while playing the sequence. Digital Performer remembers your setting changes. You can edit these in the Sequence Editor the same way you edit volume automation. (Choose the effect parameter from the Sequence Editor mode menu, discussed above for volume changes.)

DO THIS: Use at least two audio effects in your sequence. Choose the effects either from the Mixing Board or from the Audio Plug-ins menu.

Preventing Audio Clicks

If you're listening carefully, you'll probably hear clicks at the start or end of some soundbites. This usually happens when there is a sharp discontinuity in the waveform — a common occurrence when splicing bits of audio together. Sometimes you also want to crossfade two adjacent soundbites. This smooths the "joint" between them.

You get rid of clicks by applying a volume envelope to a soundbite. Even a very quick, barely noticeable attack or release can suppress a click. There are two ways to create these envelopes:

Volume ramps, as described above — not really recommended for click suppression.
Fades, created using the Fade command (Audio menu). To create a fade, select a brief time region overlapping the start or end of a soundbite, then invoke the Fade command. Make your selection in the Sequence Editor for precision. This method is preferable to the volume ramps, partly because the fades move with the soundbite when you drag it, but the volume events don't.
There is a graphical shortcut to using the Fade command: move the mouse over the edge of a soundbite, in the area just between the waveform display and the colored title bar. When you see the crossfade cursor, click and drag toward the middle of the soundbite to create a fade (or crossfade).

This may strike you as an obscure consideration. But to get professional results, you need to suppress unwanted clicks.

Combining MIDI and Audio

Once you get the hang of using soundbites, try combining several audio tracks with some MIDI tracks to create an interesting hybrid texture. There's nothing special to know about combining MIDI and audio — just work with each kind of track using the appropriate methods.

DO THIS: Add at least two MIDI tracks to your sequence. Try to find MIDI-based sounds that complement the speech. (For example, adding a pitched, sustained sound can point up the musical qualities of the speech.) You can use Reason to play some or all of your MIDI tracks, as long as you provide me with a Reason song file, which contains your rack setup.

A Note on Backing Up

When you've finished working, copy your Project folder to the Music Server. Make a duplicate backup onto a USB flash drive.

When you come in again to work on an existing project:

Copy the project folder from the Music Server to the Desktop.
Disconnect from the Music Server (drag its icon to the Trash). This is important, because it's very easy for Digital Performer to use sound files that live on another disk instead of in the project folder on the hard disk. This can lead to various problems, so be organized!

Remember to follow the assignment submission instructions above (where it says "What to turn in")!