Working with the CBC – An HTML5 audio experiment

For the last 2 days a small team and myself have been working on an HTML5 audio puzzle for a tribute to Marshall Mcluhan.  The team consisted of Chris DeCairos, Scott Downe and myself from CDOT, Bobby Richter, Mark Surman and Allen Gunn from Mozilla, and Lilly, Sean and Angela from the CBC.  We had a brief introduction on Monday morning and then went straight into brainstorming.  The group from the CBC explained to us there vision and what it is they were looking for in terms of a demo.  What they wanted was essentially an HTML5 audio puzzle.  The puzzle would consist of various short clips that were from a much larger clip.  They would be split into various sections and jumbled in a word box.  The user would then have to organize the quote of Marshall Mcluhan in what they think is the correct order and then click play.  Even if the order the user entered was wrong, it would still play the audio, just out of order and sounding weird.  THe user would then rearrange the pieces and try it out again until the demo produced a seamlessly playing clip.  At this point the user can attempt to submit there selection if they think they have it correct.  When the user clicks submit all of the correct entries will stay locked in place, highlighted green, and all of the incorrect ones will be thrown back into the word box.  This outlined the basic functionality of the game.  The CBC crew also wanted a simple way to create these short audio clips from a larger one.  So we needed to devise a plan to do so.

The CDOT team, Bobby and I quickly began throwing around ideas on what we could use to build this and what would be necessary.  We contemplated on using Popcorn.js, but in the end decided it was a bit overkill for what we were going to be doing.  We decided to write our own mini-timeupdate portion of Popcorn.js to use and went from there.  We split into two small coding groups, Scott and I were working on the user facing side of things, so we essentially handled the adding of audio clips, what clicking on a clip would do, implementing drag and drop, and adding in the logic for the puzzle.  This included locking pieces in that were in a correct spot, playing the clips regardless of what order they were in, and determining if the user has won or not.  Chris and Bobby were responsible for creating an editor in which the user could highlight a portion of an audio clip, and in doing so it would save that clips data in JSON.  The user could then select it again to enter the text that will correspond with this clip ( likely the words that are said in the clip ).  They could then click a button after they have created several clips and it would output a big JSON blurb in which they can easily plugin to what Scott and I wrote.  The goal for Monday was to on both ends, get a basic demo up and running with all of the essential features included.

In order to be accurate with clips, Scott and I decided to use request animation frame ( getting 30 frames per second i believe ).  This would allow for Chris and Bobby to create extremely accurate sound clips for us to use.  We got a small blurb of code from Paul Irish’s blog ( as apparently its now the standard for request animation frame ) and rolled with that as a starting point.  We then implemented a addEvent function to our small core that we created.  Within the hour we had our event registration system up and running, which was very similar to the structure of Popcorn.js.  We provided a start, end and text attribute for the user to fill in, and then stored these events in our core for later access.  We then added onClick functionality to each of the bin items so that the corresponding event would be fired when they were clicked, sort of like a small preview.  This was simple enough to get going and basically just required us to create a function that would be run when request animation frame was fired.  The next task was a bit more difficult.

Since the CBC was excited to use HTML5 tools and functionality, we decided to go with the HTML5 drag and drop over the jQuery one as we figured it was probably overkill again to go with jQuery.  The sad part about this is, is that the HTML5 drag and drop is pretty messy and wasn’t something that I could just common sense together ( as I found out the hard way ).  Tho it is all done through listeners, it heavily relies on some weird event functions that I’ve never used before, such as preventDefaults and stopPropogation to name a few.  Scott and I scowered over the documentation for the better part of the morning to get this working in a very basic sense.  Shortly after lunch we got it working and were ready to start implementing the actual puzzle part of the demo.

Our next step was to on a button click, attempt to play all of the clips that the user had assembled in our, for lack of a better term, timeline box.  It would go through each of the events, play it until its end time, and then go onto the next event if there was one.  This was one of the cooler pieces of code that I’ve had the opportunity to write in the last little while as it was asynchronous and involved a good bit of recursion.  The code itself wasn’t the pretty thing I’ve ever written ( as im sure Bobby can vouch for haha ), but it was fun to write and made me really sit down and think about how to solve the problem.  The code block for this was as follows:

<pre>var ridinSpinnas = function( options, cb ) {

    if ( audioElement.currentTime >= options.end || audioElement.currentTime === audioElement.duration ) {
      cb && cb();
    } else if ( !stopSpinnin ) {
      requestAnimFrame( function(){
        ridinSpinnas( options, cb );
      });
    }
  };

  var getNextChild = function( children ) {

    var increment = function() {
      playingIndex++;
      if ( playingIndex  >= itemsLength ) {
        audioElement.pause();
        playingIndex = 0;
      } else {
        getNextChild( children );
      }
    },

    itemsLength = children.length,
    childsChildren = children[ playingIndex ].children,
    currentNode = children[ playingIndex ],
    options;

    if ( childsChildren.length > 0 ) {

      options = playingEvents[ childsChildren.item( 0 ).id ];

      if ( options.start !== lastEnd ) {
        audioElement.currentTime = options.start;
      }

      lastEnd = options.end;
      audioElement.play();
      addClass( currentNode, "cbc-puzzle-playing" );
    }

    if ( options ) {

      requestAnimFrame( function() {
        ridinSpinnas( options, function() {
          removeClass( currentNode, "cbc-puzzle-playing" );
          increment();
        });
      });
    } else {
      increment();
    }
  };</pre>

Excuse the variable names, but other than that its a cool piece of code.  Essentially we call our getNextChild function, which then gets the first audio clip that we are going to play.  We pass it into our request animation frame function and let it spin ( we ridin spinnas yo ).  We also pass in a callback because of its asynchronous nature, we tell it to fire the callback once its done.  Inside our callback we then either finish or call ourself again ( getNextChild ) and continue from there.  I think its pretty neat.

It is reasons like this that I love programming so much and really reinforces why I love doing this so much.  Even tho im sure its been done 100000x before, it is cool to sit down and do it by myself and to see the outcome first hand.  I love that each and every week im being tested with new problems and have to think up new solutions to solve them. There is never a shortage of unique and challeneging problems and I love it!

The second day Scott wasn’t around due to school commitments, so I was flying solo here.  Essentially what needed to be finished was some end game logic and some logic to lock a piece in place if it is found to be in the right place when the submit button is clicked.  This took me a while to get up and running and still has a few bugs with it in some fringe cases, but for the most part is finished.  If I had one more day I think I would have it perfect, but I guess that can be said for a lot of things 😛 .  Chris and I also threw togethor a randomizer function to randomize the data being entered ( scramble the words in the word bin ) and a readInJSON function for, well, reading in JSON events.  After this we headed over to the CBC office downtown toronto and worked with Sean and another designer there on getting there design incorporated with our code.  There were a few issues with CSS collisions and such, but for the most part it all started to come togethor.  We saw our crappy looking demo looking all nice and shiny now, which is always a good feeling.  We were also able to identify a few bugs here as well, which although sort of sucks, is nice that we had the opportunity to sit down with the designers and go over everything.  All in all, it was an amazing two days and I can’t wait for when this goes live on CBC‘s website 😀

Im sure you all want to see what we have done over the last two days, so here are two small demos showcasing what we have done, enjoy!

http://scotland.proximity.on.ca/dseif/cbc-puzzle/puzzleCSS/

The above is pretty self explanatory, drag onto timeline, click play to hear, submit to validate. Fun ensues.

http://scotland.proximity.on.ca/dseif/cbc-puzzle/editor.html

Above made by Bobby and Chris, hold down the z key to capture audio fragments and let go to stop the capture.  Click on it to play the capture and hit enter when focused on it to enter text for it.

ALso, here are a few pics of the Toronto Mozilla office and one of the CBC.