Implementing video playback statistics in Firefox
Early this year I began my second course in Open Source development taught by David Humphrey. I have been working on various Firefox bugs since the beginning of the semester and have landed 2 tickets so far, with a 3rd on the way. The 3rd ticket that I am working on is almost done and just requires a few tests to be written then it should be good to go. While I muster up some motivation to write the tests I figured I may as well begin working on the biggest ticket I took on this semester, which is implementing the rest of the video playback statistics in Firefox. I was initially going to work on it alone, but Mathew Schranz expressed interest in working on it as well, so why not, the more the merrier!
We began by dividing up the remaining features that were still needed which included the following:
- bytesReceived – the number of raw bytes received from the network for decode
- downloadTime – the time since first HTTP request is sent until now or the download stops/finishes/terminates (whichever is earlier)
- networkWaitTime – the total duration of time when a playback is stalled, waiting to receive more data from network
- videoBytesDecoded – the number of bytes of video data that have been decoded
- audioBytesDecoded – the number of bytes of audio data that have been decoded
- droppedFrames – the number of frames of video that have been dropped due to performance reasons
- playbackJitter – it is useful for applications to be able obtain an overall metric for perceived playback quality and smoothness
I decided that I would take bytesReceived, downloadTime, networkWaitTime, and dropped frames while Mathew would take on videoBytesDecoded, audioBytesDecoded, and playbackJitter. I decided to simply start at the top of the list and work my way down, so bytesReceived was first. Since Humph went over this in class last week I had a good idea of where to start, so I picked up where we left off. Using MXR I looked up where the other parts of the playback statistics had been implemented and worked out from there. My guess was that I wouldn’t need to do too much heavy lifting, as much of the work was probably done for me ( I mean, they must keep track of what they are downloading somehow ) so I began digging through various mediaElement, decoder, and mediaStream files. I eventually ended up pieces of code for each of the various video encoding formats and began sifting through what was there. Eventually I found a function that seemed like something I wanted:
After looking at the function a bit closer I noticed that it called another function, AddBytes on the mPlaybackStatistics object, which seemed liked something that was related to what I was doing here ( not to mention it was referencing playback statistics! ). Looking closer at AddBytes I noticed it was adding the number of bytes passed into it to a local variable in the decoder called mAccumulatedBytes. I was pretty confident at this point that what we wanted for bytesDownloaded was simply all of the bytes that we have downloaded so far, which im pretty confident is mAccumulatedBytes. Now I was ready to start coding!
I started by defining the attribute, mozBytesDownloaded, inside of the idl file located at dom/interfaces/html/nsIDOMHTMLMediaElement.idl inside mozilla-central. I made it a readonly attribute that was an unsigned long ( as i’m pretty sure that there will be no negative values here, so why not ). Next I needed to find a way to reach into the decoder and access the value of mAccumulatedBytes, so I took a look at the route that was currently being taken by similar functions to get to the decoder. I decided to look at how Chris Pearce originally did this for the already existing portions of the video playback statistics ( mozParsedFrames, mozDecodedFrames, mozPresentedFrames, mozPaintedFrames, mozPaintDelay ). After doing another search on MXR I found that this was being done inside nsHTMLVideoElement.cpp . Seeing as the bytesDownloaded attribute applied to both audio AND video, I couldn’t do this in here but told me it needed to be done in the nsHTMLMediaElement.cpp file instead.
Inside the nsHTMLMediaElement file I saw similar getters and setters to those that were in the videoElement file, such as GetMozSampleRate, GetMozFrameBufferLength, and many more. I figured a good place for my new function, GetMozBytesDownloaded, would be at home with these guys. I did a bit of cargo culting and stole what logic I could from GetMozFrameBufferLength as it was also accessing the decoder so seemed ok to me. What I needed to do from here was find my way into the decoder and it looked like GetMozFrameBufferLength was doing so also ( it was calling GetFrameBufferLength() ) so I decided to look where I could write my own function to access the data that I needed. The method was being defined in nsMediaDecoder.h so I figured it was a good place to put mine as well. After doing a bit of poking around and checking out what else was in here I found the function that brought me down this path, NotifyBytesConsumed! I was feeling even more confident now in my function placement and created my own, getBytesDownloaded. I did some more cargo culting here and stole what I could from NotifyBytesConsumed. I figured I didn’t need the check that NotifyBytesConsumed had for mIgnoreProgressData as it looked like it was pertaining to whether the video seeked or moved the play position; we only cared about how many bytes were downloaded. Instead of calling AddBytes on mPlaybackStatistics I figured I would yet again write my own function, getBytes. Just like I did previously, I looked up how AddBytes was implemented and followed suit where I could( which was done in MediaResource ). All my function did was ensure that the video had started ( if it wasn’t started, return early ) and if it had started then return the number of bytes that we have downloaded. Alright great, now lets compile this and see what happens. Anddddd build failures 😦
I managed to fix a few failures but I am currently stuck on one that looks like the following:
Undefined symbols for architecture x86_64:
“nsMediaDecoder::bytesDownloaded()”, referenced from:
nsHTMLMediaElement::GetMozBytesDownloaded(unsigned int*)in nsHTMLMediaElement.o
ld: symbol(s) not found for architecture x86_64
It’s gotten pretty late now, so I am going to tackle this in the morning and hopefully I figure it out then.