Black Magic: Event production with Perl

TL;DR

Introduction

Five years ago, I started as “someone who knew computers” without any real knowledge of the task I was given. “You control that screen”, as they pointed at the big LED screen in the middle of the stadium overseeing center court. My job was to put content on the screen during matches, so called crowd pumpers, short videos to boost crowd engagement after points. We also switched to replays and commercial content. And of course, one of the most important things: we showed a scoreboard.

In the following days I prototyped Raspberry Pi’s to switch videos and so called crowd pumpers during one of the biggest beach tennis tournaments on the planet: The Aruba Open.

After talking to a TV producer who introduced me to vMix and BitFocus Companion, I was set. I was able to control the LED screen(s) via a couple of mobile phones and I could hand them off to others while taking bathroom breaks. Our MC’s could now also control content if needed.

This year I’ve built a fully web-based production control center to operate Jumbotrons and LED boarding. Game, set and match.

Going pro

Each year the setup improved a little. I knew the system, and I pushed it a bit further, added some some more things to it. A dedicated phone, a dedicated tablet, improved how we showed commercials and did daytime playlists.
And then three years ago I bought Blackmagic Design (BMD) HyperDecks and a BMD ATEM. For the uninitiated, these are (semi)professional hardware devices that do video playback and switching for event production, conferences and TV channels. The production company already used a BMD device, which was my gateway drug to this hardware. Only one problem: They don’t do no playlist support. I guess the TV world doesn’t need playlists, but I do. For daytime usage, we need sponsor loops and show after-movies of previous years. I had to reluctantly reach for vMix again…

HyperDeck playlist support

Last year before the Open, I developed a playlist driven playback system. HyperDecks support a text-based Ethernet protocol. And the coolest thing: You can listen to events on the decks. So I mimicked the approach I took from my i3 workspace tool and built the first version of playlist support. Actually, it was v2, I had a blocking version the year before, but I never used it. And it didn’t use the event system either.

While my parents were watching Formula 1 at my place during their vacation, I was knee-deep in AnyEvent code and celebrating little wins after I saw the correct clip play on either of my HyperDecks. I ended up with a toolkit that ran AnyEvent, exposed itself over the network with a small RPC server and could be operated by anyone who knew how to press on a button.

I added a small Intel Compute Stick, installed Samba on it, installed my little toolkit on it and I had myself two HyperDecks and Companion/StreamDeck integration. I even added automatics NAS (re)attachments so anyone could startup the HyperDeck and connect them to the Samba shares. It was a great success, I hardly had to press a button or explain things to the LED screen operator: everything just worked.

The first iteration of EventAV was born.

Watching the Event Horizon

So I stole the concept from my i3 module, I wanted a consumer to be able to subscribe to events:

$hyperdeck->subscribe(transport => sub {
    # action here
});

HyperDecks send events in the following format:

123 action:
some: DATA
other: thing
yada: yada

My parser looks for the 123 action: string, maps it to an action and then tries to parse the data that follows. The annoying thing is that the messages can be long and that I don’t capture all the data in a single read. The trick is that HyperDecks end their lines with \r\n and the final section of an event in \r\n\r\n. So I do some clever things to ensure that my parser keeps the incomplete line and then prepends it when reading the next segment(s). Until the moment comes where the segment really ends.

# Quirky
  $line =~ s/(200 ok)(\d+)/$1\r\n$2/;

  if ($_remainder) {
    $line = $_remainder . $line;
    $_remainder = undef;
  }
  my @lines = split(/\r\n/, $line);

  my $complete = 0;
  if ($line =~ /\r\n\r\n\Z/m) {
    $complete = 1;
  }
  else {
    $_remainder = pop @lines;
  }

When it does I map the code to an event, e.g. 123 action is mapped to action. And the data becomes:

{ some => 'DATA', other => 'thing', yada => 'yada' }

And now you can write a subscriber:

$hyperdeck->subscribe(action => sub {
    my $data = shift;
    return unless ($data->{some} eq 'DATA');

    # more code here
});

A qualification, a sprint race and a race further, and you suddenly have a full fledged playlist daemon.

Playlist requirements

A playlist isn’t just playing clips in a particular order. When you switch inputs or clips you cannot have that after the clip finishes, your playlist just continues. So there is logic to detect if a clip is owned by the playlist or isn’t. If the playlist owns a played clip it continues with the clip that comes after it, and if it doesn’t it just doesn’t do anything and waits for someone to actually start the playlist again.

From NAS to screen

So how is this all done? HyperDecks expose their media not based on filenames, but on numerical IDs. So once you add a new file, the IDs can change and you most likely need to reconfigure almost everything for each HyperDeck. Lucky for us, you can ask the deck to give you all the files, via two different methods, disk list and clips get which both reply with a different format. EventAV deals with both and rebuilds an internal mapping so all playlists still know exactly which file gets played based on the filename.

Last year we had to reload the configuration after we added a file. This year, we did this event based, as the decks can also notify us about disk changes if we subscribe to these events.

Once we stop a playlist mid-flight we can restart it from the same clip. And if we add more logic we can even start it from the exact moment we’ve stopped the previous clip, but that doesn’t really give us a benefit yet, so it hasn’t been built yet.

From YAML to SQLite

Playlists were configured in a simple yaml file, and once the playlist had to change, I just logged in with ssh and updated the file. I think around this time I also first actually found out YAML was created by Ingy döt Net, who I met and had dinner with in Glasgow during a YAPC a few years earlier.
This year however, I wanted a proper DB backed system because I wanted a web UI to upload files to the NAS and to automatically convert files to the right format. This will be done by using a couple of zsh scripts I wrote in the past.

To understand why I’m making this, when I started with the Aruba Open I operated LED screens. Today I not only enable LED operations with my hardware, I also operate match scheduling for over 1900 teams over 8 days, this includes frontend, backend, and mobile. I simply need to outsource some tasks to other people.

I want the person who had to press buttons to be able to change things, to upload things, without me being present and without having to know anything about the system. Push button easy. Others have to be able to change the playlist, via a browser, without having to open a text file.

Video formats

HyperDecks require some (obscure) ProRes formats which are a bit finicky. The format itself isn’t that complex, the playback requirements of the HyperDeck are weird. That is because besides playing video, they can also record video. And that is their primary use case, and thus the recording format is the playback format. And it needs to be the exact same. Otherwise you see it on the disk, but you won’t see it on the deck (and other tools). Ironically.. I managed to figure out the, in my opinion, more complex ProRes format, while I still, after three years, haven’t figured out the h264 playback configuration.

So once we get a video from anyone, it first needs to be converted to the right format, to do this, we had to convert everything before loading it on the NAS. This manual step should be eliminated.

Commercial rotation and play logging

The Aruba Open uses playlists for commercial loops throughout the day. And once the sun sets and artificial lights hit the sand, our center court changes from a serene oasis to packed colosseum. And this is where the HyperDecks and LED boarding do double duty as commercial outlets and crowd activation tools.

This means that we need to show commercials from our top paying commercial partners: With the press of a button we need to transform the stadium into “Sponsor A”. In the past we did a lot of it manual, with pen and paper. EventAV has been adapted to be able to assign sponsor tier levels, so the operator just needs to hit a button and all the visuals change based on tiers and potentially focus point for a night.

We try to mimic a commercial break like on TV, with the notable exception, we want to limit the amount of commercials, because commercials are a downer for a crowd that wants to go wild. EventAV can therefore automate button presses on BitFocus Companion so once a commercial has finished, we directly go back to a live feed (of often the crowd dancing).

To help marketing, we log the amount of plays of each clip that we play, this will allow us to show concrete numbers to our partners and also means we can debug and optimize EventAV (but it is mainly a commercial/business focused goal).

Branding the stadium

While we can independently control LED boarding and Jumbotrons, the real cool feature is to paint our center court in one color. One of our sponsors uses a bright pink color and once those colors are everywhere, it adds a little extra to the commercial experience.
We do this via a so-called weighted playlist. A weighted playlist is perhaps a bit of a weird concept, but it essentially adds “weight to a partner”, so they are shown more often than others. These partners often have their own LED boarding media. To control the LED boarding our production partner uses Resolume – it is mostly used by VJs, and production companies, that need to split video over screens and odd shaped displays. We can configure EventAV to use a Resolume column, and we can do this because EventAV has OSC support baked in. This means our operator presses a button and the partner’s clip start playing. And because the weighted playlist knows about the LED boarding, we can change the clip and the whole stadium is now “branded” to one sponsor.
EventAV can control the Resolume side directly, but Companion still handles operator controls and the ATEM transitions.

Running it at an event

What I learned

Conclusion