Welcome to the
Dec. 15th, 2021 issue of Radio World Engineer Extra
I T ’ S A L L I N W H E AT N E T- I P
THE INTELLIGENT NETWORK
Make Your Streams Stand Out Stream up to eight programs at once, each with four outputs for a total of 32 streams. Full suite of stream-specific audio processing tools. Optimize performance of audio content. AAC, MP3 and Opus encoders. Reaching a broad range of end user devices and players. Metadata agnostic. Lua transformation filters adapt metadata input from any automation system into any required output format. Cloud-ready for the future, yet compatible with standard CDN and streaming platforms now. Supports HLS, Icecast, RTMP, and RTP streams. All-inclusive Linux and AoIP appliance. No Windows® drivers, updates or PC needed. Add StreamBlade to your WheatNet-IP audio network or add Wheatstream to any existing AoIP network, including WheatNet-IP or AES67 compatible networks.
STREAMBLADE & WHEATSTREAM STREAMING AUDIO PROCESSORS wheatstone.com/stream-rw21a
BROADCAST AUDIO PERFECTIONISTS®
In-depth technology for radio engineers
radioworld.com | December 15 2021 | $5.00
Cloud-ready and CDN-compatible
Rick Bidlack offers a primer on the politics and protocols of streaming.
monsitj/Getty Images
Vol. 45 No. 31 | December 15 2021 www.radioworld.com FOLLOW US www.twitter.com/radioworld_news www.facebook.com/RadioWorldMagazine CONTENT Managing Director, Content & Editor in Chief Paul J. McLane, paul.mclane@futurenet.com, 845-414-6105 Technical Advisors Thomas R. McGinley, Doug Irwin Technical Editor, RW Engineering Extra W.C. “Cris” Alexander Contributors: Susan Ashworth, John Bisset, James Careless, Ken Deutsch, Mark Durenberger, Charles Fitch, Donna Halper, Craig Johnston, Alan Jurison, Paul Kaminski, John Kean, Peter King, Larry Langford, Mark Lapidus, Michael LeClair, Jim Peck, Mark Persons, Stephen M. Poole, James O’Neal, Rich Rarey, Jeremy Ruck, John Schneider, Dan Slentz, Randy Stine, Tom Vernon, Jennifer Waits, Steve Walker, Chris Wygal Production Manager Nicole Schilling Managing Design Director Nicole Cobban Senior Design Directors Lisa McIntosh and Will Shum ADVERTISING SALES Senior Business Director & Publisher, Radio World John Casey, john.casey@futurenet.com, 845-678-3839 Publisher, Radio World International Raffaella Calabrese, raffaella.calabrese@futurenet.com, +39-320-891-1938 SUBSCRIBER CUSTOMER SERVICE To subscribe, change your address, or check on your current account status, go to www.radioworld.com and click on Subscribe, email futureplc@computerfulfillment.com, call 888-266-5828, or write P.O. Box 1051, Lowell, MA 01853. Licensing/Reprints/Permissions Radio World is available for licensing. Contact the Licensing team to discuss partnership opportunities. Head of Print Licensing Rachel Shaw licensing@futurenet.com MANAGEMENT Senior Vice President, B2B Rick Stamberger Vice President, Sales & Publishing, B2B Aaron Kern Vice President, B2B Tech Group Carmel King Vice President, Sales, B2B Tech Group Adam Goldstein Head of Production US & UK Mark Constance Head of Design Rodney Dive FUTURE US, INC. 130 West 42nd Street, 7th Floor, New York, NY 10036
All contents ©Future US, Inc. or published under licence. All rights reserved. No part of this magazine may be used, stored, transmitted or reproduced in any way without the prior written permission of the publisher. Future Publishing Limited (company number 02008885) is registered in England and Wales. Registered office: Quay House, The Ambury, Bath BA1 1UA. All information contained in this publication is for information only and is, as far as we are aware, correct at the time of going to press. Future cannot accept any responsibility for errors or inaccuracies in such information. You are advised to contact manufacturers and retailers directly with regard to the price of products/services referred to in this publication. Apps and websites mentioned in this publication are not under our control. We are not responsible for their contents or any other changes or updates to them. This magazine is fully independent and not affiliated in any way with the companies mentioned herein. If you submit material to us, you warrant that you own the material and/or have the necessary rights/ permissions to supply the material and you automatically grant Future and its licensees a licence to publish your submission in whole or in part in any/all issues and/or editions of publications, in any format published worldwide and on associated websites, social media channels and associated products. Any material you submit is sent at your own risk and, although every care is taken, neither Future nor its employees, agents, subcontractors or licensees shall be liable for loss or damage. We assume all unsolicited material is for publication unless otherwise stated, and reserve the right to edit, amend, adapt all submissions. Radio World (ISSN: 0274-8541) is published bi-weekly with additional issues in February, April, June, August, October and December by Future US, Inc., 130 West 42nd Street, 7th Floor, New York, NY 10036. Phone: (703) 852-4600, Fax: (703) 852-4583. Periodicals postage rates are paid at New York, NY and additional mailing offices. POSTMASTER: Send address changes to Radio World, PO Box 1051, Lowell, MA 01853.
A stream of thought … on streaming It’s not the also-ran it used to be, so we’ve got to get it right
F
or radio stations, streaming of our programming has changed considerably over the past decade. In the early days, it was a novelty. It represented just another way for listeners to “tune in” Cris to their favorite stations. There was Alexander little in the way of value in an internetCPBE, AMD, DRB delivered stream, but many stations Tech Editor got on the bandwagon for various reasons, including (and perhaps especially) because the competition was doing it. Comment Those early streaming efforts were on this or any … primitive (to put it nicely). Bit rates story. Email rweetech@ were often low because bandwidth gmail.com. costs were so high in those days, and the streams sounded, well, not great. They were full of artifacts, and many exhibited that watery sound that was and is so common to low bit rate streams. Most were either unprocessed or poorly processed — wrongly processed is probably a better descriptor, and that didn’t help matters much on the listener’s end. As time went on and bandwidth costs came down, bit rates were improved and things started sounding better. We began to figure out how to monetize our streams to make them pay for themselves at the very least and be somewhat profitable at best. This roughly coincided with reduced costs for mobile data plans, making it possible for listeners to connect to our streams using their mobile devices.
Squeaks and chirps We started figuring out how to process our streams right, using processing algorithms specifically created for reduced bit rate audio. This made a big difference in our 32- and 64-kbps MP3 and AAC streams; they became downright pleasant to listen to. Gone were the watery sound and chirpy, squeaky artifacts. And because we weren’t trying to beat the guy(s) Please recycle. We are committed to only using magazine paper which is derived from responsibly managed, certified forestry and chlorine-free manufacture. The paper in this magazine was sourced and produced from sustainable managed forests, conforming to strict environmental and socioeconomic standards. The manufacturing paper mill and printer hold full FSC and PEFC certification and accreditation.
3
THIS ISSUE THIS ISSUE NEWS x 3
xxx From xxx the tech editor
FEATURES x 4
More on xxx xxx toroids
BUYERS A primer on 8 GUIDE the politics
and protocols xxx of x streaming xxx
OPINION Take a page 18
xxx from the IT x handbook xxx
From the Editor
4
across town in a loudness war, our streams could actually have some dynamic range. With the advent of hybrid radio, the quality of our streams has taken on a whole new level of importance. When the blend to the stream occurs in the listener’s radio, we want to avoid big difference in sound quality. It should be imperceptible, or almost so. The challenge is much the same as it has been for hybrid HD Radio for the past 15+ years: The blend has to sound good. Streaming encoders have come a long way, too. Most, as far as I can tell, are still software encoders running on the station’s own hardware, but both the encoding software and the hardware are getting better and better. In addition to traditional sound cards ranging from under $50 to over $1,000, stations with AoIP studio infrastructure have the option to feed their encoders via IP using PC drivers compatible with whatever AoIP system they are using. In the bigger markets, my company employs Dell Precision rack-mount workstations with some horsepower,
Readers Forum
More on toroids Nice explanation of toroid core lightning protection by Jeff Welton in the Oct. 20 issue of Radio World Engineering Extra titled, “Ferrite Toroids Can Be an Engineer’s Best Pal.” To expand on that: I like to pass conductors through toroid cores several times, when practical, to get even more protection. Also, I have always coiled 120 VAC power cords to use up extra cord length and to provide inductance to a lightning path, even when a toroid core is used. More is better. Mark Persons, CPBE, W0MH The author is a Radio World contributor, retired contract engineer and recent recipient of the SBE Lifetime Achievement Award.
Above Special low-bitrate processing algorithms properly used can make your stream sound great. Shown is the GUI for a Wheatstone processor.
dual-port NICs and employing multi-channel AoIP drivers to produce the streams. These machines can easily handle streams for multiple stations with multiple streams for each station. Streaming service for our stations is provided by Triton, and we use their proprietary encoder software on these workstations. Purpose-built hardware encoders are now available from various manufacturers that will accept a variety of input sources, including AoIP. These encoders will produce multiple streams in different formats and with different data rates. Some include integral audio processing, providing a “one-stop” encoding solution.
Making it pay As noted, monetizing our streams is important in many stations. Preroll and midroll audio or video ads, ad replacement and programmatic ad sales are elements that we have to deal with in many cases. Some of those things
Those early streaming efforts were … primitive (to put it nicely). radioworld.com | December 15 2021
From the Editor
6
require special triggers from our playout systems, and that can be really tricky. RIAA reporting for royalties is another important element that we have got to get right. Underreporting or missed reporting can get us in hot water legally; overreporting can cost us in unnecessary royalties. This reporting is a function of metadata export from our playout systems. Some of the same metadata export is used to provide title/artist display on media players, and even to trigger lookup of album art for player/app displays, another reason to get it right. Listeners get irritated when incorrect information — or no information — is displayed. I know I do. On a related note, stream listeners who don’t show up in the Nielsen data don’t count, so we’ve got to get our watermark encoding right. Program directors and station managers get really irritated when PPM doesn’t pick up on listeners that we have. In this issue of RWEE, Rick Bidlack, a development engineer at Wheatstone Corp., will walk us through the building blocks of a stream. There’s a lot at work here, from the playout system, the encoder, audio processing, metadata aggregation and conversion, and transmission to the masses. There are plenty of places for things to go wrong, but the better our understanding of the signal flow, the better equipped we as broadcast engineers will be to deal with it. I remember a decade or more ago pulling out my iPhone and telling the president of our company that this was the
“
Above Hardware like this Dell Precision rackmount workstation using an AoIP driver can handle multiple stations with multiple streams each.
“transistor radio” of the future. In a lot of ways, I missed that prediction by quite a bit; nobody walks around with an iPhone in their shirt pocket audibly playing a radio station stream. But in some ways I was right. There are plenty of folks using their smartphones to stream audio to their Bluetooth earbuds, and there are just as many that play streams through a Bluetooth-connected audio system, in cars, homes, even airplanes. And listeners in cars equipped with hybrid radio will sometimes listen to our streams and won’t even know it. The bottom line for us is that more than ever, our internet streams have got to be right. They have to be reliable, meaning that someone (or something) has got to monitor them to be sure they are up. They have to sound good, be artifact-free and carry the correct metadata. Ad replacement has to be smooth and seamless, with overlays fitting exactly. And watermark encoding has got to be done right, too. There is a lot at stake. We can’t afford to be anything but diligent.
Underreporting or missed reporting can get us in hot water legally; overreporting Learn more can cost us in unnecessary royalties.
Read about streaming trends and best practices in Radio World’s free ebook “Streaming for Radio in 2021.” Find it at www.radioworld.com/ebooks.
radioworld.com | December 15 2021
White Paper
natrot/Getty Images
8 Writer
Rick Bidlack Development Engineer, Wheatstone Corp.
A primer on the politics & protocols of streaming
What to know about getting your streams cloud-ready and CDN-compatible This article is based on a paper prepared for the 75th NAB Broadcast Engineering and IT Conference. Presentation videos and proceedings are available at https://nabpilot.org/beitcproceedings/.
T
he broadcast industry is in a transitional phase, heading toward a world in which a large portion, perhaps even a majority, of listeners will receive the station’s program via an internet stream rather than terrestrial radio broadcast. Many accomplished engineers and station managers, while experts in traditional radio technology, find themselves feeling like neophytes once again when it comes to internet streaming. This paper aspires to present an instructional overview of what’s involved in this generation of an internet audio stream, as well as a better understanding of where standards and practices are still developing.
Signal flows Streamed content starts out on the same path as broadcast content, typically originating from the automation system if it’s music or from the studio microphone if it’s voice, or both. One, however, goes to the broadcast chain and the other goes to the streaming encoder, and that one difference introduces new considerations in both processing and metadata for streaming. To understand these considerations, it helps to get a closer look at the signal flow of each.
Broadcast signal flow For broadcasting purposes, content is fed into an automation system, which sends its audio output to an FM (or AM) processor, where it is processed and multiplexed, and then passed on to a transmitter that disseminates the program to the public.
radioworld.com | December 15 2021
“ White Paper
The stream encoder is also a mediator/ translator between the automation and CDN, transforming the format of what it ingests into one that the CDN requires.
As a side channel, the automation system has also been stocked with metadata (artist, title, album, duration, label, ISRC, etc.) for every piece of audio in its playlist; this metadata is also transmitted to the FM processor, where it is turned into an RDS signal that rides as a sideband on the MPX output. There are, of course, variations in this flow path. For example, the metadata from the automation machine may be routed instead to an “aggregator” that cleans and ensures the accuracy and consistency of the metadata before it is forwarded to the FM processor. Or it might not even go to the processor, but to a separate RDS encoder, the output of which is then mixed with the MPX from the processor. The exact details may change, but the steps along the path remain essentially the same.
Stream signal flow For streaming purposes, content enters the automation system as before, but in this case the output is routed to a stream encoder (also called an “origin server,” for reasons that will be clear later), and from there to a CDN, or Content Distribution Network. The stream encoder has three jobs:
10
1. Process and condition the audio signals, optimizing it for the compression algorithms; 2. Encode, packetize and transmit the program over the public internet to the destination server (the CDN); 3. Handle the reformatting and forwarding of metadata from the automation system on to the CDN.
CDN streaming and add-ons Before looking at the inner workings of the stream encoder, let’s examine the Content Distribution Network. The CDN
Automation System Audio
FM Processor/ Multiplexer Audio MPX
Metadata
Automation System Audio Metadata
RDS
Stream Encoder
”
plays the role of the transmitter in the streaming paradigm, but its unique position in the program flow path actually gives it a bigger and more important job. The main function of the CDN is to serve your stream to thousands or tens of thousands of listeners. But the fact that your program and all associated metadata pass through the CDN’s servers, and they know who is listening from what location and for how long, gives them an opportunity to provide a whole suite of add-on services. A big one is ad replacement, which is usually geographically based but could also be tailored to whatever can be deduced about the individual listener’s tastes and habits. Geo-blocking, logging, skimming, catch-up recording and playback, access to additional metadata (e.g. album art, fan club URLs), listener statistics and click-throughs, customized Below players, royalty tracking, redundant stream failover, Typical signal flows transcoding from one format to another — these are some in broadcast, top, and in streaming. of the services that CDNs typically provide. Thus, the CDN basically controls the distribution of the stream to the listening public. Transmitter It is the responsibility of the stream encoder — the origin Listeners server to the CDN’s ingest and distribution servers — to make sure that the CDN gets the right data at the right time and in the right format. Especially with regard to metadata, the CDN determines what the format will be. The CDN stream encoder is therefore also a mediator/translator between the automation Listeners system and the CDN, as it must be able to transform the format of whatever it ingests into the format that the CDN
radioworld.com | December 15 2021
White Paper
Metadata
CDN
Lua Transform Filter
DSP
Audio
Stereo /Mono
EQ
Encoder
AGC
Packetizer
AAC OPUS MP3
Limiter
Compressed
ICE RTMP HLS
CDN
Packetizer RTP
Linear
SSL/STL
Monitor
14 Above The encoder’s central function is data compression.
requires. With that in mind, let’s take a closer look at the stream encoder.
Job 1: Data compression The stream encoder performs various functions. The central function, which all others are designed in relation to, is to apply data compression to the audio stream, thereby reducing the bandwidth required both to transmit the stream up to the CDN, and for the listener to receive it. By far, the most widely used compression codec for high-quality audio storage and transmission is AAC, but its predecessor MP3 is still used for legacy streams (to support legacy players). All compression algorithms operate by reducing the information contained in the audio signal to as few bits as possible; you therefore want to maximize the
Even in the age of digital streaming, keeping entropy at bay is still the constant battle.
value of those bits by removing extraneous artifacts from the audio signal before it hits the compression codec. That cleansing and conditioning of the raw audio signal is the job of the DSP section at the input.
Encoder handoff to CDN The compression codecs produce periodic output in the form of chunks of data, each of which represents a small segment (typically 5 to 20 milliseconds) of the continuous audio signal. These chunks are wrapped in a transport format and transmitted to the CDN’s ingest server as discrete, time-stamped packets. Common transport protocols include Icecast, RTMP, HLS, MPEGDASH and RTP. Metadata is typically received by the stream encoder on a TCP or UDP socket, and most commonly arrives formatted as XML. What happens after that depends on the transport protocol being used. For Icecast streams, metadata updates (including ad triggers) are sent to the server out-of-band, as separate HTTP messages. For RTMP, the metadata update is embedded in the transport stream itself, as a special INFO type packet. For HLS, metadata may be reformatted as ID3 data and embedded into a separate channel of the underlying MPEG2 Transport stream, and ad triggers can be woven into the manifest file as SCTE-35 “program replacement opportunities.”
radioworld.com | December 15 2021
White Paper
Read the Proceedings
Radio World thanks the organizers of the NAB Broadcast Engineering and IT Conference for their cooperation in bringing this and other event content to our pages. We encourage readers to explore the available resources at https://nabpilot. org/beitcproceedings/.
16
For all of these methods, the exact details can differ from one CDN to the next since there are no universally accepted standards for handling metadata, so setting up a stream and getting the metadata to update correctly and in sync with the audio stream often requires a short period of trial and error while negotiating protocols with the CDN. RTP is a special case. RTP is the only stream format that is commonly used to carry uncompressed, full-bandwidth audio (in the form of AES67, for example), although it can be used to carry virtually any kind of data, including AAC, Opus or MP3 audio. RTP is typically used in a studio-to-studio or studio-totransmitter link, rather than as an ingest feed to a CDN.
Metadata formatting The final area we want to examine is the metadata itself. We’ve been talking about transforming it from one format to another, but what does that mean, and why do we need to do it? To answer, let’s look at some examples of raw metadata received from various automation systems. (Names and numbers have been changed to protect the innocent.) Some systems export simple tagged text: artist=Sonny Rollins title=God Bless the Child length=00:07:29 More commonly, metadata arrives in XML or XML-like form. Note that many fields may be empty: <nowplaying><sched_time>220200</sched_ time><air_time>402000</air_time><stack_pos></ stack_pos><title>TheSkyIsANeighborhood</ title><artist>FooFighters</artist><trivia>*</ trivia><category>MNJ</category><cart>R585</ cart><intro>14000</intro><end></end><station>93.5HD1</ station><duration>243300</duration><media_type>SONG</ media_type><milliseconds_left></milliseconds_ left><Album>ConcreteandGold</Album><Field2></ Field2><ISRC>USRW9170028</ISRC><Label>RCA</ Label><Tempo></Tempo><Year>2017</Year></nowplaying> The HTML ampersand entity in this one will require special handling: <audio ID=”id_3155460704_30756200”> <type>Song</type> <status>Playing</status> <title>JINGLE BELL ROCK</title> <artist>Hall & Oates</artist> <length>00:02:03</length> <category>Classic</category> </audio>
Here’s a station ID and a liner from the same source. Some CDNs may want this data, some may not: <audio> <type>Link</type> <status>Playing</status> <artist></artist> <title>ROCK91.7 | LEGAL ID2</title> <number>120002</number> <length>00:00:11</length> </audio> <audio> <type>VoiceTrack</type> <status>Playing</status> <artist></artist> <title>VT TLA3 - Rock: 2021-03-25 08:37</title> <number></number> <length>00:00:06</length> </audio> Here’s a sweeper and an ad from another source, below. Notice that the <media_type> tag distinguishes one from the other. The arrival of a SPOT is often used to trigger an ad replacement by the CDN. Durations shown here are in milliseconds. <nowplaying><sched_time>67199000</sched_ time><air_time>67267000</air_time><stack_pos></ stack_pos><title>CYQTONEQUICKSWEEP#12DRY</ title><artist>201971108:06:28</artist><trivia></ trivia><category>IM5</category><cart>BIBE</ cart><intro>0</intro><end></end><station>95.7KQED</ station><duration>3900</duration><media_ type>UNSPECIFIED</media_type><milliseconds_ left></milliseconds_left><Album></ Album><Field2></Field2><ISRC></ISRC><Label></ Label><Tempo></Tempo><Year></Year></ nowplaying><nowplaying><sched_time>33716000</ sched_time><air_time>33721000</air_time><stack_pos></ stack_pos><title>ShopLocalAWestfieldMerchantsFriday</ title><artist>WestfieldMerchantsJune</ artist><trivia>shoplocalgreencountry</ trivia><category>COM</category><cart>4652</ cart><intro>0</intro><end></end><station>95.7KQED</ station><duration>30000</duration><media_type>SPOT</ media_type><milliseconds_left></milliseconds_ left><Album></Album><Field2></Field2><ISRC></ ISRC><Label></Label><Tempo></Tempo><Year></Year></ nowplaying> None of the metadata messages in these examples can be transmitted to the CDN ingest server in the form they are in. Instead, the relevant data must be stripped out and reformatted into a protocol that the ingest server can
radioworld.com | December 15 2021
White Paper understand. This protocol often requires that additional information, which is not available in the original metadata received from the automation system (such as login credentials, the server URL, and query parameters) be woven into the message. For an Icecast stream, that might take a form like this: http://source:abc123@ice5.streamnet. com/admin/metadata?mount=/ BRAVOFM1im&mode=updinfo&song=Foo%20 Fighters|The%20Sky%20Is%20A%20 Neighborhood|00:04:03|SONG
stream encoders often provide a menu of Lua filters that are able to transform common patterns of metadata input into similarly common output patterns. Usually all that is required to customize the transform filter for the station’s own particular needs are a few tweaks to the script. In closing, we’ll show an example of one of the pitfalls of relying on metadata for accuracy, and why many stations are now employing metadata cleaners and aggregators in their flow path. This is a real message received from an automation system (which shall remain anonymous): <audio> <type>Song</type> <status>Playing</status> <artist>Beyonc�</artist> <title>Crazy in Love</title> <number></number> <length>00:03:56</length> </audio>
Likewise, an ad trigger might look like this: http://source:abc123@ice5.streamnet. com/admin/metadata?mount=/ BRAVOFM1im&mode=updinfo&song=|Shop%20LocalA%20 %2D%20WestfieldMerchantsFriday|00:00:30|COM In the stream transmitted by the CDN, the local ad from the origin server might be replaced with another ad targeted to the geographic location of the listener. The CDN may use the duration of the ad (30 seconds in this example) as the queue for switching back to the original program, or it may wait for the next SONG update to switch back. For RTMP and HLS streams, the metadata is reformatted in a particular manner and “injected” into the stream as in-band data. For example, for RTMP, a “setDataFrame” message is assembled, typically containing an array of three strings: title, artist and “url” (a container for everything else). With the Foo Fighters example from above as our input, the schematic form of the output, ignoring the header bits demarcating the various sections, would look more or less like this: @setDataFrame onMetaData title:The Sky Is A Neighborhood artist:Foo Fighters url:http://www.blaze105. com?autoID=R585&autoCat=SONG&cat=MNJ&album= Concreteand old&label=RCA&ISRC=USRW9170028 Note that the mapping of the tags in the original message to some of the query parameters in the output “url” string is neither obvious nor predictable, and in this case was in fact determined by the CDN. The underlying implication is that the manufacturer of the stream encoder cannot know ahead of time what format either the incoming or the outgoing metadata will look like. A common solution for effecting the transformation of arbitrary input to arbitrary output is a programmable, embedded scripting language such as Lua. Current
Obviously, Beyoncé is the intended artist, but how would a computer know that? The odd character sequence at the end of her name is actually a single character: the Unicode Replacement Character /uFFFD, or one representation of it. (The lozenge-question-mark � is another representation of the same character.) How did it get there? Some automated transcription program back in the day encountered a character it wasn’t designed to handle (in this case the é), and replaced it with a wild card. The problem for any metadata transform filter is that the wild card can refer to anything — there’s actually no way to know except from the context what the original character was supposed to be. And so there we are, at the interface of data and chaos. Even in the age of digital streaming, keeping entropy at bay is still the constant battle.
“
A common solution for effecting the transformation of arbitrary input to arbitrary output is a programmable, embedded scripting language such as Lua.
radioworld.com | December 15 2021
17
Radio IT
Take a page from the IT handbook FOSS applications can be adapted to create custom solutions for broadcast
I
18
consider it my mission to stay on top of tech — always reading, watching and digging. It is my passion to bring custom solutions to broadcast customers. I frequently am inspired by software, networking and virtualization technologies used across so many industries, tried and true solutions that easily could benefit local radio. Generally, IT infrastructure already exists in a broadcast facility in the form of routers, firewalls, switches and virtualization stacks from various vendors. It is imperative to make sure that this infrastructure is designed correctly, with security and functionality in mind. In our industry, most of us have become accustomed to working with less-than-ideal equipment, not always implemented with the best, most secure, most efficient design in mind. Whenever possible, I use free and opensource software (FOSS) tools for my customers, even “retrofitting around” pre-existing equipment when a redesign is not yet feasible. With FOSS, community-driven development allows for unique and powerful features, equal to or greater than proprietary solutions. In a time where security should be of the highest priority, I like knowing that source code is freeand-clear to audit at any time by anyone in the world. FOSS can offer significant cost savings to clients, and many times, make doing a project that would otherwise be costprohibitive possible. But misunderstood or misconfigured tech is often more problematic and insecure than not having it at all. I have a lab where I can test software, virtualization, networking and proprietary broadcast hardware and software, and I have remote access to broadcast equipment in the radio stations and labs of partners and clients all over the world.
Working remotely The pandemic has taught us that remote work, even for broadcasters, is surprisingly doable. With so many at home — aside from audio transport, more on that later —
Writer
Chris Fonte
Owner, Fontastic LLC
Above WideOrbit Automation for Radio running on a laptop at talent’s house connected via OpenVPN, with joint control of the radio station, audio from the RodeCaster Pro transported back to the radio station via Luci Live SE.
“
IPSec, OpenVPN and Wireguard are some of the most common VPN technologies, as well as proprietary vendorspecific ones.
”
connecting to PCs station-side with tools like TeamViewer, VNC and LogMeIn have been the go-to solution. But broadcast engineers like Jobie Sprinkle at WFAE in Charlotte, Henrik Poulsen at Radio NORDJYSKE in Aalborg, Denmark, and Tim Aquilina at 92.7 Mix FM in Maroochydore, Australia, have been looking for a way to give talent physical control again. And they are not alone. Air talent around the world have grown up pushing physical buttons and adjusting physical faders and knobs
radioworld.com | December 15 2021
Radio IT Fontastic Lab in South Carolina
192.168.87.0/24 WheatNet Network
PC OpenVPN
pfSense Router
To connect Fontastic to radioDNA’s WheatNet network, I’ve used OpenVPN running in pfSense. The Fontastic side has a public IP, while the radioDNA side is dynamic. And so the OpenVPN Server is running on the Fontastic side. OpenVPN is running with a TAP interface, bridged to the WheatNet interface in pfSense on both sides; this way the Subnet remains the same and a Gateway does not need to be configured for all devices. L2 mode lets traffic flow freely on the WheatNet Subnet bidirectionally on both sides.
Static Public IP
PUBLIC INTERNET
radio DNA Lab in Minnesota L2 Switch 192.168.87.0/24 WheatNet Network
PC OpenVPN
pfSense Router
Dynamic Public IP ISP Router
L2 Switch
without needing to see what they are doing because of muscle-memory. Today, it is possible to build the creature comforts at home that talent has been accustomed to in studios for decades, while embracing new technologies never before possible. It can be done, even on a budget, by combining tools used by IT professionals across all industries with the plethora of tools already designed for broadcasters.
Above Fig. 1: Extending the network with a VPN tunnel.
Security concerns It has always been considered bad practice to open firewall ports to internal services across the public internet, especially when the data is unencrypted. But I still see it happening at broadcast facilities all over the world. Today, security is of the highest priority, so, using encrypted VPNs with additional internal and external firewall rules in place are the way to go. IPSec, OpenVPN and Wireguard are some of the most common VPN technologies, as well as proprietary vendor-specific ones. SD-WAN, although a buzzword of late, is very powerful and allows for global networks of all kinds of devices.
“
With free and opensource software, community-driven development allows for unique and powerful features, equal to or greater than proprietary solutions.
In all cases, encryption at the highest level available should be used.
Getting into WheatNet remotely Working from home myself, I wanted to test the reliability of remote physical control of gear from Wheatstone. In my home office lab sits a Wheatstone Sideboard. It is connected to a full Wheatstone AoIP system over 1,000 miles away. The Sideboard gives me real, physical, tactile control over a Utility Mixer — an 8-channel virtual console in a 1 RU Wheatstone Blade. This is the intended purpose of a Sideboard, and it usually happens locally, but in this case, the Blade is in another time zone. With the Sideboard, I can select any source on the remote WheatNet system and make it available in the Utility Mixer I am accessing remotely. So while I am controlling the mix locally, the mix itself is happening in the remote lab. Wheatstone supports using their automation control interface (ACI) between devices over a network, both locally and remote. the Sideboard is just one device on their list of control surfaces that can do this. To make it possible in my lab, I have chosen a stack of FOSS tools including a router/firewall from pfSense with built-in OpenVPN. I set up an openVPN server on a static public IP address in my lab. On the remote-tome lab’s side of things, it is behind a consumer firewall with a dynamic IP address. Behind that firewall is a PC connected to the internal internet network on one Interface, and the WheatNet network on another. It runs an OpenVPN Client and connects over the internet to the server in my lab. In OpenVPN on both sides, I am using TAP Interfaces, bridged to each local WheatNet network. On the tunnel, there is no routing happening, it’s all layer 2, and so the devices on either side do not need a gateway defined to be able to talk to each other. In this way, I am effectively extending the same WheatNet Network across the VPN tunnel. See Fig. 1.
radioworld.com | December 15 2021
21
Radio IT Climbing the firewall Sometimes, due to time, budget or hardware constraints, setting up VPN tunnels between two sites via dedicated hardware is not always possible. This is where a newer technology called SD-WAN, or software-defined wide-area networking, can be helpful. ZeroTier One, Nubula and Tailscale are examples of this and employ magic (UDP hole-punching) between firewalls to establish connections between devices on an internal network on either side, without the need to insecurely open firewall ports or use hardware VPNs. All data is encrypted end-to-end and allows bridging and routing securely through the internet behind dynamic IPs behind firewalls on both sides. This is an exciting technology that is making waves across all industries, though setup and configuration can be a bit more daunting than standard VPNs.
Transporting the audio
22
Solutions for getting audio from point A to point B are in huge supply. Comrex, Tieline, Barix and others have complete lines of hardware devices that do this effectively and efficiently. And Wheatstone has the Blade 4 with audio codecs built in. Real-time audio in the virtualized world — software to software, and software to hardware — has become the Holy Grail in our ongoing pandemic world, and products that do this have come a long way, too. Some are able to use the high-quality, freely available (and FOSS!) Opus codec via SIP-managed connections; others are web-based like ipDTL and Cleanfeed. LUCI Software offers solutions for mobile, PC, Mac and Linux (think LUCI Live and LUCI Studio). These can work via SIP or direct connection, and have become my go-to solution lately because of their immense flexibility and lifetime licensing fees.
Blurring the lines Early this year, I heard from a client who wanted to hire a new afternoon talent, but she is located nearly 100 miles away from the radio station. Is this doable, and on a budget? The show would continue through the pandemic and beyond. I got to work designing and implementing a costeffective solution. The station is a customer of WideOrbit automation for radio and runs version 4.0, part of the newest incarnation, and provides “joint control” of each radio station. To obtain this control remotely and securely, all that is needed is a VPN connection. I turned to my trusty pfSesne/OpenVPN combo at the radio station, with an OpenVPN client running on a station-provided laptop at the talent’s house. WideOrbit’s RadioClient, a native PC application, connects to the station-side RadioServers through the VPN tunnel, and the talent can control the radio station as if sitting in
More info
Fontastic LLC is a broadcast services company focusing on software and IT, helping radio stations around the world with integration projects. Email: chris@ fontasticllc.com Twitter: fonte935
the studio. I configured workflows on hotkeys to route talent’s audio directly to air, take the feed off the air, send caller-audio to the talent, and route the backfeed to hear pre- and post-cut audio for voice tracking. The off-the-shelf laptop is using a Roadecaster Pro for its audio interface, with an Electro-Voice RE20 plugged straight in. It has a solid DSP-based mic processor with a preset for the RE20, a listen and record bus and allows a mix-minus for the talent to hear return audio mixed with outgoing audio. The client is particularly sensitive about subscription fees, so audio transport is handled via a one-time licensed version of LUCI Live SE on the laptop. This audio stream, along with WideOrbit control, is sent via the OpenVPN tunnel. Station-side, a Windows virtual machine with an AoIP driver, is running LUCI Studio. LUCI allows different send and receive codecs, tailoring the codecs to the use-case. In both directions, I wanted the lowest latency possible. I wanted return audio to be stereo, so that the talent feels like they are mixed well with the music. And so audio received from the talent uses a low-delay, mono, high-quality AAC codec. Return audio uses a stereo AAC codec with low delay and slightly lower quality. Currently, LUCI Studio is handling one bidirectional stream, but it is capable of 64. Finally, phones are handled via Broadcast Bionics’ Caller One, running on a virtual machine at the station. Calls are answered via a web browser over the VPN remotely, with caller audio transported back to the talent via LUCI Studio, mixed on the Rodecaster Pro, sent back to the station as mix via LUCI Live SE, received by LUCI Studio, and recorded into WideOrbit as a produced package. All of this does not feel remote for the talent, does not sound remote for the listener, and does not have an ongoing cost for the client. Remote work is now a part of our lives across all industries. We live in a fantastic time of technology, where so much is available. And, now more than ever, it is possible for broadcasters, too. Doing it securely should be of the highest priority.
“
All of this does not feel remote for the talent, does not sound remote for the listener, and does not have an ongoing cost for the client.
radioworld.com | December 15 2021
9000