13 minute read
Disaster ready
In a recent report into the impact of a fire at Channel 4’s playout centre, Ofcom warned that broadcasters need to urgently look at their disaster recovery plans. TVBEurope hears from four experts as to how broadcasters can plan for such a situation and whether a move to an IP infrastructure would stop it happening again
HOW CAN BROADCASTERS ENSURE A SITUATION LIKE THE ONE THAT HAPPENED LAST YEAR DOESN’T HAPPEN TO THEM? Neil Anderson (NA), CRO, Codemill: Don’t rely on a single service provider or platform, if you can afford it! There’s a lot to unpack from the Ofcom report. It was a catalogue of errors, not just a single point of failure, that led to such a prolonged outage of access services. Risks were taken, seemingly without mitigation planning, along with a lack of disaster recovery testing by their service provider, which was a key factor. The lack of a business continuity plan for access services during disaster recovery seems to have been the single biggest strategic failure.
Ignacio Revuelto Rosello, product marketing manager at Imagine Communications: At a very basic level, broadcasters must acknowledge the potential risk and proactively plan for it. Everyone is at least aware of the concept of disaster recovery and remote back-up facilities. The problem for many, I think, is in determining what they want specifically from their disaster recovery plans.
There are two metrics used in the wider world of business continuity: recovery time objective (RTO) and recovery point objective (RPO). RTO is defined as the maximum time that the loss of a business function impacts the organisation; in broadcast, the function would be the output of a channel. RPO is the point in time to which systems and data must be recovered after a disaster; in our terms, when the content and playlists are synchronised.
Those are the key factors when you are specifying disaster recovery plans, however you choose to implement it. How fast do you need to be back in full operation?
Venugopal Iyengar, COO, new business, Planetcast International:A robust disaster recovery system harnessing cloud-based technology is one of the best, most cost-effective ways to prevent such situations. Good disaster recovery isn’t just about moving to the cloud, however. Broadcasters should implement a solution that is also facility- and location-independent to minimise risk. For instance, if the back-up is essentially in a specific building in a particular area that is close to the primary facility, then it is exposed to potential geographic risks that can impact the entire region. Indeed, this recent incident underlined the need for facility- or location-agnostic solutions enabled by a cloud-based approach.
DO THEY NEED TO MOVE FURTHER TO THE CLOUD AND AWAY FROM ON-PREMISE INFRASTRUCTURE? NA: Public cloud provides unlimited infrastructure and services, with levels of resilience, redundancy, failover and recovery that hardware vendors or service providers can never achieve, with their own infrastructure. Cloud outages can happen, but these are typically only for a short period of time (seconds, minutes or at worst hours). This risk can be easily mitigated by deploying channels in more than one availability zone (i.e. 2N), in two or more geographical regions (2N+1) or even in multiple public cloud service providers.
Failover can still be instantaneous, or disaster recovery services can be spun up in seconds, on-demand, so you only pay for it, when you need it. Overall, public cloud has made true disaster recovery and business continuity far more achievable and affordable through opex, than spending capex to build a fully redundant, hardware-based playout chain, where there will always be a single point of failure, somewhere.
Denis Genevois (DG), marketing and communications VP, Globecast: A TV platform part of a disaster recovery plan needs to be switched on when an outage occurs on the main system, or when the DR plan is tested. The second case happens to a regular schedule. So, most of the time, such a platform is in a dormant state. Cloud is a cost-effective solution for ad-hoc use of such infrastructure, with a pay-as-you-go model. But the virtual site running in the cloud is not isolated, it needs connectivity for ingesting live feeds and non-linear video content and for linking the output TV signal to TV delivery platform and monitoring/ NOC. So, the network to and from the cloud is really important.
WILL THE MOVE TO IP INFRASTRUCTURE HELP PREVENT INCIDENTS SUCH AS THE ONE AT CHANNEL 4? DG: An IP-only video network is mandatory when working with the cloud. It isn’t with back-up physical remote sites that can use traditional broadcast wiring. But, if you build a modern and new DR
plan, it should be IP-based and cloud-based. Even a physical DR site can be part of a hybrid cloud design (public cloud and on-site).
VI: The shift to IP infrastructure is fundamental to enabling effective disaster recovery today. If your infrastructure is SDI-based, you can’t virtualise or design a cost-effective multi-facility architecture, let alone harness the cloud-based workflows required to achieve a best-in-class disaster recovery position that is facility- or location-agnostic.
HOW OFTEN SHOULD BROADCASTERS BE TESTING THEIR DISASTER RECOVERY SYSTEMS? DG: It’s a learning curve. It should be more frequent during initialisation to both gain confidence and trust, but also to validate and train people to activate the DR procedure. The difficulty is to really test scenarios that are listed in the DR plan and not just a part of it. The DR plan may also not be comprehensive. It’s probably a point for improvement where the industry, broadcaster and regulatory bodies gather and share best practices and DR scenarios.
IRR: As we move towards cloud disaster recovery, I think we move away from the idea of pulling the plug out of the main playout feed and seeing how long it takes the backup to cut in. Now broadcasters need to look at audit testing, to define what level of services need to be always running, to start up everything else should it be needed.
AWS published useful references in its whitepaper titled ‘Disaster recovery workload on AWS’ about the levels of disaster recovery; starting from a back-up and restore architecture with an RTO/RPO of hours but a low cost, up to hot standby with RTO/RPO of zero but very high cost for a lot of running processes. A useful stage is what they call ‘pilot light’, which is the minimum set of functionality, defined by the audit of your workflows, which ensures content and playlists are maintained in synchronisation and which can start playout processes quickly when needed. Testing then becomes stress-testing your design to be sure that it will respond seamlessly when you need it, rather than having a monthly rehearsal of full-scale system failure.
VI: It’s critical to frequently test disaster recovery systems; at least once a month or after any significant infrastructure update. Frequent testing is essential to ensuring that your DR solution works – and works in good time – when needed.
In addition, a cloud-based disaster recovery system can be easy and less costly to test regularly. Modern cloud-based DR runs in a passive mode where content is ingested and schedules are updated continually but the playout portion is dormant, which offers a very low running cost. Once a month, the DR system can be fired up into active mode through a largely automated testing process that can ensure switchover and playout are running as expected in the event of a crisis. In fact, at Planetcast International we’ve tried to ease this burden by offering ‘pay as you go’ models which have a low maintenance cost and an event-based or requirement-based cost increment that kicks in only when used. This ‘only-when-you-need-it’ cloud-based approach can provide a more flexible commercial model for broadcasters concerned about the expense of an active/active on-premise DR set-up.
WHAT WOULD YOUR ADVICE BE TO ANY BROADCASTER WHO FEELS THEIR DISASTER RECOVERY INFRASTRUCTURE IS CURRENTLY LACKING? IRR: My advice to any broadcaster, whatever their perceived readiness, is to take a fresh audit of what they want from disaster recovery. We talk about ‘business continuity’ because these are business decisions. You must evaluate the importance of protecting your playout services, as an outage will negatively impact your brand. This has the potential to impact your ratings, market share, and indeed your revenue, because audiences won’t have seen a scheduled advert, which means you cannot bill the advertiser for this. Additionally, such an outage may force audiences to switch to more reliable competitor channels losing you much more for your business over time. So how much are you prepared to pay for what level of readiness? You should identify a technology partner – like Imagine – who can help you through this audit, to determine what levels of RTO and RPO you need to meet your business and regulatory demands, and how best that can be implemented.
VI: I would recommend doing a simple cost analysis of how the damage of an hour, a day or several days of outage would impact the business; both in terms of lost revenue and reputational damage.
Next, consider if your current disaster recovery solution – if you have one – could withstand a regional issue. Embracing a location-agnostic infrastructure set-up is key to futureproofing your disaster recovery system. Broadcasters also want service level agreement assurances, often a key concern when considering a move to cloud-based disaster recovery. With that in mind, identify a proven technology partner with the depth of international technical expertise, round-the-clock support and a demonstrated track record that ensures reliability and performance. Don’t just take a supplier’s word for it: engage in proof-of-concept demonstrations to try it out for yourself.
Broadcasters should also look to suppliers with existing cloud playout and media asset management expertise and technologies to ensure a seamless migration to cloud workflows. Make sure the parallel cloud DR solution can be integrated with your existing system; the DR needs to work with what you already have in place.
Media brands should treat the transition to cloud-based disaster recovery as part of the broader, ongoing journey to the cloud. Use cases such as pop-up channels, for which investment in heavy on-premise set-ups makes little sense, are a great use case for cost-effective delivery via cloud-based platforms. The flexibility and scalability of the cloud not only help enable cost-effective DR and pop-ups but, as part of a broader hybrid architecture, improve overall business agility. n
‘I LOVE THE INDUSTRY’S APPETITE FOR CHANGE’
Paul Clennell, chief technology officer, dock10, discusses his love for the media tech industry, and why he’s worried for the next generation
How did you get started in the media tech industry? I grew up wanting to be a musician and was classically trained for many years. In my teens, one of the orchestras I was performing with produced an LP (showing my age). Being involved in the process of recording and mixing the final result proved fascinating and set me on a new path of audio and music technology. My university studies focused on this and software engineering; leading to my first job in the industry as an embedded software engineer for an audio signal processing company called Klark Teknik. A couple of subsequent jobs and promotion to project manager together with the realisation that adding pictures (TV) opened up a whole new world of bigger opportunities and larger projects led me, eventually, to my role at dock10!
How has it changed since you started your career and what makes you passionate about working in the industry? So many changes that I’m not sure where to start! During my studies, non-linear (the ability to make instantaneous changes at any time, anywhere in a sequence) was a big thing for audio, but pretty inconceivable/cost-prohibitive in the video world. As processing power and storage increased exponentially (along with some ingenuity from the likes of Bill Warner), those principles, skills and high-level workflows transferred very readily into television; triggering a number of transformation projects that fundamentally changed how the major broadcasters and their suppliers worked. The industry and its consumers seem to have an insatiable appetite for greater resolution, be it more pixels, colours or audio samples which have generated countless standards that continue to evolve at pace to deliver an ever more immersive consumer experience. We are always exploring new ways of doing things; more recently pushing the boundaries of remote technology (and borrowing from other sectors) to overcome the obstacles presented by the global pandemic. I guess it’s one of the things I love most about this industry: its appetite for change which ensures there’s always something really exciting to get stuck into.
If you could change one thing about the media tech industry, what would it be? We know our industry isn’t anywhere near as inclusive as it should be. The entertainment and media industry has a particular responsibility because young people develop their views and take their cues from what they see and read. Sadly, certain groups do not see themselves reflected in the content that’s available and consequently do not see a path for themselves into a career creating that content. For various reasons, career opportunities are not evenly distributed and apprenticeships are very limited. There’s still a huge amount we can all do, from raising awareness of barriers to inclusivity in the workplace and actively tackling them to supporting fantastic organisations such as Rise.
How do we encourage young people that media technology is the career for them? Technology is so varied and media technology is very relatable to a huge area of our lives. We need to ensure that in schools the subject is taught with context so it doesn’t become ‘boring’ – not just how to use technology, but why – that’s the really interesting bit. Our team are keen STEM ambassadors and we all enjoy talking to students about careers in technology.
What’s the biggest topic of discussion in your area of the industry? We’re very focused on the next generation of virtual production, using the latest real-time games-engine technology from Epic and Unreal Engine with Zero Density, together with the Mo-Sys StarTracker system, and encouraging entertainment formats to adopt these exciting technologies more widely over the next few years. There is a continued pressure to reduce budgets, but channels still demand new and exciting content, and this is where virtual studio elements can make a creative difference. The big advantage with using virtual studios is the ability to refresh a set or change elements that would otherwise be extremely difficult or costly with a physical build. Virtual production technology enables our customers to make brand updates or special editions with ease and explore creative ideas and try different elements without committing to a major build cost. This is an advantage that just wasn’t possible before!
What should the industry be talking about that it isn’t at the moment? The advent of widespread remote working has been a lifesaver for our industry during the global pandemic and many (including me) continue to enjoy the added flexibility that remote working supports. Avoiding a regular commute and optimising a work/life balance is great, but I’m worried for the future. Our industry thrives on collaboration and technology is a huge enabler of this, particularly for established relationships. My concern is about the challenges the new intake faces. When I started working, it was a given that I was surrounded by colleagues I could observe, pester and bond with. We need to be talking more about how we afford this opportunity to our newest colleagues to ensure they’re not missing out. n
