Page 1

Wagner College ​hello my name is Brian Largent I'm with the ArcLight group in Tulsa Oklahoma today I'm going to talk a little bit about the allscrips ransomware it's hack that happened on about January 18 2008 teen we don't know exactly when it happened but that's when the reports kind of started rolling in all we know for sure is what all scripts has posted to the public and that is they had a ransomware attack that took some of their systems down for a small subset of their organization and they're working on getting that remedied in an operational again so what do we know from their Twitter feed well we know that certain customers are there's have have claimed to have been down for more than 24 hours with massive impact of not being able to schedule surgeries not be able to see patients and so on so I wanted to talk about sastra covery planning because this is a really important topic and and what's happening to all scripts could happen to anyone now I don't know exactly what's happening to all scripts but I know how to recover from a ransomware attack because we've had to deal with it for customers that have called us out of the blue saying they have a problem so let's dive into a it's just a 9-point PowerPoint presentation it's not a lot of PowerPoint so I won't kill you with PowerPoint but we're going to dive into this and I'm going to try to cover some of that and then we're gonna answer just a couple of questions along the way so here we go so there's three basic types of server disasters there's a hardware failure that's instantaneous there's a data corruption and then there's a computer virus we're gonna explain each of these in a little detail a hardware failure that's instant usually is a natural disaster or a accidental damage or an actual hardware failure itself where like a power supply dies or or something like that when a hardware failure happens like that it's usually pretty instantaneous data is going in and then it stops going into the system and it's dead that means that recovery is pretty simple you just take your backups your most recent backup and you restore it on to another server and you should be operational fairly quickly that is if you have good enterprisegrade backups and and systems in place the second type is data corruption now this one's a little bit messy when you have data corruption you really have to get into the system and figure out what's causing the data corruption get that remedied then have you have to figure out where the corruption resides and when it started if you can figure out where the corruption resides and when it started then you can go back to a certain point in time and restore data now you're probably going to lose some data in this type of issue because people are putting data in but it's being put in and then corrupted therefore you have nothing to back up except corrupted data so if the if the data is corrupted on old data then you can probably restore old data from a point when it was actually not corrupted so there's a little flexibility as there's a few things you're gonna have to work around but data corruptions not as bad as computer virus when you have a computer virus well let's let's talk about what is a computer virus it's malicious code or program written to alter the way a computer operates and it spreads from host to host by self-replicating and requires a host to lift a host is just a computer a server something like that so if you have a computer virus on your system it wants to replicate to everywhere in the network so trying to figure out where all the places the virus is replicated to is very difficult we've done it ourselves for customers that have called us in a panic they have a virus on their system and they need to have us go in and figure out where it's at and get it cleaned off and get their systems operational again now we've never had that for a customer that is a full service customer that that you know relies on us for protecting their network but we have had companies call us out of the blue and say this is going on please help us so we've got a lot of experience with it in the case of the allscrips attack what they had was they had a crypto ransomware virus so what it does is it wants to self-replicate across the network and then it starts to encrypt files well what kind of files is it encrypt well here's a list of the locker ransomware which is one of the more modern and up-to-date ransom words that are out using an aes encryption algorithm very hard to decrypt without a key pretty much impossible so what kind of files is it encrypting well here's a list well if you look at the gray arrows the first thing I want you to look at is the VMware it's pointing to a file called V in dk a VMDK file is a virtual server that is the entire operating system and applications anything stored in that becomes encrypted the entire server is offline we're not talking about just some data we're talking about the entire operating system the applications that are installed and possibly data if you have a data based on the same server as your app when we talk about the sequel its encrypting sequel light sequel light DB sequel MDB

files LDF MDF these are all Microsoft sequel files in various different third party sequel files if they become encrypted you've lost your data you may have your servers operational but your database is encrypted so you have to go back to a point in time and restore that so there's different ways to recover from these instances and usually companies will take that both ways and try to work down both these tracks at the same time so I have a light gray in a light blue track here if you go down the light gray track that's when you're just trying to clean the system up and get it operational so you first before you can do anything no matter what your track is you have to isolate the system you do not want information going in and out of the system when you're trying to figure out how to recover the system from a virus or in this case a ransomware virus so you isolate the system now you've got customer screaming saying we can't get our systems we're not operational so that the clock is ticking you need to get them operational as quick as possible the first thing you need to do is you need to discover the cause and the earliest instance of the virus because that's going to tell you how far back you're gonna have to go to restore your data then you're gonna try to clean your system and recover the files if it's just a couple of files got encrypted you can then possibly if you can narrow them down to know where all the infections are you can clean those off recover those files do some testing and then return everything to service you got to have a pretty high confidence level that you've got it cleaned off it's possible you might be able to do that but more often than not when you have a virus a very smart virus that knows how to self-replicate you have to go through this process and that is to isolate the systems discover the cause in the earliest instance then you have to restore your still server OS and apps that means you have to go back to your backups to a point in time before the virus showed up restore all of that data on to new hardware new infrastructure then you apply the missing updates like your windows patches and sequel updates and so on and you have to apply the patches and fixes for the application itself a company like all scripts is going to be applying updates feature releases and everything all on the way so if the virus happened on the 18th and they're restoring data they may have to restore and patch going back you know two or three days or however long before they can be operational depending on how long the virus is on there and how long it continued to run after that and people were putting data in and out of the system so once you've done that then you need to restore your most recent data depending on how your infrastructure is built will determine how that's done for instance if you've got all your sequels safe and it's secured and it didn't get touched by the virus then you might just be able to mount your databases back up to your servers in your application and get it operational pretty quick but after all that you still need to do internal testing because the applications themselves have had new patches and new updates and you may have missed something your employees or your your vendor may have missed something they know I got everything patched you don't want to just rush into production immediately so you do your internal testing if internal testing patch passes then you want to do your production testing and you don't want to roll that out usually to all of your customers you want to roll it out to a small subset of customers those customers if they if they're happy and things are working then you're going to roll it out to a few more customers and a few more customers once you've reached a saturation point where you have enough customers that are successfully getting on and everything's working you can return the entire system to service and be ready to answer any support questions as they come in so how do you keep this from happening to begin with because obviously it's going to be a nightmare to try to come out of something like this it's not like a server failure which is what we all think of whenever we have a disaster recovery plan we think of fire flood tornado hurricane but those are not what's going to get you it's going to be a virus it's going to be something like that where user interaction is creating the problem so there's some things you can do the internal defenses you can do or you lock down your systems and restrict unnecessary access do not let every employee have administrative access don't let every employee have full rights to system don't let them have full rights to their local computers if they don't need it they need to be able to do their job and the functions of their job but that should be where the the limit is it shouldn't go beyond that then you need to have a managed and monitor antivirus and all systems now this is a very important one because if you're just putting in a virus on your computers and you're not managing and monitoring it then it becomes disabled or something happens and it's no longer getting updates you're not going to know about it you need to have something that tells you when ever that antivirus has not checked in the way our system does is is if we install an anti-virus it has a cloud console and that antivirus will check in periodically periodically with the cloud console to get updates and send it a heartbeat if it becomes disabled they're no longer receiving updates or not scanning the system it notifies us because it didn't check in so it can be completely removed off the system but I'll still get an alert because my cloud console is notifying me whenever it can't connect so next you need to have active system monitoring just like the antivirus you need to be monitoring everything on the systems and when I talk about everything I'm talking about applications being installed that are unauthorized people are just going out to shareware sites and installing stuff you need to know when that happens and then you need to remediate that by removing those apps talking to the employee educating them on not installing those apps and and

that's another very positive way of protecting your environment lastly you need employee training and testing this one is very overlooked what most companies will do is they'll have a little training seminar what employee starts in and they'll they'll give them the don't install software don't take EPA chai outside the organization you know don't store credit card information you know there's just a one-time training when an employee starts and they feel like we've done everything we need need to do well there's more training you can do we provide training for customers by actually sending them malicious emails if the employee clicks on the malicious email it will automatically send them to training and show them why they fell for the email and teach them how not to fall for that religious email again it's a fascinating training it's very helpful and then finally we talked about perimeter defenses what are the things you can do at the perimeter your organization before a virus even gets in because the antivirus and the things you're doing on the systems only happen once then something has gotten into your network so your perimeter defenses are things like content filtering you're blocking malicious websites sites that where the company has no business going to you can also do what's called geo IP where you're blocking entire countries from being able to get in or out of your system you have intrusion prevention that's an active system that looks for attack vectors on your on your network people trying to get in and it'll actually block those and intrusion detection if someone tries to get in you need it to be detected and then the prevention system can kick in you need antivirus on the Gateway not just on your computers you need something one that looks at all traffic coming in and out of your network and if it looks like a virus or malicious code in some way it's going to block it and let you know and then just active system monitoring again look at your perimeter just like you look at your internal systems look for problems get alerts and react to them the final question we wanted to answer was should you move your server to a hosting environment in this case with all scripts their environment so if you have electronic medical record system and it's centricity or prime suite from Greenway or all scripts whoever it might be all those companies have hosting environments where you can not have any server infrastructure in-house and move your servers to their environment and there's the hope that that's going to be better because they have an economy of scale they're going to keep your costs down they imply that they're going to be more HIPAA compliant there's there's some positive implications on moving to a hosting environment but that being said having your server inhouse does give you a degree of control if you're having problems with your software vendor you can bring in a third party company to look at the database to look at the server performance and make sure that things are running in tiptop shape if they're not they can correct that in-house but they also can help shine a light on where your software vendors may be not taking great care of you that's something you're going to give up when you move to a hosting environment like all scripts or greenway prime suite or so on what might be a better fit for you is instead of moving to a company who actually owns and operates and manages all the infrastructure might be to move to an actual hosting data center where you own the server in the data center and a third party company manages that for you that gives you the insights and control but also keeps one company from controlling both the hardware and equipment of it with your systems as well as the application itself you need to have and this is from our own experience you need to have two companies working on your systems together to resolve issues where in our case Arclight would be working as an advocate for you to make sure that your systems are performing and if they're not would work then with your vendor be it all scripts or whoever it might be to get your systems performing properly well that's all I've got for today if you have any questions or concerns we'd like to talk to us about helping you put together an effective disaster recovery plan feel free to reach us at the number on your screen have a wonderful rest of your week you White Plains campus.