Tag Archives: disaster recovery

The Cloud is not a Miracle – Do Your Homework

As more people and more businesses embrace the cloud, the opportunity for disaster goes up.

For example, we have seen companies move to the Amazon cloud and then be surprised when their web sites go dark (see this example).

There are no silver bullets when it comes to data center availability and the cloud is not one.

The cloud can both help you and hurt you; good design and architecture still “rules”.

Here is a recent example.

MJ Freeway makes marijuana grow and dispensary software that helps businesses comply with the law and manage their businesses.  They claim to have processed $5 billion in transactions for the MJ industry,

Their solution is cloud based, making it easy for businesses to use their software.  Until they have a problem.

MJ Freeway’s cloud  based solution was hacked, blinding a thousand dispensaries – unable to track sales and manage inventories.  For many of these stores, that means closing the doors until they can get the problem resolved.

But the attack was interesting.  All the data was encrypted, so the hacker could not use the data.  That however, does not appear to be the hacker’s objective.  The attackers targeted live production servers and backup servers at the same time.

Because it took MJ Freeway several hours to discover the attack, the attackers had a head start and because they attacked the primary and backup sites, clients had an outage.

Some customers maintained their own personal, offline backups of their data.  Those customers were able to restore their data as soon as MJ Freeway had a stable web site.  While it was wonderful that these users did not lose any data, they were still down until their vendor could create a stable operating environment.

For users that depended on their cloud service provider to backup their data, they had a bigger problem.  Since the primary and backup web sites were attacked at the same time, no online copies of the data were usable.

The “seed to sale” data was, apparently, corrupted and may not ever be recoverable.  What that means to those dispensaries from a legal standpoint is not clear, but can’t be good.

If the hacker’s objective was to ruin these companies – to bankrupt them – to run them out of business – that may be a great way to do that.

If their objective is just to cause the dispensaries pain – including lost sales, lost customers forever (to competitors), lost business to MJ Freeway, fines for regulatory failures and a host of other costs, the hackers may well have succeeded.

However, this is a great lesson for all businesses – whether you are in a semi-legal business like marijuana or a totally mainstream business like retail or services – the cloud is a wonderful tool.  It is not, however, a silver bullet.

Cloud services go down.  They lose data.  Sometimes they go out of business unexpectedly.  Who is liable typically depends on the terms in the contract.  If the contract was written by the online service provider, you can count on the contract saying that the provider is not responsible for anything.

Plan for a disaster.  Plan for a cyber incident.  WHEN something unexpected happens (notice I said when and not if), you will be in a much better position to deal with it.

Two terms in the disaster recovery business should be in every business that uses cloud services (and others too) lexicon:

RTO – Recovery Time Objective – How long are you willing to be down for.  If the answer is a day or a week, how you prepare for a disaster is different than if the answer is 5 minutes or an hour.

RPO – Recovery Point Objective – How much data are you willing to lose (or how far back in time are you willing to restart at).  If you can lose (and I assume, recreate) a day’s worth of data, it is easier and cheaper to build a disaster recovery plan than if the answer is 15 minutes.

So everyone who signs up for a cloud solution, keep in mind that sometimes, where it is cloudy, it rains and when it does, if you have an umbrella (aka a disaster recovery plan) then you are likely OK;  however, if you don’t have that disaster umbrella, you are going to get wet;  possibly very wet.

As those dispensaries discovered; your profit can go up in smoke and not in a good way.

Information for this post came from Network World.

Internet of Things – The New Hacker Attack Vector

Recently, Brian Krebs (KrebsOnSecurity.com) was hit with a massive denial of service attack.  The site went down – hard – and was down for days.  His Internet Service Provider kicked him off, permanently.  The attack threw over 600 gigabits per second of traffic at the site.  There are very few web sites that could withstand such an attack.

The week after that, there was another denial of service attack – this time against French web hosting provider OVH – that was over 1 terabit per second.  Apparently, OVH was able to deal with it, but these two attacks should be a warning to everyone.

These attacks were both executed using the Mirai botnet.  Mirai used hundreds of thousands to millions of Internet of Things devices to launch this attack.    The originator released the source code to this attack because, he says, that he wants to get out of the business.

While Mirai used to control around 380,000 devices every day, some ISPs have started to take action and the number is now down to about 300,000 a day.

There are a couple of reasons why the Internet of Things presents a new problem.

The first problem is patching.  When was the last time that you patched your refrigerator?  Or TV?  I thought so!  After 10 years of berating users, desktops and laptops are being patched regularly. Phones are being patched less regularly.  Internet of Things devices are patched almost never.

The second problem is numbers.  Depending who you believe, there will be billions of new IoT devices brought online over the next few years.  These range from light bulbs to baby monitors to refrigerators.  The manufacturers are in such a hurry to get products to market and since there is almost no liability for crappy security, the manufacturers are not motivated to worry about security.

Brian Krebs, in a recent post, examined the Mirai malware and identified 68 usernames and passwords hardcoded into this “first generation” IoT malware.  For about 30 of them, he has tied the credentials to specific manufacturers.

This means that with a handful of hardcoded userids and passwords, Mirai was able to control at least hundreds of thousands of IoT devices.

How many IoT devices could a second- or third- generation version of that malware control?

The third problem is the magnitude of these attacks.  While DDoS attack prevention services like Cloudflare and Akamai have been able to handle attacks in the 500 gigabit per second range, if the growth of DDoS attacks continues and we are talking about multi-terabit attacks, how much bandwidth will these providers need to purchase to keep up with the DDoS arms race.  While the cost of bandwidth is coming down, the size of attacks may be going up faster.

Lastly, ISPs – the Internet providers that enable the Internet connection to your home or office are not stepping up to the plate quickly enough to stomp out these attacks.

The ISPs may become more motivated as soon as these rogue IoT devices that are sending out DDoS traffic force the ISPs to buy more bandwidth to keep their customers happy.

Of course, like Brian Krebs, if your company winds up being the target of one of these attacks, your ISP is likely to drop you like a hot potato.  And equally likely, they will not let you back on after the attack is over.

If being able to be connected to the Internet is important to your business – and it is for most companies – you should  have a disaster plan.

The good news is that if your servers are running out of a data center, that data center probably has a number of Internet Service Providers available and you should be able to buy services from a different provider in the same data center within a few days to a week.  Of course, your servers will be dark – down – offline – in the mean time.  Think about what that means to your business.

For your office, things are a lot more dicey.  Many office buildings only have a single service provider – often the local phone company.  Some also have cable TV providers in the building and some of those offer Internet services, but my experience says that switching to a new Internet provider in your office could take several weeks and that may be optimistic.

Having a good, tested, disaster recovery plan in place sounds like a really good idea just about now.


Information for this post came from PC World.

The Brian Krebs post can be heard here.

Don’t Let Your Website Be Held Hostage

For most businesses, their web site is the public face of the company.  If your web site is an e-commerce site, then not only is it your public face, but also the way you earn money.  If your site is down, it says something to your customers.  If your site is defaced, it says even more.

Hackers, or more accurately, extortionists, have used this fact to separate business owners from their money.  If your site is hacked – compromised – defaced – pick one, do you have a plan to respond?  What if the attack is a ransomware attack where the hackers encrypt all the code and data – even backups if those are accessible.  Remember, even if you pay the ransom, you may or may not get your site and data back.

What if they take over your site and you lose control of it so that you can’t even log on to it to fix it?  If they put an offensive message on the site (for example, what happened to Sony) and you have lost control of the site, what do you do.

Having a plan is a good idea.  Pros call this disaster recovery and business continuity – keep the business running while you get things back to normal.

Here are a few basic suggestions.

  • Keep your web site software up to date.  As soon as patches are available, test and install them.  This includes the operating system, the content management system, shopping cart and any other pieces.  Once patches are released, the attackers have a roadmap for attacking you.
  • Make sure that the source code is stored some place that is not directly accessible from the web site so that if an attacker does get in, he can’t wipe out your source code too.  I replicate my backups in three places – on the web site, in the cloud and offline.  Nothing is perfect, but when it comes to backups, more is better.
  • Replicate files and databases frequently so that even if you get compromised, you can recover.  How often you replicate is dependent on how quickly things change.  If it is an e-commerce site, you may want to replicate changes every few minutes or hourly at the most.  And, you need to do this in a way that hackers won’t be able to destroy the backups.  Sometimes, that is easier than said.
  • Minimize the software that lives your web server.  You should NEVER use it for anything other than running the web server.  Other than the people managing the web server, no one else should be able to log on to the server.  This is for both security of the data and to reduce the chance of human error.  The more software on the server, the more attack points for the hacker.  And, NO web surfing from that server.  If you need to update a program, download the updates elsewhere and bring them over.  No browsing reduces the attack surface.
  • If possible, have the web server run inside a virtual machine – either in your data center or in the cloud.  Snapshot the VM often and do not store any data inside the VM. Keep enough generations of the backups so that even if you don’t discover the problem for a while, you still have an uncorrupted backup.
  • Finally, TEST, TEST and then TEST again.  Whether your site is taken offline, compromised or defaced, you want to be able to get back to “normal” as quickly as possible.  You don’t want to be trying to restore it for the first time.


Information for this post came from TMCNet.