Category Archives: Business Continuity

Security News for the Week Ending April 29, 2022

Sungard Files for Chapter 11 Bankruptcy Protection – Again

Sungard, the king of disaster recovery and business continuity needs to figure out a new business. They previously filed for Chapter 11 in 2019 and shed $800 million in debt, but they have a fundamental problem. As businesses move from private data centers to the cloud and from offices to work-from-home, they just don’t need Sungard anymore. And, likely never will. They REALLY need to reinvent themselves. Credit: Tech Target

Any Sign of the Supply Chain Returning to ‘Normal’?

One of the lists I am on asked this question and the answer seems to still be no time soon. High end network and server gear still is between 6-12 months or ‘unknown’ out. Manufacturers are reducing their chip and system product range to focus limited supply on the more important products and some customers are getting priority based on performance penalties in long term contracts. The NY Times has an extensive piece on all of the problems, none of which are easy to fix in the short term. Credit: NY Times

AWS Locks Up NSA Cloud Deal

Years ago Amazon (AWS) locked up a deal worth up to $10 billion to provide a secure, classified cloud to the CIA. That was before the days of contract protests over the cloud. Years later, the DoD tried the same thing, called JEDI. It died due to contract protests. DoD is still trying to build a classified cloud, now called JWCC. However, now the NSA has joined the CIA and awarded AWS a $10 billion contract to build them a classified cloud. The rest of the DoD is still waiting. Credit: Meritalk

Brazil Senate Passes Bill to Regulate Cryptocurrency

The Brazilian Senate has passed a bill that regulates the cryptocurrency market in an effort to protect consumers. Crypto exchanges would fall under the regulation of Brazil’s Central Bank. As one of the leaders in the crypto market, Brazil is also set to release a cryptocurrency pegged to the real, Brazil’s currency. It is not clear to me what the value of any cryptocurrency pegged to any country’s currency, but the good news (bad news?) is that since it is based on software, all of these new cryptocurrencies will likely be hacked and the hackers will make billions. At least someone will get rich. Credit: ZDNet

China, Russia and India Do Not Agree Not to Undermine Future Elections Using Misinformation

The United States, European Union, United Kingdom and 32 other nations have committed to not interfere with future elections by running online misinformation campaigns or illegally spying on people. On the other hand, Russia, China and India, unlike these 60 other countries, did not agree to the declaration. Not really a big surprise. Credit: ZDNet

Cybersecurity News for the Week Ending April 1, 2022

How Many Times Do I Need to Say – Crypto is Software, Software Has Bugs, Your Money is at Risk

Decentralized Finance platform (DeFi) Revest Finance said that it lost $2 million due to a software bug and, oh yeah, (a) the can’t recover the funds, (b) they do not have the money to cover the losses and(c) they don’t have insurance to cover the hack. Unless we eliminate the software, we cannot eliminate all bugs. Credit: The Record

Russia Faces Internet Outages Due to Equipment Shortages

One of Russia’s tech unions says that Russian ISPs run the risk of Internet outages as the value of the Ruble goes down and foreign companies won’t sell them parts or new equipment. Right now the government is saying that is the Internet providers’ problem, but if it turns into widespread outages, they are likely to change their tune. Credit: Bleeping Computer

Cryptocurrency was Fun While it Lasted

EU Parliament committees have voted to require crypto exchanges to verify the identity of self-hosted wallets, meaning the end of anonymity for crypto transactions. The US Treasury (FinCEN) has also suggested that we do that, but it has not yet appeared in a bill. That means that the bad guys will need to do peer to peer crypto, minus the exchanges to deal in criminal activities. While this is harder than using exchanges, it is far from impossible. Given that the whole purpose (beside speculating) of crypto is to commit fraud, identifying yourself is probably not high on user’s wish lists. Credit: Vice

Senate Asks Companies About Hackers Creating Fake Warrants

Recently I wrote that hackers have figured out the the government’s search warrant process is as secure as, say, a screen door. Now that the facts have been outed and likely even more hackers will use that fact to steal even more data, a couple of Senators have started asked questions. That is a long way from Congress actually doing anything useful about it, but at least it is a start. Don’t expect anything to happen because it is a hard problem to fix. Credit: Brian Krebs

Apple Fixes More Mac, iPhone Zero Days

In case you haven’t noticed, the last 12 months have not been Apple’s friends when it comes to zero-day bugs. This week Apple patched two more that are actively being exploited in the wild and affect iPhones, iPads, iWatches and Macs. The versions you are looking for are iOS 15.4.1, iPadOS 15.4.1, and macOS Monterey 12.3.1 with improved input validation and bounds checking, respectively. Credit: Bleeping Computer

Nashville Bombing Part 2

As I said last week, while the bombing is a horrible event, it does point out how brittle our telecommunications world is. That being said, for most companies, the rest of the IT infrastructure is probably more brittle.

Companies should use this as an opportunity to review their situation and see if they can make improvements at a price that is affordable.

While AT&T was able to strike a deal with the City of Nashville to commandeer Nissan Stadium, home of the Titans, to set up a replacement central office, you probably will not get the same treatment if you asked.

AT&T was also able to deploy 25 tractor trailers of computer equipment to replace the equipment that was damaged.

Finally, AT&T was able to temporarily reassign personnel with every skill that they might possibly need from fiber techs to computer programmers. Again, you likely would not be able to do that.

The question for you to “game out” is what are my critical vendors and what would I do if they had a meltdown. I don’t mean a 30 minute outage, I mean a meltdown. We have seen, for example, tech companies that have gotten hit by ransomware.

Perhaps, like many companies, you use a managed service provider or MSP. A number of MSPs have been hit by ransomware and when they do, often so do their customers. Does your MSP have the resources to defend all (or most of) its customers from a ransomware attack at once. How long would it take your MSP to get you back to working? Even large MSPs (which equals many customers) likely don’t have the resources.

If that were to happen to you – and of course, they have the only copies of your data, right? – what would they do and what would you do?

Maybe your servers are hosted in your office. There are a lot of possible events that could occur.

Even if your servers are in a colo, things can occur that can take you down.

Here is one thing to start with –

For each key system from personnel to public web sites, both internal and at third parties, document your RECOVERY TIME OBJECTIVE or RTO. The RTO is the maximum acceptable downtime before recovering. For example, for payroll, it might be 24 hours. But what if the outage happens at noon on the day that payroll must be sent to your bank? So, think carefully about what the maximum RTO is and remember that it will likely be different for different systems.

Then, for system, document the RECOVERY POINT OBJECTIVE or RPO. The RPO is the point in time, counting backward from the event, that you are willing to lose data. For example, if this is an ecommerce system, maybe you are willing to lose 30 minutes worth of orders. Or maybe 5 minutes. If it is an accounting system, maybe it is 8 hours (rekeying one day’s worth of AR and AP may be considered acceptable). Again each system will likely be different.

Then get all of the lines of business, management and the Board (if there is one) to agree on those times. Note that shorter RTOs and RPOs mean increased cost. The business units may say that they are not willing to lose any data. If you tell them that you can do that, but it will cost them a million dollars a year, they may rethink that. Or management may rethink that for them. The key point is to get everyone on the same page.

Once you have done that, make a list of the possible events that you need to deal with.

  • Someone plants a bomb in an RV outside your building and there is severe physical damage to your building.
  • Or maybe the bomb is down the block, but the force of the blast damages the water pipes in your building .
  • Or, the bomb is down the block and there is no damage to your building, but the city has turned off water, power and gas to the building. And the building is inside a police line and will be inaccessible while the police try to figure out what is going on.
  • In the case of AT&T, they had to pump three FEET of water out of the building. Water and generators are not a good mix. Neither are water and batteries. While AT&T lost their generators as a result of the blast, their batteries were distributed around the building so they did not lose ALL of their batteries.

Note that you do not need to think up all the scenarios yourself. You can look at the news reports and after-action reports from other big, public meltdowns. Here is another article on the Nashville situation.

Now create a matrix of events and systems for your RTO and RPO numbers. In the intersection box, you can say that you already can meet those objectives or that it will cost $1.29 one time to meet it or a million dollars a year. You need to include third party providers if they run and manage any systems that are critical to you.

Once you have done all that, you can go back to management and the lines of business and tell them here is the reality – what risk are you willing to accept? This is NOT an IT problem. This is a business problem.

The business will consider the likelihood of the event – even after Nashville, an RV filled with explosives is an unlikely event and the cost to mitigate the problem is likely high. For some systems the cost may be low enough and the risk high enough that management says fix it. For other systems, probably not.

The key point is that everyone from the lines of business to management to the Board all understand what the risks are and what the mitigation costs are. From this data, they can make an informed BUSINESS decision on what to do.

If you need help with this, please contact us.

Security News for the Week Ending Jan 1, 2021

Happy New Year. May 2021 be more sane than 2020.

Microsoft Says Goal of Solar Winds Attack Was Your Cloud Data

Microsoft says that the objective of the Solar Winds Hackers was to get into a number of organizations and then pick and choose which ones to attack, leaving the back door in place at the others for future operations. One way to do that was to generate SAML login tokens from inside the network and then use those tokens to gain access to cloud resources, including, but not limited to email. The article also provides more information on how to detect if the hackers did compromise your cloud resources. Credit: Bleeping Computer

“Swatting” Moves to the Next Level

Swatting, the practice of calling in a fake 911 call so that SWAT teams are deployed to the victim’s location based on, say, a fake kidnapping, are moving to the next level. As if having the police show up unexpected with lots of guns and breaking down your door isn’t bad enough, now the perpetrators are taking advantage of the fact that people choose crappy passwords and are watching and listening to the police assault on the victim’s own smart devices. On occasion, the situation becomes deadly when the police, not really knowing what to believe, shoot the victim. On rare occasions, the swatters, as they are called, are caught and prosecuted. Credit: Bleeping Computer

I Think The Wasabi Got a Little Too Hot

Wasabi, the cloud storage competitor to Amazon S3 that claims that it is significantly cheaper than Amazon and 99.999999999% reliable just got a little less reliable. Their domain registrar notified them of some malware hosted on one of their domains. Only they sent the email to the wrong email address. The registrar, following normal procedures, suspended their domain for not responding to an email they never got, knocking many of their customers offline.

After Wasabi discovered they had been DDoSed by their domain registrar, they suspended the offending customer and asked to get their domain back. That process took over 13 hours. Are you ready for this kind of attack from your suppliers?

That attack probably knocked several of those 9’s off their reliability, depending on how the mess with the data.

Credit: Bleeping Computer

Solar Winds Troubles Are Not Over

A second piece of malware called SUPERNOVA and a zero-day vulnerability that it exploited makes it look like there may have been a second attack against Solar Winds. This appears to be a separate attack from the Russian attack. The attack vector is different too – this is not an attack against Solar Winds code base. This spells additional trouble for Solar Winds. Credit: Security Week

The Cloud Adds New Security Risks

Yesterday’s double trouble outage should remind businesses that planning for outages and continuing to operate is not optional.

The first outage was at Microsoft where it’s Active Directory services had some problems. Active Directory is used to “authenticate” users and services, so if it doesn’t work, not much else does.

The good news is that it happened towards the end of the work day (around 5:30 PM Eastern time for about 3 hours or so), so some of the pain was deflected. This particular type of outage is hard to build in redundancy for because it affected the behind the scenes infrastructure.

The second trouble was when 911 services in many communities in 14 states went down around 4:30 PM Mountain time. There was some question about whether these two were related, but based on what we are hearing, that is not the case. Losing 911 services is slightly more important than, saying, losing access to Twitter, even though the current occupant of the White House might disagree with that.

Like many companies, Public Safety Access Points or PSAPs, which is the technical name for 911 call centers, have outsourced some or all of their tech. Both companies involved with yesterday’s 911 outage have recently changed their name – likely to shed the reputations they had before. The company that the PSAPs contract with is Intrado, formerly known as West Safety Communications. Intrado says their outage was the fault of one of their vendors, Lumen. Many of you know Lumen as the company formerly known as Centurylink (actually, it is a piece of Centurylink).

The bottom line here is that whether you are a business selling or servicing widgets or a 911 operator, you are dependent on tech and more and more, you are dependent on the cloud. You are also dependent on third parties.

You need to decide how long you are willing to be down and how often. In general, cloud services are reliable. Some more than others. But you have lost some insight into tradeoffs being made by virtue of moving to the cloud and using third party vendors. These vendors are trying to save money. While you might agree with their decisions, you are never consulted and likely never informed.

You may be okay with this, but it should be a conscious decision, not something that happens accidentally.

Do you have a disaster recovery plan? Or a business continuity plan? When was it last tested? Are you happy with the results?

These outages were relatively short-lived. For most people the Microsoft outage affected them for around 3-4 hours. For the 911 outage, it lasted for around 1-2 hours. But many of these outages have lasted much longer than that.

Have you asked your vendors (cloud or otherwise) about their plans? Do you believe them? Are their meaningful penalties in the contract to cover your losses and your customers’ losses? Are you okay with the inevitable outages?

Consider this outage an opportunity. Credit: Brian Krebs

Who Wants to Hear Fiction About System Recovery Time

A survey of small and medium size businesses asked executives about their Recovery Time Objectives or RTOs. A company’s RTO represents the amount of time a system, such as a web site, can be down after an incident. The incident could be a software error, hardware failure, ransomware attack or many other things. Here are some of the answers they got.

  • 92% of SMB executives said they believe their businesses are prepared to recover from a disaster.

My first question for these executives is when was the last time you TESTED that preparation and what was the result? My guess is that the primary answer will be that it has never been tested.

20 percent say that they do not have a data backup or disaster recovery solution in place. If so, how are they prepared to recover?

  • 16% of executives say that they do not know their own recovery time objectives, but 24% expect to recover in less than 10 minutes and 29% expect to recover in less than an hour.

So, while 20% don’t even have a data backup solution in place, more than half expect to recover in less than an hour.

The results are from a survey of 500 SMB execs; 87% of which were CEOs.

  • Of those who said they knew what their RTOs are, 9% said it was less than one minute, 30% said it was under an hour and 17% said it was under a day.

Compare that to recent ransomware attacks. Atlanta took several months to recover. Travelex was down for over a month.

How do all of these SMB execs figure they are smarter than these guys who took weeks and months to recover?

Another problem is that people don’t agree on what the definition of a disaster is. Is it recovering from a data loss or recovering from a malware attack or the ability to become operational quickly or what?

Bottom line – executives need to understand this recovery thing because experience tells me that it takes way longer to recover than people seem to think it does. And, for most companies, if their systems are down, they are not making money and are spending money.

If executives think they have a handle on this – conduct a mock disaster drill and see how long recovery takes. For most companies it will not be 10 minutes or an hour.

Need some help figuring this out? Contact us. Credit: Help Net Security