Major Software & Hardware Vendors Cause Self-Inflicted Downtime

Let’s Encrypt is the free HTTPS encryption service that is used by millions of web sites. Since it started out as a good idea of two Mozilla employees in 2012, it has issued about 2 billion free TLS certificates.

The history behind this organization is long and convoluted. The industry has a high bar for entery for a new player and in 2012, they had to get someone that the industry trusted to, kind of, co-sign their HTTPS certificates.

They knew that co-sign process was a short term solution and about 4 years ago they convinced the “Internet authorities” that they were the real deal and replaced that co-signed certificate with a new one.

Browsers and other software vendors have been incorporating this new software since 2017.

Let’s Encrypt, itself, has been warning people for about a year that the old certificate was going to expire today and software vendors needed to upgrade.

We expected that old, unsupported software like Windows XP and old hardware like Android phones running Android 7, would have a problem today.

That turned out to be true.

What we did not expect is that mainstream websites like Shopify, mainstream tech vendors like Palo Alto and Cisco and mainstream service vendors like, Google Cloud monitoring and Quickbooks would be caught, napping or completely asleep at the switch.

Unfortunately, we were wrong.

These vendors and many others went dark about about 8 AM Mountain Time this morning.

Some of them fixed the issue. Shopify, for example, recovered at about 3:30 PM.

Others, like Fortinet, seem to continue to be asleep at the switch and have told their customers to turn off the security feature that warns you when you have a security issue. That is not a great solution, but for some Fortinet customers, that is their only option.

Many more likely have not been detected yet – like IoT devices that just stopped working but that no one has either noticed or figure out why.

And, importantly, if these software or hardware products are no longer supported, you are probably out of luck and will have to replace it.

In some cases, you have the ability to tell the system to ignore the error and move forward, but most of the time, that is not an option.

I am writing this because, I think, this is day one of an extended discovery process. Likely there are things that are down and people don’t know they are down or don’t know why they are down. This will take a while to discover and to fix. In some cases, the fix will be expensive and extended.

I wrote about this a few months ago. This should not have happened as the industry knew exactly what day it was going to be a problem 9 years ago. Still we, as an industry, create self inflicted wounds.

For more details, check out this article at ZDNet.

CISA Issues Cyber Goals & Objectives for Critical Infrastructure Control Systems

While goals are CURRENTLY voluntary, CISA issued guidelines for what it expects from pipelines and other critical infrastructure in light of the Colonial Pipeline attack. While it appears that the hackers were not able to take over the control systems in that attack, they did take over the control systems in the Florida and Kansas water system attacks.

And, while this legally only applies to critical infrastructure, if it makes sense, you might want to do it also.

Here are some highlights.

CISA already has a raft of documents, so they reviewed and harmonized them and came up with a single list. See the link at the end for more information. Here are some of the highlights. Each goal comes with a rationale and objectives.


GOAL: Identify and document cybersecurity risks to control systems using established recommended practices (e.g., NIST Cybersecurity Framework, NIST Risk Management Framework, International Society of Automation/International Electrotechnical Commission (ISA/IEC) 62443, NIST Special Publication (SP) 800-53, NIST SP 800-30, NIST SP 800-82) and provide dedicated resources to address cybersecurity risk and resiliency through planning, policies, funding, and trained personnel.


GOAL: Integrate cybersecurity and resilience into system architecture and design in accordance with established recommended practices for segmentation, zoning, and isolating critical systems (e.g., Industrial Control Systems-Computer Emergency Response Team Defense in Depth guide, Purdue Diagram) and review and update annually to include, as appropriate, any lessons learned from operating experience consistent with industry and federal recommendations.


GOAL: Document and control hardware and software inventory, system settings, configurations, and network traffic flows throughout control system hardware and software lifecycles.


GOAL: Physical access to systems, facilities, equipment, and other infrastructure assets, including new or replacement resources in transit, is limited to authorized users and are secured against risks associated with the physical environment.


GOAL: Protect the control system and its data against corruption, compromise, or loss.


GOAL: Implement and perform continuous monitoring of control systems cybersecurity threats and vulnerabilities.


GOAL: Train personnel to have the fundamental knowledge and skills necessary to recognize control system cybersecurity risks and understand their roles and responsibilities within established cybersecurity policies, procedures, and practices.


GOAL: Implement and test control system response and recovery plans with clearly defined roles and responsibilities.


GOAL: Risks associated with control system hardware, software, and managed services are identified and policies and procedures are in place to prevent the exploitation of systems through effective supply chain risk management consistent with best practices (e.g. NIST SP 800-161).

For more details go to this CISA web site here.

FCC is Finally Going to Help Carriers Replace Huawei Network Gear

In March of 2020 Donald Trump signed a bill requiring rural telecom companies to replace suspected vulnerable Huawei telecom equipment. The government would even pay for the replacement. For whatever reason, the Trump administration never did anything to make that happen.

Finally, 18 months later (if this is really a security threat, shouldn’t something have happened sooner?), the FCC has laid out the rules for carriers to get reimbursed.

Note that carriers have still not received a dime and likely have not replaced any vulnerable equipment unless it was unrepairable.

For carriers with less than 10 million customers (they are not helping Comcast or AT&T and probably should not) and even some schools if they are acting as an Internet provider, they can be reimbursed for the cost of removing and replacing vulnerable equipment.

Very nicely, the FCC says that if the carrier replaces any old 4G network with, say a new 5G network, those costs may be covered.

On the other hand, if the carrier was connecting towers by microwave radio and they decided to upgrade that to fiber, that is not covered.

But, if the old tower will not hold the new equipment, the FCC will consider paying for a new tower.

Carriers will be able to recover vendor installation costs and labor costs of employees who are replacing the equipment.

Simple, huh?

After delaying this for 18+ months, the FCC now says that carriers have from October 29th to January 14th to ask for reimbursement. Applicants will find out if they are a winner in Q2 2022 – two years after the law was passed.

At that point, winning carriers can start ORDERING equipment, which will likely take years to get and deploy.

I assume this is a real problem. I don’t know, but it certainly is consistent with things the Chinese might do.

Not related to this, but curiously timed, Huawei CFO Meng Wanzhou, the daughter of its founder, who has been under house arrest in Canada for several years, reached a deal to avoid prosecution by admitting that she mislead lenders about her and their relationship with Iran. While she did not plead guilty to the fraud charges, admitting that she lied to banks about their dealings with Iran in spite of there being sanctions in place at the time, doesn’t give me any warm feelings that they would not sell us out if it was convenient (or profitable) for them.

It would certainly be better if the government moved quicker. Credit: ZDNet

Security News for the Week Ending September 24, 2021

Detecting Hidden Cameras in Your Airbnb and Similar Rentals

No one wants to think about this, but it is an issue. Especially in private home/condo rentals, owners are worried about you stealing or damaging their stuff. And some of them are just stalkers. Here is a TikTok video from well known security researcher Marcus Hutchins on some things that you can do to look for hidden cameras. Credit: Hack Read

Japan Sets New Internet Speed Record – 319,000,000,000,000 bits per second

While not a security issue, it is pretty impressive. This beats the old record of 178 terabits/second. The test was carried out in a lab, but simulated a 3,000 KM fiber. This is definitely still experimental, so don’t expect to get this speed at your house any time soon. Credit: Computing (free account required)

The Internet is Going to Break

Well, I don’t think so, but some people are concerned. Let’s Encrypt is that free service that lets web site owners encrypt traffic to and from their website. Let’s Encrypt’s original ROOT CERTIFICATE is going to expire in about a week. They updated their certificate in clients like Chrome and Edge and server software like Linux Apache a long time ago, but what about users that are running old, unsupported software. In a word, they are going to be SOL. The certificate will show as expired and depending on the situation, the user likely will not be able to establish the connection. If it is a server that has that expired certificate, even if the user has been updated, things won’t work. Bottom line, this is only going to be a problem for old, unsupported systems – but there are a lot of these. Stay tuned. Old IoT devices are most likely to break. If you are responsible for systems, now would be a good time to test. Credit: Portswigger

VoIP Phone Provider Hit by Denial of Service Attack; Has Been Down for a Week

This is the downside of the cloud. has been battling a massive (they say) distributed denial of service attack since September 16th. They say they have over 80,000 (likely unhappy) customers in 125 countries. All of whom have limited voice service as a result of the attackers wanting to pay them a ransom to stop the attack. How would your business operate if it did not have phone service for a week? Credit: ZDNet

100 Million IoT Devices Affected by New Bug

NanoMQ is an OPEN SOURCE messaging processing platform that is used in many critical IoT devices like patient monitors, fire detection, car system monitors and smart city applications, among many others. Researchers form Guardara detected multiple vulnerabilities affecting as many as 100 million devices. It could cause the device to crash – that is very simple to do – or worse. Attacks on these kinds of devices are spiking and until IoT vendors get serious about security, plan on a backup system for anything that is critical. While some people continue to spread the myth that Open Source software is secure, there is not much evidence for that as we see bug after bug revealed in super popular apps, never mind the really niche ones. Credit: Threat Post

Google Says Geofence Warrants Up by 10x+

Geofence warrants are “requests” by law enforcement for information on everyone that was in a particular geographic area during a particular time window.

Typically they use the results to come up with the usual list of suspects. The initial response usually doesn’t include names and addresses; that comes after the police mine all the data that they got. Also note that they do not delete that data. Possibly for ever.

Lets say there was a burglary at 1 Park Avenue in New York on Saturday morning, maybe around 6 AM. The NYPD might ask Google to give them data on everyone in a 4 block area surrounding 1 Park between midnight and noon that day.

The police would need to convince a judge that this is reasonable, but that does not tend to be that hard.

How I know that it is not hard is by looking at the numbers.

In 2018 before geofence warrants were popular, Google responded to 982 of these warrants.

Last year, they responded to over 11,000 of them.


Google really hasn’t said much in response to this. In fairness to them, they have to comply with the law.

But the reason these are becoming very popular is the sheer amount of data we choose share with Google. From location tracking to maps to queries to all kinds of stuff, Google is awash in your data.

In one case, in 2020, the data indicated that one Zachery McCoy was the police’s prime suspect and in this case, Google told him about the warrant (they can’t and don’t always tell). He was using an app to track his bike riding and it put him near a burglary.

Ignore, for the moment, that any half-way intelligent crook will power off his or her phone before going out on the town to loot, pillage, maim and whatever.

McCoy had to spend his own money to eventually exonerate himself – after other evidence emerged.

Such is the danger of our super connected world.

Convenience and surveillance. Wonderful. Credit: Threatpost

Domain Registrar Epik Hacked

Domain registrar Epik is known for hosting certain types of domains. They call themselves the Swiss Bank of Domains – neutral in the political fights. They host the domains for right wing sites like Parler and Gab and political sites like Texas Right to Life and the Texas GOP, among many others.

The company confirmed that hackers breached their security AND downloaded customer account information.

The hackers may be affiliated with the non-group Anonymous, the loose collective of hackers that go after folks that they don’t like. They said, in a press release, that the hack was in retaliation for Epik’s habit of hosting questionable alt-right websites (their words).

“This dataset is all that’s needed to trace actual ownership and management of the fascist side of the internet,” the group said. “Time to find out who in your family secretly ran an Ivermectin horse porn fetish site, disinfo publishing outfit or yet another QAnon hellhole.”

Epik Confirms Hack, Gigabytes of Data on Offer | Threatpost

It also appears that non-customers were also swept up in hack as well and some of their data was stolen too.

Size-wise, the hackers stole 180 gigabytes of data, they say, including names, phone numbers, physical addresses, purchases and passwords.

Also apparently much of the data was not encrypted and some of it was only lightly salted (meaning that reversing it was trivial for the hackers).

It seems that the hackers are GIVING the data away for FREE. Here is what you get for free:

  • domain purchases and transfers in and out, all whois history unredacted, all DNS changes, all email forwards, payment history (without credit cards), account credentials for customers, hosting, VPN, etc., Epik’s internal servers and systems, Epik’s GoDaddy logins and more.

The hackers said “yep, these Russian developers they hired are actually just that bad.” referring to the lack of encryption and weak hashing.

They also hacked the Texas GOP web site for fun.

What does this mean to you?

First of all – vendor cyber risk management. Are your vendors secure?

Second, if you used Epik, change all affected passwords and encryption keys

Third, assume an attack like this could happen; plan for it. Then do what you can to mitigate the damage from it.

Credit: Ars Technica