With all the uproar around the election in December, the news of the SolarWinds data breach did not get the attention it deserved. Some well-resourced foreign organization, almost certainly in Russia, succeeded in infiltrating the data systems of an astounding 18,000 or more U.S. organizations. These included major federal agencies such as the Pentagon, the Department of Homeland Security, the State Department, the Department of Energy, the National Nuclear Security Administration, and the Treasury, and other big targets like Microsoft, Cisco, Intel, and Deloitte, and organizations like the California Department of State Hospitals, and Kent State University. Security watchdogs run out of adjectives (“11 out of 10”) in characterizing the magnitude of this hack.
At the same time, security experts cannot help admiring the sheer artistry of this exploit. Hackers themselves often view their codes as a work of art. According to one cybersecurity expert, “Programmers and hackers like to sign their work like artists…So they sign that code in various ways. Often, they’ll leave their initials or they’ll try to be cute and put some sort of cryptic message.” So how was this hack accomplished?
SolarWinds makes the Orion network management software, which is widely used by enterprises and government agencies to help monitor and manage IT operations at scale. Back in March 2020 or probably much earlier, attackers were able to infiltrate SolarWinds and inject malware into their product. SolarWinds had been called out earlier for sloppy security – – a security expert back in March 2019 notified them that the pitiful password “solarwinds123” to their update server was widely known on the internet. Anyway, this injected malware was then spread nationwide, as SolarWinds pushed out the next update to its software. I can imagine the hackers clinking glasses of vodka as they chortled over 18,000 earnest IT managers in the U.S. dutifully downloading the latest version of the SolarWinds package, thinking they were doing the right thing to protect their organization’s security.
Microsoft published an analysis of how this hack was done. The attackers inserted a few innocuous-looking lines of code in the program, outlined in red below:
Who would suspect a bit of code called “OrionImprovementBusinessLayer”? Well, this code activated some further inserted code that was disguised as just strings of numbers and letters. Ultimately this “backdoor” code allowed operatives on another continent to have hands-on-keyboard access within the information systems of the victim organizations. Once inside a network, attackers could elevate their privileges and move laterally to do lots of things.
This was all done with a deft, light touch. The hackers were clever about not being detected. From afar, they tested the running of their software in subtle ways, and gradually introduced more features. The malware would only run in the actual live production software; it would go dormant when the software was being tested in a side “sandbox” mode which companies use to test and examine their software. The initial malware enabled the injection of a different level of spyware known as Cobalt Strike. However, the two sets of malware were kept separated, so if Cobalt Strike were detected and removed from a victim’s system, the attackers could still exploit the initial malware. Finally, in June, 2020, after the initial malware had been distributed around the country in the update pushes from SolarWinds, the hackers slipped back into SolarWinds and removed the initial malware from the company’s version of the Orion software package. Thus, if down the road users started getting suspicious about the SolarWinds software, they could audit SolarWinds all they wanted and find no trace of tampering.
How the Hack Was Found
It is said that the perfect crime is one which is never detected. By that measure, the SolarWinds exploit came close to perfection. In an update, Microsoft has described the timeline of the later stages of this attack. From May through December of 2020, the hackers had enormous access to thousands of organizations, with no suspicions raised. Nobody wants to talk about the full extent of the data that was compromised. However damaging the ultimate results of this information theft turn out to be, it does seem that the attackers limited themselves to simply stealing information. Thus, this would be classified as cyber espionage, not a “cyberattack” where the attacker does damage like wiping out computers or taking down the electric grid (which Russia has done in the past, to Estonia and to Ukraine).
In America we have an enormous national security apparatus, which totally missed this information theft going on under its nose. The detection of this breach was due to picky, obsessive diligence on the part of a private security firm named FireEye. This company went to huge lengths to figure out a small anomaly that most organizations might have shrugged off. This is a pointed case of a sharp private enterprise out-performing large government-run organizations. Here is an interview which explains what happened:
One of the things that really has caught my attention from the beginning is that the State Department didn’t catch it. Treasury didn’t catch it. Microsoft didn’t catch it. A much smaller cybersecurity company caught it. How did that happen?
Yeah, so this is not good. Not just the companies and the agencies that you mentioned, but NSA, the National Security Agency, the home of most of the brains in the government of cybersecurity expertise, they are a customer of SolarWinds. They had this stuff, and they didn’t catch it. And one of the NSA’s many jobs is reviewing code of software suppliers to particularly DoD, one would assume, and they missed it. That’s a really big red flag for the system we have.
So what happened is that FireEye found it, and FireEye, which subsumed Mandiant a few years ago, is among the very best-known, most sophisticated cybersecurity companies on the planet. They’re quite famous in the industry, and justly so. You can imagine that they have all kinds of security, because you might think that security companies aren’t an obvious target for serious hackers, because they’re more likely to be detected. But actually, they’re a major target, because they have awesome access inside all sorts of things, and you can find out how they find out about you as a hacker. So it’s a major prize and a really effective means if you can compromise security software.
But FireEye does have good defenses, and one of the things they have is two-factor authentication for their employees. There was a notification that one of the employees had activated a new device to verify himself coming into the network. So they caught that, and they asked the employee, “Hey, do you have a new phone?” The employee said no, and then FireEye began digging. They’ve done a number of things that were really, really good on this. So one is that they didn’t ignore this as a potential false positive. They actually went nuts, and they couldn’t figure out how the bad guys had gotten in as far as they had. So they went digging for how they could have gotten in, and eliminated basically all the more straightforward ways in. Then they started tearing down the code of the software that was on the servers that were compromised, and that is a nightmare. I mean, that is not something you want to do.
FireEye found that the intruders made off with copies of the special tools the company uses to detect its clients vulnerability to being hacked. Possession of these tools can enable the attackers to mount better hacks and to avoid detection, so this loss was painful. After painstakingly digging through some 50,000 lines of code on their servers, the FireEye folks found that the malware was hidden in the SolarWinds code.
The hackers established thousands of “clean” (never used before for attacks) email addresses across the U.S. to help stage their exploit. The CEO of FireEye paid them this tribute:
The attackers tailored their world-class capabilities specifically to target and attack FireEye. They are highly trained in operational security and executed with discipline and focus. They operated clandestinely, using methods that counter security tools and forensic examination. They used a novel combination of techniques not witnessed by us or our partners in the past.
At first, FireEye was only aware that they had been hacked. It had to be embarrassing to admit, considering they were in the business of protecting organizations from being hacked, but FireEye did the right thing and promptly informed the security community in early December of its findings. But it soon became clear that the problem was way bigger than just one firm. If it were not for FireEye’s diligence, the data breach would probably still be running.
Economist Writing Every Day 