No matter what your business is, you rely on systems that routinely must be patched. Vulnerabilities exist from the hardware and operating systems to applications and everywhere in between. And vulnerabilities that remain unpatched become liabilities to the health and security of not only your systems but your data and customers as well. Virus and malware scanning isn’t enough to protect you. And with the threat of ransomware becoming more prevalent, every unpatched vulnerability puts you, your business, and your customers at further risk.

Considering the critical role of patching, why do so many vulnerabilities remain unpatched? In its latest State of Software Security (SOSS) report, CA Veracode reports that up to 70% of bugs remain unpatched four weeks after disclosure, and close to 55% are not resolved three months after discovery. Moreover, 25% of vulnerabilities given high-severity ratings are not addressed within 290 days, and a quarter of disclosed bugs that are less critical remain unpatched well after a year.

The patching deployment process itself is not hard. But implementing patches in certain environments definitely can be, such as environments that consist of thousands or tens of thousands of servers or systems where downtime is not an option. As environments increase in complexity, the patch deployment process does as well.

Considering that most enterprises have a mixed bag of systems, there can be multiple native operating system tools on hand to deploy patches, such as Systems Center Configuration Manager (SCCM) for Microsoft Windows or Yum, Apt, and Zypper for major Linux distributions. Multiple tools do not mean patching is more successful. In fact, they add new challenges such as dependencies, multiple panes of glass and inconsistent patching processes.

On top of the operating systems, there are also many applications that require frequent patching. Some applications have their own automatic updating frameworks, such as Java Runtime Environment or Adobe Flash, but allowing systems to patch themselves is problematic. What if half of the employees in an organization don’t install that critical zero-day Flash patch? What if server admins don’t install Oracle updates because they break compatibility with legacy applications? In these cases, it’s not the technical aspects of deploying patches that is challenging; it’s what must be done to ensure full and continuous vulnerability remediation.

Patching Requires Testing

Any organization with a mature patching model always tests patches in a non-production environment first, hopefully using automation. Though kernel-level crashing is less common nowadays, patches can still introduce issues at the operating system level, such as software patches breaking functionality or interoperability with other applications.

Even if a patch doesn’t completely cripple a system immediately, problems can arise after a reboot or short bake-in period. To solve this issue, many organizations roll patches out in a tapered fashion. An example of this would be first deploying all current patches to pre-production, testing and letting them run for two weeks, and then deploying them to production. This is common but requires a tremendous effort to orchestrate across multiple patching mediums.

As a result, the need for testing reduces the number of patches for two reasons:

• Patches break application functionality, meaning the vulnerability won’t be patched or is only patched later.
• Patching is delayed to allow for time to validate functionality in a non-production environment, meaning production remains unpatched longer.

Patching Can Require Downtime

There are many cases where patching requires a system reboot. It could be a hardware patch to a Storage Area Network controller, an operating system patch, or even a patch for network equipment. Some systems are architected for high availability through techniques like clustering or data replication. But these techniques also typically require patching to be deployed carefully, with rolling restarts that are orchestrated to prevent service or data loss. These rolling restarts help maintain service availability, but what about systems that aren’t as flexible?

Some systems or applications require a full outage to remediate vulnerabilities. That means that the event must be carefully planned, scheduled, and executed with roll-back plans in place. Considering the cost and orchestration required for this, it may take companies months to schedule planned downtime (for instance, on a holiday weekend), or they may decide to opt out of the patches all together.

To summarize, some vulnerabilities aren’t patched because of the need for downtime:

• Downtime is costly due to potential lost revenues, and/or lost productivity.
• Designing for high-availability can mitigate some of these costs, but rolling deployments require additional orchestration and cost.

Patching Is a Cycle

Those who have managed vulnerability remediation understand that you are in a continuous cycle of deploying patches. NIST CVE articles are published constantly on vulnerabilities for different systems or applications, with each identified vulnerability being assigned a level of criticality and required effort as well as remediation instructions. The wider the adoption of operating systems, vendor applications, and equipment, the larger the set of vulnerabilities to account for and patch.

Patching Cycles Can Be Disrupted

This struggle becomes real when coordinating the efforts of the prior two sections (testing and downtime). Companies want to deploy as many patches as possible at one time because the cost is lower in allocating resources and scheduled downtime windows are fewer. However, many companies fall under regulatory requirements, such as HIPAA, SOC, or PCI to name a few, and it is common for these regulatory accreditations to require companies to remediate vulnerabilities within a certain timeframe (e.g., critical vulnerabilities within 15 days).

Another disruption is zero-day vulnerabilities, which are always critical in nature. Both regulatory compliance and zero-days can force patching off cycle, thus mandating expedited vulnerability remediation schedules. And fixing vulnerabilities under such conditions greatly increases the pressure on companies and their systems teams.

Patch cycle disruption can be extremely difficult for organizations with thousands or tens of thousands of systems. Anyone who’s managed a large data center knows that deploying patches to reach 100% across tens of thousands of nodes is an extreme challenge. Some systems will have different versions deployed, others may be temporarily offline, some may exhibit strange behaviors, or the patch can fail to install. This is all perfectly normal and yet extremely challenging to manage properly.

Why vulnerabilities aren’t patched because of processes, obligations, and surprises:

• Zero-days require immediate action. If the test and validation processes are not excellent, fixing zero-days introduces time-induced risk.
• Zero-days are more difficult to coordinate remediation for as the number of systems grows.
• There are many factors that determine how frequently patch cycles occur. The further apart the cycles are, the more patch disruptions cause issues, and the longer systems are vulnerable to attack.
• Larger data centers can fall into the cycle of never-ending patching to try and catch up on systems or applications that fail to patch.
• Meeting regulatory objectives may place extra pressure on patching teams to mitigate patches before they are properly tested or by the next patch window.

The Key to Proactively Controlling Vulnerabilities

With so many obstacles to overcome in vulnerability remediation, it becomes critical to address all of these issues via a unified approach. The best way to traverse from vulnerability to safety is through robust orchestration. JetPatch provides such needed orchestration to allow your organization to address these challenges.

Orchestration

JetPatch has the ability to proxy multiple operating systems and application vendor patches through a single pane of glass, giving you the ability to deploy in a unified fashion via a suite of controls. This reduces the level of overhead required to coordinate patches through disparate systems, tools, and vendors. And this level of orchestration provides a unified way to understand the threat picture and where your organization stands in the vulnerability remediation cycle across all of your ecosystems. It’s much easier to understand that one system is 85% patched for all vulnerabilities, rather than 90% patched on Microsoft Windows, 45% on Oracle 12i, 70% on Java related applications, etc.

Rapid Vulnerability Remediation

Building on this orchestration, JetPatch allows you to deploy your unified patching solution with the mere push of a button. Unlike some tools that can often be delayed when deploying patches, such as SCCM, JetPatch enables you to have finer control of your patching windows. This rapid deployment technology can also assist in closing those zero-day vulnerabilities much faster.

Automated Rollouts

Wouldn’t it be nice to not have to manually deploy and oversee your patches? Using JetPatch’s technology to automate rollouts, your administrators can plan and deploy remediations during pre-determined patch windows. This means your engineers don’t have to babysit patching and can better spend their time remediating one-offs or troubleshooting issues. This technology can also simplify rolling restarts or cluster patching by eliminating the need to manually manage the process, thus reducing the risk of human error.

Centralized Governance

It is critical that security personnel and administrators understand the picture of what is currently vulnerable. The reporting and tracking tools that JetPatch provides give you this information. And this, in turn, empowers teams to make the necessary changes to either remediate vulnerabilities in different ways or simply be aware of just how much still remains to be patched in their environment. The patching policies and reporting can also be customized to generate this information from a unified picture—rather than compiling the data from several tools—to better understand the full scope of your current vulnerability threat.

Process Management

Many organizations require change control or a patching process to remediate vulnerabilities. Whether it is company policy or a regulatory requirement that certain testing, validations, or security sign-offs be met to deploy patches, JetPatch allows you to implement any such controls.

Summary

Managing vulnerabilities is difficult. It’s no wonder that so many companies fall victim to threats because of unpatched systems when you consider the challenges of not just remediating vulnerabilities, but remediating them as quickly as possible. These challenges can be overcome via tools like JetPatch, which offers a solution to simplify orchestration, improve governance, automate deployments, and adhere to processes and controls.

When considering your threat remediation challenges, consider using JetPatch’s unified toolset with robust capabilities in order to successfully avoid the chaos and worry of dealing with unpatched vulnerabilities.