In 2019, despite better software development practices, increased automation, and more security awareness, there is still an amazingly large amount of vulnerabilities. CVE Details shows that over 16,000 vulnerabilities were identified in just the last year. But what does that really mean—16,000 vulnerabilities? If you’re a large enterprise with approximately 75% of these vulnerabilities relevant to your systems, it might be alarming. Yet that data isn’t helpful in addressing the actual challenge. Regardless of the numbers, you need to resolve the underlying problem. And to do this, you need to align your best practices for patch management to close the gap on your vulnerable systems.

To better understand how to meet these challenges directly, we will briefly cover the three most important principles of patching and also cover a few resources you should know about. We’ll then dive into a few vendors who operate in the patch/vulnerability management space and then explore best practices you’ll want to adopt.

The Three Primary Principles of Patching

There are many disciplines in patch/vulnerability management, but enterprises that effectively manage their security threat picture have concrete answers to the following three principles.

1. Vulnerability Scanning and Analytics

The scanning and analysis of vulnerabilities is decades old, and both tools and threats have evolved over the years. The need to continually evaluate and remediate is just as important—if not more important—today, as enterprises are scaling out, scaling across cloud platforms, and building faster than ever before.

2. Patch Process Governance

Despite the fact that patching has become more automated, the basis of patching and remediating vulnerabilities falls into security plans, policies, and procedures. Following these patch management processes allows for effective management so that vulnerabilities are consistently mitigated or remediated.

3. End-to-End Patch Workflow Automation

As enterprises grow in size and complexity, automating the cycle of scan -> analyze -> test -> deploy -> scan is a burdensome but necessary effort. No Microsoft Windows team wants to spend every weekend of every month patching thousands of servers the way they were patched five to ten years ago.

Read more about this in The Three Principles of Patching: Ignore at Your Peril.

The Basics: Security Threat Advisories and Guides

There are many advisories and guides available to help you better understand security vulnerabilities. These resources are referential in nature and are not meant to understand which patches you should apply to what systems.

The National Institute of Standards and Technology (NIST) provides information and resources for Common Vulnerability and Exposures (CVEs) for each vulnerability, including for well-known threats such as Spectre (here and here) and Meltdown (here). NIST also provides a National Vulnerability Database (NVD), which has many other helpful resources.

The Defense Information Systems Agency (DISA) releases Security Technical Implementation Guides (STIGS), which provide extensive details on hardening operating systems, applications, networks, databases, etc. This site includes available guides such as Microsoft Windows operating STIGs.

Chasing Advisories Does Not Scale

If you’re familiar with the resources in the previous section, it becomes clear that it would be impossible to research each and every vulnerability manually. This is where tools come in. It’s impossible to research all possible vulnerabilities for your Ubuntu 16.04 systems or your Oracle 12i databases, but a vulnerability scanner will not only tell you what vulnerabilities are open or closed but also give you references to further reading. Some tools will even deploy patches or close vulnerabilities for you.

There are many vendors that perform patch and/or vulnerability monitoring and scanning. For example, Tenable Nessus and Rapid7 are excellent at scanning systems for missing patches and security vulnerabilities. JetPatch provides a suite of tools that can scan for vulnerabilities as well as manage and deploy patches to multiple operating systems and applications.

Best Practices in Patch Management and Vulnerability Remediation

So far we’ve covered three solid patching principles to follow, resources to help you understand the security threat picture and tools to aid you in remediating those threats. Now these best practices below will help you stay ahead of any possible vulnerability exposure.

Make Scanning and Remediation Part of Your CI/CD

Scanning and remediation is a lengthy process. While some enterprises scan quarterly, or even annually (or even less frequently), patching typically takes place on a monthly cycle. These time gaps were acceptable years ago when software releases were quarterly or annual. In today’s world, however, software CI/CD can mature to daily or multiple daily releases.

It’s likely that your enterprise has already moved or is moving to delivery pipelines, which are popular for good reason—they’re flexible to best control the automation of software releases. This means that all necessary patches and vulnerability remediations should be deployed as part of those delivery pipelines. Adding steps to the pipeline to re-scan for vulnerabilities post-release is a good way to ensure that a controlled baseline of remediation is moving through each software environment, i.e., from development all the way through to production.

Update Baseline Images to Speed Time-To-Provision

Many enterprises will have a mixed environment of legacy systems, long-living servers, stateless servers, containers, and perhaps even serverless workloads. Since vulnerability/patch scanning and monitoring is a lengthy process, it makes sense to have a fully scanned/patched image baseline. Often called a “gold image” or “secure baseline,” these images can reduce overhead and time-to-provision.

As enterprises move closer to stateless environments that are disposable in nature, the process of having gold images becomes more critical. As a basic example, if it takes an hour for a base Microsoft Windows 2012 R2 server to be fully patched prior to the next steps of provisioning, why not put the process of building the baseline and creating the image into an automation pipeline? That way, you have a known baseline for patching, but your further automation of applications/databases can build directly on top of that gold image.

Updating all gold images should be part of a delivery pipeline that executes no less than once a month. Ideally, the workflow might look like this:

• Brand new server launched
• Fully patched (reboots included)
• Vulnerability scan
• Integration smoke testing (such as Serverspec)
• Gold image cut

Rapid Deployment to the Network Boundary

Patch and vulnerability management at the security border of your infrastructure is the most critical. This means that the border must receive patches and remediations as quickly as possible. Unfortunately, the boundary systems for many enterprises are often the revenue point, such as an online store or a B2B integration solution. So patching externally exposed systems is extremely critical.

Building on the last two best practices, an excellent way to rapidly deploy patches to the border with confidence is having automated patch workflows. Here below are two deployment methods to help you quickly adjust to issues that can arise during this process.

Blue/Green

Whether you’re using a cloud technology like auto-scaling groups or packs of bare metal servers in your data center, you can use blue/green deployments. This simply means that you have two equal pools of systems, but only one is active at any given time. When a change (such as patching) happens, the load balancer for those systems shifts and the standby systems become active.

This gives you the advantages of flexibility (testing, rollback), speed, and durability (having a fallback environment is always handy). The disadvantage is one of cost—double the amount for systems with traditional deployments.

Canary

The canary ratio for different border systems can vary. But for the purpose of illustration, let’s say 10% of every border system is the canaries. Those systems would be patched or mitigated first, often immediately after vulnerabilities are discovered. These canaries co-exist with the other systems as part of the active pool. If the patches or hardening causes issues, it will be evident in your API/service (i.e. monitoring). Should issues arise, the canaries can be removed until the problem is addressed and another pool of 10% is selected for the next change.

The advantages here are cost, as there are either no or few additional systems, immediate feedback, and the ability to test before deploying the rest. The disadvantage is that this model does not implicitly allow for any easy rollback or failover environments.

Summary

Choosing your patch and vulnerability monitoring and scanning strategies requires a solid understanding of what must be done. You must also rely on the right resources and patch management tools that can help you understand and remove the threat along with the proper best practices to help ensure it all comes together successfully. Patch management is a constant need in every organization, and there are few vendors who can meet the needs of large enterprises and cloud-native workloads through governance and end-to-end automation.

JetPatch, however, is a vendor that can cater to these patch management needs. So when you are considering how to address the above concerns, consider getting a demo from JetPatch to complete your vulnerability management solution.

Further Reading

We mentioned that you need to choose the right patch management tools. We recently put together the ultimate patch management checklist: what to look for in a platform. For additional discussion of patch management best practices, read Time to Clean Up Your Patch Management Process.