When you walk into a building lobby, your first impression will generally be positive if attention has been paid to the little details—helpful signage, tasteful and well-maintained decorative elements, clean and comfy furniture, and, of course, free WiFi with a good signal. There’s a comforting feeling that those in charge know what they’re doing and take pride in their work. By the same token, we would argue that patch management process excellence is the trademark of a well-run IT shop. If fundamental IT best practices are firmly established and enforced, then patch processes will proceed smoothly as well. But if IT is managed with insufficient attention to detail, then patching can easily become more about putting out fires than contributing to a consistently high-security posture.
Patch management in and of itself is not rocket science. However, good patch management processes require full visibility into and categorization of all IT assets, diligent monitoring for patches and vulnerabilities, and well-defined test-deploy-document workflows that are as automated as possible and minimize downtime.
In this blog post, we provide best practices, tips, and tricks for taking your patch management process to the next level.
What Are Good Patch Management Processes?
Good patch management processes keep IT assets proactively secure against known vulnerabilities, with minimal downtime to production systems. In order to assess if your patch management processes are optimal, you should be asking questions such as:
- Is your current patch management process well-documented—and is it being implemented accordingly?
- How do you collect patches from all the relevant sources? Is it possible that a patch will be published and you won’t know about it?
- How do you test patches across your different environments? How often do you have to roll back deployed patches?
- How do you decide when to patch what? Have you ever suffered an outage or breach because a patch was not deployed in a timely manner?
- Is your entire organization ready and enabled to do critical patching of all systems in just one downtime period?
If you don’t have convincing answers to these questions, then your patch metrics and KPIs are most likely sub-optimal, and it’s time to clean up your patch process.
Discovery and Prioritization
Two critical prerequisites for an effective patch management framework are a thorough and well-maintained inventory of IT assets and clear criteria by which the assets are categorized in terms of risk and impact.
The inventory discovery process should include, at a minimum:
- All production systems, including their OS, OS version.
- Third-party apps, services, and libraries.
- All security controls (firewalls, routers, Intrusion Detection Systems, etc.) and their configurations.
- Network and edge (IoT) devices.
- Employee handheld devices that are authorized to access assets.
The more automated the discovery process, the more likely that the IT inventory will be thorough and kept up to date. In particular, watch out for hidden systems that run behind the scenes and are easily forgotten. It’s also a good practice to aggregate the information in a Configuration Management Database that is accessible to all the relevant stakeholders. Last but not least, leverage the inventory gathering process to find opportunities to standardize and harden your production systems.
Once discovered, properly categorizing IT assets is another essential step in achieving patch objectives. According to the CVE Details website, in 2018 16,555 Common Vulnerabilities and Exposures were reported, up from 14,714 in 2017. In other words, given the volume and velocity of vulnerabilities and their patches, no organization is going to be able to achieve 100% patch coverage. Hence the importance of understanding which assets and workloads have the greatest impact on the business and giving them the highest priority in terms of patch deployment.
In addition to categorizing by impact, it’s important to categorize assets by their risk profiles as well. For example, mission-critical customer-facing apps are often exposed to the Internet and untrusted networks or users. With both high impact and high risk, these apps will get the highest patching priority.
Monitoring for Patches and Vulnerabilities
You can’t patch if you don’t know what vulnerabilities are out there and what patches have been made available to address them. There are several open-source resources that track and record vulnerabilities, including:
- CVE® (Common Vulnerabilities and Exposures): A searchable, risk-ranked listing of publicly disclosed cybersecurity CVEs, each with a unique identifier. An entry may include advisories as to how to mitigate the vulnerability, including patches that are available. Compiled by CVE Numbering Authorities around the globe, the CVE® dictionary is free to use and to incorporate into products and services. It is managed by the MITRE Corporation and sponsored by various agencies within the US Department of Homeland Security.
- CWE (Common Weakness Enumeration): Also managed by the MITRE Corporation, CWE is a community-developed listing of software security weaknesses. It provides a baseline for identifying, mitigating, and preventing software-related security vulnerabilities.
- National Vulnerability Database (NVD): Managed by NIST (National Institute of Standards and Technology), the NVD is the US government’s repository of vulnerability management data. Structured under the Security Content Automation Protocol (SCAP), the NVD encourages automated vulnerability management and compliance. Providing impact scores per vulnerability, the NVD includes security checklist references as well as security-related software flaws and misconfigurations.
In theory, it is possible for an organization to build home-grown tools to track these and other resources for vulnerabilities and their patches. However, with today’s highly complex and distributed IT stacks, manual or semi-automated monitoring is unlikely to provide the coverage necessary for truly effective patch management processes. Thus, we have seen the emergence of next-generation patch management platforms, such as JetPatch, that tightly integrate cutting-edge real-time vulnerability scanning with the organization’s Configuration Management Database and rule-based patch workflows.
Scheduled Patches: Testing and Rollout
Testing and rolling out scheduled patches is among the biggest challenges of a patch management process. Each patch should be tested for each combination of OS, device, and infrastructure on which it will be deployed to ensure that it does indeed fix the vulnerability but also does not break other dependent resources.
A typical patch testing and rollout workflow takes about three weeks and comprises the following main stages:
- Test Stage: In this stage, the test environments that model all the relevant production environments are provisioned, and the test protocol is determined, including who carries out the testing and what the go/no-go criteria are. Although it might be tempting to cut corners when setting up testing labs, breaking something in the production environment due to a poorly tested patch will be much more expensive in the long run.
- Pilot Stage: After successful testing, a pilot stage should be initiated in which the patch is deployed on a limited basis for selected users. In this way, the patch team gets invaluable user feedback without risking entire departments or locations.
- Deployment Stage: After issues that arose during the pilot stage are dealt with and tested, it is time for a full rollout. The key issue to be addressed during this stage is how to deploy the patch(es) with minimal downtime. Mission-critical apps, for example, should be patched whenever possible during non-peak hours, taking into account time zones where relevant. There should also be some flexibility built into the rollout, including the circumstances under which a user can be excepted from installing a patch.
NOTE: Despite all the testing, it is still possible that problems will be encountered during deployment. Thus, a clear and tested rollback plan must be part of your patch management process. - Analysis and Reporting Stage: In this stage, the patch team continues to audit for issues and problems. It is important to gather metrics on specific patches and on the patch management process in general in order to verify that patch objectives are being achieved—or to discover where optimizations are needed.
Dealing With Emergency Patches
When dealing with routine patches, the rule of thumb regarding a patch schedule is that desktop and server OSes, server apps, malware, and virus protection tooling, VPN clients, and security tooling should be patched on a monthly basis. Physical and virtual appliances, hypervisors, and management toolings should be patched every quarter, while infrastructure firmware, drivers, and management apps should be patched every six months.
Bear in mind, however, that there will inevitably be critical, emergency patch situations where the test and rollout processes will have to be adapted and accelerated. For example, scheduled patches are highly visible to end users, who usually have the option to delay or opt out of patch installation. Emergency patches, however, must be enforced “brutally,” with little or no options to the end user in order to avoid situations like this:
Image source: Model Technology Solutions
Conclusion
JetPatch is an advanced patch management solution that helps companies large and small achieve patching excellence. JetPatch has been designed from the bottom up to address the challenges of the patch management process, as described in this article:
- JetPatch automatically discovers and maintains patch information for all relevant IT assets—OSes, servers, apps, and so on—to ensure complete, error-proof visibility.
- JetPatch also establishes rules-based workflows to implement a consistent and infrastructure-agnostic patch remediation process across an organization.
- JetPatch integrates seamlessly with all leading vulnerability scanners to continuously monitor for and prioritize existing and new vulnerabilities.
- JetPatch also integrates with your current patch management, ITSM, downtime management, and job scheduling tools for accelerated patch testing-staging-production cycles and automatic rollouts that minimize downtime.
Schedule a demo today to see firsthand how JetPatch can take your patch management process to the next level.
