“It takes a village to raise a child” is a wise African saying. This same insight can be applied to many other areas of life, including enterprise IT. There is an ever-growing body of rigorous studies that clearly show how collaboration and effective teamwork in the workplace significantly enhance employee satisfaction and are the foundation for high-performing businesses.

Similarly, effective and sustainable patch management requires collaboration across the entire enterprise. Above and beyond the security team that is directly responsible for patching operating systems and applications, there are many other stakeholders within the organization who play a critical role in patching operations—internal and external end-users, development and operations teams, customer service, business line teams, and managers.

This article discusses organizational best practices for patch management, who in the organization should be involved,  and how to foster collaboration with technology that automates an end-to-end, well-governed patch management process.

Who Are the Patching Stakeholders?

Everyone in the enterprise understands, in theory, the importance of patching to keep operating systems and other applications up-to-date and secure. But those same people will get upset if, in practice, the patching operation disrupts workflows or has a negative impact on the customer experience. This contradiction between theory and practice often places patching teams between the proverbial rock and a hard place. One of the best ways to extricate the team from this rather uncomfortable situation is to get cross-departmental inputs and buy-in into the corporate patching process.

A good place to start is to understand the impact that patching has across the organization and thus identify the stakeholders that should be brought on board, such as:

• Business line managers who, on the one hand, want the apps and services that their teams use to carry out their mandates to be secure and compliant but also want to avoid, or at least minimize, downtime. An oft-quoted metric for the average cost of IT downtime is Gartner’s estimate of $5,600 per minute. Although this estimate was calculated in the context of unplanned IT downtime, there is no question that even planned downtime for patching can have a quantifiable impact on line-of-business performance throughout the enterprise.
• The internal consumers of corporate IT services are also affected by patching downtime as well as by patches that “break” the applications they rely on. Nobody wants to see a system reboot notice pop up when they are in the middle of a task. Even more so, users don’t want to experience difficulties in getting back into an application after they have begrudgingly consented to the reboot.
• Customer and partner support teams are also highly impacted when patching affects external end-users. They need to be able to inform customers and partners of impending downtime. They also need to be given the information and tools to handle the many queries that will come flooding in should the patch affect app performance.
• DevOps teams have a lot at stake whenever an OS or other infrastructure software undergoes a patch. Today’s highly modular applications have many dependencies and a change to any component in the application layer can have unexpected results on deployed apps or apps under development. Just one simple example is that patch management teams may have to wait for developers to update their Java applications before they can apply patches for the Java Runtime Environment.
• Network support teams also need to be in the patching loop. At the very least, it is important to avoid situations where the knee-jerk reaction to a network failure is assuming that it must be the fault of the patching team. More importantly, in the case of cloud and/or hybrid infrastructures, the network team must provide insight into the organization’s complex attack surface.

Last but certainly not least, the entire C-level suite—and not just the CISO or CSO—has to buy into and support the patching process. They have a particularly important role to play if, for example, there are conflicts over patching between IT and business line managers.

Best Practices That Foster Patching Collaboration

Establishing a collaborative patching framework is a strategic endeavor. As such, it requires a clear statement of patching objectives, a carefully designed and well-documented process, effective governance to oversee execution and adapt the process to changing needs, automation, and, last but not least, communication, communication, communication.

Whether the corporate patch management policy is stand-alone or part of a broader Information Security Policy, it is important that executives across the full spectrum of business functions provide input on a statement that establishes clear patching guidelines and measurable objectives. Although it might be the CIO or CISO who initiates and leads the policy-making process, all managers with executive responsibility must sign off on and stand behind the final decisions.

Once the patching guidelines and objectives have been articulated, it is up to the various department heads and function managers to devise and document a detailed, peer-reviewed end-to-end patching process. The process should be broken down into pre-patching, patch deployment, and post-patching stages, with clear allocation of responsibility and oversight for each stage and task. The process must include the patching metrics to be tracked, how and in what resolution, and who receives the reports. The documentation should also include a cross-departmental governance framework that regularly monitors patching outcomes vs. objectives as well as updates the process to keep up with the needs of a dynamic environment.

Other best practices that foster a collaborative approach to patching are:

• Promoting cross-departmental cooperation to enhance the efficacy of patch testing before and after deployment. For example, all relevant departments and stakeholders could collectively maintain a centralized configuration management database that tracks the dependencies to be taken into account when testing patches prior to deployment. It is also an excellent practice to recruit internal end-users from across the organization as application experts who can most effectively test the impact of patches on applications.
• Encouraging standardized build environments to the greatest extent possible in order to reduce the complexity of patching.
• Grouping applications and systems to facilitate batched patch installations along with other efficiencies that minimize disruption and downtime.
• Flexible, phased scheduling of patches to mitigate their impact on 24/7 services. A more expensive but highly effective practice to minimize the impact of patches on mission-critical apps is to maintain a failover replica that is launched while the production infrastructure undergoes patching.
• An adaptive patching process that is aligned with corporate change management processes, with clear guidelines regarding patching opt-outs or exceptions based on risk assessment—including alternative controls that can be implemented instead of a patch.
• Converging security and operations teams (SecOps) as much as possible in order to mitigate possible disconnects between patch management and vulnerability resolution. For example, it is not unusual for a cumulative Microsoft update to include a CVE that requires additional steps (beyond simply installing the patch) in order to fully mitigate the vulnerability.

Finally, the backbone of collaborative patching is good communications. The more that knowledge is shared across the organization on the importance of patching, potential pitfalls, and how each team can contribute to a seamless, collaborative patching process, the less likely the direct patch management team will find itself in conflict with other lines of business.

The Importance of (True) Patch Automation

Given the scope and complexity of patch management, it is critical that the patching process be as automated as possible. True end-to-end patching automation, around which all stakeholders in the enterprise can collaborate in order to establish effective and sustainable patching, must uphold all three of the following principles:

• Vulnerability scanning and analytics that help the patch management team understand the priority of any given patch for its specific organization.
• Process governance that enforces well-defined patching workflows.
• End-to-end automation of all patching tasks, including those that occur before and after actual patch deployment.

You can learn more about the importance of these principles in the article “The Three Principles of Patching: Ignore at Your Own Peril.”

JetPatch has integrated its in-depth understanding of enterprise-grade patching best practices into an industry-leading patching automation platform. We invite you to schedule a demo during which we can show you how JetPatch allows enterprises to establish a collaborative, sustainable, and secure patching process.