The recent CrowdStrike Falcon Sensor update caused significant disruptions, highlighting the need for robust agent management strategies. Let’s explore how we can prevent and mitigate such outages in the future.
What Happened
On July 19, 2024, a channel file update from CrowdStrike triggered a logic error, which in turn caused the Falcon agent to crash Windows systems into a BSOD (blue screen of death) loop. This affected systems running Falcon sensor for Windows version 7.11 and above that downloaded the update between 04:09 UTC and 05:27 UTC. These channel file updates occur regularly and automatically as part of CrowdStrike’s protection mechanisms, but this time led to an extensive outage. For more technical details, you can visit CrowdStrike’s official technical details page.
How JetPatch’s Agent Manager Could Have Helped Mitigate the Issue
JetPatch provides a centralized view of all agent deployments across endpoints and servers, automatically identifying and fixing misbehaving agents. If JetPatch had been in place, it could have sent a command to stop or disable the Falcon service, preventing the BSOD loop. This would allow users to start using their machines again, while also allowing IT teams to remotely access the machine, remove the problematic C-00000291*.sys file, and then restart the Falcon service without further BSOD or other disruptions. This approach would have allowed the affected machines to resume in minutes instead of hours or days.
Improving CrowdStrike’s Rollout Strategy
To prevent such widespread issues in the future, CrowdStrike can enhance its rollout strategy for channel updates. Instead of pushing updates to all machines simultaneously, a staged rollout process can be implemented. This method involves deploying updates to a smaller subset of systems first, monitoring for any issues, and then gradually extending the rollout to the rest of the infrastructure. This approach allows for the identification and resolution of potential problems before they impact the entire environment, averting potential outages.
Broader Protection Against Agent Crashes from Other Vendors
This incident with CrowdStrike is just one example of how agent updates can cause significant disruptions. Similar issues can arise with other vendors’ agents due to updates from the agents themselves or from OS vendors like Microsoft. Therefore, it is essential to protect your infrastructure with comprehensive agent management.
JetPatch Agent Management Overview
JetPatch offers extensive protection and management for various agents from different vendors, ensuring a resilient IT environment. Here’s how JetPatch provides layers of protection and mitigation:
- Centralized Management: Provides a unified view of all tools and agents, ensuring comprehensive monitoring and control.
- Policy-Based Automation: Automates deployment and management based on customizable policies, ensuring consistent compliance and operation.
- Resource Monitoring: Tracks and adjusts resource consumption to prevent overuse and potential system crashes.
- Watchdog Mechanism: Ensures continuous operation of agents and services, with automatic detection and remediation of issues.
In organizations with diverse infrastructures, multiple software management tools are often required for performance monitoring, security, and backup. JetPatch centralizes the orchestration and management of these tools, ensuring proper configuration and operation without manual intervention.
Two Layers of Protection Offered by JetPatch
- Control Over Various Agents: JetPatch provides full control over agents from different vendors, monitoring and managing their behavior. For instance, if any agent starts using excessive resources, JetPatch can throttle its usage without affecting the service or device.
- Staged Agent Upgrades: Similar to the suggested improvement for CrowdStrike, JetPatch stages rollouts of agent updates, ensuring a controlled and gradual update process. This reduces the risk of widespread issues and allows for prompt mitigation of any detected problems.
For more details, please visit JetPatch Automated Agent Management.
Best regards,
Todd Kirkland
CEO, JetPatch
