Yesterday was a fun day. This might have been my favorite scenario:
Lots of affected PCs
They all have bitlocker turned on
Bitlocker self service portal and admin server on VMWare is down because Crowdstrike
VMWare environment requires daily rotating admin credentials to log into (They needed access to mount the affected drive on a good server and delete the file)
Can't get the rotated passwords because the on-prem password manager prod server is down because Crowdstrike
Can't restore the on-prem password manager because the backup server also has Crowdstrike on it
Can't use the password HA server in the cloud because Crowdstrike
Fortunately the cloud leverages that vendors nightly backup functionality which were intentionally scheduled to occur right after password rotation.
On the positive side for them the password manager solution was intentionally designed with multiple service/platform failures in mind. Not with this specific scenario in mind but because its the most critical of their IT assets.
So once they got the HA copy up the VMWare admins could finally log in and fix the Bitlocker server. Then the desktop support techs could finally get to work (Yes there were workarounds but with the scale of the issue it was tough to make a dent without the Bitlocker server)
TBH, aside from today, their sensor is pretty dang good. A lot of the installs I've seen have been in organizations with a broad range of scientific research efforts and its played well with 99.999% of rando compiled software and niche applications for research without issue. HPC? Fine. GIS? Fine. fMRI? Fine. Completely custom cluster and software? Fine.
Lots of affected PCs
They all have bitlocker turned on
Bitlocker self service portal and admin server on VMWare is down because Crowdstrike
VMWare environment requires daily rotating admin credentials to log into (They needed access to mount the affected drive on a good server and delete the file)
Can't get the rotated passwords because the on-prem password manager prod server is down because Crowdstrike
Can't restore the on-prem password manager because the backup server also has Crowdstrike on it
Can't use the password HA server in the cloud because Crowdstrike
Fortunately the cloud leverages that vendors nightly backup functionality which were intentionally scheduled to occur right after password rotation.
On the positive side for them the password manager solution was intentionally designed with multiple service/platform failures in mind. Not with this specific scenario in mind but because its the most critical of their IT assets.
So once they got the HA copy up the VMWare admins could finally log in and fix the Bitlocker server. Then the desktop support techs could finally get to work (Yes there were workarounds but with the scale of the issue it was tough to make a dent without the Bitlocker server)
I don't think the anti-virus description of these types of tools is the best analogy. It brings up memories of Norton or McAfee and there are a couple of big differences. This particular product of theirs is an "EDR" or Endpoint detection and response. It doesn't scan the computer (and take up tons of resources to do so) but monitors and blocks malicious activity (or what looks like malicious activity). So way more efficient and doesn't bog down the computer. Another big difference IT Security people need to be aware of is that most of it's settings will leave the offending file in place and it doesn't go after PUPs or PUAs. So I've seen places pair it with something like Malwarebytes - esp if the have a large Mac fleet as having a Mac fleet without giving everyone admin rights is unnecessarily hard.For the IT uninitiated what does Crowdstrike do?
TBH, aside from today, their sensor is pretty dang good. A lot of the installs I've seen have been in organizations with a broad range of scientific research efforts and its played well with 99.999% of rando compiled software and niche applications for research without issue. HPC? Fine. GIS? Fine. fMRI? Fine. Completely custom cluster and software? Fine.
Last edited: