Tracking an Adversary in Real-Time Using Velociraptor - part II
Velociraptor is a powerful open-source tool that we deploy during incident response engagements to conduct our forensic investigation of the attack. We use it to surgically collect data of high-forensic value on endpoints, and to hunt for adversary activity across endpoints.
Sometimes, when you're lucky, you get to deal with an active adversary that is still busy with achieving their mission. And when this happens you want to detect new hands-on-keyboard activity and receive an alert in near real-time. While detection and alerting is not the most typical use-case of Velociraptor, it can be used for that. And to us it has proven to be valuable during engagements when the deployed Endpoint Detection and Response (EDR) solution lacked certain detection capabilities, or when it was not deployed.
This blog post shows you two practical examples on how to detect adversary activity and alert on it using Velociraptor. It is an extension of part I, which provides two other examples. You can find part I here.
Example 3: Track registry modifications
Adversaries like to maintain persistence in the IT-infrastructure they accessed. A simple example of this is launching malware on user logon by adding a registry value 'malware.exe' to the Run key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
Of course there are more stealthy approaches for adversary to maintain persistence through the registry. One example of this uses ServiceDll's, which is a DLL loaded by a Windows Service. Microsoft uses this to load a DLL into the shared services process named svchost. Adversaries abuse this to load their malicious DLL's. The beauty of this from an adversary perspective, is that the dll is very well hidden from user interfaces of Windows. The screenshot below shows a malicious ServiceDLL named 'ieupdate.dll' that is loaded by the process svchost.exe.
The DLL with the name 'ieupdate.dll' was loaded by the Windows service 'BrowserUpdate' which is seemingly normal (see below screenshot on the left). But if you dive deeper, and look at the registry keys of the Windows Service at hand, you see an additional dll (see below screenshot on the right).
Even though it is hidden from Windows user interfaces, you can easily spot it with Velociraptor when hunting for it using the aritfact 'Windows.System.Services'. In our case you can easily spot it as it is in different folder then all other dll's, and it is the only one that is not signed.
If you have the Velociraptor client deployed across a wide range of systems, it becomes even more easy to spot as you can look for rare ServiceDll's in an IT-infrastructure. Rare ServiceDll's are more interesting because adversaries that attempt to operate in a stealthy manner, typically don't deploy malicious ServiceDll's on a lot of systems. A common hunting technique to find these rare items uses the Least Frequency of Occurrence (LFO) approach. This is a popular method suitable for finding previously unknown malware and adversary techniques. It is also suitable for finding malware that bypassed security controls such as an antivirus solution or EDR.
You can start this hunt by launching the aritfact 'Windows.System.Services' and run the following Notebook query.
SELECT count() as Count, Name,PathName,ServiceDll,CertinfoServiceDll.Trusted as Trusted
FROM source(artifact="Windows.System.Services")
WHERE ServiceDll
GROUP BY ServiceDll
ORDER BY Count
LIMIT 50
The result of this query is a list of ServiceDll's with the rare ones on top.
To toy around with ServiceDll's, you can use the instructions in the following blogposts:
- https://www.ired.team/offensive-security/persistence/persisting-in-svchost.exe-with-a-service-dll-servicemain
- https://blog.didierstevens.com/2019/10/29/quickpost-running-a-service-dll
Enable detection
Besides hunting for malicious ServiceDll's, you can also build a detection for it using Velociraptor. To receive alerts the moment the adversary plants a potentially malicious, follow the instructions below using the GUI.
- Select a random client:
- Go to ‘Client Events’ via the hamburger menu
- Choose ‘Update client monitoring table’
- Select the Label group
- Select the artifact 'Windows.Detection.Registry' (if you cannot find it, download the artifact here)
- Configure the artifact parameters:
Configure the parameters:
- Parameter 1: HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\*\\Parameters\\ServiceDll
- Parameter 2: "CertificateInfo"
- Parameter 3: "Show only executables that are not trusted by Authenticode"
The third parameter is to only detect untrusted ServiceDll's. This is a very handy parameter as our testing learns that most ServiceDll's are trusted. Meaning that you will not receive alerts when trusted ServiceDll's are placed. Of course the adversary can sign a ServiceDll, in that case you can detect based on certificate issuer (if you have that).
Continue the wizard to enable the detection.
Enable receiving alerts
Now that the detection is enable, you need to be able to receive alerts. Follow below instructions.
- In the hamburger menu of Velociraptor, go to Server Events
- Update Server Monitoring Table
- Select 'Server Internal Alerts'
Test detection
If you enabled the artifact to receive alerts, it is time to test everything. Do this by running the following command which triggers an alert.
reg add HKLM\SYSTEM\CurrentControlSet\services\BrowserUpdate\Parameters /v ServiceDll /t REG_EXPAND_SZ /d C:\ProgramData\Microsoft\Windows\Caches\ieupdate.dll
If all works well, you should see an alert similar as displayed in the screenshot below.
Testing a detection is crucial. What we do during engagements is deploying a virtual machine and install all the tools we use for detection, which typically is the EDR agent and the velociraptor client. On this virtual machine we mimic the adversary behaviour to see if it triggers an alert.
What else to detect in the registry
In this example we showed how to detect the placement of malware that uses an unsigned ServiceDll that is loaded using the registry. Other possibilities to detect registry modifications are:
- Sticky key attack
- Persistency mechanisms
- Lowering security controls like disabling the firewall or enabling WDigest
- Way, way more..
Example 4: Track mutants
Malware often uses a mutant (also called a Mutex) that serves as a flag to ensure it does not re-infect the same machine. Mike Cohen provided this clear example about mutants in his blog: "consider malware which is delivered via a malicious word document. Each time the document is opened, the malware may unnecessarily reinfect the machine, increasing its chance of detection. To avoid this, the malware attempts to open a named mutex with a predetermined hard coded name. If the CreateMutex call succeeds then the malware can continue to run. If the call fails it is most likely because another copy of the malware is already running, therefore the malware will exit."
Responders can utilise a mutex to spot the adversary. An example of a mutex is provided in the screenshot below.
Enable detection
Detecting a mutex is beneficial as you will be notified near realtime when the adversary plants malware on new systems. To configure a detection for the "evilmutant", enable the client event artifact 'Windows.Event.Mutants'. If you can't find that artifact, you can get it from here.
Enable receiving alerts
You can receive alerts by enabling the artifact: 'Server.Internal.Alerts'. If you already enabled this there is no need to follow the instructions.
- In the hamburger menu of Velociraptor, go to Server Events
- Update Server Monitoring Table
- Server Internal Alerts
Test detection
Simply run the following PowerShell command to create a Mutex.
New-Object -TypeName System.Threading.Mutex($true, "Global\evilMutant", [ref]$true)
If it works, you should see an alert similar as displayed below.
Alert Notification
After testing if the alert pops up in your Velociraptor dashboard, you might be interested in triggering a notification in Slack, Teams, and Discord - or any communication-platform that supports webhooks. Simply enable the artifact 'Server.Alerts.Notification.' If you can't find it, download it here.
Performance
Monitoring an endpoint with Velociraptor artifacts requires memory and CPU resources. The examples provided in this blog are very lightweight and don't consume a lot of resources.
To be confident that the monitoring doesn't impact systems by consuming too many resources, we always monitor the memory and CPU resource. If you want to do that for one system, select a client -> Host Information -> VQL Drilldown.
To monitor the memory and CPU footprint across all endpoints, you can use this query that you run in a Notebook.
LET CPU = SELECT * FROM foreach(row={SELECT client_id as client, os_info.fqdn as fqdn FROM clients()}, query={SELECT timestamp(epoch=Timestamp) AS Timestamp_readable, RSS / 1000000 AS MemoryUse, rate(x=CPU, y=Timestamp) * 100 As CPUPercent, fqdn FROM source(artifact='Generic.Client.Stats', client_id=client, start_time=now() - 60) WHERE CPUPercent > 0 GROUP BY ClientId} )
SELECT * FROM CPU ORDER BY MemoryUse DESC
The above query results in a table showing the memory and CPU footprint if all Velociraptor agents.
So what?
We provided two examples that you can use to detect adversary behaviour. As a responder dealing with an active adversary this is a huge benefit, as you will be notified close after the moment they conduct their malicious activity and move laterally through the environment.
While an EDR is sufficient most of the times, you may come across a scenario that is doesn’t fulfil your need to spot a adversary in near real-time. Like when the custom detection feature of the EDR is just not there or it's not mature. It could also be that the EDR deployment scope is limited or there is none at all.
To wrap it up: Don’t put all you eggs in one basket, trust but verify and have a layered approach!
