Splunk BOTSv1: A Defender's Walkthrough

Splunk's Boss of the SOC v1 (BOTSv1) is a blue team CTF built around a fictional intrusion at Wayne Enterprises. Splunk has released this dataset publicly allowing anyone with a Splunk instance to work through it locally. This is why it's become one of the recommended free CTFs for those moving into the SOC or IR work.
Scenario 1 - Web site defacement
Today is Alice's first day at the Wayne Enterprises' Security Operations Center. Lucius sits Alice down and gives her first assignment: A memo from Gotham City Police Department (GCPD). Apparently GCPD has found evidence online http://pastebin.com/Gw6dWjS9 that the website www.imreallynotbatman.com hosted on Wayne Enterprises' IP address space has been compromised. The group has multiple objectives... but a key aspect of their modus operandi is to deface websites in order to embarrass their victim. Lucius has asked Alice to determine if www.imreallynotbatman.com. (the personal blog of Wayne Corporations CEO) was really compromised.
Pre Work:
I started by looking at all the available indices:
| eventcount summarize=false index=* | dedup index | fields index

Then I viewed the source types under the first index:
| tstats count where index=botsv1 by sourcetype | sort -count

Question 101:
What is the likely IPv4 address of someone from the Po1s0n1vy group scanning imreallynotbatman[.]com for web application vulnerabilities?
Picking the relevant index and source type, I searched for the top source IPs and user agents that made HTTP requests:
index=botsv1 sourcetype=stream:http site=imreallynotbatman.com
| top src_ip, http_user_agent

There where only 2 source IPs displayed. 40.80.148.42 was the clear web vulnerability scanner as it was modifying the user agent in attempts to exploit web vulnerabilities. Real users don't rotate user agents though automated vulnerability scanning tooling often does. That a single source (src_ip) generated multiple distinct UAs (http_user_agent) in a single burst, which is not legitimate behaviour for a web browser.
Answer: 40.80.148.42
Question 102:
What company created the web vulnerability scanner used by Po1s0n1vy? Type the company name.
Since the user agent in my previous search didn't provide an answer to this question, I looked at all the HTTP headers for the web app scanner against the site:
index=botsv1 sourcetype=stream:http site=imreallynotbatman.com src_ip=40.80.148.42
| top src_headers
Answer: Acunetix
Question 103:
What content management system is imreallynotbatman.com likely using?
The answer to this question was in my screenshot above, though I wanted to make sure the request was successful and not just part of a wordlist used by the scanner.
index=botsv1 sourcetype=stream:http site=imreallynotbatman.com status=200
| top src_headers
Answer: Joomla
Question 104:
What is the name of the file that defaced the imreallynotbatman.com website? Please submit only the name of the file with extension?
I was not having much luck with finding the file with stream:http, so I switched to sourcetype=* to cover any other sources. Some of these sources required an IP rather than the site name that I was using before. So I wanted to find the source IP of the web server:
index=botsv1 sourcetype=stream:http site=imreallynotbatman.com status=200
| top dest_ip

Knowing the IP address of the web server (192.168.250.70), I took a look at the different content type requests, where the status code was 200:
index=botsv1 sourcetype=* status=200 (src_ip=192.168.250.70 OR dest_ip=192.168.250.70)
| stats count by http_content_type

There where only 25 different content types, so I picked some of the ones that were not used too often, and could also be used for defacement. There where only 54 results, so I scanned through the list until I find a suspicious file (poisonivy-is-coming-for-you-batman.jpeg), which was the answer on only line 2:
index=botsv1 sourcetype=* status=200 (src_ip=192.168.250.70 OR dest_ip=192.168.250.70) http_content_type IN ("image/jpeg", "image/png", "image/gif")
| dedup url
| table url

Answer: poisonivy-is-coming-for-you-batman.jpeg
Question 105:
This attack used dynamic DNS to resolve to the malicious IP. What fully qualified domain name (FQDN) is associated with this attack?
I already had the URL from question 104, so I just need to find the FQDN which hosted it:
index=botsv1 status=200 (src_ip=192.168.250.70 OR dest_ip=192.168.250.70) http_content_type url="/poisonivy-is-coming-for-you-batman.jpeg"
| stats count by http.hostname

Answer: prankglassinebracket.jumpingcrab.com
Question 106:
What IPv4 address has Po1s0n1vy tied to domains that are pre-staged to attack Wayne Enterprises?
A quick search for the domain (found in question 105) on VirusTotal shows that they have captured the domain as resolving to 23.22.63[.]114 on 2018-05-02:

Answer: 23.22.63.114
Question 107:
This question did not exist.
Question 108:
What IPv4 address is likely attempting a brute force password attack against imreallynotbatman.com?
I already know the destination IP (Question 104) and that the target is a webserver. Looking at the most hit urls and their source IP, the answer is found by looking at the most likely path to an authentication page:
index=botsv1 sourcetype=stream:http dest_ip=192.168.250.70 status!=404
| stats count by url, src_ip
| sort -count

Answer: 23.22.63.114
Question 109:
What is the name of the executable uploaded by Po1s0n1vy?
Since told it was uploaded, the destination IP will be the webserver, method POST (first assumption) and then can take a look at the filenames:
index=botsv1 sourcetype=stream:http dest_ip=192.168.250.70 http_method=POST
| stats count by part_filename{}

Answer: 3791.exe
Question 110:
What is the MD5 hash of the executable uploaded?
Sysmon Event ID 11 wasn't showing up, so I searched for it's execution instead:
index=botsv1 sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1 Image=*3791.exe
| stats count by MD5

Answer: AAE3F5A29935E6ABCC2C2754D12A9AF0
Question 111:
GCPD reported that common TTPs (Tactics, Techniques, Procedures) for the Po1s0n1vy APT group, if initial compromise fails, is to send a spear phishing email with custom malware attached to their intended target. This malware is usually connected to Po1s0n1vys initial attack infrastructure. Using research techniques, provide the SHA256 hash of this malware.
Knowing the attacker's IPv4 from question 106, we can search VirusTotal and click "Relations" to see related files. Looking at the first entry where the IP is found in the file contents (Files Referring), we can then collect the SHA256 hash under "Details":

Answer: 9709473ab351387aab9e816eff3910b9f28a7a70202e250ed46dba8f820f34a8
Question 112:
What special hex code is associated with the customized malware discussed in question 111?
From the VirusTotal search of the SHA256 hash in question 112, clicking on the "Community" tab and scrolling down to the first comment revels the answer:

Answer: 53 74 65 76 65 20 42 72 61 6e 74 27 73 20 42 65 61 72 64 20 69 73 20 61 20 70 6f 77 65 72 66 75 6c 20 74 68 69 6e 67 2e 20 46 69 6e 64 20 74 68 69 73 20 6d 65 73 73 61 67 65 20 61 6e 64 20 61 73 6b 20 68 69 6d 20 74 6f 20 62 75 79 20 79 6f 75 20 61 20 62 65 65 72 21 21 21
Question 113:
This question did not exist.
Question 114:
What was the first brute force password used?
In question 108 the source IP of the brute force was found, so I modified the query from that question:
index=botsv1 sourcetype=stream:http src_ip=23.22.63.114 dest_ip=192.168.250.70 status!=404 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| table form_data, _time
| sort _time

It can also be done with regex for a cleaner result (captures everything after passwd= up to the next & or end of the string):
index=botsv1 sourcetype=stream:http src_ip=23.22.63.114 dest_ip=192.168.250.70 status!=404 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| table password, _time
| sort _time

Answer: 12345678
Question 115:
One of the passwords in the brute force attack is James Brodsky's favorite Coldplay song. We are looking for a six character word on this one. Which is it?
I reused the first query, but added | where len(password)=6. Scrolling down to find the answer.
index=botsv1 sourcetype=stream:http src_ip=23.22.63.114 dest_ip=192.168.250.70 status!=404 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| where len(password)=6
| table password, _time
| sort _time

(Note: in the screenshot I replaced the last line with | sort -password, just to show the result more clearly).
Answer: yellow
Question 116:
What was the correct password for admin access to the content management system running "imreallynotbatman.com"?
Using the query from question 114, I modified to exclude login attempts from the malicious IP. This showed one, genuine login:
index=botsv1 sourcetype=stream:http src_ip!=23.22.63.114 dest_ip=192.168.250.70 status!=404 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| where len(password)=6
| table password, _time

Answer: batman
Question 117:
What was the average password length used in the password brute forcing attempt?
Using the query from question 114 again, I used | eval password_len=len(password) to create a new field for the length of each password, then | stats avg(password_len) as avg_length to find the average.
index=botsv1 sourcetype=stream:http src_ip=23.22.63.114 dest_ip=192.168.250.70 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| eval password_len=len(password)
| stats avg(password_len) as avg_length

Answer: 6
Question 118:
How many seconds elapsed between the time the brute force password scan identified the correct password and the compromised login?
From previous questions the correct password (batman) is known, so I made a query that only shows events against the web server where this password was entered:
index=botsv1 sourcetype=stream:http dest_ip=192.168.250.70 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| where password="batman"
| table password, _time

There where only 2 events, so I used | stats range(_time) as elapsed to get their difference:
index=botsv1 sourcetype=stream:http dest_ip=192.168.250.70 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| where password="batman"
| stats range(_time) as elapsed

Answer: 92.17
Question 119:
How many unique passwords were attempted in the brute force attempt?
I used my query from question 117, though I replaced the last 2 lines with | stats dc(password) as unique_passwords:
index=botsv1 sourcetype=stream:http src_ip=23.22.63.114 dest_ip=192.168.250.70 http_method=POST url=http://imreallynotbatman.com/joomla/administrator/index.php
| rex field=form_data "passwd=(?<password>[^&]+)"
| stats dc(password) as unique_passwords

Answer: 412