The CTF I chose for this little experiment was Violator (https://www.vulnhub.com/entry/violator-1,153) and the excellent walkthrough here: https://medium.com/@euri10/violator-walkthrough-7f153340659#.n4ww016z7
To recap, I gained root on the box, installed OSQuery and a custom configuration which contains queries against the logged_in_users, process_open_sockets,processes, and file_events (monitoring webroot) tables, then removed all my artifacts from the box to render it back to a pre-rooted state.
The logs were, as usual, forwarded into Splunk.
By taking various interesting fields matching the osquery _json sourcetype into a table which is then sorted through the stats command, we can get a timeline of the steps taken in the first part of the attack.
As OSQuery is an operating system instrumentation tool, the results we'll be seeing are of course, OS-centric. We won't, for example, see the details of any exploit code sent to a web application using this tool, or any attempts to brute force an FTP account. For insight at the early active stage of the so-called "kill chain", we'll have to be ingesting the httpd and FTP logs respectively. OSQuery is superb for detecting the often lengthy middle part of the attack; the dropping and execution of a backdoor, for example. Let's take a look at how that exact event looks in Splunk using OSQuery data:
1. Socket opens for listening port 21 (FTP)
2. ProFTP child process spawns in the webroot directory, terminates.
3. Apache2 child process starts in webroot as the backdoor's URL is accessed.*
4. Our backdoor is dropped, a new file is discovered in webroot.*
5. The backdoor port (1337) is connected to by the victim.
*Waiiiit a minute... This list is in the wrong order? Yes, the backdoor would appear to activate _before_ it's dropped, how is that possible? The timestamps of each event are the times which the OSQuery query is ran and the diff in state (added/removed) is update, NOT the time of the event. As queries are splayed (the time of which you can configure) to avoid queries with the same interval executing simultaneously, which if you had a lot of queries with the same interval would probably cost you in terms of performance...
Also, notice the columns.pid... erm, column. Pid 1123 is our Apache2 child process, which then goes onto connect to the attacker on 1337. This is expected; PHP is running a module within Apache. Gives us a nice basis to begin writing detection on, once you can get a baseline on how Apache's children should behave, spotting deviations should be easy.
Next, I'll follow the steps to get root and see what data that bubbles up from the victim.