Sunday, August 7, 2016

CTF for better IDS, part 2.

So, in part one, I looked at the possibility of using OSQuery to provide data in order to record and measure the activities on a target host under attack, using a vulnerable by design image from a CTF challenge.

The CTF I chose for this little experiment was Violator (,153) and the excellent walkthrough here: 

To recap, I gained root on the box, installed OSQuery and a custom configuration which contains queries against the logged_in_users, process_open_sockets,processes, and file_events (monitoring webroot) tables, then removed all my artifacts from the box to render it back to a pre-rooted state. 

The logs were, as usual, forwarded into Splunk. 

By taking various interesting fields matching the osquery _json sourcetype into a table which is then sorted through the stats command, we can get a timeline of the steps taken in the first part of the attack. 

As OSQuery is an operating system instrumentation tool, the results we'll be seeing are of course, OS-centric. We won't, for example, see the details of any exploit code sent to a web application using this tool, or any attempts to brute force an FTP account. For insight at the early active stage of the so-called "kill chain", we'll have to be ingesting the httpd and FTP logs respectively.  OSQuery is superb for detecting the often lengthy middle part of the attack; the dropping and execution of a backdoor, for example. Let's take a look at how that exact event looks in Splunk using OSQuery data:

1. Socket opens for listening port 21 (FTP)
2. ProFTP child process spawns in the webroot directory, terminates. 
3. Apache2 child process starts in webroot as the backdoor's URL is accessed.* 
4. Our backdoor is dropped, a new file is discovered in webroot.*
5. The backdoor port (1337) is connected to by the victim. 

*Waiiiit a minute... This list is in the wrong order? Yes, the backdoor would appear to activate _before_ it's dropped, how is that possible? The timestamps of each event are the times which the OSQuery query is ran and the diff in state (added/removed) is update, NOT the time of the event. As queries are splayed (the time of which you can configure) to avoid queries with the same interval executing simultaneously, which if you had a lot of queries with the same interval would probably cost you in terms of performance... 

Also, notice the erm, column. Pid 1123 is our Apache2 child process, which then goes onto connect to the attacker on 1337. This is expected; PHP is running a module within Apache. Gives us a nice basis to begin writing detection on, once you can get a baseline on how Apache's children should behave, spotting deviations should be easy. 

Next, I'll follow the steps to get root and see what data that bubbles up from the victim. 


Thursday, July 14, 2016

CTF for better IDS! Part one.

Having some experience now of using osquery ( in production coupled with Splunk to build a centralised IDS, I began to turn my thoughts to what exactly I needed to be detecting.

The problem often seen with IDS implementations (if not the products themselves), is no matter where they get located in a given stack is the fact they tend to become "Everything is OK!" alarms; generating nothing but varying levels of noise which give you a warm, fuzzy false sense of security which based only on the successful execution of your existing controls. I've had a great many conversations with people at various positions in IT who think that IDS consists of logging blocked connections at firewalls, failed auth attempts, bad HTTP requests and making them all into a pretty graph...


Don't get me wrong, those data are important for profiling your attacker and detecting the reconnaissance stage of the killchain, but in isolation they don't provide much value apart from a nice way of seeing your investment in shiny firewalls pay off; blocked connections, locked accounts and failed HTTP requests don't steal your data.

With that in mind, let's look an exercise where successfully attacking a host is the objective, a CTF. I've been playing with boot2root and other attackable image vareities from for a few years now, and the next steps seemed pretty logical...

1. Pick a nice CTF image.
2. Root it
3. Install OSQuery with a reasonably broad config.
4. Get OSQuery events into Splunk
5. Clean up any artifacts of your rooting...
6. Re-hack using the same steps in 2.
7. Analyse the lovely data.
8. Build better alerting and correlation based off the observed patterns during the attack.

Up to step 6 now. Given the variety of host OS' onto which these challenges are built, getting OSQuery to play nicely wasn't easy; you may find yourself chasing dependencies for a while. Anyway,  I have a couple of CTF images, one boot2root and another more of a vulnerability busybox, both images now running queries from the users, processes, and sockets tables in OSQuery and feeding their results into Splunk. I'm also pushing httpd requests and errors too, in order to get a side-by-side picture of the cause and effect between the attacking the web application and any generated OS artifacts.

I also have a rough timeline of various fields from queries in Splunk, set out like this:

index=* sourcetype=_json name!=info name!=rpm_packages "hostIdentifier"=* | eval hash=md5(_time._raw) | eval "action_name_host" = action+" - "+name+" - "+hostIdentifier | table "action_name_host", "calendarTime", "columns.address","columns.remote_address","columns.local_address","columns.remote_port","columns.local_port","columns.user","columns.md5","columns.target_path","columns.path","columns.mode","","columns.action" "","columns.tty", "", "event_id","unixTime",hash | stats values(*) as * by hash | sort - "unixTime" | fields - unixTime,hash

Which is giving a reasonable spread of events on the targets in a nice timeline format:

Now to get hacking...

Wednesday, March 9, 2016

Beware the hidden security control, 2

Recently, when reviewing the interim findings of a pen test of a third party's system I noticed an odd comment from the tester which suggested they were struggling with XSS injection. Something was interfering with the reflection of the injected scripts, and it wasn't encoding which was responsible.

Called up the third party to ask what validation they were using. Dev didn't even know about input validation. Huh.

So, what was going on? < and it's non identical twin, > were being stripped, along with everything inside them. Chatting further with the dev, we found the evil strings were in the database, so something was indeed stopping the attack strings getting reflected into the DOM where they would have executed.

Stripping characters isn't a great way (again, British, understatement...) to defeat XSS, and stripping just <> is likely to leave you open to a whole world of evasion techniques, but what was doing it?

Taking a look at the DOM and the requests the browser was making, it became clear jQuery was responsible for taking content from the database via some php and passing it to the client. So... does jQuery strip anything from elements?

Yep, jQuery will strip < and > from an element, unless you tell it otherwise. (Version 1.12.1):

// Handle HTML strings
if ( typeof selector === "string" ) {
if ( selector.charAt( 0 ) === "<" &&
selector.charAt( selector.length - 1 ) === ">" &&
selector.length >= 3 ) {

Risky? You betcha!* Consider:

1. Your application is only "safe" from XSS because two characters are filtered out when the strings are passed to the DOM. This won't protect you from clever encoding (JSF*ck?), reflecting the string back in CSS or js... 
2. Your one protection is based in a script which you don't own. One change, intentional or otherwise could remove this protection. 
3. The attack strings exist in your database. What if someone or something else uses your data?**

Escape, validate and encode, people. 

** Check this out, stored and latent XSS detection like a boss. 

Friday, February 26, 2016

OSQuery, Splunk and PCI

A couple of years ago over at Facebook, OSQuery was open sourced. This tool allows you to make SQL-Lite queries against tables containing information about a running Linux or OSX host. One massive advantage of this is a wide range of system attributes can be queried using a universal syntax; just imagine building (and maintaining!) even a modest sized bank of queries using native Linux tools, as well as trying to get their collective outputs into a universal format.

You can check out the tables available here: and you'll notice the file_events table, which if you are faced with PCI requirement 11.5, you'll probably find your interest starting to get piqued...

PCI 11.5: "Deploy file-integrity monitoring software to alert personnel to unauthorized modification of critical system files, configuration files, or content files, and configure the software to perform critical file comparisons at least weekly."

Previously, this involved either something like OSSec, or a Wire, which an attacker may Trip over... Both are well established tools, but both have some disadvantages on lean hosts within scalable deployments. The commercial offering by it's nature costs money, and can be tricky to configure. OSSec is an excellent project, but very much a full on server-client application with custom management and processes, which if you already have config and log management may not be desirable.

So then, back to OSQuery. By now, you've hopefully checked out the project docs and can see how it all hangs together. Let's look at a really simple config file:

 /* Configure the daemon below */
 "options": {
   "config_plugin": "filesystem",
   "logger_plugin": "filesystem",
   "events_expiry": "3600",
   "verbose": "false",
   "worker_threads": "2",
   "enable_monitor": "true"
   "schedule": {
   "ports": {
     "query": "SELECT * from listening_ports;",
     "interval": 300
   "users": {
     "query": "SELECT * from logged_in_users;",
     "interval": 300
   "file_events": {
     "query": "select * from file_events;",
     "interval": 300
"file_paths": {
   "my_app": [

[disclaimer] As you can see, OSQuery is a highly capable tool, and can return a wealth of information about a host. PCI is a very thorough compliance standard, covering all aspects of payment card security. The example in this blog is very simple, and provided for your interest only. For further information, consult your QSA. Your milage may vary.

This runs three basic queries:

users: Returns columns from the users table, notifies when a user logs in, or out of the host. (OSQuery's daemon by default, logs an "added" value when an query returns a previously unseen result, and a "removed" value when a previously seen result is no longer being returned)

ports: Returns columns from the listening_ports table, when ports are added/removed

file_events: Returns columns from the file events table when a file's attributes on the monitored paths change.

The final stanza defines the paths we want to monitor. The % and %% operators are wildcards, and recursive wildcards respectively.

Each time a query is ran the results are output to the default log location, where in our example, the Splunk Forwarder picks them up and indexes as per the inputs.conf below:

disabled = false
index = test
sourcetype = _json

json data... yeah, Splunk knows it.

If you are using Splunk Enterprise Security, you can now create a simple correlation search and generate a notable event on the fields of your choice. As we're talking FIM, let's look at all added events where a hash value for a file on the monitored path has changed (and map some fields to their Splunk CIM names so they show up in the notable event):

index=test sourcetype=_json name=file_events action="added" "columns.action"="updated" | `get_event_id` | stats values("columns.sha256") as "file_hash", values("columns.target_path") as "path", values("columns.mode") as "file_permission", values("hostIdentifier") as "orig_host", values("action") as "action" values("calendarTime") as "file_modify_time", values("columns.size") as "file_size",values("event_id") as "orig_event_id" | `map_notable_fields`

The query is returning all file change attributes, but we are only alerting on the "UPDATED" events where a hash value is added.

So great, we now have a notable event which tells us when a monitored file's contents have been changed, or a new file (ergo, a new hash) is created. Wouldn't it be nice to see what happened prior to that? Well, remember the inputs.conf? We're bringing in all OSQuery results as the same sourcetype. As the column names are consistent, we can return some interesting fields in a simple drill-down search, stats sorted by time to form a quick and dirty timeline:

index=test sourcetype=_json name=* name!=info name!=rpm_packages "hostIdentifier"=$orig_host$ | `get_event_id` | table "name", "hostIdentifier","columns.port", "columns.address","columns.user","columns.target_path","columns.mode","columns.sha256","","columns.action" "","columns.tty", "", action, "calendarTime", "event_id","unixTime" | stats values(*) as * by event_id | sort - "unixTime" | fields - event_id

Which is using (in addition to fields higher up in the json objects) fields from:

users: columns.user,,columns.tty, columns.address,
ports:columns.port, columns.address,
file_events: columns.target_path, columns.mode, columns.sha256

We're also using get_event_id to create unique ID for each matching event. Beware of creating "where" conditionals, ALL results from tables which have no "" key will be excluded.

Run this up, and we can see:

Timelines rule, dashboards drool. 

1. The attacker logs into the host, possibly using a stolen credential?
2. The file payment.html is changed
3. The file newpayment.html is created, and a php shell, because why not.
4. A listening port is opened, maybe to land shells from pivots to other hosts?

This of course, is just the start. Want to see the shell history? Add queries from shell_history table or the more detailed user_events table. Processes? Yup, there's a table for that.

Try some queries, then play the attacker and watch the results roll in. Keep tweaking the queries until you have enough data for a really nice timeline.

[edit] Ok then, who spotted the deliberate mistake? The payment.html page didn't exist before I started this demo, so you only saw the empty file get created in the previous screenshot. I logged in again (just for you, you lucky people) and tampered with the payment.html file. You can see my login, the "removed" event for the old hash, and the "added" event for the new one:

[another edit] Did you spot another strange thing? There seems to be more events than lines in the stats table. This is thanks to 'get_event_id'; each matching event gets a hash value as it's event ID, duplicate events having the same hash value are assumed to be the same event. Screenshots are from my development setup, and something with forwarding or indexing is causing duplicated events.


Monday, February 1, 2016

Beware the hidden security control...

Is your application vulnerable to SQL injection?

No? Awesome.

Can you tell me why?


In that case, you are vulnerable to SQL injection.

The hidden control which is protecting you, but you know nothing about, is only one release away from being disabled.

Sunday, January 17, 2016

Adventures in typewriters, the code.

Still pulling the typewriter apart. Have discovered a few exciting things (Exciting? Well, if you're reading this post you'll probably find them exciting too...) but in the meantime, I've made a little thing to grab RSS feeds and squirt them out to a serial terminal. Tested with an old thermal printer and working well.

Check it out:


Friday, January 15, 2016

CVE 2016 0777/8

This morning, a new and mildly concerning (I am British, therefore I genetically predisposed to the gift of the understatement: bug has been discovered in OpenSSH clients.

The feature, SSH roaming, was never fully implemented and yet seemingly the code is still present and on by default on Open SSH clients, despite not being in the server versions. It was supposed to allow for the resumption of dropped SSH sessions by allowing the client and server to re-exchange keys in the background.

The bugs themselves do not allow for an evil server or man-in-the-middle to directly request keys from a client, but they do enable a buffer overflow to be sent to the client which could allow for information resident in memory to be leaked, most worryingly this includes your SSH keys.

So, what can you do about it?

  1. Switch the vulnerable code off by adding "UseRoaming no" to your client SSH configuration.
  2. If "[connection suspended, press return to resume]" appears in stderr during a session, break the session, as this is client responding to an attempted resumption of a session by server which "supports" roaming. As roaming was never actually implemented on OpenSSH server, you can be assured this isn't a good thing...
  3. Protect your keys with a passphase, but we all do this anyway, right?
  4. Don't use keys. I shall await your fire on this one. I've always maintained that a good, well enforced, audit-able password policy > poorly implemented key management. You root into your servers using a key which is two years old and shared by all sysadmins? Then you have bigger worries than this bug.
  5. Use two-factor auth.
  6. Restrict inbound SSH connections by IP address, either at the network or application level.
My thanks to the excellent write up from Qualys here: