Monday, November 20, 2017

Docker with Splunk and Billu B0x forwarding Apache2 and mysql logs

For this post, I am going to walk through the setup of Splunk using a docker image.  With Splunk configured I am going to go back to the walk-through of Billu b0x, configure the logs to come in, and then we are going to go through the walk through and see what logs are being generated.

The goals of this post are:
1.  To show how analysts could detect the attack occurring using a SIEM
2.  To show what the attack/walkthrough would look like in a SIEM
3.  To learn what you could change in the attack/walkthough to be more stealthy in the methods utilized and how tools are used

Lab

1.  On my Kali box where docker is installed, start the service.

Command: service docker start

2.  Then search for docker images for the keyword "splunk"

Command: docker search splunk

3.  The image that I selected is called splunk/splunk.  So I am going to pull down that image.  We are trying to get version 7 of splunk.

Command: docker pull splunk/splunk

4.  After pulling the image we are going to run it.  However, prior to doing that, Splunk uses port 8000 for the web interface and also needs port 9997 for a splunk forwarder (agent) to send logs to the server. (The ports can be changed.)  With the docker image installed on Kali, the image will receive by default a 172.17.0.2 IP Address.  Billu_b0x will be in a virtual machine that will not have access to that IP Address unless we associate the ports to the IP Address of Kali.  To do that, use the -p command line switch to indicate the IP Address you want to bind to, the listening port on that IP Address forwarded to the port on the image.

Command: docker run -it -p 172.16.216.130:8000:8000 -p 172.16.216.130:9997:9997 splunk/splunk


5.  When the image loads it will have you agree to the End-User License Agreement.  After it completes loading then it will display a blinking cursor.  Use the key combination of Ctrl <p> <q> to exit out of the image while leaving it running.

6.  Because we associated the Splunk web interface with an IP Address that the host of my Kali VM can get to, let's navigate to the splunk login page on port 8000.  (You should change the password, but remember being a docker image you will loose everything when you kill the instance of the image.

URL: http://172.16.216.130:8000



7.  Then setup a receiver to listen on port 9997.  Click settings in the top right, then select forwarding and receiving.  Then click add new to receive data.  Insert port 9997 for the default port.



8.  Now, we need to load the Billu_b0x VM.  If you do not know the root password, go back and work through the VM and figure out the password.  Go ahead and login to the console and start the SSH server.

Command: /etc/init.d/ssh start

9.  Connect to the VM from the host through SSH.  This will simplify the configuration.

Command: ssh root@172.16.216.129

10.  Download the "Universal Splunk Forwarder" to the host.  This VM requires the 32 bit deb package.  After you download the file, similar to this, splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-intel.deb, copy this over to the Billu_b0x VM.

11. In a new terminal window, let's copy the file over to the VM.  To do this you can use WinSCP on windows or scp on Linux.  I am going to demonstrate using scp. 

Command:  scp splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-intel.deb root@172.16.216.129:/root

Walking through the command, secure copy the file by using the account of root to the IP Address listed and place the file in the /root directory.

12.  Then go back to the SSH session you established on step 10.  Then install the splunk forwarder.

Command:  dpkg -i splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-intel.deb

13.  Now that the forwarder is installed, we need to configure it so send logs to 172.16.216.130:9997 or the Kali box on port 9997 which then sends it to the docker image of splunk.

Command: /opt/splunkforwarder/bin/splunk add forward-server 172.16.216.130:9997


14. Verify the forward-server is configured.

Command: /opt/splunkforwarder/bin/splunk list forward-server

You shoud see it listed under inactive forwarders.  Don't worry about this yet.

15.  Now you need to add which files or directories you would like to send to Splunk.  The main reason you want to send your logs to a SIEM or central location is a miscreant will tamper with them or delete them on the box.

16.  Let's add the logs for the apache2 server for the access.log and the error.log.

Command:  /opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/access.log

Command: /opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log



17.  Now that we have configured the forwarder to send logs to the server and what logs to send to the server, let's start the splunk forwarder.

Command:  /opt/splunkforwarder/bin/splunk start splunkd


18.  After this is started you may have to wait about 2-5 minutes but then navigate in Splunk to the search box.  In the search query, type index=main and search for the last 24 hours.  You should see the logs.

19.  To generate some logs I created a simple batch script to get the home page of Billu-b0x every tenth of a second for up to 2000 times.  I ran the script from the host.



20.  If all is setup correctly, click splunk in the top-left, click app search and reporting, then in the new search insert "index=main".  You should see the logs coming in, indicating the host is "indishell".


21.  Notice that the wget tool will identify itself in what is called the user-agent.  The user-agent will describe the tool, browser, operating system and other plugins associated with the connecting device to a web server.

22.  With the tool wget you can control the user-agent that is passed.  In the terminal window I specified the user-agent to be "Hello!", then executed it.  I researched the logs and found the log entry that I caused with the tool.


23.  As a penetration tester you should understand what your tools look like in the logs.  As defenders you should know about what these tools produce and should look through logs for anomalies or unique user-agents to detect interesting activity.

24.  In the Billu b0x walk through we used nikto and dirb.  Below I am going to run both tools and we are going to look at the logs to see what is produced from the tools.

Command: nikto -h 172.16.216.129
Command: dirb http://172.16.216.129 /usr/share/wordlists/dirb/big.txt 


Before... 2,002 logs recorded


After nikto... 18,414 logs recorded (Observe the user-agent)



After dirb... 89,683 logs recorded (Observe the user-agent)



Challenge: Can you change the user-agent that is passed with nikto or dirb?

Challenge: Use Splunk to search the logs.  Try and find HTTP code 200 or web sites that exist that were accessed by Nikto or dirb.

25.  Now we are going to setup MySQL to log queries to a file and setup the splunk forwarder to collect those logs.  Login as root to Billu b0x and change to /etc/mysql and modify the my.cnf file.

Command: cd /etc/mysql
Command: vim my.cnf


26.  Scroll-down in the file to the section on "Logging and Replication".  Remove the comment or the "#" in front of "general_log_file" and "general_log".  Then save and exit from vim "<esc> :wq".


27.  Now add the file "/var/log/mysql/mysql.log" to the splunk files to be monitored, also add the error.log.

Command: /opt/splunkforwarder/bin/splunk add monitor /var/log/mysql/mysql.log

Command: /opt/splunkforwarder/bin/splunk add monitor /var/log/mysql/error.log



28.  After configuring the logging of mysql, attempt to login then use splunk to view the query to the database of the username and password.  Notice the search is specific to the mysqld log.

Search: index=main sourcetype=mysqld


29.  The query is logged and now the username and password for the user is in the logs.  Working with a SIEM you need to understand what is in the logs.  Another example is when a query contains a SSN or a credit card number.  Be aware of when this information could be gathered by a SIEM.

Developers should return as results sensitive information but be careful querying for it directly.  For example, you can query for the password of the admin user.  Then with the user input for the password and the returned password compare them and verify they match.  That is after you check to verify if the user exists in the database.

30.  As an ethical hacker or a penetration tester you may want to test your attack in a lab prior to performing it.  I also like to test for vulnerabilities with a proxy and logging enabled.  This helps me to analyze my attacks and how I have to change them to be more effective.

Challenge:  Continue working through the Billu b0x walk through.  Use burp suite and see if you can see in the logs that you are using it as a proxy.




No comments:

Post a Comment

Test Authentication from Linux Console using python3 pexpect

Working with the IT420 lab, you will discover that we need to discover a vulnerable user account.  The following python3 script uses the pex...