How to perform threat analysis on Apache log
Apache stores details of HTTP requests in its Access Log as defined by the CustomLog directive. The log can be used to check for malicious requests such as hacking or defacement attempts.
Scalp! is a security log analyzer for Apache. It automates reading the Apache log files and perform threat analysis.
Scalp! uses rulesets provided by PHP-IDS project and is available for download from GitHub.
Steps to perform threat analysis on Apache log using Scalp:
Download Scalp from GitHub.
$ git clone https://github.com/neuroo/apache-scalp scalp Cloning into 'scalp'... remote: Enumerating objects: 11, done. remote: Total 11 (delta 0), reused 0 (delta 0), pack-reused 11 Unpacking objects: 100% (11/11), 11.85 KiB | 346.00 KiB/s, done.
Related: How to install Git on Ubuntu
Download signature file from PHPIDS project on GitHub.
$ git clone https://github.com/PHPIDS/PHPIDS phpids Cloning into 'phpids'... remote: Enumerating objects: 11281, done. remote: Total 11281 (delta 0), reused 0 (delta 0), pack-reused 11281 Receiving objects: 100% (11281/11281), 4.16 MiB | 2.81 MiB/s, done. Resolving deltas: 100% (5636/5636), done.
Split Apache log file if longer than 10000 lines.
$ split -l 10000 /var/log/apache2/access_log
Analyze Apache log file using Scalp and PHPIDS signature.
$ sudo python scalp/scalp.py --log /var/log/apache2/access_log --filters phpids/lib/IDS/default_filter.xml Password: Loading XML file 'phpids/lib/IDS/default_filter.xml'... Processing the file '/var/log/apache2/access_log'... Scalp results: Processed 1318 lines over 1318 Found 6 attack patterns in 0.425544 s Generating output in /home/user/access_log_scalp_*
More options for Scalp:
Scalp the apache log! by Romain Gaucher - http://rgaucher.info usage: ./scalp.py [--log|-l log_file] [--filters|-f filter_file] [--period time-frame] [OPTIONS] [--attack a1,a2,..,an] [--sample|-s 4.2] --log |-l: the apache log file './access_log' by default --filters |-f: the filter file './default_filter.xml' by default --exhaustive|-e: will report all type of attacks detected and not stop at the first found --tough |-u: try to decode the potential attack vectors (may increase the examination time) --period |-p: the period must be specified in the same format as in the Apache logs using * as wild-card ex: 04/Apr/2008:15:45;*/Mai/2008 if not specified at the end, the max or min are taken --html |-h: generate an HTML output --xml |-x: generate an XML output --text |-t: generate a simple text output (default) --except |-c: generate a file that contains the non examined logs due to the main regular expression; ill-formed Apache log etc. --attack |-a: specify the list of attacks to look for list: xss, sqli, csrf, dos, dt, spam, id, ref, lfi the list of attacks should not contains spaces and comma separated ex: xss,sqli,lfi,ref --ignore-ip|-i: specify the list of IP Addresses to look exclude the list of IP Addresses should be comma separated and not contain spaces This option can be used in conjunction with --ignore-ip --ignore-subnet|-n: specify the list of Subnets to look exclude the list of Subnets should be comma separated and not contain spaces This option can be used in conjunction with --ignore-subnet --output |-o: specifying the output directory; by default, scalp will try to write in the same directory as the log file --sample |-s: use a random sample of the lines, the number (float in [0,100]) is the percentage, ex: --sample 0.1 for 1/1000
Review generated output.
$ cat /home/user/access_log_scalp_* # # File created by Scalp! by Romain Gaucher - http://code.google.com/p/apache-scalp # Apache log attack analysis tool based on PHP-IDS filters # Scalped file: access_log Creation date: Sat-08-Feb-2020 Attack type: files Attack Cross-Site Scripting (xss) Attack Cross-Site Request Forgery (csrf) Attack Spam (spam) Attack Local File Inclusion (lfi) ### Impact 5 127.0.0.1 - - [08/Feb/2020:07:31:12 +0800] "GET /windows/start HTTP/1.1" 200 4541 Reason: "Detects specific directory and path traversal" 127.0.0.1 - - [08/Feb/2020:07:31:16 +0800] "GET /windows/burn-iso-image HTTP/1.1" 200 4941 Reason: "Detects specific directory and path traversal" 127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /_media/windows/burn-iso-image/windows7-burn-iso-explorer-write-to-disc.png?w=400&tok=e21342 HTTP/1.1" 200 16104 Reason: "Detects specific directory and path traversal" 127.0.0.1 - - [08/Feb/2020:07:31:17 +0800] "GET /windows/disable-system-restore HTTP/1.1" 200 4679 Reason: "Detects specific directory and path traversal" 127.0.0.1 - - [08/Feb/2020:07:31:18 +0800] "GET /windows/firewall-command HTTP/1.1" 200 5511 Reason: "Detects specific directory and path traversal" 127.0.0.1 - - [08/Feb/2020:07:31:19 +0800] "GET /windows/restore-mbr HTTP/1.1" 200 5323 Reason: "Detects specific directory and path traversal" Attack SQL Injection (sqli) Attack type: format string Attack Remote File Execution (rfe) Attack Denial Of Service (dos) Attack Directory Traversal (dt) Attack Information Disclosure (id)