Tag Archives: Server Administration

Analyzing Logs stored in Synology Log Center for Abusive IP Addresses

A few days ago, I published a blog post on how one can quickly and easily setup a Synology NAS to act as a log receiver and store syslogs from remote CentOS servers.

When I wrote the post, I hadn’t planned to write a Part 2. But here we go! In this blog post, I will explain how I am analyzing the logs, stored in Synology’s Log Center, to identify abusive IP addresses and generate a list of the top offending /24 networks. The goal is to then automatically upload this list of networks back to every server we manage, and load those networks into each server’s firewall.

Relevant to this post, it should be noted that we are using a set of iptable wrapper scripts known as CSF (ConfigServer Security & Firewall). I think CSF is better than something like Fail2Ban, because it does much more than filter for abusive IP addresses. CSF also tracks for potential suspicious users and/or processes, and it includes an easy-to-configure perl config file to manage your firewall (opening and closing specific tcp and/or udp ports). And yes, CSF does support IPv6.

Synology Log Center stores its data in sqlite3 databases. My first step was to parse through the logs, and find the relevant messages where the firewall specifically blocked an IP address. Each server’s logs are represented in a different sqlite3 database.

So first, I run a simple find command to identify the databases. Then, I select the appropriate column from each database:

for i in $(find -name "*.DB"); do sqlite3 $i "SELECT msg FROM LOGS;"

(You’ll notice that my ‘for’ loop hasn’t been terminated in the above code. More on that in a second…)

Here is an example message (a row from within the sqlite3 database) that I’m interested in  (I am redacting the IP addresses and replacing the redaction with x.x.x.x or y.y.y.y).

Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=52:01:22:e0:39:21:84:b5:9c:f9:08:30:08:00 SRC=x.x.x.x DST=y.y.y.y LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=17737 PROTO=TCP SPT=48944 DPT=15132 WINDOW=1024 RES=0x00 SYN URGP=0

So now, I need to grab the SRC IP address. I initially was selecting (correct) results with the following command:

grep "SRC=" | awk '{print $7}' | sed 's/SRC=//'

… Which, when combined with my earlier code, looked like this:

for i in $(find -name "*.DB"); do sqlite3 $i "SELECT msg FROM LOGS;" | grep "SRC=" | awk '{print $7}' | sed 's/SRC=//'; done

But someone who does some work for me from time to time pointed out that, with the above command, I’m assuming CSF will always format the log message the same and that the IP address I’m interested in will always be in the 7th column. He suggested an alternative way to grab the IP address. After some testing, and verifying that each method returns the same results, I decided to go with his method, because I agree – it future proofs the code, and is a more accurate filter for the SRC IP Address.

grep "SRC=" | sed -e "s/^.*SRC=//" -e "s/ .*//"

So, combined with my earlier code, the full command syntax is now:

for i in $(find -name "*.DB"); do sqlite3 $i "SELECT msg FROM LOGS;"; done | grep "SRC=" | sed -e "s/^.*SRC=//" -e "s/ .*//"

At this point, we have a list of raw IP addresses. But now, I want to gather some statistics about these IP addresses, and identify the most abusive subnets. First, we should sort, and then we should only output unique lines (no point in displaying the same IP address twice). Then, let’s only output the first 3 octets, do another sort, make sure the results are unique, and count how many IP addresses are represented in each line:

awk -F . '{print $1"."$2"."$3}' | sort | uniq -c

Now we’re getting somewhere. The lined containing the full commands now looks like this:

for i in $(find -name "*.DB"); do sqlite3 $i "SELECT msg FROM LOGS;"; done | grep "SRC=" | sed -e "s/^.*SRC=//" -e "s/ .*//" | sort | uniq | awk -F . '{print $1"."$2"."$3}' | sort | uniq -c |

This will output a list of /24 subnets (IP addresses minus the last octet, preceded by how many times that subnet appeared in the list), i.e. like this: 5 x.x.x

Now let’s only grab subnets that seem problematic. You can pick any number, and I would recommend testing that number and being flexible with it, as you don’t want to block legitimate traffic just because it is coming from a subnet where other members of that IP space has had bad behavior. To start, I’m going with 20 for now.

Finally, let’s format the output so that it represents a valid subnet.

awk '$1 > 20' | awk '{print $2".0/24"}'

… and save it to a file.

Here’s the final command:

for i in $(find -name "*.DB"); do sqlite3 $i "SELECT msg FROM LOGS;"; done | grep "SRC=" | sed -e "s/^.*SRC=//" -e "s/ .*//" | sort | uniq | awk -F . '{print $1"."$2"."$3}' | sort | uniq -c | awk '$1 > 20' | awk '{print $2".0/24"}' > abusive-ip-addresses.txt

What will I do with the information?

I plan to use Synology’s crontab to run the above command once an hour. The resulting text file will get rsync’d to one of our servers

CSF will then be configured to block anything within the text document. According to CSF’s documentation, Blocklists are controlled by modifying /etc/csf/csf.blocklists

So, a valid entry could be: DEVELOPCENTS|86400|0|https://our.private.url/abusive_ip_addresses.txt

Share

Securely Collecting rsyslog Data onto Synology over TCP with SSL Encryption (from a CentOS Server)

If you are managing other servers, and are not exporting those server logs somewhere else, then you really should consider doing so. I won’t try to make the case for why in this blog post. You can do your own research (this might be a good place to start).

Screenshot of Synology Log Center

Click this screenshot of Synology’s Log Center to enlarge

Synology’s Log Center package can be used as a central log collector for other servers. It is certainly not elegant, it is simple, and it doesn’t have very many features. But it is easy and fast to implement, and is definitely better than not centralizing your logs at all.

Here are steps to configure CentOS 7 to securely send its log data to Synology’s Log Center package:

Prerequisites

  1. Ensure that the firewall where your Synology is located has NAT enabled for TCP/514 to send that traffic to your Synology (you do have a firewall, right? Never, ever connect your Synology directly to the internet).

Steps to perform on the Synology:

  1. Install the “Log Center” package using Synology’s Package Manager. The default log center in DSM is very limited. You’ll need the extra features that the Log Center “add-on” package provides.
  2. Open the Log Center package, and click on “Log Receiving”
  3. Click Create
  4. Give your Logging Rule a name. It can be anything (mine is named “ServerLogs”)
  5. The Log Format should be set to BSD
  6. Transfer Protocol should be changed to TCP
  7. The Default Port for syslog traffic is 514, but you can change the port to something else if you want, as long as you remember to set the correct port on the CentOS server (rsyslog client)
  8. Check the checkbox to Enable secure connection (SSL)
  9. Click OK
  10. Click the “Export Certificate” tab inside Log Center (see above screenshot, the tab is far right) and save the CA file somewhere. You’ll need to upload this to the CentOS server in a later step.

Steps to perform on the CentOS 7 Server (rsyslog client):

  1. Ensure port TCP/514 is open (incoming and outgoing). CentOS 7 uses firewalld, and if that is enabled, you can run:
    $  firewall-cmd --permanent --add-port=514/tcp
  2. Upload the CA file you saved in step 10 above into /etc/ssl/certs/synology-ca.crt
  3. Ensure rsyslog-gnutls is installed
    $ yum install rsyslog-gnutls
  4. Edit /etc/rsyslog.conf and add the following lines to the bottom of the file:
    $DefaultNetstreamDriver gtls # use gtls netstream driver
    $ActionSendStreamDriverMode 1 # require TLS for the connection
    $ActionSendStreamDriverAuthMode anon # server is NOT authenticated
    $DefaultNetstreamDriverCAFile /etc/ssl/certs/synology-ca.crt
    *.* @@Your-Synology-IP-Address:514
  5. Restart rsyslog:
    systemctl restart rsyslog

 

You’re done!

If your CentOS server ever gets hacked, or if you want to review logs from your CentOS server without having to SSH into it, you can now review those logs using Synology Log Center.

I hope that this was helpful. Visit https://developcents.com/knowledge-base/#Synology to view several other how-to tutorials that I’ve created for Synology users.

 

Share