Never Ending Security

It starts all here

Tag Archives: cron

How to install CSF firewall on your VPS


How to install CSF firewall on your VPS

Is by far the easiest firewall script to date I have worked with which even comes with a Webmin module a web interface front-end to manage the firewall configuration.  If you’re serious about your server security then add another layer of defense by installing CSF the user-friendly server firewall.  CSF has been tested to work with different Virtual Private Servers.  But I would suggest to use a Xen or KVM VPS instead of an Open VZ so you will have all the IPtables modules needed for CSF to work correctly. On a personal note after using CSF on my servers I have noticed a significant reduction of brute force attempts directed against FTP or SASL.

My CSF installation was done on an Ubuntu 12.04 LTS and Debian 7 Wheezy.

Install CSF Firewall

cd /usr/local/src

wget http://download.configserver.com/csf.tgz

tar -xzf csf.tgz

cd csf

sh install.sh

Iptables Module Test

Do a test to make sure you have the needed iptables modules installed for it to work.  This is one of the many reasons to use a Xen or KVM VPS so you don’t run into any missing iptables modules when using OpenVZ type VPS.

perl /usr/local/csf/bin/csftest.pl

You should get something like this.

iptables module test

Remove Advance Policy Firewall & Brute Force Detection

Run the this script if you already APF & BFD firewall installed like I do.

sh /usr/local/csf/bin/remove_apf_bfd.sh

Installing CSF Webmin Module

Install the Webmin module to manage the firewall through a web interface.   Since I already have Webmin installed on my server all I had to do was to go to Webmin > Webmin Configuration > Webmin Modules > Install from local file > Browse to /usr/local/csf/csfwebmin.tgz.

Click install.

install csf webmin module

CSF Firewall Webmin Menu

After it has been installed there will now be a menu call ConfigServer Security & Firewallunder the System menu.

csf webmin module

CSF Basic Security Test

CSF can perform a basic security check on your server with suggestions on how to fix any issues found.

Click  Check Server Security.

csf server security check

These were the results I got.  So I have some work to do.

csf security check results

Green indicator Firewall is running

Very important to keep on doing the test and fixing any issues found by the check until you get the OK.

One of the suggested fix is to enable the CSF upgrade connection to use SSL.

You can install the LWP perl module using Webmin’s perl module.

perl module

Then edit the csf.conf file.

vi /etc/csf.conf

ssl upgrade

green indicator

Firewall Configuration

Clicking on Firewall Configuration to make your edits.

firewall configuration

To quickly jump to sections of the firewall settings you can choose from the drop down menu.

firewall configuration menu

Before I start changing settings on the CSF firewall Webmin module I added my current IP address so I don’t lock myself out.  By clicking Quick Allow.

allow ip through

Or you could set the CSF firewall to test mode by setting the values like below.

csf testmode

Clicking on Firewall Configuration next to start managing the firewall configuration script.

firewall configuration

Using the recommended setting for RESTRICT_SYSLOG.

restrict_syslog recommended setting

Create a group for Syslog.

syslog group

Restricted UI set to the recommended setting.

restrict ui setting

Set the auto update to on so the cron script can check daily for newer versions of CSF.

auto updates on

If an update becomes available this will appear as below.  You can view details of the upgrade by clicking View ChangeLog.

Clicking Upgrade csf will perform upgrade.

csf update available

Allow which ports to receive and send connections otherwise those services will not be able to communicate.

allowed ports

I ran into an issue where my outbound SSH connections were being blocked by the firewall.  I forgot to add the new port number on the outbound TCP ports.  I am using a non-default SSH port.

outbound ssh port

Enable or disable ping replies.

allow ping replies

How many IPs to keep in the deny lists.  Change this setting depending on your server resources.

deny ip limit

The following settings are enable so LFD can check for login failures to ban.  The setting will also check to make sure CSF has not been stopped so it can be restarted.

lfd set to check for login failures

Set the default drop target for connections matching a rule.  Set it to DROP.  This will cause anyone trying to port scan your server to hang.

drop target

I like to enable the drop connections should I need to see which IPs got blocked.

drop logging

How to block countries from accessing your server

CSF, makes this very easy to do compared to other scripts I have used in the past.  You just need to add the country code separated by comma.

block countries

Blocking a specific IP address or a network

I have used this feature a lot whenever I get phishing emails or lots of spam coming from an IP address or IP addresses from the same network block I will add the IP address or network address in here with a comment.  Any IP address added here will be permanently blocked.  I have used this online whois to determine who owns the IP address and which ISP provides hosting.

deny ip

Login failure blocking when enabled will trigger LFD Login Failure Daemon to block any failed login attempts when it reaches the number of failed attempts set.

lf trigger

When you have LFD enabled you will sometimes need to add IP addresses you own in here so you don’t get locked out if you mistype a password.  Click edit then add in your IP address or network.  Then restart LFD.

lfd ignore list

Block lists

let us enable these block lists from Spamhaus, Dshield, Honeypot, Tor nodes, etc.

Clicking lfd Blocklist.  Uncommenting the blocklist you want to use.  Using this has reduced intrusion attempts against my server from compromised hosts.  What a great option to have on a firewall.  CSF makes it incredibly easy to enable.  Before you enable this blocklist or country blocking you need to consider if your server has enough to resources to handle the load.  My VPS typically have more than 3 GB of ram some even more.  I usually do not have less than 4 CPUs for my VPS.  So I am able to use all the blocklist rules with no noticeable performance hit.

Don’t forget to click change to apply the new settings.

vi /etc/csf/csf.blocklist

blocklist

If you’re curios to see what rules your CSF firewall has loaded click on view iptables rules. Depending on what you have enable be prepared to scroll for a long time.  This is just a sample of mine which shows connections from China are blocked.  I had to snipped it for the output was very long.

china blocked

If you want to see connections being dropped in real time you could do so by clicking watch system logs.  Then choosing from drop down kern.log.

watch system logs

dropped connections

If you wanted to permanently block an IP or IP range click Firewall Deny IPs.  Enter each IP or CIDR addressing one per each line.

Click change to apply configuration changes.

block ip permanently

block ip list

Login Failure Daemon (LFD)

LFD Daemon is a process which continuously scans the logs for failed login attempts the script will immediately block the offending host when a set number of failed attempts is reached per IP.  It can also detect distributed attacks.  Compared to Fail2ban which I used before the resource consumption created by LFD is much lower.

Very Important! If you want your home IP address not being blocked by LFD due to failed login attempts (You making SSH, IMAP, etc connections while putting in the wrong password) you will have to add them into csf.ignore.  Add the IPs you don’t want blocked one per each line. I learned this the hard way!

From the web interface choose from the drop down which LFD file to edit to add IP addresses you never want locked out.

lfd ignore web interface

vi /etc/csf/csf.ignore

If you end up blocking yourself you will have to login at the console to stop LFD  through init.  
/etc/init.d/lfd stop

Check if Syslog is running

syslog is running check

ConfigSecurity Firewall & LFD Brute Force Detection Blocking Specific Settings for Ubuntu & Debian

For LSF to block failed attempts against ProFTPD, SASL on Ubuntu & Debian the following log paths on CSF.conf have to be changed.

vi /etc/csf/csf.conf
HTACCESS_LOG = "/var/log/apache2/error.log"

MODSEC_LOG = "/var/log/apache2/error.log"

SSHD_LOG = "/var/log/auth.log"

SU_LOG = "/var/log/messages"

FTPD_LOG = "/var/log/proftpd/proftpd.log"

SMTPAUTH_LOG = "/var/log/mail.log"

POP3D_LOG = "/var/log/mail.log"

IMAPD_LOG = "/var/log/mail.log"

IPTABLES_LOG = "/var/log/syslog"

SUHOSIN_LOG = "/var/log/syslog"

BIND_LOG = "/var/log/syslog"

SYSLOG_LOG = "/var/log/syslog"

WEBMIN_LOG = "/var/log/auth.log"

Then on the CUSTOM LOG.

CUSTOM1_LOG = "/var/log/mail.log"

Then you will need to add the regex to catch the failed attempts against SASL.

vi /usr/local/csf/bin/regex.custom.pm

Add the following code in the middle of  “Do not edit before this point &  Do not edit beyond this point”  The numbers after “mysaslmatch” are used for the following: “1” is the number of failed attempts which triggers a block IPTable rule.  The next number indicates the port to monitor “25,58”. You could separate the multiple ports using a comma.  The next number “6000” is the time in seconds the host will be kept in the deny lists.

if (($lgfile eq $config{CUSTOM1_LOG}) and ($line =~ /^\S+\s+\d+\s+\S+ \S+ postfix\/smtpd\[\d+\]: warning:.*\[(\d+\.\d+\.\d+\.\d+)\]: SASL [A-Z]*? authentication failed/)) {

return ("Failed SASL login from",$1,"mysaslmatch","1","25","6000");

}

Restart the CSF firewall to apply settings.

csf -r

As soon as I have the SASL custom regex applied an offending host was caught abusing SASL.  The log which was emailed to me.  It has been very effective blocking brute force detection targeted against my FTP and SASL services that I decided to do away with Fail2ban.

sasl blocked host

Checking the Temporary IP Entries came up with the following results.

temporary block ips

From this window you can easily unblock or permanently ban an IP by clicking the icons.  Any hosts added to this list will be banned accessing any ports until the set banned time limit is reached.

blocked ip gui

If you want to allow only specific IPs from connecting to your SSH port you could do so by removing SSH port 22 in the IPv4 port settings.

Allow specific ips from connecting

Then adding the IP addresses you want to be able to connect to your SSH port in.

vi /etc/csf/csf.allow

###############################################################################
# Copyright 2006-2014, Way to the Web Limited
# URL: http://www.configserver.com
# Email: sales@waytotheweb.com
###############################################################################
# The following IP addresses will be allowed through iptables.
# One IP address per line.
# CIDR addressing allowed with a quaded IP (e.g. 192.168.254.0/24).
# Only list IP addresses, not domain names (they will be ignored)
#
# Advanced port+ip filtering allowed with the following format
# tcp/udp|in/out|s/d=port|s/d=ip
# See readme.txt for more information
#
# Note: IP addressess listed in this file will NOT be ignored by lfd, so they
# can still be blocked. If you do not want lfd to block an IP address you must
# add it to csf.ignore
123.123.123.124 # csf SSH installation/upgrade IP address - Wed Feb 26 13:16:28 2014
123.123.123.125 # Home IP address

 DDoS Protection

From Firewall Configuration click on drop down.

connection tracking

For some level of DDoS protection I have enabled connection tracking by doing so I am able to limit the number of connections to network services I want to limit connections it receives.  The values below are what works for my setup you will have to play around as to what settings works best for you.

CT_LIMIT = 100

CT_BLOCK_TIME = 1800 (30 mins blocked time)

CT_PORTS = 80,993

Leaving the rest of the settings to use the default values.

ct_limit

Leaving the rest of the settings up to you to change.  The CSF firewall settings are very well documented.  When you’re done making your edit apply new settings by clicking change.

apply setting changes

Command line CSF

Enable CSF

csf -e

Disable CSF

csf -x

Re-enable CSF and LFD

csf -e

Restart CSF

csf -r

Happy Fire-walling using CSF The User-friendly host-based firewall.

References:

http://forum.configserver.com/viewtopic.php?f=6&t=6968

https://www.virtualmin.com/node/13841

Monitor disk io on linux server with iotop and cron


Iotop – Disk Input Output usage

Recently my server was giving notifications of disk io activity rising above a certain threshold at regular intervals. My first guess was that some cronjob task was causing that. So I tried to check various cron tasks to find out which task or process was causing the io burst. On servers specially its always a good practice to monitor resource usage to make sure that websites work fast and well.

However searching manually is not quite easy and this is where utilities like iotop come in. iotop shows what or how much disk io are all current processes doing. Its quite easy to use. Just run it from a terminal and you should see some output like this

Total DISK READ:       0.00 B/s | Total DISK WRITE:     106.14 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                
  335 be/3 root        0.00 B/s   98.56 K/s  0.00 %  2.03 % [jbd2/sda6-8]
 4096 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
 4100 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
 4102 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
    7 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u:0H]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]

As we can see, each row shows a certain process and the amount of data it is reading or writing to. This information is actually instantaneous, so iotop keeps updating the values at certain interval like 1 second. Running iotop like this just tells the current io usage. What if we want to keep running iotop and record all io activity and analyse it later. This is where cron comes in.

Automatic logging via cron

Cron will run iotop in the background and record io usage details to a file that can be analysed later.

Here is the basic iotop command that we want to run in the background via cron.

$ iotop -botqqqk --iter=60
17:38:13   335 be/3 root        0.00 K/s    7.64 K/s  0.00 %  2.30 % [jbd2/sda6-8]
17:38:13  3296 be/4 enlighte    0.00 K/s   15.28 K/s  0.00 %  0.00 % chrome
17:38:14   335 be/3 root        0.00 K/s    7.62 K/s  0.00 %  3.35 % [jbd2/sda6-8]
17:38:14  3293 be/4 enlighte    0.00 K/s    7.62 K/s  0.00 %  0.02 % chrome
17:38:15  3319 be/4 enlighte    0.00 K/s   19.09 K/s  0.00 %  0.00 % chrome

Note that iotop must be run with root privileges. So use sudo on ubuntu for example.

Now the most important option used in the above command is the “b” option which is for batch/non-interactive mode. In batch mode iotop will keep outputting line after line instead of showing a long list that updates automatically. This is necessary when we want to log io activity over a certain period of time.

The other option called “o” will show only those processes which actually did some io activity. Otherwise iotop would show all processes. The t option adds a timestamp which adds further information if you want to track a specific high io process. The k option shows all figures in kilobytes.

To log the output we simply need to redirect it to a file. The best place is /var/log and the file could be named iotop. So here is the next command

$ iotop -botqqqk --iter=60 >> /var/log/iotop

This will run iotop for 60 seconds where each iteration takes 1 second, and the output would be logged to the file /var/log/iotop. Now the command has to be run every minute which cron would do easily.

Setup cron as root

Get into your linux server as root and create a file

/etc/cron.d/iotop

Add to this file our earlier iotop command that would log the iotop output.

* * * * * root /usr/sbin/iotop -botqqqk --iter=60 >> /var/log/iotop

Simple! Now cron would run the command every minute and every time the command would run for a minute taking samples as 1 second interval and all activity gets logged to /var/log/iotop.

Make sure to mention the full path to iotop otherwise cron might not be able to run it at all. To find the location of iotop on your linux server, use the which command

$ which iotop
/usr/sbin/iotop

Monitor only high io processes

I had to setup this io monitoring because my linux server was giving high io alerts at around 3 AM in the morning when I could not get up to check manually. And since the server alert was showing high disk io activity I decided to monitor only those processes that did high disk io.

Here is a simple command that greps the iotop output to grab process that have over 10 k/s of disk io anywhere.

$ sudo iotop -botqqqk --iter=60 | grep -P "\d\d\.\d\d K/s"
17:49:02   335 be/3 root        0.00 K/s   41.90 K/s  0.00 %  8.43 % [jbd2/sda6-8]
17:49:02  3307 be/4 enlighte    0.00 K/s  152.36 K/s  0.00 %  0.87 % chrome
17:49:10  3310 be/4 enlighte    0.00 K/s   22.80 K/s  0.00 %  0.10 % chrome
17:49:15  3319 be/4 enlighte    0.00 K/s   26.54 K/s  0.00 %  2.50 % chrome
17:49:16  3310 be/4 enlighte    0.00 K/s   19.02 K/s  0.00 %  0.00 % chrome

So it would not show those process that had less than 10 K/s of disk io. It does so by filtering the output using grep and regular expressions. This is very useful when we want to find any process that is causing very high io activity.
High activity would mean around 1 M/s of disk activity for some time. So grepping for values with 4 digits will find those processes right away.

To add the same command to cron replace the earlier command like this

* * * * * root /usr/sbin/iotop -botqqqk --iter=60 | grep -P "\d\d\.\d\d K/s"  >> /var/log/iotop

I used it, and it works quite well. Here is how the output looks when filtered using grep to show only high io process entries.

13:19:01  1325 be/4 root        0.00 K/s 1897.74 K/s  0.00 %  5.65 % [kjournald]
13:24:22  2836 be/4 mysql       0.00 K/s 1071.07 K/s  0.00 %  0.18 % mysqld
13:32:01  1325 be/4 root        0.00 K/s 1469.17 K/s  0.00 %  7.13 % [kjournald]
13:46:18 10978 be/4 binary   1634.31 K/s    0.00 K/s  0.00 % 23.87 % php-fpm: pool binary
13:47:01  2955 be/4 mysql       0.00 K/s 8738.80 K/s  0.00 %  0.00 % mysqld
14:17:01  1325 be/4 root        0.00 K/s 1354.01 K/s  0.00 %  6.84 % [kjournald]
14:23:02  1325 be/4 root        0.00 K/s 1146.18 K/s  0.00 %  4.69 % [kjournald]
14:25:01  1325 be/4 root        0.00 K/s 1494.21 K/s  0.00 % 11.05 % [kjournald]
14:34:01  9938 be/4 mysql       0.00 K/s 2878.55 K/s  0.00 %  0.00 % mysqld
14:36:01  9424 be/4 mysql       0.00 K/s 2694.21 K/s  0.00 %  0.00 % mysqld

So try it out on your server.