Never Ending Security

It starts all here

Tag Archives: Ad-blocking

Setup an Ad Blocking Machine on a Raspberry Pi


Raspberry Pi: Ad Blocking


This Ad Blocking tutorial will show you how to use your Raspberry Pi as a WiFi access point that blocks ads by default for any devices using it. This is really neat in that it would work for your Android or iOS device, your Xbox 360, TiVo, laptop, and more without needing to customize any of those devices other than to use your Raspberry Pi as the access point for WiFi. Using an ad-blocker can be useful for conserving bandwidth, helping out low-power devices, or for keeping your sanity while browsing the web!

ad blocking


This tutorial assumes you have your Pi mostly set up and ready to go.

  • Install the OS onto your SD card
  • Boot the Pi and configure
  • Don’t forget to change the default password for the ‘pi’ acccount!
  • Set up and test the Ethernet and Wifi connection
  • Set up your Raspberry Pi as an Access Point
  • Connect with a USB console cable (optional)

When done you should have a Pi that is booting Raspbian, you can connect to with a USB console cable and log into the Pi via the command line interface. Your Pi should also be already setup as a WiFi access point


Install Software

Once you’ve done all the preparation, setting your Pi up as an ad blocking AP is actually rather simple. The first thing we need to do is install the necessary software, and set it all up.

To start with, uninstall isc-dhcp-server and install dnsmasq. We’ll be using dnsmasq as your dhcp server instead of isc-dhcp-server. You may want to do a sudo apt-get update first, and then execute the following:

sudo apt-get autoremove isc-dhcp-server
sudo apt-get install -y dnsmasq dnsutils

You can test that it installed properly by checking the status:

sudo service dnsmasq status

ad blockingCreate and edit a new file for the dhcp server settings:

sudo nano /etc/dnsmasq.d/dnsmasq.custom.conf

Add the following to that file:

interface=wlan0
dhcp-range=wlan0,192.168.42.10,192.168.42.50,2h
# Gateway
dhcp-option=3,192.168.42.1
# DNS
dhcp-option=6,192.168.42.1
dhcp-authoritative

Save the file by typing in Control-X then Y then return

These are the minimum settings required to get the dhcp server setup properly. There are a lot more settings available as examples in /etc/dnsmasq.conf if you want to configure it further.

Any files added to the directory /etc/dnsmasq.d are automatically loaded by dnsmasq after a restart. This is a convenient way to override or add new configuration files in Debian. Next, let’s update the dns nameservers to route to our pixelserv IP address first. We’ll also go ahead and use the google nameservers.

Open the file with sudo nano /etc/resolv.conf and replace the contents with the following:

nameserver 192.168.42.49
nameserver 8.8.8.8
nameserver 8.8.4.4

After saving the resolv.conf file, let’s restart dnsmasq to have the settings take effect:

sudo service dnsmasq restart

Try testing out the dns by using the dig command.

Ok, now that we have all of that setup, let’s create a script that will redirect known ad servers to an internal IP address. This will essentially cause any requests to generate an HTTP 404.

Open nano and create a file with the following command:

sudo nano /usr/local/bin/dnsmasq_ad_list.sh

Copy and paste the following into the file, and save and exit:

#!/bin/bash

ad_list_url="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
pixelserv_ip="192.168.42.49"
ad_file="/etc/dnsmasq.d/dnsmasq.adlist.conf"
temp_ad_file="/etc/dnsmasq.d/dnsmasq.adlist.conf.tmp"

curl $ad_list_url | sed "s/127\.0\.0\.1/$pixelserv_ip/" > $temp_ad_file

if [ -f "$temp_ad_file" ]
then
    #sed -i -e '/www\.favoritesite\.com/d' $temp_ad_file
    mv $temp_ad_file $ad_file
else
    echo "Error building the ad list, please try again."
    exit
fi

service dnsmasq restart

Notice, in the above script there is a line starting with “#sed “. You can uncomment that, and modify it to remove your favorite sites from the ad blocking list so you can continue to support them. You can add as many of those lines as you’d like. One example would be:

sed -i -e '/ads\.stackoverflow\.com/d' $temp_ad_file

Next allow the file to be executed:

sudo chmod +x /usr/local/bin/dnsmasq_ad_list.sh

Manually run the script to test that it works:

sudo /usr/local/bin/dnsmasq_ad_list.sh

You should see the following output:

ad blockingThe above script will need to be run as root in order to properly update the configuration file. Basically, what it does it grab a pre-generated list of known ad server domain names from http://pgl.yoyo.org/adservers and save them to a file in /etc/dnsmasq.d/dnsmasq.adlist.conf. It then restarts dnsmasq.

You could manually do this every once in a while, but it’s easier if we setup a weekly cron job with the root user.

Open crontab with the following command:

sudo crontab -e

Add the following line to the end of the file and save it:

@weekly /usr/local/bin/dnsmasq_ad_list.sh

Congratulations, you should now be blocking ads with your Raspberry Pi. You can test that it’s working by executing the following:

dig doubleclick.com

ad blocking

And you should see that it gets routed to the 192.168.42.49 IP address:

One drawback to just stopping here is that we’re now timing out on requests that are going to return an HTTP code of 404 for any ad servers. This won’t be terribly fast. Another drawback is that any areas with ads will potentially still take up space in the page. We can fix this in the next section.

Improving Performance

One way to improve the performance of the ad blocker is to serve either a 1×1 transparent image, or a blank HTML page. This can be done in a number of ways so we’ll show a couple of the options available to you.


Pixelserv

The first, and maybe most common is to use pixelserv. This is a really lightweight perl web server that simply serves a 1×1 transparent GIF to any requests made to it. Thus, anytime ads get redirected by dnsmasq to pixelserv, you’ll actually receive a tiny image that won’t be visible on the page.

To start with, download the pixelserv file:

sudo curl -o /usr/local/bin/pixelserv http://proxytunnel.sourceforge.net/files/pixelserv.pl.txt

And also change the permissions:

sudo chmod 755 /usr/local/bin/pixelserv

Now, open the file with nano:

sudo nano /usr/local/bin/pixelserv

You can see that the pixelserv is a fairly small perl script. Let’s edit it to change the IP address that we’re using (192.168.42.49) to redirect the ads to. Find the line with LocalHost, and change it to the following:

$sock = new IO::Socket::INET (  LocalHost => '192.168.42.49',

Save the file by typing in Control-X then Y then return

You could try running the server now, but you’d get the following error:

pi@raspberrypi /usr/local/bin $ ./pixelserv
error : cannot bind : Cannot assign requested address exit

We can resolve this by adding this IP address to our wlan0 interface . Open nano and the interfaces file:

sudo nano /etc/network/interfaces

Update your iface wlan0 inet static section to look like the following. We’re adding the last two lines (post-up and pre-down):

iface wlan0 inet static
  address 192.168.42.1
  netmask 255.255.255.0
  post-up ip addr add dev wlan0 192.168.42.49/24
  pre-down ip addr del dev wlan0 192.168.42.49/24

Save the file by typing in Control-X then Y then return

Now reboot your Pi so the settings take effect:

sudo reboot

Once your system comes back up, try running the server. It won’t output anything in the console, but you can try refreshing a page you know that has ads that get blocked:

sudo /usr/local/bin/pixelserv

It could get annoying having to always run that command. The next logical step would be to create a service that will start the server for us whenever we start our Pi. Let’s do that.

First, kill the server you’re running by typing “Ctrl-C”.

Now, create a new file with nano:

sudo nano /etc/init.d/pixelserv

Copy and paste the following to that file:

### BEGIN INIT INFO
# Provides:          pixelserv
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: pixelserv server for ad blocking
# Description:       Server for serving 1x1 pixels
### END INIT INFO

case "$1" in
   start)
     echo "pixelserv: starting"
     /usr/local/bin/pixelserv &
     ;;
   stop)
     echo "pixelserv: stopping"
     killall pixelserv
     ;;
   *)
     echo "Usage: service $0 {start|stop}"
     exit 1
     ;;
esac

exit 0

Save the file by typing in Control-X then Y then return

Change the permissions on that file:

sudo chmod 744 /etc/init.d/pixelserv

Test that the script works by starting and stopping it:

sudo /etc/init.d/pixelserv start
sudo /etc/init.d/pixelserv stop

Now, enable the script on startup of your Pi:

update-rc.d pixelserv defaults

You can manually start/stop the pixelserv server by executing the following:

sudo service pixelserv start
sudo service pixelserv stop

If you decide you’d rather use the Apache solution, you can disable the pixelserv service on startup:

sudo update-rc.d -f pixelserv remove

Apache

As an alternative to pixelserv, we can use Apache to serve a blank html file with an HTTP 200 response. Apache is a bit heavier but likely much more stable than pixelserv.

Let’s start by installing Apache (make sure to stop and disable pixelserv if you’ve already installed that):

sudo apt-get install apache2 -y

By default apache is listening to all IP addresses on port 80. You can change this if you like, but it’s not necessary.

Test that apache is picking up our redirected requests:

curl doubleclick.com

You should get the following:

pi@raspberrypi ~ $ curl doubleclick.com
<html><body><h1>It works!</h1>
<p>This is the default web page for this server.</p>
<p>The web server software is running but no content has been added, yet.</p>
</body></html>
pi@raspberrypi ~ $

That’s a bit more than what we want. Let’s modify it so we basically just get an OK from apache instead of any content.

First enable the apache2 rewrite engine by executing the following:

sudo a2enmod rewrite

Next, let’s update the default VirtualHost. Execute the following to open the default VirtualHost file in nano:

sudo nano /etc/apache2/sites-available/default

Edit the <Directory /var/www/> section to look like the following (we’re adding the last three lines):

        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
                RewriteEngine on
                RedirectMatch 200 (.*)$
                ErrorDocument 200 " "
        </Directory>

Save the file by typing in Control-X then Y then return

Make the same change for the default-ssl file as well (sudo nano /etc/apache2/sites-available/default-ssl).

At this point if you restart Apache, you’ll get an error about not being able to determine the server’s fully qualified domain name. We can fix that by executing the following command:

echo "ServerName raspberrypi" | sudo tee -a /etc/apache2/conf.d/fqdn

Ok, now we can restart apache:

sudo service apache2 restart

And test that the response from Apache has changed:

pi@raspberrypi /etc/apache2/sites-available $ curl doubleclick.com
 pi@raspberrypi /etc/apache2/sites-available $

That’s it, you’re all setup for super-speedy ad blocking from your Raspberry Pi!

Raspberry Pi Ad blocking proxy installation using Privoxy


Raspberry Pi Ad blocking proxy installation using Privoxy

So I wanted to block adverts that always seemed to pop up whenever I visited a certain few websites (non porn related). I decided to look in to what is available for the Raspberry Pi and use it as a proxy server that would filter all of my traffic for me.

The benefits of using a proxy server ad blocker over an in browser ad blocker is that the server does all the work for you and not your client (browser). This ensures that ads are removed much earlier than during the rendering phase of your browser and therefore results in quicker page loads, well that’s my theory anyway.

So what did I find out? Well there is a very simple service called Privoxy! As always here is the marketing information from their website (which I have to say is one of the most basic sites I have ever visited):

Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data and HTTP headers, controlling access, and removing ads and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. It has application for both stand-alone systems and multi-user networks.

Excellent it ticks all of the boxes I wanted. So now all that is left to do is walk you through how to install this on to your Raspberry Pi.

Firstly SSH in to your Pi and run an update and upgrade command to ensure all libraries are up to date.

pi@raspberrypi ~ $ sudo apt-get update
pi@raspberrypi ~ $ sudo apt-get upgrade

Then we can just grab the privoxy package.

pi@raspberrypi ~ $ sudo apt-get install privoxy

You should then see the package install

Need to get 779 kB of archives.
After this operation, 2,492 kB of additional disk space will be used.
Do you want to continue [Y/n]? y

Setting up privoxy (3.0.19-2) ...

Once completed we need to make some minor changes to the privoxy configuration file. This is easy enough to do via nano.

pi@raspberrypi ~ $ sudo nano /etc/privoxy/config

You need to find the line that talks about your “listen-address” as per below:

#
#      Suppose you are running Privoxy on an IPv6-capable machine and
#      you want it to listen on the IPv6 address of the loopback device:
#
#        listen-address [::1]:8118
#
#listen-address  localhost:8118
#
#
#  4.2. toggle

Change this listen-address to be the IP address of your Pi, the internal IP that is so something along the lines of 192.168.0.10 would be mine.

Your file should look like the below extract once un-commenting and updating the IP.

#
#      Suppose you are running Privoxy on an IPv6-capable machine and
#      you want it to listen on the IPv6 address of the loopback device:
#
#        listen-address [::1]:8118
#
listen-address  192.168.0.10:8118
#
#
#  4.2. toggle

I would love to explain what else is held in the configuration file but quite honestly it is very self explanatory and also VERY long.

Next Hit Ctrl X and then Y to exit and save changes to the file.

You will then want to restart the privoxy service:

pi@raspberrypi ~ $ sudo service privoxy restart

Wonderful.. we are done for initial set up. We now need to configure our laptops, tablets, phones and whatever else we want to use the advanced filtering proxy to point towards our Raspberry Pi.

To add the proxy info:
In Google Chrome: Go to Settings > Show advanced settings… > Change proxy settings…
In Firefox: Go to Preferences > Advanced tab > Network tab > Settings button
In Internet Explorer: Go to Settings > Internet Options > Connections > LAN Settings > Tick use proxy server

Then enter the proxy server IP and Port which in our case will be the Pi IP address and Port 8118.

After that restart your browser and try and access the following page:

http://config.privoxy.org/

Hopefully you will see something along the lines of:

This is Privoxy 3.0.19 on raspberrypi.local (192.168.0.10), port 8118, enabled

Thats it! I’ll let you do the rest of the exploring.

It is worth noting that if you want to use this service when you are out and about, maybe using your cell phones mobile data service you can. All you need to do is open up the port 8118 on your router and instead of referencing your internal Pi IP address on your phones proxy server you just use your external router IP address. This will then pass any traffic from your phone through to the router and on to the Pi.