Build your own Google TV Using Raspberry Pi

Please note that this project is not intended to replicate an actual GoogleTV, but it’s simply a proof of concept using modern web technologies.

This is the new project I will ‘try-out’ in the next few days [ which I actually dig out from Donald’s Blog ]. All Credit goes to him along with a big TnX for this wonderful idea. Make sure to support the developer and visit his page and [ also fork the project]. This workshop was given at Lamba Labs Beirut First Hackerspace after a series of lightning talks check out the presentation here If you’d like to bypass the tutorial and jump into the fun stuff, you can always  fork the code on Github

What’s Google TV ?

Turned out that Google is also doing its own thing for the 10-foot screen. Google announced 2 versions of their famous new TV, the first is called the Buddy Box which is currently an expensive box manufactured by Sony and the second is an Integrated TV built right into the TV set that will be announced soon.

The Google TV looks something like that:

Google TV preview

Developers: you can start building your own Web Apps for the Google TV or renovate any android app to fit the 10′ Screen, all the resources can be found at Google’s Developers Site

Build your own Google TV

Hackers & makers like to re-invent the wheel, and it’s always fun when you do. So we’re going tobuild our own version of the Google TV using the following open source technologies:

Hardware:

• The RaspberryPi

Software Stack:

• Raspbian OS – a Debian distro specially made for the rPi
• NodeJsChromium Browser
  • Socket.io – to handle the connection between our remote and our TV via websockets
  • Express – to handle some basic http requests
  • Omxcontrol – a simple module to control the OMXPlayer which is the best video player on the rPi
• OMXPlayer
• Youtube-dl – a script that let you download youtube videos
• QuoJS – to handle swipe gestures on the mobile web app
• HTML5, CSS3 transitions, Javascript, and Moustache as a template engine
• Youtube API

The end result

Raspberry Pi TV with its special remote controller

Walkthrough

The project is divided into 4 main categories:

1. Installing the software stack
2. Basic shell commands & scripts
3. Building the backend: NodeJS + Express + Socket.io
4. Building the front end

1.Installing the software stack:

INSTALL RASPBIAN & NODEJS

Follow this tutorial to install Raspbian and Node Js on your Raspberry Pi

INSTALL CHROMIUM & YOUTUBE-DL

Install Chromium Browser for the Raspberry Pi Source

sudo apt-get install chromium-browser

In order to have a better display you can also install MC core fonts using

sudo apt-get install ttf-mscorefonts-installer

Install and Update Youtube Downloader

sudo apt-get install youtube-dl 

sudo youtube-dl -U

Note-1: There’s a problem when you want to stream videos on the RaspberryPi from youtube in Chromium, they’re extremely slow because the videos are not being rendered on the GPU. Youtube-dl comes as a quick alternative, the video is downloaded instead then played by the OMXPlayer which will render our videos on the GPU giving us a good quality of HD videos.

Note-2: The OMXPlayer is installed by default on the Raspbian.

2.Basic shell commands & scripts

If you’re using SSH to connect to your RaspberryPi you should first add “DISPLAY=:0.0″ to your env variables, by simply executing

export DISPLAY=:0.0

To check all your environment variables

env

Test Chromium in Kiosk Mode:

chromium --kiosk http://www.google.com

Test Youtube-dl

youtube-dl youtube_video_url

I’ve added few parameters to youtube-dl to change the name of the downloaded file to be just the “-o youtube ID [dot] the extension” and with the “-f /22/18 ” I can force this script to download for me a 720p version of the video. Check out the full list of supported youtube formats here

youtube-dl  -o "%(id)s.%(ext)s" -f /22/18 youtube_video_url

After downloading the video, try playing it using OMXPLayer

omxplayer youtube_video_file

Have fun trying the keyboard shortcuts to pause/resume your video and a lot more

Fancy! Let’s automate this process using Node JS

Building the backend: NodeJS + Express + Socket.io

The source code is intended to be simple for the sake of the workshop. Here’s the project’s hierarchy:

• publicapp.js
  • js
  • css
  • images
  • fonts
  • index.html
  • remote.html
• package.json

Package.json – A JSON file needed by npm to auto-install dependencies and save some basic info about your project

{
    "name": "GoogleTV-rPi",
    "version": "0.0.1",
    "private": false,
    "scripts": {
        "start": "node app.js"
    },
    "dependencies": {
    "express": "3.1.1",
    "socket.io":"0.9.14",
    "omxcontrol":"*"
    }
}

after creating this file, go to your app directory and run the following to install the dependencies.

npm install

Note-3: Notice that a folder called node_modules will be created prior to this action, if you like to use git, don’t forget to create a .gitignore file and simply write into it “node_modules” this will ignore the folder node_modules from being added to your git project

Create the app.js file and lets start by creating our basic HTTP Express Server

var express = require('express')
  , app = express()  
  , server = require('http').createServer(app)
  , path = require('path')

// all environments
app.set('port', process.env.TEST_PORT || 8080);
app.use(express.favicon());
app.use(express.logger('dev'));
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(express.static(path.join(__dirname, 'public')));

//Routes
app.get('/', function (req, res) {
  res.sendfile(__dirname + '/public/index.html');
});

app.get('/remote', function (req, res) {
  res.sendfile(__dirname + '/public/remote.html');
});

server.listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

This is our basic Express HTTP server configuration with our routes. To test what’ve done so far, you should first create the index.html and remote.html files inside the public/ directory, write your favorite “Hello, World” messages into them, then go back to your terminal and execute

node app.js

or

npm start

Note-4: That will only work if you have added the following piece of code to your package.json

...
"scripts": {
        "start": "node app.js"
    },
...

Once your server starts it will output that Express server listening on port 8080
To test your “Hello, World” pages you should run this application in the background by simply doing

node app.js &

Now this is the most primitive way to launch a Node application in the background, while learning node you might bump into some modules that automates this simple task, just likeForever.js

Now we have our Node Application up and running in the background, let’s open chromium in kiosk mode and test our Hello, World pages.

chromium --kiosk http://localhost:8080

Adding the Socket.io Magic

I strongly believe that WebSockets are the foundation of the modern web, I always like to point out the following analogy that helped me understand Socket.io

When AJAX first popped out, old skool developers felt its magic, but they’ve encountered many problems due to how different browsers handle Asynchronous JavaScript and XML requests. jQuery came with the solution by providing a nice and minimal set of functions to deal with the browsers nightmare. Socket.io did the same but for WebSockets, even more!

In order to provide realtime connectivity on every browser, Socket.IO selects the most capable transport at runtime, without it affecting the API.

1. WebSocket
2. Adobe® Flash® Socket
3. AJAX long polling
4. AJAX multipart streaming
5. Forever Iframe
6. JSONP Polling

In order to integrate Socket.io we should add the following to our app.js file:

var express = require('express')
  , app = express()  
  , server = require('http').createServer(app)
  , path = require('path')
  , io = require('socket.io').listen(server)
  , spawn = require('child_process').spawn

and to minify the logs add this:

//Socket.io Config
io.set('log level', 1);

When developing with Socket.io always think like you’re creating a Hello, World Chat Application. I’ve added a simple Chat Application done with Node & Socket.io on a github repo for the sake of this tutorial!

Our Socket.io Server is ready, but it doesn’t do anything, we should implement how we process messages and events sent from the client to the server.

Here’s how you implement this on the server’s side, note that you should also implement how you handle messages on the client’s side, we will see that as we progress throughout this tutorial.

io.sockets.on('connection', function (socket) {
    socket.emit('message', { message: 'welcome to the chat' });
    socket.on('send', function (data) {
        //Emit to all
        io.sockets.emit('message', data);
    });
});

Now our server Emits the message “message” whenever a new client is connected, and waits for an event name “send” to process the data and emit it back to all connected clients

In our case We have two types of clients: The RaspberryPi Display (Screen) and the Mobile Web Application (Remote)

var ss;
//Socket.io Server
io.sockets.on('connection', function (socket) {

 socket.on("screen", function(data){
   socket.type = "screen";
   //Save the screen socket
   ss = socket;
   console.log("Screen ready...");
 });

 socket.on("remote", function(data){
   socket.type = "remote";
   console.log("Remote ready...");
   if(ss != undefined){
      console.log("Synced...");
   }
 });
)};

Client Side Sockets Handeling

inside remote.html we should have the following:

    <script src="/socket.io/socket.io.js"> </script>
    <script>
      //use http://raspberryPi.local if your using Avahi Service 
          //or use your RasperryPi IP instead
          var socket = io.connect('http://raspberrypi.local:8080');
      socket.on('connect', function(data){
        socket.emit('screen');
      });
    </script>

On our index.html

    <script src="/socket.io/socket.io.js"> </script>
    <script>
      //use http://raspberryPi.local if your using Avahi Service 
          //or use your RasperryPi IP instead
          var socket = io.connect('http://raspberrypi.local:8080');
      socket.on('connect', function(data){
        socket.emit('screen');
      });
    </script>

Execute Shell Commands from Node Server

Node enables us to run a system command within a new child process, and listen in on its input/output. This includes being able to pass arguments to the command, and even pipe the results of one command to another. 

The basic way of executing shell commands from NodeJS is very simple

spawn('echo',['foobar']);

But if you want to pipe in the output, you should add the following function to your app.js file:

//Run and pipe shell script output 
function run_shell(cmd, args, cb, end) {
    var spawn = require('child_process').spawn,
        child = spawn(cmd, args),
        me = this;
    child.stdout.on('data', function (buffer) { cb(me, buffer) });
    child.stdout.on('end', end);
}

Adding OMXControl – the OMXPlayer controller Node Module

Luckily I found a node module on npmjs.org that let you control your OMXPlayer using Express!
just add the following to your app.js file to use it.

var omx = require('omxcontrol');

//use it with express
app.use(omx());

This will create for us the following routes, that we can use to control and play our videos:

http://localhost:8080/omx/start/:filename

http://localhost:8080/omx/pause


http://localhost:8080/omx/quit

Pretty Awesome!

Putting it all together

Our evolved app.js file

/**
 * Module dependencies.
 */

var express = require('express')
  , app = express()  
  , server = require('http').createServer(app)
  , path = require('path')
  , io = require('socket.io').listen(server)
  , spawn = require('child_process').spawn
  , omx = require('omxcontrol');

// all environments
app.set('port', process.env.TEST_PORT || 8080);
app.use(express.favicon());
app.use(express.logger('dev'));
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(express.static(path.join(__dirname, 'public')));
app.use(omx());

//Routes
app.get('/', function (req, res) {
  res.sendfile(__dirname + '/public/index.html');
});

app.get('/remote', function (req, res) {
  res.sendfile(__dirname + '/public/remote.html');
});

//Socket.io Congfig
io.set('log level', 1);

server.listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

//Run and pipe shell script output 
function run_shell(cmd, args, cb, end) {
    var spawn = require('child_process').spawn,
        child = spawn(cmd, args),
        me = this;
    child.stdout.on('data', function (buffer) { cb(me, buffer) });
    child.stdout.on('end', end);
}

//Save the Screen Socket in this variable
var ss;
//Socket.io Server
io.sockets.on('connection', function (socket) {

 socket.on("screen", function(data){
   socket.type = "screen";
   ss = socket;
   console.log("Screen ready...");
 });
 socket.on("remote", function(data){
   socket.type = "remote";
   console.log("Remote ready...");
 });

 socket.on("controll", function(data){
    console.log(data);
   if(socket.type === "remote"){

     if(data.action === "tap"){
         if(ss != undefined){
            ss.emit("controlling", {action:"enter"}); 
            }
     }
     else if(data.action === "swipeLeft"){
      if(ss != undefined){
          ss.emit("controlling", {action:"goLeft"}); 
          }
     }
     else if(data.action === "swipeRight"){
       if(ss != undefined){
           ss.emit("controlling", {action:"goRight"}); 
           }
     }
   }
 });

 socket.on("video", function(data){

    if( data.action === "play"){
    var id = data.video_id,
         url = "http://www.youtube.com/watch?v="+id;

    var runShell = new run_shell('youtube-dl',['-o','%(id)s.%(ext)s','-f','/18/22',url],
        function (me, buffer) { 
            me.stdout += buffer.toString();
            socket.emit("loading",{output: me.stdout});
            console.log(me.stdout)
         },
        function () { 
            //child = spawn('omxplayer',[id+'.mp4']);
            omx.start(id+'.mp4');
        });
    }    

 });
});

Building the front-end

Raspberry Pi TV Screen Front-end

Describing in details how I built the front-end is outside the scope of this tutorial, however I would like to point out few tips that I discovered while doing this project over the weekend.

When designing for the 10′ Screen there’s some design considerations that you should follow, Google assembled a nice set of these standards on their Developers Site

Raspberry Pi TV Remote

Instead of creating a typical remote, full of fake buttons, I decided to give QuoJS a try, it’s really fantastic and easy to use!

$$(".r-container").swipeLeft(function(){
socket.emit('control',{action:"swipeLeft"}); 
});

Here’s an example of how I send the message “Control” back to the server with the data action:”swipeLeft”
the server will handle that message by sending it to the screen, the screen client will handle this message by moving the selected square to the next app (Watch, Listen, Play)

I’ve also stumbled upon few trick that will let your iPhone mobile web app look like a native one with a nice icon and a splash screen.
Just add the following to your HTML <head></head> blocks

<link rel="apple-touch-icon" href="images/custom_icon.png"/>
<link rel="apple-touch-startup-image" href="images/startup.png">
<meta name="viewport" content="width=device-width initial-scale=1, maximum-scale=1, user-scalable=no" />
<meta name="apple-mobile-web-app-title" content="Remote">
<meta name="apple-mobile-web-app-capable" content="yes">

Wrap-up

This project is still a work in progress, updates coming soon. If you liked this tutorial please don’t forget to check the source code on Github and show some love by starring it .

Special Thanks to everyone at Lamba Labs Beirut Hackerspace , and of course Donald Derek.

I would Highly recommend this project. A lot of quality time [again]  spent  playing with RaspberryPi building interesting and very useful setup.

Onion Pi – Build a Raspberry Pi Tor Onion Router Machine

Onion Pi

Another Excellent Raspberry Pi project which is now coming bundled with Tor Onion Router which gives you opportunity to create secure network wherever you are. For more information about the project please visit Adafruit Learning System. Credit: Created by Ladyada [ Many Many TNX ] As usually discussion is open on ARRAKIS  

Feel like someone is snooping on you? Browse anonymously anywhere you go with the Onion Pi Tor proxy. This is fun weekend project that uses a Raspberry Pi, a USB WiFi adapter and Ethernet cable to create a small, low-power and portable privacy Pi.

Using it is easy-as-pie. First, plug the Ethernet cable into any Internet provider in your home, work, hotel or conference/event. Next, power up the Pi with the micro USB cable to your laptop or to the wall adapter. The Pi will boot up and create a new secure wireless access point called Onion Pi. Connecting to that access point will automatically route any web browsing from your computer through the anonymizing Tor network.

What is Tor?

Tor is an onion routing service – every internet packet goes through 3 layers of relays before going to your destination. This makes it much harder for the server you are accessing (or anyone snooping on your Internet use) to figure out who you are and where you are coming from. It is an excellent way to allow people who are blocked from accessing websites to get around those restritions.

According to the Tor website:

Journalists use Tor Onion to communicate more safely with whistleblowers and dissidents. Non-governmental organizations (NGOs) use Tor to allow their workers to connect to their home website while they’re in a foreign country, without notifying everybody nearby that they’re working with that organization.Groups such as Indymedia recommend Tor Onion for safeguarding their members’ online privacy and security. Activist groups like the Electronic Frontier Foundation (EFF) recommend Tor as a mechanism for maintaining civil liberties online. Corporations use Tor as a safe way to conduct competitive analysis, and to protect sensitive procurement patterns from eavesdroppers. They also use it to replace traditional VPNs, which reveal the exact amount and timing of communication. Which locations have employees working late? Which locations have employees consulting job-hunting websites? Which research divisions are communicating with the company’s patent lawyers?A branch of the U.S. Navy uses Tor for open source intelligence gathering, and one of its teams used Tor while deployed in the Middle East recently. Law enforcement uses Tor for visiting or surveilling web sites without leaving government IP addresses in their web logs, and for security during sting operations.

BEFORE YOU START USING YOUR PROXY – remember that there are a lot of ways to identify you, even if your IP address is ‘randomized’. Delete & block your browser cache, history and cookies – some browsers allow “anonymous sessions”. Do not log into existing accounts with personally identifying information (unless you’re sure that’s what you want to do). And read https://www.torproject.org/ for a lot more information on how to use TOR in a smart and safe way

This tutorial is a great way to make something fun and useful with your Raspberry Pi, but it is a work in progress. We can’t guarantee that it is 100% anonymous and secure! Be smart & paranoid about your TOR usage.

What you’ll need

You’ll need a few things to run this tutorial:

• Raspberry Pi model B – Ethernet is required
• Ethernet cable
• WiFi adapter – Not all WiFi adapters work !
• SD Card (4GB or greater) with Raspbian on it. You can either copy the Raspbian image onto it or buy a ready-made Raspbian card
• Power supply for your Pi & a Micro USB cable
• USB Console cable (optional) – this makes it a little easier to debug the system
  
• Case for your Pi (optional)
• A SD or MicroSD card reader (optional)

Chances are you’ve got a couple of these items already. If not, our Onion Pi starter pack has everything you need !

Preparation

This tutorial assumes you have your Pi mostly set up and have followed our “Raspberry Pi as Wifi Access Point” tutorial

Please follow these tutorials in order to

• Install the OS onto your SD card
  If you bought an SD card with Wheezy pre-burned on you can skip this step
• Boot the Pi and configure
  Don’t forget to change the default password for the ‘pi’ acccount!!!

Make sure to expand the filesystem to the entire disk or you may run out of space

• Set up and test the Ethernet and Wifi connection
  Check that you can ping from the Raspberry Pi and that your Wifi adapter is recognized and shows up as wlan0 when you run ifconfig -a
• Connect with a USB console cable (optional)
  Handy for debugging especially when connecting to the access point hosted by the Pi

When done you should have a Pi that is booting Raspbian, you can connect to with a USB console cable and log into the Pi via the command line interface.

• Then follow our Pi-as-Access-Point tutorial to set up the Pi as a wifi access point router.

When done you should be able to connect to the Pi as a WiFi access point and connect to the internet through it.

 

It is possible to do this tutorial via ssh on the Ethernet port or using a console cable.

If using a console cable, even though the diagram on the last step shows powering the Pi via the USB console cable (red wire) we suggest not connecting the red wire and instead powering from the wall adapter. Keep the black, white and green cables connected as is.

 

Install TOR

Essentially, this tutorial just follows the tor “anonymizing middlebox” writeup here.

We’ll begin by installing tor – the onion routing software.

Log into your pi by Ethernet or console cable and run

sudo apt-get install tor

Edit the tor config file by running

sudo nano /etc/tor/torrc

and copy and paste the text into the top of the file, right below the the FAQ notice.

Copy Code

Log notice file /var/log/tor/notices.log
VirtualAddrNetwork 10.192.0.0/10
AutomapHostsSuffixes .onion,.exit
AutomapHostsOnResolve 1
TransPort 9040
TransListenAddress 192.168.42.1
DNSPort 53
DNSListenAddress 192.168.42.1

Let’s edit the host access point so it is called something memorable like Onion Pi – don’t forget to set a good password, don’t use the default here!

 

Time to change our ip routing tables so that connections via the wifi interface (wlan0) will be routed through the tor software.
Type the following to flush the old rules from the ip NAT table

sudo iptables -F
 sudo iptables -t nat -F

Type the following to route all DNS (UDP port 53) from interface wlan0 to internal port 53 (DNSPort in our torrc)

sudo iptables -t nat -A PREROUTING -i wlan0 -p udp --dport 53 -j REDIRECT --to-ports 53

Type the following to route all TCP traffic from interface wlan0 to port 9040 (TransPort in our torrc)

sudo iptables -t nat -A PREROUTING -i wlan0 -p tcp --syn -j REDIRECT --to-ports 9040

Next you can check that the ip tables are right with

sudo iptables -t nat -L

If all is good, we’ll save it to our old NAT save file

sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

It will automatically get loaded when the networking is set up on reboot (as we did in the last tutorial on making a Pi access point)

Next we’ll create our log file (handy for debugging) with

sudo touch /var/log/tor/notices.log
sudo chown debian-tor /var/log/tor/notices.log
sudo chmod 644 /var/log/tor/notices.log

Check it with

ls -l /var/log/tor

Start the tor service manually

sudo service tor start

Check its really running (you can run this whenever you’re not sure, it something is wrong you’ll see a big FAIL notice

sudo service tor status

Finally, make it start on boot

sudo update-rc.d tor enable

That’s it, now you’re ready to test in the next step.

Test It!

OK now the fun part! It’s time to test your TOR anonymizing proxy. On a computer, check out the available wifi networks, you should see the Onion Pi network

Connect to it using the password you entered into the hostapd configuration file

You can open up a Terminal or command prompt and ping 192.168.42.1 to check that your connect to the Pi is working. However you won’t be able to ping outside of it because ping’s are not translated through the proxy

To check that the proxy is working, visit a website like http://www.ipchicken.com which will display your IP address as it sees it and also the matching domain name if available. The IP address should not be from your internet provider – in fact, if you reload the page it should change!

Your web browsing traffic is now anonymized!

onion onion onion onion onion

BEFORE YOU START USING YOUR PROXY – remember that there are a lot of ways to identify you, even if your IP address is ‘randomized’. Delete your browser cache, history and cookies (some browsers allow “anonymous sessions”) and read https://www.torproject.org/ for a lot more information on how to use Tor Onion Rrouter in a smart and safe way

Control Your Home Lights with a Raspberry Pi

In the past, when we’ve wanted to connect a hardware project to the internet, we’ve gone with Arduino and an ethernet shield. The problem with this solution is that the price was high and the available space for software was limited. This is where the Raspberry Pi really shines! But in order for the Raspberry Pi to provide these great benefits, you’ll need your program to be able to communicate. In particular, a web-based API will make it a snap to communicate with. Python is a popular language for using the GPIO pins and so pairing that with the Twisted networking module makes for a powerful program. 

Turning a Raspberry Pi into a networked controller

First, let’s consider a simple example of a single bulb lamp connected to a powerswitch tail, which communicates with a Raspberry Pi on GPIO pin 17. If you wanted to write a python program that would turn the lamp on, then off the code would look like this:

If you’ve programmed for an Arduino before, this code is really straightforward.

import RPi.GPIO as GPIO         #import the GPIO library
GPIO.setmode(GPIO.BCM)          #Set the pin naming scheme
GPIO.setmode(17, GPIO.OUT)      #Tell pin 17 to be an output pin

GPIO.output(17, true)           #Turn the lamp on!
GPIO.output(17, false)          #Turn the lamp off!

• The first few lines are setup, import will load the Raspberry Pi GPIO library
• The second sets the pin naming scheme
• The third tells pin 17 to be an output pin.

Once the setup is taken care of you can turn the pin off or on now, easily. Which in turn turns the lamp off and on!

Now we can control the hardware, but what good is a controller program if it isn’t  listening? Here’s where Twisted comes in, let’s say we wanted to be able to turn the lamp off and on remotely. The code to listen for input over the web looks like this:

from twisted.web.server import Site       #import twisted stuff
from twisted.web.resource import Resource
from twisted.internet import reactor

import RPi.GPIO as GPIO     #import the GPIO library
GPIO.setmode(GPIO.BCM)      #Set the pin naming scheme
GPIO.setmode(17, GPIO.OUT)  #Tell pin 17 to be an output pin

class lampAPI(Resource)
   def render_GET(self, request):
            if 'light' in request.args:                        #'light' is the URL variable
                        if request.args['light'][0] == "off":     #Did the client put 'off' in the light var?
                                GPIO.output(power_pin, False)        #turn the lamp off
                                return " light off "                 #tell the browser/client that we did it
                        if request.args['light'][0] == "on":      #Did the client put 'on' in the light var? 
                                GPIO.output(power_pin, True)         #turn the lamp on     
                                return " light on "                  #tell the browser/client that we did it

root = Resource()                #Create a root 
root.putChild("API", lampAPI())  #Create a child that will handle requests (the second argument must be the class name)
factory = Site(root)             #Initialize the twisted object 
reactor.listenTCP(80, factory)   #Choose the port to listen on (80 is standard for HTTP) 
reactor.run()                    #Start listening, this command is an infinite loop 
                                 #so don't bother putting anything after it

After including the twisted modules you’ll recognize the GPIO setup again.

The next chunk is our response class, it will define a standard ‘get’ request that we can use as an API to turn the bulb off and on. ‘lampAPI’ is the class name I chose, name it whatever you like. In the first IF statement, the first term in the condition is the URL variable. So if you want your API call to look like this ( http://127.0.0.1/API?foo=1 ) then you would put ‘foo’ in the quotes instead. Once you’ve confirmed that your variable made it into the request object, you can now check what the variable holds. That’s what the next two IF statements are doing. The variables can contain whatever you like. You could look for a zero or one, yes or no, off or on, etc. Again, you should recognize the GPIO statement from before, that will turn the light off or on, respectively. The return statement should be a string, and whatever it contains is what the client will spit out to the browser!

Lastly, the Twisted object is initialized, and the program enters an infinite loop where we check to see if there’s a new request, forever. Now you can turn your light off and on from anywhere by typing the following URL/commands:

• http://%5Bip of rasberry pi]/API?light=on (Turn the light on)
• http://%5Bip of rasberry pi]/API?light=off (Turn the light off)

Ok, this is great, but an API is an interface for robots, let’s make one for humans.

The first step is to set up the root object of the twisted object to pass files to the client, which is the primary job of a web-server. This can be done simply by changing:

root = Resource()

to

root = File(“lampwww”)

Now, create a directory called ‘lampwww’ in the same directory as your python file. Anything called from the URL of the Pi will be passed along to the browser. The last step here is simply creating an html file with links that will call up our API. In it’s simplest form, it could look like the following (create a file called index.html and put it in the lampwww dir) :

My Lamp Control
<a href="http://[ip of lamp]/API?light=on"> Turn On </a> | <a href="http://[ip of lamp]/API?light=off"> Turn Off</a>

If you browse to the IP of your lamp, you’ll see the heading and the two choices to turn the lamp on or off; Tada! Web-connected lamp. home lights

Getting more sophisticated

“I crafted this simple example to make an easy to follow tutorial but there’s a whole lot of improvements that could be made. In my Cloud Lamp project for instance, I not only had light bulbs connected, I also had a 60 ‘pixel’ string of ws2801 LED’s. So what could we do with this type of display? How about creating different modes of operation, each with it’s own unique display.”

Adding this functionality required a few key changes, first, the end of our example is an infinite loop that looks for changes from the network. We need to change this so that we can do other things besides listening. The statement,

reactor.run()

will run the loop

reactor.startRunning(false)

Now we’re not tied down to only listening for new connections. The end of our program would now be an infinite loop that includes but is not limited to, our listening statement, like this:

while True:
   reactor.iterate()

Next, how do we animate the LED’s in different modes, which is a continual process, while also listening for new connections. I opted for a simple list of IF’s that choose which animation subroutine to run on every loop iteration. A global ‘mode’ variable will decide which animation to iterate through. Each of the animation sub-routines is contained in a function that can be called on from the main while loop. In my infodisplay mode, I’m showing the state of the house (Whether all the doors are close or not, whether there’s motion in the house), the surf report and the weather report. I accomplished this by using the Beautiful Soup module to scrape a Magic Seaweed widget, Weather Underground’s API for the weather report and MicasaVerde’s json feed, all formatted to output into a serial list of multicolored LED’s.

And finally, I wrote a more sophisticated web interface, taking advantage of the ample speed and space available on the Raspberry Pi. Besides a fancier looking interface, the important part is multi-directional feedback powered by Jquery. Talking to the lamp is easy, i just use standard .click’s on the mode buttons and .get’s to call the API that I created. The web interface is also able to dynamically tell which mode it’s in and whether the bulbs are off or on. I do this by creating another API that spits out a Json feed of the state of all the lamp’s functions, like this:

def getStatus():
        global MODE
        global BULB
        global STROBE
        global BRIGHT

        jsonString = ''
        jsonString += '{"lamp":{"bulb":'
        jsonString += str(BULB)
        jsonString += ',"strobefreq":'
        jsonString += str(STROBE)
        jsonString += ',"mode":'
        jsonString += str(MODE)
        jsonString += ',"brightness":'
        jsonString += str(BRIGHT)
        jsonString += '}}'
        #print jsonString
        return jsonString

Then, use Jquery’s cross-domain capable Jsonp functionality to check the status every X number of seconds and update the interface. It’s fairly simple from a code perspective and the result is that even with multiple interfaces open, they will all update depending on the actual state of the Lamp!

Why not dig into all the code yourself? It’s up on Github, feel free to modify, use, scoff at or appreciate all of it!

The Cloud Lamp

DIG INTO ALL THE CODE FOR MY LAMP PROJECT OR GRAB THE WHOLE PACKAGE FOR YOUR MODIFYING PLEASURE!

What’s next?

In addition to turning the light-bulbs off and on and running the LED’s internally, I’ve also got a mode that allows the LED’s to be controlled externally through a TCP stream. It’s too slow to be very functional right now, however. And still requires some debugging. To try that as well, here’s an exampleProcessing.org script:

import processing.net.*; 
Client myClient; 

//Put the IP of your Cloud Lamp here in this variable
String ip="";
String datamode="-1";

int pcirc[][] = new int[60][3]; 
int LEDnum = 60;
int RGBnum = 3;

int xCenter = 300;
int yCenter = 300;
int r = 250;
int LEDdiameter = 15;

int count = 0;
int count2 = 0;

void setup()
{
  setModeData();
  delay(2000);
  myClient = new Client(this, ip, 50007);
  size(600, 600);
  ellipseMode(CENTER);

  //Create empty LED array
  for (int i = 0; i < LEDnum; i ++ ) {
    for (int j = 0; j < RGBnum; j ++ ) {
      // Initialize each object
      pcirc[i][j] = 0;
    }
  }
}

//Show the RGB spectrum sequentially, similar to the beginning of Lady Ada's ws2801 test script
void draw()
{
  background(150.0);

  if(count2 == 0)
  {
    pcirc[count][0] = 255;
    pcirc[count][1] = 0;
    pcirc[count][2] = 0;
  }
  if(count2 == 1)
  {
    pcirc[count][0] = 0;
    pcirc[count][1] = 255;
    pcirc[count][2] = 0;
  }
  if(count2 == 2)
  {
    pcirc[count][0] = 0;
    pcirc[count][1] = 0;
    pcirc[count][2] = 255;
  }

  count++;
  if(count == 60)
  {
    count = 0;
    count2++;
      if(count2 == 3)
      {
        count2 = 0;
      }
  }
  refreshCircles();
  delay(40);
  sendData();
}

This function draws and refreshes an onscreen simulation of the LED's in the Cloud Lamp
void refreshCircles()
{
  for (int i = 0; i < LEDnum; i ++ ) 
  {
    float xrad = (float) (xCenter + r * Math.cos(2 * Math.PI * i / LEDnum));
    float yrad = (float) (yCenter + r * Math.sin(2 * Math.PI * i / LEDnum));   

    int Col[] = new int[3];
    for (int j = 0; j < RGBnum; j ++ ) 
    {
      // Initialize each object
      Col[j] = pcirc[i][j];
    }

    //print("(" + Col[0] + ":" + Col[1] + ":" + Col[2] + ")|");

    //line(xCenter, yCenter, xrad, yrad);
    fill(Col[0],Col[1],Col[2]);
    //stroke(0);

    ellipse(xrad, yrad, LEDdiameter, LEDdiameter);
  }  
}

//This function sends the string data prepared by the draw loop
void sendData()
{
  String data = "";
  for (int i = 0; i < LEDnum; i ++ ) 
  {
    int Col[] = new int[3];
    for (int j = 0; j < RGBnum; j ++ ) 
    {
      // Initialize each object
      Col[j] = pcirc[i][j];
    }

    data = data + Col[0] + "," + Col[1] + "," + Col[2] + ",";
  }
    myClient.write(data);
}

//This function puts the Cloud Lamp into 'Data Stream' mode
void setModeData()
{
  Client c;
  String data;

  String domain = ip;
  String addr = "/API?mode=" + datamode;
  print(addr);
  c = new Client(this, domain, 80); // Connect to server on port 80 
  c.write("GET "+addr+" HTTP/1.1\r\n"); // Can replace / with, eg., /reference/ or similar path
  c.write("Host: "+domain+"\r\n"); // Which server is asked
  c.write("User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.04 Chromium/10.0.648.205 Chrome/10.0.648.205 Safari/534.16\r\n");
  c.write("Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n");
  c.write("Accept-Language: en-us,en;q=0.5\r\n");
  c.write("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n");
  c.write("\r\n");

  //c.stop();
}

More information at: http://falldeaf.com/2013/07/the-pi-control-script/

Raspberry Pi – OpenVPN

OpenVPN: Raspberry Pi

If you are in a public network, for example at university or the airport, your traffic can be recorded and decrypted. To prevent others from doing that you can send your traffic through a secured VPN-tunnel. The VPN-tunnel leads your traffic encrypted to a server which is processing your requests.

In the following tutorial you will learn how to run OpenVPN Server on your Raspberry Pi:

Requirements

Raspbian or a similar distribution.

Step 1

To be able to install the latest program versions we should update our packet sources:

sudo apt-get update

Step 2

Now we are installing Open VPN and OpenSSL.

sudo apt-get install openvpn openssl

Step 3

We are switching to the directory and paste a directory we will be needing later into it.

cd /etc/openvpn
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0 ./easy-rsa

Step 4

Now we open the file easy-rsa/vars with nano and apply some changes.

nano /easy-rsa/vars 
export EASY_RSA="`pwd`"
export EASY_RSA="/etc/openvpn/easy-rsa"

Step 5

We change the directory, log in as root user and execute some configurations.

cd easy-rsa
sudo su
source vars
./clean-all
./pkitool --initca
ln -s openssl-1.0.0.cnf openssl.cnf

Step 6

Now we are able to generate the components for the encryption of Open VPN. After the first input you will be asked for the abbreviation of your country (US = USA, DE – Germany, AT = Austria, CH – Switzerland). All other inputs can simply be confirmed.

./build-ca OpenVPN
./build-key-server server
./build-key client1

Step 7

The calculation of the last components can take a few minutes.

./build-dh
exit

Step 8

We have to switch the directory again and create the file openvpn.conf with the following content:

cd ..
sudo touch openvpn.conf
sudo nano openvpn.conf

dev tun
proto udp
port 1194
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
user nobody
group nogroup
server 10.8.0.0 255.255.255.0
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 3
client-to-client
push "redirect-gateway def1"
#set the dns servers
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
log-append /var/log/openvpn
comp-lzo

You can change the DNS-servers to any DNS you like.

Step 9

Now, create the internet-forwarding for the CPN clients. If you are not using an ethernet-cable (e.g. Wifi) you will have to replace “eth0″ with the name of your network device.

sudo sh -c ‘echo 1 > /proc/sys/net/ipv4/ip_forward’
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/8 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE

Step 10

One of the final steps will be to delete the “#” before net.ipv4.ip_forward=1 in sysctl.conf.

cd ..
sudo nano sysctl.conf

Step 11

A part of the above settings have to be endorsed as a crontab to work permanently. Insert following line at the end of the crontab file (replace “eth0″ if you did above):

crontab -e

@reboot sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/8 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE

Step 12

Again change to the root-user and to the directory /etc/openvpn/easy-rsa/keys in which we will create the fileraspberrypi.ovpn and fill it with the code of the second paragraph. RASPBERRY-PI-IP should be replaced by the IP address of your Pi or, if you are using a DynDNS service,  by the given domain.

sudo su
cd /etc/openvpn/easy-rsa/keys
nano raspberrypi.ovpn

dev tun
client
proto udp
remote RASPBERRY-PI-IP 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client1.crt
key client1.key
comp-lzo
verb 3

Step 13

Now create a packet with all the needed files for the client, which we will place in /home/pi and give the user pi the needed rights to the file.

tar czf openvpn-keys.tgz ca.crt ca.key client1.crt client1.csr client1.key raspberrypi.ovpn
mv openvpn-keys.tgz /home/pi
chown pi:pi /home/pi/openvpn-keys.tgz
exit

Step 14

Restart the server.

sudo /etc/init.d/openvpn start

Finished! Now we are able to download the file die openvpn-keys.tar.gz on the client and extract the files to your Open VPN client folder.

An Open VPN Client for Windows is: http://openvpn.se/
for Mac: https://code.google.com/p/tunnelblick/

Linux users simply install the packet openvpn

Raspberry Pi – HoneyPot

Now let’s have another cool setup for your Raspberry Pi! This time, we would like to introduce to you Glastopf Pi! Raspbery Pi Honeypot

Download

Glastopf is a web application honeypot project lead by Lukas Rist a.k.a glaslos of the Honeynet Project. The Glastopf project started in the year 2009. It is a simple and minimalistic web server written in Python that records information of web-based application attacks like Structured Query Language Injection (SQLI), Remote Code Execution (RCE), Local File Inclusion (LFI), Remote File Inclusion (RFI), and many more, and it emulates web application vulnerabilities, tricking attackers or scanners that it is a vulnerable web server.

Here are some snippets of the README file for this project in order to understand this web application honeypot better:

The adversaries usually use search engines and special crafted search requests to find their victims. In order to attract them, Glastopf provide those keywords (aka dork) and extracts them also from request and extends its attack surface automatically. So over time and with a growing number of attacks, the honeypot gets more and more attractive. In the feature we will make the SQL injection emulator pubic, provide IP profiling for crawler recognition and intelligent dork selection.

Glastopf has also hpfeeds, which is a central logger of the project that reports the events, but it can actually be turned off under the glastopf.cfg configuration file.

Now let’s begin with the setup!

Prerequisites

1. A Raspberry Pi board
2. The Soft-float Debian “wheezy” Linux, which can be downloaded here
3. A Micro SD card of at least 4GB in size
4. An Internet Access

Install

Add the backports repository to your sources list file, which can be found under the /etc/apt directory:

sudo echo “deb http://backports.debian.org/debian-backports squeeze-backports main” >> /etc/apt/sources.list

Now let’s install the dependencies:

sudo apt-get update
sudo apt-get install python python-openssl python-gevent libevent-dev python-dev build-essential make
sudo apt-get install python-argparse python-chardet python-requests python-sqlalchemy python-lxml
sudo apt-get install python-beautifulsoup python-pip python-dev python-numpy python-setuptools
sudo apt-get install python-numpy-dev python-scipy libatlas-dev g++ git php5 php5-dev liblapack-dev gfortran
sudo apt-get install libxml2-dev libxslt-dev
sudo pip install –upgrade distribute

Configure the PHP sandbox.
Download BFR (Better Function Replacer) by using git:

sudo apt-get install git-core
cd /opt
sudo git clone git://github.com/glastopf/BFR.git
cd BFR
sudo phpize
sudo ./configure –enable-bfr
sudo make && make install

It should have this following message after the make install:

Build complete.
Don’t forget to run ‘make test’.
Installing shared extensions: /usr/lib/php5/20100525+lfs/

Copy or append the search path to bfr.so and add it to php.ini file:

sudo echo “zend_extension = /usr/lib/php5/20100525+lfs/bfr.so” >> /etc/php5/cli/php.ini

You should see the extension on the output by using the php –version command in the terminal:

root@n0where.net:~# php –version
PHP 5.4.4-14 (cli) (built: Mar 4 2013 19:41:30)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
with Better Function Replacer (BFR) v0.1, by Lukas Rist

Install the latest stable release of Glastopf from pip:

sudo pip install glastopf

Configuration and Preparation for the Glastopf environment

cd /opt
sudo mkdir glastopfi

Stop the apache service so that the web application honeypot could listen to port 80:

sudo service apache2 stop

Now, run the web application honeypot:

sudo glastopf-runner

A new default glastopf.cfg will be created in the glastopfpi directory, which can be customized to your liking just like what port you want the application to listen on.

Sample Screenshot of the Web Server Running

Logging

The log files can be found under the log directory.

(glastopf.glastopf) Initializing Glastopf using "/opt/glastopfpi" as work directory
(glastopf.glastopf) Connecting to main database with: sqlite:///db/glastopf.db
(glastopf.modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database.
(requests.packages.urllib3.connectionpool) Starting new HTTPS connection (1): mnemosyne.honeycloud.net
(requests.packages.urllib3.connectionpool) "POST /login HTTP/1.1" 200 30
(requests.packages.urllib3.connectionpool) "GET /api/v1/aux/dorks?limit=1000 HTTP/1.1" 200 177444
(glastopf.modules.handlers.emulators.dork_list.mnem_service) Successfully retrieved 1000 dorks from the mnemosyne service.
(glastopf.glastopf) Generating initial dork pages - this can take a while.
(glastopf.modules.reporting.auxiliary.log_hpfeeds) Connecting to feed broker.
(glastopf.modules.reporting.auxiliary.log_hpfeeds) Connected to hpfeed broker.
(glastopf.glastopf) Glastopf started and privileges dropped.
(glastopf.glastopf) 192.168.1.2 requested GET / on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /style.css on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=-1%20union%20select%201,2,3,4,5,6 on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=1%20union%20select%201,2,3,4,5,6 on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=1 on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /style.css on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=1' on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /style.css on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=1%20union%20select%201,2,3,4,5 on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=ls%20-la on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /style.css on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) 192.168.1.2 requested GET /?id=http://map.honeycloud.net/ on 192.168.1.3
(glastopf.sandbox.sandbox) File successfully parsed with sandbox.
(glastopf.glastopf) 192.168.1.2 requested GET /favicon.ico on 192.168.1.3
(glastopf.glastopf) Stopping Glastopf.

Conclusion

So what can we learn or get from setting up this kind of web application honeypot? Here are some scenarios and examples:

1. Discover malicious sources, links, or scripts just like the URLs used in Timthumb Remote Code Execution attacks: http://www.target.tld/wp-content/themes/THEME/timthumb.php?src=http://blogger.com.iamahacker.com/backdoor.php
2. Capture the links or sources of possible IRC botnets.
3. Determine what kind of attacks are being thrown out in a day by attackers or scanners.
4. Live capture of SQL injection techniques in a POST request used by attackers.
5. Discover unknown or new attacks.

Setup an Ad Blocking Machine on a Raspberry Pi

Raspberry Pi: Ad Blocking

This Ad Blocking tutorial will show you how to use your Raspberry Pi as a WiFi access point that blocks ads by default for any devices using it. This is really neat in that it would work for your Android or iOS device, your Xbox 360, TiVo, laptop, and more without needing to customize any of those devices other than to use your Raspberry Pi as the access point for WiFi. Using an ad-blocker can be useful for conserving bandwidth, helping out low-power devices, or for keeping your sanity while browsing the web!

This tutorial assumes you have your Pi mostly set up and ready to go.

• Install the OS onto your SD card
• Boot the Pi and configure
• Don’t forget to change the default password for the ‘pi’ acccount!
• Set up and test the Ethernet and Wifi connection
• Set up your Raspberry Pi as an Access Point
• Connect with a USB console cable (optional)

When done you should have a Pi that is booting Raspbian, you can connect to with a USB console cable and log into the Pi via the command line interface. Your Pi should also be already setup as a WiFi access point

Install Software

Once you’ve done all the preparation, setting your Pi up as an ad blocking AP is actually rather simple. The first thing we need to do is install the necessary software, and set it all up.

To start with, uninstall isc-dhcp-server and install dnsmasq. We’ll be using dnsmasq as your dhcp server instead of isc-dhcp-server. You may want to do a sudo apt-get update first, and then execute the following:

sudo apt-get autoremove isc-dhcp-server
sudo apt-get install -y dnsmasq dnsutils

You can test that it installed properly by checking the status:

sudo service dnsmasq status

Create and edit a new file for the dhcp server settings:

sudo nano /etc/dnsmasq.d/dnsmasq.custom.conf

Add the following to that file:

interface=wlan0
dhcp-range=wlan0,192.168.42.10,192.168.42.50,2h
# Gateway
dhcp-option=3,192.168.42.1
# DNS
dhcp-option=6,192.168.42.1
dhcp-authoritative

Save the file by typing in Control-X then Y then return

These are the minimum settings required to get the dhcp server setup properly. There are a lot more settings available as examples in /etc/dnsmasq.conf if you want to configure it further.

Any files added to the directory /etc/dnsmasq.d are automatically loaded by dnsmasq after a restart. This is a convenient way to override or add new configuration files in Debian. Next, let’s update the dns nameservers to route to our pixelserv IP address first. We’ll also go ahead and use the google nameservers.

Open the file with sudo nano /etc/resolv.conf and replace the contents with the following:

nameserver 192.168.42.49
nameserver 8.8.8.8
nameserver 8.8.4.4

After saving the resolv.conf file, let’s restart dnsmasq to have the settings take effect:

sudo service dnsmasq restart

Try testing out the dns by using the dig command.

Ok, now that we have all of that setup, let’s create a script that will redirect known ad servers to an internal IP address. This will essentially cause any requests to generate an HTTP 404.

Open nano and create a file with the following command:

sudo nano /usr/local/bin/dnsmasq_ad_list.sh

Copy and paste the following into the file, and save and exit:

#!/bin/bash

ad_list_url="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
pixelserv_ip="192.168.42.49"
ad_file="/etc/dnsmasq.d/dnsmasq.adlist.conf"
temp_ad_file="/etc/dnsmasq.d/dnsmasq.adlist.conf.tmp"

curl $ad_list_url | sed "s/127\.0\.0\.1/$pixelserv_ip/" > $temp_ad_file

if [ -f "$temp_ad_file" ]
then
    #sed -i -e '/www\.favoritesite\.com/d' $temp_ad_file
    mv $temp_ad_file $ad_file
else
    echo "Error building the ad list, please try again."
    exit
fi

service dnsmasq restart

Notice, in the above script there is a line starting with “#sed “. You can uncomment that, and modify it to remove your favorite sites from the ad blocking list so you can continue to support them. You can add as many of those lines as you’d like. One example would be:

sed -i -e '/ads\.stackoverflow\.com/d' $temp_ad_file

Next allow the file to be executed:

sudo chmod +x /usr/local/bin/dnsmasq_ad_list.sh

Manually run the script to test that it works:

sudo /usr/local/bin/dnsmasq_ad_list.sh

You should see the following output:

The above script will need to be run as root in order to properly update the configuration file. Basically, what it does it grab a pre-generated list of known ad server domain names from http://pgl.yoyo.org/adservers and save them to a file in /etc/dnsmasq.d/dnsmasq.adlist.conf. It then restarts dnsmasq.

You could manually do this every once in a while, but it’s easier if we setup a weekly cron job with the root user.

Open crontab with the following command:

sudo crontab -e

Add the following line to the end of the file and save it:

@weekly /usr/local/bin/dnsmasq_ad_list.sh

Congratulations, you should now be blocking ads with your Raspberry Pi. You can test that it’s working by executing the following:

dig doubleclick.com

And you should see that it gets routed to the 192.168.42.49 IP address:

One drawback to just stopping here is that we’re now timing out on requests that are going to return an HTTP code of 404 for any ad servers. This won’t be terribly fast. Another drawback is that any areas with ads will potentially still take up space in the page. We can fix this in the next section.

Improving Performance

One way to improve the performance of the ad blocker is to serve either a 1×1 transparent image, or a blank HTML page. This can be done in a number of ways so we’ll show a couple of the options available to you.

Pixelserv

The first, and maybe most common is to use pixelserv. This is a really lightweight perl web server that simply serves a 1×1 transparent GIF to any requests made to it. Thus, anytime ads get redirected by dnsmasq to pixelserv, you’ll actually receive a tiny image that won’t be visible on the page.

To start with, download the pixelserv file:

sudo curl -o /usr/local/bin/pixelserv http://proxytunnel.sourceforge.net/files/pixelserv.pl.txt

And also change the permissions:

sudo chmod 755 /usr/local/bin/pixelserv

Now, open the file with nano:

sudo nano /usr/local/bin/pixelserv

You can see that the pixelserv is a fairly small perl script. Let’s edit it to change the IP address that we’re using (192.168.42.49) to redirect the ads to. Find the line with LocalHost, and change it to the following:

$sock = new IO::Socket::INET (  LocalHost => '192.168.42.49',

Save the file by typing in Control-X then Y then return

You could try running the server now, but you’d get the following error:

pi@raspberrypi /usr/local/bin $ ./pixelserv
error : cannot bind : Cannot assign requested address exit

We can resolve this by adding this IP address to our wlan0 interface . Open nano and the interfaces file:

sudo nano /etc/network/interfaces

Update your iface wlan0 inet static section to look like the following. We’re adding the last two lines (post-up and pre-down):

iface wlan0 inet static
  address 192.168.42.1
  netmask 255.255.255.0
  post-up ip addr add dev wlan0 192.168.42.49/24
  pre-down ip addr del dev wlan0 192.168.42.49/24

Save the file by typing in Control-X then Y then return

Now reboot your Pi so the settings take effect:

sudo reboot

Once your system comes back up, try running the server. It won’t output anything in the console, but you can try refreshing a page you know that has ads that get blocked:

sudo /usr/local/bin/pixelserv

It could get annoying having to always run that command. The next logical step would be to create a service that will start the server for us whenever we start our Pi. Let’s do that.

First, kill the server you’re running by typing “Ctrl-C”.

Now, create a new file with nano:

sudo nano /etc/init.d/pixelserv

Copy and paste the following to that file:

### BEGIN INIT INFO
# Provides:          pixelserv
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: pixelserv server for ad blocking
# Description:       Server for serving 1x1 pixels
### END INIT INFO

case "$1" in
   start)
     echo "pixelserv: starting"
     /usr/local/bin/pixelserv &
     ;;
   stop)
     echo "pixelserv: stopping"
     killall pixelserv
     ;;
   *)
     echo "Usage: service $0 {start|stop}"
     exit 1
     ;;
esac

exit 0

Save the file by typing in Control-X then Y then return

Change the permissions on that file:

sudo chmod 744 /etc/init.d/pixelserv

Test that the script works by starting and stopping it:

sudo /etc/init.d/pixelserv start
sudo /etc/init.d/pixelserv stop

Now, enable the script on startup of your Pi:

update-rc.d pixelserv defaults

You can manually start/stop the pixelserv server by executing the following:

sudo service pixelserv start
sudo service pixelserv stop

If you decide you’d rather use the Apache solution, you can disable the pixelserv service on startup:

sudo update-rc.d -f pixelserv remove

Apache

As an alternative to pixelserv, we can use Apache to serve a blank html file with an HTTP 200 response. Apache is a bit heavier but likely much more stable than pixelserv.

Let’s start by installing Apache (make sure to stop and disable pixelserv if you’ve already installed that):

sudo apt-get install apache2 -y

By default apache is listening to all IP addresses on port 80. You can change this if you like, but it’s not necessary.

Test that apache is picking up our redirected requests:

curl doubleclick.com

You should get the following:

pi@raspberrypi ~ $ curl doubleclick.com
<html><body><h1>It works!</h1>
<p>This is the default web page for this server.</p>
<p>The web server software is running but no content has been added, yet.</p>
</body></html>
pi@raspberrypi ~ $

That’s a bit more than what we want. Let’s modify it so we basically just get an OK from apache instead of any content.

First enable the apache2 rewrite engine by executing the following:

sudo a2enmod rewrite

Next, let’s update the default VirtualHost. Execute the following to open the default VirtualHost file in nano:

sudo nano /etc/apache2/sites-available/default

Edit the <Directory /var/www/> section to look like the following (we’re adding the last three lines):

        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
                RewriteEngine on
                RedirectMatch 200 (.*)$
                ErrorDocument 200 " "
        </Directory>

Save the file by typing in Control-X then Y then return

Make the same change for the default-ssl file as well (sudo nano /etc/apache2/sites-available/default-ssl).

At this point if you restart Apache, you’ll get an error about not being able to determine the server’s fully qualified domain name. We can fix that by executing the following command:

echo "ServerName raspberrypi" | sudo tee -a /etc/apache2/conf.d/fqdn

Ok, now we can restart apache:

sudo service apache2 restart

And test that the response from Apache has changed:

pi@raspberrypi /etc/apache2/sites-available $ curl doubleclick.com
 pi@raspberrypi /etc/apache2/sites-available $

That’s it, you’re all setup for super-speedy ad blocking from your Raspberry Pi!

PiMSO – A Raspberry Pi based Wi-Fi Oscilloscope

PiMSO

PiMSO, is a Raspberry Pi controlled 200 Msa/S mixed signal oscilloscope. Depending on your application, it can be configure to use the Midori browser on the Pi GUI or access remotely via the internet. Since the PiMSO GUI is web based, You can also control it from a browser on your tablet or smart phone. But what happens when you are at a location that there are no convenient access to a network? For this project I’ll demonstrate how to build a Raspberry Pi based oscilloscope that is also a Wireless Access Point.

As a bare minimum you’ll need the Raspberry Pi and a Ralink RT5370 based Wi-Fi dongle. For this project I used the Tenda W311M from MicroCenter, primarily because it is Access Point capable and inexpensive. An even less expensive dongle that based on the same Ralink RT5370 chip from Comfast can be purchased from Aliexpress for less than $5 inc S/H. A quick search on Aliexpress for RT5370 should yield a few other options. If you don’t have a MSO-28 you can still try it out, the WebMSO28 GUI will run regardless if you have the MSO-28. It’ll just show “NO DSO”.

If you don’t want to roll your own, you can grab a pre-configured Raspian distro from here and skip to Step 6. Otherwise here is how to roll your own on top of your favorite distro.

Step 1: Setting up the Raspberry Pi

1. Grab the latest version of Raspian from http://www.raspberrypi.org/downloads. Follow the raw images installation instruction provided on this site http://elinux.org/RPi_Easy_SD_Card_Setup.

2. Boot up the Raspbian and set the it to boot into command line.

Step 2: Package Installation

1. Gain root privilege.

sudo -i

2. Update the package list.

apt-get update

3. Install the necessary packages.

apt-get install git-core build-essential nginx hostapd iw minicom usbmount dnsmasq libfcgi-dev

4. Go to the home directory.

cd / home/pi

5. wiringPi is not necessary if you don’t have PiLCD, but it is nice to have for future I/O control.

git clone git://git.drogon.net/wiringPi
cd wiringPi
./build

6. Grab and install the WebMSO-config.git so the Pi will recognize MSO-28 and MSO-19

cd ..
git clone git://github.com/tkrmnz/WebMSO-config.git
cd WebMSO-config
cp /home/pi/WebMSO-config/74-linkmso.rules /etc/udev/rules.d/.

7. At this point plug the MSO-28 into the Raspberry Pi, you should see it listed under /dev as MSO-28-0.

Step 3: Setting up the web server

1. Edit the nginx default file.

nano /etc/nginx/sites-available/default

Change the nginx default root directory to /var/www

root /var/www;

2. Also add the following block for fcgi port forwarding.

location /fcgi-bin/mso28ctl.fcgi{
include fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
}

3. Copy the proxy.conf to /etc/nginx

cp proxy.conf /etc/nginx/.

The content of the proxy.conf is shown below.

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

4. Check to see if the gzip is enabled in the /etc/nginx/nginx.conf

nano /etc/nginx/nginx.conf

un-comment gzip on;

gzip on;
gzip_disable "msie6";

5. Create the webpage folder

mkdir /var/www

6. Grab and install the WebMSO28fcgi javascripts content

cd /home/pi
git clone git://github.com/tkrmnz/WebMSO28fcgi.git
cd WebMSO28fcgi
cp -rf * /var/www/.

7. Start the nginx web server.

/etc/init.d/nginx start

At this point nginx should start serving the web page. ifconfig to get your ipv4 address and access it via a browser from a different computer on the same network. pimso pimso

Step 4: Setting up the FCGI server

1. Add execution privilege of the MSO28 fcgi driver in the /var/www/fcgi-bin directory.

chmod +x /var/www/fcgi-bin/mso28ctl.fcgi

2. Create a temporary ram drive for the mso28 fcgi driver to store the captured data.

mkdir /mnt/tmp
mount -t tmpfs -o size=20m tmpfs /mnt/tmp
ln -s /mnt/tmp /var/www/fcgi-bin/tmp

3. Grab and install the spawn-fcgi and mso28ctl.fcgi

cd /home/pi
git clone git://github.com/tkrmnz/mso28fcgi.git
copy spawn-fcgi-1.6.3.tar.bz2 to /home/pi and unzip the spawn-fcgi
cp mso28fcgi/spawn-fcgi-1.6.3.tar.bz2 .
tar -xvjf spawn-fcgi-1.6.3.tar.bz2

Install the spawn-fcgi

cd spawn-fcgi-1.6.3
./configure
make
make install
cp src/spawn-fcgi /usr/bin/.

Copy the init script from /home/pi/mso28fcgi to /etc/init.d

cp /home/pi/mso28fcgi/spawn-fcgi1 to /etc/init.d/.
chmod +x /etc/init.d/spawn-fcgi1

4. Compile and copy the mso28ctl to /www/var/fcgi-bin

cd mso28fcgi
touch main.c
sh makemso2.sh
sh msocpy6.sh

5. You can test the installing by starting the spawn-fcgi using

/etc/init.d/spawn-fcgi1 start

You should be able to see the blinking indicator next to the ”’Stby”’ above the timing window of the WebMSO28 client page.

If you see No MSO, then your spwan-fcgi is not running.

To stop the spawn-fcgi ”’ctrl-z”’ and kill the process

fuser -k 9000/tcp

6. To auto start spawn-fcgi on boot.

update-rc.d spawn-fcgi1 defaults

To remove spawn-fcgi from auto boot.

update-rc.d -f spawn-fcgi1 remove

At this point server works with ethernet cable.

Step 5: PiMSO Access Point setup

1. Grab WebMSO28AP files from Github.

git clone git://github.com/tkrmnz/WebMSO28AP.git

2. Copy the various conf files to their directories.

cd WebMSO28AP
cp dnsmasq.conf /etc/.
cp hostpad.conf /etc/hostapd/.
cp hosts /etc/.
cp interfaces /etc/network/.

Add DAEMON_CONF=”/etc/hostapd/hostapd.conf” to /etc/default/hostapd

nano /etc/default/hostapd

Look for DAEMON_CONF=””, insert the following

DAEMON_CONF=”/etc/hostapd/hostapd.conf”

Enable the hostapd and restart dnsmasq

update-rc.d hostapd enable
/etc/init.d/dnsmasq restart

Step 6: Connecting to PiMSO

You should be able to see and connect to the Access Point with the SSID of “WebMSO” with your portable device. Point the brower on your portable device to

http://mso28.linkmso

you should be able to see the WebMSO28 web client.

If you have a PiLCD, go to Step 7, otherwise you are done.

Step 7: PiLCD Setup

The goal of PiLCD module is provide a convenient way for the user to interact with a headless Raspberry Pi. The PiLCD module consists of a 16×2 LCD display with a rotary encoder switch. Together they provide a way for user to input data and read the results on the display.

One application is for a headless Raspberry Pi with a Wifi dongle to scan, select and connect to nearby Wifi Access Points.

The PiMSO project demonstrates how the PiLCD module can be used to provide a convenient way for the user see the Access Point and URL info of the PiMSO, it will also display current IPV4 address of the Ethernet connection.

1. In addition to wiringPi, you will also need nslookup, which is available in the dnsuitls package.

apt-get install dnsutils

2. Grab and install the PiLCD project from Github

cd /home/Pi
git clone git://github.com/tkrmnz/PiLCD.git
cd /home/pi/PiLCD

3. Copy the LCDinit scrip to init.d

cp LCDinit.sh /etc/init.d/.
cd /home/pi/wiringPi/examples
make /home/pi/PiLCD/lcd_ip2.c
chmod +x /etc/init.d/LCDinit.sh

4. you can test PiLCD by typing

/home/pi/lcd_ip2

You should be able to see the Wifi AP and URL information and Ethernet IPV4 by spinning the knob.

The LCDinit will execute the lcd_ip2 driver from PiLCD directory.
To run the LCDinit.sh on boot, do the following

update-rc.d LCDinit.sh defaults

To remove it from boot

update-rc.d -f LCDinit.sh remove

Raspberry Pi Video Tutorials

Raspberry Pi®‎ Tutorials

Playlist: https://www.youtube.com/playlist?list=PLQVvvaa0QuDeazo-AqrVk8BGUej9NCHrQ

• 			Raspberry Pi Part 1 Introduction door sentdex		Intro to Pi®	4:14
  			Raspberry Pi Part 2: Installing an Operating System door sentdex			6:23
  			Raspberry Pi Part 3: Plugging everything in door sentdex		Plugin session.	5:05
  			Raspberry Pi Part 4: Actually Installing an Operating System door sentdex		Actually Installing an Operating System	7:22
  			Raspberry Pi Part 5: Navigating Your New Pi door sentdex		Navigating Pi	6:06
  			Raspberry Pi Part 6: Overclocking and how to overclock Raspberry Pi door sentdex		Overclocking part 1	8:29
  			Raspberry Pi Part 7: Overclocking and how to overclock Raspberry Pi part 2 door sentdex		More on overclocking	7:55
  			Raspberry Pi Part 8: First Download and Update! (Firmware) door sentdex		First download and update (firmware)	6:29
  			Raspberry Pi Part 9: Navigating the Terminal door sentdex		Basics for those new to the terminal lingo.	21:16
  			Raspberry Pi Part 10: How to set up a Linux Web Server on your Pi door sentdex		Hosting websites off your Pi	20:53
  			Raspberry Pi Part 11: Remote Desktop door sentdex			9:26
  			GPIO Tutorial for Raspberry Pi Part 1 – Setting up door sentdex		Part 1 of using the GPIO on your Raspberry Pi	5:33
  			GPIO Tutorial for Pi Part 2 – Programming the GPIO door sentdex		GPIO tutorial part 2	11:51
  			GPIO Tutorial for Raspberry Pi Part 3 – Running our GPIO door sentdex			10:07
  			How to Put a Computer in Your Car with the Raspberry Pi door sentdex			15:50

  Robotics and the Raspberry Pi

  Playlist: https://www.youtube.com/playlist?list=PLQVvvaa0QuDeJlgD1RX9_49tMLUxvIxF4

  • 			Raspberry pi with Python for Robotics – Video Streaming Pi RC Car! door sentdex			7:38
    			Raspberry pi with Python for Robotics 1 – Supplies Needed door sentdex			6:34
    			Raspberry pi with Python for Robotics 2 – Motor Control door sentdex			15:27
    			Raspberry pi with Python for Robotics 3 – Connecting 4 motors door sentdex			8:33
    			Raspberry pi with Python for Robotics 4 – Forward and Reverse Motors door sentdex			10:20
    			Raspberry pi with Python for Robotics 5 – Turning our car door sentdex			5:48
    			Raspberry pi with Python for Robotics 6 – Pivot RC Car door sentdex			5:37
    			Raspberry pi with Python for Robotics 7 – Adding User Control door sentdex			14:13
    			Raspberry pi with Python for Robotics 8 – Remote Control Car Driving door sentdex			2:19
    			Raspberry pi with Python for Robotics 9 – Adding HC-SR04 Distance Sensor door sentdex			16:55
    			Raspberry pi with Python for Robotics 10 – Programming Distance Sensor HC-SR04 door sentdex			8:05
    			Raspberry pi with Python for Robotics 11 – Autopilot/auto correct door sentdex			8:05
    			Raspberry pi with Python for Robotics 12 – Autonomous beginnings door sentdex			14:05
    			Raspberry pi with Python for Robotics 13 – Testing Autonomous Code door sentdex			3:22
    			Raspberry Pi camera module streaming video to another computer door sentdex			5:08
    			Raspberry Pi Camera Module Streaming to Browser door sentdex			8:08

    Build your Own Supercomputer

    Playlist: https://www.youtube.com/playlist?list=PLQVvvaa0QuDf9IW-fe6No8SCw-aVnCfRi

    • 			Build Your Own Supercomputer 1 – About Supercomputers door sentdex			16:20
      			Build Your Own Supercomputer 2 – Supplies needed door sentdex			8:46
      			Build Your Own Supercomputer 3 – Installing operating system door sentdex			8:48
      			Build Your Own Supercomputer 4 – Downloading/Installing an MPI door sentdex			11:26
      			Build Your Own Supercomputer 5 – Testing supercomputer process door sentdex			12:49
      			Build Your Own Supercomputer 6 – Copying Master Node door sentdex			7:28
      			Build Your Own Supercomputer 7 – Setting up Master Login door sentdex			7:34
      			Build Your Own Supercomputer 8 – Completing our 2-Node supercomputer door sentdex			12:50
      			Getting MPI4py and MPI tutorial- Supercomputing and Parallel Programming in Python and MPI 1 door sentdex			11:49
      			Getting Node Rank tutorial – Supercomputing and Parallel Programming in Python and MPI 2 door sentdex			10:09
      			Conditional Statements tutorial – Supercomputing and Parallel Programming in Python and MPI 3 door sentdex			6:16
      			Network Size tutorial- Supercomputing and Parallel Programming in Python and MPI 4 door sentdex			3:30
      			Sending and Receiving data tutorial – Supercomputing and Parallel Programming in Python and MPI 5 door sentdex			14:37
      			Dynamically sending and receiving tutorial – Parallel Programming in Python and MPI 6 door sentdex			6:48
      			Data tagging tutorial – Supercomputing and Parallel Programming in Python and MPI 7 door sentdex			8:27
      			Broadcasting with bcast tutorial – Supercomputing and Parallel Programming in Python and MPI 8 door sentdex			3:27
      			Scatter tutorial – Supercomputing and Parallel Programming in Python and MPI 9 door sentdex			7:23
      			Gather tutorial – Supercomputing and Parallel Programming in Python and MPI 10 door sentdex			4:47

FruityWifi – An Open Source Tool to Audit Wireless Networks

FruityWifi is an open source tool to audit wireless networks. It allows the user to deploy advanced attacks by directly using the web interface or by sending messages to it. Initialy the application was created to be used with the Raspberry-Pi, but it can be installed on any Debian based system. FruityWifi v2.0 has many upgrades. A new interface, new modules, Realtek chipsets support, Mobile Broadband (3G/4G) support, a new control panel, and more. A more flexible control panel. Now it is possible to use FruityWifi combining multiple networks and setups: – Ethernet Ethernet, – Ethernet 3G/4G, – Ethernet Wifi, – Wifi Wifi, – Wifi 3G/4G, etc. Within the new options on the control panel we can change the AP mode between Hostapd or Airmon-ng allowing to use more chipsets like Realtek. It is possible customize each one of the network interfaces which allows the user to keep the current setup or change it completely. NEW MODULES FruityWifi is based on modules making it more flexible. These modules can be installed from the control panel to provide FruityWifi with new functionalities. Within the available modules you can find URLsnarf, DNSspoof, Kismet, mdk3, ngrep, nmap, Squid3 y SSLstrip (code injection functionality), Captive Portal, AutoSSH, Meterpreter, Tcpdump and more. AutoSSH allows the user to create a reverse ssh connection, restarting it in case that the connection has been closed or dropped. It is useful to keep a permanent connection with FruityWifi. Meterpreter is an outstanding tool to gather information from a compromised host, manipulate system processes and/or kill them, and more. This module allows FruityWifi to compromise more hosts and use them to access more devices and networks. Nessus is a vulnerability scanner. With this module it is possible to scan hosts from FruityWifi without using the Nessus interface. We can discover the vulnerabilities present on each of the hosts to understand the attack surface and compromise them. Among the new features FruityWifi now supports Mobile Broadband (3G/4G). We can use this module to connect a 3G/4G dongle and give internet access to FruityWifi without the need of Wifi or Ethernet. The main function of Tcpdump is to analyze network traffic. With this module we can intercept the traffic passing through the device, filter it and/or store it for post analysis. Ettercap is a tool able to capture network traffic and perform different attacks. With this module we can perform MITM attacks using ARP poisoning. VIDEOS FruityWifi Modules Sample (v1.7) FruityWifi + Raspberry Pi + adafruit 16×2 lcd + keypad More information at: http://www.fruitywifi.com

Raspberry Pi Ad blocking proxy installation using Privoxy

So I wanted to block adverts that always seemed to pop up whenever I visited a certain few websites (non porn related). I decided to look in to what is available for the Raspberry Pi and use it as a proxy server that would filter all of my traffic for me.

The benefits of using a proxy server ad blocker over an in browser ad blocker is that the server does all the work for you and not your client (browser). This ensures that ads are removed much earlier than during the rendering phase of your browser and therefore results in quicker page loads, well that’s my theory anyway.

So what did I find out? Well there is a very simple service called Privoxy! As always here is the marketing information from their website (which I have to say is one of the most basic sites I have ever visited):

Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data and HTTP headers, controlling access, and removing ads and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. It has application for both stand-alone systems and multi-user networks.

Excellent it ticks all of the boxes I wanted. So now all that is left to do is walk you through how to install this on to your Raspberry Pi.

Firstly SSH in to your Pi and run an update and upgrade command to ensure all libraries are up to date.

pi@raspberrypi ~ $ sudo apt-get update
pi@raspberrypi ~ $ sudo apt-get upgrade

Then we can just grab the privoxy package.

pi@raspberrypi ~ $ sudo apt-get install privoxy

You should then see the package install

Need to get 779 kB of archives.
After this operation, 2,492 kB of additional disk space will be used.
Do you want to continue [Y/n]? y

Setting up privoxy (3.0.19-2) ...

Once completed we need to make some minor changes to the privoxy configuration file. This is easy enough to do via nano.

pi@raspberrypi ~ $ sudo nano /etc/privoxy/config

You need to find the line that talks about your “listen-address” as per below:

#
#      Suppose you are running Privoxy on an IPv6-capable machine and
#      you want it to listen on the IPv6 address of the loopback device:
#
#        listen-address [::1]:8118
#
#listen-address  localhost:8118
#
#
#  4.2. toggle

Change this listen-address to be the IP address of your Pi, the internal IP that is so something along the lines of 192.168.0.10 would be mine.

Your file should look like the below extract once un-commenting and updating the IP.

#
#      Suppose you are running Privoxy on an IPv6-capable machine and
#      you want it to listen on the IPv6 address of the loopback device:
#
#        listen-address [::1]:8118
#
listen-address  192.168.0.10:8118
#
#
#  4.2. toggle

I would love to explain what else is held in the configuration file but quite honestly it is very self explanatory and also VERY long.

Next Hit Ctrl X and then Y to exit and save changes to the file.

You will then want to restart the privoxy service:

pi@raspberrypi ~ $ sudo service privoxy restart

Wonderful.. we are done for initial set up. We now need to configure our laptops, tablets, phones and whatever else we want to use the advanced filtering proxy to point towards our Raspberry Pi.

To add the proxy info:
In Google Chrome: Go to Settings > Show advanced settings… > Change proxy settings…
In Firefox: Go to Preferences > Advanced tab > Network tab > Settings button
In Internet Explorer: Go to Settings > Internet Options > Connections > LAN Settings > Tick use proxy server

Then enter the proxy server IP and Port which in our case will be the Pi IP address and Port 8118.

After that restart your browser and try and access the following page:

http://config.privoxy.org/

Hopefully you will see something along the lines of:

This is Privoxy 3.0.19 on raspberrypi.local (192.168.0.10), port 8118, enabled

Thats it! I’ll let you do the rest of the exploring.

It is worth noting that if you want to use this service when you are out and about, maybe using your cell phones mobile data service you can. All you need to do is open up the port 8118 on your router and instead of referencing your internal Pi IP address on your phones proxy server you just use your external router IP address. This will then pass any traffic from your phone through to the router and on to the Pi.

Webmin Setup On Raspberry Pi

Todays tutorial is aimed at those of us who want to use the webmin panel on our raspberry pi. Webmin essentially gives you a great overview of all the functions you would normally need to carry out using SSH on your headless setup. Theres a raft of features ranging from memory and process usage stats right through to rebooting and shutting down your pi all from the web.

It is a great application and I highly recommend it for those that aren’t so keen on using a terminal window all of the time.

In order to get this bad boy set up carry out the following steps.

First get the file.

pi@raspberrypi / $ sudo wget http://prdownloads.sourceforge.net/webadmin/webmin-1.580.tar.gz

The file is a gz so we can use the gunzip command to get to the tar.

pi@raspberrypi / $ sudo gunzip webmin-1.580.tar.gz

Now we have our tar we need to open it up.

pi@raspberrypi / $ sudo tar xf webmin-1.580.tar

Some of the above commands may take a while as it does the unpacking but do not fear all is ok.

I always like to try and keep my web apps in one place so I will create a folder in my /var/www to hold webmin.

pi@raspberrypi / $ sudo mkdir /var/www/webmin

Now we can run the simple setup script.

pi@raspberrypi / $ cd webmin-1.580




pi@raspberrypi /webmin-1.580 $ sudo sh setup.sh /var/www/webmin

You will be presented with the following.

***********************************************************************
*            Welcome to the Webmin setup script, version 1.580        *
***********************************************************************
Webmin is a web-based interface that allows Unix-like operating
systems and common Unix services to be easily administered.

Installing Webmin from /webmin-1.580 to /var/www/webmin ...

***********************************************************************
Webmin uses separate directories for configuration files and log files.
Unless you want to run multiple versions of Webmin at the same time
you can just accept the defaults.

Config file directory [/etc/webmin]:
Log file directory [/var/webmin]:

***********************************************************************
Webmin is written entirely in Perl. Please enter the full path to the
Perl 5 interpreter on your system.

Full path to perl (default /usr/bin/perl):

Testing Perl ...
Perl seems to be installed ok

***********************************************************************
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
cat: /etc/lsb-release: No such file or directory
Operating system name:    Debian Linux
Operating system version: 7.0

***********************************************************************
Webmin uses its own password protected web server to provide access
to the administration programs. The setup script needs to know :
 - What port to run the web server on. There must not be another
   web server already using this port.
 - The login name required to access the web server.
 - The password required to access the web server.
 - If the webserver should use SSL (if your system supports it).
 - Whether to start webmin at boot time.

Web server port (default 10000):
Login name (default admin):
Login password:
Password again:
Use SSL (y/n): n
Start Webmin at boot time (y/n): y
***********************************************************************
Copying files to /var/www/webmin ..
..done

Creating web server config files..
..done

Creating access control file..
..done

Inserting path to perl into scripts..
..done

Creating start and stop scripts..
..done

Copying config files..
..done

Configuring Webmin to start at boot time..
Created init script /etc/init.d/webmin
..done

Creating uninstall script /etc/webmin/uninstall.sh ..
..done

Changing ownership and permissions ..
..done

Running postinstall scripts ..
PID 3347 from file /var/webmin/miniserv.pid is not valid

..done

Enabling background status collection ..
PID 3347 from file /var/webmin/miniserv.pid is not valid
..done

Attempting to start Webmin mini web server..
Starting Webmin server in /var/www/webmin
Pre-loaded WebminCore
..done

***********************************************************************
Webmin has been installed and started successfully. Use your web
browser to go to

http://raspberrypi:10000/

and login with the name and password you entered previously.

Thats it! Just simply follow any instructions if prompted and your app will nicely set itself up. I have to say it is excellent and requires hardly any effort to get it all working considering the amount of functionality it has.

You can access the webmin page using:

http://host:10000/

Replacing host with either your domain or ip address.

You will be presented with the following type of page.

From here explore the webmin app using the drop down navigation on the left hand side. It will automatically pick up all of your installed packages and setup. It even has a nice area to control mysql without needing to install phpmyadmin. The only thing I will point out is there is nothing to directly support lighttpd. Webmin will install perfectly on a lighttpd server but it will not list it under its server options. I have a dead install of apache in mine.

Hope this offers some use, well worth getting!

MASSIVE COLLECTIONS: Awesome, Awesome All, Awesome-Awesome, Awesome-Awesomes, Awesome Awesomeness, Awesome-Collection, Lists, Lists Of Github Lists, List of Lists, Must-Watch-List and Wiki China Lists

Awesome

A curated list of awesome lists
For more info check: https://github.com/sindresorhus/awesome

Platforms

• Node.js
• Frontend Development
• iOS
• Android
• Hybrid Apps

Programming languages

• JavaScript
• Swift
• Python
• Rust
• Haskell
• Go
• Scala
• Ruby
• Clojure
• Elixir
• Erlang
• Julia
• Lua
• C
• C/C++
• R
• D
• Common Lisp
• Perl
• Groovy
• Dart
• Java
• OCaml
• Coldfusion
• Fortran
• .NET
• PHP

Front-end development

• JavaScript Must Watch Talks
• ES6 Tools
• Web Performance Optimization
• Web Tools
• Above-the-fold CSS Tools
• React
• Web Components
• AngularJS
• Backbone
• HTML5
• Sass
• SVG
• KnockoutJS
• Dojo Toolkit
• Inspiration
• Ember

Back-end development

• Django
• Play1 Framework
• CakePHP
• Symfony2
• Laravel

Computer science

• University Courses
• Data Science
• Machine Learning
• Speech and Natural Language Processing
• Linguistics
• Cryptography
• Computer Vision

Big data

• Big Data
• Public Datasets
• Hadoop

Theory

• Papers We Love
• Free Programming Books
• Talks
• Algorithms
• Algorithm Visualizations
• Artificial Intelligence

Miscellaneous

• GitHub Cheat Sheet
• Git Cheat Sheet
• Tools
• Game Development
• Game Talks
• Shell
• Dotfiles
• Dev Env
• Docker
• Sublime Text
• Images
• Creative Commons Media
• Database
• Sysadmin
• Analytics
• Vim
• Emacs
• Cordova
• Free Software Testing Books
• Selenium
• Services Engineering
• Android Security
• Vagrant
• Awesome
• Conferences
• Remote Jobs
• Styleguides
• Radio
• SQLAlchemy
• PCAPTools
• Continuous Delivery
• Fonts
• Readme
• Science Fiction
• Browser Extensions for GitHub
• Quick Look Plugins
• Learn to Program
• Boilerplate Projects

Awesome All

A curated list of all the awesome lists of awesome frameworks, libraries and software
For more info check: https://github.com/bradoyler/awesome-all

Contributing

Please take a quick gander at the contribution guidelines first. Thanks to all contributors; you rock!

Contents

• Android
• Bash
• Big data
• C
• Clojure
• Cobol
• Common Lisp
• D
• Elixir
• Frontend Dev
• Go
• Hadoop
• iOS
• Java
• JavaScript
• Lists
• Node.js
• PHP
• Python
• Ruby
• Scala
• Swift
• Sysadmin
• Talks
• Wikipedia

Awesome-Awesome

A curated list of awesome curated lists! Inspired by inspiration.
For more info check: https://github.com/erichs/awesome-awesome

• Awesome-Activeadmin
• Awesome-Algorithms
• Awesome-Android and in Spanish
• Awesome-Analytics
• Awesome-Angular
• Awesome-Apple
• Awesome-Arabic
• Awesome-AutoHotkey
• Awesome-AVR
• Awesome-BigData
• Awesome-Bootstrap
• Awesome-C
• Awesome-CakePHP
• Awesome-Clojure
• Awesome-Cloud
• Awesome-Cloud-Storage
• Awesome-Cobol
• Awesome-ColdFusion
• Awesome-Cordova
• Awesome-Cpp
• Awesome-CSS
• Awesome-D
• Awesome-Data-Science
• Awesome-Datasets
• Awesome-Dart
• Awesome-DB
• Awesome-DDD
• Awesome-DevEnv
• Awesome-DevTools
• Awesome-Delphi
• Awesome-Django
• Awesome-Django-CMS
• Awesome-Doctrine
• Awesome-Drupal
• Awesome-Dotfiles
• Awesome-Elixir
• Awesome-Emacs, also here
• Awesome-Ember
• Awesome-Erlang
• Awesome-Flask
• Awesome-Fortran
• Awesome-Github
• Awesome-Go
• Awesome-Grails
• Awesome-Groovy
• Awesome-Hadoop
• Awesome-Haskell
• Awesome-HTML5
• Awesome-iBeacon
• Awesome-Images
• Awesome-IOS
• Awesome-Java
• Awesome-Javascript
• Awesome-Laravel, also here
• Awesome-Lua and here
• Awesome-Luvit
• Awesome-Machine-Learning
• Awesome-NodeJS
• Aweomse-ObjC-Swift
• Awesome-Ocaml
• Awesome-Perl
• Awesome-Persian
• Awesome-PHP
• Awesome-Pipeline
• Awesome-Puppet
• Awesome-Python
• Awesome-R
• Awesome-Rails and here
• Awesome-Ruby
• Awesome-Rust
• Awesome-Sass
• Awesome-Scala
• Awesome-Selenium
• Awesome-Shell
• Awesome-Swift and on ZEEF
• Awesome-Sysadmin
• Awesome-Talks
• Awesome-Unity
• Awesome-Vagrant
• Awesome-Web
• Awesome-Webservice
• Awesome-Wikipedia
• Awesome-Wordpress
• Awesome-Wx
• Awesome-Yii
• Aweszome-zsh-plugins

Awesome Awesome

A curated list of amazingly awesome curated lists of amazingly awesome libraries, resources and shiny things for various languages and frameworks.
For more info check: https://github.com/oyvindrobertsen/awesome-awesome

• Awesome Awesome
  • C
  • Clojure
  • Common Lisp
  • Go
  • Java
  • JavaScript
  • PHP
  • Python
  • Ruby
  • Scala
  • Swift

C

• Awesome C

Clojure

• Awesome Clojure

Common Lisp

• Awesome Common Lisp

Go

• Awesome Go

Java

• Awesome Java
• Awesome Android

JavaScript

• Awesome JavaScript
• Awesome Node.js

PHP

• Awesome PHP

Python

• Awesome Python

Ruby

• Awesome Ruby (markets)
• Awesome Ruby (Sdogruyol)

Scala

• Awesome Scala

Swift

• Awesome Swift

Awesome-Awesomes

Awesome collection of awesome lists of libraries, tools, frameworks and software for any programming language, or closely related :D
For more info check: https://github.com/fleveque/awesome-awesomes

Feel free to add new lists or categories! Remember, it’s not mandatory that name starts with awesome- ;)

Programming languages | Frameworks, platforms, etc | Related and useful

Programming Languages

• C
• Clojure
• Common Lisp
• D
• Elixir
• Erlang
• Go
• Haskell
• Java
• JavaScript
• PHP
• Python
• Ruby
• Scala

C

• Awesome C – A curated list of awesome C libraries, frameworks and other shinies.

Clojure

• Awesome Clojure – A curated list of awesome clojure libraries and software

Common Lisp

• Awesome Common Lisp – A curated list of awesome Common Lisp libraries, software and other shinies.

D

• Awesome D – A curated list of awesome D documents, frameworks, libraries and software

Elixir

• Awesome Elixir – A curated list of amazingly awesome Elixir libraries, resources and shiny things

Erlang

Go

• Awesome Go – A curated list of awesome Go frameworks, libraries and software

Haskell

• Awesome Haskell – A curated list of awesome Haskell frameworks, libraries and software

Java

JavaScript

• Awesome JavaScript – A curated list of amazingly awesome browser-side JavaScript libraries, resources and shiny things

PHP

• Awesome PHP – A curated list of amazingly awesome PHP libraries, resources and shiny things

Python

• Awesome Python – A curated list of awesome Python frameworks, libraries and software

Ruby

• Awesome Ruby – by @markets – A collection of awesome Ruby libraries, tools, frameworks and software
• Awesome Ruby – by @Sdogruyol – A curated list of awesome Ruby frameworks, libraries and resources
• Ruby Development Bookmarks – A list of ruby development resources I’ve collected

Scala

• Awesome Scala – A curated list of awesome Scala frameworks, libraries and software

Frameworks, platforms, etc

• Frontend
• Node.js
• Ruby on Rails
• Mobile

Frontend

• Frontdesk – Community driven list of useful things for Front End Developers
• Frontend Development – A huge list of frontend development resources collected over time. Sorted from general knowledge at the top to concrete problems at the bottom.
• Master List of HTML5, JavaScript, and CSS Resources – Intent of sharing a list of HTML5, JavaScript, and CSS3 resources that were found very useful

Node.js

• Awesome Node.js – A curated list of astonishing Node.js frameworks, libraries and resources

Ruby on Rails

• Awesome Rails – A curated list of amazingly awesome open source rails related resources

Mobile

• Android
• iOS

Related and useful

• Editors
• Environments
• Shell
• SysAdmin
• Talks
• MachineLearning
• Awesomes

Editors

Environments

• Awesome Dev Env – A curated list of awesome tools, resources and workflow tips making an awesome development environment.

Shell

• Awesome Shell – A curated list of awesome command-line frameworks, toolkits, guides and gizmos

SysAdmin

• Awesome Sysadmin – A curated list of amazingly awesome open source sysadmin resources

Talks

• Awesome Talks – List of online talks that you would love to watch

MachineLearning

• Awesome Machine Learning – A curated list of awesome machine learning frameworks, libraries and software (by language).

Awesomes

• Awesome Awesomes – This one!! ;) Awesome collection of awesome lists of libraries, tools, frameworks and software for any programming language :D
• Awesome Awesomeness – A curated list of awesome awesomeness
• Awesome Awesome – A curated list of awesome curated lists! Inspired by inspiration

Awesome Awesomeness

A curated list of amazingly awesome awesomeness. Also available on:
Awesome-Awesomeness.ZEEF.com: https://awesome-awesomeness.zeef.com/alexander.bayandin
And Github: https://github.com/bayandin/awesome-awesomeness

• Programming Languages
  • AutoHotkey
  • Assembler
  • C
  • C/C++
  • Clojure
  • ColdFusion
  • Common Lisp
  • D
  • Delphi
  • Elixir
  • Erlang
    • by @0xAX
    • by @drobakowski
    • by @unbalancedparentheses
  • Fortran
  • Go
  • Groovy
  • Haskell
  • Java
  • JavaScript
    • Node.js
      • by @sindresorhus
      • by @vndmtrx
  • Julia
  • Lua
    • by @forhappy
    • by @lewisjellis
  • .NET
  • OCaml
  • Perl
  • PHP
  • Python
    • by @kirang89
    • by @svaksha
    • by @vinta
  • R
  • Ruby
    • by @dreikanter
    • by @markets
    • by @Sdogruyol
  • Rust
  • Scala
  • Shell
  • Swift
    • by @matteocrippa
    • by @Wolg
    • from ZEEF by @Edubits
• General
  • Algorithms
    • Algorithms Visualisation
  • Analytics
  • Android
    • Android Security
    • Android UI
  • Apple
  • Big data
    • by @onurakpolat
    • by @zenkay
    • Hadoop
  • Books
    • Free Programming Books
    • Free Software Testing Books
  • Bootstrap
  • Conferences
  • Continuous Delivery
  • Cordova
  • Courses
  • Creative Commons Media
  • Data Science
  • Database
  • Datasets
  • Dev Env
  • Docker
  • Documentation
  • Dotfiles
  • Emacs
  • Frontend Dev
  • Game Development
  • GIF
  • GitHub
  • HTML5
  • Images
  • iOS
  • Koans
  • Lists
    • by @bayandin
    • by @jnv
    • by @sindresorhus
  • Machine Learning
  • Nginx
  • Open Science
  • Papers
  • Product Manager
  • Remote Job
  • Remote Work
  • Services Engineering
  • Sheet Music
  • Speech and Natural Language Processing
  • Styleguides
  • Sublime Text
  • SVG
  • Sysadmin
  • Talks
  • Tools
  • Vagrant
  • Vim
    • by @akrawchyk
    • by @matteocrippa
  • Web Performance Optimization
  • WebComponents

Awesome Awesome

A curated list of awesome curated lists of many topics, can also found on:
Github: https://github.com/emijrp/awesome-awesome

• Awesome Awesome
  • Computer management
  • Data processing
  • Programming languages
  • Sciences
  • Web browsers
  • Websites
  • Web Platforms
  • Other
• Contributing

Computer management

• awesome-shell – Command-line frameworks, toolkits, guides and gizmos.
• awesome-sysadmin – Backups, configuration management, DNS, IMAP/POP3, LDAP, monitoring, SSH, statistics, troubleshooting, virtualization, VPN and more.

Data processing

• awesome-bigdata – Big data frameworks, resources and other awesomeness.
• awesome-hadoop – Hadoop ecosystem resources.

Programming languages

• awesome-clojure – Package management, audio, HTTP, database, websocket and testing.
• awesome-c – C frameworks, libraries, resources and other cool stuff.
• awesome-cpp – C/C++ frameworks, libraries, and resources.
• awesome-cobol – Web frameworks, template engine, forms, authentication & OAuth, database, e-mail, messaging, imagery, text processing, machine learning, testing, audio, video and logging.
• awesome-common-lisp – Common Lisp frameworks, libraries, resources and other shinies.
• awesome-d – Build tools, compilers, IDE, GUI, database clients.
• awesome-elixir – Elixir libraries, resources and shiny things.
• awesome-go – Go frameworks, libraries and software.
• awesome-java – Build tool, code analysis, database, GUI, IDE, JSON, machine learning, PDF, science, testing and web crawling.
• awesome-javascript – JavaScript libraries, resources and shiny things.
• awesome-julia – List of Julia resources and packages.
• awesome-perl – Benchmarks, databases, images, logging, profiling, testing, text processing and web frameworks.
• awesome-php – Frameworks, templating, URL, e-mail, files, imagery, testing, security, documentation, geolocation, date, PDF, search and authentication.
• awesome-python – Files, dates, text processing, NLP, imagery, audio, video, geolocation, web frameworks, OAuth, web crawling, networking, GUI, game development, testing, science and data analysis and machine learning.
• [awesome-R] – Not yet! Do it yourself!
• awesome-ruby – Ruby libraries, tools, frameworks and software
  • @markets
  • @Sdogruyol
• awesome-scala – Scala frameworks, libraries and software.
• awesome-swift – Swift documentation, projects, tutorials, updates, etc

Sciences

• [awesome-biology] – Not yet! Do it yourself!
• [awesome-chemistry] – Not yet! Do it yourself!
• [awesome-geography] – Not yet! Do it yourself!
• [awesome-math] – Not yet! Do it yourself!
• [awesome-physics] – Not yet! Do it yourself!

Web browsers

• [awesome-firefox] – Not yet! Do it yourself!

Websites

• [awesome-github] – Not yet! Do it yourself!
• [awesome-flickr] – Not yet! Do it yourself!
• [awesome-twitter] – Not yet! Do it yourself!
• awesome-wikipedia – Datasets, frameworks, libraries and other software related to Wikipedia.
• [awesome-youtube] – Not yet! Do it yourself!

Web platforms

• awesome-dojo – Dojo JavaScript Toolkit

Other

• [awesome-music] – Not yet! Do it yourself!

Awesome-Collection

a list of awesome repos
For more info check: https://github.com/flyhigher139/awesome-collection

awesome lists

• Awesome – A curated list of awesome lists
• awesome-all – A curated list of awesome lists of awesome frameworks, libraries and software
• awesome-awesome by @emijrp – A curated list of awesome curated lists of many topics.
• awesome-awesome by @erichs – A curated list of awesome curated lists! Inspired by inspiration.
• awesome-awesome by @oyvindrobertsen – A curated list of curated lists of libraries, resources and shiny things for various languages.
• awesome-awesomeness – A curated list of awesome awesomeness
• awesome-awesomes – Awesome collection of awesome lists of libraries, tools, frameworks and software for any programming language
• lists – The definitive list of (awesome) lists curated on GitHub. (comment: No awesome, but more awesome)

Programming languages

• .NET
• AutoHotkey
• C/C++
• Clojure
• Coldfusion
• D
• Dart
• Delphi
• Elixir
• Erlang
• Fortran
• GitHub-repo
• Go
• Groovy
• Haskell
• Java
• JavaScript
• JavaScript
  • AngularJS
  • Node.js
    • by @sindresorhus
    • by @vndmtrx
• Julia
• Lua
  • by @forhappy
  • by @lewisjellis
• OCaml
• Perl
• Python
  • by @kirang89
  • by @svaksha
  • Django
  • flask
• PHP
  • CakePHP
• R
• Ruby
  • by @dreikanter
  • by @markets
  • by @Sdogruyol
• Rust
• Scala
• Shell
• Swift
  • by @Wolg
  • from ZEEF by @Edubits

General

• Algorithms
• Analytics
• Books
  • Free Programming Books
  • free-programming-books-zh_CN
  • Free Software Testing Books
  • Papers We Love
• Conferences
• Cordova
• Dev Env
• Data
  • Big data
    • by @onurakpolat
    • by @zenkay
    • Hadoop
  • Data Science
  • Database
  • awesome-firebase
• Dotfiles
• Editor
  • Emacs
  • Sublime Text
  • Vim
    • by @akrawchyk
    • by @matteocrippa
• Game Development
• games
• GitHub Cheat Sheet
• Git Cheat Sheet
• Images
• Machine Learning
• Platforms
  • Android
    • Android Security
  • Apple
  • iOS
• resources
• Services Engineering
• Styleguides
• Sysadmin
• Talks
• Tools
• Web development
  • Bootstrap
  • Frontend Dev
  • HTML5
  • Sass
  • React

Lists

The definitive list of (awesome) lists curated on GitHub.
For more info check: https://github.com/jnv/lists
List of useful, silly and awesome lists curated on GitHub. Contributions welcome!

• Non-technical
• Technical
  • awesome-*
• Lists of lists
  • Lists of lists of lists
    • Lists of lists of lists of lists
      • Lists of lists of lists of lists of lists :scream:

Non-technical

• 50-DI – Curated Diversity Index of grants, scholarships and FA that encourages diversity in STEM fields aimed at half the world’s population.
  • http://svaksha.github.io/50-DI/
• Annual-Reading-List – A list of things to read every year.
• awesomebandnames – The open-source list of awesome band names.
• awesome-biology – Learning resources, research papers, tools and other resources related to Biology.
• awesome-gif – GIF /dʒ/ links and resources.
• awesome-glasgow – Some highlights around Glasgow, Scotland.
• awesome-health – Useful health resources.
• awesome-images – Free (stock) photo resources for your projects.
• awesome-kimchi – Kimchi of the people, by the people, for the people.
• awesome-webcomics
• baby-sleep – Baby sleep guides curated from the best of the Internet.
• Biology – Bibliography of Biology resources.
• cocktails – Cocktail Recipes
• corporate-logos – Curated repo of publicly listed co. logos, identified by ticker. Almost 1500 logos
• creative-commons-media – Audio, graphics and other resources that provide media licensed under Creative Commons licenses.
• dissertation-tips – Resources to help PhD students complete their dissertation successfully.
• food-recipes – Honest-to-goodness “real food” recipes
• lawrence-veggie – List of vegetarian/vegan restaurants in Lawrence, KS.
• mining-resources – Natural resources mining.
• recipes by @bzimmerman – This repository contains tasty open-source recipes.
• recipes by @csswizardy – Collection of things I like cooking
• recipes by @csclug – Delicious open source
• shelfies – Bookshelves of awesome people, community-transcribed.
• SiliconValleyThingsToDo – Things to do and activities within Silicon Valley.
• soylent – Soylent recipes
• sf-vegetarian-restaurants – A list of awesome vegetarian-friendly restaurants in SF
• teesites – A public list of great sites to buy awesome t-shirts and other cool stuff.
• tacofancy – community-driven taco repo. stars stars stars.
• visual-illusions – List of Visual Illusions.

Technical

• 101 – Resources on running a software biz.
• above-the-fold-css-tools – Tools to help prioritize above-the-fold CSS.
• admesh-projects – Projects using ADMesh (a triangulated solid meshes processor).
• algovis – Algorithm Visualization.
• alternative-internet – A collection of interesting new networks and tech aiming at decentralisation (in some form).
• android-awesome-libraries – Useful Android development libraries with usage examples.
• AndroidDevTools In Chinese – SDK, development tools, libraries, and resources.
  • http://www.androiddevtools.cn/
• android-jobs In Chinese – Android positions in China.
• android-open-project In Chinese – Collect and classify android open source projects.
• android-security-awesome – “A lot of work is happening in academia and industry on tools to perform dynamic analysis, static analysis and reverse engineering of android apps.”
• angular-education – Useful resources to learn and improve your AngularJS skills.
• angularjs-fellows – Free resources for learning AngularJS.
  • http://tachun.github.io/angularjs-fellows/
• ansible-gentoo-roles – Ansible roles for Gentoo Linux.
• apiPlatforms – Cool and awesome APIs
• APTnotes – Various public documents, whitepapers and articles about APT [Advanced persistent threat] campaigns.
• Automated-SPA-Testing – Automated unit & functional testing for web applications [JavaScript et al.].
• awesomelinks – Web Development Links by @richardbenjamin.
• Backpack – Various learning resources, organized by technology/topic.
• badass-dev-resources – #bada55 front-end developer resources.
• beautiful-docs – Pointers to useful, well-written, and otherwise beautiful documentation.
• best-of-awesomeness-and-usefuless-for-webdev – Digest of the most useful tools and resources for the last year.
  • Russian version
• Best-App In Chinese – Recommendations for best desktop and mobile apps.
• bigdata-ecosystem – Big-data related projects packed into a JSON dataset.
  • http://bigdata.andreamostosi.name/
• bioinformatics-tool-list – Bioinformatics and computational biology tools.
• bookmarklets – Bookmarklets that are useful on the web
  • http://sachaschmid.ch/bookmarklets/
• bookshelf – Reading lists for learners
• channels In Russian – YouTube channels for web developers.
• ciandcd – Continuous Integration and Continuous Delivery
  • http://www.ciandcd.com/
• content-management-systems – Open source & proprietary content management systems.
• Continuous-Integration-services – Comparison of cloud based CI services.
• crypto-might-not-suck – Crypto Projects that Might not Suck.
• css-resources
• curated-list-espresso-sugar-plugins – Sugar plugins for Espresso, the code editor by MacRabbit.
• Curated-List-of-Awesomeness – Web resources.
• cyberDigest In Russian – Digest of knowledge about crypto networks (including cryptocurrencies).
• daemon – A curated list of resources for a more literate programming.
• datajournalists-toolbox – Tools for datajournalists, with examples and gists.
• datasciencemasters – The Curriculum for learning Data Science, Open Source and at your fingertips.
  • http://datasciencemasters.org/
• Developer-Conferences – List of upcoming developer conferences.
• devopsbookmarks.com – To discover tools in the devops landscape.
  • http://www.devopsbookmarks.com/
• devops_resources
• dev-resource – Resources for devs online and offline.
• EducationalToolsResources – Tools and resources for educators and virtual teachers.
• ember-links/list – Ember.js Links
• erlang-bookmarks – All about erlang programming language.
• erlang-watchlist – Where to find good code to master Erlang idioms
• es6-tools – An aggregation of tooling for ES6
• Facets – One-liners in Ruby
• fks In Chinese – Frontend Knowledge Structure.
• flat-file-cms – Stictly flat-file cms systems.
• free-programming-books
  • http://resrc.io/list/10/list-of-free-programming-books/
• free-programming-books-zh_CN In Chinese
• free-programming-courses – Online programming/CS courses.
• free-software-testing-books – List of free software testing books.
• frontdesk – Community driven list of useful things for Front End Developers
• Front-end-Developer-Interview-Questions – A list of helpful front-end related questions you can use to interview potential candidates, test yourself or completely ignore.
  • Available in various translations
• Front-end-Web-Development-Interview-Question In Chinese
• front-end-links
• frontend-dev-bookmarks – A huge list of frontend development resources I collected over time.
• frontend-resources by @JonathanZWhite
• frontend-resources by @zedix
• frontend-stuff – Framework/libraries/tools to use when building things on the web. Mostly Javascript stuff.
• frontend-tools – An opinionated list of tools for frontend (i.e. html, js, css) desktop/laptop (i.e. does not include tablet or phone yet) web development
• games – A list of popular/awesome videos games, add-on, maps, etc. hosted on GitHub.
• github-cheat-sheet – Cool features of Git and GitHub.
• github-hall-of-fame – Hall of Fame for spectacular things on Github.
  • http://mehulkar.com/github-hall-of-fame/
• hackathon_resources by @myearwood – Learning Resources for Hackathon Organizers.
• Hackathon-Resources by @xasos – Hackathon Resources for organizers.
• HeadlessBrowsers
• ideas – One Hundred Ideas for Computing
• Inspire – A collection of links to assist you in web design and development.
• InterviewThis – Developer questions to ask prospective employers
• ios-awesome-libraries – Useful iOS development libraries with usage examples.
• iOSDevResource
• iOS-Developer-and-Designer-Interview-Questions
• javascript-patterns – JavaScript Patterns
  • http://shichuan.github.io/javascript-patterns/
• javascript-resources
• jslibs – My picks of promising/useful Javascript libraries.
  • See also JSwiki
• js-must-watch – Must-watch videos about javascript.
• jsonauts – The ultimate reference for JSON tooling and specs.
  • http://jsonauts.github.io/
• jstools – Foundational JavaScript Tools
• js-type-master – JavaScript resources about web typography.
  • https://www.codefellows.org/blog/complete-list-of-javascript-tools
• Julia.jl – Curated decibans of Julia language.
  • https://github.com/svaksha/Julia.jl
• killer-talks – Talks that are worth watching.
• Laravel-Resources – Laravel Framework Resources and Blogs.
• learn-drupal – Stuff to help you learn Drupal.
• learn-python – A list of links that teach Python.
• learnxinyminutes-docs – Code documentation written as code!
  • http://learnxinyminutes.com/
• lifeofjs – Curated source for all types of awesome resources available for JavaScript.
• ListOfMinimalistFrameworks – Minimalist frameworks (simple and lightweight).
• magictools – Game Development resources to make magic happen.
• manong In Chinese – Weekly digest of technology
• markdown-resources – Markdown resources: apps, dialects, parsers, people, …
• MaterialDesignCenter In Chinese – Resources and libraries for Material Design.
• my_tech_resources by @JamesLavin
• must-watch-css – Must-watch videos about CSS.
• nashville-lispers/resources – Lisp Resources: exercises, great books, videos, etc.
• nginx-resources – Nginx web server (+ Lua), OpenResty and Tengine.
• node-daily In Chinese – Daily article about Node.js.
• node-frameworks – Comparison of server-side Node frameworks.
• nodejs-conference-cfps – NodeJS and Javascript Conference Call for Presentations.
• NodeJS-Learning – Resources to help you learn Node.js and keep up to date.
• npm-algos – Node.js Modules Implementing Computer Science Concepts.
• offline-first – Everything you need to know to create offline-first web apps.
• papers – A collection of papers found across the web.
• papers-we-love – Papers from the computer science community to read and discuss. (Contains actual papers)
• pattern_classification – A collection of tutorials and examples for solving and understanding machine learning and pattern classification tasks.
• PHPVietnam – PHP resources for Vietnamese.
• PlacesToPostYourStartup – “Where can I post my startup to get beta users?”
• programmers-proverbs – Proverbs from the programmer
• programming-talks – Awesome & Interesting Talks concerning Programming
• Projects – Practical projects that anyone can solve in any programming language.
• pycrumbs – Bits and Bytes of Python from the Internet.
• python-github-projects – Collect and classify python projects on Github.
  • http://itgeekworkhard.com/python-github-projects/
• pythonidae – Curated decibans of Python scientific programming resources.
  • http://svaksha.github.io/pythonidae/
• python_reference – Useful functions, tutorials, and other Python-related things.
• Qix In Chinese – Node, Golang, Machine Learning, PostgreSQL.
• queues.io – Job queues, message queues and other queues.
  • http://queues.io/
• resource-list – Design & Development Resources.
• resources – JavaScript resources for and by Hack Reactor 14 & 15.
• resources – Free, online resources for various technologies, languages, and tools.
• Resources – Tools for front end devs.
• ruby-bookmarks – Ruby and Ruby on Rails bookmarks collection.
• ruby-dev-bookmarks – Ruby development resources I’ve collected.
• ruby-trivia – Questions about Ruby programming you can use to quiz yourself.
• rust-lang-resources – Links related to the Rust programming language.
• rss-readers-list – Reader replacements megalist
  • http://smithbr.github.io/rss-readers-list
• scalable-css-reading-list – Collected dispatches from The Quest for Scalable CSS.
• secure-email – Overview of projects working on next-generation secure email.
• services-engineering – A reading list for services engineering, with a focus on cloud infrastructure services.
• shellshocker-pocs – Proof of concepts and potential targets for Shellshock.
• spawnedshelter – Erlang Spawned Shelter – the best articles, videos and presentations related to Erlang.
• speech-language-processing
• StartupKnowledgeBase – Insightful articles related to startups.
• startupreadings – Reading list for all things startup-related.
• startup-resources
• static-site-generator-comparison
  • http://staticsitegenerators.net/
• Static-Site-Generators
• streaming-papers – Papers on streaming algorithms.
• sublime – Some of the best Sublime Text packages, themes, and goodies.
• sublime-bookmarks – Sublime Text essential plugins and resources.
• SwiftInFlux – An attempt to gather all that is in flux in Swift.
• tech-weekly – List of weekly technical newsletters
• Toolbox – Open source iOS stuff.
• tool_lists – Links to tools by theme. Verification, synthesis, and static analysis.
• tools – Tools for web.
• ToolsOfTheTrade – Tools of The Trade, from Hacker News.
• twofactorauth – Sites with two factor auth support which includes SMS, email, phone calls, hardware, and software.
  • http://twofactorauth.org/
• type-findings – Posts about web typography.
• WebComponents-Polymer-Resources
• webcomponents-the-right-way – Introduction to Web Components.
• webdevresourcecuration
• web-dev-resources – A table of contents for web developer resources across the internet.
• web-development-resources
• weekly In Chinese – Weekly summary of articles and resources.

awesome-*

• awesome – Useful plugins and libraries.
• awesome-2048-and-beyond – Waste and lose at least 8 hours of your life… then multiply it…
• awesome-acf – Add-ons for the Advanced Custom Field plugin for WordPress.
• awesome-activeadmin – Active Admin resources, extensions, posts and utilities. For Rails.
• awesome-algorithms – Places to learn and/or practice algorithms.
• awesome-analytics – Analytics services, frameworks, software and other tools.
• awesome-android by @Jackgris In Spanish.
• awesome-android by @JStumpp
• awesome-android by @snowdream Partially in Chinese
• awesome-android-libraries – General Android libraries.
• awesome-android-ui – UI/UX libraries for Android.
• awesome-angular by @hugoleodev
• awesome-angularjs by @gianarb
• awesome-apple – 3rd party libraries and tools for Apple platforms development.
• awesome-arabic – Arabic supporting tools, fonts, and development resources.
• awesome-asciidoctor – Collection of asciidoctor’s intros, exampes and usages.
• awesome-asm – Assembly language
• awesome-AutoHotkey – AutoHotkey libraries, library distributions, scripts, tools and resources.
• awesome-avr
• awesome-backbone – Resources for Backbone.js
• awesome-bem – Tools, sites, articles about BEM (frontend development method).
• awesome-bigdata – Big data frameworks, resources and other awesomeness.
• Awesome-Bioinformatics – Open-source bioinformatics software and libraries.
• awesome-bootstrap – Free Bootstrap themes I think are cool.
• awesome-c – C frameworks, libraries, resources etc.
• awesome-cakephp – Plugins and resources for CakePHP 2 and 3.
• awesome-cassandra
• awesome-chef – Cookbooks, handlers, add-ons and other resources for Chef, a configuration management tool.
• awesome-ci – Continuous integation services.
• awesome-cl – Common Lisp
• awesome-clojure
• awesome-cloud – Delightful cloud services.
• awesome-cloud-storage – Comparison of online storage platforms.
• awesome-code-formatters
• awesome-coldfusion
• awesome-comparisons – Framework and code comparison projects, like TodoMVC and Notejam.
• awesome-conference-playlists – Video playlists for conferences.
• awesome-conferences
• awesome-computer-vision
• awesome-cordova Apache Cordova / PhoneGap
• awesome-courses – University Computer Science courses across the web.
• awesome-cpp – C/C++ frameworks, libraries, resources.
• awesome-css by @deepakbhattarai
• awesome-css by @ikkou
• awesome-d
• awesome-dart
• awesome-dashboard – Dashboards/visualization resources.
• awesome-datascience – An open source DataScience repository to learn and apply for real world problems.
• awesome-datasets – Datasets for papers/experiments/validation.
• awesome-db – Databse libraries and resources.
• awesome-ddd – Domain Driven Design resources.
• awesome-delphi
• awesome-derby – Components for DerbyJS.
• awesome-design – Resources for digital designers.
• awesome-devenv – Tools, resources and workflow tips making an awesome development environment.
• awesome-devtools – In-browser bookmarklets, tools, and resources for front-end devs.
• awesome-django – Django apps and projects.
• awesome-django-cms – django CMS add-ons.
• awesome-docker by @anildigital
• awesome-docker by @veggiemonk
• awesome-doctrine – Doctrine ORM libraries and resources.
• awesome-dojo – Dojo JavaScript Toolkit resources and libraries.
• awesome-dotfiles
• awesome-dotnet – .NET libraries, tools, frameworks and software.
• awesome-drupal
• awesome-educate – Education resources online.
• awesome-elm – Learning resources and programs for Elm, a functional reactive language.
• awesome-emacs from @emacs-tw
• awesome-emacs by @sefakilic
• awesome-emacs by @tacticiankerala
• awesome-ember – Ember.js things.
• awesome-elixir
• awesome-endless-codeforall-list – Every tool that civic hackers worldwide use to work.
• awesome-erlang
• awesome-flask – Flask Python web framework resources and plugins.
• awesome-fortran
• awesome-github by @fffaraz – Git and GitHub references.
• awesome-github by @Kikobeats – GitHub secrets and goodies.
• awesome-github-repo – GitHub repositories; various topics like study materials, Raspberry Pi etc.
• awesome-go by @avelino
  • http://awesome-go.com/
• awesome-go-books – Online and free golang books.
• awesome-grails
• awesome-groovy
• awesome-hadoop – Hadoop and Hadoop ecosystem resources.
• awesome-haskell
• awesome-hierarchical-matrices – Hierarchical matrices frameworks, libraries, and software.
• awesome-html5
• awesome-hybrid – Hybrid Applications (web + native)
• awesome-ios
• awesome-IoT – Internet of Things platforms, libraries and apps.
• awesome-java
• awesome-javascript
• awesome-javascript-books – Online and free JavaScript books.
• awesome-julia
• awesome-knockout – Plugins for Knockout MVVM framework.
• awesome-koans – Programming kōans in various languages.
• awesome-languages – Open-source programming languages.
• awesome-laravel by @chiraggude
• awesome-laravel by @TimothyDJones
• awesome-linguistics – Tools, theory and platforms for linguistics.
• awesome-linux – Linux software.
  • http://dhamaniasad.github.io/awesome-linux
• awesome-logging
• awesome-lua by @forhappy
• awesome-lua by @LewisJEllis
• awesome-luvit – Luvit frameworks, libraries and software.
• awesome-machine-learning
• awesome-meteor
• awesome-mesos by @dharmeshkakadia
• awesome-mesos by @parolkar
• awesome-nodejs by @sindresorhus
• awesome-nodejs by @vndmtrx
• awesome-objc-swift
• awesome-ocaml
• awesome-open-science
• awesome-pcaptools – Tools to process network traces.
• awesome-pentest – Penetration testing resources and tools.
• awesome-perfocards See perfokaart.
• awesome-perl
• awesome-persian – Persian/Farsi supporting tools, fonts, and development resources.
• awesome-php
• awesome-pipeline – Pipeline toolkits.
• awesome-play1 – Play Framework 1.x modules, tools, and resources.
• awesome-ProductManager – Books and tools for Product Managers.
• awesome-puppet
• awesome-pyramid – Resources for Pyramid Python web framework.
• awesome-python by @kevmo
• awesome-python by @vinta
• awesome-python-talks – An opinionated list of awesome videos related to Python, with a focus on training and gaining hands-on experience.
• awesome-public-datasets – (Large-scale) public datasets on the Internet.
• awesome-qt – Libraries and resources for the Qt framework.
• awesome-R
• awesome-radio – Radio and citizens band (CB) radio resources.
• awesome-rails by @dpaluy
• awesome-rails by @ekremkaraca – Projects and sites made with Rails.
• awesome-rails by @ruby-vietnam – Rails libraries/app examples/ebooks/tutorials/screencasts/magazines/news.
• awesome-react – ReactJS tools, resources, videos.
• awesome-recommender-system – Recommender System frameworks, libraries and software.
• awesome-RemoteWork – Books and links about and for remote work.
• awesome-ruby by @markets
  • http://marcanguera.net/awesome-ruby/
• awesome-ruby by @Sdogruyol
• awesome-rust
• awesome-sarl – Resources for SARL Agent-Oriented Programming Language.
• awesome-sass – Sass/SCSS extensions, articles, and examples.
• awesome-scala
• awesome-security – Software, libraries, documents, books, resources and cool stuff about security.
• awesome-selenium
• awesome-services – Services that make a painful programmer’s life easier.
• awesome-sharepoint by @sergey-tihon
• awesome-SharePoint by @siaf
• awesome-shell – Command-line frameworks, toolkits, guides and gizmos.
• awesome-sheet-music – Tools to create, edit and display sheet music.
• awesome-sketch – Guides, articles, videos about Sketch 3.
• awesome-sphinxdoc – Tools for Sphinx Python Documentation Generator.
• awesome-sqlalchemy – Extra libraries for SQLAlchemy, a Python ORM.
• awesome-static-generators – Static web site generators.
• awesome-stars In Chinese – Useful libraries with personal remarks.
• awesome-styleguides
• awesome-svg
• awesome-swift by @matteocrippa
• awesome-swift by @Wolg
• awesome-symfony2 – Symfony 2 bundles, utilities and resources.
• awesome-sysadmin – Open source sysadmin resources.
• awesome-talks
• awesome-umbraco – Resources for Umbraco 7, a .NET CMS.
• awesome-unity – Assets and resources for Unity game engine.
• awesome-vagrant
• awesome-vim by @akrawchyk
• awesome-vim by @matteocrippa
• awesome-webservice – Web and cloud services, SaaS.
• awesome-weekly – Quality weekly subscription newsletters from the software world.
• awesome-wikipedia – Wikipedia-related frameworks, libraries, software, datasets and references.
• awesome-wordpress from @dropndot
• awesome-wordpress by @endel
• awesome-wordpress by @miziomon
• awesome-wpo – Web Performance Optimization
• awesome-yamada Dancing Yamada
• awesome-yii – Yii Framework extensions, tutorials and other nice things.
• awesome-yii-1 – Libraries, books, sites and team of Yii Framework v1.x.
• awesome-zsh-plugins

Lists of lists

• awesome – A curated list of awesome lists.
• awesome-all – A curated list of awesome lists of awesome frameworks, libraries and software
• awesome-awesome by @emijrp – A curated list of awesome curated lists of many topics.
• awesome-awesome by @erichs – A curated list of awesome curated lists! Inspired by inspiration.
• awesome-awesome by @oyvindrobertsen – A curated list of curated lists of libraries, resources and shiny things for various languages.
• awesome-awesomeness – A curated list of awesome awesomeness
• awesome-awesomes – Awesome collection of awesome lists of libraries, tools, frameworks and software for any programming language
• awesome-collection – A list of awesome repos.
• ListOfGithubLists – List of github lists
• list-of-lists – A meta list of lists of useful open source projects and developer tools.
• must-watch-list – List of must-watch lists.
• this one
• wiki In Chinese – A curated list of awesome lists.

Lists of lists of lists

• awesome-awesome-awesome – A a curated list of curated lists of awesome lists.
• awesomecubed – A curated list of awesome awesomeness awesomenesses.
• lologl – List of Lists of Github Lists.
• meta-awesome
• the one above

Lists of lists of lists of lists

• awesome-awesome-awesome-awesome
• the one above

Lists of lists of lists of lists of lists

• awesome-power-of-5
• the one above

List of github lists

Creating a github list is so trendy nowadays, so here’s another one.
Fore more info check: https://github.com/asciimoo/ListOfGithubLists

Pull requests are welcome

Lists

• A collection of awesome Ruby libraries, tools, frameworks and software
• A collection of awesome open-source languages. Inspired by other awesome stuffs
• A collection of awesome projects, sites made with Rails
• A curated list of amazingly awesome PHP libraries, resources and shiny things
• A curated list of amazingly awesome open source sysadmin resources inspired by Awesome PHP
• A curated list of awesome Artificial Intelligence (AI) courses, books, video lectures and papers
• A curated list of awesome C/C++ frameworks, libraries, resources, and shiny things. Inspired by awesome-… stuff
• A curated list of awesome Clojure libraries and software
• A curated list of awesome Django apps. Inspired by and based on awesome-python
• A curated list of awesome GitHub tools, libraries, resources, and shiny things
• A curated list of awesome Go frameworks, libraries and software. Inspired by awesome-python
• A curated list of awesome Machine Learning frameworks, libraries and software
• A curated list of awesome Python frameworks, libraries and software
• A curated list of awesome Scala frameworks, libraries and software
• A curated list of awesome Swift frameworks, libraries and software. Inspired by awesome-php
• A curated list of awesome big data frameworks, ressources and other awesomeness
• A curated list of awesome command-line frameworks, toolkits, guides and gizmos. Inspired by awesome-php
• A curated list of awesome curated lists! Inspired by inspiration
• A curated list of awesome extra libraries for SQLAlchemy
• A curated list of awesome tools, resources and workflow tips making an awesome development environment
• A curated list of curated lists of awesome lists
• A curated list of delightful Node.js packages and resources
• A curated list of quality Lua packages and resources
• A definitive list of tools for generating static websites
• Collection and classification of Python open projects on GitHub
• Collection of interesting new networks and tech aiming at decentralisation (in some form)
• Collection of useful .gitignore templates
• Collection of useful scripts, tutorials, and other Python-related things
• Frontend development bookmarks
• HN Tools of The Trade, 2014 Edition
• HackerNews Bookmark
• Landscaping With Front-end Tools
• List of DevOps resources – podcasts, blogs, newsletters, etc
• List of Dirty, Naughty, Obscene, and Otherwise Bad Words
• List of Free Learning Resources
• List of Game Development resources to make magic happen
• List of Lists of Github Lists
• List of US states
• List of cool features of Git and GitHub
• List of countries
• List of crypto projects that might not suck
• List of github lists
• List of helpful front-end related questions you can use to interview potential candidates
• List of minimalist frameworks
• List of must-watch videos devoted to JavaScript
• List of popular/awesome videos games, add-on, maps, etc
• List of questions about Ruby programming
• List of ruby development resources
• List of useful things for front-end developers
• Master List of HTML5, JavaScript, and CSS Resources
• Mega Project List
• OSI Licenses
• One Hundred Ideas for Computing
• Papers from the computer science community to read and discusse
• Python Repositories Collection and Classification
• Rss-readers-list
• Ruby and Ruby on Rails bookmarks collection

List-Of-Lists

A meta list of lists of useful open source projects and developer tools
For more info check: https://github.com/cyrusstoller/list-of-lists

Tools

• Sysadmin tools: https://github.com/kahun/awesome-sysadmin
• Font-end Tools: https://github.com/codylindley/frontend-tools
• List of languages that compile to JS: https://github.com/jashkenas/coffeescript/wiki/List-of-languages-that-compile-to-JS
• Landscaping With Front-end Tools: https://github.com/codylindley/frontend-tools
• List of RSS readers: https://github.com/smithbr/rss-readers-list
• List of Job queues, message queues and other queues – http://queues.io/
• List of Tools of the Trade – https://github.com/cjbarber/ToolsOfTheTrade

Frameworks / Libraries

• Front-End CSS Frameworks: http://usablica.github.io/front-end-frameworks/compare.html
• Static Site Generators: http://staticsitegenerators.net
• List of minimalist frameworks: https://github.com/neiesc/ListOfMinimalistFrameworks
• List of useful php libraries: https://github.com/ziadoz/awesome-php
• List of python Github projects: https://github.com/checkcheckzz/python-github-projects
• List of Go projects: https://code.google.com/p/go-wiki/wiki/Projects
• Database of JS Librariers, Frameworks, and Plugins – http://www.jsdb.io/
• Database of Node.js packages – https://github.com/sindresorhus/awesome-nodejs

Resources

• A list of free programming books: https://github.com/vhf/free-programming-books/blob/master/free-programming-books.md
• A list of front end development resources: https://gist.github.com/dypsilon/5819504
• Places To Post Your Startup: https://github.com/mmccaff/PlacesToPostYourStartup
• Frontend development bookmarks: https://github.com/dypsilon/frontend-dev-bookmarks
• HackerNews Bookmark: https://github.com/praveenhm/HackerNews-Bookmark
• Front End Developer Interview Questions: https://github.com/darcyclarke/Front-end-Developer-Interview-Questions
• List of useful things for front-end developers: https://github.com/miripiruni/frontdesk
• List of CS papers: https://github.com/papers-we-love/papers-we-love
• Must-watch videos about javascript – https://github.com/bolshchikov/js-must-watch
• List of Game Development resources – https://github.com/ellisonleao/magictools
• List of Web Business Models – https://gist.github.com/ndarville/4295324
• Big list of d3.js examples – http://christopheviau.com/d3list/
• 319 useful Twitter Bootstrap resources – http://www.bootstraphero.com/the-big-badass-list-of-twitter-bootstrap-resources
• A curated list of amazingly awesome awesomeness – https://github.com/bayandin/awesome-awesomeness
• Markdown resources – https://github.com/rhythmus/markdown-resources
• List of React tools, resources, videos and shiny things – https://github.com/enaqx/awesome-react
• List of awesome CS courses – https://github.com/prakhar1989/awesome-courses

Other lists of lists

• https://github.com/asciimoo/ListOfGithubLists/
• https://github.com/jnv/lists
• https://github.com/leereilly/games

Misc

• https://github.com/phiat/markdown-lists/blob/master/tools.md

must-watch-list

A list of must-watch lists
For more info check: https://github.com/adrianmoisey/must-watch-list

• CSS
• Django
• Go
• Javascript
  • by AllThingsSmitty
  • by bolshchikov
• Python
  • by primalpop
  • by s16h
• Rails
• Startup

Overview of all lists from this post:
Awesome: https://github.com/sindresorhus/awesome
Awesome All: https://github.com/bradoyler/awesome-all
Awesome-Awesome: https://github.com/erichs/awesome-awesome
Awesome Awesome: https://github.com/oyvindrobertsen/awesome-awesome
Awesome-Awesomes: https://github.com/fleveque/awesome-awesomes
Awesome-Awesomeness: https://github.com/bayandin/awesome-awesomeness
Awesome Awesome: https://github.com/emijrp/awesome-awesome
Awesome-Collection: https://github.com/flyhigher139/awesome-collection
Lists: https://github.com/jnv/lists
List Of Github Lists: https://github.com/asciimoo/ListOfGithubLists
List-Of-Lists: https://github.com/cyrusstoller/list-of-lists
Must-Watch-List: https://github.com/adrianmoisey/must-watch-list
Wiki China Lists: https://github.com/huguangju/wiki

Awesome-Awesomeness (zeef): https://awesome-awesomeness.zeef.com/alexander.bayandin

Debugging and Detecting problems with SH, Bash or the Shell

Spellcheck, paste the shell script with errors or problems on the spellcheck website and it automatically detects the problem.

Go to Spellcheck by visiting this site:
http://www.shellcheck.net

Or you can watch the source code on github at:
https://github.com/koalaman/shellcheck

Bash Pitfalls is an collection or errors that can show up in the Bash, complete with a description of the error.
It can be found on:
http://mywiki.wooledge.org/BashPitfalls

The Complete MagPi Series Collection

The MagPi is a Digital Magazine that is focusing it’s content on the Raspberry Pi.

All The Issues Can Be Downloaded From:
http://www.themagpi.com/issues

NTap: The Raspberry Pi Network Tap

NTap is a very simple configuration to make a Raspberry Pi act as a transparent network tap.

If you’re interested to verify whether one of your devices (being a laptop, router or else) is connecting to unknown destinations or it’s performing some unusual network activity (for example as a result of a compromise), you can use NTap to intercept and store transiting traffic and later inspect it.

You’ll just need a Raspberry Pi with a default Raspbian installation, a USB Ethernet adapter and two cables.

The needed files and more information can be found on:
https://github.com/botherder/ntap