Never Ending Security

It starts all here

Monthly Archives: June 2015

NISTFOIA: FOIA for NIST documents related to the design of Dual EC DRBG


Results of a recent FOIA for NIST documents related to the design of Dual EC DRBG.

These FOIA results are the combined result of two separate requests. Thanks to the following requestors:

  • Matthew Stoller and Rep. Alan Grayson
  • Andrew Crocker and Nate Cardozo of EFF

I have contributed only OCR and hosting. Happy hunting,

Matt Green, 6/5/2014

1.15.2015 production/9.1.2 Keyless Hash Function DRBG.pdf
1.15.2015 production/ANSI X9.82 Discussions.pdf
1.15.2015 production/ANSI X9.82, Part 3 DRBGs Powers point July 20, 2004.pdf
1.15.2015 production/Appendix E_ DRBG Selection.pdf
1.15.2015 production/Comments on X9.82, Part 4_Constructions.pdf
1.15.2015 production/E1 Choosing a DRBG Algorithm.pdf
1.15.2015 production/Five DRBG Algorithms Kelsey, July 2004.pdf
1.15.2015 production/Hash Funciton chart.pdf
1.15.2015 production/Letter of transmittal 1.15.2015 .pdf
1.15.2015 production/Part 4_Constructions for Building and Validating RBG Mechanisms.pdf
1.15.2015 production/Scan_2015_01_27_13_05_55_026.pdf
1.15.2015 production/Validation Testing and NIST Statistical Test Suite July 22, 2004.pdf
1.22.2015 production/10.1.2 Hash function DRBG Using HMAC.pdf
1.22.2015 production/10.1.3 KHF_DRBG.pdf
1.22.2015 production/8.6.7 Nonce.pdf
1.22.2015 production/8.7 Prediction Resistance and Backtracking Resistance.pdf
1.22.2015 production/ANSI X9.82 Part 3 Draft July 2004.pdf
1.22.2015 production/Annex G_Informative DRBG mechanism Security Properties.pdf
1.22.2015 production/Appendix G Informative DRBG Selection.pdf
1.22.2015 production/Comments on X9.82 Part 1, Barker May 18, 2005.pdf
1.22.2015 production/Cryptographic security of Dual_EC_DRBG.pdf
1.22.2015 production/D.1 Choosing a DRBG Algorithm.pdf
1.22.2015 production/DRBG Issues Power Point July 20, 2004.pdf
1.22.2015 production/Draft X9.82 Part 3 Draft May 2005.pdf
1.22.2015 production/E.1 Choosing a DRBG Algorithm (2).pdf
1.22.2015 production/E.1 Choosing a DRBG Algorithm.pdf
1.22.2015 production/Final SP 800-90 Barker May 26, 2006.pdf
1.22.2015 production/Fwd_Final SP 800-90 Barker May 26, 2006.pdf
1.22.2015 production/Kelsey comments on SP April 12, 2006.pdf
1.22.2015 production/Latest SP 800-90 Barker May 5, 2006.pdf
1.22.2015 production/Letter of transmittal 1.22.2015.pdf
1.22.2015 production/SP 800-90 Barker June 28, 2006.pdf
1.22.2015 production/SP 800-90_pre-werb version> Barker May 9, 2006.pdf
1.22.2015 production/Terse Description of two new hash-based DRGBs Kelsey, January 2004.pdf
1.22.2015 production/Two New proposed DRBG Algorithms Kelsey January 2004.pdf
1.22.2015 production/X9.82, RGB, Issues for the Workshop.pdf
6.4.2014 production/001 – Dec 2005 -NIST Recomm Random No. Gen (Barker-Kelsey).pdf
6.4.2014 production/002 – Dec 2005 – NIST Recomm Random No. Gen (Barker-Kelsey)(2).pdf
6.4.2014 production/003 – Sept 2005 – NIST Recomm Random No. Gen (Barker-Kelsey).pdf
6.4.2014 production/004 – Jan 2004 – Terse Descr. of Two New Hash-Based DRBGs.pdf
6.4.2014 production/005 – Proposed Changes to X9.82 Pt. 3 (Slides).pdf
6.4.2014 production/006 – NIST Chart 1.pdf
6.4.2014 production/007 – RNG Standard (Under Dev. ANSI X9F1) – Barker.pdf
6.4.2014 production/008 – Random Bit Gen. Requirements.pdf
6.4.2014 production/009 – Seed File Use.pdf
6.4.2014 production/010 – NIST Chart 2.pdf
6.4.2014 production/011 – 9.12 Choosing a DRBG Algorithm.pdf
6.4.2014 production/012 – May 14 2005 – Comments on ASC X9.82 Pt. 1 – Barker.pdf
6.4.2014 production/013 – X9.82 Pt. 2 – Non-Deterministic Random Bit Generators.pdf

More info you can find on:




  1. Crosby, Goldberg, Johnson, Song, Wagner: Cryptanalyzing HDCP (2001)

  2. Wagner, Schneier: Analysis of the SSL 3.0 Protocol

  3. Lucks, Schuler, Tews, Weinmann, Wenzel: Security of DECT

  4. Kohno: Analysis of WinZip Encryption

  5. Stubblefield, Ioannidis, Rubin: Breaking WEP

  6. Bellare, Kohno, Namprempre: Breaking and Repairing SSH

  7. Burrows, Abadi and Needham: A Logic of Authentication

  8. DTLA: DTCP Additional Localization Protocol

Side Channel Attacks

  1. Bar-el: Introduction to Side Channel Attacks (white paper)

  2. Kocher: Timing attack on RSA & DL systems

  3. Brumley, Boneh: Remote Timing Attacks are Practical

  4. Bernstein: Cache Timing Attack on AES.  Osvik, Shamir, Tromer: Attacks and Countermeasures

  5. Eisenbarth, Kasper, Moradi, Paar, Salmasizadeh, Shalmani: Attacking KeeLoq (SpringerLink)

  6. Shamir, Tromer: Acoustic Cryptanalysis

  7. Pellegrini, Bertacco, Austin: Fault-Based Attack of RSA Authentication

  8. Aciicmez, Koc, Seifert: Branch Prediction Analysis (very advanced)

Dictionary Attacks: Optimization & Mitigation

  1. Alexander: Password Protection for Modern OSes

  2. RSA Laboratories: PKCS #5 2.0: Password-Based Cryptography Standard

  3. Provos and Mazières: “Future-adaptable” password schemes

  4. Stamp: Once Upon a Time Space Tradeoff

  5. Oeschslin: Rainbow Tables (includes papers & demo)

  6. Canetti, Halevi, Steiner: Mitigating (offline) Dictionary Attacks with Reverse-Turing Tests

    Securing Internet Infrastructure

  7. Jackson, Barth, Bortz, Shao, Boneh: Protecting Browsers from DNS Rebinding Attacks

  8. Kaminsky: It’s the End of the (DNS) Cache As We Know It (Black Hat 2008 – 101MB)

  9. DNS Security Extensions (standards & resources)

  10. Ptacek: A case against DNSSEC

  11. Kent, Lynn and Seo: Secure BGP

  12. Secure BGP resources

Digital Rights Management & Conditional Access

  1. Lawson: Designing and Attacking DRM (presentation)

  2. Edwards: A technical description of the Content Scrambling System (CSS)

  3. Henry, Sui, Zhong: Overview of AACS — and full AACS Specification

  4. ISE: A Comparison of SPDC (technology behind BD+) and AACS (2005)

  5. Craver, Wu, Liu, Stubblefield, Swartzlander, Wallach, Dean, Felten: Watermarking & SDMI

  6. Kuhn: Analysis of the Nagravision Video Scrambling Method (analog scrambling)

  7. Naor, Naor and Lotspiech: Revocation and Tracing Schemes for Stateless Receivers

Software, Physical Security, Backdoors

  1. Halderman et al.: Cold Boot Attacks on Encryption Keys & RSA Key Reconstruction

  2. Young, Yung: Cryptovirology: extortion-based security threats and countermeasures (IEEE)

  3. Dowd: Application-Specific Attacks: Leveraging the ActionScript Virtual Machine

  4. Steil: 17 Mistakes Microsoft Made in the XBox Security (2005)

  5. Bartolozzo et al.: Attacking and Fixing PKCS#11 Security Tokens

  6. Bardou et al.: Efficient Padding Oracle Attacks on Cryptographic Hardware

Privacy and Anonymity

  1. Dingledine, Mathewson, Syverson: Tor: The Second Generation Onion Router

  2. McCoy, Bauer, Grunwald, Kohno, Sicker: Analyzing Tor Usage

  3. Murdoch, Danezis: Low-cost Traffic Analysis of Tor

  4. Murdoch: Hot Or Not: Using clock skew to locate hidden services

  5. Wang, Chen, Jajodia: Tracking Anonymized VoIP Calls

Hash Functions and Random Oracles

  1. Coron, Dodis, Malinaud, Puniya: Merkle-Damgård Revisited

  2. Wang, Yu: How to break MD5 and other hash functions

  3. Stevens, Lenstra, de Weger: Target collisions for MD5

  4. Kaminsky: MD5 To Be Considered Harmful Someday

  5. Sotirov et al.: MD5 considered harmful today (building a rogue CA cert)

  6. Wang, Yin, Yu: SHA1 broken (at least, on its way…)

  7. NIST: “SHA3” competition: list of first round candidates (December 2008)

  8. Canetti, Goldreich, Halevi: Random oracles revisited, and…

  9. Bellare, Boldyreva, Palacio: A more natural uninstantiable Random-Oracle-Model scheme

  10. Coron, Patarin, Seurin: The random oracle model and the ideal cipher model are equivalent

  11. Bellare, Canetti, Krawczyk: HMAC

Symmetric Crypto

  1. Bellare, Namprempre: Authenticated encryption, generic composition

  2. Ferguson: Authentication weaknesses in GCM.  McGrew, Viega: Response & Update.

Public Key Crypto

Bleichenbacher: CCA Attacks against Protocols (SSL) based on PKCS #1

Bellare, Rogaway: Optimal Asymmetric Encryption Padding (OAEP)

Manger: CCA Attacks against Implementations of OAEP

Bernstein: An Introduction to Post-Quantum Cryptography

Random Number Generation

  1. Dorrendorf, Gutterman, Pinkas: RNG Weaknesses in Windows 2000

  2. Gutterman, Pinkas: Flaws in the Linux RNG

  3. Barker, Kelsey: NIST Special Pub. 800-90: Recommendations for PRNGs

  4. Kelsey, Schneier, Wagner, Hall: Cryptanalytic attacks on PRNGs

  5. Schoenmakers, Sidorenko: Dual EC not kosher

  6. Shumow, Ferguson: There May Be a Backdoor in Dual EC.

  7. Keller: ANSI X9.31 (Block cipher-based PRNG). Various artists: FIPS 186-2 (see Appendix 3)

Implementation Issues

  1. Gutmann: Lessons Learned in Implementing and Deploying Crypto Software

  2. Berson: Security Evaluation of Skype (2005, conducted at Skype’s request)

  3. Biondi, Desclaux: Silver Needle in the Skype (2006, REing of Skype binary)

Financial Services

  1. Berkman, Ostrovsky: The Unbearable Lightness of PIN cracking

  2. Bond, Zieliński: Decimalisation table attacks for PIN cracking

  3. Murdoch, Drimer, Anderson, Bond: Chip and PIN is Broken

RFID and Wireless

  1. Nohl, Evans, Starbug, Plötz: Reverse-Engineering a Cryptographic RFID Tag

  2. Bono, Green, Stubblefield, Juels, Rubin, Szydlo: Security Analysis of TI DST Tags


  1. Halperin et al.: Pacemakers and ICDs (no crypto)

  2. Ellis: Non-secret Encryption (historically very interesting)

  3. TheGrugq: Opsec for Freedom Fighters

The Logjam Attack

In case you haven’t heard, there’s a new SSL/TLS vulnerability making the rounds. Nicknamed Logjam, the new attack is ‘special’ in that it may admit complete decryption or hijacking of any TLS connection you make to an improperly configured web or mail server. Worse, there’s at least circumstantial evidence that similar (and more powerful) attacks might already be in the toolkit of some state-level attackers such as the NSA.

This work is the result of an unusual collaboration between a fantastic group of co-authors spread all around the world, including institutions such as the University of Michigan, INRIA Paris-Rocquencourt, INRIA Paris-Nancy, Microsoft Research, Johns Hopkins and the University Of Pennsylvania. It’s rare to see this level of collaboration between groups with so many different areas of expertise, and I hope to see a lot more like it. (Disclosure: I am one of the authors.)

The absolute best way to understand the Logjam result is to read the technical research paper. This post is mainly aimed at people who want a slightly less technical form. For those with even shorter attention spans, here’s the TL;DR:

It appears that the the Diffie-Hellman protocol, as currently deployed in SSL/TLS, may be vulnerable to a serious downgrade attack that restores it to 1990s “export” levels of security, and offers a practical “break” of the TLS protocol against poorly configured servers. Even worse, extrapolation of the attack requirements — combined with evidence from the Snowden documents — provides some reason to speculate that a similar attack could be leveraged against protocols (including TLS, IPSec/IKE and SSH) using 768- and 1024-bit Diffie-Hellman.

I’m going to tackle this post in the usual ‘fun’ question-and-answer format I save for this sort of thing.

What is Diffie-Hellman and why should I care about TLS “export” ciphersuites?

Diffie-Hellman is probably the most famous public key cryptosystem ever invented. Publicly discovered by Whit Diffie and Martin Hellman in the late 1970s (and a few years earlier, in secret, by UK GCHQ), it allows two parties to negotiate a shared encryption key over a public connection.

Diffie-Hellman is used extensively in protocols such as SSL/TLS and IPSec, which rely on it to establish the symmetric keys that are used to transport data. To do this, both parties must agree on a set of parameters to use for the key exchange. In traditional (‘mod p‘) Diffie-Hellman, these parameters consist of a large prime number p, as well as a ‘generator’ g. The two parties now exchange keys as shown below:

Classical Diffie-Hellman (source).

TLS supports several variants of Diffie-Hellman. The one we’re interested in for this work is the ‘ephemeral’ non-elliptic (“DHE”) protocol variant, which works in a manner that’s nearly identical to the diagram above. The server takes the role of Alice, selecting (p, g, ga mod p)and signing this tuple (and some nonces) using its long-term signing key. The client responds gb mod p and the two sides then calculate a shared secret.

Just for fun, TLS also supports an obsolete ‘export’ variant of Diffie-Hellman. These export ciphersuites are a relic from the 1990s when it was illegal to ship strong encryption out of the country. What you need to know about “export DHE” is simple: it works identically to standard DHE, but limits the size of p to 512 bits. Oh yes, and it’s still out there today. Because the Internet.

How do you attack Diffie-Hellman?

The best known attack against a correct Diffie-Hellman implementation involves capturing the value gand solving to find the secret key a. The problem of finding this value is known as the discrete logarithm problem, and it’s thought to be a mathematically intractable, at least when Diffie-Hellman is implemented in cryptographically strong groups (e.g., when p is of size 2048 bits or more).

Unfortunately, the story changes dramatically when p is relatively small — for example, 512 bits in length. Given a value gmod p for a 512-bit p, itshould at least be possible to efficiently recover the secret a and read traffic on the connection.

Most TLS servers don’t use 512-bit primes, so who cares?

The good news here is that weak Diffie-Hellman parameters are almost never used purposely on the Internet. Only a trivial fraction of the SSL/TLS servers out there today will organically negotiate 512-bit Diffie-Hellman. For the most part these are crappy embedded devices such as routers and video-conferencing gateways.
However, there is a second class of servers that are capable of supporting 512-bit Diffie-Hellman when clients request it, using a special mode called the ‘export DHE’ ciphersuite. Disgustingly, these servers amount to about 8% of the Alexa top million sites (and a whopping 29% of SMTP/STARTLS mail servers). Thankfully, most decent clients (AKA popular browsers) won’t willingly negotiate ‘export-DHE’, so this would also seem to be a dead end.
It isn’t.
ServerKeyExchange message (RFC 5246)
You see, before SSL/TLS peers can start engaging in all this fancy cryptography, they first need to decide which ciphers they’re going to use. This is done through a negotiation process in which the client proposes some options (e.g., RSA, DHE, DHE-EXPORT), and the server picks one.

This all sound simple enough. However, one of the early, well known flaws in SSL/TLS is the protocol’s failure to properly authenticate these ‘negotiation’ messages. In very early versions of SSL they were not authenticated at all. SSLv3 and TLS tacked on an authentication process — but one that takes place only at the end of the handshake.*

This is particularly unfortunate given that TLS servers often have the ability to authenticate their messages using digital signatures, but don’t really take advantage of this. For example, when two parties negotiate Diffie-Hellman, the parameters sent by the server are transmitted within a signed message called the ServerKeyExchange (shown at right). The signed portion of this message covers the parameters, but neglects to include any information about which ciphersuite the server thinks it’s negotiating. If you remember that the only difference between DHE and DHE-EXPORT is the size of the parameters the server sends down, you might start to see the problem.

Here it is in a nutshell: if the server supports DHE-EXPORT, the attacker can ‘edit’ the negotiation messages sent from the a client — even if the client doesn’t support export DHE — replacing the client’s list of supported ciphers with only export DHE. The server will in turn send back a signed 512-bit export-grade Diffie-Hellman tuple, which the client will blindly accept — because it doesn’t realize that the server is negotiating the export version of the ciphersuite.From its perspective this message looks just like ‘standard’ Diffie-Hellman with really crappy parameters.

Overview of the Logjam active attack (source: paper).

All this tampering should run into a huge snag at the end of the handshake, when he client and server exchange Finished messages embedding include a MAC of the transcript. At this point the client should learn that something funny is going on, i.e., that what it sent no longer matches what the server is seeing. However, the loophole is this: if the attacker can recover the Diffie-Hellman secret quickly — before the handshake ends — she can forge her own Finished messages. In that case the client and server will be none the wiser.

The upshot is that executing this attack requires the ability to solve a 512-bit discrete logarithm before the client and server exchange Finished messages. That seems like a tall order.

Can you really solve a discrete logarithm before a TLS handshake times out?

In practice, the fastest route to solving the discrete logarithm in finite fields is via an algorithm called the Number Field Sieve (NFS). Using NFS to solve a single 512-bit discrete logarithm instance requires several core-years — or about week of wall-clock time given a few thousand cores — which would seem to rule out solving discrete logs in real time.

However, there is a complication. In practice, NFS can actually be broken up into two different steps:

  1. Pre-computation (for a given prime p). This includes the process of polynomial selection, sieving, and linear algebra, all of which depend only on p. The output of this stage is a table for use in the second stage.
  2. Solving to find a (for a given gmod p). The final stage, called the descent, uses the table from the precomputation. This is the only part of the algorithm that actually involves a specific g and ga.
The important thing to know is that the first stage of the attack consumes the vast majority of the time, up to a full week on a large-scale compute cluster. The descent stage, on the other hand, requires only a few core-minutes. Thus the attack cost depends primarily on where the server gets its Diffie-Hellman parameters from. The best case for an attacker is when p is hard-coded into the server software and used across millions of machines. The worst case is when p is re-generated routinely by the server.

I’ll let you guess what real TLS servers actually do.
In fact, large-scale Internet scans by the team at University of Michigan show that most popular web servers software tends to re-use a small number of primes across thousands of server instances. This is done because generating prime numbers is scary, so implementers default to using a hard-coded value or a config file supplied by your Linux distribution. The situation for export Diffie-Hellman is particularly awful, with only two (!) primes used across up 92% of enabled Apache/mod_ssl sites.

Number of seconds to solve a
512-bit discrete log (source: paper).

The upshot of all of this is that about two weeks of pre-computation is sufficient to build a table that allows you to perform the downgrade against most export-enabled servers in just a few minutes (see the chart at right). This is fast enough that it can be done before the TLS connection timeout. Moreover, even if this is not fast enough, the connection can often be held open longer by using clever protocol tricks, such as sending TLS warning messages to reset the timeout clock.

Keep in mind that none of this shared prime craziness matters when you’re using sufficiently large prime numbers (on the order of 2048 bits or more). It’s only a practical issue you’re using small primes, like 512-bit, 768-bit or — and here’s a sticky one I’ll come back to in a minute — 1024 bit.

How do you fix the downgrade to export DHE?

The best and most obvious fix for this problem is to exterminate export ciphersuites from the Internet. Unfortunately, these awful configurations are the default in a number of server software packages (looking at you Postfix), and getting people to update their configurations is surprisingly difficult (see e.g., FREAK).

A simpler fix is to upgrade the major web browsers to resist the attack. The easy way to do this is to enforce a larger minimum size for received DHE keys. The problem here is that the fix itself causes some collateral damage — it will break a small but significant fraction of lousy servers that organically negotiate (non-export) DHE with 512 bit keys.

The good news here is that the major browsers have decided to break the Internet (a little) rather than allow it to break them. Each has agreed to raise the minimum size limit to at least 768 bits, and some to a minimum of 1024 bits. It’s still not perfect, since 1024-bit DHE may not be cryptographically sound against powerful attackers, but it does address the immediate export attack. In the longer term the question is whether to use larger negotiated DHE groups, or abandon DHE altogether and move to elliptic curves.

What does this mean for larger parameter sizes?

The good news so far is that 512-bit Diffie-Hellman is only used by a fraction of the Internet,even when you account for active downgrade attacks. The vast majority of servers use Diffie-Hellman moduli of length at least 1024 bits. (The widespread use of 1024 is largely due to a hard-cap in older Java clients. Go away Java.)

While 2048-bit moduli are generally believed to be outside of anyone’s reach, 1024-bit DHE has long been considered to be at least within groping range of nation-state attackers. We’ve known this for years, of course, but the practical implications haven’t been quite clear. This paper tries to shine some light on that, using Internet-wide measurements and software/hardware estimates.

If you recall from above, the most critical aspect of the NFS attack is the need to perform large amounts of pre-computation on a given Diffie-Hellman prime p, followed by a relatively short calculation to break any given connection that uses p. At the 512-bit size the pre-computation only requires about a week. The question then is, how much does it cost for a 1024-bit prime, and how common are shared primes?

While there’s no exact way to know how much the 1024-bit attack would cost, the paper attempts to provide some extrapolations based on current knowledge. With software, the cost of the pre-computation seems quite high — on the order of 35 million core-years. Making this happen for a given prime within a reasonable amount of time (say, one year) would appear to require billions of dollars of computing equipment if we assume no algorithmic improvements.Even if we rule out such improvements, it’s conceivable that this cost might be brought down to a few hundred million dollars using hardware. This doesn’t seem out of bounds when you consider leaked NSA cryptanalysis budgets.

What’s interesting is that the descent stage, required to break a given Diffie-Hellman connection, is much faster. Based on some implementation experiments by the CADO-NFSteam, it may be possible to break a Diffie-Hellman connection in as little as 30 core-days, with parallelization hugely reducing the wall-clock time. This might even make near-real-time decryption of Diffie-Hellman connections practical.

Is the NSA actually doing this?

So far all we’ve noted is that NFS pre-computation is at least potentially feasible when 1024-bit primes are re-used. That doesn’t mean the NSA is actually doing any of it.

There is some evidence, however, that suggests the NSA has decryption capability that’s at least consistent with such a break. This evidence comes from a series of Snowden documents published last winter in Der Spiegel. Together they describe a large-scale effort at NSA and GCHQ, capable of decrypting ‘vast’ amounts of Internet traffic, including IPSec, SSH and HTTPS connections.

NSA slide illustrating exploitation
of IPSec encrypted traffic (source: Spiegel).

While the architecture described by the documents mentions attacks against many protocols, the bulk of the energy seems to be around the IPSec and IKE protocols, which are used to establish Virtual Private Networks (VPNs) between individuals and corporate networks such as financial institutions.

The nature of the NSA’s exploit is never made clear in the documents, but diagram at right gives a lot of the architectural details. The system involves collecting Internet Key Exchange (IKE) handshakes, transmitting them to the NSA’s Cryptanalysis and Exploitation Services (CES) enclave, and feeding them into a decryption system that controls substantial high performance computing resources to process the intercepted exchanges. This is at least circumstantially consistent with Diffie-Hellman cryptanalysis.

Of course it’s entirely possible that the attack is based on a bad random number generator, weak symmetric encryption, or any number of engineered backdoors. There are a few pieces of evidence that militate towards a Diffie-Hellman break, however:

  1. IPSec (or rather, the IKE key exchange) uses Diffie-Hellman for every single connection, meaning that it can’t be broken without some kind of exploit, although this doesn’t rule out the other explanations.
  2. The IKE exchange is particularly vulnerable to pre-computation, since IKE uses a small number of standardized prime numbers called the Oakley groups, which are going on 17 years old now. Large-scale Internet scanning by the Michigan team shows that a majority of responding IPSec endpoints will gladly negotiate using Oakley Group 1 (768 bit) or Group 2 (1024 bit), even when the initiator offers better options.
  3. The NSA’s exploit appears to require the entire IKE handshake as well as any pre-shared key (PSK). These inputs would be necessary for recovery of IKEv1 session keys, but are not required in a break that involves only symmetric cryptography.
  4. The documents explicitly rule out the use of malware, or rather, they show that such malware (‘TAO implants’) is in use — but that malware allows the NSA to bypass the IKE handshake altogether.

I would stipulate that beyond the Internet measurements and computational analysis, this remains firmly in the category of  ‘crazy-eyed informed speculation’. But while we can’t rule out other explanations, this speculation is certainly consistent with a hardware-optimized break of Diffie-Hellman 768 and 1024-bit, along with some collateral damage to SSH and related protocols.

So what next?

The paper gives a detailed set of recommendations on what to do about these downgrade attacks and (relatively) weak DHE groups. The website provides a step-by-step guide for server administrators. In short, probably the best long-term move is to switch to elliptic curves (ECDHE) as soon as possible. Failing this, clients and servers should enforce at least 2048-bit Diffie-Hellman across the Internet. If you can’t do that, stop using common primes.

Making this all happen on anything as complicated as the Internet will probably consume a few dozen person-lifetimes. But it’s something we have to do, and will do, to make the Internet work properly.


* There are reasons for this. Some SSL/TLS ciphersuites (such as the RSA encryption-based ciphersuites) don’t use signatures within the protocol, so the only way to authenticate the handshake is to negotiate a ciphersuite, run the key exchange protocol, then use the resulting shared secret to authenticate the negotiation messages after the fact. But SSL/TLS DHE involves digital signatures, so it should be possible to achieve a stronger level of security than this. It’s unfortunate that the protocol does not.

How Do We Build Encryption Backdoors?

They say that history repeats itself, first as tragedy, then as farce. Never has this principle been more apparent than in this new piece by Washington Post reporters Ellen Nakashima and Barton Gellman: ‘As encryption spreads, U.S. grapples with clash between privacy, security‘.

The subject of the piece is a renewed effort by U.S. intelligence and law enforcement agencies to mandate ‘backdoors’ in modern encryption systems. This is ostensibly a reaction to the mass adoption of strong encryption in smartphones, and a general fear that police are about to lose wiretapping capability they’ve come to depend on.

This is not the first time we’ve been here. Back in the 1990s the Federal government went as far as to propose a national standard for ‘escrowed’ telephone encryption called the ‘Clipper’ chip. That effort failed in large part because the technology was terrible, but also because — at least at the time — the idea of ordinary citizens adopting end-to-end encryption was basically science fiction.

Thanks to the advent of smartphones and ‘on-by-default’ encryption in popular systems like Apple’s iMessage, and WhatsApp, Americans are finally using end-to-end encryption at large scale and seem to like it. This is scaring the bejesus out of the powers that be.

Hence crypto backdoors.

As you might guess, I have serious philosophical objections to the idea of adding backdoors to any encryption system — for excellent reasons I could spend thousands of words on. But I’m not going to do that. What I’d like to do in this post is put aside my own value judgements and try to take these government proposals at face value.

Thus the question I’m going to consider in this post:

Let’s pretend that encryption backdoors are a great idea. From a purely technical point of view, what do we need to do to implement them, and how achievable is it?

First some background.

End-to-end encryption 101

Modern encryption schemes break down into several categories. For the purposes of this discussion we’ll consider two: those systems for which the provider holds the key, and the set of systems where the provider doesn’t.

We’re not terribly interested in the first type of encryption, which includes protocols like SSL/TLS and Google Hangouts, since those only protect data at the the link layer, i.e.,until it reaches your provider’s servers. I think it’s fairly well established that if Facebook, Apple, Google or Yahoo can access your data, then the government can access it as well — simply by subpoenaing or compelling those companies. We’ve even seen how this can work.

The encryption systems we’re interested all belong to the second class — protocols where even the provider can’t decrypt your information. This includes:

This seems like a relatively short list, but in practice w’re talking about an awful lot of data. The iMessage and WhatsApp systems alone process billions of instant messages every day, and Apple’s device encryption is on by default for everyone with a recent(ly updated) iPhone.

How to defeat end-to-end encryption

If you’ve decided to go after end-to-end encryption through legal means, there are a relatively small number of ways to proceed.

By far the simplest is to simply ban end-to-end crypto altogether, or to mandate weak encryption. There’s some precedent for this: throughout the 1990s, the NSA forced U.S. companies to ship ‘export‘ grade encryption that was billed as being good enough for commercial use, but weak enough for governments to attack. The problem with this strategy is that attacks only get better — but legacy crypto never dies.

Fortunately for this discussion, we have some parameters to work with. One of these is that Washington seems to genuinely want to avoid dictating technological designs to Silicon Valley. More importantly, President Obama himself has stated that “there’s no scenario in which we don’t want really strong encryption“. Taking these statements at face value should mean that we can exclude outright crypto bans, mandated designs, and any modification has the effect of fundamentally weakening encryption against outside attackers.

If we mix this all together, we’re left with only two real options:

  1. Attacks on key distribution. In systems that depend on centralized, provider-operated key servers, such as WhatsApp, Facetime, Signal and iMessage,** governments can force providers to distribute illegitimate public keys, or register shadow devices to a user’s account. This is essentially a man-in-the-middle attack on encrypted communication systems.
  2. Key escrow. Just about any encryption scheme can be modified to encrypt a copy of a decryption (or session) key such that a ‘master keyholder’ (e.g., Apple, or the U.S. government) can still decrypt. A major advantage is that this works even for device encryption systems, which have no key servers to suborn.

Each approach requires some modifications to clients, servers or other components of the system.

Attacking key distribution

Key lookup request for Apple iMessage. The phone
number is shown at top right, and the response at bottom left.

Many end-to-end encrypted messaging systems, including WhatsApp and iMessage, generate a long-term public and secret keypair for every device you own. The public portion of this keypair is distributed to anyone who might want to send you messages. The secret key never leaves the device.

Before you can initiate a connection with your intended recipient, you first have to obtain a copy of the recipient’s public key. This is commonly handled using a key server that’s operated by the provider. The key server may hand back one, or multiple public keys (depending on how many devices you’ve registered). As long as those keys all legitimately belong to your intended recipient, everything works fine.

Intercepting messages is possible, however, if the provider is willing to substitute its own public keys — keys for which it (or the government) actually knows the secret half. In theory this is relatively simple — in practice it can be something of a bear, due to the high complexity of protocols such as iMessage.

Key fingerprints.

The main problem with key distribution attacks is — unlike a traditional wiretap — substitute keys are at least in theory detectable by the target. Some communication systems, like Signal, allow users to compare key fingerprints in order to verify that each received the right public key. Others, like iMessage and WhatsApp, don’t offer this technology — but could easily be modified to do so (even using third party clients). Systems like CONIKS may even automate this process in the future — allowing applications to monitor changes to their own keys in real time as they’re distributed by a server.

A final, and salient feature on the key distribution approach is that it allows only prospective eavesdropping — that is, law enforcement must first target a particular user, and only then can they eavesdrop on her connections. There’s no way to look backwards in time. I see this is a generally good thing. Others may disagree.

Key Escrow 

Structure of the Clipper ‘LEAF’.

The techniques above don’t help much for systems without public key servers, Moreover, they do nothing for systems that don’t use public keys at all, the prime example being device encryptionIn this case, the only real alternative is to mandate some sort of key escrow.

Abstractly, the purpose of an escrow system is to place decryption keys on file (‘escrow’ them) with some trusted authority, who can break them out when the need arises. In practice it’s usually a bit more complex.

The first wrinkle is that modern encryption systems often feature many decryption keys, some of which can be derived on-the-fly while the system operates. (Systems such as TextSecure/WhatsApp actually derive new encryption keys for virtually every message you send.) Users with encrypted devices may change their password from time to time.

To deal with this issue, a preferred approach is to wrap these session keys up (encrypt them) under some master public key generated by the escrow authority — and to store/send the resulting ciphertexts along with the rest of the encrypted data. In the 1990s Clipperspecification these ciphertexts were referred to as Law Enforcement Access Fields, or LEAFs.***

With added LEAFs in your protocol, wiretapping becomes relatively straightforward. Law enforcement simply intercepts the encrypted data — or obtains it from your confiscated device — extract the LEAFs, and request that the escrow authority decrypt them. You can find variants of this design dating back to the PGP era. In fact, the whole concept is deceptively simple — provided you don’t go farther than the whiteboard. 

Conceptual view of some encrypted data (left) and a LEAF (right).

It’s only when you get into the details of actually implementing key escrow that things get hairy. These schemes require you to alter every protocol in your encryption system, at a pretty fundamental level — in the process creating the mother of all security vulnerabilities — but, more significantly, they force you to think very seriously about who you trust to hold those escrow keys.

Who does hold the keys?

This is the million dollar question for any escrow platform. The Post story devotes much energy to exploring various proposals for doing this.

Escrow key management is make-or-break, since the key server represents a universal vulnerability in any escrowed communication system. In the present debate there appear to be two solutions on the table. The first is to simply dump the problem onto individual providers, who will be responsible for managing their escrow keys — using whatever technological means they deem appropriate. A few companies may get this right. Unfortunately, most companies suck at cryptography, so it seems reasonable to believe that the resulting systems will be quite fragile.

The second approach is for the government to hold the keys themselves. Since the escrow key is too valuable to entrust to one organization, one or more trustworthy U.S. departments would hold ‘shares‘ of the master key, and would cooperate to provide decryption on a case-by-case basis. This was, in fact, the approach proposed for the Clipper chip.

The main problem with this proposal is that it’s non-trivial to implement. If you’re going to split keys across multiple agencies, you have to consider how you’re going to store those keys, and how you’re going to recover them when you need to access someone’s data. The obvious approach — bring the key shares back together at some centralized location — seems quite risky, since the combined master key would be vulnerable in that moment.

A second approach is to use a threshold cryptosystem. Threshold crypto refers to a set of techniques for storing secret keys across multiple locations so that decryption can be done in place without recombining the key shares. This seems like an ideal solution, with only one problem: nobody has deployed threshold cryptosystems at this kind of scale before. In fact, many of the protocols we know of in this area have never even been implemented outside of the research literature. Moreover, it will require governments to precisely specify a set of protocols for tech companies to implement — this seems incompatible with the original goal of letting technologists design their own systems.

Software implementations

A final issue to keep in mind is the complexity of the software we’ll need to make all of this happen. Our encryption software is already so complex that it’s literally at the breaking point. (If you don’t believe me, take a look at OpenSSL’s security advisories for the last year) While adding escrow mechanisms seems relatively straightforward, it will actually require quite a bit of careful coding, something we’re just not good at.

Even if we do go forward with this plan, there are many unanswered questions. How widely can these software implementations be deployed? Will every application maker be forced to use escrow? Will we be required to offer a new set of system APIs in iOS, Windows and Android that we can use to get this right? Answering each of these questions will result in dramatic changes throughout the OS software stack. I don’t envy the poor developers who will have to answer them.

How do we force people to use key escrow?

Leaving aside the technical questions, the real question is: how do you force anyone to do this stuff? Escrow requires breaking changes to most encryption protocols; it’s costly as hell; and it introduces many new security concerns. Moreover, laws outlawing end-to-end encryption software seem destined to run afoul of the First Amendment.

I’m not a lawyer, so don’t take my speculation too seriously — but it seems intuitive to me that any potential legislation will be targeted at service providers, not software vendors or OSS developers. Thus the real leverage for mandating key escrow will apply to the Facebooks and Apples of the world. Your third-party PGP and OTR clients would be left alone, for the tiny percentage of the population who uses these tools.

Unfortunately, even small app developers are increasingly running their own back-end servers these days (e.g., Whisper Systems and Silent Circle) so this is less reassuring than it sounds. Probably the big takeaway for encryption app developers is that it might be good to think about how you’ll function in a world where it’s no longer possible to run your own back-end data transport service — and where other commercial services may not be too friendly to moving your data for you.

In conclusion

If this post has been more questions than answers, that’s because there really are no answers right now. A serious debate is happening in an environment that’s almost devoid of technical input, at least from technical people who aren’t part of the intelligence establishment.

And maybe that by itself is reason enough to be skeptical.


  • Not an endorsement. I have many thoughts on Telegram’s encryption protocols, but they’re beyond the scope of this post.

** Telegram is missing from this list because their protocol doesn’t handle long term keys at all. Every single connection must be validated in person using a graphical key fingerprint, which is, quite frankly, terrible.

*** The Clipper chip used a symmetric encryption algorithm to encrypt the LEAF, which meant that the LEAF decryption key had to be present inside of every consumer device. This was completely nuts, and definitely a bullet dodged. It also meant that every single Clipper had to be implemented in hardware using tamper resistant chip manufacturing technology. It was a truly awful design.

Transport Layer Protection Cheat Sheet


This cheat sheet provides a simple model to follow when implementing transport layer protection for an application. Although the concept of SSL is known to many, the actual details and security specific decisions of implementation are often poorly understood and frequently result in insecure deployments. This article establishes clear rules which provide guidance on securely designing and configuring transport layer security for an application. This article is focused on the use of SSL/TLS between a web application and a web browser, but we also encourage the use of SSL/TLS or other network encryption technologies, such as VPN, on back end and other non-browser based connections.

Architectural Decision

An architectural decision must be made to determine the appropriate method to protect data when it is being transmitted. The most common options available to corporations are Virtual Private Networks (VPN) or a SSL/TLS model commonly used by web applications. The selected model is determined by the business needs of the particular organization. For example, a VPN connection may be the best design for a partnership between two companies that includes mutual access to a shared server over a variety of protocols. Conversely, an Internet facing enterprise web application would likely be best served by a SSL/TLS model.

This cheat sheet will focus on security considerations when the SSL/TLS model is selected. This is a frequently used model for publicly accessible web applications.

Providing Transport Layer Protection with SSL/TLS


The primary benefit of transport layer security is the protection of web application data from unauthorized disclosure and modification when it is transmitted between clients (web browsers) and the web application server, and between the web application server and back end and other non-browser based enterprise components.

The server validation component of TLS provides authentication of the server to the client. If configured to require client side certificates, TLS can also play a role in client authentication to the server. However, in practice client side certificates are not often used in lieu of username and password based authentication models for clients.

TLS also provides two additional benefits that are commonly overlooked; integrity guarantees and replay prevention. A TLS stream of communication contains built-in controls to prevent tampering with any portion of the encrypted data. In addition, controls are also built-in to prevent a captured stream of TLS data from being replayed at a later time.

It should be noted that TLS provides the above guarantees to data during transmission. TLS does not offer any of these security benefits to data that is at rest. Therefore appropriate security controls must be added to protect data while at rest within the application or within data stores.

Basic Requirements

The basic requirements for using TLS are: access to a Public Key Infrastructure (PKI) in order to obtain certificates, access to a directory or an Online Certificate Status Protocol (OCSP) responder in order to check certificate revocation status, and agreement/ability to support a minimum configuration of protocol versions and protocol options for each version.


The terms, Secure Socket Layer (SSL) and Transport Layer Security (TLS) are often used interchangeably. In fact, SSL v3.1 is equivalent to TLS v1.0. However, different versions of SSL and TLS are supported by modern web browsers and by most modern web frameworks and platforms. For the purposes of this cheat sheet we will refer to the technology generically as TLS. Recommendations regarding the use of SSL and TLS protocols, as well as browser support for TLS, can be found in the rule below titled “Only Support Strong Protocols”.

Cryptomodule Parts and Operation

When to Use a FIPS 140-2 Validated Cryptomodule

If the web application may be the target of determined attackers (a common threat model for Internet accessible applications handling sensitive data), it is strongly advised to use TLS services that are provided by FIPS 140-2 validated cryptomodules.

A cryptomodule, whether it is a software library or a hardware device, basically consists of three parts:

  • Components that implement cryptographic algorithms (symmetric and asymmetric algorithms, hash algorithms, random number generator algorithms, and message authentication code algorithms)
  • Components that call and manage cryptographic functions (inputs and outputs include cryptographic keys and so-called critical security parameters)
  • A physical container around the components that implement cryptographic algorithms and the components that call and manage cryptographic functions

The security of a cryptomodule and its services (and the web applications that call the cryptomodule) depend on the correct implementation and integration of each of these three parts. In addition, the cryptomodule must be used and accessed securely. The includes consideration for:

  • Calling and managing cryptographic functions
  • Securely Handling inputs and output
  • Ensuring the secure construction of the physical container around the components

In order to leverage the benefits of TLS it is important to use a TLS service (e.g. library, web framework, web application server) which has been FIPS 140-2 validated. In addition, the cryptomodule must be installed, configured and operated in either an approved or an allowed mode to provide a high degree of certainty that the FIPS 140-2 validated cryptomodule is providing the expected security services in the expected manner.

If the system is legally required to use FIPS 140-2 encryption (e.g., owned or operated by or on behalf of the U.S. Government) then TLS must be used and SSL disabled. Details on why SSL is unacceptable are described in Section 7.1 of Implementation Guidance for FIPS PUB 140-2 and the Cryptographic Module Validation Program.

Further reading on the use of TLS to protect highly sensitive data against determined attackers can be viewed inSP800-52 Guidelines for the Selection and Use of Transport Layer Security (TLS) Implementations

Secure Server Design

Rule – Use TLS for All Login Pages and All Authenticated Pages

The login page and all subsequent authenticated pages must be exclusively accessed over TLS. The initial login page, referred to as the “login landing page”, must be served over TLS. Failure to utilize TLS for the login landing page allows an attacker to modify the login form action, causing the user’s credentials to be posted to an arbitrary location. Failure to utilize TLS for authenticated pages after the login enables an attacker to view the unencrypted session ID and compromise the user’s authenticated session.

Rule – Use TLS on Any Networks (External and Internal) Transmitting Sensitive Data

All networks, both external and internal, which transmit sensitive data must utilize TLS or an equivalent transport layer security mechanism. It is not sufficient to claim that access to the internal network is “restricted to employees”. Numerous recent data compromises have shown that the internal network can be breached by attackers. In these attacks, sniffers have been installed to access unencrypted sensitive data sent on the internal network.

Rule – Do Not Provide Non-TLS Pages for Secure Content

All pages which are available over TLS must not be available over a non-TLS connection. A user may inadvertently bookmark or manually type a URL to a HTTP page (e.g. within the authenticated portion of the application. If this request is processed by the application then the response, and any sensitive data, would be returned to the user over the clear text HTTP.

Rule – Do Not Mix TLS and Non-TLS Content

A page that is available over TLS must be comprised completely of content which is transmitted over TLS. The page must not contain any content that is transmitted over unencrypted HTTP. This includes content from unrelated third party sites.

An attacker could intercept any of the data transmitted over the unencrypted HTTP and inject malicious content into the user’s page. This malicious content would be included in the page even if the overall page is served over TLS. In addition, an attacker could steal the user’s session cookie that is transmitted with any non-TLS requests. This is possible if the cookie’s ‘secure’ flag is not set. See the rule ‘Use “Secure” Cookie Flag’

Rule – Use “Secure” Cookie Flag

The “Secure” flag must be set for all user cookies. Failure to use the “secure” flag enables an attacker to access the session cookie by tricking the user’s browser into submitting a request to an unencrypted page on the site. This attack is possible even if the server is not configured to offer HTTP content since the attacker is monitoring the requests and does not care if the server responds with a 404 or doesn’t respond at all.

Rule – Keep Sensitive Data Out of the URL

Sensitive data must not be transmitted via URL arguments. A more appropriate place is to store sensitive data in a server side repository or within the user’s session. When using TLS the URL arguments and values are encrypted during transit. However, there are two methods that the URL arguments and values could be exposed.

  1. The entire URL is cached within the local user’s browser history. This may expose sensitive data to any other user of the workstation.
  2. The entire URL is exposed if the user clicks on a link to another HTTPS site. This may expose sensitive data within the referral field to the third party site. This exposure occurs in most browsers and will only occur on transitions between two TLS sites.

For example, a user following a link on which leads to would expose the full URL of (including URL arguments) in the referral header (within most browsers). This would not be the case if the user followed a link on to

Rule – Prevent Caching of Sensitive Data

The TLS protocol provides confidentiality only for data in transit but it does not help with potential data leakage issues at the client or intermediary proxies. As a result, it is frequently prudent to instruct these nodes not to cache or persist sensitive data. One option is to add anticaching headers to relevant HTTP responses, (for example, “Cache-Control: no-cache, no-store” and “Expires: 0” for coverage of many modern browsers as of 2013). For compatibility with HTTP/1.0 (i.e., when user agents are really old or the webserver works around quirks by forcing HTTP/1.0) the response should also include the header “Pragma: no-cache”. More information is available in HTTP 1.1 RFC 2616, section 14.9.

Rule – Use HTTP Strict Transport Security

See: HTTP Strict Transport Security

Rule – Use Public Key Pinning

See: Certificate and Public Key Pinning

Server Certificate

Note: If using a FIPS 140-2 cryptomodule disregard the following rules and defer to the recommended configuration for the particular cryptomodule. Nevertheless we recommend to use this rules to audit your configuration.

Rule – Use Strong Keys & Protect Them

The private key used to generate the cipher key must be sufficiently strong for the anticipated lifetime of the private key and corresponding certificate. The current best practice is to select a key size of at least 2048 bits. Additional information on key lifetimes and comparable key strengths can be found in [1], NIST SP 800-57. In addition, the private key must be stored in a location that is protected from unauthorized access.

Rule – Use a Certificate That Supports Required Domain Names

A user should never be presented with a certificate error, including prompts to reconcile domain or hostname mismatches, or expired certificates. If the application is available at both and then an appropriate certificate, or certificates, must be presented to accommodate the situation. The presence of certificate errors desensitizes users to TLS error messages and increases the possibility an attacker could launch a convincing phishing or man-in-the-middle attack.

For example, consider a web application accessible at and One certificate should be acquired for the host or server; and a second certificate for host or In both cases, the hostname would be present in the Subject’s Common Name (CN).

Alternatively, the Subject Alternate Names (SANs) can be used to provide a specific listing of multiple names where the certificate is valid. In the example above, the certificate could list the Subject’s CN as, and list two and These certificates are sometimes referred to as “multiple domain certificates”.

Rule – Use Fully Qualified Names in Certificates

Use fully qualified names in the DNS name field, and do not use unqualifed names (e.g., ‘www’), local names (e.g., ‘localhost’), or private IP addresses (e.g., in the DNS name field. Unqualifed names, local names, or private IP addresses violate the certificate specification.

Rule – Do Not Use Wildcard Certificates

You should refrain from using wildcard certificates. Though they are expedient at circumventing annoying user prompts, they also violate the principal of least privilege and asks the user to trust all machines, including developer’s machines, the secretary’s machine in the lobby and the sign-in kiosk. Obtaining access to the private key is left as an exercise for the attacker, but its made much easier when stored on the file system unprotected.

Statistics gathered by Qualys for Internet SSL Survey 2010 indicate wildcard certificates have a 4.4% share, so the practice is not standard for public facing hosts. Finally, wildcard certificates violate EV Certificate Guidelines.

Rule – Do Not Use RFC 1918 Addresses in Certificates

Certificates should not use private addresses. RFC 1918 is Address Allocation for Private Internets. Private addresses are Internet Assigned Numbers Authority (IANA) reserved and include 192.168/16, 172.16/12, and 10/8.

Certificates issued with private addresses violate EV Certificate Guidelines. In addition, Peter Gutmann writes in inEngineering Security: “This one is particularly troublesome because, in combination with the router-compromise attacks… and …OSCP-defeating measures, it allows an attacker to spoof any EV-certificate site.”

Rule – Use an Appropriate Certification Authority for the Application’s User Base

An application user must never be presented with a warning that the certificate was signed by an unknown or untrusted authority. The application’s user population must have access to the public certificate of the certification authority which issued the server’s certificate. For Internet accessible websites, the most effective method of achieving this goal is to purchase the TLS certificate from a recognize certification authority. Popular Internet browsers already contain the public certificates of these recognized certification authorities.

Internal applications with a limited user population can use an internal certification authority provided its public certificate is securely distributed to all users. However, remember that all certificates issued by this certification authority will be trusted by the users. Therefore, utilize controls to protect the private key and ensure that only authorized individuals have the ability to sign certificates.

The use of self signed certificates is never acceptable. Self signed certificates negate the benefit of end-point authentication and also significantly decrease the ability for an individual to detect a man-in-the-middle attack.

Rule – Always Provide All Needed Certificates

Clients attempt to solve the problem of identifying a server or host using PKI and X509 certificate. When a user receives a server or host’s certificate, the certificate must be validated back to a trusted root certification authority. This is known as path validation.

There can be one or more intermediate certificates in between the end-entity (server or host) certificate and root certificate. In addition to validating both endpoints, the user will also have to validate all intermediate certificates. Validating all intermediate certificates can be tricky because the user may not have them locally. This is a well-known PKI issue called the “Which Directory?” problem.

To avoid the “Which Directory?” problem, a server should provide the user with all required certificates used in a path validation.

Rule – Be aware of and have a plan for the SHA-1 deprecation plan

In order to avoid presenting end users with progressive certificate warnings, organizations must proactively address the browser vendor’s upcoming SHA-1 deprecation plans. The Google Chrome plan is probably the most specific and aggressive at this point: Gradually sunsetting SHA-1

If your organization has no SHA256 compatibility issues then it may be appropriate to move your site to a SHA256 signed certificate/chain. If there are, or may be, issues – you should ensure that your SHA-1 certificates expire before 1/1/2017.

Server Protocol and Cipher Configuration

Note: If using a FIPS 140-2 cryptomodule disregard the following rules and defer to the recommended configuration for the particular cryptomodule. Nevertheless we recommend to use this rules to audit your configuration.

Rule – Only Support Strong Protocols

SSL/TLS is a collection of protocols. Weaknesses have been identified with earlier SSL protocols, including SSLv2and SSLv3 hence SSL versions 1, 2 and 3 should not longer be used. The best practice for transport layer protection is to only provide support for the TLS protocols – TLS1.0, TLS 1.1 and TLS 1.2. This configuration will provide maximum protection against skilled and determined attackers and is appropriate for applications handling sensitive data or performing critical operations.

Nearly all modern browsers support at least TLS 1.0. As of February 2013, contemporary browsers (Chrome v20+, IE v8+, Opera v10+, and Safari v5+) support TLS 1.1 and TLS 1.2. You should provide support for TLS 1.1 and TLS 1.2 to accommodate clients which support the protocols. The client and server (usually) negotiate the best protocol, that is supported on both sides.

TLS 1.0 is still widely used as ‘best’ protocol by a lot of browsers, that are not patched to the very latest version. It suffers CBC Chaining attacks and Padding Oracle attacks. TLSv1.0 should only be used only after risk analysis and acceptance.

Under no circumstances neither SSLv2 nor SSLv3 should be enabled as a protocol selection:

Rule – Prefer Ephemeral Key Exchanges

Ephemeral key exchanges are based on Diffie-Hellman and use per-session, temporary keys during the initial SSL/TLS handshake. They provide perfect forward secrecy (PFS), which means a compromise of the server’s long term signing key does not compromise the confidentiality of past session (see following rule). When the server uses an ephemeral key, the server will sign the temporary key with its long term key (the long term key is the customary key available in its certificate).

Use cryptographic parameters (like DH-parameter) that use a secure length that match to the supported keylength of your certificate (>=2048 bits or equivalent Elliptic Curves). As some middleware had some issues with this, upgrade to the latest version. Note: There are some legacy browsers or old Java versions that are not capable to cope with DH-Params >1024 bits, please read the following rule how this can be solved.

Do not use standardized DH-parameters like they are defined by RFCs 2409, 3526, or 5114. Generate your individual DH-parameters to get unique prime numbers (this may take a long time):

openssl dhparam 2048 -out dhparam2048.pem

Set the path to use this parameter file, e.g. when using Apache:

SSLOpenSSLConfCmd DHParameters <path to dhparam2048.pem>

If you have a server farm and are providing forward secrecy, then you might have to disable session resumption. For example, Apache writes the session id’s and master secrets to disk so all servers in the farm can participate in resuming a session (there is currently no in-memory mechanism to achieve the sharing). Writing the session id and master secret to disk undermines forward secrecy.

Rule – Only Support Strong Cryptographic Ciphers

Each protocol (TLSv1.0, TLSv1.1, TLSv1.2, etc) provides cipher suites. As of TLS 1.2, there is support for over 300 suites (320+ and counting), including national vanity cipher suites. The strength of the encryption used within a TLS session is determined by the encryption cipher negotiated between the server and the browser. In order to ensure that only strong cryptographic ciphers are selected the server must be modified to disable the use of weak ciphers and to configure the ciphers in an adequate order. It is recommended to configure the server to only support strong ciphers and to use sufficiently large key sizes. In general, the following should be observed when selecting CipherSuites:

  • Use the very latest recommendations, they may be volantile these days
  • Setup your Policy to get a Whitelist for recommended Ciphers, e.g.:
    • Activate to set the Cipher Order by the Server
    • Highest Priority for Ciphers that support ‘Forward Secrecy’ (-> Support ephemeral Diffie-Hellman key exchange, see rule above) [2]
    • Favor DHE over ECDHE (and monitor the CPU usage, see Notes below), ECDHE lacks now of really reliable Elliptic Curves, see discussion about secp{224,256,384,521}r1 and secp256k1, cf. [3], [4]. The solution might be to use Brainpool Curves [German], defined for TLS in RFC 7027, or Edwards Curves. The most promising candidates for the latter are ‘Curve25519’ and Ed448-Goldilocks (see DRAFT-irtf-cfrg-curves), that is not yet defined for TLS, cf. IANA
    • Use RSA-Keys (no DSA/DSS: they get very weak, if a bad entropy source is used during signing, cf. [5], [6])
    • Favor GCM over CBC regardless of the cipher size. In other words, use Authenticated Encryption with Associated Data (AEAD), e.g. AES-GCM, AES-CCM.
    • Watch also for Stream Ciphers which XOR the key stream with plaintext (such as AES/CTR mode)
    • Priorize the ciphers by the sizes of the Cipher and the MAC
    • Use SHA1 or above for digests, prefer SHA2 (or equivalent)
    • Disable weak ciphers (which is implicitly done by this whitelist) without disabling legacy browsers and bots that have to be supported (find the best compromise), actually the cipher TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) does this job.
      • Disable cipher suites that do not offer encryption (eNULL, NULL)
      • Disable cipher suites that do not offer authentication (aNULL). aNULL includes anonymous cipher suites ADH (Anonymous Diffie-Hellman) and AECDH (Anonymous Elliptic Curve Diffie Hellman).
      • Disable export level ciphers (EXP, eg. ciphers containing DES)
      • Disable key sizes smaller than 128 bits for encrypting payload traffic (see BSI: TR-02102 Part 2 (German))
      • Disable the use of MD5 as a hashing mechanism for payload traffic
      • Disable the use of IDEA Cipher Suites (see [7])
      • Disable RC4 cipher suites (see [8], [9])
    • Ciphers should be usable for DH-Pamameters >= 2048 bits, without blocking legacy browsers (The cipher ‘DHE-RSA-AES128-SHA’ is suppressed as some browsers like to use it but are not capable to cope with DH-Params > 1024 bits.)
  • Define a Cipher String that works with different Versions of your encryption tool, like openssl
  • Verify your cipher string

#add optionally ‘:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:!ADH:!IDEA’ to protect older Versions of OpenSSL
#you may use openssl ciphers -V “…” for openssl >= 1.0.1:

0x00,0x9F - DHE-RSA-AES256-GCM-SHA384   TLSv1.2 Kx=DH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
0x00,0x9E - DHE-RSA-AES128-GCM-SHA256   TLSv1.2 Kx=DH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x6B - DHE-RSA-AES256-SHA256       TLSv1.2 Kx=DH     Au=RSA  Enc=AES(256)    Mac=SHA256
0x00,0x39 - DHE-RSA-AES256-SHA          SSLv3   Kx=DH     Au=RSA  Enc=AES(256)    Mac=SHA1
0x00,0x67 - DHE-RSA-AES128-SHA256       TLSv1.2 Kx=DH     Au=RSA  Enc=AES(128)    Mac=SHA256
0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH   Au=RSA  Enc=AESGCM(256) Mac=AEAD
0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH   Au=RSA  Enc=AESGCM(128) Mac=AEAD
0xC0,0x28 - ECDHE-RSA-AES256-SHA384     TLSv1.2 Kx=ECDH   Au=RSA  Enc=AES(256)    Mac=SHA384
0xC0,0x14 - ECDHE-RSA-AES256-SHA        SSLv3   Kx=ECDH   Au=RSA  Enc=AES(256)    Mac=SHA1
0xC0,0x27 - ECDHE-RSA-AES128-SHA256     TLSv1.2 Kx=ECDH   Au=RSA  Enc=AES(128)    Mac=SHA256
0xC0,0x13 - ECDHE-RSA-AES128-SHA        SSLv3   Kx=ECDH   Au=RSA  Enc=AES(128)    Mac=SHA1
0x00,0x9D - AES256-GCM-SHA384           TLSv1.2 Kx=RSA    Au=RSA  Enc=AESGCM(256) Mac=AEAD
0x00,0x9C - AES128-GCM-SHA256           TLSv1.2 Kx=RSA    Au=RSA  Enc=AESGCM(128) Mac=AEAD
0x00,0x35 - AES256-SHA                  SSLv3   Kx=RSA    Au=RSA  Enc=AES(256)    Mac=SHA1
0x00,0x2F - AES128-SHA                  SSLv3   Kx=RSA    Au=RSA  Enc=AES(128)    Mac=SHA1
0x00,0x0A - DES-CBC3-SHA                SSLv3   Kx=RSA    Au=RSA  Enc=3DES(168)   Mac=SHA1


  • According to my researches the most common browsers should be supported with this setting, too (see also SSL Labs: SSL Server Test -> SSL Report -> Handshake Simulation).
  • Monitor the performance of your server, e.g. the TLS handshake with DHE hinders the CPU abt 2.4 times more than ECDHE, cf. Vincent Bernat, 2011, nmav’s Blog, 2011.
  • Use of Ephemeral Diffie-Hellman key exchange will protect confidentiality of the transmitted plaintext data even if the corresponding RSA or DSS server private key got compromised. An attacker would have to perform active man-in-the-middle attack at the time of the key exchange to be able to extract the transmitted plaintext. All modern browsers support this key exchange with the notable exception of Internet Explorer prior to Windows Vista.

Additional information can be obtained within the TLS 1.2 RFC 5246, SSL Labs: ‘SSL/TLS Deployment Best Practices’, BSI: ‘TR-02102 Part 2 (German)’, ENISA: ‘Algorithms, Key Sizes and Parameters Report’, RFC 7525: Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS)and FIPS 140-2 IG.

Rule – Support TLS-PSK and TLS-SRP for Mutual Authentication

When using a shared secret or password offer TLS-PSK (Pre-Shared Key) or TLS-SRP (Secure Remote Password), which are known as Password Authenticated Key Exchange (PAKEs). TLS-PSK and TLS-SRP properly bind the channel, which refers to the cryptographic binding between the outer tunnel and the inner authentication protocol. IANA currently reserves 79 PSK cipehr suites and 9 SRP cipher suites.

Basic authentication places the user’s password on the wire in the plain text after a server authenticates itself. Basic authentication only provides unilateral authentication. In contrast, both TLS-PSK and TLS-SRP provide mutual authentication, meaning each party proves it knows the password without placing the password on the wire in the plain text.

Finally, using a PAKE removes the need to trust an outside party, such as a Certification Authority (CA).

Rule – Only Support Secure Renegotiations

A design weakness in TLS, identified as CVE-2009-3555, allows an attacker to inject a plaintext of his choice into a TLS session of a victim. In the HTTPS context the attacker might be able to inject his own HTTP requests on behalf of the victim. The issue can be mitigated either by disabling support for TLS renegotiations or by supporting only renegotiations compliant with RFC 5746. All modern browsers have been updated to comply with this RFC.

Rule – Disable Compression

Compression Ratio Info-leak Made Easy (CRIME) is an exploit against the data compression scheme used by the TLS and SPDY protocols. The exploit allows an adversary to recover user authentication cookies from HTTPS. The recovered cookie can be subsequently used for session hijacking attacks.

Test your overall TLS/SSL setup and your Certificate

This section shows the most common references only. For more tools and such, please refer to Tools.

Client (Browser) Configuration

The validation procedures to ensure that a certificate is valid are complex and difficult to correctly perform. In a typical web application model, these checks will be performed by the client’s web browser in accordance with local browser settings and are out of the control of the application. However, these items do need to be addressed in the following scenarios:

  • The application server establishes connections to other applications over TLS for purposes such as web services or any exchange of data
  • A thick client application is connecting to a server via TLS

In these situations extensive certificate validation checks must occur in order to establish the validity of the certificate. Consult the following resources to assist in the design and testing of this functionality. The NIST PKI testing site includes a full test suite of certificates and expected outcomes of the test cases.

As specified in the above guidance, if the certificate can not be validated for any reason then the connection between the client and server must be dropped. Any data exchanged over a connection where the certificate has not properly been validated could be exposed to unauthorized access or modification.

Additional Controls

Extended Validation Certificates

Extended validation certificates (EV Certificates) proffer an enhanced investigation by the issuer into the requesting party due to the industry’s race to the bottom. The purpose of EV certificates is to provide the user with greater assurance that the owner of the certificate is a verified legal entity for the site. Browsers with support for EV certificates distinguish an EV certificate in a variety of ways. Internet Explorer will color a portion of the URL in green, while Mozilla will add a green portion to the left of the URL indicating the company name.

High value websites should consider the use of EV certificates to enhance customer confidence in the certificate. It should also be noted that EV certificates do not provide any greater technical security for the TLS. The purpose of the EV certificate is to increase user confidence that the target site is indeed who it claims to be.

Client-Side Certificates

Client side certificates can be used with TLS to prove the identity of the client to the server. Referred to as “two-way TLS”, this configuration requires the client to provide their certificate to the server, in addition to the server providing their’s to the client. If client certificates are used, ensure that the same validation of the client certificate is performed by the server, as indicated for the validation of server certificates above. In addition, the server should be configured to drop the TLS connection if the client certificate cannot be verified or is not provided.

The use of client side certificates is relatively rare currently due to the complexities of certificate generation, safe distribution, client side configuration, certificate revocation and reissuance, and the fact that clients can only authenticate on machines where their client side certificate is installed. Such certificates are typically used for very high value connections that have small user populations.

Certificate and Public Key Pinning

Hybrid and native applications can take advantage of certificate and public key pinning. Pinning associates a host (for example, server) with an identity (for example, certificate or public key), and allows an application to leverage knowledge of the pre-existing relationship. At runtime, the application would inspect the certificate or public key received after connecting to the server. If the certificate or public key is expected, then the application would proceed as normal. If unexpected, the application would stop using the channel and close the connection since an adversary could control the channel or server.

Pinning still requires customary X509 checks, such as revocation, since CRLs and OCSP provides real time status information. Otherwise, an application could possibly (1) accept a known bad certificate; or (2) require an out-of-band update, which could result in a lengthy App Store approval.

Browser based applications are at a disadvantage since most browsers do not allow the user to leverage pre-existing relationships and a priori knowledge. In addition, Javascript and Websockets do not expose methods to for a web app to query the underlying secure connection information (such as the certificate or public key). It is noteworthy that Chromium based browsers perform pinning on selected sites, but the list is currently maintained by the vendor.

For more information, please see the Pinning Cheat Sheet.

Providing Transport Layer Protection for Back End and Other Connections

Although not the focus of this cheat sheet, it should be stressed that transport layer protection is necessary for back-end connections and any other connection where sensitive data is exchanged or where user identity is established. Failure to implement an effective and robust transport layer security will expose sensitive data and undermine the effectiveness of any authentication or access control mechanism.

Secure Internal Network Fallacy

The internal network of a corporation is not immune to attacks. Many recent high profile intrusions, where thousands of sensitive customer records were compromised, have been perpetrated by attackers that have gained internal network access and then used sniffers to capture unencrypted data as it traversed the internal network.




Related Articles

Certificate and Public Key Pinning

Certificate and Public Key Pinning is a technical guide to implementing certificate and public key pinning as discussed at the Virginia chapter’spresentation Securing Wireless Channels in the Mobile Space. This guide is focused on providing clear, simple, actionable guidance for securing the channel in a hostile environment where actors could be malicious and the conference of trust a liability. Additional presentation material includedsupplement with code excerpts, Android sample program, iOS sample program, .Net sample program, and OpenSSL sample program.

A cheat sheet is available at Pinning Cheat Sheet.


Secure channels are a cornerstone to users and employees working remotely and on the go. Users and developers expect end-to-end security when sending and receiving data – especially sensitive data on channels protected by VPN, SSL, or TLS. While organizations which control DNS and CA have likely reduced risk to trivial levels under most threat models, users and developers subjugated to other’s DNS and a public CA hierarchy are exposed to non-trivial amounts of risk. In fact, history has shown those relying on outside services have suffered chronic breaches in their secure channels.

The pandemic abuse of trust has resulted in users, developers and applications making security related decisions on untrusted input. The situation is somewhat of a paradox: entities such as DNS and CAs are trusted and supposed to supply trusted input; yet their input cannot be trusted. Relying on untrusted input for security related decisions is not only bad karma, it violates a number of secure coding principals (see, for example, OWASP’sInjection Theory and Data Validation).

Pinning effectively removes the “conference of trust”. An application which pins a certificate or public key no longer needs to depend on others – such as DNS or CAs – when making security decisions relating to a peer’s identity. For those familiar with SSH, you should realize that public key pinning is nearly identical to SSH’s StrictHostKeyChecking option. SSH had it right the entire time, and the rest of the world is beginning to realize the virtues of directly identifying a host or service by its public key.

Others who actively engage in pinning include Google and its browser Chrome. Chrome was successful in detecting the DigiNotar compromise which uncovered suspected interception by the Iranian government on its citizens. The initial report of the compromise can be found at Is This MITM Attack to Gmail’s SSL?; and Google Security’s immediate response at An update on attempted man-in-the-middle attacks.

What’s the problem?

Users, developers, and applications expect end-to-end security on their secure channels, but some secure channels are not meeting the expectation. Specifically, channels built using well known protocols such as VPN, SSL, and TLS can be vulnerable to a number of attacks.

Examples of past failures are listed on the discussion tab for this article. This cheat sheet does not attempt to catalogue the failures in the industry, investigate the design flaws in the scaffolding, justify the lack of accountability or liability with the providers, explain the race to the bottom in services, or demystify the collusion between, for example, Browsers and CAs. For additional reading, please visit PKI is Broken and The Internet is Broken.

Patient 0

The original problem was the Key Distribution Problem. Insecure communications can be transformed into a secure communication problem with encryption. Encrypted communications can be transformed into an identity problem with signatures. The identity problem terminates at the key distribution problem. They are the same problem.

The Cures

There are three cures for the key distribution problem. First is to have first hand knowledge of your partner or peer (i.e., a peer, server or service). This could be solved with SneakerNet. Unfortunately, SneakerNet does not scale and cannot be used to solve the key distribution problem.

The second is to rely on others, and it has two variants: (1) web of trust, and (2) hierarchy of trust. Web of Trust and Hierarchy of Trust solve the key distribution problem in a sterile environment. However, Web of Trust and Hierarchy of Trust each requires us to rely on others – or confer trust. In practice, trusting others is showing to be problematic.

What Is Pinning?

Pinning is the process of associating a host with their expected X509 certificate or public key. Once a certificate or public key is known or seen for a host, the certificate or public key is associated or ‘pinned’ to the host. If more than one certificate or public key is acceptable, then the program holds a pinset (taking from Jon Larimer and Kenny Root Google I/O talk). In this case, the advertised identity must match one of the elements in the pinset.

A host or service’s certificate or public key can be added to an application at development time, or it can be added upon first encountering the certificate or public key. The former – adding at development time – is preferred since preloading the certificate or public key out of band usually means the attacker cannot taint the pin. If the certificate or public key is added upon first encounter, you will be using key continuity. Key continuity can fail if the attacker has a privileged position during the first first encounter.

Pinning leverages knowledge of the pre-existing relationship between the user and an organization or service to help make better security related decisions. Because you already have information on the server or service, you don’t need to rely on generalized mechanisms meant to solve the key distribution problem. That is, you don’t need to turn to DNS for name/address mappings or CAs for bindings and status. One exception is revocation and it is discussed below in Pinning Gaps.

It is also worth mention that Pinning is not Stapling. Stapling sends both the certificate and OCSP responder information in the same request to avoid the additional fetches the client should perform during path validations.

When Do You Pin?

You should pin anytime you want to be relatively certain of the remote host’s identity or when operating in a hostile environment. Since one or both are almost always true, you should probably pin all the time.

A perfect case in point: during the two weeks or so of preparation for the presentation and cheat sheet, we’ve observed three relevant and related failures. First was Nokia/Opera willfully breaking the secure channel; second was DigiCert issuing a code signing certificate for malware; and third was Bit9’s loss of its root signing key. The environment is not only hostile, its toxic.

When Do You Whitelist?

If you are working for an organization which practices “egress filtering” as part of a Data Loss Prevention (DLP) strategy, you will likely encounterInterception Proxies. I like to refer to these things as “good” bad guys (as opposed to “bad” bad guys) since both break end-to-end security and we can’t tell them apart. In this case, do not offer to whitelist the interception proxy since it defeats your security goals. Add the interception proxy’s public key to your pinset after being instructed to do so by the folks in Risk Acceptance.

Note: if you whitelist a certificate or public key for a different host (for example, to accommodate an interception proxy), you are no longer pinning the expected certificates and keys for the host. Security and integrity on the channel could suffer, and it surely breaks end-to-end security expectations of users and organizations.

For more reading on interception proxies, the additional risk they bestow, and how they fail, see Dr. Matthew Green’s How do Interception Proxies fail? and Jeff Jarmoc’s BlackHat talk SSL/TLS Interception Proxies and Transitive Trust.

How Do You Pin?

The idea is to re-use the existing protocols and infrastructure, but use them in a hardened manner. For re-use, a program would keep doing the things it used to do when establishing a secure connection.

To harden the channel, the program would take advantage of the OnConnect callback offered by a library, framework or platform. In the callback, the program would verify the remote host’s identity by validating its certificate or public key. While pinning does not have to occur in an OnConnectcallback, its often most convenient because the underlying connection information is readily available.

What Should Be Pinned?

The first thing to decide is what should be pinned. For this choice, you have two options: you can (1) pin the certificate; or (2) pin the public key. If you choose public keys, you have two additional choices: (a) pin the subjectPublicKeyInfo; or (b) pin one of the concrete types such asRSAPublicKey or DSAPublicKey.

The three choices are explained below in more detail. I would encourage you to pin the subjectPublicKeyInfo because it has the public parameters (such as {e,n} for an RSA public key) and contextual information such as an algorithm and OID. The context will help you keep your bearings at times, and Figure 1 below shows the additional information available.


For the purposes of this article, the objects are in X509-compatible presentation format (PKCS#1 defers to X509, both of which use ASN.1). If you have a PEM encoded object (for example, -----BEGIN CERTIFICATE-----, -----END CERTIFICATE-----), then convert the object to DER encoding. Conversion using OpenSSL is offered below in Format Conversions.

A certificate is an object which binds an entity (such as a person or organization) to a public key via a signature. The certificate is DER encoded, and has associated data or attributes such as Subject (who is identified or bound), Issuer (who signed it), Validity (NotBefore and NotAfter), and a Public Key.

A certificate has a subjectPublicKeyInfo. The subjectPublicKeyInfo is a key with additional information. The ASN.1 type includes an Algorithm ID, aVersion, and an extensible format to hold a concrete public key. Figures 1 and 2 below show different views of the same RSA key, which is the subjectPublicKeyInfo. The key is for the site, and it is used in the sample programs and listings below.

Figure 1: subjectPublicKeyInfo dumped with dumpans1

Figure 2: subjectPublicKeyInfo under a hex editor

The concrete public key is an encoded public key. The key format will usually be specified elsewhere – for example, PKCS#1 in the case of RSA Public Keys. In the case of an RSA public key, the type is RSAPublicKey and the parameters {e,n} will be ASN.1 encoded. Figures 1 and 2 above clearly show the modulus (n at line 28) and exponent (e at line 289). For DSA, the concrete type is DSAPublicKey and the ASN.1 encoded parameters would be {p,q,g,y}.

Final takeaways: (1) a certificate binds an entity to a public key; (2) a certificate has a subjectPublicKeyInfo; and (3) a subjectPublicKeyInfo has an concrete public key. For those who want to learn more, a more in-depth discussion from a programmer’s perspective can be found at the Code Project’s article Cryptographic Interoperability: Keys.



The certificate is easiest to pin. You can fetch the certificate out of band for the website, have the IT folks email your company certificate to you, use openssl s_client to retrieve the certificate etc. When the certificate expires, you would update your application. Assuming your application has no bugs or security defects, the application would be updated every year or two.

At runtime, you retrieve the website or server’s certificate in the callback. Within the callback, you compare the retrieved certificate with the certificate embedded within the program. If the comparison fails, then fail the method or function.

There is a downside to pinning a certificate. If the site rotates its certificate on a regular basis, then your application would need to be updated regularly. For example, Google rotates its certificates, so you will need to update your application about once a month (if it depended on Google services). Even though Google rotates its certificates, the underlying public keys (within the certificate) remain static.

Public Key

Public Key

Public key pinning is more flexible but a little trickier due to the extra steps necessary to extract the public key from a certificate. As with a certificate, the program checks the extracted public key with its embedded copy of the public key.

There are two downsides two public key pinning. First, its harder to work with keys (versus certificates) since you usually must extract the key from the certificate. Extraction is a minor inconvenience in Java and .Net, buts its uncomfortable in Cocoa/CocoaTouch and OpenSSL. Second, the key is static and may violate key rotation policies.


While the three choices above used DER encoding, its also acceptable to use a hash of the information (or other transforms). In fact, the original sample programs were written using digested certificates and public keys. The samples were changed to allow a programmer to inspect the objects with tools like dumpasn1 and other ASN.1 decoders.

Hashing also provides three additional benefits. First, hashing allows you to anonymize a certificate or public key. This might be important if you application is concerned about leaking information during decompilation and re-engineering.

Second, a digested certificate fingerprint is often available as a native API for many libraries, so its convenient to use.

Finally, an organization might want to supply a reserve (or back-up) identity in case the primary identity is compromised. Hashing ensures your adversaries do not see the reserved certificate or public key in advance of its use. In fact, Google’s IETF draft websec-key-pinning uses the technique.

What About X509?

PKI{X} and the Internet form an intersection. What Internet users expect and what they receive from CAs could vary wildly. For example, an Internet user has security goals, while a CA has revenue goals and legal goals. Many are surprised to learn that the user is often required to perform host identity verification even though the CA issued the certificate (the details are buried in CA warranties on their certificates and their Certification Practice Statement (CPS)).

There are a number of PKI profiles available. For the Internet, “Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL)”, also known as RFC 5280, is of interest. Since a certificate is specified in the ITU’s X509 standard, there are lots of mandatory and optional fields available for validation from both bodies. Because of the disjoint goals among groups, the next section provides guidance.

Mandatory Checks

All X509 verifications must include:

  • A path validation check. The check verifies all the signatures on certificates in the chain are valid under a given PKI. The check begins at the server or service’s certificate (the leaf), and proceeds back to a trusted root certificate (the root).
  • A validity check, or the notBefore and notAfter fields. The notAfter field is especially important since a CA will not warrant the certificate after the date, and it does not have to provide CRL/OCSP updates after the date.
  • Revocation status. As with notAfter, revocation is important because the CA will not warrant a certificate once it is listed as revoked. The IETF approved way of checking a certificate’s revocation is OCSP and specified in RFC 2560.

Optional Checks

[Mulling over what else to present, and the best way to present it. Subject name? DNS lookups? Key Usage? Algorithms? Geolocation based on IP? Check back soon.] In the model which pre-dated PKIX RFC-5280, X.509v1 there was strong binding of the certificate Subject name to the X.500 Directory. With the update to X.509v3, the Directory is still the standard for authentication of caCertificate attributes, versus accepting a self signed root. Geo-location is important, the fake certificate for Google was given a location of Florida, instead of Mountain View, CA. The binding of the certificate to the Directory can anchor the root caCertificate, in effect “pin” it, to a valid entity that can have demonstrable attributes such as location. This is detailed in RFC-1255. Additional fields specified, such as the subject alternative field, for example a RFC-822 email address, or DNS name, can be located in the DNS, but the actual heavy lifting is done by the X.500 Directory, which is used currently as a cross-certificate trust conduit at the Federal Bridge between major communities of interest, that are not Internet focused. While those cross-certificates are valuable in validation between trust communities, a self-signed root, still needs to be either pinned, curated in trust bundle such as in web browser software secure storage or represented by a federated community. The Directory can play a role to fill in gaps to validate caCertificates, either locally, or nationally under an administrative domain such as c=US. By divorcing the subject from the Directory entry, problems begin to arise in which pinning plays a key role to ensure that client and server have the same reference points.

Public Key Checks

Quod vide (q.v.). Verifying the identity of a host with knowledge of its associated/expected public key is pinning.

Examples of Pinning

This section demonstrates certificate and public key pinning in Android Java, iOS, .Net, and OpenSSL. All programs attempt to connect and fetch bytes (Dr. Mads Haahr participates in AOSP’s pinning program, so the site should have a static key). The programs enjoy a pre-existing relationship with the site (more correctly, a priori knowledge), so they include a copy of the site’s public key and pin the identity on the key.

Parameter validation, return value checking, and error checking have been omitted in the code below, but is present in the sample programs. So the sample code is ready for copy/paste. By far, the most uncomfortable languages are C-based: iOS and OpenSSL.

HTTP pinning

RFC 7469 introduced a new HTTP header that allows SSL servers to declare hashes of their certificates with time scope in which these certificates should not be changed. For example:

      Public-Key-Pins: max-age=2592000;

Please note that RFC 7469 is controversial since it allows overrides for locally installed authorities. That is, it allows an adversary or other party who successfully phishes the user to override a known good pinset with non-authentic or fraudulent information. Second, the reporting mechanism is suppressed from broken pinsets, so a complying user agent will be complicit in the cover up after the fact. That is, the reporting of the broken pinset is called out as MUST NOT report [1].


Pinning in Android is accomplished through a custom X509TrustManager. X509TrustManager should perform the customary X509 checks in addition to performing the pin.

Download: Android sample program

public final class PubKeyManager implements X509TrustManager
  private static String PUB_KEY = "30820122300d06092a864886f70d0101" +
    "0105000382010f003082010a0282010100b35ea8adaf4cb6db86068a836f3c85" +
    "5a545b1f0cc8afb19e38213bac4d55c3f2f19df6dee82ead67f70a990131b6bc" +
    "ac1a9116acc883862f00593199df19ce027c8eaaae8e3121f7f329219464e657" +
    "2cbf66e8e229eac2992dd795c4f23df0fe72b6ceef457eba0b9029619e0395b8" +
    "609851849dd6214589a2ceba4f7a7dcceb7ab2a6b60c27c69317bd7ab2135f50" +
    "c6317e5dbfb9d1e55936e4109b7b911450c746fe0d5d07165b6b23ada7700b00" +
    "33238c858ad179a82459c4718019c111b4ef7be53e5972e06ca68a112406da38" +
    "cf60d2f4fda4d1cd52f1da9fd6104d91a34455cd7b328b02525320a35253147b" +

  public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException
    if (chain == null) {
      throw new IllegalArgumentException("checkServerTrusted: X509Certificate array is null");

    if (!(chain.length > 0)) {
      throw new IllegalArgumentException("checkServerTrusted: X509Certificate is empty");

    if (!(null != authType && authType.equalsIgnoreCase("RSA"))) {
      throw new CertificateException("checkServerTrusted: AuthType is not RSA");

    // Perform customary SSL/TLS checks
    try {
      TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
      tmf.init((KeyStore) null);
      for (TrustManager trustManager : tmf.getTrustManagers()) {
        ((X509TrustManager) trustManager).checkServerTrusted(chain, authType);
    } catch (Exception e) {
      throw new CertificateException(e);

    // Hack ahead: BigInteger and toString(). We know a DER encoded Public Key begins
    // with 0x30 (ASN.1 SEQUENCE and CONSTRUCTED), so there is no leading 0x00 to drop.
    RSAPublicKey pubkey = (RSAPublicKey) chain[0].getPublicKey();
    String encoded = new BigInteger(1 /* positive */, pubkey.getEncoded()).toString(16);

    // Pin it!
    final boolean expected = PUB_KEY.equalsIgnoreCase(encoded);
    if (!expected) {
      throw new CertificateException("checkServerTrusted: Expected public key: "
                + PUB_KEY + ", got public key:" + encoded);

PubKeyManager would be used in code similar to below.

TrustManager tm[] = { new PubKeyManager() };

SSLContext context = SSLContext.getInstance("TLS");
context.init(null, tm, null);

URL url = new URL( "" +

HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();

InputStreamReader instream = new InputStreamReader(connection.getInputStream());
StreamTokenizer tokenizer = new StreamTokenizer(instream);


iOS pinning is performed through a NSURLConnectionDelegate. The delegate must implementconnection:canAuthenticateAgainstProtectionSpace: and connection:didReceiveAuthenticationChallenge:. Withinconnection:didReceiveAuthenticationChallenge:, the delegate must call SecTrustEvaluate to perform customary X509 checks.

Download: iOS sample program.

    NSString* requestString = @"
    NSURL* requestUrl = [NSURL URLWithString:requestString];

    NSURLRequest* request = [NSURLRequest requestWithURL:requestUrl

    NSURLConnection* connection = [[NSURLConnection alloc] initWithRequest:request delegate:self];

-(BOOL)connection:(NSURLConnection *)connection canAuthenticateAgainstProtectionSpace:
    return [[space authenticationMethod] isEqualToString: NSURLAuthenticationMethodServerTrust];

- (void)connection:(NSURLConnection *)connection didReceiveAuthenticationChallenge:
                   (NSURLAuthenticationChallenge *)challenge
  if ([[[challenge protectionSpace] authenticationMethod] isEqualToString: NSURLAuthenticationMethodServerTrust])
      SecTrustRef serverTrust = [[challenge protectionSpace] serverTrust];
      if(nil == serverTrust)
        break; /* failed */

      OSStatus status = SecTrustEvaluate(serverTrust, NULL);
      if(!(errSecSuccess == status))
        break; /* failed */

      SecCertificateRef serverCertificate = SecTrustGetCertificateAtIndex(serverTrust, 0);
      if(nil == serverCertificate)
        break; /* failed */

      CFDataRef serverCertificateData = SecCertificateCopyData(serverCertificate);
      [(id)serverCertificateData autorelease];
      if(nil == serverCertificateData)
        break; /* failed */

      const UInt8* const data = CFDataGetBytePtr(serverCertificateData);
      const CFIndex size = CFDataGetLength(serverCertificateData);
      NSData* cert1 = [NSData dataWithBytes:data length:(NSUInteger)size];

      NSString *file = [[NSBundle mainBundle] pathForResource:@"random-org" ofType:@"der"];
      NSData* cert2 = [NSData dataWithContentsOfFile:file];

      if(nil == cert1 || nil == cert2)
        break; /* failed */

      const BOOL equal = [cert1 isEqualToData:cert2];
        break; /* failed */

      // The only good exit point
      return [[challenge sender] useCredential: [NSURLCredential credentialForTrust: serverTrust]
                    forAuthenticationChallenge: challenge];
    } while(0);

    // Bad dog
    return [[challenge sender] cancelAuthenticationChallenge: challenge];


.Net pinning can be achieved by using ServicePointManager as shown below.

Download: .Net sample program.

// Encoded RSAPublicKey
private static String PUB_KEY = "30818902818100C4A06B7B52F8D17DC1CCB47362" +
    "C64AB799AAE19E245A7559E9CEEC7D8AA4DF07CB0B21FDFD763C63A313A668FE9D764E" +
    "D913C51A676788DB62AF624F422C2F112C1316922AA5D37823CD9F43D1FC54513D14B2" +
    "9E36991F08A042C42EAAEEE5FE8E2CB10167174A359CEBF6FACC2C9CA933AD403137EE" +

public static void Main(string[] args)
  ServicePointManager.ServerCertificateValidationCallback = PinPublicKey;
  WebRequest wr = WebRequest.Create("");

public static bool PinPublicKey(object sender, X509Certificate certificate, X509Chain chain,
                                SslPolicyErrors sslPolicyErrors)
  if (null == certificate)
    return false;

  String pk = certificate.GetPublicKeyString();
  if (pk.Equals(PUB_KEY))
    return true;

  // Bad dog
  return false;


Pinning can occur at one of two places with OpenSSL. First is the user supplied verify_callback. Second is after the connection is established via SSL_get_peer_certificate. Either method will allow you to access the peer’s certificate.

Though OpenSSL performs the X509 checks, you must fail the connection and tear down the socket on error. By design, a server that does not supply a certificate will result in X509_V_OK with a NULL certificate. To check the result of the customary verification: (1) you must callSSL_get_verify_result and verify the return code is X509_V_OK; and (2) you must call SSL_get_peer_certificate and verify the certificate is non-NULL.

Download: OpenSSL sample program.

int pkp_pin_peer_pubkey(SSL* ssl)
    if(NULL == ssl) return FALSE;
    X509* cert = NULL;
    FILE* fp = NULL;
    /* Scratch */
    int len1 = 0, len2 = 0;
    unsigned char *buff1 = NULL, *buff2 = NULL;
    /* Result is returned to caller */
    int ret = 0, result = FALSE;
        /* */
        cert = SSL_get_peer_certificate(ssl);
        if(!(cert != NULL))
            break; /* failed */
        /* Begin Gyrations to get the subjectPublicKeyInfo       */
        /* Thanks to Viktor Dukhovni on the OpenSSL mailing list */
        /* */
        len1 = i2d_X509_PUBKEY(X509_get_X509_PUBKEY(cert), NULL);
        if(!(len1 > 0))
            break; /* failed */
        /* scratch */
        unsigned char* temp = NULL;
        /* */
        buff1 = temp = OPENSSL_malloc(len1);
        if(!(buff1 != NULL))
            break; /* failed */
        /* */
        len2 = i2d_X509_PUBKEY(X509_get_X509_PUBKEY(cert), &temp);

        /* These checks are verifying we got back the same values as when we sized the buffer.      */
        /* Its pretty weak since they should always be the same. But it gives us something to test. */
        if(!((len1 == len2) && (temp != NULL) && ((temp - buff1) == len1)))
            break; /* failed */
        /* End Gyrations */
        /* See the warning above!!!                                            */
        /* */
        fp = fopen("random-org.der", "rx");
        if(NULL ==fp) {
            fp = fopen("random-org.der", "r");
        if(!(NULL != fp))
            break; /* failed */
        /* Seek to eof to determine the file's size                            */
        /* */
        ret = fseek(fp, 0, SEEK_END);
        if(!(0 == ret))
            break; /* failed */
        /* Fetch the file's size                                               */
        /* */
        long size = ftell(fp);

        /* Arbitrary size, but should be relatively small (less than 1K or 2K) */
        if(!(size != -1 && size > 0 && size < 2048))
            break; /* failed */
        /* Rewind to beginning to perform the read                             */
        /* */
        ret = fseek(fp, 0, SEEK_SET);
        if(!(0 == ret))
            break; /* failed */
        /* Re-use buff2 and len2 */
        buff2 = NULL; len2 = (int)size;
        /* */
        buff2 = OPENSSL_malloc(len2);
        if(!(buff2 != NULL))
            break; /* failed */
        /* */
        /* Returns number of elements read, which should be 1 */
        ret = (int)fread(buff2, (size_t)len2, 1, fp);
        if(!(ret == 1))
            break; /* failed */
        /* Re-use size. MIN and MAX macro below... */
        size = len1 < len2 ? len1 : len2;
        /*****    PAYDIRT    *****/
        if(len1 != (int)size || len2 != (int)size || 0 != memcmp(buff1, buff2, (size_t)size))
            break; /* failed */
        /* The one good exit point */
        result = TRUE;
    } while(0);
    if(fp != NULL)
    /* */
    if(NULL != buff2)
    /* */
    if(NULL != buff1)
    /* */
    if(NULL != cert)
    return result;

Pinning Alternatives

Not all applications use split key cryptography. Fortunately, there are protocols which allow you to set up a secure channel based on knowledge of passwords and pre-shared secrets (rather than putting the secret on the wire in a basic authentication scheme). Two are listed below – SRP and PSK. SRP and PSK have 88 cipher suites assigned to them by IANA for TLS, so there’s no shortage of choices.

Figure 3: IANA reserved cipher suites for SRP and PSK


Secure Remote Password (SRP) is a Password Authenticated Key Exchange (PAKE) by Thomas Wu based upon Diffie-Hellman. The protocol is standardized in RFC 5054 and available in the OpenSSL library (among others). In the SRP scheme, the server uses a verifier which consists of a{salt, hash(password)} pair. The user has the password and receives the salt from the server. With lots of hand waving, both parties select per-instance random values (nonces) and execute the protocol using g{(salt + password)|verifier} + nonces rather than traditional Diffie-Hellman using gab.


Diffie-Hellman based schemes are part of a family of problems based on Discrete Logs (DL), which are logarithms over a finite field. DL schemes are appealing because they are known to be hard (unless P=NP, which would cause computational number theorists to have a cow).


PSK is Pre-Shared Key and specified in RFC 4279 and RFC 4764. The shared secret is used as a pre-master secret in TLS-PSK for SSL/TLS; or used to key a block cipher in EAP-PSK. EAP-PSK is designed for authentication over insecure networks such as IEEE 802.11.


This sections covers administrivia and miscellaneous items related to pinning.

Ephemeral Keys

Ephemeral keys are temporary keys used for one instance of a protocol execution and then thrown away. An ephemeral key has the benefit of providing forward secrecy, meaning a compromise of the site or service’s long term (static) signing key does not facilitate decrypting past messages because the key was temporary and discarded (once the session terminated).

Ephemeral keys do not affect pinning because the Ephemeral key is delivered in a separate ServerKeyExchange message. In addition, the ephemeral key is a key and not a certificate, so it does not change the construction of the certificate chain. That is, the certificate of interest will still be located at certificates[0].

Pinning Gaps

There are two gaps when pinning due to reuse of the existing infrastructure and protocols. First, an explicit challenge is not sent by the program to the peer server based on the server’s public information. So the program never knows if the peer can actually decrypt messages. However, the shortcoming is usually academic in practice since an adversary will receive messages it can’t decrypt.

Second is revocation. Clients don’t usually engage in revocation checking, so it could be possible to use a known bad certificate or key in a pinset. Even if revocation is active, Certificate Revocation Lists (CRLs) and Online Certificate Status Protocol (OCSP) can be defeated in a hostile environment. An application can take steps to remediate, with the primary means being freshness. That is, an application should be updated and distributed immediately when a critical security parameter changes.

No Relationship ^@$!

If you don’t have a pre-existing relationship, all is not lost. First, you can pin a host or server’s certificate or public key the first time you encounter it. If the bad guy was not active when you encountered the certificate or public key, he or she will not be successful with future funny business.

Second, bad certificates are being spotted quicker in the field due to projects like Chromium and Certificate Patrol, and initiatives like the EFF’sSSL Observatory.

Third, help is on its way, and there are a number of futures that will assist with the endeavors:

  • Public Key Pinning ( – an extension to the HTTP protocol allowing web host operators to instruct user agents (UAs) to remember (“pin”) the hosts’ cryptographic identities for a given period of time.
  • DNS-based Authentication of Named Entities (DANE) ( – uses Secure DNS to associate Certificates with Domain Names For S/MIME, SMTP with TLS, DNSSEC and TLSA records.
  • Sovereign Keys ( – operates by providing an optional and secure way of associating domain names with public keys via DNSSEC. PKI (hierarchical) is still used. Semi-centralized with append only logging.
  • Convergence ( – different [geographical] views of a site and its associated data (certificates and public keys). Web of Trust is used. Semi-centralized.

While Sovereign Keys and Convergence still require us to confer trust to outside parties, the parties involved do not serve share holders or covet revenue streams. Their interests are industry transparency and user security.

More Information?

Pinning is an old new thing that has been shaken, stirred, and repackaged. While “pinning” and “pinsets” are relatively new terms for old things, Jon Larimer and Kenny Root spent time on the subject at Google I/O 2012 with their talk Security and Privacy in Android Apps.

Format Conversions

As a convenience to readers, the following with convert between PEM and DER format using OpenSSL.

# Public key, X509
$ openssl genrsa -out rsa-openssl.pem 3072
$ openssl rsa -in rsa-openssl.pem -pubout -outform DER -out rsa-openssl.der
# Private key, PKCS#8
$ openssl genrsa -out rsa-openssl.pem 3072
$ openssl pkcs8 -nocrypt -in rsa-openssl.pem -inform PEM -topk8 -outform DER -out rsa-openssl.der


Mitigation Guidelines for Advanced Persistent Threats


This Technical Report is intended for IT professionals and managers within federal, provincial/territorial and municipal governments; critical infrastructure; and other related industries.


Recent media disclosures have reported numerous high-profile computer compromises attributed to entities identified as Advanced Persistent Threat (APT). The intent of this product is to define APT, to describe typical APT attack methodologies and to introduce mitigation and monitoring techniques that may reduce the risk to organizations. While APT actors have traditionally targeted government, military and defence industrial sectors, any organization may be of interest to and be targeted by APT actors.

Defining APT

APT can be defined by breaking it down into its component parts:

Threat. The Government of Canada defines a threat as “an event or act, deliberate or accidental, that could cause injury to people, information, assets or services.” Since APTs are fundamentally human threat activity with a purpose, they are considered deliberate. As such, APT actors have a capability—resources, skills and knowledge—as well as intent. The level of capability and intent of APT actors are generally associated to those possessed by a nation-state. Historically, reported targeted sectors consisted of government, military and defence industrial base organizations. Increasingly, other sectors, including natural resources, energy , transportation, communications and information security , as well as research and development and law firms involved in international mergers and acquisitions, are being targeted. The intent is usually to obtain information from targeted organizations in order to provide the APT sponsor with a strategic, diplomatic, military, competitive, technological or economic advantage.

Advanced. Advanced can be broken down into two sub-components. It is generally thought that “advanced” refers to the malicious code associated with an APT. This is partially true in that APT actors may have access to complex customized code that is readily adaptable to evade antivirus and intrusion detection defences. However, in many cases, APT actors use code that is obtained from publicly available malicious code “kits” and software developed to exploit common vulnerabilities. The second sub-component relates to the techniques used by APT actors to select their targets and refine the attack such that selected individuals in targeted organizations become the conduit to the organization’s information assets. These social engineering attacks are well-crafted, focus on a topic of interest to the selected individual and contain information that appears credible to the recipient.

Persistent. Persistent consists of two sub-components as well. First, the APT actor may conduct significant and focussed reconnaissance on the target organization in order to select appropriate recipients and to obtain sufficient information to entice them unknowingly to the initial attack delivery mechanism. Second, following a successful delivery of the initial attack and compromise of an organization, APT actors may stealthily move laterally within the network to collect information about its topology, technologies and privileged account holders, while minimizing the digital footprints associated with their activities. This low-and-slow approach may help the APT actor remain undetected and maintain long-term access to the compromised organization.

Chronology of an APT attack

In a recent report, the antivirus firm McAfee has provided analysis of a specific targeted attack. This report, along with other findings such as those from the U.S.-China Economic and Security Review Commission can be summarized into the following steps:

  • Reconnaissance: An APT actor will seek to find individuals that may be a viable conduit into the targeted organization. APT actors will use information available on an organization’s website, partner websites and social media sites to develop an organization chart. They may also use business cards, conference registration information or information obtained from a previous cyber compromise. Emails and instant messages may also be used to provide targeting information. These sources may also be used to obtain details of an organization’s objectives, projects, contracts, partners and customers to develop practical social engineering attacks. In some targeted attacks, an employee may even be recruited or blackmailed into providing access.
  • Social engineering and targeted malicious code delivery. Using the information obtained during the reconnaissance phase, the APT actor may send emails to specific individuals within the organization. These emails may appear to have originated from a known or trusted source, may contain a subject line and text relevant to the recipient and may even contain a valid signature block. These emails typically contain an attachment or web hyperlink that, when accessed, potentially performs various steps aimed at compromising the recipient’s workstation. The attachment may be an original document from the organization or a partner that has been modified with malicious code (i.e. trojanized). In many reported cases, the malicious code exploited a vulnerability for which a vendor patch was readily available.
  • Establish a covert backdoor. Once a system has been compromised, the APT actor may attempt to gain elevated privileges. The APT actor may move laterally throughout the network and install additional malicious code where this can be done without raising suspicion or alarms. Time delay may be used to ensure pre-coded external malicious infrastructure components awaiting connections from compromised hosts are not all accessed at once and easily identified.
  • Establish command and control infrastructure. Once sufficient privileges have been obtained, the APT actor may install additional tools, such as keyloggers and remote administration tools (RAT), and establish an encrypted communications path to the APT command and control infrastructure.
  • Achieve objective. Depending on the objectives, the APT actor may exfiltrate information, modify documents or take control of critical systems. Most reported compromises attributed to APT actors have resulted in data exfiltration. A search for files potentially containing the targeted information, such as productivity software suite documents and emails, may be conducted, and the results transferred to an exfiltration point, or staging server, within the compromised network. This server may be selected among those normally associated with high volumes of traffic to avoid suspicion and limit the number of channels to the external command and control infrastructure. Files of interest are generally compressed and encrypted before being exfiltrated.
  • Maintain presence. Once the targeted information has been exfiltrated, the APT may undertake considerable effort to maintain a long-term presence. This may include minimizing command and control communications, re-compromising restored systems, updating installed malicious code to evade antivirus detection, and monitoring systems for new passwords and other changes. Techniques such as rootkit installation using trojanized binaries, registry modification and use of Microsoft Windows Services are leveraged to maintain a hidden presence.

High return on investment strategies to reduce APT risk

This section outlines high return on investment strategies to reduce the risk presented by APTs. For a more comprehensive list of mitigation strategies, refer to the Comprehensive Mitigation Strategies list provided at the end of this document.

  1. Education and awareness.  Although its effectiveness may be questioned, a well-designed and implemented user awareness program focussed at raising the diligence and suspicion of employees may be the most cost-effective sensor an organization can have against APTs.  A physical analogy is having cameras, access controls and locks but having an unaware employee “hold the door open” for someone unknown.  Employees should be suspicious of emails containing attachments or hyperlinks.  Some specific points to emphasize include:
    • a. Have employees ask themselves: “In my role in the organization, should I expect to receive an email with this information?”  For example, does it make sense for an assistant employed in Human Resources to receive a corporate intelligence report from an overseas partner?
      *Note: In many organizations, executive assistants and other administrative staff may be tasked with reviewing emails and associated file attachments. For these positions, consider using a non-standard application viewer capable of displaying common file types such as MS Office and Adobe documents.
    • b. Have employees look at the sending email address.  It may contain the correct name of a supervisor or partner. The email may even contain a valid signature block.  But, if the sending email address is from a free email service, such as Hotmail, Yahoo! or Gmail, the employee should ask themselves: “Would my manager send a business-related email from one of these services”?
    • c. Have employees seek confirmation.  When an employee is in doubt, phone the sender and ask “Did you send me this email?”  In an organization educated and aware of APTs, this is not be perceived as an odd question.
    • d. Make employees aware of Help Desk policies.  Employees should be aware that the Help Desk will never call or send an email asking for a password.  Train employees to report suspicious activity to the Help Desk, prior to opening attachments or clicking on links contained in an email.
    • e. Monitor and validate the organization’s awareness.  Review publicly accessible directories, organization charts and documents.  APT actors often use publicly available information to develop targeted attacks. Consider developing test phishing emails and using them to conduct routine user-awareness validation for high-risk employees such as executives and network administrators.
  2. Patch systems.  While some APT attacks utilize zero-day exploits , many reported APT compromises have exploited a vulnerability where the vendor patch was available. Special attention should be paid to patching end-user baseline applications such as Microsoft Office and Adobe products.
  3. Configure systems securely.  In addition to patching systems quickly, there are a number of basic configuration best practices that can reduce the risk of a compromise by an APT.  The following recommendations are generic in nature.  Guidance documentation is available from a number of sources depending on the operating system used in your organization.
    • a. Minimize administrative privileges.  The likelihood of a successful compromise by an APT is reduced if the APT actor cannot obtain administrative credentials.  Few employees should have a legitimate requirement for these privileges in the performance of their day-to-day activities.  For those who do, a separate account should be provided.  The administrator account should not be used for email or web browsing.
    • b. Whitelist applications.  Tools and configuration options are available to prevent the unauthorized installation of software.
    • c. Encrypt data.  When at rest or in transit across a network, an approved encryption algorithm should be used to preserve the confidentiality of information.
    • d. Disable unnecessary services.
    • e. Implement file hashing on critical devices.  APT actors may use installation or modification of files or services on compromised hosts to maintain persistence.  A file integrity checker can help identify files that have been modified.
  4. Maintain the network perimeter. A properly configured perimeter can reduce the risk presented by an APT. The configuration should balance legitimate business requirements with information security best practices. The devices on the perimeter must be monitored regularly. The following are some APT and general cyber security-related perimeter guidelines.
    • a. Deploy an e-mail gateway. The gateway should include an antivirus scanner, preferably different from the antivirus solution used internally. Use a Sender Policy Framework. . Configure the gateway to reject emails originating externally but with an internal email address. Block executable or otherwise dangerous attachments.
    • b. Deploy a web content filter. The filter should include an active content scanning capability. Examine the feasibility of whitelisting approved websites to support business and acceptable usage policies and then block all others. This is especially relevant for HTTPS-enabled websites and dynamic DNS websites. Block the download of executable, difficult-to-inspect files such as password-protected compressed files, and otherwise dangerous file types.
    • c. Actively monitor perimeter devices. Preferably via a centralized logging capability integrated with an event correlation engine.
    • d. Implement DNS proxy with sinkhole capability. An attacker’s infrastructure may leverage fluxing of the resolving IP address to impair investigative efforts. Since domain names may be associated with an APT, a DNS sinkhole can be rapidly leveraged to prevent successful connections to these domains and monitor internal hosts that are potentially infected.

Indications of potential APT compromise

For the detection of any malicious activity, centralized logging of events and the deployment of a correlation engine are valuable tools to provide the required visibility.  The resulting events, such as alerts generated by Antivirus and Intrusion Detection/Prevention Systems, must be monitored and investigated accordingly. Additional indicators of the APT presence include events associated with abnormal host behaviours, especially after hours, such as sudden changes in server performance—CPU usage, disk space and disk input-output operations of key information stores and network traffic. More specifically, organizations may consider the following indicators:

  1. Unexpected encrypted traffic leaving the organization. Encrypted traffic, using SSL/TLS, should only be permitted to authorized websites that support organizational objectives or acceptable use policy. HTTPS traffic to other websites, or encrypted traffic detected during a regular HTTP session, should be investigated.
  2. Large outbound data transfers via HTTP/HTTPS. HTTP and HTTPS are characterized by asymmetric traffic during most expected use cases. Typically, a small outbound request by a user generates a large response in return. Unless the communicating device is a web server, any device that sends large volumes of data outbound via either HTTP or HTTPS should be examined for compromise.
  3. Login to a critical network device that is not attributable to an authorized employee. Critical devices include domain controllers, routers, network security devices and password vaults.  All login events, whether successful or unsuccessful, should be logged on a centralized logging device (i.e. not the device that may just have been compromised). These logs should be reviewed and reconciled with administrator activity regularly.
  4. Unexplained remote access activity.  Service Desks will often use Remote Desktop Protocol (RDP) or VNC to troubleshoot computer problems.  The use of these tools by Service Desk staff should also be logged. Unexpected RDP or VNC traffic should be investigated.
  5. Unusual web communications, DNS resolution and user agent strings. Hosts attempting to establish a communication channel using DNS requests to unknown DNS servers, trying to connect directly rather than use enterprise web proxy, or using non-standard User Agent strings such as one that includes the internal host name may be signs of security issues that could be associated with APT activity. Knowing a network traffic baseline is important in order to determine whether such unusual behaviour is legitimate or not.

Comprehensive mitigation strategies

Australian Government’s Defence Signals Directorate (DSD) has produced, and regularly updates, a valuable list of recommended strategies to mitigate APTs.  Below is the DSD list in order of effectiveness, as assessed by DSD .  Reportedly, as much as 85 percent of targeted attacks could have been mitigated by implementing the first four strategies.

  1. Patch applications e.g. PDF viewer, Flash Player, Microsoft Office and Java. Patch or mitigate within two days for high risk vulnerabilities. Use the latest version of applications.
  2. Patch operating system vulnerabilities. Patch or mitigate within two days for high risk vulnerabilities. Use the latest operating system version.
  3. Minimise the number of users with domain or local administrative privileges. Such users should use a separate unprivileged account for email and web browsing.
  4. Application whitelisting to help prevent malicious software and other unapproved programs from running e.g. by using Microsoft Software Restriction Policies or AppLocker.
  5. Host-based Intrusion Detection/Prevention System to identify anomalous behaviour such as process injection, keystroke logging, driver loading and call hooking.
  6. Whitelisted email content filtering allowing only attachment types required for business functionality. Preferably convert/sanitise PDF and Microsoft Office attachments.
  7. Block spoofed emails using Sender Policy Framework checking of incoming emails, and a “hard fail” SPF record to help prevent spoofing of your organisation’s domain.
  8. User education e.g. Internet threats and spear phishing socially engineered emails. Avoid: weak passphrases, passphrase reuse, exposing email addresses, unapproved USB devices.
  9. Web content filtering of incoming and outgoing traffic, using signatures, reputation ratings and other heuristics, and whitelisting allowed types of web content.
  10. Web domain whitelisting for all domains, since this approach is more proactive and thorough than blacklisting a tiny percentage of malicious domains.
  11. Web domain whitelisting for HTTPS/SSL domains, since this approach is more proactive and thorough than blacklisting a tiny percentage of malicious domains.
  12. Workstation inspection of Microsoft Office files for abnormalities e.g. using the Microsoft Office File Validation feature.
  13. Application based workstation firewall, configured to deny traffic by default, to protect against malicious or otherwise unauthorised incoming network traffic.
  14. Application based workstation firewall, configured to deny traffic by default, that whitelists which applications are allowed to generate outgoing network traffic.
  15. Network segmentation and segregation into security zones to protect sensitive information and critical services such as user authentication and user directory information.
  16. Multi-factor authentication especially implemented for when the user is about to perform a privileged action, or access a database or other sensitive information repository.
  17. Randomised local administrator passphrases that are unique and complex for all computers. Use domain group privileges instead of local administrator accounts.
  18. Enforce a strong passphrase policy covering complexity, length, and avoiding both passphrase reuse and the use of dictionary words.
  19. Border gateway using an IPv6-capable firewall to prevent computers directly accessing the Internet except via a split DNS server, an email server, or an authenticated web proxy.
  20. Data Execution Prevention using hardware and software mechanisms for all software applications that support DEP.
  21. Antivirus software with up to date signatures, reputation ratings and other heuristic detection capabilities. Use gateway and desktop antivirus software from different vendors.
  22. Non-persistent virtualised trusted operating environment with limited access to network file shares, for risky activities such as reading email and web browsing.
  23. Centralised and time-synchronised logging of allowed and blocked network activity, with regular log analysis, storing logs for at least 18 months.
  24. Centralised and time-synchronised logging of successful and failed computer events, with regular log analysis, storing logs for at least 18 months.
  25. Standard Operating Environment with unrequired operating system functionality disabled e.g. IPv6, autorun and Remote Desktop. Harden file and registry permissions.
  26. Workstation application security configuration hardening e.g. disable unrequired features in PDF viewers, Microsoft Office applications, and web browsers.
  27. Restrict access to NetBIOS services running on workstations and on servers where possible.
  28. Server application security configuration hardening e.g. databases, web applications, customer relationship management and other data storage systems.
  29. Removable and portable media control as part of a Data Loss Prevention strategy, including storage, handling, whitelisting allowed USB devices, encryption and destruction.
  30. TLS encryption between email servers to help prevent legitimate emails being intercepted and used for social engineering. Perform content scanning after email traffic is decrypted.
  31. Disable LanMan password support and cached credentials on workstations and servers, to make it harder for adversaries to crack password hashes.
  32. Block attempts to access web sites by their IP address instead of by their domain name.
  33. Network-based Intrusion Detection/Prevention System using signatures and heuristics to identify anomalous traffic both internally and crossing network perimeter boundaries.
  34. Gateway blacklisting to block access to known malicious domains and IP addresses, including dynamic and other domains provided free to anonymous Internet users.
  35. Full network traffic capture to perform post-incident analysis of successful intrusions, storing network traffic for at least the previous seven days.

Additional resources

CCIRC recommends that computer owners implement security best practices such as those mentioned above to help ensure their computer systems and associated infrastructure remain secure.

Mac Linux USB Loader

Mac Linux USB Loader

Tool allowing you to put a Linux distribution on a USB drive and make it bootable on Intel Macs using EFI.

Mac Linux USB Loader logo

General Information

This is the Mac Linux USB Loader, a tool allowing you to take an ISO of a Linux distribution and make it boot using EFI. It requires a single USB drive formatted as FAT with at least 2 GB free recommended. Mac Linux USB Loader is available under the 3-clause BSD license.

The tool is necessary to make certain Linux distributions boot that do not have EFI booting support. Many distributions are adding this with the release of Windows 8, but it has not been finalized and is still nonstandard by most distributions. Many common distributions are supported, like Ubuntu and Linux Mint.

If you wish to contribute to the code or fork the repository, please do so. All development currently takes place on the master branch, and this is where code should be submitted for pull requests. The legacybranch contains the code for pre-3.0 versions of Mac Linux USB Loader; it will not be maintained and is present for historical interest only.

I created this tool, if you care, for several reasons:

  • None of the other tools available (esp. unetbootin) feel native and operate as you would expect on the Mac platform.
  • None of the other methods of which I am aware have the ability to make the archives boot on Intel Macs.
  • It was personally a pain in the neck getting Linux distributions to boot via USB on Macs.

That being said, it does have a few shortcomings:

  • Linux fails to have graphics on some Macs (i.e Macbook Pros with nVidia graphics), which in some cases prevents boot, but this is not necessarily an issue with Mac Linux USB Loader as much as it is an issue with the video drivers that ship with most distributions. Luckily, with Enterprise, which has been included with Mac Linux USB Loader since 2.0, you can use persistence to install the necessary video drivers on distributions like Ubuntu, helping to alleviate the issue.

Building from Source

Requirements: Xcode 6, OS X 10.10 SDK. OS X 10.8+ required to run built app

  1. Clone from git: git clone
  2. Run pod install (requires Cocoapods).
  3. Open Mac Linux USB Loader.xcworkspace and do an archive build, or simply run and debug it with Xcode


  • Used some icons from KDE’s Oxygen. link
  • Special thanks to Leander Lismond for translating the application into Dutch!

More information can be found on:

Linux super-duper admin tools: lsof

lsof is one of the more important tools you can use on your Linux box. Its name is somewhat misleading. lsof stands for lisopen files, but the term files fails to impact the true significance of power. That is, unless you remember the fundamental lesson, in Linux everything is a file.

We have had several super-duper admin articles, focusing around tools that help us understand better the behavior of our system, try to identify performance bottlenecks and solve issues that do not have an apparent, immediate presence in the logs. Save for vague, indirect symptoms, you might be struggling to understand what is happening under the hood.


lsof, alongside strace and OProfile, is another extremely versatile, powerful weapon in the arsenal of a system administrator and the curious engineer. Used correctly, it can yield a wealth of information about your machine, helping you narrow down the problem solving and maybe even expose the culprit.

So let’s see what this cool tool can do.

Why is lsof so important?

I did say lsof is important, but I did not say why. Well, the thing is, with lsof you can do pretty much anything. It encompasses the functionality of numerous other tools that you may be familiar with.

For example, lsof can provide the same information netstat offers. You can use lsof to find mounts on your machine, so it supplements both /etc/mtab and /proc/mounts. You can use lsof to learn what open files a processes holds. In general, pretty much anything you can find under the /proc filesystem, lsof can display in a very simple, centric manner, without writing custom scripts for looping through the sub-directories and parsing and filtering content.

lsof allows you to display information for particular users, processes, show only traffic for certain network protocols, file handles, and more. Used effectively, it’s the Swiss Knife of admin utilities.

lsof in action

A few demonstrations are in order.

Run without any parameters, lsof will display all of the information for all of the files. At this point, I should reiterate the fact there are many types of files. While most uses treat their music and Office documents as files, the generic description goes beyond that. Devices, sockets, pipes, and directories are also files.

lsof output explained

Before we dig in, let’s take a look at a basic output:

Basic usage

Command is the name of the process. It also includes kernel threads. PID is the process ID. USER is the owner of the process. FD is the first truly interesting field.

The FD stands for File Descriptor, an abstract indicator for accessing of files. File descriptors are indexes in kernel data structures called file descriptor tables, which contain details of all open files. Each process has its file descriptor table. User applications that wish to read and write to files will instead read to and write from file descriptors using system calls. The exact type of the file descriptor will determine what the read and write operations really mean.

In our example, we have several different values of FD listed. If you have ever looked under the /proc filesystem and examined the structure of a process, some of the entries will be familiar. For instance, cwdstands for Current Working Directory of the listed process. txt is the Text Segment or the Code Segment (CS), the bit of the object containing executable instructions, or program code if you will. mem stands for Data Segments and Shared Objects loaded into the memory. 10u refers to file descriptor 10, open for both reading and writing. rtd stands for root directory.

As you can see, you need to understand the output, but once you get the hang of it, it’s a blast. lsof provides a wealth of information, formatted for good looks, without too much effort. Now, it’s up to you to put the information to good use.

The fifth column, TYPE is directly linked to the FD column. It tells us what type of file we’re working with. DIR stands for directory. REG is a regular file or a page in memory. FIFO is a named pipe. Symbolic links, sockets and device files (block and character) are also file types. unknown means that the FD descriptor is of unknown type and locked. You will encounter these only with kernel threads.

For more details, please read the super-extensive man page.

Now, we’re already having a much better picture of what lsof tells us. For instance, 10u is a pipe used by initctl, a process control initialization utility that facilitates the startup of services during bootup. All in all, it may not mean anything at the moment, but if and when you have a problem, the information will prove useful.

The DEVICE column tells us what device we’re working on. The two numbers are called major and minor numbers. The list is well known and documented. For instance, major number 8 stands for SCSI block device. For comparison, IDE disks have a major number 3. The minor number indicates one of the 15 available partitions. Thus (8,1) tell us we’re working on sda1.

(0,16), the other interesting device listed refers to unnamed, non-device mounts.

For detailed list, please see:

SIZE/OFF is the file size. NODE is the Inode number. Name is the name of the file. Again, do not be confused. Everything is a file. Even your computer monitor, only it has a slightly different representation in the kernel.

Now, we know everything. OK, unfiltered output is too much to digest in one go. So let’s start using some flags for smart filtering of information.

Per process

To see all the open files a certain process holds, use -p:

lsof -p <pid>

lsof -p

Per user

Similarly, you can see files per user using the -u flag:

lsof -u <name>

lsof -u

File descriptors

You can see all the processes holding a certain fie descriptor with -d <number>:

lsof -d <number>

lsof -d 3

This is very important if you have hung NFS mounts or blocked processes in uninterruptible sleep (D state) refusing to go away. Your only way to start solving the problem is do dig into lsof and trace down the dependencies, hopefully finding processes and files that can be killed and closed. Alternatively, you can also display all the open file descriptors:

Rising number

Notice that the number is rising in sequence. In general, Linux kernel will give the first available file descriptor to a process asking for one. The convention calls for file descriptors 0, 1 and 2 to be standard input (STDIN), standard output (STDOUT) and standard error (STDERR), so normally, file descriptor allocation will start from 3.

If you’ve ever wondered what we were doing when we devnull-ed both the standard output and the standard error in the strace examples, this ought to explain it. We had the following:

something > /dev/null 2>&1

In other words, we redirected standard output to /dev/null, and then we redirected file descriptor 2 to 1, which means standard error goes to standard output, which itself is redirected to the system black hole.

Finding file descriptors can be quite useful, especially if some applications are hard-coding their use, which can lead to problems and conflicts. But that’s a different story altogether.

One more thing notable from the above screenshot are the unix and CHR FD types, which we have not yet seen. unix stands for UNIX domain socket, an interprocess communication socket, similar to Internet sockets, only without using a network protocol. CHR stands for a character device. Character devices allow the transmission of a single bit of data; typical examples are terminals, keyboard, mouse, and similar peripherals, where the order of data is critical.

Do not confuse domain sockets with classic sockets, which is an end-point consisting of an IP address and a port.

Netstat-like behavior

lsof can also provide lots of information similar and identical to netstat. You can dump the listing of all files and then grep for relevant information, like LISTEN, ESTABLISHED, IPV4, or any other network related term.


Internet protocols & ports

Specifically, lsof can also show you the open ports for either IPv4 or IPv6 protocols, much like nmap scan against the localhost:

lsof -i<protocol>

lsof -i

Directory search

lsof also supports a number of flags that are enabled with + and disabled with – signs, rather than the typical use of single or double dash (-) characters as option separators.

One of these is +d (and +D), which lets you show all the processes holding a certain directory. The capital D also lets you recurse and expands all the files in the directory and its sub-directories, whereas lower d will just show the directories and no files.

lsof +d <dir name> or lsof +D <dirname>

Dir search

Practical example

I’ve given you two juicy examples when I wrote the strace tutorial. I skimped a bit with OProfile, because finding simple and relevant problems that can be quickly demonstrated with a profiler tool are not easy to come by – but do not despair, there shall be an article.

Now, lsof allows a plenty of demo space. So here’s one.

How do you handle a stuck mount?

Let’s say you have a mount that refuses to go down. And you don’t really know what’s wrong. For some reason, it won’t let you unmount it.



You tried the umount command, but it does not really work:


Luckily for you, openSUSE recommends using lsof, but let’s ignore that for a moment.

Anyhow, your mount won’t come down. In desperation and against better judgment, you also try forcing the unmounting of the mount point with -f flag, but it still does not help. Not only the mount is refusing to let go, you may have also corrupted the /etc/mtab file by issuing the force mount command. Just some food for thought.

Now, how do you handle this?

The hard way

If you’re experienced and know your way about /proc, then you can do the following:

Under /proc, examine the current working directories and file descriptors holding the mount point. Then, examine the process table and see what the offending processes are and if they can be killed.

ls -l /proc/*/cwd | grep just



ls -l /proc/*/fd | grep just


Finally, in our example:

ps -ef | grep -E ‘10878|10910’


And problem solved …

Note: sometimes, especially if you have problems with mounts or stuck processes, lsof may not be the best tool, as it too may get stuck trying to recurse. In these delicate cases, you may want to use the -n and -l flags. -n inhibits the conversion of network IP addresses to domain names, making lsof work faster and avoids lockups due to name lookup not working properly. -l inhibits conversion of user IDs to names, quite useful if name lookup is working slowly or improperly, including problems with nscd daemon, connectivity to NIS, LDAP or whatever, and other issues. However, sometimes, in extreme cases, going for /proc may be the most sensible option.

The easy (and proper) way

By the book, using lsof ought to do it:

lsof | grep just

lsof just

And problem solved. Well, we still need to free the mount by closing or killing the process and the files held under the mount point, but we know what to do. Not only do we get all the information we need, we do this quickly, efficiently.

Knowing the alternative methods is great, but you should always start smart and simple, with lsof, exploring, narrowing down possibilities and converging on the root cause.

I hope you liked it.


There you go,a wealth of information about lsof and what it can do for you. I bet you won’t easily find detailed explanation about lsof output elsewhere, although examples about the actual usage are aplenty. Well, my tutorial provides you with both.

Now, the big stuff is ahead of you. Using lsof to troubleshoot serious system problems, without wasting time going through /proc and trying to find relevant system information, when it’s all there, hidden under just one mighty command.

checkinstall – Smartly manage your installations

The best way to install applications in Linux is by using the package managers. It’s the simplest, safest and most foolproof way of obtaining and maintaining the programs you need. You install them using a friendly and intuitive interface and you uninstall them using the same interface. The dependencies are automatically solved. The program revision is tracked. Whenever you can, use the package manager to get what you need.

There are many package managers available – and they come in two forms: the core utility, which is command-line and the front-end (GUI), which calls on the command-line tool to do the job. In openSUSE, you have the YaST/zypper combo, in Ubuntu, you have Synaptic/apt, in Fedora, you have Pirut/yum, and so forth.


But sometimes, the program you want will not be found in the repositories, even the extras ones like Medibuntu or RPMForge. You will have to download the sources and compile them and install your package manually.

The problem with this approach is that your manually installed programs will not be visible in your package manager. They won’t show up, nor be available for upgrades or removal, creating a potential clutter/security issue for you, especially if there are many such programs you must use.

Luckily for you, there’s a solution: a utility that can package the sources into installer files that your package manager will recognize and be able to catalog. This utility is called checkinstall.

Enter checkinstall

checkinstall works by functioning as a wrapper for your typical installation from sources. It will follow after the third stage in the configure, make, make install chain and keep track of every change made to the system. Once the installation is done, it will create a package compatible with your package management. checkinstall works with RPM, Debian and Slackware packages, covering a rather large install base.

OK, let’s see this thing in action!

Install checkinstall

The first thing is: install checkinstall. A sort of a chicken and an egg problem. You should probably use your package manager to get checkinstall installed.


Install program from source

Your next step is to find the application you want to install, which is not found in the repositories. This is not an east task nowadays. I spent quite a bit of time hunting for a program that I want. Eventually, I settled forGuake, a Quake-like drop-down terminal utility. Please note that it DOES exist in the repositories, but it was a good choice as any.

So I started the usual chain, with configure and make …



Please note that these two steps may fail, depending on the configuration of your system. Some of your libraries may be missing, outdated or too new for the sources you’re trying to compile. Then, the sources themselves might be written badly, with errors and whatnot.

But assuming that everything went smoothly, your next step is to invoke checkinstall as root (or sudo):


Run checkinstall

A short wizard will guide you through the installation & package creation process. If the package documentation directory does not exist, it will ask you to create one.

You will then have the opportunity to write your own documentation:


Then comes the installation and the creation of the package. You can change the options if you like. Normally, I would not recommend changing any of the values unless you really know what you’re doing.

Debian package

And soon, you will have the program installed:


You can even check in your package manager now, to see whether the package is listed and installed as expected. Yup, there it is! Notice our very own documentation!

In package manager

And the application in the menu:


From now on, the manually installed program is just like any other program. Your package manager will maintain it, sparing you the grueling manual work. Excellent!


checkinstall is a great addition to the Linux user’s arsenal of handy tools, especially experienced users with peculiar taste for non-conventional installations of programs not readily available in the repositories. It allows you to easily keep track and order of all your applications, whether they come as package installers or from sources.

Linux super-duper admin tools: OProfile

It’s time to step up the geeky fun a notch and learn about OProfile.


OProfile is a Linux system-wide profiling tool that you can use to, uh, profile and analyze performance and runtime problems with your applications, or even the kernel itself. It’s very simple to use and does not require any special preparations. No need to patch the kernel or use debug symbols. Just insert the module and start running.

OProfile uses the hardware performance counters of the CPU to enable profiling of a wide variety of statistics, which you can then use for profiling of the kernel and your applications. In fact, OProfile works with everything, including hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.

After you’ve collected the data you need, you can run reports against it, even produce graphs showing you a vizualization of the profiled runs.

For more details, please read the Novell Cool Solutions OProfile article and visit the official website, where you can find lots of useful information about the tool, including its numerous features and advantages.

Now, let’s use it.


I must warn you though. Officially, OProfile is alpha software. Although it has been tested to work well with a wide range of architecture platforms and kernels, there’s no guarantee it will do what’s expected of it. You may even break your system.

My experience shows no issues with OProfile, but you may not be so lucky. Now, if you’re brave enough, proceed.

Regardless, using OProfile is a very geeky thing that will surely impress everyone and may even provide you with yet another powerful tool for making your environment smarter, faster and safer. Home users will probably never need it, but they just might be piqued enough to give it a try, especially if they’re facing severe performance problems. System admin wise, OProfile probably falls into the Level II-III support, so you won’t be using it that often, but when you do, it should come quite handy.

Install OProfile

The tool comes available in the repositories of many distributions, so you will not have to manually download and compile. If you do, consider using checkinstall to have it registered in your software database.


Running OProfile

Now, we need to start using the tool.

The first question you need to ask yourselves is: do you also want to profile the kernel? Your answer will determine the OProfile command line.

If you wish to run OProfile without profiling the kernel, then:

opcontrol –no-vmlinux

If you do want to profile the kernel, as well, then:

opcontrol –vmlinux=/boot/vmlinux-`uname -r`


The above command will set the Linux kernel to the running version of your kernel in uncompressed form. To make sure such a kernel exists, please take a look under your /boot directory.

Modern Linux distributions ship with the kernel archived (zipped) to conserve space, so you will have to unzip it before it can be used:

Boot dir

So we do have the kernel available, it’s vmlinux-<whatever>.gz. We need to unzip it. This is done with the gunzip command:

gunzip vmlinux-<something>.gz

Like this:


Now, if needed, rerun the opcontrol command from before to set the kernel. Once you’ve done this, launch the tool. It will start collecting the data.


Let it run for a while before stopping it and profiling the data. In fact, to make things meaningful, we’ll run a little compilation in the background, so that our report contains more data, as well as more meaningful data. A good example is the MPlayer.


After a while you can simply dump the collected data and continue profiling or stop the profiler altogether. To just dump the data, use:

opcontrol –dump

To stop it, use:

opcontrol –stop


If, at any given moment you wish to resrt your profiling counters and start fresh, then you can reset the OProfile deamon:

opcontrol –reset

And to shut it down altogether:

opcontrol –shutdown


Once you’ve collected enough data, time for a report. Just run:


To get what you need. This is where the real fun begins, analyzing the report and trying to understand the problems you’re facing.

Anyhow, here’s a sample of what it may look like. Screenshot taken on another host, so please don’t mind the differences in host names and CPU speeds and such.


More data

In the leftmost column, you will get the exact number of samples collected. In the middle one, the percentage of time spent using different libraries. In the rightmost column, the actual process name. Usually, you will see a whole bunch of glibc, perl and other libraries. If you’re compiling a tool that also uses GUI, then other interesting bits, too.

Since I was also listening to Youtube in Firefox while compiling and doing a few more things, you will notice calls to Nvidia driver, Flash player and so forth.

Now, opreport may not report everything you want and it may even exit with an error. Sometimes, the amount of time spent profiling may not be enough to produce a meaningful report. At other times, you may need packages with debug symbols or even a complete debug kernel.

In that case, you will want to run opcontrol –symbols or opcontrol -l to get more data.


And you may even want to create a call graph. This is done by running opcontrol -c.

Call graph

You will need a tool that can read and display call graphs. There are many, many options available. For example, an oldie but goodie Kpl comes to mind:


You may also want to use KCachegrind, but this one requires the call graphs to be in the Valgrind format, which will require a conversion tool. This is where the real hacking kicks in, but this is beyond the scope of this article. We’ll talk about advanced system debugging in following articles. Consider strace and OProfile a sort of a long and expensive warmup.


OProfile is a very handy, useful tool. In the hands of a smart system administrator, it can be used to detect application slowness problems, analyze system bottlenecks, optimize system performance and utilization, and resolve resource/usage conflicts. Combined with a range of other admin programs, some of which we’ve talked about and others we are yet to see, OProfile is a must-have item on the system debugging checklist.

I hope you’ve enjoyed this article. Many more are yet to come, exposing you to the realm of uber-hack. As to profiling itself, we’ll talk about some other cool programs, including Valgrind and Linux Trace Toolkit (LTT).

Linux super-duper admin tools: screen

Time to learn about yet another cool little admin application that will change the way you think and work. We had strace, a mighty, versatile debugging tool that helped us diagnose and categorize system programs quickly and effectively and point us in the right direction in our investigation of problems. We had OProfile, a powerful profiling utility that can be used to time the system and application performance and identify chokepoints and bottlenecks in program executions. Time to step back and appraise screen.



screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. Each virtual terminal provides the functions of the legendary DEC VT100 terminal.

Additionally, the utility has insert/delete line, support for multiple character sets, a scrollback history buffer for each virtual terminal, and a copy-and-paste mechanism that allows moving text regions between windows.

When screen is called, it creates a single window with a shell in it and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new full-screen windows with other programs in them, including more shells, kill existing windows, view a list of windows, turn output logging on and off, copy & paste text between windows, view the scrollback history, switch between windows in whatever manner you wish, etc.

All windows run their programs completely independent of each other. Programs continue to run when their window is currently invisible and even when the whole screen session is detached from the user’s terminal. When a program terminates, screen kills the window that contained it. If this window was in the foreground, the display switches to the previous window; if none are left, screen exits.

In a nutshell, if Ctrl + F buttons allow you to switch between up to seven virtual consoles, horizontally, screen lets you create an infinite vertical stack of consoles in each one of these.

Home users running full GUI desktops and playing with tabbed terminal utilities would be hard-tempted to find merit in screen, but when you’re running in runlevel 3 and the monitor space is limited, screen is a blessing.

Screen in action

Let’s begin with a few screenshots. To start screen, just type screen in any one console windows, be it gnome-terminal, xterm, Konsole, or any other.


This will display an introduction messages. Press Enter to exit.


You’re inside a new virtual console. Why not fire another?


And here’s the second:


Using the right keyboard shortcuts, we can switch between them, back and forth. Use Ctrl + a then 0 to go to the zeroth (first) screen, Ctrl + a then 1 to go to the second one, and so forth.

First toggled

Now, demonstrating screen with still images is difficult, so here’s a Flash movie! Created using Wink, which served us well so many times, including the tutorial itself, as well as the Windows PowerShell article, and a few others.

So here we go:

Lovely, right! Damn right!

Help window

Don’t hesitate to call for help. Ctrl + a, then ? will pop the help screen.


Of course, you can also read the man page for more details. There’s a plenty you can do with screen, attach/detach/reattach sessions, specify the history scrollback buffer, turn login mode on and off, suppress error messages, and more. screen is a powerful marvel and you should start using it.


Yet another powerful tool mastered. Our list grows bigger, and so does our knowledge. screen may seem trivial to you, but what if you need to debug problems across multiple session and you can’t afford to have tons of Konsole or xterm windows strewn about the desktop like mad. Then, there’s the issue of practical visibility. Never take your eyes off the screen and yet enjoy full multi-view console.

I hope you liked this little surprise. Now, off to new wonders. Stay tuned for many more articles of great admin tools, aptly called super-duper, by me. Be excellent to each other and party on.

Collecting And Analyzing Linux Kernel Crashes – LKCD

Table of Contents

  1. LKCD – Introduction
    1. How does LKCD work?
  2. LKCD Installation
  3. Basic prerequisites
    1. A Linux operating system configured to use LKCD:
    2. LKCD configuration
  4. LKCD local dump procedure
    1. Required packages
    2. Configuration file
    3. Activate dump process (DUMP_ACTIVE)
    4. Configure the dump device (DUMP_DEVICE)
    5. Configure the dump directory (DUMPDIR)
    6. Configure the dump level (DUMP_LEVEL)
    7. Configure the dump flags (DUMP_FLAGS)
    8. Configure the dump compression level (DUMP_COMPRESS)
    9. Additional settings
    10. Enable core dump capturing
    11. Configure LKCD dump utility to run on startup
  5. LKCD netdump procedure
  6. Configure LKCD netdump server
    1. Required packages
    2. Configuration file
    3. Configure the dump flags (DUMP_FLAGS)
    4. Configure the source port (SOURCE_PORT)
    5. Make sure dump directory is writable for the netdump user
    6. Configure LKCD netdump server to run on startup
    7. Start the server
  7. Configure LKCD client for netdump
    1. Configure the dump device (DUMP_DEV)
    2. Configure the target host IP address (TARGET_HOST)
    3. Configure target host MAC address (ETH_ADDRESS)
    4. Configure target host port (TARGET_PORT)
    5. Configure the source port (SOURCE_PORT)
    6. Enable core dump capturing
    7. Configure LKCD dump utility to run on startup
    8. Start the lkcd-netdump utility
  8. Test functionality
    1. Example of unsuccessful netdump to different network segment
  9. Conclusion
  10. Download

LKCD – Introduction

LKCD stands for Linux Kernel Crash Dump. This tool allows the Linux system to write the contents of its memory when a crash occurs, so that they can be later analyzed for the root cause of the crash.

Ideally, kernels never crash. In reality, the crashes sometimes occur, for whatever reason. It is in the best interest of people using the plagued machines to be able to recover from the problem as quickly as possible while collecting as much data available. The most relevant piece of information for system administrators is the memory dump, taken at the moment of the kernel crash.

How does LKCD work?

You won’t notice LKCD in your daily work. Only when a kernel crash occurs will LKCD kick into action. The kernel crash may result from a kernel panic or an oops or it may be user-triggered. Whatever the case, this is when LKCD begins working, provided it has been configured correctly.

LKCD works in two stages:

Stage 1

This is the stage when the kernel crashes. Or more correctly, a crash is requested, either due to a panic, an oops or a user-triggered dump. When this happens, LKCD kicks into action, provided it has been enabled during the boot sequence.

LKCD copies the contents of the memory to a temporary storage device, called the dump device, which is usually a swap partition, but it may also be a dedicated crash dump collection partition.

After this stage is completed, the system is rebooted.

Stage 2

Once the system boots back online, LKCD is initiated. On different systems, this takes a different startup script. For instance, on a RedHat machine, LKCD is run by the /etc/rc.sysinit script.

Next, LKCD runs two commands. The first command is lkcd config, which we will review more intimately later. This commands prepares the system for the next crash. The second command is lkcd save, which copies the crash dump data from its temporary storage on the dump device to the permanent storage directory, called dump directory.

Along with the dump core, an analysis file and a map file are created and copied; we’ll talk about these separately when we review the crash analysis.

A completion of this two-stage cycle signifies a successful LKCD crash dump.

Here’s an illustration:


Some reservations:

LKCD is a somewhat old utility. It may not work well with newer kernels.

All right, now that we know what we’re talking about, let us setup and configure LKCD.

LKCD Installation

You will have to forgive me, but I will NOT demonstrate the LKCD installation. There are several reasons for this untactical evasion on my behalf. I do not expect you to forgive me, but I do hope you will listen to my points:

The LKCD installation requires kernel compilation. This is a lengthy and complex procedure that takes quite a bit of time. It is impossible to explain how LKCD can be installed without showing the entire kernel compilation in detail. For now, I will have to skip this step, but I promise you a tutorial on kernel compilation.

Furthermore, the official LKCD documentation does cover this step. In fact, the supplied IBM tutorial is rather good. However, like most advanced technical papers geared toward highly experienced system administrators, it lacks actual usage examples.

Therefore, I will assume you have a working system compiled with LKCD. So the big question is, what now? How do you use this thing?

This tutorial will try to answer the questions in a linear fashion, explaining how to configure LKCD for local and network dumping of the memory core.

Basic prerequisites

A Linux operating system configured to use LKCD:

Most home users will probably not be able to meet this demand. On the other hand, when you think about it, the collection and analysis of kernel crashes is something you will rarely do at home. For home users, kernel crashes, if they ever occur within the limited scope of desktop usage, are just an occasional nuisance, the open-source world equivalent of the BSOD.

However, if you’re running a business, having your mission-critical systems go down can have a negative business impact. This means that you should be running the “right” kind of operating system in your work environment, configured to suit your needs.

LKCD configuration

LKCD dumps the system memory to a device. This device can be a local partition or a network server. We will discuss both options.

LKCD local dump procedure

Required packages

The host must have the lkcdutils package installed.

Configuration file

The LKCD configuration is located under /etc/sysconfig/dump. Back this up before making any changes! We will have to make several adjustments to this file before we can use LKCD. So let us begin.

Activate dump process (DUMP_ACTIVE)

To be able to use LKCD when crashes occur, you must activate it.


Configure the dump device (DUMP_DEVICE)

You should be very careful when configuring this directive. If you choose the wrong device, its contents will be overwritten when a crash is saved to it, causing data loss.

Therefore, you must make sure that the DUMPDEV is linked to the correct dump device. In most cases, this will be a swap partition, although you can use any block device whose contents you can afford to overwrite. Accidentally, this section partially explains why the somewhat nebulous and historic requirement for a swap partition to be 1.5x the size of RAM.

What you need to do is define a DUMPDEV device and then link it to a physical block device; for example,/dev/sdb1. Let’s use the LKCD default, which calls the DUMPDEV directive to be set to /dev/vmdump.


Now, please check that /dev/vmdump points to the right physical device. Example:

ls -l /dev/vmdump
lrwxrwxrwx 1 root root 5 Nov 6 21:53 /dev/vmdump ->/dev/sda5

/dev/sda5 should be your swap partition or a disposable crash partition. If the symbolic link does not exist, LKCD will create one the first time it is run and will link /dev/vmdump to the first swap partition found in the/etc/fstab configuration file. Therefore, if you do not want to use the first swap partition, you will have to manually create a symbolic link for the device configured under the DUMPDEV directive.

Configure the dump directory (DUMPDIR)

This is where the memory images saved previously to the dump device will be copied and kept for later analysis. You should make sure the directory resides on a partition with enough free space to contain the memory image, especially if you’re saving all of it. This means 2GB RAM = 2GB space or more.

In our example, we will use /tmp/dump. The default is set to /var/log/dump.


And a screenshot of the configuration file in action, just to make you feel comfortable:


Configure the dump level (DUMP_LEVEL)

This directive defines what part of the memory you wish to save. Bear in mind your space restrictions. However, the more you save, the better when it comes to analyzing the crash root cause.

Do nothing, just return if called
Dump the dump header and first 128K bytes out
Everything in DUMP_HEADER and kernel pages only
Everything except kernel free pages
All memory

Configure the dump flags (DUMP_FLAGS)

The flags define what type of dump is going to be saved. For now, you need to know that there are two basic dump device types: local and network.

Local block device
Network device

Later, we will also use the network option. For now, we need local.


Configure the dump compression level (DUMP_COMPRESS)

You can keep the dumps uncompressed or use RLE or GZIP to compress them. It’s up to you.


I would call the settings above the “must-have” set. You must make sure these directives are configured properly for the LKCD to function. Pay attention to the devices you intend to use for saving the crash dumps.

Additional settings

There are several other directives listed in the configuration file. These other directives are all set to the the configuration defaults. You can find a brief explanation on each below. If you find the section inadequate, please email me and I’ll elaborate.

These include:

  • DUMP_SAVE=”1″ – Save the memory image to disk
  • PANIC_TIMEOUT=”5″ – The timeout (in seconds) before a reboot after panic occurs
  • BOUNDS_LIMIT =”10″ – A limit on the number of dumps kept
  • KEXEC_IMAGE=”/boot/vmlinuz” – Defines what kernel image to use after rebooting the system; usually, this will be the same kernel used in normal production
  • KEXEC_CMDLINE=”root console=tty0″ – Defines what parameters the kernel should use when booting after the crash; usually, you won’t have to tamper with this setting – but if you have problems, email me.

In general, we’re ready to use LKCD. So let’s do it.

Enable core dump capturing

The first step we need to do is enable the core dump capturing. In other words, we need to sort of source the configuration file so the LKCD utility can use the values set in it. This is done by running the lkcd configcommand, followed by lkcd query command, which allows you to see the configuration settings.

lkcd config
lkcd query

The output is as follows:

Configured dump device: 0xffffffff
Configured dump flags: KL_DUMP_FLAGS_DISKDUMP
Configured dump level: KL_DUMP_LEVEL_HEADER| >>
Configured dump compression method: KL_DUMP_COMPRESS_GZIP

Configure LKCD dump utility to run on startup

To work properly, the LKCD must run on boot. On RedHat machines, you can use the chkconfig utility to achieve this:

chkconfig boot.lkcd on

After the reboot, your machine is ready for crashing … I mean crash dumping. We can begin testing the functionality. However …


Disk-based dumping may not always succeed in all panic situations. For instance, dumping on hung systems is a best-effort attempt. Furthermore, LKCD does not seem to like the md RAID devices, presenting another problem into the equation. Therefore, to overcome the potentially troublesome situations where you may end up with failed crash collections to local disks, you may want to consider using the network dumping option. Therefore, before we demonstrate the LKCD functionality, we’ll study the netdump option first.

LKCD netdump procedure

Netdump procedure is different from the local dump in having two machines involved in the process. One is the host itself that will suffer kernel crashes and whose memory image we want to collect and analyze. This is the client machine. The only difference from a host configured for local dump is that this machine will use another machine for storage of the crash dump.

The storage machine is the netdump server. Like any server, this host will run a service and listen on a port to incoming network traffic, particular to the LKCD netdump. When crashes are sent, they will be saved to the local block device on the server. Other terms used to describe the relationship between the netdump server and the client is that of source and target, if you will: the client is a source, the machine that generates the information; the server is the target, the destination where the information is sent.

We will begin with the server configuration.

Configure LKCD netdump server

Required packages

The server must have the following two packages installed: lkcdutils and lkcdutils-netdump-server.

Configuration file

The configuration file is the same one, located under /etc/sysconfig/dump. Again, back this file up before making any changes. Next, we will review the changes you need to make in the file for the netdump to work. Most of the directives will remain unchanged, so we’ll take a look only at those specific to netdump procedure, on the server side.

Configure the dump flags (DUMP_FLAGS)

This directive defines what kind of dump is going to be saved to the dump directory. Earlier, we used the local block device flag. Now, we need to change it. The appropriate flag for network dump is 0x40000000.


Configure the source port (SOURCE_PORT)

This is a new directive we have not seen or used before. This directive defines on which port the server should listen for incoming connections from hosts trying to send LKCD dumps. The default port is 6688. When configured, this directive effectively turns a host into a server – provided the relevant service is running, of course.


Make sure dump directory is writable for the netdump user

This directive is extremely important. It defines the ability of the netdump service to write to the partitions / directories on the server. The netdump server run as the netdump user. We need to make sure this user can write to the desired destination (dump) directory. In our case:

install -o netdump -g dump -m 777 -d /tmp/dump

You may also want to ls the destination directory and check the owner:group. It should be netdump:dump. Example:

ls -ld dump
drwxrwxrwx 3 netdump dump 96 2009-02-20 13:35 dump

You may also try getting away with manually chowning and chmoding the destination to see what happens.

Configure LKCD netdump server to run on startup

We need to configure the netdump service to run on startup. Using chkconfig to demonstrate:

chkconfig netdump-server on

Start the server

Now, we need to start the server and check that it’s running properly. This includes both checking the status and the network connections to see that the server is indeed listening on port 6688.

/etc/init.d/netdump-server start
/etc/init.d/netdump-server status


netstat -tulpen | grep 6688
udp 0 0* 479 37910 >>
>> 22791/netdump-server

Everything seems to be in order. This concludes the server-side configurations.

Configure LKCD client for netdump

Client is the machine (which can also be a server of some kind) that we want to collect kernel crashes for. When kernel crashes for whatever reason on this machine, we want it to send its core to the netdump server. Again, we need to edit the /etc/sysconfig/dump configuration file. Once again, most of the directives are identical to previous configurations.

In fact, by changing just a few directives, a host configured to save local dumps can be converted for netdump.

Configure the dump device (DUMP_DEV)

Earlier, we have configured our clients to dump their core to the /dev/vmdump device. However, network dump requires an active network interface. There are other considerations in place as well, but we will review them later.


Configure the target host IP address (TARGET_HOST)

The target host is the netdump server, as mentioned before. In our case, it’s the server machine we configured above. To configure this directive – and the one after – we need to go back to our server and collect some information, the output from the ifconfig command, listing the IP address and the MAC address. For example:

inet addr:
HWaddr 00:12:1b:40:c7:63

Therefore, our target host directive is set to:


Alternatively, it is also possible to use hostnames, but this requires the use of hosts file, DNS, NIS or other name resolution mechanisms properly set and working.

Configure target host MAC address (ETH_ADDRESS)

If this directive is not set, the LKCD will send a broadcast to the entire neighborhood, possibly inducing a traffic load. In our case, we need to set this directive to the MAC address of our server:



Please note that the netdump functionality is currently limited to the same subnet that the server runs on. In our case, this means /24 subnet. We’ll see an example for this shortly.

Configure target host port (TARGET_PORT)

We need to set this option to what we configured earlier for our server. This means port 6688.


Configure the source port (SOURCE_PORT)

Lastly, we need to configure the port the client will use to send dumps over network. Again, the default is 6688.


And image example:

Source port

This concludes the changes to the configuration file.

Enable core dump capturing

Perform the same steps we did during the local dump configuration: run the lkcd config and lkcd querycommands and check the setup.

lkcd config
lkcd query

The output is as follows:

Configured dump device: 0xffffffff
Configured dump flags: KL_DUMP_FLAGS_NETDUMP
Configured dump level: KL_DUMP_LEVEL_HEADER| >>
Configured dump compression method: KL_DUMP_COMPRESS_GZIP

Configure LKCD dump utility to run on startup

Once again, the usual procedure:

chkconfig lkcd-netdump on

Start the lkcd-netdump utility

Start the utility by running the /etc/init.d/lkcd-netdump script.

/etc/init.d/lkcd-netdump start

Watch the console for successful configuration message. Something like this:


This means you have successfully configured the client and can proceed to test the functionality.

Test functionality

To test the functionality, we will force a panic on our kernel. This is something you should be careful about doing, especially on your production systems. Make sure you backup all critical data before experimenting.

To be able to create panic, you will have to enable the System Request (SysRq) functionality on the desired clients, if it has not already been set:

echo 1 > /proc/sys/kernel/sysrq

And then force the panic:

echo c > /proc/sysrq-trigger

Watch the console. The system should reboot after a while, indicating a successful recovery from the panic. Furthermore, you need to check the dump directory on the netdump server for the newly created core, indicating a successful network dump. Indeed, checking the destination directory, we can see the memory core was successfully saved. And now we can proceed to analyze it.


Example of unsuccessful netdump to different network segment

As mentioned before, the netdump functionality seems limited to the same subnet. Trying to send the dump to a machine on a different subnet results in an error (see screenshot below). I have tested this functionality for several different subnets, without success. If anyone has a solution, please email it to me.

Here’s a screenshot:



LKCD is a very useful application, although it has its limitations.

On one hand, it provides with the critical ability to perform indepth forensics on crashed systems post-mortem. The netdump functionality is particularly useful in allowing system administrators to save memory images after kernel crashes without relying on the internal hard disk space or the hard disk configuration. This can be particularly useful for machines with very large RAM, when dumping the entire contents of the memory to local partitions might be problematic. Furthermore, the netdump functionality allows LKCD to be used on hosts configured with RAID, since LKCD is unable to work with md partitions, overcoming the problem.

However, the limitation to use within the same network segment severely limits the ability to mass-deploy the netdump in large environments. It would be extremely useful if a workaround or patch were available so that centralized netdump servers can be used without relying on specific network topography.

Lastly, LKCD is a somewhat old utility and might not work well on the modern kernels. In general, it is fairly safe to say it has been replaced by the more flexible Kdump, which we will review in the next article

Linux Super-Duper Admin Tools: GNU Debugger (gbd)

Let’s talk debug. So you wrote a piece of code and you want to compile it and run it. Or you have a binary and you just run it. The only problem is, the execution fails with a segmentation fault. For all practical purposes, you call it a day.

Luckily for you, the ultimate combination of the GNU Debugger (gdb) and Dedoimedo tutorials will help you overcome the problem. Today, we will learn how to handle misbehaving binary code, how to examine its execution step by step, how to interpret errors and problems, and we will even step into the assembly code and hunt for problems there. This won’t be easy, but it sure will be one of the best super-duper admin guides you have read so far.



I repeat: this will not be easy. Working with gdb is not something anyone can do at their leisure. There are many requirements you must meet before you can have a successful session.


You can debug code without having access to source files. However, your task will be more difficult, because you will not be able to refer to the actual code and try to understand if there’s any kind of logical fallacy in the execution. You will only be able to follow symptoms and try to figure out where things might be wrong, but not why.

Sources compiled with symbols

On top of that, you will want sources with symbols, so you can map instructions in the binary program to their corresponding functions and lines in the source code. Otherwise, you will be sort of groping in the dark.

Understanding of gdb

This tutorial will teach you a handful of basic and intermediate commands, so you need not worry too much about that. However, if you really find the concepts alien and you struggle with compilations and working on the command line in general, perhaps this topic is a little advanced for you at the moment.

Understanding of Linux system

This is probably the most important element. First, you will need some core knowledge of the memory management in Linux. Then, the fundamental concepts like code, data, heap, stack and whatnot. You should also be able to navigate /proc with some degree of comfort. You should also be familiar with the AT&T Assembly syntax, which is the syntax used in Linux, as opposed to Intel syntax, for example.

All right, if you meet all of the above – or wish to – then you can proceed.

Simple example

We will begin with a simple example – a null pointer. In layman’s terms, null pointer is a pointer to an address in the memory space that does not have a meaningful value and cannot be referenced by the calling program, for whatever reason. This will normally lead to an unhandled error, resulting in a segmentation fault.

Here’s our source code:

Source code

#include <stdio.h>

int main (int argc, char* argv[])
int* boom=0;
printf(“hello %d”,*boom);

Now, let’s us compile it, with symbols. This is done by using the -g flag when running gcc. We have seen this before, in the Linux Kernel Crash Book examples.


gcc -g source.c -o naughty-file.bin

And then, we run it and get a nasty segmentation fault:


Now, you may want to try to debug this problem using standard tools, like perhaps strace, ltrace, maybe lsof, and a few others. Normally, you would do this, because having a methodical approach to problem solving is always good, and you should start with simple things first. However, I will purposefully not do that right now to keep the mind clobber at a minimum. As we advance in the tutorial, we will see more complex examples and the use of other tools, too.

All right, so now we need to start using the GNU Debugger. We will invoke the program once again, this time through gdb. The syntax is simple:

gdb <program>

And so we do it.

Invoked gdb

For the time being, nothing happens. The important thing is that gdb has read symbols from our binary. The next step is to run the program and reproduce the segmentation fault. To do this, simply use the commandrun inside gdb.


We see several important details. One, that separate debuginfo (symbols) for third-party libraries, which are not part of our own code, are missing. This means that we can hook into their execution, but we won’t see any symbols. We’ll see an example soon. Two, we see that our program crashes. The problem is in the sixth line of source, as shown in the image, our printf line. Does this mean there’s a problem with printf? Probably not, but something in the variable that printf is trying to use, most likely. The plot thickens.

What we learn here is that we have symbols, that gdb won’t run automatically and that we have a meaningful way of reproducing the problem. This is very important to remember, but we will recap this when we discuss when to run or not to run gdb.


Running through the program does not yield enough meaningful information. We need to halt the execution just before the printf line. Enter breakpoints, just like when working with a compiler. We will break into the main function and then advance step by step until the problem occurs again, then rerun and break, then execute commands one at a time just short of the segmentation fault.

To this end, we need the break command, which lets you specify breakpoints either against functions, your own or third-party loaded by external libraries or against specific lines of code in your sources – an example is on the way. Then, we will use info command to examine our breakpoints.

We will place the break point in the main() function. As a rule of thumb, it’s always a good place to start.

break main

Break point

Now we run again. The execution halts when we reach main().

Run with break

Step by step, oooh babe

Now that we have stopped at the entry to main, we will step through code line by line, using the nextcommand. Luckily for us, there isn’t that much code to walk through. After just two steps, we segfault. Good.


We will now rerun the code, break in the main(), do a single next that will lead us to printf, and then we will halt and examine the assembly code no less!



Indeed, at this stage, there’s nothing else the code can tell us. We have exhausted our understanding of what happens in the code. Seemingly, there doesn’t seem to be any great problem, or rather, we can’t see it yet, supposedly.

So we will use the disassemble command, which will dump the assembly code. In a way, it’s no different than what we did when using objdump against a binary in the kernel crash example. The big difference is, you have a full control of your execution here, so you don’t need to understand everything, just limit your work to a small subset of code.

Just type disassemble inside gdb and this will dump the assembly instructions that your code uses. It will look like the screenshot below.


This is probably the most difficult part of the tutorial yet. Assembly code is not easy to digest and looks like Rain Man’s afternoon fun. Let’s try to understand what we see here, again in very simplistic terms.

On the left, we have memory addresses. The second column shows increments in the memory space from the starting address. The third column shows the mnemonic. The fourth column includes actual registers and values.

If you feel lost, consider reading the TL;DR section below, to get even more lost.

All right, there’s a little arrow pointing at the memory address where our execution is right now. We are at offset 40054b, and we have have moved the value that is stored 8 bytes below the base pointer into the RAX register.One line before that, we moved the value 0 into the RBP-8 address. So now, we have the value 0 in the RAX register.

0x00000000400543 <+15> movq $0x0,-0x8(%rbp)
0x0000000040054b <+23> mov  -0x8(%rbp),%rax

Our next instruction is the one that will cause the segmentation fault, as we have seen earlier while next-ing through the code.

0x0000000040054f <+27> mov  (%rax),%edx

So we need to understand what’s wrong here. Let’s examine the EDX register, which is supposed to get this new value. We can do this by using the examine or x command. You can use all kinds of output formats, but that’s not important right now.

x $edx

Register values

And we get a message that we cannot access memory at the specified address. This is the clue right there, problem solved. We tried fondling memory that is not to be fondled. As to why we breached our allocation and how we can know that, we will learn soon.

Not so simple example

Now, we do something more complex. We’ll create a dynamic array called pointer, which also happens to be a pointer, there’s a punny pun right there. We’ll use the standard malloc subroutine for this. We will then loop, incrementing i values by 1 every iteration, then let pointer exceed its allowed memory space, AKA heap overflow. Understandable as a lab case, but let’s see this happen in real life and how we can handle problems like these. Most importantly, we will learn additional gdb commands.

Here’s the source:

#include <stdio.h>
#include <stdlib.h>

int *pointer;
int i;
pointer = malloc(sizeof(int));
for (i = 0; 1; i++)
printf(“pointer[%d] = %d\n”, i, pointer[i]);

Let’s compile:

gcc -g seg.c -o seg

When we run it, we see something like this:


pointer[33785] = 33785
pointer[33786] = 33786
pointer[33787] = 33787
Segmentation fault

Now, before we hit gdb and assembly, let’s try some normal debugging. Ley’s say you want to try to solve the problem with one of the standard system admin and troubleshooting tools like strace. After having heard of strace on Dedoimedo, you know the tool’s worth and you want to attempt the simple steps first. Indeed, strace works well in most cases. But here, it’s of no use.

15715 write(1, “pointer[33784] = 33784\n”, 23) = 23
15715 write(1, “pointer[33785] = 33785\n”, 23) = 23
15715 write(1, “pointer[33786] = 33786\n”, 23) = 23
15715 write(1, “pointer[33787] = 33787\n”, 23) = 23
15715 — SIGSEGV (Segmentation fault) @ 0 (0) —
15715 +++ killed by SIGSEGV +++

Nothing useful there really. In fact, no classic tool will give you any indication what happens here. So we need a debugger, gdb in our case. Load the program.

gdb /tmp/seg


Like before, we set a breakpoint. However, using main() is not going to be good for us, because the program will enter main() once and then loop, never going back to the set breakpoint. So we need something else. We need to break in a specific line of code.

To determine the best place, we could run and try to figure out where the problem occurs. We can also take a look at our code and make an educated guess. This should be somewhere in the for loop of course. So perhaps, the start of it?

Break line



All right, but this is not good enough. We will have a break point at every entry to our loop, and from the execution run, we see there are going to be some 30K + iterations. We cannot possibly manually type contand hit Enter every time. So we need a condition, an if statement that will break only if a specific condition is met.

From our sample run, we see that the problem occurs when i reaches the value of 33787, so we’ll place aconditional break some one or two loop iterations before that. Conditions are set per breakpoint. Notice the breakpoint number, after it is set, because we need that number to set a condition.

break 10
Breakpoint 1 at …

And then:

condition 1 i == 33786


If you had multiple breakpoints and you wanted to set multiple conditions, then you would invoke the correct breakpoint number. All right, we’re ready to roll, hit run and let the for loop churn for a while.

Condition reached

All right, now we walk through the code, step by step using the next command.


All right, we know the problem occurs after pointer[i]=i is set, when the i value is 33787. Which means, we will rerun the program and then stop just short of executing the pointer[i]=i line of code after a successful print of pointer[33787] = 33787.

Now, the next time we reach this point, we create the assembly dump.


We know the problem occurs at offset 4005bc, where we mov %eax value into %rdx. This is similar to what we saw earlier. But we need to understand what happens before that, one or two instructions back.

Stepping through assembly dump

To this end, we will use the stepi command, which can walk the assembly dump, line by line. It’s like next in a way, but you can control individual registers, so to speak.

Take a look at the dump. The last line in the dump is the jump (jmp) instruction back to offset <main+29>, which brings us to mov 0xfffffffffffffffc(%rbp), %eax. This is effectively our for loop. Now, when we hit stepi, we will execute line 4005ac. I omitted the line that reads cltq, because it merely extends the 2-byte EAX into a 4-byte value, that’s because we’re on a 64-bit system.


Now, we have several lines where the i value is incremented and whatnot. But the crucial line is just one short of the segmentation fault. We need to understand what’s inside those registers or if we can access them at all.

RDX register

And turns out we can’t. It’s like we had earlier. But why? How can we know that this address is off limits? How do we know that?

proc mappings

In Linux, you can view the memory maps of any process through /proc/<pid>/maps. It is important to understand what a sample output provides before we can proceed. I’m not going to elaborate too much, but basically:

/proc maps

The first line is the code (or text), the actual binary instructions. The second line shows data, which stores all initialized global variables. The third section is the heap, which is used for dynamic allocations, like malloc. Sometimes, it also includes the .bss segment, which stores statically linked variables and uninitialized global variables. When the .bss segment is small, it can reside inside the data segment.

After that, you get shared libraries, and the first one is the dynamic linker itself. Finally, you get the stack. The two last lines are the Linux gating mechanisms for fast system calls, which replaces the int 0x80 system call that was used in olden days. As you may notice, there are still more memory addresses above the last line, reserved by the kernel.

So here, at a glance, you can examine how your process resides in the memory. When a program is executed through gdb, you can view its memory allocations using the info proc mappings command.

info proc mappings


Three lines, code, data, heap. And for heap, we can see that the end address is 0x522000. And we can’t be using that, so we get our lovely segmentation fault. Back to C code, we will need to figure out what we did wrong, whether we molested our integer, tried an illegal allocation or double freeing or whatever.

Now, if you really wanna go Rain Man on this, you can start counting bytes. In general, we use a single page for code, because our executable is small. We use a single page for data. And then, there’s some heap space, a total of 0x21000, which is 132KB or more specifically 135168 bytes.

On the other hand, we ran through 33788 iterations of the for loop, each 4 bytes in size, as we’re on a 64-bit system. Not 33787 as you may assume from the print output in our program run, but one more, because we started counting i at value 0.

So we get 135152 bytes, which is 16 bytes less that our heap. So you may ask, where did the extra 16 bytes go? Well, we can use the examine command again and check more accurately what happens at the start address.


We print eight 4-byte hexadecimal values. The first 16 bytes are the heap header and the count starts at address 0x501010. So we’re all good here, and we know why we got our nasty segmentation fault. We can examine our source code and try to figure out what we did wrong. Two examples, two problems solved.

Now, we will talk some more about using gdb in general, including collecting application cores and analyzing them, attaching to running processes, more tips on popular gdb commands, and we’ll see yet another example, which shows when gdb is not really useful and yet the assembly dump will tell us all we need, even if we do not have sources.

General advice & more stuff

Analyzing application cores

Similar to kernel crashes, application crashes can create cores that you can analyze later on. There are a few things that you need to make sure are properly set in the system before you can analyze cores.

Enable application cores

You will have to make sure that you can create cores. This is governed via sysctl, but you can also make changes on the fly. Depending on your shell, you will use either the limit or ulimit builtins.

ulimit -c unlimited

And for TCSH:

limit coredumpsize unlimited

Core format

By default, the core will be dumped in the current directory where the binary was executed. But the core name might not be useful of meaningful. So you can change its format, which is governed by the core_pattern setting under /proc. For example:

echo “/tmp/core-%p-%u” > /proc/sys/kernel/core_pattern

This will dump a core under /tmp, with the PID and UID suffixed. There are many other available options. You can also set this option permanently via sysctl.conf. For more reading, perhaps you want to consult thiscyberciti article and this Novell cool solution.

Core dumped

Invoke gdb against core file

Next, your application will crash and create a core. Then, use gdb as follows:

gdb <binary> <core>

Read core

The important thing is that gdb successfully read and loaded the symbols. We can now proceed with the analysis, like before. Some functions will not be available to us, as the core is not a running application, but we will still be able to figure out what went wrong.

Attach to a running process

Similarly, you may want to attach gdb to a running process. As it happens, you may have a problem right now, so you cannot restart the program and try to reproduce the issue at leisure. This may not be the most effective way of debugging problems, but it could give you additional information that may not be available otherwise.

The simplest way to demonstrate this is by altering our example with an extra sleep somewhere. Then, while the program is running, find its PID and attach to it.

gdb -p <process id>

Attach to running process

This example also shows the fact the third-party libraries are stripped, so you get function names, but you don’t know the exact lines of code or the variables. Moreover, using the backtrace (bt) command, we see we’re currently sleeping.

Other useful commands

Let’s list down a few other commands you may want to try and use.

show lets you show contents, as simple as that. set lets you configure variables. For example, you may want to see the initial arguments your program started with and then change them. In our heap overflow example, we could try altering the value of i to see if that affects the program.

show & set

The syntax for setting variables is quite simple. set i=4 would do. You can also set registers, but don’t do this if you don’t know what you’re doing. list lets you dump your code. You can list individual lines, specific functions or entire code. By default, you get ten lines printed, sort of like tail.


Another thing you may want to do is inspect stack frames in detail. We’re already familiar with the infocommand, so what we need now is to invoke it against specific frames, as listed in the backtrace (bt) command. In our heap overflow example, there’s only a single frame.

We break in main, run, display the backtrace and then check info frame 0, as shown in the screenshot below. You get a wealth of information, including the instruction pointer (RIP), the saved instruction pointer from a previous frame, the address and the list of arguments, the address and the list of local variables, the previous stack pointer, and saved registers.

info frame

I mentioned backtrace (bt) earlier, and indeed, it is a most valuable command and best used when you don’t know what your program is doing. External commands can be executed using the shell command. For instance, showing the /proc/PID/maps can also be done by using the shell cat /proc/PID/maps instead of info proc mappings as we did before. If for some reason you cannot use either, then you might want to resort toreadelf to try to decipher the binary. Like we used next and stepi, you can use nexti and step. Let’s not forget finish, jump, until, and call. whatis lets you examine variables.

And that’s enough for this section, I guess.

When to use or not use gdb?

All right, gdb is useful when you have reproducible problems and your binaries have been compiled with symbols. You can also try using gdb against third-party functions, but this won’t guarantee much success.

For instance, we know we’re using printf() in our code. So maybe we need to break there? Well, gdb will informs us that the function is not defined and will create a breakpoint pending on future shared library load. Not a bad idea, but do notice that we don’t see any function names for, because we don’t have symbols, and for that matter, we might not even have sources. Without either, it will not be easy figuring out what went wrong.

(gdb) break printf
Function “printf” not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (printf) pending.
(gdb) run
Starting program: /tmp/segfaults/seg
Breakpoint 2 at 0x2aaaaac113f0
Pending breakpoint “printf” resolved

Breakpoint 2, 0x00002aaaaac113f0 in printf () from /lib64/

Finally, using gdb for sporadic, random problems that are not easily reproduced or those that might stem from hardware problems is a hard, grueling task that will yield few results. Even the trivial examples are not so trivial, so imagine what happens on a real production system, with binaries compiled from sources with thousands of lines of code. Still, you get a taste of the goodness, and you’re hooked now.

Lastly, let’s see an example where gdb is both the worst and best tool for analysis. We’ll create an infinite loop program, nothing sinister, just a bit of while true thingie. This kind of program will loop forever, churning CPU.

If you try strace, you see this – useless:


If you try gdb, it works, we can break in main just fine, but after several next commands, even gdb seems to hang. In this case, you will have to interrupt the execution to get back to your code. Now, with symbols, it’s trivial seeing where the problem lies. But let’s assume that we have nothing; we only know where the problem manifests.


All right, enter disassembly once again. The important thing is that our program revolves around two instructions. You have the comparison (cmpl) and the jump (je). We’re in a tight loop. Naughty. But even if you don’t have the sources, even if the program has no symbols and you don’t know what it’s doing, you can still figure out what’s wrong.

forever, disassemble

TL;DR – Read only if you’re bored

Here’s a bit more about what we saw in the disassemble section. It may help you understand a little better how things work.

Let’s examine the top three lines.

0x00000000400534 <+0> push %rbp
0x00000000400535 <+1> mov  %rsp,%rbp
0x00000000400538 <+4> sub  $0x20,%rsp

These three lines are called the Function Prologue, and it’s automatically added by the GCC compiler on the standard x86 (32-bit) and x86_64 (64-bit) architectures. I’m not sure how things behave on other processors.

The Function Prologue has one function [sic] – to preserve the value of the base pointer of the previous frame on the stack, or in other words, the calling function’s stack frame. On the 32-bit architecture, the EBP register is used for this purpose, on the 64-bit architecture, the RBP register.

So the first instructions pushes (decrements) the stack pointer. Note: the stack grows downwards, so to speak, but this is part of those boring prerequisites we talked about earlier. Then, the stack pointer value is copied into the base pointer register. The third instruction allocates space of 20 bytes for function’s local variables. sub $0x20,%rsp can also be translated as %rsp-20. The actual value will depend on the function’s declaration.

Similarly, at the end of the assembly dump, there’s the Function Epilogue, which does exactly the same like the Prologue, in reverse. The epilogue consists of the leave and ret instructions. In our example, leaveq,retq.

In between, we have all kinds of instructions that depend entirely on how the function is written and what it does. We’ve seen enough earlier, so no need to elaborate more.

To more know about stack frames, you use the backtrace (bt) command and then info. In our two examples, the trace is simple, with just one frame each, but some programs may have 10-15 frames, etc. This is especially useful if you don’t know what the binary is supposed to be doing.


Some more reading

You can read more about AT&T Assembly in this quick ‘n’ dirty guide.

You can also read more about the x86 (and x86_64) assembly language on Wikipedia.

And let’s not forget Call stack.


This was ultra-geeky, I admit, but I think you liked it, if you got this far. Working with gdb is not a simple deal, but the tool is extremely versatile and powerful. You will need a lot of time mastering its commands and abilities to the fullest, best done with real problems that truly emphasize what you’re doing. For starters, grab some bad C code and start practicing.

We learned how to work our way through the code, with break points and conditions, next and stepi commands, assembly dumps, info on various important elements, and whatnot. Alongside the vast array of other powerful tools and admin hacks that I’ve taught you before, your Linux pimpage ought to climb majestically. Stay sweet.

A thorough Introduction Guide To Docker Containers

Let me start with a big promise. You will absolutely LOVE this article today. It’s going to be long, detailed and highly useful. Think GRUB, GRUB2. The same thing here. Only we will tackle Docker, a nice distribution platform that wraps the Linux Containers (LXC) technology in a simple, convenient way.

I will show you how to get started, and then we will create our own containers with SSH and Apache, learn how to use Dockerfiles, expose service ports, and solve an immense number of little bugs and problems that normally never get addressed in public forums. Please, without further ado, follow me.


Table of Contents

  1. Introduction
  2. Docker implementation
  3. Getting started
  4. Docker commands
  5. Pull image
  6. Start a Docker container
  7. Install Apache & SSH
    1. Start service
    2. Apache service
    3. SSH service
  8. Check if Web server is up
    1. Expose incoming ports
    2. Check IP address
    3. Testing new configuration
  9. Check if SSH works
    1. Wait, what is the root password?
  10. Commit image
  11. Dockerfile
    1. Build image
    2. Test image
  12. Alternative build
    1. COPY instruction
  13. Advantages of containers
  14. Problems you may encounter & troubleshooting
  15. Additional commands
    1. Differences between exec and attach
    2. Differences between start and run
    3. Differences between build and create
  16. This is just a beginning …
  17. More reading
  18. Conclusion


I have given a brief overview of the technology in a Gizmo’s Freeware article sometime last year. Now, we are going to get serious about using Docker. First, it is important to remember that this framework allows you to use LXC in a convenient manner, without having to worry about all the little details. It is the next step in this world, the same way OpenStack is the next evolutionary step in the virtualization world. Let me give you some history and analogies.

Virtualization began with software that lets you abstractize your hardware. Then, to make things speedier, virtualization programs began using hardware acceleration, and then you also got paravirtualization. In the end, hypervisors began popping up like mushrooms after rain, and it became somewhat difficult to provision and manage them all. This is the core reason for concepts like OpenStack, which hide different platforms under a unified API.

The containers began their way in a similar manner. First, we had the chroot, but processes running inside the jailed environment shared the same namespace and fought for the same resources. Then, we got the kexec system call, which let us boot into the context of another kernel without going through the BIOS. Then, control groups came about, allowing us to partition system resources like CPU, memory and others into subgroups, thus allowing better control, hence the name, of processes running on the system.

Later on, the Linux kernel began offering a full isolation of resources, using cgroups as the basic partitioning mechanism. Technically, this is a system-level virtualization technology, allowing you to run multiple instances of the running kernel on top of the control host inside self-contained environments, with the added bonus of very little performance penalty and overhead.

Several competing technologies tried to offer similar solutions, like OpenVZ, but the community eventually narrowed down its focus to the native enablement inside the mainline kernel, and this seems to be the future direction. Still, LXC remains somewhat difficult to use, as a fair amount of technical knowledge and scripting is required to get the containers running.

This is where Docker comes into place. It tries to take away the gritty pieces and offer a simple method of spawning new container instances without worrying about the infrastructure backend. Well, almost. But the level of difficulty is much less.

Another strong advantage of Docker is a widespread community acceptance, as well as the emphasis on integration with cloud services. Here we go full buzzword, and this means naming some of the big players like AWS, Hadoop, Azure, Jenkins and others. Then we can also talk about Platform as a Service (Paas), and you can imagine how much money and focus this is going to get in the coming years. The technological landscape is huge and confusing, and it’s definitely going to keep on changing and evolving, with more and more concepts and wrapper technologies coming into life and building on top of Docker.

But we want to focus on the technological side. Once we master the basic, we will slowly expand and began utilizing the strong integration capabilities, the flexibility of the solution, and work on making our cloud ecosystem expertise varied, automated and just pure rad. That won’t happen right now, but I want to help you navigate the first few miles, or should we say kilometers, of the muddy startup waters, so you can begin using Docker in a sensible, efficient way. Since this is a young technology, it’s Wild West out there, and most of the online documentation, tips, tutorials and whatnot are outdated, copy & paste versions that do not help anyone, and largely incomplete. I want to fix that today.

Docker implementation

A bit more boring stuff before we do some cool things. Anyhow, Docker is mostly about LXC, but not just. It’s been designed to be extensible, and it can also interface with libvirt and systemd. In a way, this makes it almost like a hyper-hypervisor, as there’s potential for future growth, and when additional modules are added, it could effectively replace classic hypervisors like Xen or KVM or anything using libvirt and friends.

Docker diagram

This be a public domain image, if you wondered.

Getting started

We will demonstrate using CentOS 7. Not Ubuntu. Most of the online stuff focuses on Ubuntu, but I want to show you how it’s done using as-near-as-enterprise flavor of Linux as possible, because if you’re going to be using Docker, it’s gonna be somewhere business like. The first thing is to install docker:

yum install docker-io

Once the software is installed, you can start using it. However, you may encounter the following two issues the first time you attempt to run docker commands:

docker <any one command>
FATA[0000] Get http:///var/run/docker.sock/v1.18/images/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?

And the other error is:

docker <any one command>
FATA[0000] Get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

The reason is, you need to start the Docker service first. Moreover, you must run the technology as root, because Docker needs access to some rather sensitive pieces of the system, and interact with the kernel. That’s how it works.

systemctl start docker

Now we can go crazy and begin using Docker.

Docker commands

The basic thing is to run docker help to get the available list of commands. I will not go through all the options. We will learn more about them as we go along. In general, if you’re ever in doubt, you should consult the pretty decent online documentation. The complete CLI reference also kicks ass. And then, there’s also an excellent cheat sheet on GitHub. But our first mission will be to download a new Docker image and then run our first instance.

Pull image

There are many available images. We want to practice with CentOS. This is a good starting point. An official repository is available, and it lists all the supported images and tags. Indeed, at this point, we need to understand how Docker images are labeled.

The naming convention is repository:tag, for example centos:latest. In other words, we want the latest CentOS image. But the require image might as well be centos:6.6. All right, let’s do it.

Pulling image

Now let’s list the images by running the docker images command:


Start a Docker container

As we’ve seen in my original tutorial, the simplest example is to run a shell:

docker run -ti centos:centos7 /bin/bash

So what do we have here? We are running a new container instance with its own TTY (-t) and STDIN (-i), from the CentOS 7 image, with a BASH shell. With a few seconds, you will get a new shell inside the container. Now, it’s a very basic, very stripped-down operating system, but you can start building things inside it.

Run container

Container running top

Install Apache & SSH

Let’s setup a Web server, which will also have SSH access. To this end, we will need to do some rather basic installations. Grab Apache (httpd) and SSHD (openssh-server), and configure them. This has nothing to do with Docker, per se, but it’s a useful exercise.

How, some of you may clamor, wait, you don’t need SSH inside a container, it’s a security risk and whatnot. Well, maybe, yes and no, depending on what you need and what you intend to use the container for. But let’s leave the security considerations aside. The purpose of the exercise is to learn how to setup and run ANY service.

Start service

You might want to start your Apache using an init script or a systemd command. This will not quite work. Specifically for CentOS, it comes with systemd, but more importantly, the container does not have its own systemd. If you try, the commands will fail.

systemctl start httpd
Failed to get D-Bus connection: No connection to service manager.

There are hacks around this problem, and we will learn about some of these in a future tutorial. But in general, given the lightweight and simple nature of containers, you do not really need a fully fledged startup service to run your processes. This does add some complexity.

Apache service

To run Apache (HTTPD), just execute /usr/sbin/httpd – or an equivalent command in your distro. The service should start, most likely with a warning that you have not configured your ServerName directive in httpd.conf. We have learned how to do this in my rather extensive Apache guide.

AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using Set the ‘ServerName’ directive globally to suppress this message

SSH service

With SSHD, run /usr/sbin/sshd.

/usr/sbin/sshd -f /etc/ssh/sshd_config
Could not load host key: /etc/ssh/ssh_host_rsa_key
Could not load host key: /etc/ssh/ssh_host_dsa_key
Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

You will also fail, because you won’t have all the keys. Normally, startup scripts take of this, so you will need to run the ssh-keygen command once before the service starts correctly. Either one of the two commands will work:

/usr/bin/ssh-keygen -t rsa -f <path to file>

/usr/bin/ssh-keygen -A
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA ED25519

Check if Web server is up

Now, inside the container, we can see that Apache is indeed running.

ps -ef|grep apache
apache      87    86  0 10:47 ?        00:00:00 /usr/sbin/httpd
apache      88    86  0 10:47 ?        00:00:00 /usr/sbin/httpd
apache      89    86  0 10:47 ?        00:00:00 /usr/sbin/httpd
apache      90    86  0 10:47 ?        00:00:00 /usr/sbin/httpd
apache      91    86  0 10:47 ?        00:00:00 /usr/sbin/httpd

But what if we want to check external connectivity? At this point, we have a couple of problems at our hand. One, we have not setup any open ports, so to speak. Two, we do not know what the IP address of our container is. Now, if you try to run the ifconfig inside the BASH shell, you won’t get anywhere, because the necessary package containing the basic networking commands is not installed. Good, because it makes our container slim and secure.

Expose incoming ports

Like with any Web server, we will need to allow incoming connections. We will use the default port 80. This is no different than port forwarding in your router, allowing firewall policies and whatnot. With Docker, there are several ways you can achieve the desired result.

When starting a new container with the run command, you can use -p option to specify which ports to open. You can choose a single port or a range of ports, and you can also map both the host port (hostPort) and container port (containerPort). For instance:

  • -p 80 will expose container port 80. It will be automatically mapped to a random port on the host. We will learn later on how to identify the correct port.
  • -p 80:80 will map the container port to the host port 80. This means you do not need to know the internal IP address of the container. There is an element of internal NAT involved, which goes through the Docker virtual interface. We will discuss this soon. Moreover, if you use this method, only a single container will be able to bind to port 80. If you want to use multiple Web servers with different IP addresses, you will have to set them up each on a different port.

docker run -ti -p 22:22 -p 80:80 image-1:latest
FATA[0000] Error response from daemon: Cannot start container 64bd520e2d95a699156f5d40331d1aba972039c3c201a97268d61c6ed17e1619: Bind for failed: port is already allocated

There are many additional considerations. IP forwarding, bridged networks, public and private networks, subnet ranges, firewall rules, load balancing, and more. At the moment, we do not need to worry about these.

There is also an additional method of how we can expose port, but we will discuss that later on, when we touch on the topic of Dockerfiles, which are templates for building new images. For now, we need to remember to run our images with the -p option.

Check IP address

If you want to leave your host ports free, then you can omit the hostPort piece. In that case, you can connect to the container directly, using its IP address and Web server port. To do that, we need to figure our the container details:

docker inspect <container name or ID>

This will give a very long list of details, much like the KVM XML config, except this one is written in JSON, which is another modern and ugly format for data. Readable but extremely ugly.

docker inspect distracted_euclid
“AppArmorProfile”: “”,
“Args”: [],
“Config”: {
“AttachStderr”: true,
“AttachStdin”: true,
“AttachStdout”: true,
“Cmd”: [
“CpuShares”: 0,
“Cpuset”: “”,
“Domainname”: “”,
“Entrypoint”: null,
“Env”: [

“ExposedPorts”: {
“80/tcp”: {}
“Hostname”: “43b179c5aec7”,
“Image”: “centos:centos7”,
“Labels”: {},
“MacAddress”: “”,

We can narrow it down to just the IP address.

docker inspect <container name or ID> | grep -i “ipaddr”
“IPAddress”: “”,

Testing new configuration

Let’s start fresh. Launch a new instance, setup Apache, start it. Open a Web browser and test. If it works, then you have properly configured your Web server. Exactly what we wanted.

docker run -it -p 80:80 centos:centos7 /bin/bash

If we check the running container, we can see the port mapping – the output is split over multiple lines for brevety, so please excuse that. Normally, the all-uppercase titles will show as the row header, and then, you will get all the rest printed below, one container per line.

# docker ps
CONTAINER ID        IMAGE               COMMAND
43b179c5aec7        centos:centos7      “/bin/bash”

CREATED             STATUS              PORTS
2 hours ago         Up 2 hours>80/tcp

NAMES               distracted_euclid

And in the browser, we get:

Web server running

Optional: Now, the internal IP address range will only be accessible on the host. If you want to make it accessible from other machines, you will need your NAT and IP forwarding. And if you want to use names, then you will need to properly configure the /etc/hosts as well as DNS. For container, this can be done using the –add-host=”host:IP” directive when running a new instance.

Another note: Remember that Docker has its own internal networking, much like VirtualBox and KVM, as we’ve seen in my other tutorials. It’s a fairly extensive /16 network, so you have quite a lot of freedom. On the host:

# /sbin/ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet  netmask  broadcast
inet6 fe80::5484:7aff:fefe:9799  prefixlen 64  scopeid 0x20<link>
ether 56:84:7a:fe:97:99  txqueuelen 0  (Ethernet)
RX packets 6199  bytes 333408 (325.5 KiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 11037  bytes 32736299 (31.2 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Check if SSH works

We need to do the same exercise with SSH. Again, this means exposing port 22, and we have several options available. To make it more interesting, let’s try with a random port assignment:

docker run -ti -p 20 -p 80 centos:centos7 /bin/bash

And if we check with docker ps, specifically for ports:>22/tcp,>80/tcp   boring_mcclintock

This means you can connect to the docker0 IP address, ports as specified above in the docker ps command output, and this equivalent to actually connecting to the container IP directly, on their service port. This can be useful, because you do not need to worry about the internal IP address that your container uses, and it can simplify forwarding. Now, let’s try to connect. We can use the host port, or we can use the container IP directly.

ssh -p 49117

Either way, we will get what we need, for instance:

The authenticity of host ‘ (’ can’t be established. ECDSA key fingerprint is 00:4b:de:91:60:e5:22:cc:f7:89:01:19:3e:61:cb:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘’ (ECDSA) to the list of known hosts.
root@’s password:

Wait, what is the root password?

We will fail because we do not have the root password. So what do we do now? Again, we have several options. First, try to change the root password inside the container using the passwd command. But this won’t work, because the passwd utility is not installed. We can then grab the necessary RPM and set it up inside the container. On the host, check the dependencies:

rpm -q –whatprovides /etc/passwd

But this is a security vulnerability. We want our containers to be lean. So we can just copy the password hash from /etc/shadow on the host into the container. Later, we will learn about a more streamlined way of doing it.

Another thing that strikes quite clearly is that we are repeating all our actions. This is not efficient, and this is why we want to preserve changes we have done to our container. The next section handles that.

SSH success

Commit image

After you’ve made changes to the container, you may want to commit it. In other words, when starting a new container later on, you will not need to repeat all the steps from scratch, you will be able to reuse your existing work and save time and bandwidth. You can commit an image based on its ID or its alias:

docker commit <container name or ID> <new image>

For example, we get the following:

docker commit 43b179c5aec7 myapache3

Commit image

Check the list of images again:

Image committed


A more streamlined way of creating your images is to use Dockerfiles. In a way, it’s like using Makefile for compilation, only in Docker format. Or an RPM specfile if you will. Basically, in any one “build” directory, create a Dockerfile. We will learn what things we can put inside one, and why we want it for our Apache + SSH exercise. Then, we will build a new image from it. We can combine it with our committed images to preserve changes already done inside the container, like the installation of software, to make it faster and save network utilization.

Before we go any further, let’s take a look at a Dockerfile that we will be using for our exercise. At the moment, the commands may not make much sense, but they soon will.

FROM myhttptest2:latest


CMD [“/usr/sbin/sshd”, “-D”]


RUN mkdir -p /run/httpd
CMD [“/usr/sbin/httpd”, “-D”, “FOREGROUND”]

What do we have here?

  • The FROM directory tells us what repo:tag to use as the baseline. In our case, it’s one of the committed images that already contains the httpd and sshd binaries, SSH keys, and a bit more.
  • EXPOSE 22 – This line exposes port 22 inside the container. We can map it further using the -p option at runtime. The same is true for EXPOSE 80, which is relevant for the Web server.
  • CMD [“/usr/sbin/sshd”, “-D”] – This instructions runs an executable, with optional arguments. It is as simple as that.
  • RUN mkdir -p /run/httpd – This instruction runs a command in a new layer on top of the base image – and COMMITS the results. This is very important to remember, as we will soon discuss what happens if you don’t use the RUN mkdir thingie with Apache.
  • CMD [“/usr/sbin/httpd”, “-D”, “FOREGROUND”] – We run the server, in the foreground. The last bit is optional, but for the time being, you can start Apache this way. Good enough.

As you can see, Dockerfiles aren’t that complex or difficult to write, but they are highly useful. You can pretty much add anything you want. Using these templates form a basis for automation, and with conditional logic, you can create all sorts of scenarios and spawn containers that match your requirements.

Build image

Once you have a Dockerfile in place, it’s time to build a new image. Dockerfiles must follow a strict convention, just like Makefiles. It’s best to keep different image builds in separate sub-directories. For example:

docker build -t test5 .
Sending build context to Docker daemon 41.47 kB
Sending build context to Docker daemon
Step 0 : FROM myapache4:latest
—> 7505c70235e6
Step 1 : EXPOSE 22 80
—> Using cache
—> 58f11217c3e3
Step 2 : CMD /usr/sbin/sshd -D
—> Using cache
—> 628c3d6b5399
Step 3 : RUN mkdir -p /run/httpd
—> Using cache
—> 5fc118f61a4d
Step 4 : CMD /usr/sbin/httpd -D FOREGROUND
—> Using cache
—> d892acd86198
Successfully built d892acd86198

The command tells us the following: -t repository name from a Dockerfile stored in the current directory (.). That’s all. Very simple and elegant.

Test image

Run a new container from the created image. If everything went smoothly, you should have both SSH connectivity, as well as a running Web server in place. Again, all the usual network related rules apply.

Running successfully, built from Dockerfile

Alternative build

Once you have the knowledge how do it on your own, you can try one of the official Apache builds. Indeed, the Docker repository contains a lot of good stuff, so you should definitely invest time checking available templates. For Apache, you only need the following in your Dockerfile – the second like is optional.

FROM httpd:2.4
COPY ./public-html/ /usr/local/apache2/htdocs/

COPY instruction

What do we have above? Basically, in the Dockerfile, we have the declaration what template to use. And then, we have a COPY instructions, which will look for a public-html directory in the current folder and copy it into the container during the build. In the same manner, you can also copy your httpd.conf file. Depending on your distribution, the paths and filenames might differ. Finally, after building the image and running the container:

docker run -ti -p 22 -p 80 image-1:latest
AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using Set the ‘ServerName’ directive globally to suppress this message
[Thu Apr 16 21:08:35.967670 2015] [mpm_event:notice] [pid 1:tid 140302870259584] AH00489: Apache/2.4.12 (Unix) configured — resuming normal operations
[Thu Apr 16 21:08:35.976879 2015] [core:notice] [pid 1:tid 140302870259584] AH00094: Command line: ‘httpd -D FOREGROUND’

Default HTTPD works

Advantages of containers

There are many good reasons why you want to use this technology. But let’s just briefly focus on what we gain by running these tiny, isolated instances. Sure, there’s a lot happening under the hood, in the kernel, but in general, the memory footprint of spawned containers is fairly small. In our case, the SSH + Apache containers use a tiny fraction of extra memory. Compare this to any virtualization technology.

Container memory

Low memory usage

Problems you may encounter & troubleshooting

Let’s go back to the Apache example, and now you will also learn why so many online tutorials sin the sin of copy & pasting information without checking, and why most of the advice is not correct, unfortunately. It has to do with, what do you do if your Apache server seems to die within a second or two after launching the container? Indeed, if this happens, you want to step into the container and troubleshoot. To that end, you can use the docker exec command to attach a shell to the instance.

docker exec -ti boring_mcclintock /bin/bash

Then, it comes down to reading logs and trying to figure out what might have gone wrong. If your httpd.conf is configured correctly, you will have access and error logs under /var/log/httpd:

[auth_digest:error] [pid 25] (2)No such file or directory: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.25

A typical problem is that you may be a missing /run/httpd directory. If this one does not exist in your container, httpd will start and die. Sounds so simple, but few if any reference mentions this.

While initially playing with containers, I did encounter this issue. Reading online, I found several suggestions, none of which really helped. But I do want to elaborate on them, and how you can make progress in your problem solving, even if intermediate steps aren’t really useful.

Suggestion 1: You must use -D FOREGROUND to run Apache, and you must also use ENTRYPOINT rather than CMD. The difference between the two instructions is very subtle. And it does not solve our problem in any way.

ENTRYPOINT [“/usr/sbin/httpd”]

Suggestion 2: Use a separate startup script, which could work around any issues with the starting or restarting of the httpd service. In other words, the Dockerfile becomes something like this:

COPY ./ /
RUN chmod -v +x /
CMD [“/”]

And the contents of the script are along the lines of:


rm -rf /run/httpd/*

exec /usr/sbin/apachectl -D FOREGROUND

Almost there. Remove any old leftover PID files, but these are normally not stored under /run/httpd. Instead, you will find them under /var/run/httpd. Moreover, we are not certain that this directory exists.

Finally, the idea is to work around any problems with the execution of a separation shell inside which the httpd thread is spawned. While it does provide us with additional, useful lessons on how to manage the container, with COPY and RUN instructions, it’s not what we need to fix the issue.

Step 3 : EXPOSE 80
—> Using cache
—> 108785c8e507
Step 4 : COPY ./ /
—> 582d795d59d4
Removing intermediate container 7ff5b58b40bf
Step 5 : RUN chmod -v +x /
—> Running in 56fadf4dd2d4
mode of ‘/’ changed from 0644 (rw-r–r–) to 0755 (rwxr-xr-x)
—> 928640f680cf
Removing intermediate container 56fadf4dd2d4
Step 6 : CMD /
—> Running in f9c6b30795e2
—> b2dcc2818a27
Removing intermediate container f9c6b30795e2
Successfully built b2dcc2818a27

This won’t work, because apachectl is an unsupported command for managing httpd, plus we have seen problems using startup scripts and utilities earlier, and we will work on fixing this in a separate tutorial.

docker run -ti -p 80 image-2:latest
Passing arguments to httpd using apachectl is no longer supported. You can only start/stop/restart httpd using this script. If you want to pass extra arguments to httpd, edit the /etc/sysconfig/httpd config file.

But it is useful to try these different things, to get the hang of it. Unfortunately, it also highlights the lack of maturity and the somewhat inadequate documentation for this technology out there.

Additional commands

There are many ways you can interact with your container. If you do not want to attach a new shell to a running instance, you can use a subset of docker commands directly against the container ID or name:

docker <command> <container name or ID>

For instance, to get the top output from the container:

docker top boring_stallman

If you have too many images, some of which have just been used for testing, then you can remove them to free up some of your disk space. This can be done using the docker rmi command.

# docker rmi -f test7
Untagged: test7:latest
Deleted: d0505b88466a97b73d083434b2dd0e7b59b9a5e8d0438b1bf8c6c
Deleted: 5fc118f61bf856f6f3d90e0e71076b737fa7cc58cd56785ea7904
Deleted: 628c3d6b53992521c9c1fdda4148693347c3d10b1d130f7e091e7
Deleted: 58f11217c3e31206b4e41d07100a797cd4d17e4569b0fdb8b7a18
Deleted: 7505c70235e638c54028ea5b63eba2b691de6bee67c2cb5e2861a

Then, you can also run your containers in the background. Using the -d flag will do exactly that, and you will get the shell prompt back. This is also useful if you do not mask signals, so if you accidentally break in your shell, you might kill the container when it’s running in the foreground.

docker run -d -ti -p 80 image-3:latest

You can also check events, examine changes inside a container’s filesystem as well as check history, so you basically have a version control in place, export or import tarred images to and from remote locations, including over the Web, and more.

Differences between exec and attach

If you read through the documentation, you will notice you can connect to a running container using eitherexec or attach commands. So what’s the difference, you may ask? If we look at the official documentation, then:

The docker exec command runs a new command in a running container. The command started using docker exec only runs while the container’s primary process (PID 1) is running, and it is not restarted if the container is restarted.

On the other hand, attach gives you the following:

The docker attach command allows you to attach to a running container using the container’s ID or name, either to view its ongoing output or to control it interactively. You can attach to the same contained process multiple times simultaneously, screen sharing style, or quickly view the progress of your daemonized process. You can detach from the container (and leave it running) with CTRL-p CTRL-q (for a quiet exit) or CTRL-c which will send a SIGKILL to the container. When you are attached to a container, and exit its main process, the process’s exit code will be returned to the client.

In other words, with attach, you will get a shell, and be able to do whatever you need. With exec, you can issue commands that do not require any interaction, but with you use a shell in combination with exec, you will achieve the same result as if you used attach.

Differences between start and run

Start is used to resume the execution of a stopped container. It is not used to start a fresh instance. For that, you have the run command. The choice of words could have been better.

Differences between build and create

The first command is used to create a new image from a Dockerfile. On the other hand, the latter is used to create a new container using command line options and arguments. Create lets you specify container settings, too, like network configurations, resource limitations and other settings, which affect the container from the outside, whereas the changes implemented by the build command will be reflected inside it, once you start an instance. And by start, I mean run. Get it?

This is just a beginning …

There are a million more things we can do: using systemd enabled containers, policies, security, resource constraints, proxying, signals, other networking and storage options including the super-critical question of how to mount data volumes inside containers so that data does not get destroyed when containers die, additional pure LXC commands, and more. We’ve barely scratched the surface. But now, we know what to do. And we’ll get there. Slowly but surely.

More reading

I recommend you allocate a few hours and then spend some honest time reading all of the below, in detail. Then practice. This is the only way you will really fully understand and embrace the concepts.

Dockerizing an SSH Deamon Service

Differences between save and export in Docker

Docker Explained: Using Dockerfiles to Automate Building of Images


We’re done with this tutorial for today. Hopefully, you’ve found it useful. In a nutshell, it does explain quite a few things, including how to get started with Docker, how to pull new images, run basic containers, add services like SSH and Apache, commit changes to a file, expose incoming ports, build new images with Dockerfiles, lots of troubleshooting of problems, additional commands, and more. Eventful and colorful, I’d dare say.

In the future, we will expand significantly on what we learned here, and focus on various helper technologies like supervisord for instance, we will learn how to mount filesystems, work on administration and orchestration, and many other cool things. Docker is a very nice concept, and if used correctly, it can make your virtual world easier and more elegant. The initial few steps are rough, but with some luck, this guide will have provided you with the right dose of karma to get happily and confidently underway. Ping me if you have any requests or desires. Technology related, of course. We’re done.

Easily manage Linux services with chkconfig and sysv-rc-conf utilities

If you want to manage the services in your Linux distribution without manually hacking the scripts in runlevel directories, you may want to use the chkconfig and sysv-rc-conf utilities. chkconfig is used on RedHat-based distros, sysv-rc-conf is for Debian variants.

Follow me.


The utility is included by default with RedHat (and sons) and SUSE. This means that system administration is a simple affair on these distributions. All you have to do is learn the basic chkconfig syntax. The commands must be run as root.

chkconfig –levels <levels> <service> <switch>

What do we have here?

–levels <levels> defines the runlevels in which you want the particular service switch to be turned. The runlevels are listed one next to another without spaces, commas or any other delimiter. Simply a string of numbers, e.g. 34 – signifies runlevels 3 and 4.

<service> is the actual service we want enabled/disabled for the listed runlevels.

<switch> is on/off, meaning enabled/disabled service for the particular runlevels.

Let’s see a practical example:


Here, we turned the SSH service (daemon called sshd) off and on in runlevel 2. Simple, eh?

And we can also list all services using the –list flag:

chkconfig –list


Let’s see what sysv-rc-conf offers:


Let’s take a look on sysv-rc-conf on Ubuntu.

By default, the utility is not installed. We will install it via Synaptic.


The usage model is practically identical to chkconfig. The only difference is that sysv-rc-conf also has a text-based GUI. You can use a simple, table-like interface to manage services, using Spacebar to toggle their status (enable/disable), with letter X as the switch.


The other functions are identical, including –list and –levels:



One important thing: please note in the screenshot above that I’ve written ssh instead of sshd. This is a mistake, but there was no alert from the system. Indeed, both chkconfig and sysv-rc-conf will ignore your attempts for non-existent services, so pay attention to possible mistakes!

What do they do?

You may be asking yourselves what these tools do. Well, without getting too technical, I’ll try to portray a simple enough picture.

Using a classic model for example, Linux works in runlevels, sets of modes similar somewhat to Windows Safe mode with/without command prompt and/or networking. Runlevels define what runs when the system boots. Runlevel 1 is the maintenance mode, also known as single mode, with root access only and minimal shell. Runlevel 3 is the system level with networking and no GUI. Modern desktop distros usually boot in runlevel 5, which is the full system level, with networking and GUI. And so forth.

Definitions for what services should boot in each runlevel are listed under /etc/rcX.d/ directories, where X signifies the runlevel. Inside these directories, you will find scripts, starting with letter S and a two-digit number, e.g. S04somescript. These are the Start scripts for relevant services. Starting according to their number, from the lowest to the highest, the scripts are invoked and run on startup.

There is also a set of Kill scripts, which begin with K and are used to stop the services when the system is being shut down, rebooted or runlevels switched.

What the chkconfig and sysv-rc-conf do is merely create or remove these scripts from relevant directories. Simple and beautiful.

For more details, you may want to read the man pages.


Admining Linux systems is a serious, difficult task. You can do without the tedious, futile tasks like manually configuring scripts for the system services, worrying about spelling mistakes or bad syntax. Let the machine work hard, instead of you.

chkconfig and sysv-rc-conf are a must in the hands of even the most bored system administrator. They can greatly simplify the routine of properly configuring the systems for your environment.

Connect to WiFi Network From Command Line In Linux

How many of you failed to connect to WiFi network in Linux? Did you bumped into issues like the followings in different forums, discussion page, blogs? I am sure everyone did at some point. Following list shows just the results from Page 1 of a Google search result with “Unable to connect to WiFi network in Linux” keywords.Connect to WiFi network in Linux from command line - blackMORE Ops

  1. Cannot connect to wifi at home after upgrade to ubuntu 14.04
  2. Arch Linux not connecting to Wifi anymore
  3. I can’t connect to my wifi
  4. Cannot connect to WiFi
  5. Ubuntu 13.04 can detect wi-fi but can’t connect
  6. Unable to connect to wireless network ath9k
  7. Crazy! I can see wireless network but can’t connect
  8. Unable to connect to Wifi Access point in Debian 7
  9. Unable to connect Wireless

Following guide explains how you can connect to a WiFi network in Linux from command Line. This guide will take you through the steps for connecting to a WPA/WPA2 WiFi network.


  • WiFi network from command line – Required tools
  • Linux WPA/WPA2/IEEE 802.1X Supplicant
    • iw – Linux Wireless
    • ip – ip program in Linux
    • ping
  • Step 1: Find available WiFi adapters – WiFi network from command line
  • Step 2: Check device status – WiFi network from command line
  • Step 3: Bring up the WiFi interface – WiFi network from command line
  • Step 4: Check the connection status – WiFi network from command line
  • Step 5: Scan to find WiFi Network – WiFi network from command line
  • Step 6: Generate a wpa/wpa2 configuration file – WiFi network from command line
  • Step 7: Connect to WPA/WPA2 WiFi network – WiFi network from command line
  • Step 8: Get an IP using dhclient – WiFi network from command line
  • Step 9: Test connectivity – WiFi network from command line
  • Conclusion

WiFi network from command line – Required tools

Following tools are required to connect to WiFi network in Linux from command line

  1. wpa_supplicant
  2. iw
  3. ip
  4. ping

Before we jump into technical jargons let’s just quickly go over each item at a time.

Linux WPA/WPA2/IEEE 802.1X Supplicant

wpa_supplicant is a WPA Supplicant for Linux, BSD, Mac OS X, and Windows with support for WPA and WPA2 (IEEE 802.11i / RSN). It is suitable for both desktop/laptop computers and embedded systems. Supplicant is the IEEE 802.1X/WPA component that is used in the client stations. It implements key negotiation with a WPA Authenticator and it controls the roaming and IEEE 802.11 authentication/association of the wlan driver.

iw – Linux Wireless

iw is a new nl80211 based CLI configuration utility for wireless devices. It supports all new drivers that have been added to the kernel recently. The old tool iwconfing, which uses Wireless Extensions interface, is deprecated and it’s strongly recommended to switch to iw and nl80211.

ip – ip program in Linux

ip is used to show / manipulate routing, devices, policy routing and tunnels. It is used for enabling/disabling devices and it helps you to find general networking informations. ip was written by Alexey N. Kuznetsov and added in Linux 2.2. Use man ip to see full help/man page.


Good old ping For every ping, there shall be a pong …. ping-pong – ping-pong – ping-pong … that should explain it.

BTW man ping helps too …

Step 1: Find available WiFi adapters – WiFi network from command line

This actually help .. I mean you need to know your WiFi device name before you go an connect to a WiFi network. So just use the following command that will list all the connected WiFi adapters in your Linux machines.

root@kali:~# iw dev
    Interface wlan0
        ifindex 4
        type managed

Let me explain the output:

This system has 1 physical WiFi adapters.

  1. Designated name: phy#1
  2. Device names: wlan0
  3. Interface Index: 4. Usually as per connected ports (which can be an USB port).
  4. Type: Managed. Type specifies the operational mode of the wireless devices. managed means the device is a WiFi station or client that connects to an access point.

Connect to WiFi network in Linux from command line - Find WiFi adapters - blackMORE Ops-1

Step 2: Check device status – WiFi network from command line

By this time many of you are thinking, why two network devices. The reason I am using two is because I would like to show how a connected and disconnected device looks like side by side. Next command will show you exactly that.

You can check that if the wireless device is up or not using the following command:

root@kali:~# ip link show wlan0
4: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DORMANT qlen 1000
    link/ether 00:60:64:37:4a:30 brd ff:ff:ff:ff:ff:ff

As you can already see, I got once interface (wlan0) as state UP and wlan1 as state DOWN.

Look for the word “UP” inside the brackets in the first line of the output.

Connect to WiFi network in Linux from command line - Check device status- blackMORE Ops-2

In the above example, wlan1 is not UP. Execute the following command to

Step 3: Bring up the WiFi interface – WiFi network from command line

Use the following command to bring up the WiFI interface

root@kali:~# ip link set wlan0 up

Note: If you’re using Ubuntu, Linux Mint, CentOS, Fedora etc. use the command with ‘sudo’ prefix

Connect to WiFi network in Linux from command line - Bring device up - blackMORE Ops-3

If you run the show link command again, you can tell that wlan1 is now UP.

root@kali:~# ip link show wlan0
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT qlen 1000
    link/ether 00:60:64:37:4a:30 brd ff:ff:ff:ff:ff:ff

Step 4: Check the connection status – WiFi network from command line

You can check WiFi network connection status from command line using the following command

root@kali:~# iw wlan0 link
Not connected.

Connect to WiFi network in Linux from command line - Check device connection - blackMORE Ops-4

The above output shows that you are not connected to any network.

Step 5: Scan to find WiFi Network – WiFi network from command line

Scan to find out what WiFi network(s) are detected

root@kali:~# iw wlan0 scan
BSS 9c:97:26:de:12:37 (on wlan0)
    TSF: 5311608514951 usec (61d, 11:26:48)
    freq: 2462
    beacon interval: 100
    capability: ESS Privacy ShortSlotTime (0x0411)
    signal: -53.00 dBm 
    last seen: 104 ms ago
    Information elements from Probe Response frame:
    SSID: blackMOREOps
    Supported rates: 1.0* 2.0* 5.5* 11.0* 18.0 24.0 36.0 54.0 
    DS Parameter set: channel 11
    ERP: Barker_Preamble_Mode
    RSN:     * Version: 1
         * Group cipher: CCMP
         * Pairwise ciphers: CCMP
         * Authentication suites: PSK
         * Capabilities: 16-PTKSA-RC (0x000c)
    Extended supported rates: 6.0 9.0 12.0 48.0 
---- truncated ----

The 2 important pieces of information from the above are the SSID and the security protocol (WPA/WPA2 vs WEP). The SSID from the above example is blackMOREOps. The security protocol is RSN, also commonly referred to as WPA2. The security protocol is important because it determines what tool you use to connect to the network.

— following image is a sample only —

Connect to WiFi network in Linux from command line - Scan Wifi Network using iw - blackMORE Ops - 5

Step 6: Generate a wpa/wpa2 configuration file – WiFi network from command line

Now we will generate a configuration file for wpa_supplicant that contains the pre-shared key (“passphrase“) for the WiFi network.

root@kali:~# wpa_passphrase blackMOREOps >> /etc/wpa_supplicant.conf
(where 'abcd1234' was the Network password)

wpa_passphrase uses SSID as a string, that means you need to type in the passphrase for the WiFi networkblackMOREOps after you run the command.

Connect to WiFi network in Linux from command line - Connect to WPA WPA2 WiFi network - blackMORE Ops - 6

Note: If you’re using Ubuntu, Linux Mint, CentOS, Fedora etc. use the command with ‘sudo’ prefix

wpa_passphrase will create the necessary configuration entries based on your input. Each new network will be added as a new configuration (it wont replace existing configurations) in the configurations file /etc/wpa_supplicant.conf.

root@kali:~# cat /etc/wpa_supplicant.conf 
# reading passphrase from stdin

Step 7: Connect to WPA/WPA2 WiFi network – WiFi network from command line

Now that we have the configuration file, we can use it to connect to the WiFi network. We will be usingwpa_supplicant to connect. Use the following command

root@kali:~# wpa_supplicant -B -D wext -i wlan0 -c /etc/wpa_supplicant.conf
ioctl[SIOCSIWENCODEEXT]: Invalid argument 
ioctl[SIOCSIWENCODEEXT]: Invalid argument 


  • -B means run wpa_supplicant in the background.
  • -D specifies the wireless driver. wext is the generic driver.
  • -c specifies the path for the configuration file.

Connect to WiFi network in Linux from command line - Connect to WPA WPA2 WiFi network - blackMORE Ops - 7

Use the iw command to verify that you are indeed connected to the SSID.

root@kali:~# iw wlan0 link
Connected to 9c:97:00:aa:11:33 (on wlan0)
    SSID: blackMOREOps
    freq: 2412
    RX: 26951 bytes (265 packets)
    TX: 1400 bytes (14 packets)
    signal: -51 dBm
    tx bitrate: 6.5 MBit/s MCS 0

    bss flags:    short-slot-time
    dtim period:    0
    beacon int:    100

Step 8: Get an IP using dhclient – WiFi network from command line

Until step 7, we’ve spent time connecting to the WiFi network. Now use dhclient to get an IP address by DHCP

root@kali:~# dhclient wlan0
Reloading /etc/samba/smb.conf: smbd only.

You can use ip or ifconfig command to verify the IP address assigned by DHCP. The IP address is below.

root@kali:~# ip addr show wlan0
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:60:64:37:4a:30 brd ff:ff:ff:ff:ff:ff
    inet brd scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::260:64ff:fe37:4a30/64 scope link 
       valid_lft forever preferred_lft forever


root@kali:~# ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:60:64:37:4a:30 
 inet addr: Bcast: Mask:
 inet6 addr: fe80::260:64ff:fe37:4a30/64 Scope:Link
 RX packets:23868 errors:0 dropped:0 overruns:0 frame:0
 TX packets:23502 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000 
 RX bytes:22999066 (21.9 MiB) TX bytes:5776947 (5.5 MiB)


Add default routing rule.The last configuration step is to make sure that you have the proper routing rules.

root@kali:~# ip route show 
default via dev wlan0 dev wlan0  proto kernel  scope link  src 

Connect to WiFi network in Linux from command line - Check Routing and DHCP - blackMORE Ops - 8

Step 9: Test connectivity – WiFi network from command line

Ping Google’s IP to confirm network connection (or you can just browse?)

root@kali:~# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_req=3 ttl=42 time=265 ms
64 bytes from icmp_req=4 ttl=42 time=176 ms
64 bytes from icmp_req=5 ttl=42 time=174 ms
64 bytes from icmp_req=6 ttl=42 time=174 ms
--- ping statistics ---
6 packets transmitted, 4 received, 33% packet loss, time 5020ms
rtt min/avg/max/mdev = 174.353/197.683/265.456/39.134 ms


This is a very detailed and long guide. Here is a short summary of all the things you need to do in just few line.

root@kali:~# iw dev
root@kali:~# ip link set wlan0 up
root@kali:~# iw wlan0 scan
root@kali:~# wpa_passphrase blackMOREOps >> /etc/wpa_supplicant.conf
root@kali:~# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
root@kali:~# iw wlan0 link
root@kali:~# dhclient wlan0
root@kali:~# ping
(Where wlan0 is wifi adapter and blackMOREOps is SSID)
(Add Routing manually)
root@kali:~# ip route add default via dev wlan0

At the end of it, you should be able to connect to WiFi network. Depending on the Linux distro you are using and how things go, your commands might be slightly different. Edit commands as required to meet your needs.

Setup DHCP Or Static IP Address From Command Line In Linux

Did you ever had trouble with Network Manager and felt that you need to try to setup DHCP orstatic IP address from command Line in Linux? I once accidentally removed Gnome (my bad, wasn’t paying attention and did an apt-get autoremove -y .. how bad is that.. ) So I was stuck, I couldn’t connect to Internet to reinstall my Gnome Network Manager because I’m in TEXT modenetwork-manager was broken.  I learned a good lesson. you need internet for almost anything these days unless you’ve memorized all those manual command.

This guide will guide you on how to setup DHCP or static IP address from command Line in Linux. It saved me when I was in trouble, hopefully you will find it useful as well.

Note that my network interface is eth0 for this whole guide. Change eth0 to match your network interface.

Static assignment of IP addresses is typically used to eliminate the network traffic associated with DHCP/DNS and to lock an element in the address space to provide a consistent IP target.

Step 1 : STOP and START Networking service

Some people would argue restart would work, but I prefer STOP-START to do a complete rehash. Also if it’s not working already, why bother?

# /etc/init.d/networking stop
 [ ok ] Deconfiguring network interfaces...done.
 # /etc/init.d/networking start
 [ ok ] Configuring network interfaces...done.

Step 2 : STOP and START Network-Manager

If you have some other network manager (i.e. wicd, then start stop that one).

# /etc/init.d/network-manager stop
 [ ok ] Stopping network connection manager: NetworkManager.
 # /etc/init.d/network-manager start
 [ ok ] Starting network connection manager: NetworkManager.

Just for the kicks, following is what restart would do:

 # /etc/init.d/network-manager restart
 [ ok ] Stopping network connection manager: NetworkManager.
 [ ok ] Starting network connection manager: NetworkManager.

Step 3 : Bring up network Interface

Now that we’ve restarted both networking and network-manager services, we can bring our interface eth0 up. For some it will already be up and useless at this point. But we are going to fix that in next few steps.

# ifconfig eth0 up 
# ifup eth0

The next command shows the status of the interface. as you can see, it doesn’t have any IP address assigned to it now.

 # ifconfig eth0
 eth0      Link encap:Ethernet  HWaddr aa:bb:cc:11:22:33
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Setup DHCP or static IP address from command Line in Linux - blackMORE Ops - 6

Step 4 : Setting up IP address – DHCP or Static?

Now we have two options. We can setup DHCP or static IP address from command Line in Linux. If you decide to use DHCP address, ensure your Router is capable to serving DHCP. If you think DHCP was the problem all along, then go for static.

Again, if you’re using static IP address, you might want to investigate what range is supported in the network you are connecting to. (i.e. some networks uses, some uses etc. ranges). For some readers, this might be trial and error method, but it always works.

Step 4.1 – Setup DHCP from command Line in Linux

Assuming that you’ve already completed step 1,2 and 3, you can just use this simple command

The first command updates/etc/network/interfaces file with eth0 interface to use DHCP.

# echo “iface eth0 inet dhcp” >>/etc/network/interfaces

The next command brings up the interface.

# ifconfig eth0 up 
# ifup eth0

With DHCP, you get IP address, subnet mask, broadcast address, Gateway IP and DNS ip addresses. Go to step xxx to test your internet connection.

Step 4.2 – Setup static IP, subnet mask, broadcast address in Linux

Use the following command to setup IP, subnet mask, broadcast address in Linux. Note that I’ve highlighted the IP addresses in red. You will be able to find these details from another device connected to the network or directly from the router or gateways status page. (i.e. some networks uses, some uses etc. ranges)

 # ifconfig eth0
 # ifconfig eth0 netmask
 # ifconfig eth0 broadcast

Next command shows the IP address and details that we’ve set manually.

# ifconfig eth0
 eth0     Link encap:Ethernet  HWaddr aa:bb:cc:11:22:33
 inet addr:  Bcast:  Mask:
 RX packets:19325 errors:0 dropped:0 overruns:0 frame:0
 TX packets:19641 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Because we are doing everything manually, we also need to setup the Gateway address for the interface. Use the following command to add default Gateway route to eth0.

# route add default gw eth0

We can confirm it using the following command:

# route -n
 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface         UG    0      0        0 eth0   U     0      0        0 eth0

Step 4.3 – Alternative way of setting Static IP in a DHCP network

If you’re connected to a network where you have DHCP enabled but want to assign a static IP to your interface, you can use the following command to assign Static IP in a DHCP network, netmask and Gateway.

# echo -e “iface eth0 inet dhcp\n address\n netmask\n gateway″>>/etc/network/interfaces 

At this point if your network interface is not up already, you can bring it up.

# ifconfig eth0 up 
# ifup eth0

Step 4.4 –  Fix missing default Gateway

Looks good to me so far. We’re almost there.

Try to ping (cause if is down, Internet is broken!):

# ping
 PING ( 56(84) bytes of data.
 64 bytes from ( icmp_req=1 ttl=49 time=520 ms
 64 bytes from ( icmp_req=2 ttl=49 time=318 ms
 64 bytes from ( icmp_req=3 ttl=49 time=358 ms
 64 bytes from ( icmp_req=4 ttl=49 time=315 ms
 --- ping statistics ---
 4 packets transmitted, 4 received, 0% packet loss, time 3002ms
 rtt min/avg/max/mdev = 315.863/378.359/520.263/83.643 ms

It worked!

Step 5 : Setting up nameserver / DNS

For most users step 4.4 would be the last step. But in case you get a DNS error you want to assign DNS servers manually, then use the following command:

# echo “nameserver\n nameserver″ >>/etc/resolv.conf

This will add Google Public DNS servers to your resolv.conf file. Now you should be able to ping or browse to any website.


Losing internet connection these days is just painful because we are so dependent on Internet to find usable information. It gets frustrating when you suddenly lose your GUI and/or your Network Manager and all you got is either an Ethernet port or Wireless card to connect to the internet. But then again you need to memorize all these steps.

I’ve tried to made this guide as much generic I can, but if you have a suggestion or if I’ve made a mistake, feel free to comment. Thanks for reading. Please share & RT.

Linux hacks you probably did not know about

This article is a compilation of several interesting, unique command-line tricks that should help you squeeze more juice out of your system, improve your situational awareness of what goes on behind the curtains of the desktop, plus some rather unorthodox solutions that will melt the proverbial socks off your kernel.

Follow me for a round of creative administrative hacking.

1. Run top in batch mode

top is a handy utility for monitoring the utilization of your system. It is invoked from the command line and it works by displaying lots of useful information, including CPU and memory usage, the number of running processes, load, the top resource hitters, and other useful bits. By default, top refreshes its report every 3 seconds.


Most of us use top in this fashion; we run it inside the terminal, look on the statistics for a few seconds and then graciously quit and continue our work.

But what if you wanted to monitor the usage of your system resources unattended? In other words, let some system administration utility run and collect system information and write it to a log file every once in a while. Better yet, what if you wanted to run such a utility only for a given period of time, again without any user interaction?

There are many possible answers:

  • You could schedule a job via cron.
  • You could run a shell script that runs ps every X seconds or so in a loop, incrementing a counter until the desired number of interactions elapsed. But you would also need uptime to check the load and several other commands to monitor disk utilization and what not.

Instead of going wild about trying to patch a script, there’s a much, much simpler solution: top in batch mode.

top can be run non-interactively, in batch mode. Time delay and the number of iterations can be configured, giving you the ability to dictate the data collection as you see fit. Here’s an example:

top -b -d 10 -n 3 >> top-file

We have top running in batch mode (-b). It’s going to refresh every 10 seconds, as specified by the delay (-d) flag, for a total count of 3 iterations (-n). The output will be sent to a file. A few screenshots:

Batch mode 1

Batch mode 2

And that does the trick. Speaking of writing to files …

2. Write to more than one file at once with tee

In general, with static data, this is not a problem. You simply repeat the write operation. With dynamic data, again, this is not that much of a problem. You capture the output into a temporary variable and then write it to a number of files. But there’s an easier and faster way of doing it, without redirection and repetitive write operations. The answer: tee.

tee is a very useful utility that duplicates pipe content. Now, what makes tee really useful is that it can append data to existing files, making it ideal for writing periodic log information to multiple files at once.

Here’s a great example:

ps | tee file1 file2 file3

That’s it! We’re sending the output of the ps command to three different files! Or as many as we want. As you can see in the screenshots below, all three files were created at the same time and they all contain the same data. This is extremely useful for constantly changing output, which you must preserve in multiple instances without typing the same commands over and over like a keyboard-loving monkey.

tee 1

tee 2

tee 3

Now, if you wanted to append data to files, that is periodically update them, you would use the -a flag, like this:

ps | tee -a file1 file2 file3 file4

3. Unleash the accounting power with pacct

Did you know that you can log the completion of every single process running on your machine? You may even want to do this, for security, statistical purposes, load optimization, or any other administrative reason you may think of. By default, process accounting (pacct) may not be activated on your machine. You might have to start it:

/usr/sbin/accton /var/account/pacct

Once this is done, every single process will be logged. You can find the logs under /var/account. The log itself is in binary form, so you will have to use a dumping utility to convert it to human-readable form. To this end, you use the dump-acct utility.

dump-acct pacct

The output may be very long, depending on the activity on your machine and whether you rotate the logs, which you should, since the accounting logs can inflate very quickly.


And there you go, the list of all processes ran on our host since the moment we activated the accounting. The output is printed in nice columns and includes the following, from left to right: process name, user time, system time, effective time, UID, GID, memory, and date. Other ways of starting accounting may be in the following forms:

/etc/init.d/psacct start


/etc/init.d/acct start

In fact, starting accounting using the init script is the preferred way of doing things. However, you should note that accounting is not a service in the typical form. The init script does not look for a running process – it merely checks for the lock file under /var. Therefore, if you turn the accounting on/off using the accton command, the init scripts won’t be aware of this and may report false results.

BTW, turning accounting off with accton is done just like that:


When no file is specified, the accounting is turned off. When the command is run against a file, as we’ve demonstrated earlier, the accounting process is started. You should be careful when activating/deactivating the accounting and stick to one method of management, either via the accton command or using the init scripts.

4. Dump utmp and wtmp logs

Like pacct, you can also dump the contents of the utmp and wtmp files. Both these files provide login records for the host. This information may be critical, especially if applications rely on the proper output of these files to function.

Being able to analyze the records gives you the power to examine your systems in and out. Furthermore, it may help you diagnose problems with logins, for example, via VNC or ssh, non-console and console login attempts, and more.

You can dump the logs using the dump-utmp utility. There is no dump-wtmp utility; the former works for both.

Dump utmp

You can also do the following:

dump-utmp /var/log/wtmp

Here’s what the sample file looks like:

utmp log

5. Monitor CPU and disk usage with iostat

Would you like to know how your hard disks behave? Or how well does your CPU churn? iostat is a utility that reports statistics for CPU and I/O devices on your system. It can help you identify bottlenecks and mis-tuned kernel parameters, allowing you to boost the performance of your machine.

On some systems, the utility will be installed by default. Ubuntu 9.04, for example, requires that you installsysstat package, which, by the way, contains several more goodies that we will soon review:

Install sysstat

Then, we can start monitoring the performance. I will not go into details what each little bit of displayed information means, but I will focus on one item: the first output reported by the utility is the average statistics since the last reboot.

Here’s a sample run of iostat:

iostat -x 10 10

The utility runs 10 times, every 10 seconds, reporting extended (-x) statistics. Here’s what the sample output to terminal looks like:

iostat example

6. Monitor memory usage with vmstat

vmstat does the similar job, except it works with the virtual memory statistics. For Windows users, please note the term virtual does not refer to the pagefile, i.e. swap. It refers to the logical abstraction of memory in kernel, which is then translated into physical addresses.

vmstat reports information about processes, memory, paging, block IO, traps, and CPU activity. Again, it is very handy for detecting problems with system performance. Here’s a sample run of vmstat:

vmstat -x 10 10

The utility runs 10 times, reporting every 1 second. For example, we can see that out system has taken some swap, but it’s not doing anything much with it, there’s approx. 35MB free memory and there’s very little I/O activity, as there are no blocked processes. The CPU utilization spikes from just a few percents to almost 90% before calming down.

Nothing specially exciting, but in critical situations, this kind of information can be critical.

vmstat example

7. Combine the power of iostat and vmstat with dstat

dstat aims to replace vmstat, iostat and ifstat combined. It also offers exporting data into .csv files that can then be analyzed using spreadsheet software. dstat uses a pleasant color output in the terminal:


Plus you can make really nice graphs. The spike in the graph comes from opening the Firefox browser, for instance.



8. Collect, report or save system activity information with sar

sar is another powerful, versatile system. It is a sort of a jack o’ all trades when it comes to monitoring and logging system activity. sar can be very useful for trying to analyze strange system problems where normal logs like boot.msg, messages or secure under /var/log do not yield too much information. sar writes the daily statistics into log files under /var/log/sa. Like we did before, we can monitor CPU utilization, every 2 seconds, 10 times:

sar -u 2 10

CPU example

Or you may want to monitor disk activity (10 iterations, every 5 seconds):

sar -d 5 10

Disk example

Now for some really cool stuff …

9. Create UDP server-client – version 1

Here’s something radical: create a small UDP server that listens on a port. Then configure a client to send information to the server. All this without root access!

Configure server with netcat

netcat is an incredibly powerful utility that can do just about anything with TCP or UDP connections. It can open connections, listen on ports, scan ports, and much more, all this with both IPv4 and IPv6.

In our example, we will use it to create a small UDP server on one of the non-service ports. This means we won’t need root access to get it going.

netcat -l -u -p 42000

Here’s what we did:

-l tells netcat to listen, -u tells it to use UDP, -p specifies the port (42000).

Netcat idle

We can indeed verify with netstat:

netstat -tulpen | grep 42000

And we have an open port:


Configure client

Now we need to configure the client. The big question is how to tell our process to send data to a remote machine, to a UDP port? The answer is quite simple: open a file descriptor that points to the remote server. Here’s the actual BASH script that we will use to test our connection:

Client script

The most interesting bit is the line that starts with exec.

exec 104<> /dev/udp/$1

We created a file descriptor 104 that points to our server. Now, it is possible that the file descriptor number 104 might already be in use, so you may want to check first with lsof or randomize the choice of the descriptor. Furthermore, if you have a name resolution mechanism in place, you can use a hostname instead of an IP. If you wanted to use a TCP connection, you would use /dev/tcp.

The choice of the port is defined by the $1 variable, passed as a command-line argument. You can hard code it – or make everything configurable by the user at runtime. The rest of the code is unimportant; we do something and then send information to our file descriptor, without really caring what it is. Again, we need no root access to do this.

Test connection

Now, we can see the server-client connection in action. Our server is a Ubuntu 8.10 machine, while our client is a Fedora 11. We ran the script on the client:

Script running

And watch the command-line on the server:

Server working

To make it even more exciting, I’ve created a small Flash demo with Wink. You are welcome to play the file, if you’re interested:

Cool, eh?

10. Configure UDP server-client – version 2

The limitation with the exercise above is that we do not control over some of the finer aspects of our connection. Furthermore, the connection is limited to a single end-point. If one client connects, others will be refused. To make things more exciting, we can improve our server. Instead of using netcat, we will write one of our own – in Perl.

Perl is a powerful programming language, very flexible, very neat. I must admin I have only recently began dabbling in it, so do not expect any miracles, but here’s one way of creating a UDP server in Perl – there are tons of other implementations available, better, smarter, faster, and more elegant.

The code is very simple. First, let’s take a look at the entire file and then examine sections of code. Here it is:


use IO::Socket;

$server = IO::Socket::INET->new(LocalPort => ‘50060’,
Proto => “udp”)
or die “Could not create UDP server on port
$server_port : $@n”;

my $datagram;
my $MAXSIZE = 16384; #buffer size

while (my $data=$server->recv($datagram,$MAXSIZE))
print $datagram;

my $logdate=`date +”%m-%d-%H:%M:%S”`;

my $filename=”file.$logdate”;
print FD $datagram;


The code begins with the standard Perl declaration. If you want extra debugging, you can add the -w flag. If you want to use strict code, then you may also want to add use strict; declaration. I warmly recommend this.

The next important bit is this one:

use IO::Socket;

This one tells Perl to use the IO::Socket object interface. You can also use IO:Socket::INET specifically for domain sockets. For more information, please check the official Perl documentation.

The next bit is the creation of the socket, i.e. server:

$server = IO::Socket::INET->new(LocalPort => ‘50060’,
Proto => “udp”)
or die “Could not create UDP server on port
$server_port : $@n”;

We are trying to open the local UDP port 50060. If this cannot be done, the script will die with a rather descriptive message.

Next, we define a variable that will take incoming data (datagram) and the buffer size. The buffer size might be limited by the network implementation or network restrictions on your router/switch or the kernel itself, so some values might not work for you.

And then, we have the server doing some hard work. It prints the data to the screen. But it also creates a log file with a time stamp and prints the data to the file as well.

The beauty of this implementation is that the server permits multiple incoming connections. Of course, you will have to decide how you want to differentiate the data sent by different clients, whether by a message header or using additional IO:Socket:INET objects like PeerAddr.

On the client side, nothing changes.


That’s it for now. This crazy collection should help you impress your boyfriends and girlfriends, evoke a smile with your peers or even your boss and help you be more detailed and productive when it comes to system administration tasks. Some of the utilities and tricks presented here are tremendously useful.

If you’re wondering what distribution you may need to be running to get these things done, don’t worry. You can get them working on all distros. Throughout this document, I demonstrated using Ubuntu 8.10, Ubuntu 9.04 and Fedora 11. Debian-based or RedHat-based, there’s something for everyone.

In the next article, we will also talk about other crazy hacks and tips, including a very, very useful utility calledsec – Simple Event Correlator. That’s just a brain teaser for now. I hope you enjoyed this article. See you around.

Hello there, dear readers. Time for the second article of highly useful, cool and fun utilities, commands, and tricks that should help you gain better productivity and understand your system better. In the first part, we learned about a whole bunch of great things, including top in batch mode, how to read process account logs, how to measure system activity with a range of programs, and how to write a simple UDP server-client.

Now, let’s see a few more tricks that will help you master a higher, cooler level of Linux knowledge and allow you to impress you significant others, including your boss.

1. Sparse files

What they be, you’re askin’. Well, sparse files are normal files – except that blocks containing only zeros are not really counted. In other words, empty space inside sparse files is just listed, without actually taking any physical space. This, in contrast to regular files, where everything is preallocated, including bits that hold no data.

If you’re a fan of virtualization, you have come across sparse files – virtual machines disks can be sparse files. If you’re creating virtual machines with, say 10GB space, but do not preallocate it, then you have witnessed sparse files in action! Dynamically expanding virtual disks are sparse files.

Sparse files have an advantage of conserving space until needed, but if you convert them back to raw format, like during the conversion of VMDK virtual disks to AM2 format for the use in Amazon EC2 cloud, then the files will be inflated back to their normal size. Now, the big question is, why sparse files, and what are they good for?

Well, sparse files are definitely useful in virtualization, but they have other uses. For example, when creating archives or copying files, you may or may not want to use the sparse option, depending on your requirements. Let’s see how we can create sparse and identify sparse files, so we can treat them accordingly.

Create sparse files

Creating sparse files is very simple. Just move the pointer to the end of the file.

dd if=/dev/zero of=file bs=1 count=0 seek=1M

For example, here we have created a zero-size file, except the metadata, which by default will take the customary block size (say 4096 bytes). But we have also moved the pointer to the end of the file, at 1M location, this creating a virtual 1MB file.

Sparse create

Now, using the ls command, you may think it’s a regular file:

Reported size

But you need the -s flag in the ls command options to really know what’s happening. The first field in the output will be the file size, in KB:


Similarly, you can use the du command to get the accurate report:

du command

Just for comparison, here’s what a real, 1MB file reports:

ls real file

du real file

Pay attention to this when working with files. Do not get confused by crazy ls readings, because you may end up with a total that exceeds the real disk size. Use the appropriate flags to get the real status.

Moreover, pay attention when working with file handling, compressing and archiving tools, like cp, tar, zip, and others. For instance, cp has an option that specifies how the sparse files should be handled.

man page

2. Having fun with atop

It’s not a spelling error, there’s no space missing between the letter a and top. atop is a top utility, with some spice. The full description is AT Computing’s System & Process Monitor, an interactive utility to view the load on a Linux system. It can do everything top does, and then some.

atop is a very useful program and you’ll fall in love instantly. The main view is very similar to the original tool, except you have more info and it’s arranged in a more intuitive fashion. You’ll also have color readings for critical percentage of resource usage.


In the bottom half of the main view, you will be able to sort the process table based on different columns, like memory or disk. Press m to sort by memory in the descending order. Press d to sort by disk activity in the descending order.



You can save data into flat files, any which way you want.


Better yet, you can also write data to logs in compressed, binary form and then parse relevant fields, compiling useful time-dependent statistics about your system load and usage, helping identify bottlenecks and problems. The manual page is very details and provides examples to get you started instantly.

For instance, the following command:

atop -w /tmp/atop.raw 30 10

will collect the raw data every thirty seconds a total of ten times. Very similar to iostat and vmstat, as we’ve seen the last time. Afterwards, you can pull out desired subsets very easily.

For example, to view the processor and disk utilization of this file in parseable format:

atop -PCPU,DSK -r /tmp/atop.raw

Here’s what the data looks like:


Now, if you don’t like the separator, just remove it with some simple sed-ing.

sed -e ‘/^SEP$/d’ /tmp/atop.raw > /tmp/f-clean.csv

Then, you can open this file in, say OpenOffice and create some impressive graphs:



3. ASCII art

ASCII art won’t make you an expert, but it can be fun. Oh, I’m not talking about using high-end tools like GIMP; anyone can do that. I’m talking about deriving fun ASCII art from the command line.

There are several ways you can achieve this, we will see two.


boxes is a neat little utility that lets you create your own command-line fortune cookies, similar to what Linux Mint does. The tool has a number of template ASCII figures available, on top of which you add your own little slogans.

boxes is available in most repositories, so go grab it. Then, start playing. For example, to have a cute little kitten write something witty in your terminal, run boxes -d cat, type your own message and hit Ctrl + D to end. Soon thereafter, a little cat will show in the terminal, along with your own message.

boxes cat

Innocent, sweet and fun.


This ominous sounding command is not one of those robots in Star Wars. It’s a utility that can convert JPEG images, any one you want, into ASCII art. Very useful and impressive.

For example, take your stock Tux. Now, the image I found was in the PNG format and jp2a does not handle these. So I had to convert the image to JPEG first.


And then, just run the command against the image name and Voila! Tux is your uncle!

Tux converted

4. xargs

xargs sounds like a peon curse from Warcraft I-III, but it’s in fact a very powerful and useful command that builds and executes commands from the standard input. In other words, when you use complex chains of commands in Linux, sometimes separated by the pipe symbol (|), you may want to feed the output of the last command into the input of the next one. But things can get complicated.

Luckily, xargs can do everything you need. Let’s see a few simple examples.

Example 1:

We will display all the users listed in the /etc/passwd file. Then, we will sort them and print them to the console, each on a separate line.

The command we need is:

cut -d: -f1 < /etc/passwd | sort | xargs echo |
tr ‘ ‘ ‘\n’

Example 1

xargs takes the list of usernames, one by one, echoes them to the console, while the tr command separates into each line, replacing the space delimiter with a new line feed.

Example 2:

Here’s another example. xargs is particularly useful when run with the find command and quite often sed. Let’s say you want to find a list of certain files in your directory and then manipulate them, including changing their permissions, deleting them or just listing them.

With xargs, you can make this affair a one-liner.

find . -type f -print0 | xargs -0 ls

Example 2

Here we’re using xargs with the -0 flag, which instructs it to ignore whitespaces and treat slashes and backslashes literally, making it quite useful if you expect your files to contain quotes, spaces and other exotic characters. To do this, xargs requires the find command to provide input in the right format, which is exactly what the -print0 flag does.

If you’re not convinced xargs is mighty, try doing a few exercises without it and see if you can manage to get the job done in a single line of shell code.

5. Swapon/swapoff

Another allegory, Karate Kid. Wax on, wax off. Except that we’re dealing with the command that handles swap files on Linux. I do not know how often you will have to handle swap manually, but if you’re using live CDs or work with RAID, then you just might.

swapon/swapoff allows you to turn on/off swap devices, set their priority and just plain list them. Changing the priority could be useful if you have swaps of different sizes or set on disks with different speeds.

For example, to view all swap devices:

swapon -s

A screenshot of a typical output:


And sometimes, you just may want to turn swap off. For example, swap may be used by the live CD, preventing you from unmounting the disk for partitioning, which could lead to errors. In this case, a simple swapoff will do the trick.

Speaking of disks and speeds …

6. Use ramdisk for lightning-fast execution

RAM is not cheap and you should not waste it as simple storage space if you need not to, but sometimes, just sometimes, you may be in a bit of a hurry and would like to get your project completed as soon as possible. If your work entails quite a bit of disk activity, which is usually the bottleneck of the program execution on modern machines, then using a ramdisk could help.

ramdisk is a file system created in the system memory (RAM) and treated as a regular disk device, hence its name. For all practical purposes, if you give someone a system with a RAM disk, they won’t know the difference, except the speed. ramdisks are much faster.

Here’s a little demo.

First, let’s create a ramdisk (as root or sudo):

sudo mount -t tmpfs none /tmp/ramdisk -o size=50M


We created a 50M disk and mounted it under /tmp/ramdisk. And now, let’s compare some basic writes …

Normal disk:


RAM disk:


Of course, the results will depend on many factors, including system load, disk type and speed, memory type and speed, and whatnot, but even my 23-second demonstration shows that using ramdisk you can boost your performance by 50% of more. And if you attempt repetitive serial tasks like copy, you will be able to improve your execution time by perhaps an order of magnitude.

7. Perl timeout (alarm) function

Again, Perl as the last item. Now, I have to reiterate, I’m not a skilled Perl writer. I am a cunning linguist and a master debater, but my Perl skills are moderate, so don’t take my perling advice as a holy grail. But you should definitely be familiar with the timeout function, or rather – alarm.

Why alarm?

Well, it allows you to gracefully terminate a process with SIGALARM after a given timeout period, without having your program stuck forever, waiting for something to happen.

Now, let’s see an example. If you’ve read my strace article, then this little demo should remind you of some of the things we’ve seen there.


use strict;
my $debug=1;

eval {
local $SIG{ALRM} = sub { die “alarm\n” }; # NB: \n required
alarm 5; # timeout after 5 seconds without response
system(“/bin/ping -c 1 @ARGV[0] > /dev/null”);
alarm 0;

if ($@) {
die unless $@ eq “alarm\n”;   # propagate unexpected errors
print “\nWe could not ping the desired address!\n\n” if $debug;
# timed out

else {
print “\nWe’re good!\n\n” if $debug;

What do we have here? Well, a rather simple program. Let’s examine the different bits separately. The first few lines are quite basic. We have the perl declaration, the use of strict coding, which is always recommended, and a debug flag, which will print all kinds of debugging messages when set to true. Rather useful when testing your own stuff.

Next, the eval function, which tells the program to die with ALRM signal if the desired functionality is not achieved within the given time window (in seconds). Our example is a simple ping command, which takes the IP address as the input argument and tries to get a reply within five seconds.

eval {
local $SIG{ALRM} = sub { die “alarm\n” }; # NB: \n required
alarm 5; # timeout after 5 seconds without response
system(“/bin/ping -c 1 @ARGV[0] > /dev/null”);
alarm 0;

Next, we set the program to exit if there are error messages ($@), printing a message to the user that informs him/er that we could not ping the desired address. What more, if the program execution got botched for some reason other than our timed alarm, we will terminate the execution, thus covering all angles. If successful, we continue with our work, plus some encouraging messages.

if ($@) {
die unless $@ eq “alarm\n”;   # propagate unexpected errors
print “\nWe could not ping the desired address!\n\n” if $debug;
# timed out

else {
print “\nWe’re good!\n\n” if $debug;

Some screenshots … Here’s the perl code. P.S. Just noticed the 10 seconds in the comment after alert 5; Well, it’s an innocent error, but it does not affect the code, so you can ignore it.


Then, we have a good example:


And a bad one:


And just to show you it’s a five-second timeout we’re talking about, I’ve used the time command to … well, time the execution of the script run:


ping is just a silly example, but you can use other, more complex functions. For example, mount commands. In combination with strace, which we’ve seen a few weeks ago, you can have a powerful trapping mechanism for efficient system debugging.

To read more about alarm, try the official documentation: perldoc -f alarm. To this end, you will need the perl documentation package installed on your system.

Why this exercise?

Well, it emphasizes the importance of proper checks when coding programs that use external inputs and outputs to work. Since you cannot guarantee that the other bits of code will cooperate with yours, you need to place failsafe checks to make sure you can gracefully complete the run without getting stuck. Along with input validation, timeouts and error exits are an integral part of cavalier programming.


That’s it, seven lovelies this time. A magnificent seven. I did promise you sec, but it’s too large to be just a bullet item. We will have a separate article soon, probably as a super-duper admin tool.

Anyhow, today you’ve learned several more useful tools, tricks and commands that should help you understand better your environment, work more smartly and be able to control and monitor your systems more effectively. Best of all, the tips given do not really require any specific platform. Any Linux will do. I used openSUSE 11.2, Ubuntu Jaunty and Ubuntu Karmic for these demos.

I hope you appreciate the combined effort. Stay tuned for more. We’ll have several more compilations as well as dedicated, detailed articles on some of the more powerful programs available, including both mid-end and high-end tools, as well as advanced system debugging utilities.

Welcome to the third installment in the Linux cool hacks series. Like the previous two, this article is all about cool things you can do with your Linux that are not well known and yet rather useful. When I say cool, this applies to laughing hard at XKCD’s sudo make me a sandwich style of people rather than someone wearing Zara flipflops, although those are not mutually exclusive.

Anyhow, we’ve had some 17 tips so far. Let’s try a few more. I will demonstrate using Ubuntu, openSUSE andCentOS, to show you that the choice of the system does not really make much difference. So please join me. Tomorrow, after having read and practiced these tricks, you will be able to impress your significant others and colleagues and there ought to be much rejoicing.

1. Show (kernel) functions in ps output

This is an interesting need. Say you have a program that is misbehaving. You do not want or cannot attach the debugger to it, as you fear you may disrupt some delicate time-race condition or possibly even crash the application. Or it may be stuck in a non-debuggable state. Or it may not have symbols or deny ptrace hooks or who knows what else. All in all, lots of geek lingo, the bottom line is, you just want to know at what stage the execution of the software is stuck, in the quickest, least intrusive way possible. ps will do.

This one specific example is even written in the man page:

ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm

And you will be able to see in the WCHAN column, the last function being used by your process. Most of the time, this will be completely meaningless, but if you have an inkling of understanding how your process ought to behave or you might be a developer, this could be useful information.

ps, wchan

2. Nohup

Nohup is a special Linux command that lets you detach processes from their shell, allowing them to run in what you might want to refer to as the background service mode. Indeed, if you take a look at the process table (ps), you will see a lot of processes that were spawned by the system and run without a tty.

When you start a program from the command line, it will live within the shell of your terminal window, even if you background it with &. When you kill the shell, all of its children processes will die too. In a few select cases, we want to avoid this, so we need a mechanism that will detach processes from their shell. A simple method is to create a startup script and add it to /etc/init.d, but this should really be reserved to services.

So nohup will daemonize our processes – make them daemons. Sounds scary, but it’s just geek lingo designed to impress girls. Anyhow, nohup is invoked against the desired binary or script. You need a full path if the binary or script are not presented in the PATH. You must also background nohup itself, so that it detached from the shell.

nohup <command> &

Nohup will redirect the output to nohup.out in the current directory. You should also make sure to use the proper redirection for the standard input, output and error to avoid hangs.

Here’s an example. Notice that runs without a terminal, as denoted by ? in the sixth column. For instance, the grep command runs on the virtual terminal pts/3. Moreover, is parented by init (PID = 1). And you can also see the nohup output, which is just a silly echo in this example.


3. Fallocate

Fallocate sounds a meme, but it is a very neat command that can save you a lot of time. To prove that, let me ask you a question first. What do you do if you need to create a very large file, which cannot be sparse? You use dd and source the bit stream from /dev/zero, but this takes a long time. It’s normally limited by the device speed, which is about 80MB/s for most disks. So if you need to create an 80GB file, you will need some twenty minutes to do that, in the best case. With USB connections and slower disks, this can grow to 40 minutes or longer. fallocate solves the problem by preallocating blocks instantly.

This is a relatively new command and system call in the Linux kernel, available since revision 2.6.23. All right, let us demonstrate.

First, we create a 10MB file. Nothing special. But then, to show you how powerful this command really is, we will compare with dd. While files this small could easily be written to disk cache, masking the true speed, the demonstration is powerful enough without having to use large files.

fallocate -l 10m 10mbfile


Now, the comparison. Notice the actual time differences between fallocate and dd. Even for such a tiny file, the difference is huge. fallocate is some 70 times faster in terms of system time, even though the entire operation took a fraction of the second.

fallocate speed

dd speed

Now, fallocate will remain as fast, without any regard to file size, while dd times will increase. When you have to create files that are several GB is size or much larger, you will appreciate this capability. For example, you may need to create swap files in this manner and preallocate them to partitions during the installation setup. You might not be able to wait long minutes or possibly hours for this operation to complete. Fallocate resolves the problem.

4. Debug filesystems (debugfs)

Debugfs is an interactive tool for managing EXT filesystems. Invoked from the command line, it allows you to change the mode, block size, write to the superblock, force the filesystem to execute specific commands, and more. Naturally, this kind of work means you know what you’re doing and you’re well aware of the potential hazards of data corruption when working against devices and their filesystems in a sort of live operation mode.

debugfs is invoked against the desired target device. By default, it will open the filesystem in read-only mode, as a precaution. This is quite useful for trying to salvage data from corrupted filesystems. Other commands that come into mind when trying to work with filesystems include tune2fs and resize2fs.


5. Blacklisting drivers

The Linux kernel comes with a ton of drivers, some compiled into the kernel, during the kernel compilation, which is done by specifying Y, some available as dynamically loadable modules, which is done by specifying M. The modules will later show under /lib/modules, matching your kernel.

Now, the kernel footprint could be big and contain too many drivers that you do not need or even contain conflicting drivers that interfere with your work. For instance, you might not want ipv6, which is something we tried in my Realtek network troubleshooting on Kubuntu Natty on my latest desktop, or perhaps you might not want the Nouveau graphics driver, as it conflicts with the Nvidia driver and prevents its installation, as we have seen in my CentOS Nvidia guide.

There are several ways you can disable drivers – by blacklisting them. Not a new thing, we’ve done the same back in 2006 with my Linux guide of highly useful configurations. You can make permanent changes by editing files on your system or pass parameters to the kernel commandline in the GRUB menu.

Using the CentOS example, you can disable the Nouveau driver by appending the following string to the kernel command line:

kernel /boot/vmlinuz <all kinds of options> rdblacklist=nouveau

Oncer your system boots and you are 100% confident the change works well, then you can make the change permanent, either by editing the GRUB menu or by editing the driver to the /etc/modprobe.d/blacklist or /etc/modprobe.d/blacklist.conf file, depending on your distribution.

echo “driver name” >> /etc/modprobe.d/blacklist

Please make sure you have backups before you permanently alert your system. Finally, some drivers will have writable parameters exposed under /proc and /sys, allowing you to echo new values on the fly and make changes as necessary. We will discuss that a while later.

6. Browsing the kernel stuff

This is a vague title, but what I’m referring to is the capability to quickly inspect kernel functions, check header files, determine whether your applications are trying to run code that belongs to the kernel or something else and so forth. To this end, there are many tools you can use. We’ll examine a few.

First, you can go online lxr – The Linux Cross Reference site, which indexes all source code in the kernel repositories. So if you’re looking some function, just input the name or part thereof into the search box and start reading.

LXR site

Then, there’s cscope, which we saw in the Kernel Crash Book. If you have kernel sources installed on your machine, you will be able to check what functions, text strings, symbols and definitions are declared in different source files. This is quite useful if you are trying to debug problems with your applications or perhaps even kernel crashes. To that end, you might also be interested in ctags.


7. Some extras

The tips listed below will probably not serve you that often, but it is good to know about them. Almost like hoarding water for the nuclear winter, so to speak, only more fun. Now, please note that you cannot follow the advice below at all!

It’s a sort of a paradox, but unlike so many people out there, I will not give you blanket suggestions on how to utilize your machines, as every single use case is different. Saying that X will speed Y is utterly and morally wrong. One man’s tweak blessing is another’s curse. Do not even change configuration because someone somewhere said it ought to work, make your system work faster, be more responsive, etc. 99% of these wild and happy recommendations are valid for single home machines with no regard to reality, especially not businesses with heavily loaded production servers. Therefore, be aware of the possibilities, study them carefully and then apply your best formula.

/proc and /sys tunables

Explaining what /proc and /sys do is beyond the scope of this article by three whole quantum leaps. But they are very important pseudo-filesystems that let you tweak all kinds of things, on the fly, no reboot required.

In this section, I will try to elaborate on several useful features, like CPU affinity, memory tunables, scheduling, and a few other items that will normally earn you a good beating your neighbors if you ever speak of them in public. Let’s do it.

For example, if you have a multi-processor system that does very specific tasks, you might want to bypass the internal scheduling mechanisms and force your cores to process only certain workloads. Normally, this tradeoff usually has more problems than benefits, so please don’t make any changes just for the sake of being cool.

To give you a practical example, you might want to assign interrupt handling for most heavily used network channels to CPU1, while allowing the rest of the tasks to work on CPU2. Indeed, if you have a box that has several network devices and churns data like mad, loading one specific processor might be a good idea in ensuring the quality of service for other tasks. Then again, you could ruin everything, so be careful.

To get this going, you need the processor bitmask, which you can derive from the number of available processors on your box, as well as the corresponding interrupt for the channel you wish to assign to a specific processor.

cat /proc/cpuinfo

cat /proc/interrupts


And then, we do the magic – force IRQ 30 (Wireless, iwlagn) to processor 1:

echo 1 > /proc/irq/30/smp_affinity

Of course, your kernel must be capable of symmetric multi-processing, which is a default in all new kernels. It’s not a given for older kernels like 2.6.16 and 2.6.18 in previous but still much used enterprise editions of SUSE and RedHat.

More reading here:

Memory management

Linux memory management is the blackest of magics in the world. But it’s a fun thing, especially if you know what you’re doing. Like I mentioned before, no one setting will work for everyone. There’s no golden rule. The system defaults are as good as empirically possible for the widest range of uses, so you should stick with that.

If however, you feel really adventurous, you might want to explore the kernel tunables under /proc/sys/vm. There are several of those.

The swappiness parameter tells you how aggressively your system will try to swap pages. The values range from 0 to 100. In most cases, your disk will always be the bottleneck, so it will make little difference. Then, there’s the dirty_ratio tunable, which tells the percentage of total system memory that can be taken by dirty pages. Once this limit is hit, the system will start flushing data to the disk. Another parameter that is closely related to the dirty_ratio is dirty_expire_centisecs, which determines the max. age of dirty pages before they are flushed. The system will commit the dirty data based on the first of the two parameters to be met, which will most likely be the expire time.

Centisecs value

A mental exercise: the default dirty_ratio on Linux is 40%, while the default expire tunable is set to 3000 centiseconds. A centisecond is 1/100 of a second or 10ms, so we have 30 seconds total. If you have a machine with 4GB RAM, then 1.6GB will be dedicated to dirty pages at most. Now, this means that whatever you’re writing, it needs to create some 55MB of data every second to exceed this threshold in the thirty-second period for the kernel flushing thread to wake and start writing to the disk. In most cases, you will rarely have such aggressive writes. Some notable examples include large copies, video rendering and alike. In daily use, hardly ever. If you have more than 4GB RAM, say 8-16GB, then this becomes even less likely.

This exercise also tells you whether you really need that high dirty_ratio, how to set the other tunables and more. Having too many dirty pages also means very long and sustained writes when the time comes to commit them to disk. Food for thought, fellas. There’s no golden rule.

As you can see, I’m breezing through these extremely lengthy and complex topics, but the idea is not to write a PhD on memory management, but give you a very brief sampling of the possibilities, so you can later explore and use them.

You can make changes by echoing values to /proc or using sysctl.

A very geeky read (direct link) for RHEL4, but still very much relevant today.

Another thing you may want to attempt is to allow/disallow memory overcommitment. Normally, Linux uses smart heuristics for managing overcommittment, but if you are really worried about how your system handles out its quiche to processes, then you can disable the overcommittment or set a ratio. I would recommend against any changes, unless you have very strict requirements, you cannot afford OOM mechanism to work, etc.

I/O scheduling

Another geeky item, best left alone. But if you must, please read on. First of all, most I/O elevator algorithms assume platter-based disks, so if you’re running with SSD, the rules of the game changes, but this has been taken into account in recent kernels. Assuming you’re running on plain old mechanical hardware, then you have one simple goal: as few seeks as possible to minimize access times and wear, which translate into user responsiveness latency. But then, some of your machines might be running pure computation tasks, so the responsiveness might not be an issue.

But in general, we want to perform write operations in bursts, as much data as possible. There are four available schedulers: noop – most basic, dispatches requests as they come, normally good for disks on key and systems with heavy CPU usage; anticipatory – longer delays, so there’s more chance for starvation, however it tries to maximize throughput and reduce seeks; cfq – better known as completely fair queue scheduler, it relies on processes behavior and can be used with ionice to achieve balanced throughputs. It does not prefer writes or reads; deadline – this one tries to dispatch as quickly as possible, treating tasks as real-time, in order to avoid process starvation.

You can issue the change per disk:

echo <scheduler> > /sys/block/<device>/queue/scheduler

For instance:

echo cfq > /sys/block/sdb/queue/scheduler

All this sounds dandy, but the real challenge is figuring out what your machines are doing and match the behavior accordingly. After you have made the change, you will need to test your results. In the Linux world, you will most commonly find cfq or anticipatory as the default choice.

Of course, if you make changes to the scheduler, then you might also want to tweak the readahead settings, both the readahead max. value and the throughput value, as well as the number of simultaneous I/O requests. The corresponding tunables include nr_requests, read_ahead_kb and inode_readahead_blks. Some of the values will be limited by the filesystem choice. Let me disappoint you and tell you that you will have to work hard to see significant improvements.

Some reading on schedulers: Linux Journal – I/O Schedulers.

Filesystem mount options

Like the disk, we want speed. That’s the basic driver here. So let’s see what kind of options we can use. The most notable focus is on the journaling capabilities of modern filesystems.

This is another black magic, but something you can test with relative safety. Choose any old disk, preferably with a single partition to avoid masking results by typical disk speed bottlenecks. Then, test various mount options. Some of the notable performance boosters so to speak include:

writeback mode – only the metadata is journaled, and the data blocks are written directly to their location on the disk. This preserves the filesystem structure and avoids corruption, but data corruption can occur. For example, if the system crashes after the metadata is journaled but before the data block is written.

ordered mode – metadata journaling is done after the data is written to the disk. In this way, data and filesystem are guaranteed consistent after a recovery.

data mode – both metadata and data are journaled. This mode offers the greatest protection against file system corruption and data loss but can suffer from performance degradation, as all data is written twice (first to the journal, then to the disk).

Some more reading: Anatomy of Linux journaling filesystems.

All right, now that we know what we need, we can simply mount a filesystem with the  writeback option. You should test extensively, to make sure things work out find, or at the very least, use this option for filesystems with heavy access but that might not be containing critical data.

mount -o data=writeback /dev/<device>  /<mountpoint>

You might also want to consider noatime and nodiratime, but again don’t listen to one geek trying to impress you with words, do your own testing and prove everyone else wrong.

And I guess that would be enough for today. Other items that you might want to look at include slabinfo/slabtop, huge pages and Table Lookaside Buffers (TLB). That’s different from LTB, which stands for Tomato Lettuce and Bacon, a different kind of hack. Some screenshots and we’re done here.

slabinfo, slabtop

Huge pages config


There you go, another lovely set of geekiness. Again, the real value in these hacks is the exposure not the actual application. Be aware of the functionality, study it, and then apply it to your personal or business needs one day. And remember that no two computers and use cases are the same, so blind copy & paste will not work.

That would be all, I guess. You are also welcome to check the first and the second article, as well as the whole series of so-called super-duper admin tools. We will also have an extensive review on the Gnu Debugger (gdb) soon. Stay pretty.

Once article numbers start to run high, people tend to start paying less attention to the content. However, by no means does that make this article any less useful or interesting. I happen to have a fresh new bunch of tips and tricks that ought to increase your Linux street credit.

In the first two parts, we focused on system administration mostly. The third part focused on system internals. This fourth chapter will elaborate on compilation and fiddling with Linux binaries, specifically the ELF format. Again, not everyone’s lunch or dinner, but some of you may appreciate the extra geekiness I devoted to making your lives easier. So please follow me.


1. Learn more about the file – no strings attached

Say you have a binary of some sort – a utility, a shared object, a kernel module, maybe even an entire kernel. Now, using file will give some very basic information on what kind of object you’re dealing with. But there’s more. Strings. Now, the subtitle makes a lot of punny sense, hihihihihihi.

Strings is a very useful command that can pull out all printable characters out of binary files. This can be quite useful if you need to know the would-be meta data, like compiler versions, compilation options, author, etc. For example, here’s what it looks like for a kernel vmlinuz file. Some of you may actually recognize some of the print messages there.


2. Debugging symbols

Now, say you wish to debug your faulty application, but for some reason all of the functions in the backtrace come out with ?? marks. The simple reason is that you may not have debug symbols installed. But how would you know?

Well, apart from checking the installed database of RPM of DEB files, you may want to query the files directly. Again, we will use the file command, and then delve deeper into the system. Here’s an example:

Stripped object

What we see here is that we have a 32-bit Little Endian shared object for the Intel architecture, stripped of symbols. That’s what the last word tell us. This means the binary was compiled without symbols or they have been removed afterward to conserve space and improve performance. We discussed symbols in the Kernel Crash book, too.

So how do you go about having or not having debug symbols? Another highly useful tool that should let you get binary symbols is nm. This tool is specifically designed to get symbols from various sections in the executable file format that is typical on Linux.

For instance, -b flag lets you get symbols for uninitialized global variables in the data section, also known as bss. -C lets you query common symbols, or rather uninitialized data. In our example, there are none available, because our shared library is stripped.

nm example

However, if you query with -D flag, you will get symbols in the initialized data section.

Global table

For most people, this information is completely useless. But for senior system admins and software developers, knowing exactly the mapping of code in a binary and translation of memory addresses to function names is essential.

Playing with symbols – objdump, objcopy, readelf

We can add and remove them, as we please, after the compilation. To that end, we will use several handy utilities, including objcopy and readelf. The first allows manipulating object files. The second lets you read data from binary files in a structured human readable format.

We will begin with readelf. The simplest way is to dump everything. This can be done using -a flag, but beware the torrents of information, which probably won’t mean much to anyone but developers and hackers. Still good to know and impress girls.

readelf, all

Another useful flag is –debug-dump=info. You might be interested in debuginfo only. Here, specifically, we compile our test tool with debug symbols, and then display the info. Please note that we have a lot of information here:

Debuginfo not stripped

Now, objcopy can manipulate files so that above information is shown, not shown or used elsewhere. For instance, you might want to compile a binary with debug symbols for testing purposes, but distribute a stripped version to your customers. Let’s see a few practical use cases.

To remove debug info from the original binary:

objcopy –strip-debug foo

This will result in a stripped binary, just like we saw earlier. But then, you might not want to toss away those symbols permanently. To that end, you can extract debug info and keep it into separate file:

objcopy –only-keep-debug foo foo.dbg

And then, you can link debug info back to the stripped binary when you need it:

objcopy –add-gnu-debuglink=foo.dbg foo

On the far end of the spectrum, we get objdump, another handy utility. Again, we used the program before, when playing with kernel crashes, so we are no strangers to its power and functionality. Similar to readelf, objdump let us obtain information from object files. For example, you may be interested in the comment section of your binary:

objdump, comments

Or you may want everything:


Combined example

Now, let’s see this in practice. First, we compile our code with -g flag. The binary weighs some 18299 bytes. Then, we strip debug information using objcopy. The resulting binary is now much smaller, at 13042 bytes. And readelf shows nothing, unlike before.

Remove symbols

3. Compilation optimization tips

When compiling your code, there are a billion flags you can use to make you code more efficient, leaner, more compact, easier to debug, or something else entirely. What I want to focus on here is the optimization during the compilation. GCC, which can be considered a de-facto compiler on pretty much any Linux, has the ability to optimize your code. Quoting from the original website:

Without any optimization option, the compiler’s goal is to reduce the cost of compilation and to make debugging produce the expected results. Statements are independent; if you stop the program with a breakpoint, you can then assign a new value to any variable or change the program counter to any other statement in the function and get exactly the results you would expect from the source code. Turning on optimization flags makes the compiler attempt to improve the performance and/or code size at the expense of compilation time and possibly the ability to debug the program.

In other words, the compiler can perform optimizations based on the knowledge it has of the program. This is done by intermixing your C language with Assembly in numerous ways. For example, simple arithmetic procedures of constant values can be skipped altogether and the final results returned, saving time.

Optimizations can affect binary file size and its speed or both. At the same time, it will be much harder to debug, because some of the instructions may be omitted. Moreover, the compilation time will probably be longer. Overall, -O2 levels offers a good compromise between user’s ability to debug, size and performance. It is also possible to recompile code with -O0 level for debugging purposes only and ship to customers with the lean image.

GCC optimization

Here’s another interesting article on optimizations.

4. LDD (List Dynamic Dependencies)

When you try to run your applications, they may sometimes refuse to start, complaining about missing libraries. This can happen for several reasons, including permissions, badly configured path or an actual missing library. To be to know exactly what’s going on, there’s a neat little utility called LDD. It allows you to print shared library dependencies for your binaries. You should use it.



As I’ve mentioned just moments earlier, the system path can impact the successful startup of applications. For example, you may have several libraries under /opt, but /opt is not defined in the search path, which may only include /lib and /lib64, for instance. When you try to fire up your program, it will fail, not having found the libraries, even though they are physically there. You can work around this issue without copying files around by initializing environment variables that will tell the system where to look.

The word system sounds almighty here, so perhaps a short introduction in how things work might be in order. In Linux, there’s the super-tool called dynamic linker/loader, which does the task of finding and loading libraries for programs to run. is a smart and efficient tool, so it does not perform a full-system search every time it needs to fire up a binary. Instead, it has its own mini-database, stored under /etc/, which contains a compiled list of search libraries and an ordered list of candidate libraries. It’s somewhat similar to the locate program.

This list is updated by running ldconfig, which most Linux systems execute either during startup or shutdown, but it can be manually run whenever the /etc/ file, which contains the list of search libraries, is updated. This also happens after installations of software.

If the linker cannot find libraries, the loading of the program will fail. And you can use LDD to see exactly what gives. Then, you can use the environment variables LD_PRELOAD and LD_LIBRARY_PATH to force loading of libraries outside the search path.

There is some difference between the two. LD_PRELOAD will force loading of these libraries before any other. LD_LIBRARY_PATH is similar to standard PATH. There are many other variables you can change, but that’s what the man page is for.

One last hack that you might be interested in is rpath. It allows hard-coding runtime search paths directly into the executable, which might be necessary if you’re using several versions of the same shared library, for instance.

Recursive implementation

LDD displays only unique values. But you might be interested in a recursive implementation. To that end, you might want to check the Recursive LDD tool, available for download at It’s a simple Perl script, with some nice tweaks and options. Quite useful for debugging software problems.

Recursive LDD

5. Some more gdb tips

We learned a lot about gdb. Now, let’s learn some more. Specifically, I want to talk to you about the Text User Interface (TUI) functionality. What you want to do is fire up the venerable debugger with -tui option. Then, you will have a sort of a split-screen view of both your code and the gdb prompt, allowing you to debug with higher visual clarity. All the usual tricks still apply.

GDB -tui

You might also be interested in this article.

6. Other tips

The one last extra tip is about translating addresses into file names and line numbers. addr2line translates addresses into file names and line numbers. Given an address in an executable or an offset in a section of a relocatable object, it uses the debugging information to figure out which file name and line number are associated with it.

addr2line <addr> -e <executable>

A geeky example; say you have a misbehaving program. And then you run it under a debugger and get a backtrace. Now, let’s assume we have a problematic frame:

# C  []  gzdirect+0x28

All right, so we translate (-e tells us the name of the object). Works both ways. You can translate from offsets to functions and line numbers and vice versa. Again, this can be quite handy for debugging, but you must be familiar with the application and its source.

addr2line 0xa910 -e

addr2line -f -e 0xa910
gzdirect  ? function name

More reading

You might also want to check these:

Linux super-duper admin tools: strace and lsof

Linux system debugging super tutorial

Highly useful Linux commands & configurations


I assume this article is only for the brave, bold and beautiful. It’s definitely not something the absolute majority of you will ever want, need, see, try, require, or anything of that sort. But then, if you’re after impressing girls, there’s no better way of doing it.

Along that noble cause, this tutorial also presents some handy tips for software development and debugging, which, combined with a deep understanding of system internals and wise use of tools like strace, lsof, gdb, and others, can provide an awesome wealth of useful information. We learned how to read and extract information from files, how to work with symbols, how to read the binary format, compilation tips, dynamic dependencies, and several other tweaks and hacks. That should keep you busy for a week or there until you figure out everything. Meanwhile, do send me any ideas you may have on similar topics, if you feel there ought to be a tutorial out there. And see you around.

Highly Useful Linux Commands & Configurations

Oh, you’re gonna love this article! Even though there are many websites hawking similar content, with varying degree of clarity and quality, I want to offer a short, easy-to-use guide to some of the most common yet highly useful commands that could help make your Linux experience more joyful.

Now that you have read some of my installation guides, you have probably setup your system and configured the basic settings. However, I’m positive that some of you must have encountered certain difficulties – a missing package, a missing driver. The initial effort required of a Linux novice can appear daunting, especially after many years of Windows discipline.

Therefore, this article was born, in order to offer simple solutions to some of the more widespread problems that one might face during and immediately after a Linux installation. It is intended for the beginner and intermediate users, who still feel slightly uncomfortable with meddling in command line, scripts or configuration files.

This article will refer to Ubuntu Linux distribution as the demonstration platform. However, all of these commands will work well with many other Linux distributions, with only small changes in syntax, at most. I have personally tested and used all of the commands and configurations in both Debian-based and RedHat-based distributions with success.

What am I going to write about?

Here are the topics. If you want to skip through some of the paragraphs, you can use the table of contents further below, but I recommend you read everything.

  • Basic tips – avoiding classic mistakes.
  • Commands – an introduction to the command line.
  • Installation of software – including extraction of archives and compilation of sources.
  • Installation of drivers – including compilation, loading, configuration, and addition of drivers to the bootup chain, writing of scripts and addition to the bootup chain.
  • Mounting of drives – including NTFS and FAT32 filesystems and read/write permissions.
  • Installation of graphic card drivers – including troubleshooting of stubborn common problems.
  • Network sharing – how to access shared folders in Windows and Linux from one another.
  • Printer sharing – how to share printers in Windows and Linux from one another.
  • Some other useful commands.

Table of contents

  1. Basic tips
  2. Commands
    1. Asking for help
  3. Installation of software
    1. What should you choose?
    2. Discipline
    3. Unpacking an archive
    4. Zipped archives
    5. Installation
    6. Compilation (from sources)
    7. Summary of installation procedures
  4. Installation of drivers
    1. Installation
    2. Loading drivers
    3. Configuration of drivers
    4. Scripts
  5. Mounting a drive
    1. Other options
  6. Installation of graphic card drivers
  7. Network sharing
    1. Windows > Linux
    2. Linux > Windows
  8. Printer sharing
  9. Other useful commands
    1. Switching between runlevels
    2. Backing up the X Windows configuration file (useful before graphic drivers update)
    3. Display system environment information
    4. Listing information about files and folders
    5. Kill a process

Basic tips

There are some things you need to know before heading into the deep waters of the Command Line:

  • Linux commands are cAse-sensitive (dedoimedo and Dedoimedo are two different files).
  • It is best to create folders and files in Linux WITHOUT spaces. For example: Red Gemini.doc is a valid Windows filename, but you might have problems accessing it from the command line in Linux; you should rename the file to RedGemini.doc. Users of the DOS command line are also familiar with this problem – commands will fail on folders and files with more than a single word, unless explicitly declared with double quotation marks (“like this”).
  • Pressing TAB when typing a command will auto-complete the command. For example: if you have a single file in a certain folder that begins with the letter p, typing p then TAB will automatically complete the name regardless of its length; if you have more than one file, the command will complete the maximum available part of the string that matches all relevant filenames (s + TAB for smirk and smile will auto-complete to smi).
  • Before copying, moving, deleting, or tweaking any file, especially scripts and configuration files, it is best to back them up first.
  • Do NOT stop the commands while they are running (by pressing Ctrl + C). Even though you may not see the HDD light blinking and the execution takes a very long time, do not assume the system is frozen. Unlike Windows, Linux almost never gets stuck. Let the command complete, be it 5 seconds or 5 hours. Just for reference, compilation of certain programs can take a few days to complete.


To be able to use the command line, you need to be familiar with some rudimentary Linux commands. Former users of DOS will find the transition very simple. Below you can find links to some of the basic Linux commands:

Alphabetic Directory of Linux Commands

An A-Z Index of the Linux BASH command line

Some Useful Linux Commands

Asking for help

First, anything and everything you could ever probably think of has already been answered at least once in a Linux forum; use the forums to find solutions to … everything. Copy & paste your error code / message into a search engine of your choosing (e.g. Google) and you will find links to answers in 99.9996532% of cases.

Locally, help is one of the most useful features available to the command line user. If, for some reason, you cannot figure out the syntax required to use the file, you can ask for help. There are two ways of doing it:

man some_command

The above usage will display a full help file for the command in question in Vi text editor. You can learn more about Vi from An Extremely Quick and Simple Introduction to the Vi Text Editor.

some_command –help

The above usage will display a summary of available options for the command in question, inside the command line terminal. You will most likely prefer to use this second way.

Installation of software

Although most Linux distributions offer a wealth of useful programs, you will probably be compelled to try new products. Some programs will be available for download via package managers, like Synaptic. Others will only be found on the developer’s site, most likely packaged inside an archive.

You probably ask yourself: What now? The answer is very simple. There are three versions to your downloads, from the easiest to hardest:

  1. Compiled packages, usually with .rpm or .deb extension. These packages are identical to Windows .exe installers and will unpack and install automatically. The upside of the packages is the relative use of their deployment; the downside is that the user has no control over the installation script.
  2. Compiled archives, called tarballs, with .tar extension. These archives will contain all of the necessary files required to make a program run, but the user will have to install them manually, from the command line, after unpacking the archive. These archives will also most likely be compressed and bear a double extension like tar.gz or tar.bz2. This option offers more control during the installation.
  3. Sources, usually archived. The user will have to unpack the archives and then compile the sources before being able to actually install the program. In addition to better control of the installation, the user will also benefit from software optimized to his hardware configuration.

What should you choose?

The logical choice for the novice user should be 1 > 2 > 3. Intermediate users will probably try 2 > 3. Geeks will most likely ever only compile from sources.


This may sound harsh or strict, but certain unspoken rules are followed, which simplifies the use of software downloads.

  • The program itself will almost always be accompanied with a how-to, usually in a form of a text file that explains what a user should do, prior, during and after the installation. The how-tos are most often found on the site you download the software from, either as a standalone file, an explanatory text on the download page or bundled with the download.
  • You should read this how-to FIRST before downloading / manipulating the software.
  • A secondary how-to will most often be packed with the program, explaining the installation process itself.
  • You should read this how-to FIRST before installing the software.

Unpacking an archive

The exact syntax will differ from one package to another. But the general idea is the same for all. The only difference will be in the arguments used for unpacking. Here are a few common examples:

tar zxf some_software.tar.gz
tar -xjf some_software.tar.bz2

You can read in detail about the handling of tarballs on the Wikipedia site.

Zipped archives

Some archives will be zipped rather than tarred. This might put you off. But the problem is easily solvable. If you recall, we have the ability to “ask” for help for each unknown command. In our case, we need to know how to unzip an archive.

unzip –help

Here’s a screenshot I took, depicting the very dilemma we are facing – and its solution:

Linux commands - unzip

A possible usage will then be:

unzip -d /tmp

Reading from the help screen above, we want to unpack our archive into a folder. The argument -d tells us that the contents of the archive will be extracted into a destination directory (folder), in our case a temporary folder called /tmp.


After unpacking the archive, you will now have to install the software. Usually, the installation is invoked by using a script. The exact name of the script will vary from one program to another, as well as its extension, depending on the language used to write it.

For example, the following command will invoke the script named (written in Perl). Dot and trailing slash indicate that the script will be executed within the current directory.


Compilation (from sources)

Sometimes, the programs will not be compiled and ready to install. The archives will contain lots of files with curious extensions like .c, .h and .o. If you are not a programmer, you should not bother understanding what they are and what they do. Likewise, you need not understand how the compilation of sources is made. You just need to remember three simple commands:

This first command will generates files required to build the software and setup system-wide parameters.


This second command will build the libraries and applications.


This third command will install the libraries and applications.

make install

For homework, you could use some reading:

Compiling and installing software from source in Linux

There is no guarantee that the compilation will succeed. Some sources are broken! In that case, you should make note of the errors and post them in relevant forums, where you are most likely to find an answer rather quickly.

Summary of installation procedures

To make things easier to understand, below are two examples showing the list of necessary commands required to run to successfully install a downloaded application (please note these are ONLY examples!). Most likely, you will need root privileges (su or sudo) to be able to install software. An archive containing compiled program:

tar zxf some_software.tar.gz
tar -xjf some_software.tar.bz2

cd some_software_directory

An archive containing sources:

tar zxf some_software.tar.gz
tar -xjf some_software.tar.bz2

cd some_software_directory
make install

Installation of drivers

Drivers are programs, like any software. The only difference is – you do not actively use them. They serve the purpose of making your hardware components understand each other. As simple as that. You need them to enhance your usage of the operating system.

Most often, the necessary drivers will be included with the distribution and installed during the setup. Sometimes, you might not be so lucky and will reach a newly installed desktop without sound, network or video drivers.

I will not go into details explaining how specific drivers are installed. You should contact your vendors for that information. I will explain how to install the drivers, how to load them, and then how to add them to startup, so they will load automatically every time your machine starts.


Just like any software, drivers may be compiled or not. Most often, they will not be. Drivers will usually be distributed as sources, in order to achieve maximal possible compatibility with the hardware on the installation platform. This means you will have to compile from sources. Piece of cake. We already know how to do that.

If the vendor is benevolent, it is possible that the driver will be accompanied with a self-installation script. In other words, you will need to run only one command, which will in turn extract the archive, compile, install, and load it. But this might not be the case – or might not even work. I have personally witnessed a driver self-installation script go wrong. Therefore, for all practical purposes, you should probably manually install the driver.

After successfully extracting the archive and compiling the sources (./configure, make, make install), you will most likely be faced with three choices:

  • The driver will be fully configured and copied to default directories and the system paths updated. You will not need do anything special to use the driver.
  • The driver will be auto-configured and the system paths updated. This means you will only have to add the driver name to the list of drivers loaded during the boot to enable it every time the machine starts.
  • The driver will be ready to use, but will not be configured nor system paths updated. You will have to manually load the driver and then update the list of drivers loaded during the boot to enable it every time the machine starts.

The second option will make the installation process probably look like this:

tar zxf some_driver.tar.gz
tar -xjf some_driver.tar.bz2

cd some_driver_directory
make install



All that remains is to add this driver to the list of drivers loaded at bootup. In Linux, the drivers are often referred to as modules.

You need to open the configuration file containing the list of modules. You should refer to your specific distribution for exact name and location of this file. In Ubuntu, the file is called modules.conf and is found in /etcdirectory (/etc/modules.conf). We will update this file, but first we will back it up! Please remember that you need root privileges to meddle with the configuration files.

This is what our procedure would look like:

cp /etc/modules.conf /etc/modules.conf.bak
gedit /etc/modules.conf

The above commands will open the file modules.conf inside the gedit text editor. Simply add your driver in an empty line below the existing drivers, save the file, exit the text editor, and reboot for the change to take effect. That’s all!

Here’s an example of a modules.conf file for a Kubuntu Linux, installed as a virtual machine. To add a new driver, we would simply write its name below the existing entries. Of course, you need to know the EXACT name of the driver in question.

Linux commands - modules.conf

The third option is a bit more complex.

Loading drivers

You have successfully compiled the driver, but nothing has happened yet. This is because the driver is not yet enabled. Looking inside the directory, you will notice a file with .ko extension. This is your driver and you need to manually load it.

We need to install the driver into the kernel. This can be done using the insmod command.

cd driver_directory
insmod driver.ko

After the driver is loaded, it can be configured. To verify that the driver is indeed present, you can list all the available modules:


If by some chance you have made a terrible mistake and you wish to remove the driver, you can use thermmod command:


Configuration of drivers

Configuring the driver requires a bit of knowledge into its functionality. Most often, instructions will be included in the how-to text files.

Below, the example demonstrates how the network card is configured after the network driver is loaded. The network card is assigned an identifier and an IP address. In this particular case, eth0 was the selected device name, although it could be also eth1, eth2 or any the name. The assigned IP address tells us the machine will be part of a LAN network.

ifconfig eth0

After a reboot, you will realize that you no longer enjoy a network connection. This is because your driver has not been created in a common default directory and the system does not know where to look for it. You will have to repeat the entire procedure again:

cd driver_directory
insmod driver.ko
ifconfig eth0

You now realize that an automated script would be an excellent idea for solving this problem. This is exactly what we’re going to do – write a script and add it to bootup.


Like in DOS and Windows, scripts can be written in any text editor. However, special changes are needed to separate between text files and scripts. In the Windows department, simply renaming the .txt extension to .bat will convert the file to a script. In Linux, things are a bit different.

Linux command line lives inside a shell – or more precisely Shell. There are several Shells, each with a unique set of commands. The most common (and default) Linux Shell is the BASH. We need to add this information to our script, if we wish to make it communicate with our Shell. Therefore, the above commands + Shell addition will make the following script:


cd driver_directory
insmod driver.ko
ifconfig eth0

We can also make it shorter:


insmod /home/roger/driver_directory/driver.ko
ifconfig eth0

Now, we have a script. Or rather a text file that contains the relevant commands. We need to make it into an executable file. First, we need to save the file. Let’s call it network_script. To make the script executable:

chmod +x network_script

Now we have a real script. We need to place it in the /etc/init.d directory so that it will be run during bootup.

cp network_script /etc/init.d/

And finally, we need to update the system, so it will take our script into consideration.

update-rc.d network_script defaults

After you reboot, you will realize that your driver loads automatically and that your network card is configured! Alternatively, it is possible that the make install of the driver will place in the default directory:

/lib/modules/<KERNEL VERSION>/kernel/drivers/net/driver.ko

Or you could place the driver in this directory by yourself. This way, you will be able to avoid the step of writing the script. However, my method, even if not the most elegant one, has one advantage: Drivers that you have manually compiled and placed into the default directories will be lost every time you update the kernel. This means you will have to reinstall them again after every such update. My method un-elegantly escapes this problem.

Mounting a drive

If you run a dual-boot system, it is entirely possible that you have installed your Linux before you have formatted all the Windows drives. This means that some of these drives might not be mounted – or accessible – when you’re booted in Linux. Alternatively, you might have formatted the drives, but you have resized and relettered and renamed the partitions and they are no longer recognized by Linux. Furthermore, you just might be unlucky and your Linux refuses to see the drives despite your best efforts. Finally, you might be able to see them, but you cannot write to the NTFS drives and this irks you so. Compared to the above tasks, mounting drives is a simple job.

To be able to do this correctly, you need to know how your drives are ordered and what they are called, both in Windows and Linux. This requires that you be able to correlate between Windows partitions (E:\, G:\, K:\ etc.) and Linux partitions (hda1, hda4, hdb2 etc.).

First, make sure you know the order of your partitions in Windows. Then, when booted in Linux, list the Partition Tables:

fdisk -l

The above command will display all the available partitions on your system. In this example, you see only the Linux partitions present, but there might be other (Windows) partitions.

Linux commands - fdisk

For the sake of this exercise, let’s assume that Linux partitions are hda4-6, while Windows partitions are hda1-3.


  • hda1 will be Windows C:\ drive.
  • hda2 will be Windows F:\ drive – also called Data.
  • hda3 will be Windows G:\ drive – also called Games.
  • hda4 will be Linux swap / Solaris.
  • hda5 will be Linux (your /root).
  • hda6 will be Linux (your /home).

Now, before you mount a drive, you need to create a mount point. This is most conveniently done by assigned a directory within the /media directory. For example:

mkdir /media/data

The name data is arbitrary, but it can help relate the mounted drive to its Windows designation. Now, we need to mount the drive that corresponds to data. In our case, this is hda2.

There are several ways of mounting the drive. By default, NTFS partitions are mounted as read-only, although write access can also be enabled. FAT32 partitions are writable by default.

Like before, mounting the drive only once will hold valid for the current session. After reboot, the changes will be lost. Therefore, we need to add the mounting of the relevant partitions to the boot chain. The configuration file that holds this crucial information is called fstab and is located under /etc (/etc/fstab).

Therefore, in order to mount the NTFS drive (Windows F:\ drive called data) as read-only we need to:

  • Create a directory called data within /media.
  • Backup fstab.
  • Add a new line to the fstab file – that will mount the NTFS drive hda2 (Windows F:\data) as read-only.
mkdir /media/data
cp /etc/fstab /etc/fstab.bak
gedit /etc/fstab

After opening the file in the text editor, we need to add the mount command. NTFS read-only:

/dev/hda2 /media/data ntfs nls=utf8,umask=0222 0 0

The necessary commands, as well as procedures are well-documented in the Unofficial Ubuntu 6.10 (Edgy Eft) Starter Guide. Here, you can see the sample fstab file inside Kate text editor, for Kubuntu Linux.

Linux commands - fstab

Other options

Alternatively, if you have partitions formatted with FAT32 file system or you wish to be able to write to NTFS partitions from within Linux, you can use the following commands:

FAT32 read/write:

/dev/hda2 /media/data vfat iocharset=utf8,umask=000 0 0

NTFS read/write – requires installation of software that can write to NTFS drives.

apt-get install ntfs-3g
/dev/hda1 /media/data ntfs-3g defaults,locale=en_US.utf8 0 0

An exercise: Let’s assume we wish to be able to write to NTFS partition C, read-only NTFS partition F and use FAT32 partition G. In that case, the list of commands that we need to execute is:

apt-get install ntfs-3g

mkdir /media/windows
mkdir /media/data
mkdir /media/games

cp /etc/fstab /etc/fstab.bak
gedit /etc/fstab


/dev/hda1 /media/windows ntfs-3g defaults,locale=en_US.utf8 0 0
/dev/hda2 /media/data ntfs nls=utf8,umask=0222 0 0
/dev/hda3 /media/games vfat iocharset=utf8,umask=000 0 0

Installation of graphic card drivers

Please note that commands used in this subsection are for Nvidia drivers ONLY – I have several computers, ALL of which have Nvidia graphic cards – but some of the solutions presented work for both Nvidia and ATI cards.

Although I have already discussed the installation of graphic card drivers in my Installing SUSE Linux andInstalling Kubuntu Linux articles, I think a bit of extra guidance will not hurt anyone.

Basically, you can install the graphic card drivers using a Package Manager or via the command line. For most people, the first method should work flawlessly. The first method is embodied in these two commands – the download of the required package and the installation of the driver:

apt-get install nvidia-glx
nvidia-glx-config enable

Some people might prefer to install the drivers manually, with the X Windows stopped. To do this, you literally need to stop the desktop from running.

/etc/init.d/gdm stop
/etc/init.d/kdm stop
/etc/init.d/xdm stop

The desktop should vanish and be replaced with a command line. You will probably need to login. It is possible that you will only see a black screen and no command prompt. Do not be alarmed! Linux operating system usually has 7 virtual consoles. The first six consoles provide a text terminal with a login prompt to a UNIX shell. The 7th virtual console is used to start the X Windows.

In other words, it may occur that by stopping the X Windows you will have simply switched off the graphics AND remain in the 7th virtual console, therefore having no command line to work with. All you need to do is switch to one of the text consoles by pressing Alt + F1-6 on the keyboard. Now, you need to install your driver:


After the installation is complete, you should simply restart the X Windows.

/etc/init.d/gdm start
/etc/init.d/kdm start
/etc/init.d/xdm start

If you see an Nvidia splash logo, it means the driver has been successfully installed. Reboot your machine just to make sure. This is where you might encounter a problem.

Instead of the Nvidia logo, you will see an error message indicating that the X Server has been disabled and that you need to manually edit the settings in the xorg.conf file before being able to proceed to the desktop. Now, there are many possible reasons for such an error and trying to provide a general solution is impossible.

However, I have found the following argument to hold true for many cases: If you have setup your Linux distribution using the GUI installer, you will have probably used the default configurations and the generic kernel will have been installed. I this case, sometimes, the built-in Nvidia driver (nv) might interrupt with the installation. There are two methods for solving this problem.

Method 1: Alberto Milone’s envy package

Envy is a command-line application that will download the latest drivers for your card, clean up old drivers and install the new ones. Instructions for the usage can be found below the download links.

Method 2: Do it yourself

First, download the required driver. Then, execute the following commands:

The offending built-in driver needs to be disabled.

gedit /etc/default/linux-restricted-modules-common

Change the last line to DISABLED_MODULES=”nv”. This will prevent the built-in driver from loading and interrupting with your own installed driver.

Linux commands - linux-restricted

Now, you should remove all conflicting files from your system:

apt-get install linux-headers-`uname -r` build-essential gcc gcc-3.4 xserver-xorg-dev

apt-get –purge remove nvidia-glx nvidia-settings nvidia-kernel-common

rm /etc/init.d/nvidia-*

After the offenders are removed, you should install the drivers from the command line:

/etc/init.d/gdm stop
nvidia-xconfig –add-argb-glx-visuals
/etc/init.d/gdm start

Again, you should see the Nvidia splash logo. Reboot just to make sure there are no more surprises. This should get you up and running with the latest graphic card driver.

Network sharing

If you have more than one computer, you are probably sharing resources among them.There is no reason why you should not continue doing this if one of the machines is running a Linux distribution. Sharing can be accomplished in many ways. Perhaps the simplest is using Samba server. First, install Samba:

apt-get install samba

After the Samba server is installed, you will need to edit a few options in the configuration file to allow sharing privileges.

cp /etc/samba/smb.conf /etc/samba/smb.conf.bak
gedit /etc/samba/smb.conf

In the configuration file, you will need to setup a number of parameters:

  • workgroup = workgroup_name – the name of the Workgroup for your LAN (e.g. HOME)
  • netbios name = netbios_name – without spaces; computer alias by which you will be able to call it across the network
  • security = user

After saving the configuration file, you will have to restart the Samba server:

/etc/init.d/samba restart

Now, select a folder that you wish to share.

Linux commands - samba share 1

If you have ticked the option Writable, you will be able to modify the contents of this folder. Finally, to be able to connect to this share from Windows, you will have to create a Samba user:

smbpasswd -a ‘name’

Under ‘name’ you should specify an existing UNIX user (e.g. roger). Do not forget the apostrophes! You will be asked to create a password. And finally, restart the Samba server again, for the changes to take effect. Now, the sharing itself. Very simple.

Windows > Linux

Start > Run > \\ OR \\netbios_name

When asked for username and password, provide the Samba user name, e.g. roger and the relevant password. And that’s it. Browse to the shared folder. If the shared folder is writable, you will be able to modify the contents.

Linux > Windows

Press Alt + F2. This will bring up the Run Command window. In the Command line, specify the IP address or the name of the computer that you wish to connect. You can see an example below:

Linux commands - samba sharing 2

And that’s it. Easy peasy lemon squeasy!

Printer sharing

Well now, folder and file sharing is really easy. What about the printers? Again, it is very simple. If you have a printer installed on a Windows machine, accessing it from a Linux machine will be easy. The rougher side of the coin is accessing a printer installed on a Linux machine from a Windows machine.First, you will have to allow your printer to be shared. Backup and then edit the Common UNIX Printer System configuration file.

cp /etc/cups/cupds.conf /etc/cups/cupsd.conf.bak
gedit /etc/cups/cupsd.conf

In the file, search for the entry #Listen and add or change as follows:

#Listen OR localhost:631 OR *:631
Listen /var/run/cups/cups.sock

CUPS listens on the port 631. If you use a static IP address for the Linux machine, you can specify only that IP. Otherwise, you might need to use a wildcard. Of course, you should be aware that an open port means a wee less security than before, so keep that in mind. After saving the changes, you will have to restart CUPS:

/etc/init.d/cupsys restart

Now that the printer is available, you will have to add it for the Windows machine.

Start > Settings > Printers and Faxes
File > Add Printer

… A network printer, or a printer attached to another computer …
… Connect to a printer on the Internet or on a home or office network …

When prompted for the driver, either select from a list or install it from a disk (like CD). And that’s it! You can now print from a Windows machine on a printer connected to a Linux machine.

Tip: If you are using a Lexmark printer, you will probably not be able to find the right Linux drivers for your printer. Worry not! Using generic drivers for Hewlett Packard printers will work remarkably well.

Other useful commands

Here’s a tiny sampling of some other useful tools that you might want to know. Be aware that the commands are presented in a generic way only. A variety of options (switches) can be used in conjunction with many of the commands to make their usage far more complex and effective.

Switching between runlevels

init 0-6


telinit 0-6

Backing up the X Windows configuration file

cp /etc/X11/xorg.conf /etc/X11/xorg.conf.bak

Sometimes, you may need or want to configure the X Windows manually:

dpkg-reconfigure xserver-xorg

Display system environment information

You can use the cat (concatenate) command, which will print the contents of the files into the terminal. To display the CPU parameters:

cat /proc/cpuinfo

To display the memory parameters:

cat /proc/meminfo

To find the version of your kernel and the GCC compiler:

cat /proc/version

Furthermore, to find out the version of your kernel:

uname -r

Listing information about files and folders

This command is the equivalent of the DOS dir command.


To display hidden files as well (starting with dot).

ls -a

Kill a process

Sometimes, you may start an application … only it does not really start. So you try again. But this time, your distro informs you that the process is already running. This can also happen in Windows. Sometimes, processes remain open and need to be killed. Before you can kill a process, you need to know its ID. The command below will list all running processes:

ps -elf

Then, kill the offending process by its ID.

kill PID

Alternatively, you can kill a process by its name. The below command will terminate all processes with the corresponding name (or names).

killall process_name


Well, that’s it, for now. Hopefully you have learned something.

If you have had problems with your software installations, compilation from sources, drivers, partitions, and sharing, this article may have helped you overcome some of the problems. Personally, the above tips cover about 90% of tasks that a normal user would have to confront as a part of his/her daily usage. Isn’t Linux so much fun? Well, have fun tweaking.