Never Ending Security

It starts all here

Tag Archives: Recon-ng Framework

The Recon-ng Framework – A full-featured Web Reconnaissance framework written in Python.

5 Comments Posted by on 7 April 2015

The Recon-ng Framework

Featured | News | Credits | Usage Guide | Development Guide

Featured

Recon-ng is regarded as one of the top tools for open source reconnaisance and is featured in the following resources:

Training Courses

Books

  • “Advanced OSINT Target Profiling” by Shane MacDougall
  • Various titles available through Amazon.

News

Credits

A special thanks to my good friend Ethan Robish (@EthanRobish) who has been a technical advisor for this project since its inception. Without Ethan, Recon-ng simply would not be the tool that it is today. Thanks Ethan. For other credits, please see the various accepted pull requests and module meta data. Thank you to all who have contributed. Recon-ng is truly a community project.

Getting Started

Installation – Kali Linux

  • Install Recon-ng
    • apt-get update && apt-get install recon-ng

Installation – Source

  • Clone the Recon-ng repository.
    • git clone https://LaNMaSteR53@bitbucket.org/LaNMaSteR53/recon-ng.git
  • Change into the Recon-ng directory.
    • cd recon-ng
  • Install dependencies.
    • pip install -r REQUIREMENTS
  • Launch Recon-ng.
    • ./recon-ng
  • Use the “-h” switch for information on runtime options.
    • ./recon-ng -h

Dependencies

  • There is no guarantee that the included 3rd party libraries will work on all systems and architectures. If load errors are encountered, try downloading, compiling, and replacing the library which is raising exceptions.

Usage Notes

Below are a few helpful nuggets for getting started with the Recon-ng framework. While not all features are covered, the following notes will help make sense of a few of the frameworks more helpful and complex features.

  • Users will likely create and share custom modules that are not merged into the master branch of the framework. In order to allow for the use of these modules without interfering with installed package, the framework allows for the use of a custom module tree placed in the user’s “home” directory. In order to leverage this feature, a directory named “modules” must be created underneath the “.recon-ng” directory, i.e. “~/.recon-ng/modules/”. Custom modules that are added to the “~/.recon-ng/modules/” directory are loaded into the framework at runtime. Where the modules are placed underneath the “~/.recon-ng/modules/” directory doesn’t effect functionality, but things will look much nicer in the framework if the proper module directory tree is replicated and the modules are placed in the proper category.
  • Modules are organized to facilitate the flow of a penetration test, and there are separate module branches within the module tree for each methodology step. Reconnaissance, Discovery, Exploitation and Reporting are steps 1, 3, 4 and 5 of the Web Application Penetration Testing Methodology. Therefore, each of these steps has their own branch in the module tree. It is important to understand the difference between Reconnaissance and Discovery. Reconnaissance is the use of open sources to gain information about a target, commonly referred to as “passive reconnaissance”. Discovery, commonly referred to as “active reconnaissance”, occurs when packets are explicitly sent to the target network in an attempt to “discover” vulnerabilities. While Recon-ng is a reconnaissance framework, elements from the other steps of the methodology will be included as a convenient place to leverage the power of Python.
  • After loading a module, the context of the framework changes, and a new set of commands and options are available. These commands and options are unique to the module. Use the “help” and “show” commands to gain familiarity with the framework and available commands and options at the root (global) and module contexts.
  • The “info” and “source” subcommands of “show” (available only in the module context) are particularly helpful ways to discover the capabilities of the framework. The “show info” command will return detailed information about the loaded module, and the “show source” command will display its source code. Spend some time exploring modules with the “show info” and “show source” commands to get a sense for how information flows through the framework.
  • The “query” command assists in managing and understanding the data stored in the database. Users are expected to know and understand Structured Query Language (SQL) in order to interact with the database via the “query” command. The “show schema” command provides a graphical representation of the database schema to assist in building SQL queries. The “show schema” command creates the graphical representation dynamically, so as the schema of the database changes, so will the result of the command.
  • Pay attention to the global options. Global options are the options that are available at the root (global) context of the framework. Global options have a global effect on how the framework operates. Global options such as “VERBOSE” and “PROXY” drastically change how the modules present feedback and make web requests. Explore and understand the global options before diving into the modules.
  • The modular nature of the framework requires frequently switching between modules and setting options unique to each one. It can become taxing having to repeatedly set module options as information flows through the framework. Therefore, option values for all contexts within the framework are stored locally and loaded dynamically each time the context is loaded. This provides persistence to the configration of the framework between sessions.
  • Workspaces help users to conduct multiple simultaneous engagements without having to repeatedly configure global options or databases. All of the information for each workspace is stored in its own directory underneath the “~/.recon-ng/workspaces/” folder. Each workspace consists of it’s own instance of the Recon-ng database, a configuration file for the storage of configuration options, reports from reporting modules, and any loot that is gathered from other modules. To create a new workspace, use the “workspaces” command, workspaces add <name>. Loading an existing workspace is just as easy, workspaces select <name>. To view a list of available workspaces, see the “workspaces list” command or the “show workspaces” alias. To delete a workspace, use the “workspaces delete” command, workspaces delete <name>. Workspaces can also be created or loaded at runtime by invoking the “-w <workspace>” argument when executing Recon-ng, ./recon-ng -w bhis.
  • The “search” command provides the capability to search the names of all loaded modules and present the matches to the user. The “search” command can be very helpful in determining what do do next with the information that has been harvested, or identifying what is required to get the desired information. The “recon” branch of the module tree follows the following path structure:recon/<input table>-<output table>/<module>. This provides simplicity in determining what module is available for the action the user wants to take next. To see all of the modules which accept a domain as input, search for the input table name “domains” followed by a dash: search domains-. To see all of the modules which result in harvested hosts, search for the output table name “hosts” with a preceding dash: search -hosts.
  • The entire framework is equipped with command completion. Whether exploring standard commands, or passing parameters to commands, tap the “tab” key several times to be presented with all of the available options for that command or parameter.
  • Even with command completion, module loading can be cumbersome because of the directory structure of the module tree. To make module loading easier, the framework is equipped with a smart loading feature. This feature allows modules to be loaded by referring to a keyword unique to the desired module’s name. For instance, use namechk will load the “recon/contacts-contacts/namechk” module without requiring the full path since it is the only module containing the string “namechk”. Attempting to smart load with a string that exists in more than one module name will result in a list of all possible modules for the given keyword. For example, there are many modules whose names contain the string “pwned”. Therefore, the command use pwned would not load a module, but return a list of possible modules for the user to reference by full module name.
  • Every piece of information stored in the Recon-ng database is a potential input “seed” from which new information can be harvested. The “add” command allows users to add initial records to the database which will become input for modules. Modules take the seed data, transform it into other data types, and store the data in the database as potential input for other modules. Each module has a “SOURCE” option which determines the seed data. The “SOURCE” option provides flexibiliy in what the user can provide to modules as input. The “SOURCE” option allows users to select “default”, which is seed data from the database as determined by the module developer, a single entry as a string, the path to a file, or a custom SQL query. The framework will detect the source and provide it as input to the module. Changing the “SOURCE” option of a module does not effect how the module handles the resulting information.
  • While the “shell” command and “!” alias give users the ability to run system commands on the local machine from within the framework, neither of these commands is necessary to achieve this functionality. Any input that the framework does not understand as a framework command is executed as a system command. Therefore, the only time that “shell” or “!” is necessary is when the desired command shares the same name as a framework command.
  • A recorded session of all activity is essential for many penetration testers, but builtin OS tools like “tee” and “script” break needed functionality, like tab completion, and muck with output formatting. To solve this dilema, the framework is equipped with the ability to spool all activity to a file for safe keeping. The “spool” command gives users the ability to start and stop spooling, or check the current spooling status. The destination file for the spooled data is set as a parameter of the “spool start” command, spool start <filename>. Use help spool for more information on the “spool” command.
  • Developers have the ability to create new tables and columns in the database dynamically as information is harvested from various resources. Users should pay attention to local variables and run the “show schema” command often to check for new data being stored in the database. Any new table that is created will automatically be added to the list of “show” commands for quick access to the information.

Acquiring API Keys

  • Bing API Key (bing_api) – Sign up for the free subscription to the Bing Search API here. Sign in to the Windows Azure Marketplace and go to the “My Account” tab. The API key will be available under the “Account Keys” page.
  • BuiltWith API Key (builtwith_api) – Sign up for a free account here. Sign in to the application. The API key will be available in the upper right hand portion of the screen.
  • Facebook API Key (facebook_api) – TBD
  • Facebook Secret (facebook_secret) – TBD
  • Flickr API Key (flickr_api) – TBD
  • Google API Key (google_api) – Create an API Project here. The API key will be available in the project management console.
  • Google Custom Search Engine (CSE) ID (google_cse) – Create a CSE here. The CSE ID will be available in the CSE management console. Read here for guidance on configuring the CSE to search the entire web. Otherwise, the CSE will be restricted to only searching domains specified within the CSE management console. This will drastically effect the results of any module which leverages the CSE.
  • Instagram API Key (instagram_api) – Log in to http://instagram.com/developer/clients/register/ with an existing Instagram account and register a new application. Add http://localhost:31337 as the “OAuth redirect_uri”. Click “Manage Clients” at the top of the screen and the API key will be available as the “CLIENT ID”.
  • Instagram Secret (instagram_secret) – Log in to http://instagram.com/developer/. Click “Manage Clients” at the top of the screen and the Secret key will be available as the “CLIENT SECRET”.
  • IPInfoDB API Key (ipinfodb_api) – Create a free account here. Log in to the application here. The API key will be available on the “Account” tab.
  • Jigsaw API Key (jigsaw_api) – Create an account and sign up for the $1,500/year plan here. A corporate email address is preferred. Submit a request for an API token here using the same email address that was used to create the paid account. The Jigsaw API team will look up the account to validate that it is a paid membership and issue an API token.
  • LinkedIn API Key (linkedin_api) – Log in to the developer portal with an existing LinkedIn account and add a new application. Click on the application name. Add http://localhost:31337 to the list of “OAuth 2.0 Redirect URLs”. The API key will be available underneath the “OAuth Keys” heading.
    • As of November 4th, 2013, the People Search API (required for all LinkedIn related modules) has been added to the Vetted API Access program. As a result, a Vetted API Access request must be submitted and approved for the application in order for the associated API key to function properly with the LinkedIn modules.
  • LinkedIn Secret (linkedin_secret) – The Secret key will be available underneath the “OAuth Keys” heading for the application created above.
  • PwnedList API Key (pwnedlist_api) – Contact PwnedList directly regarding API access.
  • PwnedList Initialization Vector (pwnedlist_iv) – Contact PwnedList directly regarding API access.
  • PwnedList Secret (pwnedlist_secret) – Contact PwnedList directly regarding API access.
  • Shodan API Key (shodan_api) – Create an account or sign in to Shodan using one of the many options available here. The API key will be available on the right side of the screen. An upgraded account is required to access advanced search features.
  • Twitter Consumer Key (twitter_api) – Create an application here. The Consumer key will be available on the application management page.
  • Twitter Consumer Secret (twitter_secret) – The Consumer secret will be available on the application management page for the application created above.
  • VirusTotal API Key (virustotal_api) – Create a free account by clicking the “Join our community” button here. Log in to the application and select “My API key” from the user menu. The API key will be visible towards the top of the page.

Scripting the Framework

  • The entire framework is scriptable through the use of a resource file. A resource file is a plain text file containing a list of commands for the framework. By referencing the resource file when executing Recon-ng, ./recon-ng -r <filename>, the framework will read in the list of commands from the file and feed them to the command interpreter, in sequence. The resource file does not have to end by exiting the framework. The framework will automatically detect the end of the resource file and hand stdin back over to the terminal session for user input. The script is complete when the framework prompt looks like this: recon-ng > EOF.
  • To make it easy to create resource files, the framework is equipped with the ability to record commands. The “record” command gives users the ability to start and stop command recording, or check the current recording status. The destination file for the recorded commands is set as a parameter of the “record start” command, record start <filename>. Use help record for more information on the “record” command.
  • If external shell scripting is preferred, the framework includes a tool called ./recon-cli.py which makes all of the functionality of the Recon-ng framework accessible from the command line. Use ./recon-cli.py -h for information on runtime options.

Analytics

The Recon-ng project consists of a one-man development team in terms of sustaining the framework. When things break, as they often do when dealing with evolving web technologies, users don’t got to the module developer, they go to the Recon-ng Issue Tracker or directly to me. As the framework grows, module issues become more and more frequent. I needed a way to “trim the fat” in the framework and determine the best approach to maintaining broken modules. Therefore, I decided to add an analytics element (eab6307) which would allow me to track the most commonly used modules. That way, when a user comes to me and says, “There is a problem with module X.” I can look at my analytics and determine whether or not it is worth the effort to fix myself, ask them to fix, or remove from the framework all together.

Initially, I had a few folks notice and complain, citing issues with custom modules being reported and not having the ability to disable the system completely. Both of these items have been addressed (717c7c6). Overall, it is a system that helps me more efficiently maintain the framework. The only thing attributable to the user is their IP address. To me, this is a non-issue. People use the Internet everyday to visit web pages and logically attribute themselves to shady places. There is no such thing as “leaking” an external IP address. It is a part of layer 3 communication and the way the Internet works. If you don’t want your IP leaked, don’t use the Internet, or use an anonymizing service. There is no targeting or harvested information included in the analytics. I encourage users to watch the traffic and validate for themselves.

The first time Recon-ng runs, it creates a file in the user’s home ~/.recon-ng directory called .cid. It is a randomly generated UUID that is non-specific to the system. That UUID is sent with each anaytics request to differentiate users, allowing me to track how many different users are using the module. There is a big difference between one person using a module 3,000 times in a day, and 3,000 users using a module once a day. I need to know this in order to make good maintenance decisions. Analytics requests are sent each time a module is loaded using the load or use command. The analytics request includes the UUID, the module name, and the version of Recon-ng. No analytics requests are made when loading custom modules (modules that reside in the users home ~/.recon-ng/modules/ directory), and the entire system can be disabled by running Recon-ng with the --no-analytics flag. See the ./recon-ng -h help menu for more information.

Additional Help

Recon-ng has an official IRC channel (#recon-ng) located on the Freenode network. For additional help, information, and general discussion about the framework, connect to Freenode and join the channel using the /join #recon-ng command.

More information can be found at: https://bitbucket.org/LaNMaSteR53/recon-ng

Digital Forensics, Guides and Tutorials, Tools