Internet Scanners

The great thing about Internet Scanners is that when you want to find out what services your target has exposed (open ports, protocols, applications, content), they do most of the heavy lifting for you so you just need to query their database instead of performing port scans yourself. The other major benefit: it’s completely passive so your target won’t know anything about your search.

I performed a simple search of ‘spiderfoot’ in SHODAN which reveals some people running it openly exposed on the Internet. 👀

SHODAN: Probably the undisputed king of Internet scanners, provides a rich query language, API and most importantly a ton of data to sift through.

Censys: This platform is growing fast; high quality data and a nice interface and API to work with.

BinaryEdge: These guys have uncovered a number of leaks through their platform and have recently opened up free access to the public. They also include BitTorrent data and have an API.

Passive DNS

Passive DNS services tap into Internet DNS traffic to build up a history of DNS resolutions. We know that with DNS we can resolve a name to an IP, or an IP to a name. But what if we want to find all the names that resolve to a given IP? Or see what IPs a name has historically resolved to? Passive DNS solves this problem. No single service will ever have a complete picture and the freshness of data will vary, so it’s often best to query multiple Passive DNS services to get the most comprehensive result.

Searching for my personal website binarypool.com on SecurityTrails reveals DNS records going back to 2008!

SecurityTrails: Putting SecurityTrails under the Passive DNS category is pigeon-holing a bit as they do much more than passive DNS. They have a ton of data, a rich API and a nice UI. They are also growing fast as they build out their platform and provide a free usage tier.

Robtex: Robtex was my first introduction to passive DNS and got me hooked. It’s used by pretty much all OSINT tools because it’s been around so long, is free and has a lot of high quality data going back years. Don’t be fooled by the plain UI — it’s rich with data, and they provide an API.

HackerTarget: Another rich source of free passive DNS data and available through an API. Worth also checking out all the other tools they freely provide.

Reputation Systems

If one of your main goals with OSINT is threat intelligence, you’re in luck because the number and quality of sources is huge, so I’ve cheated a bit here and listed my two favourites plus a link to a service which aggregates all of them and compares their originality for you.

Looking up a suspicious domain name in VirusTotal reveals reputation information plus much more.

VirusTotal: Google-owned at Google-scale — a huge platform providing reputation data, passive DNS and more. Access is free but query volumes tightly controlled with throttling. Shouldn’t be an issue if you’re only using the UI.

Greynoise: A newcomer focused more on identifying Internet scanners (like those mentioned in the section above). If you’re investigating suspicious IPs then this is a great resource to eliminate false positives (“anti-threat intelligence”).

FireHOL IP Lists: As stated on the site, “ The objective is to create a blacklist that can be safe enough to be used on all systems, with a firewall, to block access entirely, from and to its listed IPs.” They also maintain historic data, analyse for data uniqueness and more.

Reverse Whois

Reverse Whois is one of my favorite OSINT resources because it’s so powerful and often yields surprising (and funny) results. While regular Whois only provides searching by the domain name, Reverse Whois resources enable you to search current and historic Whois records by fields such as name, phone number and email address. More concretely, if abc@xyz.com is the contact for the domain you’re investigating, you can find all the other domains registered under that e-mail address. This is incredibly useful for when attempting to identify shadow IT issues or discovering the full perimeter of your target beyond the primary domain name(s).

Searching for generalcounsel@trumporg.com reveals other domains registered under that address.

ViewDNS.info: Many free tools are provided on their website, and pricing for the API is quite reasonable. Without the API you can still query quite a bit from the website and the data goes very far back. You can also download data in bulk.

WhoXY: Their data covers over 2000 TLDs, is available via an API and in bulk. You can also pay for query volumes in chunks at a rate of $2 per 1000 queries. Not bad at all.

SecurityTrails: I had to mention SecurityTrails again because yes, they provide reverse Whois data too.

Automation Tooling

With the ever increasing number of OSINT data sources you will eventually need some automation so that you’re not spending all your time switching between browser windows, copying and pasting and increasing the chance of human error. Tools will enable you collect, structure, correlate and visualise OSINT data and even monitor for changes over time.

There are a growing number of tools entering the OSINT arena, covering sub-domain enumeration, social media correlation and so on. Sometimes a general broad-coverage tool is good to give you access to a lot of data points at once, and other times you want a very narrow and specific tool for just one data point. One critical theme however is freshness — try and use tools that are actively maintained so that you can be confident they are using the latest APIs of the data sources they integrate with.

SpiderFoot is available as open source, or a cloud-hosted version (SpiderFoot HX, above) which is currently in Private Beta.

SpiderFoot: Being the author of SpiderFoot, it’s impossible for me to be impartial so I’ll just say that it provides a web-based UI, CLI, is actively maintained and has over 150 modules for integrating with and analysing OSINT data sources. Also, the SpiderFoot HX Private Beta is still open at the time of writing, which has much more functionality.

Maltego: Available as Community Edition (free) or commercial. Maltego has impressive visualisation capabilities and works with the model of OSINT “transforms” which transform one type of data (e.g. e-mail address) to another (e.g. a person’s name).

theHarvester: Very popular open source pure CLI OSINT tool which integrates with a number of data sources including several mentioned in this article.

Community

One of the best things to happen in the OSINT space over the past couple of years is the growth in the community. Blog posts, chat groups, aggregated resource lists and even Podcasts are now available:

A podcast 100% dedicated to OSINT.