Recently a friend of mine told me about Project Sonar by Rapid7. The purpose of this project is to enumerate as many as possible services online.

The enumeration happens by scanning all the IPs and determining what services are running on those. This is done from multiple subnets, so they will be able to collect as much information as possible. On Patrik Hudak’s website, there is an in depth explanation of how the project works, like on the Rapid7 website.

Querying domains

How to get the data

I downloaded the Forward DNS records, and specifically the 2018-11-23-1542931676-fdns_any.json.gz , that like it’s visible it is a compressed file and has size 25.6 GB .

Forward DNS JSON scheme

Below is the structure of the JSON file.

{ "$id": "https://opendata.rapid7.com/sonar.fdns_v2/", "type": "object", "definitions": {}, "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "properties": { "timestamp": { "$id": "/properties/timestamp", "type": "string", "description": "The time when this response was received in seconds since the epoch" }, "name": { "$id": "/properties/name", "type": "string", "description": "The record name" }, "type": { "$id": "/properties/type", "type": "string", "description": "The record type" }, "value": { "$id": "/properties/value", "type": "string", "description": "The response received for a record of the given name and type" } } }

Parse results from the file