Rapid7 Open Data provides commercial access to data from Project Sonar, which conducts internet-wide surveys to gain insights into global exposure to common vulnerabilities.
Rapid7 Open Data provides commercial access to Project Sonar's internet telemetry data to enable security organizations to map and understand the global internet attack surface. Data access is provided with appropriate security controls to protect privacy as well as use case review to ensure that the data is being used in alignment with the project's goals.
Open Data is designed for organizations seeking to map the internet attack surface and improve their security posture:
If you have a commercial use case for Project Sonar data, please contact us at opendata[at]rapid7.com. We work with organizations to understand their specific needs and determine how our data can support their security initiatives.
To ensure data is used appropriately and to protect privacy, we have implemented the following restrictions:
Feel free to contact research[at]rapid7.com regarding further questions.
To inquire about commercial access to our Open Data datasets, please contact opendata[at]rapid7.com and provide the following:
All access requests will be reviewed to ensure alignment with appropriate security use cases. Access is provided through a commercial data licensing agreement which includes privacy protections, usage restrictions, and other terms. Pricing and terms are determined based on the specific use case and data requirements.
Project Sonar is a security research project by Rapid7 that conducts internet-wide surveys across different services and protocols to gain insights into global exposure to common vulnerabilities. The data collected is available via Open Data in an effort to enable security research.
This page contains a condensed version of the project activities. Please visit the following posts for further details and the motivation behind Project Sonar:
Project Sonar gathers data in two stages. In the first stage, this involves scanning all public IPv4 addresses in an attempt to determine which have the respective service port open. Once an IP is identified as meeting these criteria, collection activities take place which involve connecting to and communicating with the service.
Project Sonar performs its scans from several different subnets, which can be allowlisted or blocklisted at your preference:
Project Sonar performs its collection activities from AWS EC2 us-west-1, us-west-2 and us-east-1 instances with non-static IP addresses, and as such cannot be readily allowlisted or blocklisted themselves, however it is sufficient to blocklist or allowlist the scan ranges listed above.
At no point does Sonar bypass any technical barriers or otherwise access non-public-facing computers. We are doing everything possible to reduce impact on remote networks and we follow best practices as outlined by the ZMap developers.
Sonar collects all SSL certificates visible on public IPv4 HTTPS web servers and certain non-HTTP services, such as SSL and STARTTLS-enabled email services like SMTP, IMAP and POP. This data can be used to detect changes such as malicious replacement of certificates or reveal the revocation of a compromised previous certificate. This data is complementary to the Electronic Frontier Foundation's SSL Observatory project. Other purposes include detection of insecurely reused or still actively used revoked certificates. In addition, with the Sonar data one can see all IP addresses / services that claim to represent a particular domain - which in turn can be used for asset identification and detection of malicious certificate usage. Also the certificate fields can be used for soft- and hardware identification in specific situations.
Sonar performs several HTTP studies that collect the HTML content of all public IPv4 web servers. The main HTTP study requests the index page (“/”) on TCP port 80, and other studies request other specific pages potentially on other TCP ports. This behavior is similar to what search engines do, except that Sonar does not crawl the servers beyond the initial requested page. One of the potential uses of this data set is the identification of compromised web servers and injected malicious HTML snippets such as "iframes" to non-advertisement web servers. We found several instances of Javascript and direct IFrames pointing to so-called "exploit kits" that try to infect client computers. We also use this data to identify vulnerable embedded devices through fingerprinting the content and headers of the HTTP response
Sonar gathers the reverse DNS records for all IPv4 addresses. This data enables organizational asset discovery and can help identify misconfigurations and possibly DNS hijacking attempts.
Sonar uses the domain names gathered from the above processes as well as certain TLD zone files to conduct DNS record requests for many common DNS record types. This data is also useful for asset discovery and the identification of phishing portals, as well as new malicious domains matching algorithmic patterns.
Sonar scans a growing number of TCP and UDP services. TCP studies include SSH, SMB, Telnet, RDP, Mongo, Redis, CouchDB, and more. UDP studies include NetBIOS, DNS, NTP, IPMI, NAT-PMP, BACNet, SIP, SNMP, MDNS, and quite a few others. We use the metadata from these publicly exposed services to identify large-scale misconfigurations and vulnerabilities in consumer, enterprise, and critical infrastructure systems.
In case you would like to be excluded from some or all of our probes please let us know at research[at]rapid7.com - make sure to mention your CIDR blocks / list of IP addresses and affiliation.
Please note that as part of the opt-out process we attempt to verify that the requestor has been delegated or otherwise controls the network addresses in the opt-out request. We typically perform this verification via WHOIS and other tools. If we cannot verify delegation or ownership we are unlikely to opt-out the requested addresses. As a note, we periodically review our Opt-out list and remove stale entries where the WHOIS record has changed or if we can no longer verify ownership, control, or affiliation. The opt-out can be requested again in the future.
Project Sonar employs a range of open-source tools, most notably the ZMap software developed by Zakir Durumeric, Eric Wustrow, and J. Alex Halderman at the University of Michigan. We publish a few of our own tools as well, including DAP and Recog, both of which are used in the processing stage of our scanning system. Learn more about the Rapid7 researchers maintaining and extracting insights from Project Sonar.
