Rapid7 Open Data offers researchers and partners access to data from Project Sonar, which conducts internet-wide surveys to gain insights into global exposure to common vulnerabilities.
Rapid7 Open Data provides access to Project Sonar's internet telemetry data in order to help security researchers and advocates advance security on a societal level. Data access is provided with appropriate balancing security controls to protect privacy as well as use case review to ensure that the data is being used in alignment with the project's goals.
Open Data is mostly aligned with the following use cases:
Rapid7 customers can access Project Sonar data relating to their assets through Project Doppler, a free tool that provides more curated insight into an organization’s external exposures and attack surface. We are investigating ways to extend Project Doppler access to non-customer internal InfoSec teams while still balancing privacy concerns.
If you have a use case for Project Sonar data that does not fit into one of the categories above, please contact us at research[at]rapid7.com. We welcome any opportunity to better understand how our data may be useful and we want to continue to advance security and support the security community as best we can.
In order to ensure that data is only used in support of the project's goals and to help protect privacy we have implemented the following general restrictions:
Feel free to contact research[at]rapid7.com regarding further questions.
If you would like access to our Open Data datasets please contact research[at]rapid7.com and provide the following:
Note that all requests for data via Open Data will be vetted to ensure that they align with its goals of advancing security. All recipients must enter into a data sharing agreement which requires, among other things, commitments to limiting the negative privacy impact of the data's use. For some use cases we will limit what is shared by geo-filtering the data.
Project Sonar is a security research project by Rapid7 that conducts internet-wide surveys across different services and protocols to gain insights into global exposure to common vulnerabilities. The data collected is available via Open Data in an effort to enable security research.
This page contains a condensed version of the project activities. Please visit the following posts for further details and the motivation behind Project Sonar:
Project Sonar gathers data in two stages. In the first stage, this involves scanning all public IPv4 addresses in an attempt to determine which have the respective service port open. Once an IP is identified as meeting these criteria, collection activities take place which involve connecting to and communicating with the service.
Project Sonar performs its scans from several different subnets, which can be allowlisted or blocklisted at your preference:
Project Sonar performs its collection activities from AWS EC2 us-west-1, us-west-2 and us-east-1 instances with non-static IP addresses, and as such cannot be readily allowlisted or blocklisted themselves, however it is sufficient to blocklist or allowlist the scan ranges listed above.
At no point does Sonar bypass any technical barriers or otherwise access non-public-facing computers. We are doing everything possible to reduce impact on remote networks and we follow best practices as outlined by the ZMap developers.
Sonar collects all SSL certificates visible on public IPv4 HTTPS web servers and certain non-HTTP services, such as SSL and STARTTLS-enabled email services like SMTP, IMAP and POP. This data can be used to detect changes such as malicious replacement of certificates or reveal the revocation of a compromised previous certificate. This data is complementary to the Electronic Frontier Foundation's SSL Observatory project. Other purposes include detection of insecurely reused or still actively used revoked certificates. In addition, with the Sonar data one can see all IP addresses / services that claim to represent a particular domain - which in turn can be used for asset identification and detection of malicious certificate usage. Also the certificate fields can be used for soft- and hardware identification in specific situations.
Sonar gathers the reverse DNS records for all IPv4 addresses. This data enables organizational asset discovery and can help identify misconfigurations and possibly DNS hijacking attempts.
Sonar uses the domain names gathered from the above processes as well as certain TLD zone files to conduct DNS record requests for many common DNS record types. This data is also useful for asset discovery and the identification of phishing portals, as well as new malicious domains matching algorithmic patterns.
Sonar scans a growing number of TCP and UDP services. TCP studies include SSH, SMB, Telnet, RDP, Mongo, Redis, CouchDB, and more. UDP studies include NetBIOS, DNS, NTP, IPMI, NAT-PMP, BACNet, SIP, SNMP, MDNS, and quite a few others. We use the metadata from these publicly exposed services to identify large-scale misconfigurations and vulnerabilities in consumer, enterprise, and critical infrastructure systems.
In case you would like to be excluded from some or all of our probes please let us know at research[at]rapid7.com - make sure to mention your CIDR blocks / list of IP addresses and affiliation.
Please note that as part of the opt-out process we attempt to verify that the requestor has been delegated or otherwise controls the network addresses in the opt-out request. We typically perform this verification via WHOIS and other tools. If we cannot verify delegation or ownership we are unlikely to opt-out the requested addresses. As a note, we periodically review our Opt-out list and remove stale entries where the WHOIS record has changed or if we can no longer verify ownership, control, or affiliation. The opt-out can be requested again in the future.
Project Sonar employs a range of open-source tools, most notably the ZMap software developed by Zakir Durumeric, Eric Wustrow, and J. Alex Halderman at the University of Michigan. We publish a few of our own tools as well, including DAP and Recog, both of which are used in the processing stage of our scanning system. Learn more about the Rapid7 researchers maintaining and extracting insights from Project Sonar.