World Web Security Survey of 5 Million Sites — Interim Report: What Changed When 1 Million Common Crawl Domains Were Added

Also published on Site Scouter.

Summary

Interim report of a plan to observe 5 million websites worldwide with one identical method. After stage 1 (Tranco top 1 million and other lists, 1,057,185 domains), stage 2 added 1,001,045 unobserved domains from the top of the Common Crawl domain ranking. 2,058,230 domains are observed so far; 2,948,955 remain in progress. The comparisons in this report use the 911,485 Tranco-derived domains of stage 1 and the 1,001,045 domains of stage 2.
Stage 2 reachability (DNS resolved and an HTTP response received) was 85.3% (stage 1: 85.3%). DNS failures were 8.9%: roughly one in ten names on the ranking no longer exists.
HTTPS enforcement (95.4%) is nearly the same as in stage 1, and HSTS (29.7%) is 5.4 points lower. CSP fell to 16.0% (-9.9 points), X-Frame-Options to 25.2% (-17.0 points) and sites behind a WAF/CDN to 35.2% (-15.7 points). Measures that require deliberate configuration drop as the population widens.
In Tranco, HSTS and CSP adoption fall with rank (HSTS 44.9% in the top 100k vs 31.9% at 600k–1M). Across Common Crawl rank bands the relation is not monotonic: flat at around 34% down to rank 600k, a drop to 23.6% at 600k–1M, then back to 31.0% beyond 1M. The two rankings mean different things by “top”.
9.8% of responding stage 2 sites announce a PHP version (stage 1: 6.8%), and 43.5% of those are end-of-life versions (stage 1: 48.5%). 10.5% of responding sites announce a version within the range of a published CVE (upper bound), about the same as stage 1 (10.6%).
This is an interim report. The remaining 2.95 million stage 2 domains and the pending ICANN zone-file request will be covered in the final report.

1. The survey and where this report sits

The survey observes 5 million websites worldwide for HTTPS, HSTS, security headers, TLS, certificates, server version disclosure and PHP support status, with the same method in the same period. It extends the earlier “Security Exposure Survey 2026” (10 regions, 260k sites) to world scale. The population is built in two stages.

Stage 1: Tranco top 1 million (911,485 domains, 139 countries plus gTLDs), observed 2026-10-22 to 2026-10-28. Stage 1 also carried over 145,700 domains from the earlier survey (Japanese sites from search history and Majestic, among others); their population differs, so they are excluded from the comparisons here.
Stage 2: the Common Crawl domain ranking, taking 3.95 million registrable domains not in stage 1, in rank order. This report covers the first 1,001,045 of them (ranks 398 to 1,256,831).

An ICANN CZDS (zone file) request is also pending; no approval had arrived at the time of writing. Zone files contain many registered names that are not public websites, so even once approved they will be treated as a separate population from Common Crawl.

Figure 11. Progress of the 5-million-site plan (observed and remaining)

2. Method

Each site receives one DNS lookup, one TLS handshake and one HTTP GET: the same as opening the site once in a browser. No port scanning, vulnerability probing, authentication attempts or exploit traffic. Observation runs from workers spread across several domestic connections; one hub server hands out a queue and collects results into an append-only table.

De-duplication: where a host was observed more than once, the latest row that is not a DNS failure is used, otherwise the latest row.
Common Crawl rank: the harmonic-centrality rank from the domain-level web graph of cc-main-2026-jul-aug-sep. Tranco is a “popularity” order that merges several ranking sources; the Common Crawl rank measures centrality in the link graph reached by the crawl, which is not the same as popularity. All comparisons in this report assume that difference.
Stage 1 scanners: stage 1 switched its worker implementation from PHP to Python part-way through, so some failure labels differ (PRIVATE_OR_INVALID_IP and HTTP_FAILED come from the PHP scanner). Reachability and adoption definitions are identical.
Metric definitions follow the Security Exposure Survey 2026: HTTPS enforced = an HTTP request is redirected to HTTPS; HSTS = the response carries Strict-Transport-Security; version concealed = no version number in Server or X-Powered-By; CVE exposure = the announced version falls inside the affected range of a published CVE (upper bound).

3. Reachability — one in ten ranked names is gone

Of the 1,001,045 stage 2 domains, 853,736 (85.3%) returned an HTTP response. DNS failures were 8.9%, connection failures 3.1% and TLS errors 2.8%. Stage 1 reached 85.3% with 6.8% DNS failures (Figure 1).

Figure 1. Reachability and failure breakdown by population

By rank band, Tranco reaches fewer sites in its top 100k (80.8%) than at 600k–1M (87.7%): the top of the list holds many CDN delivery domains and API hostnames that are not browser-facing sites. Common Crawl reaches close to 90% down to rank 300k (89.9% in the top 100k), falls to 81.6% at 600k–1M and recovers to 86.8% beyond 1M (Figure 2). Why the 600k–1M band alone is lower will be checked in the final report.

Figure 2. Reachability by rank band

4. Adoption of security measures — what changed and what did not

Figure 3 shows adoption as a share of responding sites.

Figure 3. Adoption of security measures, stage 1 vs stage 2

Measure	Stage 1 Tranco (n=777,491)	Stage 2 Common Crawl (n=853,736)	Diff.
HTTPS enforced	96.1%	95.4%	-0.7
HSTS	35.1%	29.7%	-5.4
HSTS (max-age ≥ 1 year)	23.1%	22.1%	-1.0
CSP	25.9%	16.0%	-9.9
X-Frame-Options	42.2%	25.2%	-17.0
X-Content-Type-Options	42.1%	31.7%	-10.4
Referrer-Policy	29.8%	19.2%	-10.6
Permissions-Policy	14.5%	11.1%	-3.4
Version concealed	74.3%	72.6%	-1.7
WAF/CDN in front	50.9%	35.2%	-15.7
TLS 1.3 negotiated	85.5%	86.6%	+1.1

HTTPS enforcement is nearly identical across the two populations: HTTPS has reached well beyond the top of the rankings. HSTS is 5.4 points lower, but the larger gaps appear in measures a site has to configure deliberately: CSP (-9.9 points), X-Frame-Options (-17.0 points), Referrer-Policy (-10.6 points) and the share of sites behind a WAF/CDN (-15.7 points). The WAF/CDN gap is explained by Cloudflare appearing in the Server header of 42.8% of stage 1 sites but only 27.7% of stage 2 sites. A front-end such as Cloudflare adds some headers without any action by the site, so when the front-end share falls, header adoption falls with it.

Among sites that set a cookie (244,340 in stage 2), Secure was present on 65.5%, HttpOnly on 62.0% and an explicit SameSite on 49.9%. Against stage 1 (64.5% / 67.9% / 57.8%), Secure is about the same while HttpOnly and SameSite are a few points lower.

5. Does adoption fall with rank? The two rankings differ

In Tranco, HSTS and CSP fall as rank falls: HSTS from 44.9% in the top 100k to 31.9% at 600k–1M, CSP from 31.8% to 23.5%. That is the intuitive pattern: the more a site is used, the more likely it is hardened.

Across Common Crawl rank bands the pattern does not appear. HSTS is 34.4% in the top 100k, 34.5% at 100k–300k and 33.5% at 300k–600k — flat — then drops to 23.6% at 600k–1M and returns to 31.0% beyond 1M. CSP follows the same shape (Figure 4). Harmonic centrality measures how easily a domain is reached by links from many pages, which is not the same as having many users. Directory-style sites with dense mutual links and older sites with link lists rank high, forming a population distinct from a popularity order.

Figure 4. HSTS, CSP and version concealment by rank band (solid: Common Crawl, dashed: Tranco)

This determines how to read the move from 1 million to 5 million. Stage 2 figures move closer to a “whole web” average, while “top N” must be treated as a different thing in Tranco and in Common Crawl.

6. TLS and certificates

Stage 2 negotiated TLS 1.3 on 86.6% of responding sites and TLS 1.2 on 7.1% (stage 1: 85.5% / 9.3%). With this method a site that only speaks TLS 1.0/1.1 fails to connect and cannot be told apart from other failures. DV certificates were 92.2% and OV 1.5% (stage 1 OV: 4.0%). The wider the population, the smaller the OV/EV share and the larger the share of sites running on free DV certificates (Figure 5).

Figure 5. TLS version and certificate validation level

7. Server products and version disclosure

In stage 2 the Server header named Cloudflare on 27.7%, nginx on 20.2%, Apache on 20.3% and LiteSpeed on 5.0% of responding sites. Compared with stage 1, Cloudflare fell while Apache and LiteSpeed rose — the profile of shared hosting. A version number was included by 21.6% of nginx, 17.4% of Apache and 99.9% of IIS sites (IIS announces its version by default).

Figure 6. Product named in the Server header and version disclosure

WordPress was identified on 25.0% of responding stage 2 sites (stage 1: 13.4%): widening the population raises the WordPress share considerably. Shopify fell from 3.8% in stage 1 to 0.5%.

8. PHP versions and support status

9.8% of responding stage 2 sites (83,813) announce a PHP version in X-Powered-By or similar, more than stage 1 (6.8%). Of the announced versions, 43.5% (36,498 sites) are end of life (PHP 8.1 or earlier), lower than stage 1 (48.5%). 8,042 sites still run PHP 5 and 20,006 run PHP 7. 11,483 announce 8.2 (security fixes only until 31 December 2026) and 35,830 announce 8.3 or later (Figure 7).

Figure 7. Support status of PHP-disclosing hosts and the distribution of disclosed versions

Version disclosure rose while the end-of-life share fell. Both the spread of PHP version switching on shared hosting and the number of environments that do not hide the version by default appear to contribute. Announcing a version is not a vulnerability in itself, but announcing an end-of-life version tells attackers where to look.

9. CVE exposure (upper bound)

10.5% of responding stage 2 sites (89,793) announce a version that falls within the affected range of a published CVE. 3.9% include a remote-code-execution class CVE and 8.2% include a CRITICAL one. Stage 1 was 10.6% / 3.3% / 7.4%: the level did not change with the wider population (Figure 8).

Figure 8. CVE exposure (upper bound)

By product, 51.4% of sites announcing a PHP version, 86.9% of those announcing Apache and 58.5% of those announcing nginx were within a published CVE range. These are upper bounds based on the version string; distributions such as Debian and RHEL back-port fixes without changing the version number, which cannot be reflected. Exploitation was not tested. The most common RCE-class match, CVE-2024-3566, mostly concerns PHP-CGI on Windows and usually does not apply to Linux shared hosting.

10. Baseline score

Seven items — HTTPS enforced, HSTS, CSP, X-Frame-Options, X-Content-Type-Options, version concealed and a valid certificate — give a 0–7 score. The stage 2 mean is 3.64 (stage 1: 4.10). Stage 2 concentrates at 3 points (35.7%), and a full 7 fell to 4.6% (stage 1: 10.2%) (Figure 9). The most common shape is “HTTPS, certificate and version hiding done; headers untouched”.

Figure 9. Distribution of the baseline score (0–7)

11. By TLD

Figure 10 compares HSTS and CSP adoption for the TLDs most common in the Common Crawl population against the same TLDs in stage 1. .jp sits at HSTS 15.0% and CSP 6.9%, down from stage 1 (27.6% / 11.0%) and the lowest among the major TLDs. .de fell from 45.0% to 32.8% HSTS, a large gap between its top sites and the rest. Newer gTLDs such as .io and .app show high HSTS adoption in both stages.

Figure 10. HSTS and CSP adoption by TLD

12. Progress and next steps

Stage 2 started on 2026-10-27 21:24 and had observed 1,001,045 domains at the cut-off for this report (2026-10-28 03:46). Observation of the remaining 2,948,955 continues, and a final 5-million-site report follows completion. The final report will add country (ccTLD) breakdowns, WordPress and PHP version combinations and CVE breakdowns by product. If the ICANN CZDS request is approved, the zone-file population will be handled as a separate stage 3.

13. Limits and caveats

The stage 2 population follows Common Crawl harmonic-centrality rank, not popularity. Do not read “top N” as in Tranco.
Reachability is from the observation points (domestic connections); sites that did not respond because of geo-restrictions or bot protection count as failures.
Stage 1 changed scanner implementation part-way, so some failure labels differ. Reachability and adoption definitions are the same.
CVE exposure is an upper bound based on version strings and cannot reflect back-ported fixes. Exploitation was not tested.
This is an interim count; figures will move as the rest of stage 2 is observed. The final report’s values take precedence.
No individual site is named in this report.

14. Method notes, disclaimer and FAQ

Is this an attack? No. Each site received the same access as opening it once in a browser, and only public HTTP responses and DNS data were used. No port scanning or vulnerability probing was performed.

Why Common Crawl? Tranco stops at 1 million. Reaching 5 million needs a large, public, reproducible list. The Common Crawl domain ranking needs no application, anyone can download the same file, and it is limited to names the crawl actually reached, which makes it closer to “public websites” than a zone file.

Can the two populations be added into a “world average”? Their nature differs, so this report keeps them apart. The final report will state the weighting and also give combined values.

Can I check my own site the same way? Yes. Enter a URL in Site Scouter’s server information check to see TLS, HSTS, headers and version disclosure with the same non-intrusive method as this survey.

Was the data altered? Results are stored in an append-only table, and the aggregation scripts and CSVs are kept in a form that can be recomputed.