PATERNOGA Research · 2026 study

How do DAX 40 companies control AI crawlers?

For a clearly defined sample of public corporate websites, PATERNOGA checked which crawler policies and discovery files were reachable on 30 August 2026. The study measures published configuration — not actual bot visits, indexing or citations.

Central observation

A usable robots.txt file was available on 33 of 40 domains. None of those 33 files blocked the tested homepage for Googlebot, Bingbot, OAI-SearchBot, GPTBot or PerplexityBot. Policy remains unknown for 7 domains because of a 403 response or timeout.

usable robots.txt: 33; of 40 domains
explicit OAI-SearchBot rules: 2; in usable policies
usable llms.txt: 6; of 40 domains
sitemaps declared in robots.txt: 28; of 40 domains

What the data supports — and what it does not

Observation

No complete homepage block for the five tested user agents appeared in the readable subsample.

Interpretation

Most access came from generic rules. Only a few companies named AI crawlers explicitly. This shows configuration, not intent.

Not measured

The study does not show whether a bot actually visited the site, a CDN allowed it, content was indexed or an answer cited it.

No extrapolation

DAX 40 companies are not a representative sample of all German companies or the Mittelstand.

Method

Collected on 30 August 2026 at 22:31. Domains were processed sequentially. Exactly four public resources were requested once per domain: robots.txt, llms.txt, /sitemap.xml and the homepage.

1
Sample
The 40 DAX constituents observed on the collection date. One public corporate domain was defined in advance for each company.
2
robots.txt
Rules were evaluated by group for the / path. Specific user-agent groups take precedence over the generic wildcard group.
3
Discovery files
The study recorded sitemap declarations, a recognisable URL set or sitemap index at /sitemap.xml, and usable plain text at /llms.txt.
4
Homepage
The study only recorded whether JSON-LD syntax was present. It did not assess content quality or schema validity.

Detailed findings

28 of 40 robots.txt files declared a sitemap.
20 of 40 domains returned a recognisable URL set or sitemap index at /sitemap.xml. A sitemap at another undeclared path may therefore be missed.
6 of 40 domains returned usable plain text at /llms.txt. No ranking or citation effect is inferred.
16 of 40 homepages contained recognisable JSON-LD. Presence is not a quality assessment.
3 policies named GPTBot explicitly, 2 named OAI-SearchBot and 3 named PerplexityBot.

All 40 observations

Company	robots.txt	OAI Search	GPTBot	Perplexity	Sitemap declared	llms.txt
adidas	—	unknown	unknown	unknown	no	no
Airbus	200	allowed	allowed	allowed	yes	no
Allianz	403	unknown	unknown	unknown	no	no
BASF	200	allowed	allowed	allowed	yes	no
Bayer	403	unknown	unknown	unknown	no	no
Beiersdorf	200	allowed	allowed	allowed	yes	no
BMW	200	allowed	allowed	allowed	yes	no
Brenntag	200	allowed	allowed	allowed	yes	no
Commerzbank	200	allowed	allowed	allowed	yes	no
Continental	200	allowed	allowed	allowed	yes	no
Daimler Truck	200	allowed	allowed	allowed	yes	no
Deutsche Bank	200	allowed	allowed	allowed	yes	yes
Deutsche Börse	200	allowed	allowed	allowed	yes	no
Deutsche Telekom	200	allowed	allowed	allowed	yes	no
DHL Group	—	unknown	unknown	unknown	no	no
E.ON	403	unknown	unknown	unknown	no	no
Fresenius	200	allowed	allowed	allowed	no	no
Fresenius Medical Care	200	allowed	allowed	allowed	yes	no
GEA	200	allowed	allowed	allowed	yes	no
Hannover Rück	200	allowed	allowed	allowed	yes	no
Heidelberg Materials	200	allowed	allowed	allowed	yes	yes
Henkel	200	allowed	allowed	allowed	yes	no
HOCHTIEF	200	allowed	allowed	allowed	no	no
Infineon	200	allowed	allowed	allowed	yes	no
Mercedes-Benz Group	403	unknown	unknown	unknown	no	no
Merck KGaA	—	unknown	unknown	unknown	no	no
MTU Aero Engines	200	allowed	allowed	allowed	no	no
Munich Re	200	allowed	allowed	allowed	yes	no
QIAGEN	200	allowed	allowed	allowed	yes	no
Rheinmetall	200	allowed	allowed	allowed	yes	no
RWE	200	allowed	allowed	allowed	yes	no
SAP	200	allowed	allowed	allowed	yes	yes
Scout24	200	allowed	allowed	allowed	yes	no
Siemens	200	allowed	allowed	allowed	yes	yes
Siemens Energy	200	allowed	allowed	allowed	yes	no
Siemens Healthineers	200	allowed	allowed	allowed	yes	no
Symrise	200	allowed	allowed	allowed	no	no
Volkswagen	200	allowed	allowed	allowed	yes	yes
Vonovia	200	allowed	allowed	allowed	yes	yes
Zalando	200	allowed	allowed	allowed	no	no

Study limitations

Public policy, not observed bot traffic
No IP verification of external crawlers
No WAF, CDN or rendering simulation
No assessment of indexing or citation
Point-in-time measurement; domains and rules may change
Not representative of German companies as a whole

Raw data and reproducibility

The complete observations, status codes, destination URLs, methodology version and technical limitations are available as JSON and CSV. The collection script remains versioned with the project so a later edition can use the same logic.

Download raw JSON Download raw CSV

Sample and technical primary sources

Crawler access is a prerequisite, not a visibility guarantee

A credible assessment separates policy, server access, indexing, content, sources and observed answer patterns.

Read the crawler guide Review AI crawlability

Central observation

What the data supports — and what it does not

Method

Sample

robots.txt

Discovery files

Homepage