{"article":{"title":"Which AI crawlers the Tranco top 1,000 block in robots.txt","path":"/lab/which-ai-crawlers-the-tranco-top-1000-block-in-robotstxt"},"name":"top-sites-robots","sites":[{"domain":"google.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cloudflare.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ezviz7.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazonaws.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"facebook.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"fbcdn.net","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"googleapis.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"gstatic.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"hicloudcam.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"youtube.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"microsoft.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"apple.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"akamai.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"instagram.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"akamaiedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"twitter.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"linkedin.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"azure.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"triplinkintl.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"gwfb.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"akadns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"domaincontrol.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"googlevideo.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"googletagmanager.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"appsflyersdk.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"gtld-servers.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cloudfront.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"mail.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"office.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sharepoint.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"live.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"github.com","robots":true,"blocked":["Bytespider"],"llmsTxt":true},{"domain":"aaplimg.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"shiabank.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"trafficmanager.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"lsrelayaccess.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cloud.microsoft","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wikipedia.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"w2rt6byl5nf1opmc-a.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"fastly.net","robots":true,"blocked":[],"llmsTxt":true},{"domain":"apple-dns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"netflix.com","robots":true,"blocked":["Perplexity-User","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"dzen.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"office.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"whatsapp.net","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Google-Extended","Applebot-Extended","Amazonbot"],"llmsTxt":false},{"domain":"bing.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"pv-cdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"x.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"googleusercontent.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"doubleclick.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"googlesyndication.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"youtu.be","robots":true,"blocked":[],"llmsTxt":false},{"domain":"pinterest.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"windows.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"icloud.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"myfritz.net","robots":true,"blocked":["meta-externalagent"],"llmsTxt":false},{"domain":"cloudflare.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"zoom.us","robots":true,"blocked":[],"llmsTxt":false},{"domain":"goo.gl","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adobe.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"h264.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tiktok.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"microsoftonline.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"whatsapp.com","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Google-Extended","Applebot-Extended","Amazonbot"],"llmsTxt":false},{"domain":"windowsupdate.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wordpress.org","robots":true,"blocked":[],"llmsTxt":true},{"domain":"gandi.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"yahoo.com","robots":true,"blocked":["GPTBot","ChatGPT-User","ClaudeBot","PerplexityBot","Google-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"googledomains.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wa.me","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gvt1.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"skype.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"edgekey.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"digicert.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"tiktokcdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"spotify.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"a2z.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"opera.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"b-cdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"roblox.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazonvideo.com","robots":true,"blocked":["ChatGPT-User","Claude-User","PerplexityBot","Perplexity-User","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"epicgames.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tiktokv.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vimeo.com","robots":true,"blocked":["CCBot","Bytespider"],"llmsTxt":false},{"domain":"ntp.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"aws.dev","robots":false,"blocked":[],"llmsTxt":false},{"domain":"3gppnetwork.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"workers.dev","robots":false,"blocked":[],"llmsTxt":false},{"domain":"baidu.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"blogspot.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"qq.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mozilla.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cdn77.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nginx.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"msn.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"f5.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"t-online.de","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"wordpress.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"akam.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dotaplabs.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"t.me","robots":false,"blocked":[],"llmsTxt":false},{"domain":"intuit.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"sentry.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"lencr.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"virginm.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nginx.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yandex.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"reddit.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"cdninstagram.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"edgesuite.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"2mdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ui.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"europa.eu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"chatgpt.com","robots":true,"blocked":["PerplexityBot","Google-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"gvt2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bit.ly","robots":true,"blocked":[],"llmsTxt":false},{"domain":"aliyuncs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yandex.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"azurefd.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.dev","robots":false,"blocked":[],"llmsTxt":false},{"domain":"fastly-edge.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"samsung.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"webex.com","robots":false,"blocked":[],"llmsTxt":true},{"domain":"openai.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"adobe.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"office365.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ipify.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nih.gov","robots":false,"blocked":[],"llmsTxt":false},{"domain":"github.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dropbox.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"registrar-servers.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"android.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"outlook.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ytimg.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nflxso.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"root-servers.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"windows.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cloudflare-dns.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"akamaized.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"static.microsoft","robots":false,"blocked":[],"llmsTxt":false},{"domain":"okcdn.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"duckdns.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"apache.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tm-azurefd.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"supertms.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"skybridge.click","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"discord.gg","robots":true,"blocked":[],"llmsTxt":false},{"domain":"archive.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"azurewebsites.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dns-parking.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vk.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"forms.gle","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yccdn.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nic.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tumblr.com","robots":true,"blocked":["ClaudeBot","Google-Extended","Applebot-Extended","CCBot","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"telekom.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"crashlytics.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nytimes.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"snapchat.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"jomodns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vk.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"telegram.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"godaddy.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"samsungcloud.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ubuntu.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"gravatar.com","robots":true,"blocked":["GPTBot","Google-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":true},{"domain":"meraki.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"shopify.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dnsowl.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pages.dev","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon-adsystem.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"spo-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ksyuncdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"astrumu.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"google-analytics.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"paypal.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"tbcache.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ax-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"example.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"creativecommons.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"flickr.com","robots":true,"blocked":["GPTBot","ClaudeBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"forter.com","robots":true,"blocked":["CCBot","Bytespider","meta-externalagent"],"llmsTxt":true},{"domain":"discord.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"scorecardresearch.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"comcast.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"firefox-portal-detection.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"w3.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nflximg.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"awswaf.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"reg.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cnn.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"quickconnect.to","robots":true,"blocked":[],"llmsTxt":false},{"domain":"kaspersky.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"trbcdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pki.goog","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ebay.com","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"skyhigh.cloud","robots":false,"blocked":[],"llmsTxt":false},{"domain":"brave.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"omtrdc.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"macromedia.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"soundcloud.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"theguardian.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"youtube-nocookie.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hichina.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"unity3d.com","robots":true,"blocked":["Bytespider"],"llmsTxt":true},{"domain":"doi.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hcaptcha.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"medium.com","robots":true,"blocked":["GPTBot","ClaudeBot","Applebot-Extended","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"afternic.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"stripe.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"azureedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"oracle.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mtgglobals.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ttdns2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"publicnode.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bbc.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"roku.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wikimedia.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"app-measurement.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cdngslb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"licdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mts.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"weather.com","robots":true,"blocked":["CCBot","Bytespider","Amazonbot"],"llmsTxt":true},{"domain":"amplitude.com","robots":true,"blocked":["Bytespider"],"llmsTxt":true},{"domain":"vkuserphoto.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"criteo.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"forbes.com","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"twitch.tv","robots":true,"blocked":[],"llmsTxt":false},{"domain":"googletagservices.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sciencedirect.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"researchgate.net","robots":true,"blocked":["Applebot-Extended"],"llmsTxt":false},{"domain":"namecheap.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"xiaomi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mit.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nist.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ailawandorder.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"rbxcdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"charter.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bbc.co.uk","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"applovin.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"playstation.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cisco.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"iiko.it","robots":false,"blocked":[],"llmsTxt":false},{"domain":"avcdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wsdvs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"t.co","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"shifen.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"msftconnecttest.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adtrafficquality.google","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sourceforge.net","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dnsmadeeasy.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bybanking.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"xcal.tv","robots":false,"blocked":[],"llmsTxt":false},{"domain":"salesforce.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cpanel.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"akamaihd.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"alibabadns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sfx.ms","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hubspot.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"icanhazip.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bytefcdn-oversea.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"aliyuncsslbintl.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"imdb.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"cdn-apple.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"linktr.ee","robots":true,"blocked":["Bytespider","meta-externalagent"],"llmsTxt":true},{"domain":"googleblog.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"who.int","robots":true,"blocked":["CCBot"],"llmsTxt":false},{"domain":"name-services.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mybluehost.me","robots":false,"blocked":[],"llmsTxt":false},{"domain":"alidns.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"namebrightdns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wixsite.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ttvnw.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"googleadservices.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"datadoghq.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"geobasket.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"demdex.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ibm.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"centinel-analytica.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"yandexcloud.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"slack.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"hp.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"launchdarkly.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"reuters.com","robots":true,"blocked":["GPTBot","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"ozon.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"online-metrix.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"telegram.me","robots":false,"blocked":[],"llmsTxt":false},{"domain":"booking.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"launchpad.net","robots":true,"blocked":["GPTBot","OAI-SearchBot","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"myshopify.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"archive.ph","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nature.com","robots":true,"blocked":["GPTBot","ChatGPT-User","ClaudeBot","PerplexityBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"www.gov.uk","robots":true,"blocked":[],"llmsTxt":false},{"domain":"vedcdnlb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"steampowered.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"app-analytics-services.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ggpht.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"media-amazon.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tinyurl.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gamepass.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"onelink.me","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tiktokcdn-us.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ip-api.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"wac-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"grammarly.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"jsdelivr.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"springer.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"canva.com","robots":true,"blocked":["GPTBot","ClaudeBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"alibaba.com","robots":true,"blocked":["GPTBot","Google-Extended"],"llmsTxt":false},{"domain":"msftncsi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"crcdn.pro","robots":false,"blocked":[],"llmsTxt":false},{"domain":"byteoversea.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"manulife-sinochem.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cloudflareinsights.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"miui.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wiley.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"avast.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"casalemedia.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"plex.direct","robots":false,"blocked":[],"llmsTxt":false},{"domain":"anthropic.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"zendesk.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"douyincdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"debian.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vercel-dns-016.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vungle.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"weibo.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"volcfcdndvs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"userapi.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazon.co.uk","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"heytapmobile.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"harvard.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"rubiconproject.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"arubanetworks.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"archive.is","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nflxvideo.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"espn.com","robots":true,"blocked":["GPTBot","ChatGPT-User","Google-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"autodesk.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"etsy.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"imcmdb.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"steamserver.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"php.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"unisys.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"paloaltonetworks.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"trustpilot.com","robots":true,"blocked":["Perplexity-User","Applebot-Extended","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"clarity.ms","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ok.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"naver.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"smartadserver.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"withgoogle.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"telecid.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ea.com","robots":true,"blocked":["Bytespider","meta-externalagent"],"llmsTxt":true},{"domain":"chatgpt.site","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bilibili.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"okta.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"branch.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"nvidia.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"gnu.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tradingview.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gmail.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tp-link.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"checkpoint.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dell.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"grammarly.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dotomi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"weebly.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cdc.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"dynatrace.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"browser-intake-datadoghq.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"elasticbeanstalk.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"un.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"aliyun.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"wsj.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"on.aws","robots":false,"blocked":[],"llmsTxt":false},{"domain":"shopifysvc.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cdnbuild.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"atlassian.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bsky.app","robots":true,"blocked":[],"llmsTxt":false},{"domain":"netease.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"worldnic.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pubmatic.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"google.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.de","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"go.com","robots":true,"blocked":["GPTBot","Google-Extended"],"llmsTxt":false},{"domain":"awsglobalaccelerator.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"globo.com","robots":true,"blocked":["GPTBot","Google-Extended","CCBot"],"llmsTxt":false},{"domain":"dns.google","robots":false,"blocked":[],"llmsTxt":false},{"domain":"stanford.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gitlab.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"adnxs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"allawnos.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wps.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"myhuaweicloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"xboxlive.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"impervadns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"issuu.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"opendns.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"live-video.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bloomberg.com","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"tiktokv.us","robots":false,"blocked":[],"llmsTxt":false},{"domain":"name.com","robots":true,"blocked":["meta-externalagent"],"llmsTxt":true},{"domain":"cookiedatabase.org","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"samsungcloudsolution.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazontrust.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"githubusercontent.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nasa.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"washingtonpost.com","robots":true,"blocked":["ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"twimg.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"claude.ai","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","Google-Extended"],"llmsTxt":false},{"domain":"plesk.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"spotifycdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ibyteimg.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"scdn.co","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nexthink.cloud","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adobe.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yandex.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gcdn.co","robots":false,"blocked":[],"llmsTxt":false},{"domain":"alicdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"capcutapi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"calendly.com","robots":true,"blocked":["CCBot"],"llmsTxt":true},{"domain":"substack.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"firetvcaptiveportal.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"duckduckgo.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"163.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"adobedc.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"aol.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"temu.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"line.me","robots":true,"blocked":[],"llmsTxt":false},{"domain":"steamcommunity.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"indeed.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"unsplash.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Bytespider"],"llmsTxt":false},{"domain":"tencent-cloud.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"taobao.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tiktokcdn-eu.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"expireddomains.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nmrodam.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"telekom.de","robots":false,"blocked":[],"llmsTxt":false},{"domain":"kwai-pro.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ripn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tiktokv.eu","robots":false,"blocked":[],"llmsTxt":false},{"domain":"beian.gov.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dailymotion.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"netcraze.io","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"heytapdl.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"aliexpress.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"myqcloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vercel-dns-017.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tiktokpangle.us","robots":false,"blocked":[],"llmsTxt":false},{"domain":"visualstudio.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"statsigapi.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"run.app","robots":false,"blocked":[],"llmsTxt":false},{"domain":"arxiv.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"intel.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"sberbank.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"eye4.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mzstatic.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mcafee.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dyndns.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"g.page","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yahoo.co.jp","robots":false,"blocked":[],"llmsTxt":false},{"domain":"spaceweb.pro","robots":false,"blocked":[],"llmsTxt":false},{"domain":"featureassets.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bunnyinfra.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pixabay.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"vkuser.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"lijit.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"prodregistryv2.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"duolingo.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"appsflyer.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"trendmicro.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dropcatch.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"speedtest.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"drom.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"azure-dns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bluehost.com","robots":false,"blocked":[],"llmsTxt":true},{"domain":"nr-data.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"braze.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"bx-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"shein.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"gandi-ns.fr","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dual-s-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"walmart.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"huawei.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"facebook.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"kwai.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"goodreads.com","robots":true,"blocked":["GPTBot","CCBot"],"llmsTxt":false},{"domain":"cnbc.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ddnss.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ietf.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"netangels.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"wp.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"pushy.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"businessinsider.com","robots":true,"blocked":["ClaudeBot","CCBot","Bytespider"],"llmsTxt":true},{"domain":"scribd.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.co.jp","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"force.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sina.com.cn","robots":true,"blocked":[],"llmsTxt":false},{"domain":"eventbrite.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"markmonitor.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hostgator.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"footprintdns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"quora.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Applebot-Extended","Bytespider"],"llmsTxt":false},{"domain":"teamviewer.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"kaspersky-labs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"openstreetmap.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"foxnews.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"statista.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mangosip.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"list-manage.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tplinkcloud.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"oraclecloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wbx2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"addtoany.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"squarespacedns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"playstation.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"indiatimes.com","robots":true,"blocked":["meta-externalagent"],"llmsTxt":false},{"domain":"share.google","robots":false,"blocked":[],"llmsTxt":false},{"domain":"npr.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"blogger.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"google.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"openx.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amzn.to","robots":true,"blocked":[],"llmsTxt":false},{"domain":"klaviyo.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"shop.app","robots":true,"blocked":[],"llmsTxt":true},{"domain":"spov-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ui-dns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adsafeprotected.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"behance.net","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"docker.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"inmobi.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"taboola.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"fandom.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bdydns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tandfonline.com","robots":true,"blocked":["GPTBot"],"llmsTxt":false},{"domain":"mega.co.nz","robots":true,"blocked":[],"llmsTxt":false},{"domain":"capcut.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"usgovcloudapi.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"rocket-cdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mysql.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amagi.tv","robots":false,"blocked":[],"llmsTxt":false},{"domain":"selectel.ru","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dnspod.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sc-gw.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ft.com","robots":true,"blocked":["ClaudeBot","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"mozilla.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"miit.gov.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ipv4only.arpa","robots":false,"blocked":[],"llmsTxt":false},{"domain":"stackoverflow.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"firefox.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"oup.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adobedtm.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"google.com.hk","robots":true,"blocked":[],"llmsTxt":false},{"domain":"fast.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nease.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"noaa.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"svc.ms","robots":false,"blocked":[],"llmsTxt":false},{"domain":"icloud-content.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"juicyscore.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"slideshare.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cornell.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"e2ro.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"discord.media","robots":false,"blocked":[],"llmsTxt":false},{"domain":"eset.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"exp-tas.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wired.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"threads.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"3lift.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"comcast.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"atlassian.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"sohu.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"media.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"qlivecdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"byteglb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ca.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"deviantart.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"immedia-semi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"time.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"sophos.com","robots":true,"blocked":["Amazonbot"],"llmsTxt":true},{"domain":"dailymail.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":true},{"domain":"360yield.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"uol.com.br","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"samsungqbe.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"herokudns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"rakuten.co.jp","robots":true,"blocked":[],"llmsTxt":false},{"domain":"goskope.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adsrvr.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tripadvisor.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"squarespace.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wix.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"live.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mediatek.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"chinamobile.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dreamhost.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ieee.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"conviva.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"vercel.app","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adriver.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"timeweb.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"target.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ks-cdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"aboutads.info","robots":false,"blocked":[],"llmsTxt":false},{"domain":"supercell.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"maricopa.gov","robots":false,"blocked":[],"llmsTxt":false},{"domain":"rlcdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mdpi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"xiaomi.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cloudns.net","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wildberries.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"digitalocean.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"jquery.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"crpt.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"flurry.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"unesco.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"samsungapps.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vidaahub.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hotjar.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"dyingbirds.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"whecloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sagepub.com","robots":true,"blocked":["CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"techcrunch.com","robots":true,"blocked":["GPTBot","ChatGPT-User","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"intercom.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"usatoday.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"ezvizlife.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nintendo.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"telegraph.co.uk","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"trueconf.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"optimizely.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"onetrust.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ikea.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"usercontent.goog","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vercel-dns-3.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.fr","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"free.fr","robots":true,"blocked":[],"llmsTxt":false},{"domain":"telephony.goog","robots":false,"blocked":[],"llmsTxt":false},{"domain":"berkeley.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"accuweather.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"stackadapt.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"bamgrid.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"telefonica.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"box.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"merriam-webster.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazon.ca","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"msidentity.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"doubleverify.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"edgecdn.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"avito.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ampproject.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bytedns1.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"iso.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"translate.goog","robots":false,"blocked":[],"llmsTxt":false},{"domain":"smaato.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"jotform.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"newrelic.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"verisign.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ups.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"share-dns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ring.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"service.gov.uk","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ivi.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"sentinelone.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"meta.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gtm-a2b2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"loc.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"hicloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"t-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"imrworldwide.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tencent.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hstgr.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"oneme.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dashnxdomain.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"easebar.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nike.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazon.in","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"sknt.ru","robots":true,"blocked":[],"llmsTxt":true},{"domain":"elpais.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"bidswitch.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"redhat.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"zpath.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"inner-active.mobi","robots":false,"blocked":[],"llmsTxt":false},{"domain":"primevideo.com","robots":true,"blocked":["ChatGPT-User","Claude-User","PerplexityBot","Perplexity-User","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"britannica.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cambridge.org","robots":true,"blocked":["ChatGPT-User"],"llmsTxt":false},{"domain":"surveymonkey.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"vecdnlb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"xhamster.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"uber.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mobile.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"spiegel.de","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"supabase.co","robots":true,"blocked":[],"llmsTxt":true},{"domain":"kick.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"cbsnews.com","robots":true,"blocked":["GPTBot"],"llmsTxt":false},{"domain":"wcdnga.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"gosuslugi.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"lemonde.fr","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"yelp.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"stripchat.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bandcamp.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"yahoodns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"deepl.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"fda.gov","robots":false,"blocked":[],"llmsTxt":false},{"domain":"fb.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"typekit.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"rfc-editor.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"hugedomains.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"anydesk.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dnse0.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"netflix.net","robots":true,"blocked":["Perplexity-User","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"resolver.arpa","robots":false,"blocked":[],"llmsTxt":false},{"domain":"beget.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"fontawesome.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"bugsnag.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"certcenter.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"reverse.it","robots":false,"blocked":[],"llmsTxt":false},{"domain":"globalsign.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"itch.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"alipaydns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.es","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"outbrain.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gumgum.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"shopeemobile.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"binance.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cdn20.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"quantserve.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"ya.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tubi.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cookielaw.org","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"pvp.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pinimg.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"asus.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"jamfcloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dailymail.co.uk","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":true},{"domain":"sharethrough.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"bidr.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"palmplaystore.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.it","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"rings.solutions","robots":false,"blocked":[],"llmsTxt":false},{"domain":"craigslist.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"independent.co.uk","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mediafire.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"360.cn","robots":true,"blocked":[],"llmsTxt":false},{"domain":"netexpert.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"creativecdn.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"googlezip.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ovh.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nintendo.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bitrix24.ru","robots":true,"blocked":[],"llmsTxt":true},{"domain":"agkn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"python.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"shopee.co.id","robots":true,"blocked":[],"llmsTxt":false},{"domain":"hihonorcloud.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"zoho.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"rustore.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"dv.tech","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cnet.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"shalltry.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bild.de","robots":true,"blocked":["PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"homedepot.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"nel.goog","robots":false,"blocked":[],"llmsTxt":false},{"domain":"dbankcloud.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"statuspage.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"samsungacr.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wattpad.com","robots":true,"blocked":["GPTBot"],"llmsTxt":false},{"domain":"cdnhwc1.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"livejournal.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ovscdns.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"1rx.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"twc.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"singtelchina.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"2gis.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"safebrowsing.apple","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sedo.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"g.co","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mailchimp.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"nikkei.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"google.co.uk","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cursor.sh","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wpguardian.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"apnews.com","robots":true,"blocked":["GPTBot","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Amazonbot"],"llmsTxt":false},{"domain":"herokuapp.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"marriott.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"linode.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"viber.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"wpguardian.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"criteo.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bestbuy.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"welt.de","robots":true,"blocked":["PerplexityBot","Perplexity-User","Applebot-Extended","CCBot","Bytespider","Amazonbot"],"llmsTxt":false},{"domain":"ad-tech.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mckinsey.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"auth0.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"shopee.com.br","robots":true,"blocked":[],"llmsTxt":false},{"domain":"eeroup.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"zillow.com","robots":true,"blocked":["Applebot-Extended","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"crwdcntrl.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"incapdns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"imgur.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"service-now.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"steamstatic.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"t-s1-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"enacdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"netlify.app","robots":true,"blocked":[],"llmsTxt":true},{"domain":"lenovo.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"arcgis.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cloudinary.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"everesttech.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"moloco.com","robots":true,"blocked":["Bytespider"],"llmsTxt":true},{"domain":"wswebcdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"agora.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"wp.pl","robots":true,"blocked":["GPTBot","CCBot"],"llmsTxt":false},{"domain":"mlb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"discordapp.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"prnewswire.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"samsungosp.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"life360.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ameblo.jp","robots":true,"blocked":[],"llmsTxt":false},{"domain":"fbsbx.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"uk.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"recaptcha.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"rakuten.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"liadm.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adjust.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"33across.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sc-cdn.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"akamaitech.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"focus.de","robots":false,"blocked":[],"llmsTxt":false},{"domain":"onlyfans.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"disneyplus.com","robots":true,"blocked":["Perplexity-User"],"llmsTxt":false},{"domain":"erome.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"people.com","robots":true,"blocked":["ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"digitaloceanspaces.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"azure-devices.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"perplexity.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"fiverr.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"px-cloud.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"a-mo.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"appspot.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mixpanel.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ted.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"ebay.de","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"wb.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"salesforceliveagent.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"vtwenty.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"teads.tv","robots":false,"blocked":[],"llmsTxt":false},{"domain":"privacy-mgmt.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"yieldmo.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"airbnb.com","robots":true,"blocked":["GPTBot","ClaudeBot","Applebot-Extended"],"llmsTxt":false},{"domain":"cloudsink.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"usda.gov","robots":false,"blocked":[],"llmsTxt":false},{"domain":"usps.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"kleinanzeigen.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"flipkart.com","robots":true,"blocked":["Amazonbot"],"llmsTxt":false},{"domain":"deloitte.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"vivo.com.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pendo.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"seznam.cz","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ahacdn.me","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cdnhwc2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"adform.net","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"gpcloudservice.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"chess.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ddns.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"google.com.br","robots":true,"blocked":[],"llmsTxt":false},{"domain":"imgsmail.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"discogs.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"theverge.com","robots":true,"blocked":["ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"academia.edu","robots":true,"blocked":["GPTBot","ClaudeBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"allegro.pl","robots":true,"blocked":[],"llmsTxt":true},{"domain":"www.gov.br","robots":true,"blocked":[],"llmsTxt":false},{"domain":"frontiersin.org","robots":true,"blocked":[],"llmsTxt":true},{"domain":"myspace.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"parklogic.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"daum.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"turn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"attn.tv","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bytefcdn.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"my.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"spamhaus.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"akamaitechnologies.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"unpkg.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"me.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"zoom.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"agoda.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"adgrx.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"pexels.com","robots":true,"blocked":["GPTBot","ClaudeBot","Applebot-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"xvideos.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"spectrum.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"biblegateway.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazon.com.br","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"contentsquare.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ebay.co.uk","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"deepseek.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ssl-images-amazon.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"readthedocs.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"onetag-sys.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"poki.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"unrulymedia.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"otto.de","robots":true,"blocked":[],"llmsTxt":true},{"domain":"max.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"worldbank.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nbcnews.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"instructure.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ozone.ru","robots":false,"blocked":[],"llmsTxt":false},{"domain":"vivoglobal.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"liftoff.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"bankofamerica.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"dnse2.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"smilewanted.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"porkbun.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"epa.gov","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ngenix.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"visa.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"threema.ch","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nstld.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mercadolivre.com.br","robots":true,"blocked":["GPTBot","ChatGPT-User","ClaudeBot","Claude-User","PerplexityBot","Perplexity-User","Amazonbot"],"llmsTxt":false},{"domain":"appcenter.ms","robots":false,"blocked":[],"llmsTxt":false},{"domain":"company-target.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"hbr.org","robots":true,"blocked":["GPTBot","Google-Extended","CCBot"],"llmsTxt":false},{"domain":"pikabu.ru","robots":true,"blocked":[],"llmsTxt":false},{"domain":"0xrpc.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"fwmrm.net","robots":true,"blocked":[],"llmsTxt":true},{"domain":"pornhub.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"figma.com","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","CCBot","Bytespider"],"llmsTxt":false},{"domain":"classlink.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"lg.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"princeton.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"manutouch.com.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"wyzecam.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"latimes.com","robots":true,"blocked":["GPTBot","ClaudeBot","Bytespider"],"llmsTxt":false},{"domain":"allaboutcookies.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"kunluncan.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"tailscale.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"disqus.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"msftauth.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"paramountplus.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"qualtrics.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"networkadvertising.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nexusmods.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"seedtag.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"no-ip.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"coupang.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"bund.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"amazonalexa.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"volcgslb.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"patreon.com","robots":true,"blocked":["PerplexityBot"],"llmsTxt":false},{"domain":"note.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"unruly.co","robots":false,"blocked":[],"llmsTxt":false},{"domain":"repubblica.it","robots":true,"blocked":["GPTBot","PerplexityBot","Google-Extended","Applebot-Extended","CCBot","Bytespider"],"llmsTxt":true},{"domain":"flashtalking.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mayoclinic.org","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cootlogix.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"state.gov","robots":false,"blocked":[],"llmsTxt":false},{"domain":"cpmstar.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"genius.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"youku.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"character.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tiktokw.us","robots":false,"blocked":[],"llmsTxt":false},{"domain":"eu-1-id5-sync.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"leiniao.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"onet.pl","robots":true,"blocked":["Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"pbs.org","robots":true,"blocked":["GPTBot","PerplexityBot","CCBot","Bytespider"],"llmsTxt":false},{"domain":"yellowblue.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"corriere.it","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"huffpost.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"docker.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"chaturbate.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ancestry.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"amazon.com.au","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","CCBot","Bytespider","meta-externalagent"],"llmsTxt":false},{"domain":"eporner.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"go-mpulse.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"odoo.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mercadolibre.com.ar","robots":true,"blocked":["GPTBot","ChatGPT-User","ClaudeBot","Claude-User","PerplexityBot","Perplexity-User","Amazonbot"],"llmsTxt":false},{"domain":"curseforge.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"jetbrains.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"webempresa.eu","robots":false,"blocked":[],"llmsTxt":false},{"domain":"lowes.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"change.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"hm.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"app-ads-services.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"washington.edu","robots":true,"blocked":[],"llmsTxt":false},{"domain":"twilio.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"xhamster46.desi","robots":true,"blocked":[],"llmsTxt":false},{"domain":"buydomains.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"mailchi.mp","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"caixa.gov.br","robots":false,"blocked":[],"llmsTxt":false},{"domain":"ctecdn.cn","robots":false,"blocked":[],"llmsTxt":false},{"domain":"opera-api.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"xnxx.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"n-tv.de","robots":true,"blocked":["GPTBot","ClaudeBot","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"amemv.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"btloader.com","robots":true,"blocked":["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-SearchBot","Claude-User","PerplexityBot","Perplexity-User","Google-Extended","Applebot-Extended","CCBot","Bytespider","meta-externalagent","Amazonbot"],"llmsTxt":false},{"domain":"clever.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"jd.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"a-msedge.net","robots":false,"blocked":[],"llmsTxt":false},{"domain":"sofascore.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"4dex.io","robots":false,"blocked":[],"llmsTxt":false},{"domain":"flightradar24.com","robots":true,"blocked":["GPTBot","CCBot"],"llmsTxt":true},{"domain":"playfabapi.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"synology.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"trendyol.com","robots":true,"blocked":[],"llmsTxt":false}],"agents":[{"share":"21.7%","token":"GPTBot","purpose":"training","blocking":107,"operator":"OpenAI"},{"share":"11.5%","token":"OAI-SearchBot","purpose":"search","blocking":57,"operator":"OpenAI"},{"share":"13.4%","token":"ChatGPT-User","purpose":"user request","blocking":66,"operator":"OpenAI"},{"share":"21.3%","token":"ClaudeBot","purpose":"training","blocking":105,"operator":"Anthropic"},{"share":"13.6%","token":"Claude-SearchBot","purpose":"search","blocking":67,"operator":"Anthropic"},{"share":"13.8%","token":"Claude-User","purpose":"user request","blocking":68,"operator":"Anthropic"},{"share":"17.4%","token":"PerplexityBot","purpose":"search","blocking":86,"operator":"Perplexity"},{"share":"15.0%","token":"Perplexity-User","purpose":"user request","blocking":74,"operator":"Perplexity"},{"share":"18.0%","token":"Google-Extended","purpose":"training","blocking":89,"operator":"Google"},{"share":"18.4%","token":"Applebot-Extended","purpose":"training","blocking":91,"operator":"Apple"},{"share":"23.7%","token":"CCBot","purpose":"training","blocking":117,"operator":"Common Crawl"},{"share":"24.5%","token":"Bytespider","purpose":"training","blocking":121,"operator":"ByteDance"},{"share":"19.4%","token":"meta-externalagent","purpose":"training","blocking":96,"operator":"Meta"},{"share":"16.8%","token":"Amazonbot","purpose":"training","blocking":83,"operator":"Amazon"}],"method":"On 2026-11-12 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for the top 1000 domains of the Tranco list PY2NJ, following redirects, with a 12 second timeout and the user agent AEOFootnotesBot. A robots.txt counts when it answers 200 with a text body. A crawler counts as blocked when the robots.txt group that applies to it under RFC 9309 (its own user-agent group, else the * group) disallows / without allowing /. An llms.txt counts when it answers 200 with a body that starts with a markdown H1, as the proposal specifies. Shares are of the sites that serve a robots.txt.","population":1000,"withRobots":494,"collectedAt":"2026-11-12","withLlmsTxt":116}