# robots.txt for www.onevalor.com - AI & MCP Optimized # Last updated: 07/29/2025 # Optimized for AI search engines, crawlers, and Model Context Protocol (MCP) servers # ======================================== # HOST DIRECTIVE (ONLY ONE ALLOWED) # ======================================== Host: www.onevalor.com # ======================================== # COMPREHENSIVE SITEMAP (All content in one file) # ======================================== Sitemap: https://www.onevalor.com/sitemap.xml # ======================================== # BLOCK EXTERNAL PDFS & TRACKING LINKS # ======================================== # Block tracking links to external PDFs Disallow: /l/1055883/ # Block PDF files from indexing Disallow: *.pdf # ======================================== # DEFAULT USER AGENT (ALL BOTS) # ======================================== User-agent: * Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # Block incomplete image asset paths that are being crawled as pages Disallow: /assets/images/valor-operator-services Disallow: /assets/images/Best Disallow: /assets/images/Inc. # Block malformed mineral buyer URLs with double dashes Disallow: /mineral-buyers/*--* Crawl-delay: 1 # ======================================== # GOOGLE SEARCH ENGINES # ======================================== User-agent: Googlebot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # Block incomplete image asset paths that are being crawled as pages Disallow: /assets/images/valor-operator-services Disallow: /assets/images/Best Disallow: /assets/images/Inc. # Block malformed mineral buyer URLs with double dashes Disallow: /mineral-buyers/*--* Crawl-delay: 1 User-agent: Googlebot-Image Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: Googlebot-News Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: Googlebot-Video Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: Google-Extended Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # MICROSOFT BING SEARCH ENGINES # ======================================== User-agent: Bingbot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: msnbot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: BingPreview Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # OPENAI & CHATGPT BOTS # ======================================== User-agent: GPTBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: ChatGPT-User Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: OpenAI-SearchBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # ANTHROPIC CLAUDE BOTS # ======================================== User-agent: Claude-Web Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: ClaudeBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: Anthropic-AI Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # META AI BOTS # ======================================== User-agent: Meta-ExternalAgent Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: FacebookBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: Meta-ExternalFetcher Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # PERPLEXITY AI BOTS # ======================================== User-agent: PerplexityBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: Perplexity-AI Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # BRAVE SEARCH BOTS # ======================================== User-agent: brave-search Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: BraveBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 # ======================================== # YANDEX SEARCH BOTS # ======================================== User-agent: YandexBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: YandexImages Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # DUCKDUCKGO BOTS # ======================================== User-agent: DuckDuckBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: DuckDuckGo-Favicons-Bot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # BAIDU SEARCH BOTS # ======================================== User-agent: Baiduspider Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 3 User-agent: Baiduspider-image Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # APPLE SIRI & SPOTLIGHT # ======================================== User-agent: Applebot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: AppleBot-Extended Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # AMAZON ALEXA BOTS # ======================================== User-agent: ia_archiver Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: AlexaBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # MICROSOFT COPILOT & EDGE # ======================================== User-agent: Microsoft-CopilotBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: EdgeBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # AI BOTS (COHERE, HUGGING FACE, ETC.) # ======================================== User-agent: cohere-ai Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: CohereBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: HuggingFaceBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: StabilityAI Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 User-agent: MistralBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 2 # ======================================== # MCP SERVERS & DEVELOPMENT TOOLS # ======================================== User-agent: mcp-server Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: mcp-client Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: firecrawl Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 User-agent: tavily Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 1 # ======================================== # SEO TOOLS & SEMRUSH # ======================================== User-agent: SemrushBot-SA Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 3 User-agent: SemrushBot-SI Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 3 User-agent: AhrefsBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 3 User-agent: MJ12bot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 10 # ======================================== # SOCIAL MEDIA CRAWLERS # ======================================== User-agent: Twitterbot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: LinkedInBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: WhatsApp Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* User-agent: TelegramBot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* # ======================================== # ARCHIVE & RESEARCH BOTS # ======================================== User-agent: archive.org_bot Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 5 User-agent: Wayback Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 5 User-agent: CommonCrawl Allow: / Disallow: /get-started/ Disallow: /intelligence/ Disallow: /?cst&query* Disallow: /?query* Disallow: /county-clerk/oklahoma-old/ Disallow: /county-clerk/texas-old/ Disallow: /admin/ Disallow: /farm-ranch/ Disallow: /private/ Disallow: /temp/ Disallow: /*.json$ Disallow: /api/ Disallow: /search? Disallow: /*?utm_* Disallow: /*?ref=* Disallow: /*?source=* Disallow: /*?campaign=* Crawl-delay: 10 # ======================================== # BLOCKED BOTS (AGGRESSIVE/UNWANTED) # ======================================== User-agent: DotBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: BLEXBot Disallow: / User-agent: SiteAuditBot Disallow: / # ======================================== # AI & LLM DISCOVERY # ======================================== # LLMs.txt file for AI systems and chatbots # Provides structured company information for enhanced AI responses # Available at: https://www.onevalor.com/llms.txt # ======================================== # END OF ROBOTS.TXT # ========================================