
AI crawlers and robots.txt: allow AI search, block training
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explained, with copy-paste robots.txt rules to keep AI search and opt out of training.
Stories and guides about htaccess.
15 articles

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explained, with copy-paste robots.txt rules to keep AI search and opt out of training.

Critical error after an update? Collect a private WordPress log, interpret the fatal error carefully, isolate the suspected component and clean up safely.

Add HSTS, nosniff and Referrer-Policy on Apache, verify real public responses, understand subdomain scope and prepare a rollback that browsers can receive.

Create security.txt with a real reporting contact and expiry. Check the correct host, plain-text response and file contents, then keep the channel current.

Stage report-only CSP on Apache with a working collector; test violations and critical flows before enforcement, with rollback ready.

Set browser caching for static assets, keep HTML revalidating, and avoid the WordPress ?ver= trap with a private backup and server-specific checks.

Configure HTTP 503 maintenance with Retry-After, no-store caching, Apache/Nginx rules, Express middleware, a Cloudflare edge fallback and recovery checks.

Patch PHP first, then audit WordPress permissions and file-editor access. Keep wp-config backups private and apply server rules only after checking your setup.

Check whether Jetpack or a mobile app needs xmlrpc.php, then block or rate-limit it at the server and verify the response and dependent apps.
Explore hosting built for your next chapter.