robots.txt
The oldest and most widely honoured crawler convention, and the first place most well-behaved crawlers look. No file means no stated rules, and some crawlers treat that as a reason to stay away.
- What the test asks for
- We request
/robots.txton your domain over the network, exactly the way an outside client would, and read what comes back. A setting that is switched on but unreachable is not a pass. - Can QWeb Agent Ready add it for me?
- Yes. The free plugin publishes this. The plugin adds a robots.txt through WordPress, keeping whatever your SEO plugin already put there.
- How settled is this?
- A published standard.
Does your site pass this one?
The free test checks it against your live URL, along with 18 others, and tells you how many the free plugin can fix.