For website owners
CostaYGolfBot, the Costa y Golf crawler
If you found this page in your server logs, our crawler has visited your website. We read golf course websites to prepare their guide pages: layouts, green fees, facilities and directions. What we publish is always our own writing, checked by hand; we never copy your text.
How it identifies itself
Every request carries this User-Agent header:
CostaYGolfBot/1.0 (+https://costaygolf.com/bot)How it behaves
- It follows robots.txt and stays out of whatever you disallow.
- It only reads the domain it was asked to visit and never follows links to other sites.
- It waits 1.5 seconds between two requests to the same server.
- It stops reading a page once it reaches its size limit and does not follow links that repeat endlessly, such as infinite calendars or looping paths.
How to block it
Add these lines to your robots.txt and it will not read your website again:
User-agent: CostaYGolfBot
Disallow: /Contact
If the crawler causes you any trouble, or you would like us to correct something on your course's page, write to us at bot@costaygolf.com.
bot@costaygolf.com