FAQ off Coding spurts
#6
It probably doesn't matter to you, but just in case....

both these solutions only 'ask' bots not to crawl 

It only works if you're talking about the main search engines. 

A private bot or scraper will just ignore the request. As I said, probably unnecessary for most sites as they won't be interested enough in your site to bother looking, but if you want to be ultra-careful, you should probably 'physically' stop bots with something like password protection.
Reply


Messages In This Thread
FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 09:06 AM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 12:27 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 01:11 PM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 02:21 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 03:07 PM
RE: FAQ off Coding spurts - by fuzzbox - 05-09-2024, 03:35 PM
RE: FAQ off Coding spurts - by Fulham Fallout - 05-09-2024, 03:47 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 05:13 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 07:07 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-10-2024, 07:22 AM

Forum Jump:


Users browsing this thread: 1 Guest(s)