WBAUnofficial
FAQ off Coding spurts - Printable Version

+- WBAUnofficial (https://wbaunofficial.org.uk)
+-- Forum: WBAUnofficial (https://wbaunofficial.org.uk/forumdisplay.php?fid=3)
+--- Forum: WBAUnofficial (https://wbaunofficial.org.uk/forumdisplay.php?fid=4)
+--- Thread: FAQ off Coding spurts (/showthread.php?tid=33646)



FAQ off Coding spurts - CarlosCorbewrong - 05-09-2024

How do you make a webpage invisible to search bots please?


RE: FAQ off Coding spurts - waffle - 05-09-2024

<meta name="robots" content=noindex, nofollow />

In the <head> section of the html


RE: FAQ off Coding spurts - CarlosCorbewrong - 05-09-2024

(05-09-2024, 12:27 PM)waffle Wrote: <meta name="robots" content=noindex, nofollow />

In the <head> section of the html

That simple?  And it covers the major search engines? Cheers!


RE: FAQ off Coding spurts - waffle - 05-09-2024

(05-09-2024, 01:11 PM)CarlosCorbewrong Wrote:
(05-09-2024, 12:27 PM)waffle Wrote: <meta name="robots" content=noindex, nofollow />

In the <head> section of the html

That simple?  And it covers the major search engines? Cheers!

Yep - https://developers.google.com/search/docs/crawling-indexing/block-indexing


RE: FAQ off Coding spurts - MrBater - 05-09-2024

You can also use a robots.txt and in it list the URLs the robots aren't allowed to scrape. E.g. disallow /URL but it only prevents a page from being crawled if it hasn't already been crawled.... Might be easier if it's new pages... No good if existing ones that gave been crawled previously


RE: FAQ off Coding spurts - fuzzbox - 05-09-2024

It probably doesn't matter to you, but just in case....

both these solutions only 'ask' bots not to crawl 

It only works if you're talking about the main search engines. 

A private bot or scraper will just ignore the request. As I said, probably unnecessary for most sites as they won't be interested enough in your site to bother looking, but if you want to be ultra-careful, you should probably 'physically' stop bots with something like password protection.


RE: FAQ off Coding spurts - Fulham Fallout - 05-09-2024

(05-09-2024, 09:06 AM)CarlosCorbewrong Wrote: How do you make a webpage invisible to search bots please?

You could have entered the same sentence with one less word (please) in to Google and got the answer immediately.

Just saying loike Big Grin


RE: FAQ off Coding spurts - CarlosCorbewrong - 05-09-2024

(05-09-2024, 03:47 PM)Fulham Fallout Wrote:
(05-09-2024, 09:06 AM)CarlosCorbewrong Wrote: How do you make a webpage invisible to search bots please?

You could have entered the same sentence with one less word (please) in to Google and got the answer immediately.

Just saying loike Big Grin

And you could have logged off and shut the fuck up.

(05-09-2024, 03:35 PM)fuzzbox Wrote: It probably doesn't matter to you, but just in case....

both these solutions only 'ask' bots not to crawl 

It only works if you're talking about the main search engines. 

A private bot or scraper will just ignore the request. As I said, probably unnecessary for most sites as they won't be interested enough in your site to bother looking, but if you want to be ultra-careful, you should probably 'physically' stop bots with something like password protection.

Cheers. I just don't want google to point people to my flag sharing messageboard so the coding above should be OK I think


RE: FAQ off Coding spurts - MrBater - 05-09-2024

Flagshaggers.co.uk ??


RE: FAQ off Coding spurts - CarlosCorbewrong - 05-10-2024

(05-09-2024, 07:07 PM)MrBater Wrote: Flagshaggers.co.uk ??

Flagshaggers.co.uk/secretboard