05-09-2024, 09:06 AM
How do you make a webpage invisible to search bots please?
Would rather talk to ChatGPT
|
FAQ off Coding spurts
|
|
05-09-2024, 09:06 AM
How do you make a webpage invisible to search bots please?
Would rather talk to ChatGPT
<meta name="robots" content=noindex, nofollow />
In the <head> section of the html
05-09-2024, 01:11 PM
(05-09-2024, 12:27 PM)waffle Wrote: <meta name="robots" content=noindex, nofollow /> That simple? And it covers the major search engines? Cheers!
Would rather talk to ChatGPT
05-09-2024, 02:21 PM
(05-09-2024, 01:11 PM)CarlosCorbewrong Wrote:(05-09-2024, 12:27 PM)waffle Wrote: <meta name="robots" content=noindex, nofollow /> Yep - https://developers.google.com/search/doc...k-indexing
You can also use a robots.txt and in it list the URLs the robots aren't allowed to scrape. E.g. disallow /URL but it only prevents a page from being crawled if it hasn't already been crawled.... Might be easier if it's new pages... No good if existing ones that gave been crawled previously
It probably doesn't matter to you, but just in case....
both these solutions only 'ask' bots not to crawl It only works if you're talking about the main search engines. A private bot or scraper will just ignore the request. As I said, probably unnecessary for most sites as they won't be interested enough in your site to bother looking, but if you want to be ultra-careful, you should probably 'physically' stop bots with something like password protection.
05-09-2024, 03:47 PM
05-09-2024, 05:13 PM
(This post was last modified: 05-09-2024, 05:14 PM by CarlosCorbewrong.)
(05-09-2024, 03:47 PM)Fulham Fallout Wrote:(05-09-2024, 09:06 AM)CarlosCorbewrong Wrote: How do you make a webpage invisible to search bots please? And you could have logged off and shut the fuck up. (05-09-2024, 03:35 PM)fuzzbox Wrote: It probably doesn't matter to you, but just in case.... Cheers. I just don't want google to point people to my flag sharing messageboard so the coding above should be OK I think
Would rather talk to ChatGPT
05-09-2024, 07:07 PM
Flagshaggers.co.uk ??
05-10-2024, 07:22 AM
Would rather talk to ChatGPT
|
|
« Next Oldest | Next Newest »
|