You can also use a robots.txt and in it list the URLs the robots aren't allowed to scrape. E.g. disallow /URL but it only prevents a page from being crawled if it hasn't already been crawled.... Might be easier if it's new pages... No good if existing ones that gave been crawled previously
|
FAQ off Coding spurts
|
|
« Next Oldest | Next Newest »
|
| Messages In This Thread |
|
FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 09:06 AM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 12:27 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 01:11 PM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 02:21 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 03:07 PM
RE: FAQ off Coding spurts - by fuzzbox - 05-09-2024, 03:35 PM
RE: FAQ off Coding spurts - by Fulham Fallout - 05-09-2024, 03:47 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 05:13 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 07:07 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-10-2024, 07:22 AM
|
Users browsing this thread:

