FAQ off Coding spurts
#5
You can also use a robots.txt and in it list the URLs the robots aren't allowed to scrape. E.g. disallow /URL but it only prevents a page from being crawled if it hasn't already been crawled.... Might be easier if it's new pages... No good if existing ones that gave been crawled previously
Reply


Messages In This Thread
FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 09:06 AM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 12:27 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 01:11 PM
RE: FAQ off Coding spurts - by waffle - 05-09-2024, 02:21 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 03:07 PM
RE: FAQ off Coding spurts - by fuzzbox - 05-09-2024, 03:35 PM
RE: FAQ off Coding spurts - by Fulham Fallout - 05-09-2024, 03:47 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-09-2024, 05:13 PM
RE: FAQ off Coding spurts - by MrBater - 05-09-2024, 07:07 PM
RE: FAQ off Coding spurts - by CarlosCorbewrong - 05-10-2024, 07:22 AM

Forum Jump:


Users browsing this thread: