Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

What's the state of the art in keeping scraping bots out of your webservers?

3 Posts 3 Posters 0 Views

Y This user is from outside of this forum
Y This user is from outside of this forum
CMDR Yojimbosan 🅅⁂

wrote last edited by

#1

What's the state of the art in keeping scraping bots out of your webservers?
Rate limiting, a restrictive robots.txt plus user agent matching, plus a proof-of-work page loader?
Is there a well-updated resource for the user agents?
T A 2 Replies Last reply

0
Y CMDR Yojimbosan 🅅⁂

What's the state of the art in keeping scraping bots out of your webservers?
Rate limiting, a restrictive robots.txt plus user agent matching, plus a proof-of-work page loader?
Is there a well-updated resource for the user agents?
T This user is from outside of this forum
T This user is from outside of this forum
Tekniquelly correct

wrote last edited by

#2

@yojimbo This is what I'm using: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.html
1 Reply Last reply

0
Y CMDR Yojimbosan 🅅⁂

What's the state of the art in keeping scraping bots out of your webservers?
Rate limiting, a restrictive robots.txt plus user agent matching, plus a proof-of-work page loader?
Is there a well-updated resource for the user agents?
A This user is from outside of this forum
A This user is from outside of this forum
Alex

wrote last edited by

#3

@yojimbo https://github.com/TecharoHQ/anubis
1 Reply Last reply
1
0
R ActivityRelay shared this topic

Log in to reply