wget – ignore robots.txt rules when grabbing site files

Simply do:

wget -r -e robots=off --wait 0.25 http://thesite.com




No Comments


No comments yet.



Leave a Reply

Your email address will not be published. Required fields are marked *