wget – ignore robots.txt rules when grabbing site files

Simply do:

wget -r -e robots=off --wait 0.25 http://thesite.com




No Comments


You can leave the first : )



Leave a Reply

Your email address will not be published. Required fields are marked *