wget robots.txt (1)


wget – ignore robots.txt rules when grabbing site files

Simply do: wget -r -e robots=off –wait 0.25 http://thesite.com

Read more...