wget – ignore robots.txt rules when grabbing site files September 14, 2013 Stuart Leave a comment Simply do: wget -r -e robots=off --wait 0.25 http://thesite.com