标签：post min net robot ref pos body txt gpo
scrapy爬虫出现Forbidden by robots.txt
原文地址：http://www.cnblogs.com/hfultrastrong/p/8040310.html
