1.配置
http://www.cnblogs.com/likai198981/archive/2013/01/11/2856902.html
2.扩展
http://www.ibm.com/developerworks/cn/opensource/os-cn-heritrix/
3.elfhash
http://hi.baidu.com/prince_hyai/item/ce9256d111cc3b3d2b35c7c6
http://biancheng.dnbcw.info/1000wen/405339.html
4.java
http://huangwen200301930.blog.163.com/blog/static/484849812011111142257943/
5.robot.txt
http://neilone-cn.iteye.com/blog/761617