苹果cms降低服务器负载教程

robots规则并不能防止垃圾蜘蛛爬站，因此建议略微放宽ua黑名单后，自行添加robots规则，当然小白不需要谷歌/bing蜘蛛带来的流量的话可以选择一刀切。（必应需要手动提交网址后才能收录，相对麻烦，我也没提交过收录所以直接切了必应，如有误杀请及时移除！）

第一步：修改或覆盖robots.txt内的规则

User/agent: *

Disallow: /cache

Disallow: /images

Disallow: /inc

Disallow: /js

Disallow: /player

Disallow: /template

Disallow: /a

Disallow: /.

Disallow: /vod/play

Allow: /vod/type/id/*.html$

Allow: /vod/type/id/*/pg/*.html$

Allow: /vod/detail/id/*.html$

Allow: /vod/search$

Allow: /vod/list/id/*/pg/*/order/*/by/*/class/*/year/*/letter/*/area/*/lang/.html$

Allow: /gbook/index.html$

Sitemap: /rss/index.xml$

第二步：添加宝塔ua黑名单规则

["(AhrefsBot|GoogleBot|aliyun|bingbot|crawler|CipaCrawler|commoncrawl|Digital AlphaServer|DomainCrawler|DotBot|dacongyun|daum|Epiphany|feedly|Go-http-client|GroceryHouse|greatdealshop|http-client|KOCMOHABT|ltx71|inoreader.com|msnbot|Miniflux|MJ12bot|magpie-crawler|mail.ru|Media Center PC 6.0|MSIE 6.0|MegaIndex.ru|Nimbostratus-Bot|Nexus 7 BuildNimbostratus|opensiteexplorer|pingbot|PhantomJS|Python-urllib|python-requests|python|php-market|Qwantify|rssbot|Scrapy|SemrushBot|subscribers|t.me|uptime|WinHttp|x09Chrome|YandexBot|zgrab|9.1.0.0 Safari|17.0.963.56|34.0.1847.116|41.0.2227.1|45.0.2454.93|57.0.2987.133 Safari|59.0.3071.115 Safari|535.11|20101213)"]