識别User Agent屏蔽一些Web爬蟲防采集

　　自從做網站以來，大量自動抓取我們内容的爬蟲一直是個問題，防範采集是個長期任務，這篇是我5年前的博客文章：《Apache中設置屏蔽IP地址和URL網址來禁止采集》，另外，還可以識别User Agent來辨别和屏蔽一些采集者，在Apache中設置的代碼例子如下：

RewriteCond %{HTTP_USER_AGENT} ^(.*)(DTS\sAgent|Creative\sAutoUpdate|HTTrack|YisouSpider|SemrushBot)(.*)$
RewriteRule .* - [F,L]

　　屏蔽User Agent為空的代碼：

RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule .* - [F]

　　屏蔽Referer和User Agent都為空的代碼：

RewriteCond %{HTTP_REFERER} ^$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^$ [NC]
RewriteRule .* - [F] 

　　下面把一些可以屏蔽的常見采集軟件或者機器爬蟲的User Agent的特征關鍵詞列一下供參考：

• User-Agent
• DTS Agent
• HttpClient
• Owlin
• Kazehakase
• Creative AutoUpdate
• HTTrack
• YisouSpider
• baiduboxapp
• Python-urllib
• python-requests
• SemrushBot
• SearchmetricsBot
• MegaIndex
• Scrapy
• EMail Exractor
• 007ac9

• ltx71

　　其它也可以考慮屏蔽的：

• Mail.RU_Bot：http://go.mail.ru/help/robots
• Feedly
• ZumBot
• Pcore-HTTP
• Daum
• your-server
• Mobile/12A4345d
• PhantomJS/2.1.1
• archive.org_bot
• AcooBrowser
• Go-http-client
• Jakarta Commons-HttpClient
• Apache-HttpClient
• BDCbot
• ECCP
• Nutch
• cr4nk
• MJ12bot
• MOT-MPx220
• Y!OASIS/TEST
• libwww-perl

　　一般不要屏蔽的主流搜索引擎特征：

• Google
• Baidu
• Yahoo
• Slurp
• yandex
• YandexBot

• MSN

　　一些常見浏覽器或者通用代碼也不要輕易屏蔽：

• FireFox
• Apple
• PC
• Chrome
• Microsoft
• Android
• Mail
• Windows
• Mozilla
• Safar
• Macintosh

　　有的時候是采集者單獨設置的User Agent，也可以通過分析後進行屏蔽，例如：

RewriteCond %{HTTP_USER_AGENT} ^(.*)(\'Mozilla\/5\.0|\'Mozilla\'|\'Moz\'|\'Mozil\'|\'(.+)\'|Mobile\/13G34|Chrome\/53\.0\.2785\.143)(.*)$
RewriteRule .* - [F,L]

　　或者與HTTP_USER_AGENT一起考慮其它的因素再聯合判斷檢測、屏蔽，例如：

RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{HTTP_USER_AGENT} ^(.*)(Firefox\/44\.0|Safari\/537\.36)(.*)$
RewriteCond %{REQUEST_URI} ^(.*)\/comment\/reply\/(.*)$
RewriteRule .* - [F,L]

　　上面這是遇到反複POST提交留言的情況，判斷特征進行屏蔽。

　　網上也找了一些其它的代碼，列出供參考：

RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC]
RewriteRule ^(.*)$ - [F]

　　除了修改.htaccess文件以外，還可以通過修改httpd.conf配置文件來實現：

DocumentRoot /home/wwwroot/xxx
<Directory "/home/wwwroot/xxx">
SetEnvIfNoCase User-Agent ".*(FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)" BADBOT
        Order allow,deny
        Allow from all
       deny from env=BADBOT
</Directory>

　　這樣修改後需要重啟Apache。别人列出的需要屏蔽特征：

 

• FeedDemon             内容采集
• BOT/0.1 (BOT for JCE) sql注入
• CrawlDaddy            sql注入
• Java                  内容采集
• Jullo                 内容采集
• Feedly                内容采集
• UniversalFeedParser   内容采集
• ApacheBench           cc攻擊器
• Swiftbot              無用爬蟲
• YandexBot             無用爬蟲
• AhrefsBot             無用爬蟲
• YisouSpider           無用爬蟲（已被UC神馬搜索收購，此蜘蛛可以放開！）
• MJ12bot               無用爬蟲
• ZmEu phpmyadmin       漏洞掃描
• WinHttp               采集cc攻擊
• EasouSpider           無用爬蟲
• HttpClient            tcp攻擊
• Microsoft URL Control 掃描
• YYSpider              無用爬蟲
• jaunty                wordpress爆破掃描器
• oBot                  無用爬蟲
• Python-urllib         内容采集
• Indy Library          掃描
• FlightDeckReports Bot 無用爬蟲
• Linguee Bot           無用爬蟲
   

　　繼續補充：

WinHttp|WebZIP|FetchURL|node-superagent|java/|FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT/0.1|YandexBot|FlightDeckReports|Linguee Bot

　　還有：

Aboundex
80legs
^Java
^Cogentbot
^Alexibot
^asterias
^attach
^BackDoorBot
^BackWeb
Bandit
^BatchFTP
^Bigfoot
^Black.Hole
^BlackWidow
^BlowFish
^BotALot
Buddy
^BuiltBotTough
^Bullseye
^BunnySlippers
^Cegbfeieh
^CheeseBot
^CherryPicker
^ChinaClaw
Collector
Copier
^CopyRightCheck
^cosmos
^Crescent
^Custo
^AIBOT
^DISCo
^DIIbot
^DittoSpyder
^Download\ Demon
^Download\ Devil
^Download\ Wonder
^dragonfly
^Drip
^eCatch
^EasyDL
^ebingbong
^EirGrabber
^EmailCollector
^EmailSiphon
^EmailWolf
^EroCrawler
^Exabot
^Express\ WebPictures
Extractor
^EyeNetIE
^Foobot
^flunky
^FrontPage
^Go-Ahead-Got-It
^gotit
^GrabNet
^Grafula
^Harvest
^hloader
^HMView
^HTTrack
^humanlinks
^IlseBot
^Image\ Stripper
^Image\ Sucker
Indy\ Library
^InfoNaviRobot
^InfoTekies
^Intelliseek
^InterGET
^Internet\ Ninja
^Iria
^Jakarta
^JennyBot
^JetCar
^JOC
^JustView
^Jyxobot
^Kenjin.Spider
^Keyword.Density
^larbin
^LexiBot
^lftp
^libWeb/clsHTTP
^likse
^LinkextractorPro
^LinkScan/8.1a.Unix
^LNSpiderguy
^LinkWalker
^lwp-trivial
^LWP::Simple
^Magnet
^Mag-Net
^MarkWatch
^Mass\ Downloader
^Mata.Hari
^Memo
^Microsoft.URL
^Microsoft\ URL\ Control
^MIDown\ tool
^MIIxpc
^Mirror
^Missigua\ Locator
^Mister\ PiX
^moget
^Mozilla/3.Mozilla/2.01
^Mozilla.*NEWT
^NAMEPROTECT
^Navroad
^NearSite
^NetAnts
^Netcraft
^NetMechanic
^NetSpider
^Net\ Vampire
^NetZIP
^NextGenSearchBot
^NG
^NICErsPRO
^niki-bot
^NimbleCrawler
^Ninja
^NPbot
^Octopus
^Offline\ Explorer
^Offline\ Navigator
^Openfind
^OutfoxBot
^PageGrabber
^Papa\ Foto
^pavuk
^pcBrowser
^PHP\ version\ tracker
^Pockey
^ProPowerBot/2.14
^ProWebWalker
^psbot
^Pump
^QueryN.Metasearch
^RealDownload
Reaper
Recorder
^ReGet
^RepoMonkey
^RMA
Siphon
^SiteSnagger
^SlySearch
^SmartDownload
^Snake
^Snapbot
^Snoopy
^sogou
^SpaceBison
^SpankBot
^spanner
^Sqworm
Stripper
Sucker
^SuperBot
^SuperHTTP
^Surfbot
^suzuran
^Szukacz/1.4
^tAkeOut
^Teleport
^Telesoft
^TurnitinBot/1.5
^The.Intraformant
^TheNomad
^TightTwatBot
^Titan
^True_Robot
^turingos
^TurnitinBot
^URLy.Warning
^Vacuum
^VCI
^VoidEYE
^Web\ Image\ Collector
^Web\ Sucker
^WebAuto
^WebBandit
^Webclipping.com
^WebCopier
^WebEMailExtrac.*
^WebEnhancer
^WebFetch
^WebGo\ IS
^Web.Image.Collector
^WebLeacher
^WebmasterWorldForumBot
^WebReaper
^WebSauger
^Website\ eXtractor
^Website\ Quester
^Webster
^WebStripper
^WebWhacker
^WebZIP
Whacker
^Widow
^WISENutbot
^WWWOFFLE
^WWW-Collector-E
^Xaldon
^Xenu
^Zeus
ZmEu
^Zyborg
Acunetix
FHscan

 

　　臨時屏蔽（返回503錯誤），而不是長期屏蔽的代碼：

RewriteCond %{HTTP_USER_AGENT} ^.*(bot|crawl|spider).*$ [NC]
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* - [R=503,L]

 

 

 

• 發表新回應

评论1

今天屏蔽的如下：

RewriteCond %{HTTP_USER_AGENT} ^(.*)(User-Agent|DTS\sAgent|HttpClient|Owlin|Kazehakase|Creative\sAutoUpdate|HTTrack|YisouSpider|Python-urllib|python-requests|SemrushBot|SearchmetricsBot|MegaIndex|Scrapy|EMail\sExractor|007ac9|ltx71|Feedly|ZumBot|Pcore-HTTP|Daum|Mobile\/12A4345d|PhantomJS\/2\.1\.1|archive\.org_bot|AcooBrowser|Go-http-client|Jakarta\sCommons-HttpClient|Apache-HttpClient|BDCbot|Nutch|cr4nk|MJ12bot|MOT-MPx220|Y!OASIS\/TEST|libwww-perl|Indy\sLibrary|Alexa\sToolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|UniversalFeedParser|ApacheBench|Microsoft\sURL\sControl|Swiftbot|ZmEu|oBot|jaunty|lightDeckReports\sBot|YYSpider|DigExt|heritrix|EasouSpider|Ezooms|FeedDemon|BOT\sfor\sJCE|Jullo|UniversalFeedParser|WinHttp|FlightDeckReports|Linguee\sBot|JikeSpider|node-superagent|WebZIP|FetchURL|Apache-HttpAsyncClient|Aboundex|80legs|Cogentbot|Alexibot|asterias|BackDoorBot|BackWeb|Bandit|BatchFTP|Bigfoot|Black\.Hole|BlackWidow|BlowFish|BotALot|BuiltBotTough|Bullseye|BunnySlippers|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Collector|Copier|CopyRightCheck|cosmos|Crescent|Custo|AIBOT|DISCo|DIIbot|DittoSpyder|Download\sDemon|Download\sDevil|Download\sWonder|dragonfly|Drip|eCatch|EasyDL|ebingbong|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|Express\sWebPictures|Extractor|EyeNetIE|Foobot|flunky|FrontPage|Go-Ahead-Got-It|gotit|GrabNet|Grafula|Harvest|hloader|HMView|humanlinks|IlseBot|Image\sStripper|Image\sSucker|InfoNaviRobot|InfoTekies|Intelliseek|InterGET|Internet\sNinja|Iria|Jakarta|JennyBot|JetCar|JOC|JustView|Jyxobot|Kenjin\.Spider|Keyword\.Density|larbin|LexiBot|lftp|libWeb\/clsHTTP|likse|LinkextractorPro|LinkScan|LNSpiderguy|LinkWalker|lwp-trivial|LWP::Simple|Magnet|Mag-Net|MarkWatch|Mass\sDownloader|Mata\.Hari|Memo|Microsoft\.URL|Microsoft\sURL\sControl|MIDown\stool|MIIxpc|Mirror|Missigua\sLocator|Mister\sPiX|moget|NAMEPROTECT|Navroad|NearSite|NetAnts|Netcraft|NetMechanic|NetSpider|Net\sVampire|NetZIP|NextGenSearchBot|NICErsPRO|niki-bot|NimbleCrawler|NPbot|Octopus|Offline\sExplorer|Offline\sNavigator|Openfind|OutfoxBot|PageGrabber|Papa\sFoto|pavuk|PHP\sversion\stracker|Pockey|ProPowerBot\/2\.14|ProWebWalker|psbot|Pump|QueryN\.Metasearch|RealDownload|Reaper|Recorder|ReGet|RepoMonkey|RMA|Siphon|SiteSnagger|SlySearch|SmartDownload|Snake|Snapbot|Snoopy|SpaceBison|SpankBot|spanner|Sqworm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|suzuran|Szukacz\/1\.4|tAkeOut|Teleport|Telesoft|TurnitinBot\/1\.5|The\.Intraformant|TheNomad|TightTwatBot|Titan|True_Robot|turingos|TurnitinBot|URLy\.Warning|Vacuum|VCI|VoidEYE|Web\sImage\sCollector|Web\sSucker|WebAuto|WebBandit|Webclipping\.com|WebCopier|WebEMailExtrac|WebEnhancer|WebFetch|WebGo\sIS|Web\.Image\.Collector|WebLeacher|WebmasterWorldForumBot|WebReaper|WebSauger|Website\seXtractor|Website\sQuester|Webster|WebStripper|WebWhacker|Whacker|Widow|WISENutbot|WWWOFFLE|WWW-Collector-E|Xaldon|Xenu|Zeus|Zyborg|Acunetix|FHscan)(.*)$
RewriteRule .* - [F,L]

 

• 回覆

您的名字

评论

關於文字格式

Plain text

• 不允許使用 HTML 標籤。
• 網頁和電子郵件地址自動轉換為連結。
• 自動斷行和分段。

CAPTCHA

此问题用于测试您是否是人类访问者并防止自动提交垃圾信息。