11-07-2017 дата публикации
Номер: KR1020170080954A
Принадлежит:
The present invention relates to web crawler monitoring system and method, which monitor malfunctions of a static crawler of a web site and prevent damage caused by excessive traffic to a uniform resource locator (URL) server. The web crawler monitoring system comprises a first monitoring unit, a second monitoring unit, and database. The first monitoring unit delivers the URL to the crawler, stores the crawling process information (PID) of the URL and the crawling start time as a file, and when a predetermined crawling operation time elapses, forcibly terminates the crawling process, and then deletes the file and terminates the operation. The second monitoring unit reads the crawling process information (PID) and the crawling start time from the file stored by the first monitoring unit, and when the predetermined operation time elapses, forcibly terminates the crawling process and the operation of the first monitoring unit. The database stores the URL and the file. COPYRIGHT KIPO 2017 ( ...
Подробнее