edu.uci.ics.crawler4j.crawler

Class WebCrawler

java.lang.Object

edu.uci.ics.crawler4j.crawler.WebCrawler

All Implemented Interfaces:

Runnable

public class WebCrawler
extends Object
implements Runnable

WebCrawler class in the Runnable class that is executed by each crawler thread.

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected static uk.org.lidalia.slf4jext.Logger	logger
protected CrawlController	myController The controller instance that has created this crawler thread.
protected int	myId The id associated to the crawler thread running this instance

Constructor Summary

Constructors

Constructor and Description
WebCrawler()

Method Summary

Modifier and Type	Method and Description
CrawlController	getMyController()
int	getMyId() Get the id of the current crawler instance
Object	getMyLocalData() The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread.
Thread	getThread()
protected void	handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) This function is called once the header of a page is fetched.
protected WebURL	handleUrlBeforeProcess(WebURL curURL) This function is called before processing of the page's URL It can be overridden by subclasses for tweaking of the url before processing it.
void	init(int id, CrawlController crawlController) Initializes the current instance of the crawler
boolean	isNotWaitingForNewURLs()
void	onBeforeExit() This function is called just before the termination of the current crawler instance.
protected void	onContentFetchError(WebURL webUrl) This function is called if the content of a url could not be fetched.
protected void	onPageBiggerThanMaxSize(String urlStr, long pageSize) This function is called if the content of a url is bigger than allowed size.
protected void	onParseError(WebURL webUrl) This function is called if there has been an error in parsing the content.
void	onStart() This function is called just before starting the crawl by this crawler instance.
protected void	onUnexpectedStatusCode(String urlStr, int statusCode, String contentType, String description) This function is called if the crawler encountered an unexpected http status code ( a status code other than 3xx)
protected void	onUnhandledException(WebURL webUrl, Throwable e) This function is called when a unhandled exception was encountered during fetching
void	run()
void	setThread(Thread myThread)
boolean	shouldVisit(Page referringPage, WebURL url) Classes that extends WebCrawler should overwrite this function to tell the crawler whether the given url should be crawled or not.
void	visit(Page page) Classes that extends WebCrawler should overwrite this function to process the content of the fetched and parsed page.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final uk.org.lidalia.slf4jext.Logger logger

myId

protected int myId

The id associated to the crawler thread running this instance

myController

protected CrawlController myController

The controller instance that has created this crawler thread. This reference to the controller can be used for getting configurations of the current crawl or adding new seeds during runtime.

Constructor Detail

WebCrawler

public WebCrawler()

Method Detail

init

public void init(int id,
                 CrawlController crawlController)

Initializes the current instance of the crawler

Parameters:

id - the id of this crawler instance

crawlController - the controller that manages this crawling session

getMyId

public int getMyId()

Get the id of the current crawler instance

Returns:

the id of the current crawler instance

getMyController

public CrawlController getMyController()

onStart

public void onStart()

This function is called just before starting the crawl by this crawler instance. It can be used for setting up the data structures or initializations needed by this crawler instance.

onBeforeExit

public void onBeforeExit()

This function is called just before the termination of the current crawler instance. It can be used for persisting in-memory data or other finalization tasks.

handlePageStatusCode

protected void handlePageStatusCode(WebURL webUrl,
                                    int statusCode,
                                    String statusDescription)

This function is called once the header of a page is fetched. It can be overridden by sub-classes to perform custom logic for different status codes. For example, 404 pages can be logged, etc.

Parameters:

webUrl - WebUrl containing the statusCode

statusCode - Html Status Code number

statusDescription - Html Status COde description

handleUrlBeforeProcess

protected WebURL handleUrlBeforeProcess(WebURL curURL)

This function is called before processing of the page's URL It can be overridden by subclasses for tweaking of the url before processing it. For example, http://abc.com/def?a=123 - http://abc.com/def

Parameters:

curURL - current URL which can be tweaked before processing

Returns:

tweaked WebURL

onPageBiggerThanMaxSize

protected void onPageBiggerThanMaxSize(String urlStr,
                                       long pageSize)

This function is called if the content of a url is bigger than allowed size.

Parameters:

urlStr - - The URL which it's content is bigger than allowed size

onUnexpectedStatusCode

protected void onUnexpectedStatusCode(String urlStr,
                                      int statusCode,
                                      String contentType,
                                      String description)

This function is called if the crawler encountered an unexpected http status code ( a status code other than 3xx)

Parameters:

urlStr - URL in which an unexpected error was encountered while crawling

statusCode - Html StatusCode

contentType - Type of Content

description - Error Description

onContentFetchError

protected void onContentFetchError(WebURL webUrl)

This function is called if the content of a url could not be fetched.

Parameters:

webUrl - URL which content failed to be fetched

onUnhandledException

protected void onUnhandledException(WebURL webUrl,
                                    Throwable e)

This function is called when a unhandled exception was encountered during fetching

Parameters:

webUrl - URL where a unhandled exception occured

onParseError

protected void onParseError(WebURL webUrl)

This function is called if there has been an error in parsing the content.

Parameters:

webUrl - URL which failed on parsing

getMyLocalData

public Object getMyLocalData()

The CrawlController instance that has created this crawler instance will call this function just before terminating this crawler thread. Classes that extend WebCrawler can override this function to pass their local data to their controller. The controller then puts these local data in a List that can then be used for processing the local data of crawlers (if needed).

Returns:

currently NULL

run

public void run()

Specified by:

run in interface Runnable

shouldVisit

public boolean shouldVisit(Page referringPage,
                           WebURL url)

Classes that extends WebCrawler should overwrite this function to tell the crawler whether the given url should be crawled or not. The following default implementation indicates that all urls should be included in the crawl.

Parameters:

url - the url which we are interested to know whether it should be included in the crawl or not.

referringPage - The Page in which this url was found.

Returns:

if the url should be included in the crawl it returns true, otherwise false is returned.

visit

public void visit(Page page)

Classes that extends WebCrawler should overwrite this function to process the content of the fetched and parsed page.

Parameters:

page - the page object that is just fetched and parsed.

getThread

public Thread getThread()

setThread

public void setThread(Thread myThread)

isNotWaitingForNewURLs

public boolean isNotWaitingForNewURLs()