edu.uci.ics.crawler4j.fetcher

Class PageFetcher

java.lang.Object

edu.uci.ics.crawler4j.crawler.Configurable

edu.uci.ics.crawler4j.fetcher.PageFetcher

public class PageFetcher
extends Configurable

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.http.impl.conn.PoolingHttpClientConnectionManager	connectionManager
protected IdleConnectionMonitorThread	connectionMonitorThread
protected org.apache.http.impl.client.CloseableHttpClient	httpClient
protected long	lastFetchTime
protected static org.slf4j.Logger	logger
protected Object	mutex

Fields inherited from class edu.uci.ics.crawler4j.crawler.Configurable

config

Constructor Summary

Constructors

Constructor and Description
PageFetcher(CrawlConfig config)

Method Summary

Modifier and Type	Method and Description
PageFetchResult	fetchPage(WebURL webUrl)
protected org.apache.http.client.methods.HttpUriRequest	newHttpUriRequest(String url) Creates a new HttpUriRequest for the given url.
void	shutDown()

Methods inherited from class edu.uci.ics.crawler4j.crawler.Configurable

getConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

connectionManager

protected org.apache.http.impl.conn.PoolingHttpClientConnectionManager connectionManager

httpClient

protected org.apache.http.impl.client.CloseableHttpClient httpClient

mutex

protected final Object mutex

lastFetchTime

protected long lastFetchTime

connectionMonitorThread

protected IdleConnectionMonitorThread connectionMonitorThread

Constructor Detail

PageFetcher

public PageFetcher(CrawlConfig config)

Method Detail

fetchPage

public PageFetchResult fetchPage(WebURL webUrl)
                          throws InterruptedException,
                                 IOException,
                                 PageBiggerThanMaxSizeException

Throws:

InterruptedException

IOException

PageBiggerThanMaxSizeException

shutDown

public void shutDown()

newHttpUriRequest

protected org.apache.http.client.methods.HttpUriRequest newHttpUriRequest(String url)

Creates a new HttpUriRequest for the given url. The default is to create a HttpGet without any further configuration. Subclasses may override this method and provide their own logic.

Parameters:

url - the url to be fetched

Returns:

the HttpUriRequest for the given url