CrawlConfig (crawler4j 4.2 API)

java.lang.Object
- edu.uci.ics.crawler4j.crawler.CrawlConfig

```
public class CrawlConfig
extends Object
```

Constructor Summary

Constructors
Constructor and Description

CrawlConfig()

Constructors
Constructor and Description
`CrawlConfig()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addAuthInfo(AuthInfo authInfo)`
`List<AuthInfo>`	`getAuthInfos()`
`int`	`getConnectionTimeout()`
`String`	`getCrawlStorageFolder()`
`Collection<org.apache.http.message.BasicHeader>`	`getDefaultHeaders()` Return a copy of the default header collection.
`int`	`getMaxConnectionsPerHost()`
`int`	`getMaxDepthOfCrawling()`
`int`	`getMaxDownloadSize()`
`int`	`getMaxOutgoingLinksToFollow()`
`int`	`getMaxPagesToFetch()`
`int`	`getMaxTotalConnections()`
`int`	`getPolitenessDelay()`
`String`	`getProxyHost()`
`String`	`getProxyPassword()`
`int`	`getProxyPort()`
`String`	`getProxyUsername()`
`int`	`getSocketTimeout()`
`String`	`getUserAgentString()`
`boolean`	`isFollowRedirects()`
`boolean`	`isIncludeBinaryContentInCrawling()`
`boolean`	`isIncludeHttpsPages()`
`boolean`	`isOnlineTldListUpdate()`
`boolean`	`isProcessBinaryContentInCrawling()`
`boolean`	`isResumableCrawling()`
`boolean`	`isShutdownOnEmptyQueue()`
`void`	`setAuthInfos(List<AuthInfo> authInfos)`
`void`	`setConnectionTimeout(int connectionTimeout)`
`void`	`setCrawlStorageFolder(String crawlStorageFolder)` The folder which will be used by crawler for storing the intermediate crawl data.
`void`	`setDefaultHeaders(Collection<? extends org.apache.http.Header> defaultHeaders)` Set the default header collection (creating copies of the provided headers).
`void`	`setFollowRedirects(boolean followRedirects)`
`void`	`setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)`
`void`	`setIncludeHttpsPages(boolean includeHttpsPages)`
`void`	`setMaxConnectionsPerHost(int maxConnectionsPerHost)`
`void`	`setMaxDepthOfCrawling(int maxDepthOfCrawling)` Maximum depth of crawling For unlimited depth this parameter should be set to -1
`void`	`setMaxDownloadSize(int maxDownloadSize)`
`void`	`setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)`
`void`	`setMaxPagesToFetch(int maxPagesToFetch)` Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
`void`	`setMaxTotalConnections(int maxTotalConnections)`
`void`	`setOnlineTldListUpdate(boolean online)` Should the TLD list be updated automatically on each run? Alternatively, it can be loaded from the embedded tld-names.txt resource file that was obtained from https://publicsuffix.org/list/effective_tld_names.dat
`void`	`setPolitenessDelay(int politenessDelay)` Politeness delay in milliseconds (delay between sending two requests to the same host).
`void`	`setProcessBinaryContentInCrawling(boolean processBinaryContentInCrawling)` Should we process binary content such as images, audio, ...
`void`	`setProxyHost(String proxyHost)`
`void`	`setProxyPassword(String proxyPassword)` If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
`void`	`setProxyPort(int proxyPort)`
`void`	`setProxyUsername(String proxyUsername)`
`void`	`setResumableCrawling(boolean resumableCrawling)` If this feature is enabled, you would be able to resume a previously stopped/crashed crawl.
`void`	`setShutdownOnEmptyQueue(boolean shutdown)` Should the crawler stop running when the queue is empty?
`void`	`setSocketTimeout(int socketTimeout)`
`void`	`setUserAgentString(String userAgentString)` user-agent string that is used for representing your crawler to web servers.
`String`	`toString()`
`void`	`validate()` Validates the configs specified by this instance.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - CrawlConfig
```
public CrawlConfig()
```
- Method Detail
  - validate
```
public void validate()
              throws Exception
```
    Validates the configs specified by this instance.
    
    Throws:
    
    Exception - on Validation fail
  - getCrawlStorageFolder
```
public String getCrawlStorageFolder()
```
  - setCrawlStorageFolder
```
public void setCrawlStorageFolder(String crawlStorageFolder)
```
    The folder which will be used by crawler for storing the intermediate crawl data. The content of this folder should not be modified manually.
    
    Parameters:
    
    crawlStorageFolder - The folder for the crawler's storage
  - isResumableCrawling
```
public boolean isResumableCrawling()
```
  - setResumableCrawling
```
public void setResumableCrawling(boolean resumableCrawling)
```
    If this feature is enabled, you would be able to resume a previously stopped/crashed crawl. However, it makes crawling slightly slower
    
    Parameters:
    
    resumableCrawling - Should crawling be resumable between runs ?
  - getMaxDepthOfCrawling
```
public int getMaxDepthOfCrawling()
```
  - setMaxDepthOfCrawling
```
public void setMaxDepthOfCrawling(int maxDepthOfCrawling)
```
    Maximum depth of crawling For unlimited depth this parameter should be set to -1
    
    Parameters:
    
    maxDepthOfCrawling - Depth of crawling (all links on current page = depth of 1)
  - getMaxPagesToFetch
```
public int getMaxPagesToFetch()
```
  - setMaxPagesToFetch
```
public void setMaxPagesToFetch(int maxPagesToFetch)
```
    Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
    
    Parameters:
    
    maxPagesToFetch - How many pages to fetch from all threads together ?
  - getUserAgentString
```
public String getUserAgentString()
```
    Returns:
    
    userAgentString
  - setUserAgentString
```
public void setUserAgentString(String userAgentString)
```
    user-agent string that is used for representing your crawler to web servers. See http://en.wikipedia.org/wiki/User_agent for more details
    
    Parameters:
    
    userAgentString - Custom userAgent string to use as your crawler's identifier
  - getDefaultHeaders
```
public Collection<org.apache.http.message.BasicHeader> getDefaultHeaders()
```
    Return a copy of the default header collection.
  - setDefaultHeaders
```
public void setDefaultHeaders(Collection<? extends org.apache.http.Header> defaultHeaders)
```
    Set the default header collection (creating copies of the provided headers).
  - getPolitenessDelay
```
public int getPolitenessDelay()
```
  - setPolitenessDelay
```
public void setPolitenessDelay(int politenessDelay)
```
    Politeness delay in milliseconds (delay between sending two requests to the same host).
    
    Parameters:
    
    politenessDelay - the delay in milliseconds.
  - isIncludeHttpsPages
```
public boolean isIncludeHttpsPages()
```
  - setIncludeHttpsPages
```
public void setIncludeHttpsPages(boolean includeHttpsPages)
```
    Parameters:
    
    includeHttpsPages - Should we crawl https pages?
  - isIncludeBinaryContentInCrawling
```
public boolean isIncludeBinaryContentInCrawling()
```
  - setIncludeBinaryContentInCrawling
```
public void setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)
```
    Parameters:
    
    includeBinaryContentInCrawling - Should we fetch binary content such as images, audio, ...?
  - isProcessBinaryContentInCrawling
```
public boolean isProcessBinaryContentInCrawling()
```
  - setProcessBinaryContentInCrawling
```
public void setProcessBinaryContentInCrawling(boolean processBinaryContentInCrawling)
```
    Should we process binary content such as images, audio, ... using TIKA?
  - getMaxConnectionsPerHost
```
public int getMaxConnectionsPerHost()
```
  - setMaxConnectionsPerHost
```
public void setMaxConnectionsPerHost(int maxConnectionsPerHost)
```
    Parameters:
    
    maxConnectionsPerHost - Maximum Connections per host
  - getMaxTotalConnections
```
public int getMaxTotalConnections()
```
  - setMaxTotalConnections
```
public void setMaxTotalConnections(int maxTotalConnections)
```
    Parameters:
    
    maxTotalConnections - Maximum total connections
  - getSocketTimeout
```
public int getSocketTimeout()
```
  - setSocketTimeout
```
public void setSocketTimeout(int socketTimeout)
```
    Parameters:
    
    socketTimeout - Socket timeout in milliseconds
  - getConnectionTimeout
```
public int getConnectionTimeout()
```
  - setConnectionTimeout
```
public void setConnectionTimeout(int connectionTimeout)
```
    Parameters:
    
    connectionTimeout - Connection timeout in milliseconds
  - getMaxOutgoingLinksToFollow
```
public int getMaxOutgoingLinksToFollow()
```
  - setMaxOutgoingLinksToFollow
```
public void setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)
```
    Parameters:
    
    maxOutgoingLinksToFollow - Max number of outgoing links which are processed from a page
  - getMaxDownloadSize
```
public int getMaxDownloadSize()
```
  - setMaxDownloadSize
```
public void setMaxDownloadSize(int maxDownloadSize)
```
    Parameters:
    
    maxDownloadSize - Max allowed size of a page. Pages larger than this size will not be fetched.
  - isFollowRedirects
```
public boolean isFollowRedirects()
```
  - setFollowRedirects
```
public void setFollowRedirects(boolean followRedirects)
```
    Parameters:
    
    followRedirects - Should we follow redirects?
  - isShutdownOnEmptyQueue
```
public boolean isShutdownOnEmptyQueue()
```
  - setShutdownOnEmptyQueue
```
public void setShutdownOnEmptyQueue(boolean shutdown)
```
    Should the crawler stop running when the queue is empty?
  - isOnlineTldListUpdate
```
public boolean isOnlineTldListUpdate()
```
  - setOnlineTldListUpdate
```
public void setOnlineTldListUpdate(boolean online)
```
    Should the TLD list be updated automatically on each run? Alternatively, it can be loaded from the embedded tld-names.txt resource file that was obtained from https://publicsuffix.org/list/effective_tld_names.dat
  - getProxyHost
```
public String getProxyHost()
```
  - setProxyHost
```
public void setProxyHost(String proxyHost)
```
    Parameters:
    
    proxyHost - If crawler should run behind a proxy, this parameter can be used for specifying the proxy host.
  - getProxyPort
```
public int getProxyPort()
```
  - setProxyPort
```
public void setProxyPort(int proxyPort)
```
    Parameters:
    
    proxyPort - If crawler should run behind a proxy, this parameter can be used for specifying the proxy port.
  - getProxyUsername
```
public String getProxyUsername()
```
  - setProxyUsername
```
public void setProxyUsername(String proxyUsername)
```
    Parameters:
    
    proxyUsername - If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the username.
  - getProxyPassword
```
public String getProxyPassword()
```
  - setProxyPassword
```
public void setProxyPassword(String proxyPassword)
```
    If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
    
    Parameters:
    
    proxyPassword - String Password
  - getAuthInfos
```
public List<AuthInfo> getAuthInfos()
```
    Returns:
    
    the authentications Information
  - addAuthInfo
```
public void addAuthInfo(AuthInfo authInfo)
```
  - setAuthInfos
```
public void setAuthInfos(List<AuthInfo> authInfos)
```
    Parameters:
    
    authInfos - authenticationInformations to set
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class CrawlConfig

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

CrawlConfig

Method Detail

validate

getCrawlStorageFolder

setCrawlStorageFolder

isResumableCrawling

setResumableCrawling

getMaxDepthOfCrawling

setMaxDepthOfCrawling

getMaxPagesToFetch

setMaxPagesToFetch

getUserAgentString

setUserAgentString

getDefaultHeaders

setDefaultHeaders

getPolitenessDelay

setPolitenessDelay

isIncludeHttpsPages

setIncludeHttpsPages

isIncludeBinaryContentInCrawling

setIncludeBinaryContentInCrawling

isProcessBinaryContentInCrawling

setProcessBinaryContentInCrawling

getMaxConnectionsPerHost

setMaxConnectionsPerHost

getMaxTotalConnections

setMaxTotalConnections

getSocketTimeout

setSocketTimeout

getConnectionTimeout

setConnectionTimeout

getMaxOutgoingLinksToFollow

setMaxOutgoingLinksToFollow

getMaxDownloadSize

setMaxDownloadSize

isFollowRedirects

setFollowRedirects

isShutdownOnEmptyQueue

setShutdownOnEmptyQueue

isOnlineTldListUpdate

setOnlineTldListUpdate

getProxyHost

setProxyHost

getProxyPort

setProxyPort

getProxyUsername

setProxyUsername

getProxyPassword

setProxyPassword

getAuthInfos

addAuthInfo

setAuthInfos

toString