us.codecraft.webmagic.scheduler.component

类 BloomFilterDuplicateRemover

java.lang.Object

us.codecraft.webmagic.scheduler.component.BloomFilterDuplicateRemover

所有已实现的接口:

DuplicateRemover

public class BloomFilterDuplicateRemover
extends Object
implements DuplicateRemover

BloomFilterDuplicateRemover for huge number of urls.

从以下版本开始:

0.5.1

作者:

code4crafer@gmail.com

构造器概要

构造器

构造器和说明
BloomFilterDuplicateRemover(int expectedInsertions)
BloomFilterDuplicateRemover(int expectedInsertions, double fpp)

方法概要

限定符和类型	方法和说明
int	getTotalRequestsCount(Task task) Get TotalRequestsCount for monitor.
protected String	getUrl(Request request)
boolean	isDuplicate(Request request, Task task) Check whether the request is duplicate.
protected com.google.common.hash.BloomFilter<CharSequence>	rebuildBloomFilter()
void	resetDuplicateCheck(Task task) Reset duplicate check.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions)

BloomFilterDuplicateRemover

public BloomFilterDuplicateRemover(int expectedInsertions,
                                   double fpp)

参数:

expectedInsertions - the number of expected insertions to the constructed

fpp - the desired false positive probability (must be positive and less than 1.0)

方法详细资料

rebuildBloomFilter

protected com.google.common.hash.BloomFilter<CharSequence> rebuildBloomFilter()

isDuplicate

public boolean isDuplicate(Request request,
                           Task task)

从接口复制的说明: DuplicateRemover

Check whether the request is duplicate.

指定者:

isDuplicate 在接口中 DuplicateRemover

参数:

request - request

task - task

返回:

true if is duplicate

getUrl

protected String getUrl(Request request)

resetDuplicateCheck

public void resetDuplicateCheck(Task task)

从接口复制的说明: DuplicateRemover

Reset duplicate check.

指定者:

resetDuplicateCheck 在接口中 DuplicateRemover

参数:

task - task

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

从接口复制的说明: DuplicateRemover

Get TotalRequestsCount for monitor.

指定者:

getTotalRequestsCount 在接口中 DuplicateRemover

参数:

task - task

返回:

number of total request