us.codecraft.webmagic.scheduler

类 DuplicateRemovedScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

所有已实现的接口:

Scheduler

直接已知子类:

PriorityScheduler, QueueScheduler

public abstract class DuplicateRemovedScheduler
extends Object
implements Scheduler

Remove duplicate urls and only push urls which are not duplicate.

从以下版本开始:

0.5.0

作者:

code4crafer@gmail.com

字段概要

字段

限定符和类型	字段和说明
protected org.slf4j.Logger	logger

构造器概要

构造器

构造器和说明
DuplicateRemovedScheduler()

方法概要

限定符和类型	方法和说明
DuplicateRemover	getDuplicateRemover()
void	push(Request request, Task task) add a url to fetch
protected void	pushWhenNoDuplicate(Request request, Task task)
DuplicateRemovedScheduler	setDuplicateRemover(DuplicateRemover duplicatedRemover)
protected boolean	shouldReserved(Request request)

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 us.codecraft.webmagic.scheduler.Scheduler

poll

字段详细资料

logger

protected org.slf4j.Logger logger

构造器详细资料

DuplicateRemovedScheduler

public DuplicateRemovedScheduler()

方法详细资料

getDuplicateRemover

public DuplicateRemover getDuplicateRemover()

setDuplicateRemover

public DuplicateRemovedScheduler setDuplicateRemover(DuplicateRemover duplicatedRemover)

push

public void push(Request request,
                 Task task)

从接口复制的说明: Scheduler

add a url to fetch

指定者:

push 在接口中 Scheduler

参数:

request - request

task - task

shouldReserved

protected boolean shouldReserved(Request request)

pushWhenNoDuplicate

protected void pushWhenNoDuplicate(Request request,
                                   Task task)