On some test runs CrawlTestCase.test_fixed_delay fails with

>       yield self._test_delay(total=3, delay=0.1)
E   twisted.trial.unittest.FailTest: twisted.trial.unittest.FailTest: True is not false : test total or delay values are too small

Scraping a Google search results page with HTML links containing attributes href and ping, such as:

<a href="https://en.wikipedia.org/wiki/Go_(programming_language)" ping="/url?sa=t&amp;source=web&amp;rct=j&amp;url=https://en.wikipedia.org/wiki/Go_(programming_language)&amp;ved=2ahUKEwi-yY2t5eTeAhUzNX0KHXbrD7cQFjADegQIDRAB"><h3 class="LC20lb">Go (programming language) - Wikipedia</h3

If one opens the link to the docs provided in README the Readme opens on readthedocs.io. There is no navigation bar to find where one can browse to quick start page or advanced. You can only go there if one searches quick start and click on the page. Then there are navigation links for browsing through the docs.

Jus for the record:
I'm using Firefox (60.9.0 esr) on Windows 10 Pro.

Really gr

问题是这样的，我想爬取商品分页的信息于是我用了个for循环，执行document = Jsoup.connect(domain+reviews+String.format(b, p)).get()改变p的值来改变评论的页码。

但是当爬完第一页后再爬取第二页评论时（没准备爬取一页评论时都会执行这句document = Jsoup.connect(domain+reviews+String.format(b, p)).get();）出现了这样的错误：
java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.s

Reflect this kind of things

{
method : 'POST',
form : { key: 'value', key2: 'value'}
}

Bug 描述
按教程文档说明的，使用docker-compose up -d 安装启动后，直接执行task报错
不知道哪里有问题呢？
我的docker运行环境是win10

`2020-02-15 15:58:04 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: xueqiu)
22020-02-15 15:58:04 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.6.9 (default, Nov 7 2019, 10:44:02) - [GCC 8.3.0], pyOpenSSL 19

What is the current behavior?

The Crawl-Delay is ignored.

What is the expected behavior?

The Crawl-Delay should be honored, it can be retrieved using getCrawlDelay() on the robots parser.

What is the motivation / use case for changing the behavior?

A bot is bound to respect all the directives of the robots.txt

In the quick start section, it seems forgot to mention the pipeline setting, and without the setting seems will cause yield item appear wrong result. Just like #137, please update the document, if need help, I can do the contribution as well.

Describe the bug
When using the cdp driver, during closing of a browser page, this error sometimes appears.

{"level":"warn","time":"x","url":"x","error":"rpcc: the connection is closing","time":"x","message":"failed to close browser page"}
{"level":"error","time":"x","error":": rpcc: the connection is closing: session: detach timed out for session 5C391DF4E758E985AE3CBAA03774E562","t

There are several things not accurately documented/outdated:

• -v2 is used the examples but does not work
• # duckduckgo not supported although it is in the list of supported search engines
• To get a list of all search engines --config is suggested but that just fails

re.findall issue

I reviewed the tests in this project after experiencing issues with my regex also catching some html as part of the process.

So I reviewed this test file: https://github.com/gaojiuli/gain/blob/master/tests/test_parse_multiple_items.py and catched the response of [abstract_url.py ](https://github.com/gaojiuli/gain/blob/2c8160c92943837613a773f681fb190a8c434bb2/gain/parser.

DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10

Infected files in ProxyBroker:

• cli.py
• api.py