Skip to content

/ scrapy

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistence is not fully persistent when using JOBDIR setting #4106

Open
JSPromisel opened this issue Oct 25, 2019 · 2 comments
Open

Persistence is not fully persistent when using JOBDIR setting #4106

JSPromisel opened this issue Oct 25, 2019 · 2 comments
Labels
docs enhancement

Comments

@JSPromisel
Copy link

@JSPromisel JSPromisel commented Oct 25, 2019 •
edited by elacuesta

Description

I have been experimenting with keeping a persistent state for a spider by using the JOBDIR setting, so if it crashes or I stop it, I can start it up again from the same point it left off. I notice that the seen request fingerprints are persisted, as well as an active.json file holding information about the queue. However, in my testing, I find that the existence of the queue is inconsistent and when I restart a spider, the queue has not persisted, it just starts from the beginning and filters out all the seen requests.

Steps to Reproduce

  1. Create a spider with settings:
custom_settings = {
    'JOBDIR': f'/tmp/requests',
    'LOG_LEVEL': 'DEBUG
}
  1. Use the following start_requests:
def start_requests(self):
    # Get a sampling of different links
    for key in range(40):
        url = f'https://httpbin.org/links/{key}/0'
        cb_kwargs = {'key': key}
        yield scrapy.Request(url, cb_kwargs=cb_kwargs)

and parse functions:

def parse(self, response, key):
    self.logger.info(f'Requested {response.url}')

    # Close spider w/o finishing at 4, retry should
    # pick up at the next request
    if key == 4:
        raise CloseSpider('cancelled')
  1. Run the spider, then run it again after it closes.

Expected behavior: [What you expect to happen]
For true persistency, I would expect the spider to pick up the queue and begin making requests that hadn't already been made when the spider closed, then continue until the spider is finished.

Actual behavior: [What actually happens]
The output of the spider on the second run is shown below:

2019-10-25 16:24:01 [scrapy.core.engine] INFO: Spider opened
2019-10-25 16:24:01 [scrapy.core.scheduler] INFO: Resuming crawl (1 requests scheduled)
2019-10-25 16:24:01 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://httpbin.org/links/0/0> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/22/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/23/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/30/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/24/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/25/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/31/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/26/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/27/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/32/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/33/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/28/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/34/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/35/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/29/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/36/0> (referer: None)
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/37/0> (referer: None)
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/22/0
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/38/0> (referer: None)
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/23/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/30/0
2019-10-25 16:24:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://httpbin.org/links/39/0> (referer: None)
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/24/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/25/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/31/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/26/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/27/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/32/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/33/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/28/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/34/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/35/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/29/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/36/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/37/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/38/0
2019-10-25 16:24:02 [root] INFO: Requested https://httpbin.org/links/39/0
2019-10-25 16:24:02 [scrapy.core.engine] INFO: Closing spider (finished)
2019-10-25 16:24:02 [scrapy.core.engine] INFO: Spider closed (finished)

As the debug logging shows, the spider picks up the queue from the disk and continues the crawl:
[scrapy.core.scheduler] INFO: Resuming crawl (1 requests scheduled)
However, after that, it redoes the first request and filters it:
[scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://httpbin.org/links/0/0> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

I understand that keeping the entire queue for a spider is probably not feasible, I just want to make sure I understand this correctly, as the documentation claims:

Sometimes, for big sites, it’s desirable to pause crawls and be able to resume them later.

Scrapy supports this functionality out of the box by providing the following facilities:

a scheduler that persists scheduled requests on disk
a duplicates filter that persists visited requests on disk
an extension that keeps some spider state (key/value pairs) persistent between batches

I feel I am either misunderstanding the persistence functionality or the persistence is not complete.
Any feedback is greatly appreciated.

Reproduces how often: [What percentage of the time does it reproduce?]
Everytime

Versions

Scrapy       : 1.7.3
lxml         : 4.3.3.0
libxml2      : 2.9.9
cssselect    : 1.0.3
parsel       : 1.5.1
w3lib        : 1.20.0
Twisted      : 19.2.0
Python       : 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) - [Clang 6.0 (clang-600.0.57)]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.7
Platform     : Darwin-19.0.0-x86_64-i386-64bit
@shubidubapp
Copy link

@shubidubapp shubidubapp commented Feb 18, 2020

From what I have read from docs and issues here so far. I understand that currently scheduler has 2 queues, one in memory and this query holds currently active requests the other one is in disk. When a spider is being (gracefully) closed scrapy tries to finish currently active (in memory queue) requests and then it shuts down because, if a JOBDIR setting is being used, on the next run we can continue from disk queue.
The problems I see are;

  • Disk queue isn't saved unless the app gracefully closed.
  • In memory queue is never saved regardless of how it is closed.

Which brings the issues #2399 (sadly, this one is an issue created in 2016 and still not resolved), this issue and maybe some other that I have missed to notice.
I know that memory queue is eventually healthier then accessing disk for a Request each time but we should accept that we can't always expect to have a code gracefully closing and for example in my current case I am crawling relatively big files from the web and waiting for them to gracefully close every time I use Ctrl-C is actually not acceptable. I have thought of creating some kind of a backup mechanism that would do what gracefully closing does lets say every X Request or every X seconds etc. so when an unexpected problem occurs and crawling stops we wont have to restart all over again and may be go back for 1 minutes of progress most importantly we wouldn't lose the in memory requests which might be really important. I actually thought of creating such an extension but this brought me to an other issue.
I have no idea how gracefully closing handles this so I can't imitate it and make it work every X seconds. Documentation doesn't go much further in this problem other then giving the "JOBDIR" setting. When I am trying to follow from source codes, I can see where the active query priorities are saved (active.json) but I have no idea where the disk queue is saved and as far as I have seen what happens in close functions of some objects would actually render the code unable to continue, like close function of dupefilter actually closes the file used for seen requests therefore saves it's contents but renders the dupefilter unable to continue to run.
Whereas to save the contents and be able to continue after that would require one to use flush on file object.

So other then overwriting the close functions on many classes and may be adding some other alternative functions for saving the contents of a spider. Do we have a better solution?
@kmike kmike added bug and removed bug labels Aug 21, 2020
@caffeinatedMike
Copy link

@caffeinatedMike caffeinatedMike commented Sep 14, 2020 •
edited

This is plaguing me big time lately with large-scale site crawls that typically take 2+ weeks. I thought that having JOBDIR enabled would help alleviate loss of progress when the ill-configured windows server (of which i have no control over) forces a system reboot when updating. Unfortunately, I've run into the same situation @shubidubapp has described; leading to me having to restart the entire site crawl over twice (even after the latest round of collecting 625k+ items).

Can someone please help introduce some sort of redundancy for this? I'm currently at a loss as to how to prevent this.

It seems the JOBDIR setting could use some well-needed TLC for a few important aspects of its usage (see #4748 for another important one, while not as vital as this issue) as it's a cornerstone for many large-scale scraping projects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
docs enhancement
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@kmike @Gallaecio @shubidubapp @caffeinatedMike @JSPromisel
You can’t perform that action at this time.