Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SCRAPY_CHECK is not set while running contract #4222

Open
Lightjohn opened this issue Dec 8, 2019 · 11 comments · May be fixed by #4298
Open

SCRAPY_CHECK is not set while running contract #4222

Lightjohn opened this issue Dec 8, 2019 · 11 comments · May be fixed by #4298

Comments

@Lightjohn
Copy link

@Lightjohn Lightjohn commented Dec 8, 2019

Description

Hi, it seems that #3739 is not doing what the documentation describe:

os.environ.get('SCRAPY_CHECK') is returning None in my contract check.

Steps to Reproduce

  1. Create a project from scratch
  2. Add a random spider
  3. Contract code is as follow
    def parse(self, response):
        """
        @url http://www.amazon.com/s?field-keywords=selfish+gene
        @returns requests 1 1
        """
        print("test", os.environ.get('SCRAPY_CHECK'))
        if os.environ.get('SCRAPY_CHECK'):
            yield scrapy.Request(url="next_url")

Expected behavior: Request should be yielded as per the documentation

Actual behavior: Nothing happen

Reproduces how often: In my local project and with fresh project

Versions

Windows

(globenv) C:\Users\johnl>scrapy version --verbose
Scrapy       : 1.8.0
lxml         : 4.4.1.0
libxml2      : 2.9.5
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 19.10.0
Python       : 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.7
Platform     : Windows-10-10.0.18362-SP0

Linux

scrapy version --verbose
Scrapy       : 1.8.0
lxml         : 4.4.1.0
libxml2      : 2.9.9
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 19.7.0
Python       : 3.6.8 (default, Oct  7 2019, 12:59:55) - [GCC 8.3.0]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1d  10 Sep 2019)
cryptography : 2.8
Platform     : Linux-4.4.0-18362-Microsoft-x86_64-with-Ubuntu-18.04-bionic
@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Dec 9, 2019

The variable is only defined at initialization, not at run time. If you have a look at the code example from the documentation, you’ll notice that it is used in __init__, not in a callback. I think this is on purpose, although it could be more clear in the documentation.

I’m turning this ticket into a documentation enhancement request. If you would rather it be a request to have the environment variable also available at run time, from callback, please let me know.

@Lightjohn
Copy link
Author

@Lightjohn Lightjohn commented Dec 9, 2019

Thanks for the quick answer.

I think it could be nice to have it available for the callback running in check mode.
At least it would make sense to me that the ENV variable is set all the time while check.

But from there I can definitely build the check awareness feature that I want from __init__ very easily.

EDIT: After giving a little bit more thought I think I will stay with doc enhancement as in my case it's more a trick in my spider that is forcing another trick!

@elacuesta
Copy link
Member

@elacuesta elacuesta commented Dec 10, 2019

While I agree that the current behaviour does match the docs, I wonder if this isn't a reasonable use case. From a very quick look, it seems like it could be solved by just including the next code block in the set_environ context manager:

diff --git scrapy/commands/check.py scrapy/commands/check.py
index 9d4437a4..09a76ca7 100644
--- scrapy/commands/check.py
+++ scrapy/commands/check.py
@@ -78,19 +78,19 @@ class Command(ScrapyCommand):
                 elif tested_methods:
                     self.crawler_process.crawl(spidercls)
 
-        # start checks
-        if opts.list:
-            for spider, methods in sorted(contract_reqs.items()):
-                if not methods and not opts.verbose:
-                    continue
-                print(spider)
-                for method in sorted(methods):
-                    print('  * %s' % method)
-        else:
-            start = time.time()
-            self.crawler_process.start()
-            stop = time.time()
-
-            result.printErrors()
-            result.printSummary(start, stop)
-            self.exitcode = int(not result.wasSuccessful())
+            # start checks
+            if opts.list:
+                for spider, methods in sorted(contract_reqs.items()):
+                    if not methods and not opts.verbose:
+                        continue
+                    print(spider)
+                    for method in sorted(methods):
+                        print('  * %s' % method)
+            else:
+                start = time.time()
+                self.crawler_process.start()
+                stop = time.time()
+
+                result.printErrors()
+                result.printSummary(start, stop)
+                self.exitcode = int(not result.wasSuccessful())

Any thoughts?

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Dec 10, 2019

I have no strong opinion either way.

@Matthijsy Do you remember if you had a reason not to apply the environment variable at run time?

@Matthijsy
Copy link
Contributor

@Matthijsy Matthijsy commented Dec 11, 2019

I am not sure why I did it that way. I think it would be fine to include the check run itself in the environment variable block as well.

@ivange94
Copy link

@ivange94 ivange94 commented Mar 5, 2020

Is someone working on this? If not I'd like to work on this.

@gigatesseract
Copy link

@gigatesseract gigatesseract commented Mar 11, 2020

@Gallaecio @Matthijsy
The referenced PR has no diff. Does the PR fix the issue? What am I missing?

@Matthijsy
Copy link
Contributor

@Matthijsy Matthijsy commented Mar 11, 2020

The code has an added tab to include it within the context of set_environ(SCRAPY_CHECK='true'):, this makes sure that the environment variable is set during the execution of the code

@gigatesseract
Copy link

@gigatesseract gigatesseract commented Mar 12, 2020

Got it. My bad.
Thanks for clarifying.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 12, 2020

#4298, from @faizan2700, is only missing tests. If @faizan2700 is busy, maybe someone else can work on tests, and send a pull request to the corresponding branch in his fork, or create a new pull request built on top of his changes.

@faizan2700
Copy link

@faizan2700 faizan2700 commented Mar 18, 2020

when i run tests using tox command in scrapy directory instead of running tests I am getting exception (ValueError : No closing quotation). I tried to check this online but couldn't solve the problem. Can someone please help me on this (@Gallaecio ) . Thank You

@Gallaecio Gallaecio added the contracts label Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

7 participants
You can’t perform that action at this time.