Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEED_URI_PARAMS: custom params throws KeyError #4962

Open
jas0nkim opened this issue Jan 22, 2021 · 9 comments · May be fixed by #4966
Open

FEED_URI_PARAMS: custom params throws KeyError #4962

jas0nkim opened this issue Jan 22, 2021 · 9 comments · May be fixed by #4966
Labels
bug good first issue

Comments

@jas0nkim
Copy link

@jas0nkim jas0nkim commented Jan 22, 2021 •
edited

Description

With Scrapy version 2.4.1, any custom feed uri params (FEED_URI_PARAMS) throws KeyError.

Steps to Reproduce

  1. on myproject/settings.py
    FEED_URI_PARAMS = 'myproject.utils.uri_params'

  2. on myproject/utils.py
    def uri_params(params, spider):
    return {**params, 'spider_name': spider.name}

  3. scrapy crawl <spider_name> -o "%(spider_name)s.jl"

Expected behavior: Should export the items to a jsonline file having the spider name.

Actual behavior:
ERROR: Error caught on signal handler: <bound method FeedExporter.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fc68cb20bb0>>
Traceback (most recent call last):
File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/utils/defer.py", line 157, in maybeDeferred_coro
result = f(*args, **kw)
File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/PROJECT_ROOT/.venv/lib/python3.8/site-packages/scrapy/extensions/feedexport.py", line 294, in open_spider
uri=uri % uri_params,
KeyError: 'spider_name'

Reproduces how often: Everytime

Versions

Scrapy : 2.4.1
lxml : 4.6.1.0
libxml2 : 2.9.10
cssselect : 1.1.0
parsel : 1.6.0
w3lib : 1.22.0
Twisted : 20.3.0
Python : 3.8.5 (default, Jul 28 2020, 12:59:40) - [GCC 9.3.0]
pyOpenSSL : 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020)
cryptography : 3.2.1
Platform : Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.29

Additional context

on line 480 in /scrapy/extensions/feedexport.py,
returns params should be changed to return uripar_function(params, spider)
@Gallaecio Gallaecio added the bug label Jan 23, 2021
@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Jan 23, 2021

on line 480 in /scrapy/extensions/feedexport.py,
returns params should be changed to return uripar_function(params, spider)

Indeed:

scrapy/scrapy/extensions/feedexport.py

Lines 487 to 488 in 85604e1

uripar_function(params, spider)
return params

For your specific use case, though, I think you can just use %(name)s, based on these lines above:

scrapy/scrapy/extensions/feedexport.py

Lines 480 to 481 in 85604e1

for k in dir(spider):
params[k] = getattr(spider, k)

@jas0nkim
Copy link
Author

@jas0nkim jas0nkim commented Jan 26, 2021

Yes, %(name)s was my workaround solution. Thanks for your comment, Gallaecio.

@inspurwusixuan
Copy link

@inspurwusixuan inspurwusixuan commented Jan 27, 2021

Hi @jas0nkim @Gallaecio,

I'm new here. Can I take this issue? :)

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Jan 28, 2021 •
edited

@inspurwusixuan Please go ahead! 🎉

inspurwusixuan added a commit to inspurwusixuan/scrapy that referenced this issue Jan 28, 2021
@inspurwusixuan
fix FEED_URI_PARAMS: custom params throws KeyError closes scrapy#4962
b1d97e8
@Gallaecio Gallaecio linked a pull request that will close this issue Jan 28, 2021
Open
@larue-jordan
Copy link

@larue-jordan larue-jordan commented May 4, 2021

Is this issue still available?

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented May 4, 2021

Not really, there is already a pull request, #4966, which only needs 1 more maintainer review before being merged.

@larue-jordan
Copy link

@larue-jordan larue-jordan commented May 4, 2021 •
edited

@Gallaecio Ok. Do you know if there is currently an issue that is available that I might be able to contribute to as a first time open source developer?

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented May 4, 2021

We run out of good first issues quite quickly, and I’m not sure we have anything like that open at the moment 😑

@larue-jordan
Copy link

@larue-jordan larue-jordan commented May 4, 2021

Ok. Thank you

Gallaecio added a commit to inspurwusixuan/scrapy that referenced this issue Jul 14, 2021
@Gallaecio
Merge branch 'master' into scrapy#4962
35f03fe
@Gallaecio Gallaecio mentioned this issue Oct 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug good first issue
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Fix FEED_URI_PARAMS: custom params throws KeyError
4 participants
@Gallaecio @jas0nkim @larue-jordan @inspurwusixuan