Skip to content

/ scrapy

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Should we delete the deprecated code in Scrapy 2.0? #4356

Open
noviluni opened this issue Feb 20, 2020 · 8 comments · May be fixed by #4178
Open

Should we delete the deprecated code in Scrapy 2.0? #4356

noviluni opened this issue Feb 20, 2020 · 8 comments · May be fixed by #4178
Labels
cleanup good first issue

Comments

@noviluni
Copy link
Member

@noviluni noviluni commented Feb 20, 2020 •
edited by Gallaecio

I recopilated some of the old (before 2018) deprecation warnings to discuss if we can delete them in the upcoming Scrapy 2.0 release (chronological order):

BaseSgmlLinkExtractor and SgmlLinkExtractor (classes)

  • Deprecation warnings: 07/2014
  • Links:
    • https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/sgml.py#L23
    • https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/sgml.py#L116
  • PR: #4679

Crawler.spiders (property)

  • Deprecation warning: 08/2014
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/crawler.py#L74
  • PR: #4398

HtmlParserLinkExtractor (class)

  • Deprecation warning: 05/2015
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/htmlparser.py#L22
  • PR: #4674

stringify_dict (function)

  • Deprecation warning: 07/2015
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L296
  • PR: #4362

is_writable (function)

  • Deprecation warning: 07/2015
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L311
  • PR: #4362

setattr_default (function)

  • Deprecation warning: 07/2015
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L322
  • PR: #4362

Initialising scrapy.extensions.feedexport.S3FeedStorage without AWS keys is deprecated. (procedure)

  • Deprecation warning: 11/2015
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/extensions/feedexport.py#L103
  • PR: #4411

scrapy.utils.datatypes.MultiValueDict, MultiValueDictKeyError and SiteNode (classes)

  • Deprecation warnings: 01/2016
  • Links:
    • https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L46
    • https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L19
    • https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L161
  • PR: #4400

isbinarytext (function)

  • Deprecation warning: 03/2016
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L168
  • PR: #4362

scrapy.downloadermiddlewares.chunked (module)

  • Deprecation warning: 11/2016
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/downloadermiddlewares/chunked.py#L7
  • PR: #4431

Spider.make_requests_from_url (method)

  • Deprecation warning: 02/2017
  • Link: https://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/spiders/init.py#L66
  • PR: #4178

There are other deprecation warnings but as they are from 2018 to the present I think we can leave it there for a while. However, as we are going to release a major version maybe is a good moment to do it.

Do you see any inconvenience?

Any thoughts? Let me know what you think.
@noviluni noviluni mentioned this issue Feb 20, 2020
Merged
@kmike
Copy link
Member

@kmike kmike commented Feb 20, 2020

I think that's fine to remove most of them, better in separate PRs.

It can be nice to do this for 2.0, but it is not a blocker for release; we can remove them at any time.

For Spider.make_requests_from_url there are few PRs already; it turns out a bit more tricky, as not all usages were properly deprcated.
@noviluni noviluni mentioned this issue Feb 21, 2020
Merged
@nyov
Copy link
Contributor

@nyov nyov commented Feb 21, 2020

Scrapy 2.0 sounds like the perfect release target to do some breaking changes and get rid of more of the cruft. But that's just radical me talking, as usual.

I don't see a problem with all of these getting removed wholesale, nothing here has any bearing on external lib versions, as xlib did or the Selectors shortcut. Most are merely internal renames anyway, and legacy code shouldn't have much trouble with the rest? Off with their heads! Errr... :)
This was referenced Feb 25, 2020
Closed
Merged
@nyov
Copy link
Contributor

@nyov nyov commented Mar 4, 2020

Ah, looks like we didn't get all of them in 2.0 ;)
I've done Crawler.spiders in #4398.
The LinkExtractors could possibly still be useful? But the code is in the history for whoever wants to extract it there, I'm still for removing them.

Added #4400 for MultiValueDict and SiteNode.
@nyov
Copy link
Contributor

@nyov nyov commented Mar 7, 2020

Another one from the list: #4411 for scrapy.extensions.feedexport.S3FeedStorage without AWS keys.

But I noticed I might be cleaning up easy first issues for gsoc candidates here, oops. 😊
@Gallaecio, do you think this warrants a "good first issue" tag?

So for completeness sake:
Spider.make_requests_from_url is referenced in #4178

That means, currently remaining on the list of easy pickings, here, are: scrapy.downloadermiddlewares.chunked and BaseSgmlLinkExtractor, SgmlLinkExtractor, HtmlParserLinkExtractor.
@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 12, 2020

do you think this warrants a "good first issue" tag?

Nah, let us just get rid of them in time for 2.1.
@evanjs evanjs mentioned this issue Mar 14, 2020
Merged
0 of 10 tasks complete
@elacuesta elacuesta mentioned this issue Mar 15, 2020
Merged
@elacuesta elacuesta mentioned this issue Mar 27, 2020
Closed
@noviluni
Copy link
Member Author

@noviluni noviluni commented Apr 29, 2020

Hi @nyov @elacuesta @Gallaecio @kmike , thank you for taking this!! 🚀

I updated the issue description with the PRs.
ashellunts added a commit to ashellunts/scrapy that referenced this issue Jul 10, 2020
@ashellunts
Remove deprecated class HtmlParserLinkExtractor
a6a5fa9 
Issue scrapy#4356
@ashellunts ashellunts mentioned this issue Jul 10, 2020
Merged
ashellunts added a commit to ashellunts/scrapy that referenced this issue Jul 14, 2020
@ashellunts
Remove deprecated class SgmlLinkExtractor (scrapy#4356)
8714879 
BaseSgmlLinkExtractor is also removed.
ashellunts added a commit to ashellunts/scrapy that referenced this issue Jul 16, 2020
@ashellunts
Remove deprecated class SgmlLinkExtractor (scrapy#4356)
8ee89c9 
BaseSgmlLinkExtractor is also removed.
@ashellunts ashellunts mentioned this issue Jul 17, 2020
Merged
Gallaecio pushed a commit that referenced this issue Jul 17, 2020
@ashellunts
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor a…
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
62a4ede 
…nd SgmlLinkExtractor (#4356)
@ashellunts ashellunts mentioned this issue Jul 21, 2020
Merged
@ashellunts
Copy link
Contributor

@ashellunts ashellunts commented Aug 17, 2020 •
edited

So except Spider.make_requests_from_url everything else is done. For #4178 looks like it is under discussion now.
So shall we close this issue?
@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Aug 17, 2020

Let’s just leave it open until we fix #4178.
@elacuesta elacuesta linked a pull request that will close this issue Aug 17, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cleanup good first issue
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Remove deprecated Spider.make_requests_from_url method
5 participants
@kmike @nyov @Gallaecio @ashellunts @noviluni
You can’t perform that action at this time.