Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outdated item pipeline example about Deferred #4514

Open
elacuesta opened this issue Apr 27, 2020 · 4 comments
Open

Outdated item pipeline example about Deferred #4514

elacuesta opened this issue Apr 27, 2020 · 4 comments

Comments

@elacuesta
Copy link
Member

@elacuesta elacuesta commented Apr 27, 2020

The Take screenshot of item example says "This example demonstrates how to return a Deferred from the process_item() method", but that is no longer the case (async/await syntax is currently used).
I think we should remove that sentence and add a reminder about enabling the asyncio reactor.

Also, I wonder if it'd be good to also restore the previous Deferred example and keep both of them, since not all projects might be able/willing to switch to the asyncio reactor.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Apr 27, 2020

I believe that the example now uses coroutines, but not asyncio, so it should work with a regular Twisted reactor.

PS: I used to mix the two of them as well until @kmike and @wRAR illuminated me.

@elacuesta
Copy link
Member Author

@elacuesta elacuesta commented Apr 27, 2020

I've read this article a few times, which shows how to use async/await independently from asyncio, but I still get confused sometimes. Might be time to read it again!
In any case, there is no Deferred in play anymore, that needs to be addressed.

@incognitozen
Copy link

@incognitozen incognitozen commented May 9, 2020

The docs need an example of a spider with asynio enabled and used.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented May 9, 2020

A simple spider does not require asyncio, and a complex spider would not make for a good example.

On the other hand, it may be useful to cover, for example, an example item pipeline that uses an asyncio-based SQL client to record items in a database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.