Btw, I needed hundreds of different spiders, because I needed to use hundreds of different cookies, it wasn't for different domains as in docs example. If there's better way I should approach this, please feel free to tell me :)

Hi @madvas,

Maybe you can use cookiejar meta key?

But I'm curious why starting a few hundreds at once didn't work.

Hello @kmike,

Well, I don't know neither, I wasn't digging too much into it, cuz I didn't want to overload server. Maybe with cookiejar it would be possible also. Basically what I needed was to log in with around hundred of different accounts and then do scraping. I found this quite clean solution for this.

@kmike
I use cookiejar for this kind of situationm, but it seems that you cannot specify CONCURRENT_REQUESTS and DOWNLOAD_DELAY per cookiejar.

@jmaynier you can set custom request.meta['download_slot'] for requests, an unique value per cookiejar. This way concurrency settings will work per cookiejar.

Thanks @kmike, it seems to be exactly what I was looking for ! I will try it right away :-)
It should be part of scrapy documentation.