Everything looks good to me 👍
@hoadlck can you double check?

@paulodiovani By the way, I really appreciate the detailed and well written PR. It is very easy to understand :)

What is the goodness of tests/.excluded_sites? If a site is not responding, then we need to remove it. I do disable selective sites sometimes during testing, but I just edit the data.json. But, I guess it may be a preference thing.

As far as the change to the tests goes, the load of the tests/.excluded_sites is not in the correct place. It should be loaded in the setUp() method, and used to set an attribute of the SherlockBaseTest() class. As it is, every sub-test is going to be hitting the file system, while the information is not going to change.

@hoadlck
data.json is versioned, and sometimes a site will fail due to local connection instability (a backbone down, a weird configured proxy, etc) so I think its easier to have a simple text file with a site per line for that without the risk of committing data.json changes by mistake.

I'll do the changes you requested and move the file read to setup.

Some websites may fail to respond to the requests at all, as a result the process will be stuck, a simple solution is to set a timeout, as supported by the requests module.

Changes in this pull request

• Add a fixed timeout to all requests (currently set to 10s)
  Note: In my tests, 10s was enough even with Tor. Sometimes a site may take longer to respond, but a longer timeout will make the script takes to much time (waiting + retries) and the retries will usually handle intermittent connections.
• Change Flipboard unclaimed username
• Add a tests/.excluded_sites list to allow pass tests for unresponsive sites

Fixed issues

This PR shall fix these issues.
It would be nice if the reporters could check if they still persist.

• #323 (@MinePro120)
• #357 (@eanstandig41)

Yes, the patch works successfully. I confirm.

In regards to the timeout of 10.0s...in my test setup, I get many timeout errors even increasing the timeout up to 90s. My connection is not that terrible, so I don't think that 10s is a good number.

In addition to this, we should not be using any global variable in Sherlock. I know that there are some now: this is one of the pains that I am having in doing the restructuring.

I decided to make the timeout a command line option. By default, the timeout will be disabled (just as it was before). But, a user can specify something shorter if they like.

I think that there are performance problems with some of the sites that have been added recently. If that keeps up, they are going to have to be pruned.

I am also seeing anomalies in how the timeout functions. When I set the timeout to 90s, I get completion times of less than that, with multiple error connecting problems. I think that there is something more basic causing problems: probably a manifestation of the problem noticed in #323. I am thinking that the naive way that Sherlock spawns a thread for all sites at the beginning of the query is overwhelming people's computers. We now have enough sites that we are hitting this limit.

This is something I am working on in the restructuring.

The "time delay" patch works for me successfully, but after a few times I get multiple "connection errors".

Maybe it is better to add "exclusion of bad sites" to cli? For example,

python3 sherlock --bad piling telegram

No: if we have bad sites, they need to be pruned. I think that many of the recently added sites have slow response. I did not have time to look into it yesterday, but I hope to today.

Mmm, I don’t know how else to explain. The "pling" site is not bad, but it does not work in the ru location (3000000 ip addresses are blocked in the ru location) and sherlock stumbles on it. And in China, for example, perhaps half of the list does not work, but this does not mean that the site is bad.

I'm a little late, but it works fine now, yes. Issue closed.