Hi I'd like to work on this issue

In regards to running run_test.sh I got two questions.

1. The parameter values specified in build don't adress $TEST_ARGS and $COVERAGE. What values should these have?
2. When running pytest it reports an "unrecognized arguments: --strict-data-files" error. -strict-data-files is specified as an addopt in the config file. Looking at the options page for pytest I can't find --strict-data-files as an addopt. Am I misssing something?

1. $COVERAGE is usually empty, except for the job that computed and uploads the coverage, then it's true: https://github.com/pandas-dev/pandas/blob/master/.travis.yml#L47 Similar for $TEST_ARGS, not sure which job uses it, but it's usually empty, and some build will use it to provide extra args to pytest (if I'm not wrong).
2. It looks like you don't have the right dependencies or the right version of pytest. Use conda for it, in the root of pandas run conda env update if you already created the conda environment, conda env create if you didn't. Make sure you run conda activate pandas-dev after that.

To work in an issue, you can write take in a comment and it'll be assigned to you. Please unassign it if you stop working on it. Thanks!

Thanks, I'll remember that ;)

I'd love to take a stab at this. I'll take this issue and try to get it sorted.

Thanks @lambertsbennett. I assigned you the issue, you can do it yourself next time by writing a comment with just the word take. If you can't finally finish this would be great if you unassign yourself from the issue, so someone else can give a try. Thanks!

Hi, sorry to ask, but it's my first time trying something like this. I carried out the following steps:

1. Forked and cloned pandas.
   2)Navigated to ci/deps/ and created an environment with Anaconda using the file travis-36-cov.yaml.
2. Activated the environment.
3. Ran python setup.py build_ext --inplace -j 4
4. Ran pip install -e .
5. Navigated to ci/
6. Set variable PATTERN="((not slow and not network) or (single and db))" as indicated in the build file.
7. Set variable COVERAGE="true" as indicated in the build file.

When I execute run_tests.sh I get the same error Tiore96 indicated above. When I run the pytest command alone in the directory above, the tests will run for some time and then begin to throw many errors. I'm not sure if I'm doing something fundamentally stupid here.

The first thing I'd recommend is to make sure you use Miniconda (not Anaconda) and that you never install any dependency in the base environment, that's the cause of many errors, and if misleading error messages. And often errors related to pytest.

But I think in this particular case not sure if the problem may be the path where you're calling the script from. I think it should be from the root of this project, so it's called using ci/run_tests. py. The error you get seems to be caused by pandas/conftest. py not being found by pytest, and I guess the path is the problem.

You can see in the Travis config how it is called, and used the same way.

Hope this helps.

Ok thanks for the pointers, I'll keep on it!

Hi again,

So I've worked at splitting test files to decrease the build time of the coverage build. To do so, I did the following:

1. Ran pytest and analysed which test files take the longest to run.
2. Split the the files with the longest run times (about 10 of the original test files) into as many smaller units as possible while maintaining some sort of logical system. (Often placing test classes into separate files).
   3)Split some test classes into multiple classes across files when tests were in a large monolithic class.
3. Re-ran travis CI to check with each set of changes.

At this point, each coverage build takes between 30-35 minutes. It is not the best gain in build time, but I worry that splitting things even further will create a bit of a mess. I was surprised to see that splitting the lengthiest files more than 5 times each still didn't have much of an effect on the overall build time.

Thanks for the effort @lambertsbennett.

I think we're missing something here. My expectation was that the times would improve by just splitting couple of files in two. So if after what you did there is not a clear gain, I think there is something that we're not understanding.

I think what would be useful is first to gather all the information about what is going on. How long is every file taking, how many parallel jobs are being executed in the CI, and if it's not too complex to obtain, which files are being executed in each process and how long the total for each process is.

I guess you already have some of this info. I think when we have all this, we should be able to understand what's going on, and what's the best solution.

Thanks!

OK, I will get into rounding up this info!

Looking at the TravisCI output, it looks like only two jobs are being split with each job taking half the tests. Maybe splitting the files more doesn't make much of a difference as a result of too few threads/jobs?

Another thing that I noticed, is that when the coverage reports are detected that there are several 'failed to upload' messages followed by 'sleeping for 30s and trying again...'. Depending on how long it attempts to upload prior to failing this could add a decent amount of time?

However, comparing to the '3.6, locale' build there are a similar number of tests run (difference of about 160), but pytest executes in 17 min 30 s vs 23 min 30 s, so there is definitely a hang up in terms of pytest taking longer.

Thanks for the research @lambertsbennett

If the "sleeping for 30s" is happening often, we should have a look and see what's going on.

My understanding is that until recently, pytest was splitting all the tests in two processes individually. So, if there are 1,000 tests, it'd take 500 randomly for one process, and send the other 500 to the other. That won't make both processes have the same amount of work and take the same, but the difference should be big. The slowest tests take around 30 seconds, so even if we're unlucky and one process has several slow tests more than the other, the difference won't be more than couple of minutes.

You can see the slowest tests for example here: https://travis-ci.org/pandas-dev/pandas/jobs/635275875#L2143

Now, we're instead splitting by file. I'm not sure how long it takes for each file, but I assume the slowest are several minutes. If we have 100 files, pytest should divide them in 50 and 50 randomly, and if the tests in each split take around the same, all will be good. But if there are some files that are much slower, taking for example 5 minutes, if there are couple of those more in one process than the other, one process will finish quite fast, but the other will take 10 minutes more to finish, and the job as a whole will be taking much longer (5 minutes more) than before.

I think the solution is to detect how long is taking every file, and try to make sure that there are not files that take a lot of time. I think even having only 2 jobs, if we have more files, and all them take a short period, is very unlikely that the difference between both processes is big.

Does this make sense?

Sorry for the delayed response. This totally makes sense, I had calculated the run length of the files prior to splitting, but I will check again now that I've split files to see how long the new set of files takes.