Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upReduce dependence of unit tests on external sites #606
Labels
Milestone
Comments
|
You might also think about creating an injectable component to handle these
kinds of operations. For example, you might have an interface (lets call
it ITestRig, with a LocalTestRig, and RemoteTestRig) implementation. The
interface could have a method which gives you an inputstream with the test
data. You could have a DefaultTestRig which basically switches between
implementations at run-time depending on whether or not it detects a flag
like
-Dtestmode=Local. It would hide the implementations from the developer,
but still provide a consistent test interface.
A typical test might look like this:
// declare the test rig
ITestRig testRig = new DefaultTestRig("/some/local/kras.xml", "
http://mygene.info?symbol=KRAS");
@test
public void testParseEntrezGene(){
MyParser parser = new MyParser();
testRig.testmode = System.getProperty("testmode"); // if not set,
defaults to LOCAL
Gene gene = parser.parse(testRig.getInputStream());
validate(gene);
}
Hope this helps.
Cheers,
Mark
…On Thu, Nov 24, 2016 at 6:49 AM, Spencer Bliven ***@***.***> wrote:
This has been discussed numerous times, but I didn't see an issue devoted
to it.
The unit tests currently depend heavily on external resources, e.g. for
downloading structures or genbank files. This means that tests cannot be
run offline, and that temporary outages cause the build to fail. I suggest
1. Wherever possible, include test resources directly in the
repository rather than downloading. Large files (e.g. SCOP files) should be
truncated into manageable subsets where possible.
2. Unit tests that legitimately test connections to external resources
should be moved to biojava-integrationtest. They should also test
connections using junit assume statements (although travis also fails for
failed assumptions).
3. All modules except integrationtest should pass when offline
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#606>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABOqRyC6gENoqX-ibKvysJIBpD-KMJndks5rBaP8gaJpZM4K7uqs>
.
|
|
You might also consider a caching strategy which would obviate the need to
store some of the files in git. The test rig would determine if the local
file exists, if not it would download and cache it.
Cheers,
Mark
…On Nov 26, 2016 6:41 PM, "Mark Fortner" ***@***.***> wrote:
You might also think about creating an injectable component to handle
these kinds of operations. For example, you might have an interface (lets
call it ITestRig, with a LocalTestRig, and RemoteTestRig) implementation.
The interface could have a method which gives you an inputstream with the
test data. You could have a DefaultTestRig which basically switches
between implementations at run-time depending on whether or not it detects
a flag like
-Dtestmode=Local. It would hide the implementations from the developer,
but still provide a consistent test interface.
A typical test might look like this:
// declare the test rig
ITestRig testRig = new DefaultTestRig("/some/local/kras.xml", "
http://mygene.info?symbol=KRAS");
@test
public void testParseEntrezGene(){
MyParser parser = new MyParser();
testRig.testmode = System.getProperty("testmode"); // if not set,
defaults to LOCAL
Gene gene = parser.parse(testRig.getInputStream());
validate(gene);
}
Hope this helps.
Cheers,
Mark
On Thu, Nov 24, 2016 at 6:49 AM, Spencer Bliven ***@***.***>
wrote:
> This has been discussed numerous times, but I didn't see an issue devoted
> to it.
>
> The unit tests currently depend heavily on external resources, e.g. for
> downloading structures or genbank files. This means that tests cannot be
> run offline, and that temporary outages cause the build to fail. I suggest
>
> 1. Wherever possible, include test resources directly in the
> repository rather than downloading. Large files (e.g. SCOP files) should be
> truncated into manageable subsets where possible.
> 2. Unit tests that legitimately test connections to external
> resources should be moved to biojava-integrationtest. They should also test
> connections using junit assume statements (although travis also fails for
> failed assumptions).
> 3. All modules except integrationtest should pass when offline
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#606>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABOqRyC6gENoqX-ibKvysJIBpD-KMJndks5rBaP8gaJpZM4K7uqs>
> .
>
|
|
Switching between local and remote test modes sounds like it would just require testing everything twice to assure complete coverage. Caching can speed up the tests when run locally, but it since Travis starts with a clean environment it doesn't reduce the dependence on remote resources. I think the main task should be separating the local and remote tests so that the cause of travis failures is clearer. I would feel fine merging a PR if I knew the only error was that SCOPe is down, as long as the PR wasn't modifying the RemoteSCOPInstalllation class. I've moved the integration tests to run last, so that we can see if all the non-integration tests pass successfully. |
|
The recent test failure (fixed by @valasatava in e38a0f5) highlights that this is important for reproducability, not just performance. Basically, one of the PDB entries was remediated in a way that broke our tests. This is now fixed in master, but it means that any commit prior to the fix will now fail the tests, as will any branch from previous versions. |
|
For the last week or two travis is having problems downloading external resources, so this issue has become higher priority. We'll need to work towards reducing downloads as much as possible. It seems that downloading large files is what travis can't/won't do anymore. Some tests to look at:
|
|
Jose,
Github has this feature called LFS (git large file storage
https://git-lfs.github.com/), that might be an a way to solve the problem
(assuming that the problem is intermittent access to files needed for unit
tests). You could download the files that you want to test with and store
them there, and be assured that you could always access the files when you
need to run unit tests.
Cheers,
Mark
…On Sat, Mar 10, 2018 at 10:00 AM, Jose Manuel Duarte < ***@***.***> wrote:
For the last week or two travis is having problems downloading external
resources, so this issue has become higher priority. We'll need to work
towards reducing downloads as much as possible.
It seems that downloading large files is what travis can't/won't do
anymore. Some tests to look at:
- SiftsChainToUniprotMappingTest: consistently fails with a connection
timeout... The file to download is a few MB.
- SCOP tests: review if they are still enabled and still download
large files or not
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#606 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABOqR7HjMNc_Uk7fpBD7MMDnWtu6YnZCks5tdBTRgaJpZM4K7uqs>
.
|
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This has been discussed numerous times, but I didn't see an issue devoted to it.
The unit tests currently depend heavily on external resources, e.g. for downloading structures or genbank files. This means that tests cannot be run offline, and that temporary outages cause the build to fail. I suggest