Skip to content

/ utPLSQL

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restartable tests #1029

Open
mrhawk555 opened this issue Dec 6, 2019 · 1 comment
Open

Restartable tests #1029

mrhawk555 opened this issue Dec 6, 2019 · 1 comment
Labels
discussion

Comments

@mrhawk555
Copy link

@mrhawk555 mrhawk555 commented Dec 6, 2019

In our project, we have a group of tests that measures the working time of sql queries and compares them with the expected working time. The total number of such tests is approaching 300.
Based on regular runs, we realized that often different tests do not fit the expected time. It would be a solution for us if this test were to be run in automatic mode again after an unsuccessful result. And if in two attempts we did not reach the expected execution time, then this is a problem that needs attention.

This could be an additional test setup. For example:

package test_package is
  --%suite               --This is wrong as suite annotation is not a procedure annotation

  --%test()   
  --%restartable
  procedure my_first_procedure;

end;
@pesse
Copy link
Member

@pesse pesse commented Dec 6, 2019

Hi @mrhawk555 ,
IMO this is what we call "flaky" tests - tests that sometimes work and sometimes not.
It is highly discouraged among the tester community to accept flaky tests and it is equally discouraged to handle them by automatic retries.
This is especially true for the kind of tests that are meant to be created with utPLSQL: Assert-based tests, checking deterministic functionality.

I understand your intention and need for such a feature, but utPLSQL is definitely not the right tool for these kinds of performance tests.
I don't know about good performance testing frameworks, maybe one of the peers here knows.

One of the biggest problems that occur with a growing test-base is when they contain mixed-up, not clearly defined types of tests (unit, integration, performance). You will do your team and your future self a big favor by carefully separating and maintaining them.
One option to do so would be to use tags.
You can tag performance-test suites with a performance tag and exclude them during normal runs.
You could then create your CI pipeline in a way that performance tests are run independently, possibly with retries.

I hope this helps,
Samuel
@jgebal jgebal added the discussion label Jan 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discussion
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@jgebal @pesse @mrhawk555
You can’t perform that action at this time.