Lucene development

Dawid Weiss (JIRA | 21 Jun 2012 13:50

Picon

[jira] [Created] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min

Dawid Weiss created LUCENE-4160:
-----------------------------------

             Summary: Bring back the functional equivalent of tests.iters.min
                 Key: LUCENE-4160
                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
             Project: Lucene - Java
          Issue Type: Bug
          Components: general/test
            Reporter: Dawid Weiss
            Assignee: Dawid Weiss
            Priority: Trivial
             Fix For: 5.0

What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".

Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't understand
how "X" is related to the original question.

I propose to implement a boolean "tests.fastfail" which would ignore any tests running on the same JVM
after the first failure has been hit.

Those with fond memories of "tests.iters.min" speak up, please.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Michael McCandless (JIRA | 21 Jun 2012 14:08

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398370#comment-13398370
] 

Michael McCandless commented on LUCENE-4160:
--------------------------------------------

bq. I propose to implement a boolean "tests.fastfail"

+1, I think that's a nice simplification over tests.iters.min.

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't
understand how "X" is related to the original question.
> I propose to implement a boolean "tests.fastfail" which would ignore any tests running on the same JVM

(Continue reading)

Hoss Man (JIRA | 25 Jun 2012 18:57

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400644#comment-13400644
] 

Hoss Man commented on LUCENE-4160:
----------------------------------

bq. "tests.iters.min=X" which is (still) kind of confusing to me because I don't understand how "X" is
related to the original question.

i think maybe you simplified the statement of the original question?

With tests.iters.min=X tests.iters=Y you were saying "attempt to run this test Y times, and even if it
fails, run it a minimum of X times." (at least, that was the theory as i understood it, maybe it never
actually worked that way).

bq. I propose to implement a boolean "tests.fastfail"

...the key here being that tests.iters today already repeats the test the specified number of times, even
if it fails -- and a new tests.fastfail" would default to "false" but if it was true then you could have the
old behavior of tests.iters.min=1.

seems fine to me ... i think hte main motivation of tests.iters.min isn't really applicable anymore since
the default logic for running multiple iterations is basically the reverse of what it use to be. 

an alternative way to think about it would be to add a "tests.iters.max", since tests.iters already
behaves similar to what tests.iters.min use to do.

Or just say to hell with it, and people who want the X,Y equivilent from before can run...

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 11:31

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402952#comment-13402952
] 

Dawid Weiss commented on LUCENE-4160:
-------------------------------------

Thanks for the clarification, Hoss. We could also do:

{noformat}
-Dtests.iters=X -Dtests.minfailures=Y
{noformat}

What this would do is repeat everything X times but ignore anything after the first Y failures... So if you'd
like to quickly abort after the first failure, you'd do:

{noformat}
-Dtests.iters=X -Dtests.minfailures=1
{noformat}

and if you wanted more (for whatever reason) you could wait for more, but still abort earlier than X. I'll
just provide a patch for this and we'll see how it turns out in practice.

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 11:31

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402953#comment-13402953
] 

Dawid Weiss commented on LUCENE-4160:
-------------------------------------

Now that I think of it minfailures or maxfailures depends on how you look at the problem :)

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't
understand how "X" is related to the original question.
> I propose to implement a boolean "tests.fastfail" which would ignore any tests running on the same JVM
after the first failure has been hit.
> Those with fond memories of "tests.iters.min" speak up, please.

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 12:57

Picon

[jira] [Updated] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


     [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dawid Weiss updated LUCENE-4160:
--------------------------------

    Attachment: LUCENE-4160.patch

Patch against trunk.
{noformat}
ant -Dtests.maxfailures=M
{noformat}

will cause any test after the first M failures to be assumption-ignored. Example:

{noformat}
   [junit4] IGNOR/A 0.03s | TestBuhu.testFailSometimes {#96 seed=[15604D6381DA415B:BE5164F081683030]}
   [junit4]    > Assumption #1: Ignored, failures limit reached (1 >= 1).
   [junit4] IGNOR/A 0.01s | TestBuhu.testFailSometimes {#97 seed=[15604D6381DA415B:E515E49B8D52E84E]}
   [junit4]    > Assumption #1: Ignored, failures limit reached (1 >= 1).
   [junit4] IGNOR/A 0.02s | TestBuhu.testFailSometimes {#98 seed=[15604D6381DA415B:2128D6F71D04F74D]}
   [junit4]    > Assumption #1: Ignored, failures limit reached (1 >= 1).
   [junit4] IGNOR/A 0.01s | TestBuhu.testFailSometimes {#99 seed=[15604D6381DA415B:EBBC938E1A402A5E]}
   [junit4]    > Assumption #1: Ignored, failures limit reached (1 >= 1).
   [junit4]    > ( <at> AfterClass output)
   [junit4]   2> NOTE: test params are: codec=Lucene40: {},
sim=RandomSimilarityProvider(queryNorm=false,coord=true): {}, locale=hu, timezone=US/Arizona
   [junit4]   2> NOTE: Linux 3.0.0-21-generic amd64/Oracle Corporation 1.7.0 (64-bit)/cpus=2,threads=1,free=50888552,total=78512128
   [junit4]   2> NOTE: All tests run in this JVM: [TestBuhu]

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 12:59

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402999#comment-13402999
] 

Dawid Weiss commented on LUCENE-4160:
-------------------------------------

One note -- this property is per-jvm-global in the sense that you don't need -Dtests.iters, you can wait for
M failures in general, so for example:
{noformat}
ant test-core -Dtests.maxfailures=1
{noformat}
will ignore any remaining tests after the first failure. This applies per-JVM though, so if you're running
with > 1 fork ed JVM then only that JVM's tests will be ignored. 

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>         Attachments: LUCENE-4160.patch

(Continue reading)

Robert Muir (JIRA | 28 Jun 2012 13:03

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403004#comment-13403004
] 

Robert Muir commented on LUCENE-4160:
-------------------------------------

What is the use case of the min/maxfailures (besides just your idea of a fastfail boolean, which I liked)

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>         Attachments: LUCENE-4160.patch
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't
understand how "X" is related to the original question.
> I propose to implement a boolean "tests.fastfail" which would ignore any tests running on the same JVM

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 13:11

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403008#comment-13403008
] 

Dawid Weiss commented on LUCENE-4160:
-------------------------------------

Maybe somebody would want to wait for 5 failures instead of 1 to get a bunch of stack traces? A boolean can be
emulated at ant level, it's basically -Dtests.maxfailures=1... Don't know, really -- I admit I don't
need this so I'm shooting at the dark here, it seemed to be useful for folks. To me it can be a boolean as well.

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>         Attachments: LUCENE-4160.patch
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't

(Continue reading)

Robert Muir (JIRA | 28 Jun 2012 13:17

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403012#comment-13403012
] 

Robert Muir commented on LUCENE-4160:
-------------------------------------

I think i originally caused the complexity by wanting to still have a way to run a test like 1000 times and look
at the failure rate. This is occasionally useful: e.g. this test fails 2% of the time and I improved the test
to fail 10% of the time or whatever :)

But really I think its more useful in general to have 'fastfail' on by default... especially now that in
general tests are reproducing a lot better than before...

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>         Attachments: LUCENE-4160.patch
>

(Continue reading)

Hoss Man (JIRA | 28 Jun 2012 18:09

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403187#comment-13403187
] 

Hoss Man commented on LUCENE-4160:
----------------------------------

bq. I think i originally caused the complexity by wanting to still have a way to run a test like 1000 times and
look at the failure rate.

right, the original driving usecase is already possible with the way tests.iters works now; it just so
happened that the way it worked before you could not only say "run this test at least 1000 times, even if it
fails, so i can compute a pass/fail rate and look for patterns" you could say "try to run this test at least
1000 times, even if it fails, so i can compute a pass/fail rate and look for patterns -- but if it doesn't
fail, just keep on trying up to 5000 times for good measure.  I'm going to lunch anyway."

if tests.maxfailures is just as easy to implement as tests.fastfail (and already implemented in this
patch) then i say go with that.

we can always add ant sugar so that -Dtests.failfast=true sets tests.maxfailures=1

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test

(Continue reading)

Dawid Weiss (JIRA | 28 Jun 2012 19:34

Picon

[jira] [Commented] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


    [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403259#comment-13403259
] 

Dawid Weiss commented on LUCENE-4160:
-------------------------------------

Yes, this patch is already working. I'll make it an alias so that -Dtests.failfast=[on/true/yes] will
also work and commit it in.

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 5.0
>
>         Attachments: LUCENE-4160.patch
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't
understand how "X" is related to the original question.

(Continue reading)

Dawid Weiss (JIRA | 29 Jun 2012 11:11

Picon

[jira] [Resolved] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


     [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dawid Weiss resolved LUCENE-4160.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 4.0

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 4.0, 5.0
>
>         Attachments: LUCENE-4160.patch
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't
understand how "X" is related to the original question.
> I propose to implement a boolean "tests.fastfail" which would ignore any tests running on the same JVM

(Continue reading)

Hoss Man (JIRA | 12 Jul 2012 01:15

Picon

[jira] [Updated] (LUCENE-4160) Bring back the functional equivalent of tests.iters.min


     [
https://issues.apache.org/jira/browse/LUCENE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated LUCENE-4160:
-----------------------------

    Fix Version/s:     (was: 4.0-ALPHA)
                   4.0

hoss20120711-manual-post-40alpha-change

> Bring back the functional equivalent of tests.iters.min
> -------------------------------------------------------
>
>                 Key: LUCENE-4160
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4160
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Trivial
>             Fix For: 4.0, 5.0
>
>         Attachments: LUCENE-4160.patch
>
>
> What is needed is effectively saying: "repeat this test N times, but stop once you hit a failure".
> Previously it was "tests.iters.min=X" which is (still) kind of confusing to me because I don't

(Continue reading)

Return

Return to gmane.comp.jakarta.lucene.devel.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane