Development discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Mihai Bivol | 18 Jul 2012 03:28

Dynamic Snippets - small scale result testing

Hello,

I've made a public small scale test[1] with different parameters for
generating snippets.

There is a set of queries that return different documents from
Wikipedia. For each document there are 6 snippets that I would like to
be ranked from 1 to 5.

1 - Worst snippet from the set, totally irrelevant to the _query_.
2 - Worse than the majority of the snippets from the set.
3 - Quite relevant snippet to the _query_. Not different than the
other ones from the set.
4 - Better than other relevant snippets from the set.
5 - Snippet that is totally relevant to the _query_ and document
content, best from the set.

If anyone has suggestions or any kind of feedback, please reply to this thread.
If you complete the assessment, please fill in the name field with
your IRC nickname. In case of unwanted submit (or an already existing
submit with your name), please notify me with the name of the submit
so I won't take it into account.

Thanks,
Mihai Bivol
[1] http://swarm.cs.pub.ro/~mbivol/snippet_assessor/

headers

Peter Karman | 3 Feb 2013 03:39

Re: Dynamic Snippets - small scale result testing

Mihai Bivol wrote on 7/17/12 8:28 PM:
> Hello,
> 
> I've made a public small scale test[1] with different parameters for
> generating snippets.
> 
> There is a set of queries that return different documents from
> Wikipedia. For each document there are 6 snippets that I would like to
> be ranked from 1 to 5.
> 
> 1 - Worst snippet from the set, totally irrelevant to the _query_.
> 2 - Worse than the majority of the snippets from the set.
> 3 - Quite relevant snippet to the _query_. Not different than the
> other ones from the set.
> 4 - Better than other relevant snippets from the set.
> 5 - Snippet that is totally relevant to the _query_ and document
> content, best from the set.
> 
> If anyone has suggestions or any kind of feedback, please reply to this thread.
> If you complete the assessment, please fill in the name field with
> your IRC nickname. In case of unwanted submit (or an already existing
> submit with your name), please notify me with the name of the submit
> so I won't take it into account.

was there ever any final assessment of this project?

--

-- 
Peter Karman  .  http://peknet.com/  .  peter <at> peknet.com

headers

Mihai Bivol | 3 Feb 2013 10:32

Re: Dynamic Snippets - small scale result testing

2013/2/3 Peter Karman <peter <at> peknet.com>:
> Mihai Bivol wrote on 7/17/12 8:28 PM:
>> Hello,
>>
>> I've made a public small scale test[1] with different parameters for
>> generating snippets.
>>
>> There is a set of queries that return different documents from
>> Wikipedia. For each document there are 6 snippets that I would like to
>> be ranked from 1 to 5.
>>
>> 1 - Worst snippet from the set, totally irrelevant to the _query_.
>> 2 - Worse than the majority of the snippets from the set.
>> 3 - Quite relevant snippet to the _query_. Not different than the
>> other ones from the set.
>> 4 - Better than other relevant snippets from the set.
>> 5 - Snippet that is totally relevant to the _query_ and document
>> content, best from the set.
>>
>> If anyone has suggestions or any kind of feedback, please reply to this thread.
>> If you complete the assessment, please fill in the name field with
>> your IRC nickname. In case of unwanted submit (or an already existing
>> submit with your name), please notify me with the name of the submit
>> so I won't take it into account.
>
>
> was there ever any final assessment of this project?

The final results, though with a small number of assessments was a bit
over 3. So the snippets were quite relevant, but not the best that

(Continue reading)

headers

Olly Betts | 13 Mar 2013 03:07

Re: Dynamic Snippets - small scale result testing

On Sun, Feb 03, 2013 at 11:32:44AM +0200, Mihai Bivol wrote:
> 2013/2/3 Peter Karman <peter <at> peknet.com>:
> > was there ever any final assessment of this project?

In GSoC terms, Mihai passed both the mid-term and final evaluation.

In terms of "does the code work", the snippets produced seemed good to
me.  

> The final results, though with a small number of assessments was a bit
> over 3. So the snippets were quite relevant, but not the best that
> could be selected. The constants that yielded the best results are
> present in the repository in which I worked.

I think this feature is working well enough that we really should get
it merged to trunk.  If there's room for further improvements, that
can then happen on trunk.  If those improvements need API tweaks, we
can cope with that, at least until we come to releasing 1.4.0.

It looks like Mihai started to work towards merging (I see a gsoc-merge
branch in his git repo), so I think I may have dropped the ball on this
one, and I'm very sorry if that's the case.

Mihai - can you confirm if the gsoc-merge branch is what we should be
looking at merging to trunk?  If it is, we can get on with reviewing
and applying those changes.

Cheers,
    Olly

(Continue reading)