General discussion about Xapian, an Open Source Probabilistic Information Retrieval library

Andrew Betts | 17 Feb 13:54

DatabaseModifiedError on get_data - best practice?

Hi,

I have previously had a problem with getting this error on a get_mset
call, and solved it by subclassing XapianEnquire with a
backoff-and-retry algorithm (as suggested by this list, many thanks!).
 However, I now get it intermittently when calling get_data on a
XapianDocument.  The same solution doesn't seem to be quite as easy in
this case, because:

1. The document is not instantiated by my code, it's returned from the
Iterator, so I can't easily subclass it without editing the bindings.

2. The document doesn't have a reference to the database, so I can't
reopen it from that scope.

So, first is it necessary to reopen the database in these situations,
or could I simply call get_data on the same document object after a
brief delay?  And second, how/where would you suggest I insert the
retry procedure?  Currently I can only see a few options, none of
which seem very good, and the first two don't solve the reopen
problem):

A) Subclass XapianDocument, and in order to make the bindings use the
subclass, also subclass the iterator, matchset and enquire.
B) Hack the bindings to insert the retry into the existing
XapianDocument::get_data method.
C) Add retry at the application level (need to add to several dozen projects!)

Any ideas much appreciated.

(Continue reading)

headers

Olly Betts | 8 Mar 01:57

Re: DatabaseModifiedError on get_data - best practice?

On Fri, Feb 17, 2012 at 12:54:36PM +0000, Andrew Betts wrote:
> I have previously had a problem with getting this error on a get_mset
> call, and solved it by subclassing XapianEnquire with a
> backoff-and-retry algorithm (as suggested by this list, many thanks!).
>  However, I now get it intermittently when calling get_data on a
> XapianDocument.  The same solution doesn't seem to be quite as easy in
> this case, because:
> 
> 1. The document is not instantiated by my code, it's returned from the
> Iterator, so I can't easily subclass it without editing the bindings.

Also, subclassing Xapian classes which aren't intended to be subclassed
means that your additions will be ignored if you pass an object of the
subclass to Xapian.  I don't think anything takes an Enquire object, so
that's actually OK (at least currently), but for Xapian::Document it's
an issue.

I'd probably recommend wrapping by making the Xapian object a member
rather than a base class, to make it more obvious what will happen.
But wrapping isn't the answer here (if it was, the job the wrapper
does could be done in the Xapian library).

> So, first is it necessary to reopen the database in these situations,
> or could I simply call get_data on the same document object after a
> brief delay?

The data you wanted has been overwritten, and retrying won't magically
recover it.  You need to reopen the database and start again.

> C) Add retry at the application level (need to add to several dozen

(Continue reading)

headers

Andrew Betts | 8 Mar 10:43

Re: DatabaseModifiedError on get_data - best practice?

>> C) Add retry at the application level (need to add to several dozen
>> projects!)
>
> This is the way to do it, at least until we track which older versions
> are still in use.

This is indeed what I ended up doing, thanks.
**********************************************************************************
This email was sent by a company owned by Pearson plc, registered office at 80 Strand, London WC2R 0RL.
Registered in England and Wales with company number 53723