headers
John Wiegley | 9 Sep 2011 10:43
Picon
Gravatar

texpire deleting more articles than it should

I use leafnode both to read groups, and to archive groups (and it's awesome
for doing both).  I even use local groups to archive mailing list traffic so
that it doesn't fill up my IMAP box.

However, the other day I ran texpire and saw this:

   texpire 2.0.0.alpha20110815a.luascript: check mtime only
   wg21.c++.news: 8 articles deleted, 170 kept
   wg21.c++.lib: 1839 articles deleted, 16232 kept
   wg21.c++.ext: 1608 articles deleted, 4314 kept
   wg21.c++.date-lib: 0 articles deleted, 202 kept
   wg21.c++.core: 412 articles deleted, 9203 kept
   wg21.c++.compat: 11 articles deleted, 411 kept
   wg21.c++.all: 3 articles deleted, 922 kept

But I have these relevant settings in my config:

   expire = 36500
   groupexpire = wg21.* -1

As far as I understand, texpire should just be leaving these groups alone,
right?

Luckily I had the groups in question under Git, so I resurrected the articles
easily, but I had to use this invocation to trick texpire into rebuilding my
message.id without deleting those articles again:

   texpire -vvv -f -r comp.std.c++

What can I do to fully stop texpire from deleting anything, ever, except for
(Continue reading)

Permalink | Reply |
headers
Matthias Andree | 9 Sep 2011 19:37
Picon
Picon

Re: texpire deleting more articles than it should

Am 09.09.2011 10:43, schrieb John Wiegley:
> I use leafnode both to read groups, and to archive groups (and it's awesome
> for doing both).  I even use local groups to archive mailing list traffic so
> that it doesn't fill up my IMAP box.
> 
> However, the other day I ran texpire and saw this:
> 
>    texpire 2.0.0.alpha20110815a.luascript: check mtime only
>    wg21.c++.news: 8 articles deleted, 170 kept
>    wg21.c++.lib: 1839 articles deleted, 16232 kept
>    wg21.c++.ext: 1608 articles deleted, 4314 kept
>    wg21.c++.date-lib: 0 articles deleted, 202 kept
>    wg21.c++.core: 412 articles deleted, 9203 kept
>    wg21.c++.compat: 11 articles deleted, 411 kept
>    wg21.c++.all: 3 articles deleted, 922 kept
> 
> But I have these relevant settings in my config:
> 
>    expire = 36500
>    groupexpire = wg21.* -1
> 
> As far as I understand, texpire should just be leaving these groups alone,
> right?
> 
> Luckily I had the groups in question under Git, so I resurrected the articles
> easily, but I had to use this invocation to trick texpire into rebuilding my
> message.id without deleting those articles again:
> 
>    texpire -vvv -f -r comp.std.c++
>
(Continue reading)

Permalink | Reply |
headers
John Wiegley | 10 Sep 2011 01:08
Picon
Gravatar

Re: texpire deleting more articles than it should

>>>>> Matthias Andree <matthias.andree@...> writes:

> sorry for that.  Are there logs beyond what you've quoted?

I found most of my problem: duplicate message-ids being strangely handled.

I had two files, a and b, both with the same message-id.  a was linked into
message.id, but b wasn't.

When I ran texpire -D 257 -vvv -f -e -n wg21.c++.all, it told me it was going
to delete a, but that it was *also* going to delete b because it wasn't in
message.id.

Then I ran texpire -vvv -f -r -n wg21.c++.all, which fixed up the hard links
in message.id.  *Then*, when I ran the first command again, it reported it was
only going to delete a now, and not b anymore.

Otherwise, the deletions it proposed all seem like legitimate duplicates after
analysis.  The only bug here is that it would have deleted both a and b until
I repaired the message.id hard links for that group.  In essence this means
that, to be safe, I have to always run this:

  texpire -vvv -f -r -n
  texpire -vvv -f

> Are you running leafnode on a 64-bit computer?  Please show me the output of
> "leafnode-version -v".

Yes:
(Continue reading)

Permalink | Reply |
headers
Matthias Andree | 10 Sep 2011 12:51
Picon
Picon

Re: texpire deleting more articles than it should

Am 10.09.2011 01:08, schrieb John Wiegley:
>>>>>> Matthias Andree <matthias.andree@...> writes:
> 
>> sorry for that.  Are there logs beyond what you've quoted?
> 
> I found most of my problem: duplicate message-ids being strangely handled.
> 
> I had two files, a and b, both with the same message-id.  a was linked into
> message.id, but b wasn't.
> 
> When I ran texpire -D 257 -vvv -f -e -n wg21.c++.all, it told me it was going
> to delete a, but that it was *also* going to delete b because it wasn't in
> message.id.
> 
> Then I ran texpire -vvv -f -r -n wg21.c++.all, which fixed up the hard links
> in message.id.  *Then*, when I ran the first command again, it reported it was
> only going to delete a now, and not b anymore.

John,

So something on your computer has broken the hard links.

Does Git handle and restore hard links?  I never bothered to check
because I never needed such a functionality.  If Git instead handles a
hard linked file as two separate files that happen to have identical
content, it would store it quite efficiently (because the delta is null
which compresses quite well too) but break leafnode's spool.

Have you ever copied or moved the spool?
(Continue reading)

Permalink | Reply |
headers
John Wiegley | 10 Sep 2011 23:29
Picon
Gravatar

Re: texpire deleting more articles than it should

>>>>> Matthias Andree <matthias.andree@...> writes:

> So something on your computer has broken the hard links.

> Does Git handle and restore hard links?  I never bothered to check because I
> never needed such a functionality.  If Git instead handles a hard linked
> file as two separate files that happen to have identical content, it would
> store it quite efficiently (because the delta is null which compresses quite
> well too) but break leafnode's spool.

Git isn't modifying anything within the working tree.  I even told it to
ignore the /message.id/ directory.

> Have you ever copied or moved the spool?

I put a ton of articles into in.coming and using "fetchnews" to read them into
local groups.  It was this process that failed to establish hard links for the
duplicated messages.

> Restored things from backup?

Nope.

> Or upgraded from an older leafnode version without running texpire -r?

I upgraded to 2.x, but did texpire -r first thing.

> Typically fetchnews will NOT download duplicates (that can happen if you
> upgrade from leafnode-1 or across particular bug fixes in ancient leafnode-2
> versions).
(Continue reading)

Permalink | Reply |
headers
Matthias Andree | 11 Sep 2011 17:55
Picon
Picon

Re: texpire deleting more articles than it should

Am 10.09.2011 23:29, schrieb John Wiegley:
>>>>>> Matthias Andree <matthias.andree@...> writes:
> 
>> So something on your computer has broken the hard links.
> 
>> Does Git handle and restore hard links?  I never bothered to check because I
>> never needed such a functionality.  If Git instead handles a hard linked
>> file as two separate files that happen to have identical content, it would
>> store it quite efficiently (because the delta is null which compresses quite
>> well too) but break leafnode's spool.
> 
> Git isn't modifying anything within the working tree.  I even told it to
> ignore the /message.id/ directory.
> 
>> Have you ever copied or moved the spool?
> 
> I put a ton of articles into in.coming and using "fetchnews" to read them into
> local groups.  It was this process that failed to establish hard links for the
> duplicated messages.

Thanks, I'll have to look at that.

> These articles didn't come from a download, but from in.coming.

To narrow down the problem: were there copies of articles already ion
the spool (duplicates) in in.coming/?
Permalink | Reply |
headers
John Wiegley | 12 Sep 2011 07:21
Picon
Gravatar

Re: texpire deleting more articles than it should

>>>>> Matthias Andree <matthias.andree@...> writes:

> To narrow down the problem: were there copies of articles already ion the
> spool (duplicates) in in.coming/?

Yes.  And also, in.coming itself contained some duplicates.  I had thought
that all the duplicates would get ignored by fetchnews, but it looks like it
actually fed them into the local groups.

John
Permalink | Reply |
headers
Adam Funk | 10 Sep 2011 22:30
Favicon

Re: texpire deleting more articles than it should

On 2011-09-09, John Wiegley wrote:

>>>>>> Matthias Andree <matthias.andree@...> writes:
>
>> sorry for that.  Are there logs beyond what you've quoted?
>
> I found most of my problem: duplicate message-ids being strangely handled.
>
> I had two files, a and b, both with the same message-id.  a was linked into
> message.id, but b wasn't.

Just curious/nosy --- are the two files with the same MID actually
separate duplicate copies of the same message, or different messages
with a MID collision?
Permalink | Reply |
headers
John Wiegley | 10 Sep 2011 23:29
Picon
Gravatar

Re: texpire deleting more articles than it should

>>>>> Adam Funk <a24061@...> writes:

> Just curious/nosy --- are the two files with the same MID actually separate
> duplicate copies of the same message, or different messages with a MID
> collision?

They are gone now, but I think they were real dups.

John
Permalink | Reply |
headers
Adam Funk | 11 Sep 2011 17:17
Favicon

Re: texpire deleting more articles than it should

On 2011-09-10, John Wiegley wrote:

>>>>>> Adam Funk <a24061@...> writes:
>
>> Just curious/nosy --- are the two files with the same MID actually separate
>> duplicate copies of the same message, or different messages with a MID
>> collision?
>
> They are gone now, but I think they were real dups.

Weird, I can't think how that happened.
Permalink | Reply |
headers
Matthias Andree | 11 Sep 2011 17:45
Picon
Picon

Re: texpire deleting more articles than it should

Am 11.09.2011 17:17, schrieb Adam Funk:
> On 2011-09-10, John Wiegley wrote:
> 
>>>>>>> Adam Funk <a24061@...> writes:
>>
>>> Just curious/nosy --- are the two files with the same MID actually separate
>>> duplicate copies of the same message, or different messages with a MID
>>> collision?
>>
>> They are gone now, but I think they were real dups.
> 
> Weird, I can't think how that happened.

I posted possibilities to this thread:

http://www.dt.e-technik.uni-dortmund.de/pipermail/leafnode-list/2011q3/002400.html
Permalink | Reply |
headers
Adam Funk | 11 Sep 2011 21:13
Favicon

Re: texpire deleting more articles than it should

On 2011-09-11, Matthias Andree wrote:

> Am 11.09.2011 17:17, schrieb Adam Funk:
>> On 2011-09-10, John Wiegley wrote:
>> 
>>>>>>>> Adam Funk <a24061@...> writes:
>>>
>>>> Just curious/nosy --- are the two files with the same MID actually separate
>>>> duplicate copies of the same message, or different messages with a MID
>>>> collision?
>>>
>>> They are gone now, but I think they were real dups.
>> 
>> Weird, I can't think how that happened.
>
> I posted possibilities to this thread:
>
> http://www.dt.e-technik.uni-dortmund.de/pipermail/leafnode-list/2011q3/002400.html

OK, I was aware of the possibility (and the fix, using "texpire -r")
when copying or moving the spool (which I've done, when replacing a
hard drive), and I recall that some (but not many) version changes
require the same repair operation, but real duplication like that
can't happen on an unchanging (except for the spool content) system,
can it?

AIUI, leafnode checks messages against MIDs already in the spool when
downloading from upstream servers, right?

Does it reject messages from local news clients that have MIDs that
(Continue reading)

Permalink | Reply |

Gmane