Discussion about the notmuch email software ()

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 05/14] new: use new URL-based filenames for messages

This commit breaks a bunch of tests; fixes follow.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 notmuch-new.c |   27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 938ae29..1f11b2c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
 <at>  <at>  -287,7 +287,7  <at>  <at>  add_files (notmuch_database_t *notmuch,
 {
     DIR *dir = NULL;
     struct dirent *entry = NULL;
-    char *next = NULL;
+    char *next = NULL, *path_uri = NULL;
     time_t fs_mtime, db_mtime;
     notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
     notmuch_message_t *message = NULL;
 <at>  <at>  -315,7 +315,16  <at>  <at>  add_files (notmuch_database_t *notmuch,

     fs_mtime = st.st_mtime;

-    status = notmuch_database_get_directory (notmuch, path, &directory);
+    /* maildir URIs should never have a hostname component, but
+     * uriparser doesn't parse paths correctly if they start with //,
+     * as in scheme://host//path.
+     */
+    if (path[0] == '/')

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 06/14] maildir URIs can be used in tags_to_maildir_flags

A better fix would probably be based on scheme.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 lib/message.cc |   51 ++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 46 insertions(+), 5 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index c9857f5..8ecec71 100644
--- a/lib/message.cc
+++ b/lib/message.cc
 <at>  <at>  -23,6 +23,7  <at>  <at> 

 #include <stdint.h>

+#include <uriparser/Uri.h>
 #include <gmime/gmime.h>

 struct visible _notmuch_message {
 <at>  <at>  -1093,7 +1094,6  <at>  <at>  notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
     {
 	filename = notmuch_filenames_get (filenames);
 	dir = _filename_is_in_maildir (filename);
-
 	if (! dir)
 	    continue;

 <at>  <at>  -1304,12 +1304,46  <at>  <at>  _new_maildir_filename (void *ctx,
     return filename_new;
 }

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 07/14] Update tests that need to see filenames to use URIs

This fixes all tests except atomicity, which should be next.

Signed-off-by: Ethan Glasser-Camp <ethan <at> betacantrips.com>
---
 test/emacs                   |    2 +-
 test/json                    |    4 ++--
 test/maildir-sync            |    7 ++++---
 test/multipart               |    4 ++--
 test/new                     |    6 +++---
 test/search-folder-coherence |    2 +-
 test/search-output           |    4 ++--
 test/test-lib.sh             |    3 +++
 8 files changed, 18 insertions(+), 14 deletions(-)

diff --git a/test/emacs b/test/emacs
index e9f954c..c08791e 100755
--- a/test/emacs
+++ b/test/emacs
 <at>  <at>  -621,7 +621,7  <at>  <at>  Stash my stashables
 id:"bought"
 bought
 inbox,stashtest
-${gen_msg_filename}
+${gen_msg_uri}
 http://mid.gmane.org/bought
 http://marc.info/?i=bought
 http://mail-archive.com/search?l=mid&q=bought
diff --git a/test/json b/test/json
index 6439788..be29fac 100755
--- a/test/json

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 08/14] Don't cache corpus.mail

corpus.mail has already been processed by notmuch-new, so it seems
like a good target to cache, but since filenames are no longer being
stored relative to the database, it isn't. Recopy on each test, or
else filenames from other tests will show up.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 test/test-lib.sh |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 195158c..def3760 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
 <at>  <at>  -441,7 +441,7  <at>  <at>  add_email_corpus ()
     else
 	cp -a $TEST_DIRECTORY/corpus ${MAIL_DIR}
 	notmuch new >/dev/null
-	cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
+	#cp -a ${MAIL_DIR} $TEST_DIRECTORY/corpus.mail
     fi
 }

--

-- 
1.7.9.5

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 09/14] Fix atomicity test to work without relocatable mailstores

Instead of assuming that the mailstore doesn't store its absolute
filenames, we use a symlink that can change back and forth. As long as
filenames contain this symlink, they can work in either the real
database, or the current snapshot.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 test/atomicity     |   10 +++++-----
 test/atomicity.gdb |   11 ++++++++---
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/test/atomicity b/test/atomicity
index 6df0a00..7b62ec7 100755
--- a/test/atomicity
+++ b/test/atomicity
 <at>  <at>  -49,13 +49,13  <at>  <at>  if test_require_external_prereq gdb; then
     rm $MAIL_DIR/.remove-dir/remove-directory-duplicate:2,
     rmdir $MAIL_DIR/.remove-dir

-    # Prepare a snapshot of the updated maildir.  The gdb script will
-    # update the database in this snapshot as it goes.
+    # Copy the mail database. We will run on this database concurrently.
     cp -ra $MAIL_DIR $MAIL_DIR.snap
-    cp ${NOTMUCH_CONFIG} ${NOTMUCH_CONFIG}.snap
-    NOTMUCH_CONFIG=${NOTMUCH_CONFIG}.snap notmuch config set database.path $MAIL_DIR.snap
-

+    # Use a symlink instead of the real path. This way, we can change the symlink,
+    # without filenames having to change.
+    mv $MAIL_DIR $MAIL_DIR.real

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 10/14] new: add "scan" option

This is just a quick hack to get started on adding an mbox backend.

The fact that the default maildir is scanned "automagically" is a
little weird, but it doesn't do any harm unless you decide to put mail
there that you really don't want indexed.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 notmuch-client.h |    9 +++++++++
 notmuch-config.c |   30 +++++++++++++++++++++++++++++-
 notmuch-new.c    |   18 ++++++++++++++++++
 test/config      |    1 +
 4 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 9b63eae..9d922fe 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
 <at>  <at>  -256,6 +256,15  <at>  <at>  notmuch_config_set_new_ignore (notmuch_config_t *config,
 			       const char *new_ignore[],
 			       size_t length);

+const char **
+notmuch_config_get_new_scan (notmuch_config_t *config,
+			       size_t *length);
+
+void
+notmuch_config_set_new_scan (notmuch_config_t *config,
+			       const char *new_scan[],
+			       size_t length);

(Continue reading)

Mark Walters | 28 Jun 2012 22:54

Picon

Re: [RFC PATCH 10/14] new: add "scan" option

On Mon, 25 Jun 2012, Ethan Glasser-Camp
<ethan.glasser.camp@...> wrote:
> This is just a quick hack to get started on adding an mbox backend.
>
> The fact that the default maildir is scanned "automagically" is a
> little weird, but it doesn't do any harm unless you decide to put mail
> there that you really don't want indexed.

I was a bit caught out by the way this worked. I expected that I would
add 
scan=mbox:///full_path_of_mbox 
but instead it seems that I need to add
scan=mbox:///full_path_of_directory_containing_the_mbox

I am happy with either (and yours is probably easier) but it should be
documented somewhere and I found the line
> +    "\tscan	A list (separated by ';') of mail URLs to scan.\n"

confusing.

Best wishes

Mark

> Signed-off-by: Ethan Glasser-Camp <ethan@...>
> ---
>  notmuch-client.h |    9 +++++++++
>  notmuch-config.c |   30 +++++++++++++++++++++++++++++-
>  notmuch-new.c    |   18 ++++++++++++++++++
>  test/config      |    1 +

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 11/14] notmuch-new: pull out useful bits of add_files_recursive

This is part of notmuch-new refactor phase 1: make add_files stuff
safe for other backends. add_files_recursive is essentially a
maildir-crawling function that periodically adds files to the database
or adds filenames to remove_files or remove_directory lists. I don't
see an easy way to adapt add_files_recursive for other backends who
might not have concepts of directories with other directories inside
of them, so instead just provide an add_files method for each backend.

This patch pulls some bits out of add_files_recursive which will be
useful for other backends: two reporting functions
_report_before_adding_file and _report_added_file, as well as
_add_message, which actually does the message adding.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 notmuch-new.c |  192 +++++++++++++++++++++++++++++++++++----------------------
 1 file changed, 119 insertions(+), 73 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 57b27bf..1bf4e25 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
 <at>  <at>  -249,6 +249,122  <at>  <at>  add_files_uri (unused(notmuch_database_t *notmuch),
     return NOTMUCH_STATUS_SUCCESS;
 }

+/* Progress-reporting function.
+ *
+ * Can be used by any mailstore-crawling function that wants to alert
+ * users what message it's about to add. Subsequent errors will be due

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 12/14] mailstore: support for mbox:// URIs


Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 lib/mailstore.c |   85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/lib/mailstore.c b/lib/mailstore.c
index ae02c12..e8d9bc1 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
 <at>  <at>  -19,6 +19,7  <at>  <at> 
  */
 #include <uriparser/Uri.h>
 #include <stdio.h>
+#include <glib.h>

 #include "notmuch-private.h"

 <at>  <at>  -28,6 +29,74  <at>  <at>  notmuch_mailstore_basic_open (const char *filename)
     return fopen (filename, "r");
 }

+/* Since we have to return a FILE*, we use fmemopen to turn buffers
+ * into FILE* streams. But when we close these streams, we have to
+ * free() the buffers. Use a hash to associate the two.
+ */
+static GHashTable *_mbox_files_to_strings = NULL;
+
+static void
+_ensure_mbox_files_to_strings () {

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 13/14] Tests for mbox support

These need to be improved, rather than hard-coding byte offsets.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 test/mbox         |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 test/notmuch-test |    1 +
 2 files changed, 60 insertions(+)
 create mode 100755 test/mbox

diff --git a/test/mbox b/test/mbox
new file mode 100755
index 0000000..f03f887
--- /dev/null
+++ b/test/mbox
 <at>  <at>  -0,0 +1,59  <at>  <at> 
+#!/usr/bin/env bash
+#
+# Copyright (c) 2005 Junio C Hamano
+#
+
+test_description='basic mbox support'
+. ./test-lib.sh
+
+mkdir -p $MAIL_DIR/some-mboxes/subdir $MAIL_DIR/database $MAIL_DIR/corpus
+
+# The Content-Length headers here include the final newline (added later).
+generate_message '[body]="Mbox message 1."' '[header]="Content-Length: 16"' "[dir]=corpus"
+generate_message '[body]="Mbox message 2. Longer."' '[header]="Content-Length: 24"' "[dir]=corpus"
+generate_message '[body]="Mbox message 3."' "[dir]=corpus"
+generate_message '[body]="Mbox message 4."' "[dir]=corpus"

(Continue reading)

Ethan Glasser-Camp | 25 Jun 2012 22:51

Picon

[RFC PATCH 14/14] new: Add scan support for mbox:// URIs

A lot of code is duplicated from maildir, I don't think I handled all
those errors correctly, and I didn't report any progress.

Signed-off-by: Ethan Glasser-Camp <ethan@...>
---
 notmuch-new.c |  299 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 289 insertions(+), 10 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 1bf4e25..36fee34 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
 <at>  <at>  -19,6 +19,7  <at>  <at> 
  */

 #include "notmuch-client.h"
+#include <uriparser/Uri.h>

 #include <unistd.h>

 <at>  <at>  -239,16 +240,6  <at>  <at>  _entry_in_ignore_list (const char *entry, add_files_state_t *state)
     return FALSE;
 }

-/* Call out to the appropriate add_files function, based on the URI. */
-static notmuch_status_t
-add_files_uri (unused(notmuch_database_t *notmuch),
-	       unused(const char *uri),
-	       unused(add_files_state_t *state))
-{

(Continue reading)

Return

Return to gmane.mail.notmuch.general.

Project Web Page

Discussion about the notmuch email software ()

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane