hi,

tl;dr: i propose this patch to Text/JSON/String.hs and would like to
know why it is needed:

<at> <at> -375,7 +375,7 <at> <at>
   where
   go s1 =
     case s1 of
-      (x   :xs) | x < '\x20' || x > '\x7e' -> '\\' : encControl x (go xs)
+      (x   :xs) | x < '\x20' -> '\\' : encControl x (go xs)
       ('"' :xs)              -> '\\' : '"'  : go xs
       ('\\':xs)              -> '\\' : '\\' : go xs
       (x   :xs)              -> x    : go xs


i recently stumbled upon CouchDB telling me i'm sending invalid json.

i basically read lines from a utf8 file with german umlauts and send
them to CouchDB using Text.JSON and Database.CouchDB.

  $ file lines.txt
  lines.txt: UTF-8 Unicode text

lets take 'ö' as an example. i use LANG=de_DE.utf8

ghci tells

> 'ö'
'\246'

> putChar '\246'
ö

> putChar 'ö'
ö

> :m + Text.JSON Database.CouchDB
> runCouchDB' $ newNamedDoc (db "foo") (doc "bar") (showJSON $ toJSObject [("test","ö")])
*** Exception: HTTP/1.1 400 Bad Request
Server: CouchDB/1.2.1 (Erlang OTP/R15B03)
Date: Mon, 11 Feb 2013 13:24:49 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 48
Cache-Control: must-revalidate

couchdb log says:

  Invalid JSON: {{error,{10,"lexical error: invalid bytes in UTF8 string.\n"}},<<"{\"test\":\"<F6>\"}">>}

this is indeed hex ö:

> :m + Numeric
> putChar $ toEnum $ fst $ head $ readHex "f6"
ö

if i apply the above patch and reinstall JSON and CouchDB the doc
creation works:

> runCouchDB' $ newNamedDoc (db "db") (doc "foo") (showJSON $ toJSObject [("test", "ö")])
Right someRev

but i dont get back the ö i expected:

> Just (_,_,x) <-runCouchDB' $ getDoc (db "foo") (doc "bar") :: IO (Maybe (Doc,Rev,JSObject String))
> let Ok y = valFromObj "test" =<< readJSON x :: Result String
> y
"\195\188"
> putStrLn y
ü

apperently with curl everything works fine:

$ curl localhost:5984/db/foo -XPUT -d '{"test": "ö"}'
{"ok":true,"id":"foo","rev":"someOtherRev"}
$ curl localhost:5984/db/foo
{"_id":"bars","_rev":"someOtherRev","test":"ö"}

so how can i get my precious ö back? what am i doing wrong or does Text.JSON need another patch?

another question: why does encControl in Text/JSON/String.hs handle the
cases x < '\x100' and x < '\x1000' even though they can never be
reached with the old predicate in encJSString (x < '\x20')

finally: is '\x7e' the right literal for the job?

thanks for reading

have fun
martin

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

hi,

tl;dr: i propose this patch to Text/JSON/String.hs and would like to
know why it is needed:

<at> <at> -375,7 +375,7 <at> <at>
   where
   go s1 =
     case s1 of
-      (x   :xs) | x < '\x20' || x > '\x7e' -> '\\' : encControl x (go xs)
+      (x   :xs) | x < '\x20' -> '\\' : encControl x (go xs)
       ('"' :xs)              -> '\\' : '"'  : go xs
       ('\\':xs)              -> '\\' : '\\' : go xs
       (x   :xs)              -> x    : go xs


i recently stumbled upon CouchDB telling me i'm sending invalid json.

i basically read lines from a utf8 file with german umlauts and send
them to CouchDB using Text.JSON and Database.CouchDB.

  $ file lines.txt
  lines.txt: UTF-8 Unicode text

lets take 'ö' as an example. i use LANG=de_DE.utf8

ghci tells

> 'ö'
'\246'

> putChar '\246'
ö

> putChar 'ö'
ö

> :m + Text.JSON Database.CouchDB
> runCouchDB' $ newNamedDoc (db "foo") (doc "bar") (showJSON $ toJSObject [("test","ö")])
*** Exception: HTTP/1.1 400 Bad Request
Server: CouchDB/1.2.1 (Erlang OTP/R15B03)
Date: Mon, 11 Feb 2013 13:24:49 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 48
Cache-Control: must-revalidate

couchdb log says:

  Invalid JSON: {{error,{10,"lexical error: invalid bytes in UTF8 string.\n"}},<<"{\"test\":\"<F6>\"}">>}

this is indeed hex ö:

> :m + Numeric
> putChar $ toEnum $ fst $ head $ readHex "f6"
ö

if i apply the above patch and reinstall JSON and CouchDB the doc
creation works:

> runCouchDB' $ newNamedDoc (db "db") (doc "foo") (showJSON $ toJSObject [("test", "ö")])
Right someRev

but i dont get back the ö i expected:

> Just (_,_,x) <-runCouchDB' $ getDoc (db "foo") (doc "bar") :: IO (Maybe (Doc,Rev,JSObject String))
> let Ok y = valFromObj "test" =<< readJSON x :: Result String
> y
"\195\188"
> putStrLn y
ü

apperently with curl everything works fine:

$ curl localhost:5984/db/foo -XPUT -d '{"test": "ö"}'
{"ok":true,"id":"foo","rev":"someOtherRev"}
$ curl localhost:5984/db/foo
{"_id":"bars","_rev":"someOtherRev","test":"ö"}

so how can i get my precious ö back? what am i doing wrong or does Text.JSON need another patch?

another question: why does encControl in Text/JSON/String.hs handle the
cases x < '\x100' and x < '\x1000' even though they can never be
reached with the old predicate in encJSString (x < '\x20')

finally: is '\x7e' the right literal for the job?

thanks for reading

have fun
martin

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe