Discussion of Python object-relational mapper SQLObject

Tom Coetser | 31 May 2012 08:08

PostgreSQL warnings filling up the logs with PickleCol

Hi all,

I am getting a warning in the Postgres log for every time a PickleCol is 
written to the database. Since my app uses a pickled col extensively in one of 
the more frequently used tables, the Postgres logs are filled with these 
warnings:

----------------------
WARNING:  nonstandard use of \\ in a string literal at character 50
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
----------------------

This is with SQLObject v1.3.0, but it has been there from at least v0.12.4 
from which I am upgrading to 1.3.0 now.

PostgresSQL is v8.4.11 on Debian Squeeze.

An example test case to illustrate the problem is inline below.

Any help on how to solve this so I can get my log files cleaned up, in order 
to debug another connection problem, would be greatly appreciated.

Thanks,
 Tom

-------------------------------
from sqlobject import *

db = dict(user='tomc',
          passw='',

(Continue reading)

Oleg Broytman | 31 May 2012 09:44

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Hello!

On Thu, May 31, 2012 at 08:08:42AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> I am getting a warning in the Postgres log for every time a PickleCol is 
> written to the database. Since my app uses a pickled col extensively in one of 
> the more frequently used tables, the Postgres logs are filled with these 
> warnings:
> 
> ----------------------
> WARNING:  nonstandard use of \\ in a string literal at character 50
> HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
> ----------------------

   Thank you for an example of a good bug report! An error message and a
short test case - exactly what I need.
   Can you apply and test the attached patch?

> Any help on how to solve this so I can get my log files cleaned up, in order 
> to debug another connection problem, would be greatly appreciated.

   What log files? /var/log/postgresql/*.log?

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

Attachment (pgconnection.py.patch): text/x-diff, 465 bytes

------------------------------------------------------------------------------

(Continue reading)

Tom Coetser | 31 May 2012 10:31

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi Oleg,

Thanks for the quick reply.

On Thursday 31 May 2012 09:44:05 Oleg Broytman wrote:
> Hello!
> 
> On Thu, May 31, 2012 at 08:08:42AM +0200, Tom Coetser <subs <at> icave.net> 
wrote:
> > I am getting a warning in the Postgres log for every time a PickleCol is
> > written to the database. Since my app uses a pickled col extensively in
> > one of the more frequently used tables, the Postgres logs are filled
> > with these warnings:
> > 
> > ----------------------
> > WARNING:  nonstandard use of \\ in a string literal at character 50
> > HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
> > ----------------------
> 
>    Thank you for an example of a good bug report! An error message and a
> short test case - exactly what I need.
>    Can you apply and test the attached patch?

Thanks, seems to solve the problem nicely. I assume this will go into the next 
release for 1.3, correct?

> > Any help on how to solve this so I can get my log files cleaned up, in
> > order to debug another connection problem, would be greatly appreciated.
> 
>    What log files? /var/log/postgresql/*.log?

(Continue reading)

Oleg Broytman | 31 May 2012 10:43

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

On Thu, May 31, 2012 at 10:31:33AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> Thanks, seems to solve the problem nicely. I assume this will go into the next 
> release for 1.3, correct?

   It will go to the repository today, but formal releases will be much
later. I'll be on vacation (and completely off-line) until June 20.

> > > Any help on how to solve this so I can get my log files cleaned up, in
> > > order to debug another connection problem, would be greatly appreciated.
> > 
> >    What log files? /var/log/postgresql/*.log?
> 
> Yes, but specifically /var/log/postgresql/postgresql-8.4-main.log on Debian 
> Squeeze.

/etc/init.d/postgresql stop
rm /var/log/postgresql/* # or rm *.gz && gzip *.log
/etc/init.d/postgresql start

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

(Continue reading)

Oleg Broytman | 31 May 2012 10:57

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

On Thu, May 31, 2012 at 11:44:05AM +0400, Oleg Broytman <phd <at> phdru.name> wrote:
>    Can you apply and test the attached patch?

   Please be warned - the patch causes failures of test_blob,
test_picklecol and test_validation. Seems I have to investigate it
deeper.

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Tom Coetser | 31 May 2012 11:35

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi Oleg,

On Thursday 31 May 2012 10:57:11 Oleg Broytman wrote:
> On Thu, May 31, 2012 at 11:44:05AM +0400, Oleg Broytman <phd <at> phdru.name> 
wrote:
> >    Can you apply and test the attached patch?
> 
>    Please be warned - the patch causes failures of test_blob,
> test_picklecol and test_validation. Seems I have to investigate it
> deeper.

Thanks for letting me know. So far unit tests on my app seems to be OK with 
the patch, but I'll wait to hear more from your side before I deploy the 
change.

Thanks,
 Tom

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Oleg Broytman | 31 May 2012 12:01

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

On Thu, May 31, 2012 at 11:35:31AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> On Thursday 31 May 2012 10:57:11 Oleg Broytman wrote:
> > On Thu, May 31, 2012 at 11:44:05AM +0400, Oleg Broytman <phd <at> phdru.name> 
> wrote:
> > >    Can you apply and test the attached patch?
> > 
> >    Please be warned - the patch causes failures of test_blob,
> > test_picklecol and test_validation. Seems I have to investigate it
> > deeper.
> 
> Thanks for letting me know. So far unit tests on my app seems to be OK with 
> the patch, but I'll wait to hear more from your side before I deploy the 
> change.

   I use utf-8 encoding, and SQLObject tests generates binary data. The
reason for failures is that the patch makes SQLObject produce invalid
utf-8 sequences. If your databases and client/server encoding is ascii
or latin1 - I think the patch is ok.

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

(Continue reading)

Oleg Broytman | 31 May 2012 16:16

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Ok, found the problem - str() removes one level of backslashes so I have
to double backslashes once again. See the attached patch - it passes all
tests.

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

Attachment (pgconnection.py.patch): text/x-diff, 668 bytes

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss

Tom Coetser | 1 Jun 2012 10:06

Re: PostgreSQL warnings filling up the logs with PickleCol

Hey Oleg,

On Thursday 31 May 2012 16:16:41 Oleg Broytman wrote:
> Ok, found the problem - str() removes one level of backslashes so I have
> to double backslashes once again. See the attached patch - it passes all
> tests.

When I use the new patch on my test case, after adding some extended 
characters to the pickle column and setting the encoding to utf-8 (see below), 
i get this error:

------------------------
$ python pickleTest.py 
 1/QueryIns:  INSERT INTO server (id, config, name) VALUES (1,

E'\\\\200\\\\002}q\\\\001(U\\\\005otherq\\\\002X\\\\010\\\\000\\\\000\\\\000f\\\\303\\\\266\\\\303\\\\266barq\\\\003U\\\\006remoteq\\\\004\\\\211U\\\\003dnsq\\\\005}q\\\\006(U\\\\006domainq\\\\007U\\\\007foo.barq\\\\010U\\\\010hostnameq\\\\011NuU\\\\010servicesq\\\\012]q\\\\013(U\\\\003webq\\\\014U\\\\010databaseq\\\\015eU\\\\006loggerq\\\\016NU\\\\005portsq\\\\017K\\\\002u.'::bytea, 
'foo')
 1/COMMIT  :  auto
 1/QueryOne:  SELECT name, config FROM server WHERE ((server.id) = (1))
 1/QueryR  :  SELECT name, config FROM server WHERE ((server.id) = (1))
 1/COMMIT  :  auto
Traceback (most recent call last):
  File "pickleTest.py", line 37, in <module>
    s1 = Server(name='foo')
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1223, in 
__init__
    self._create(id, **kw)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1271, in _create
    self._SO_finishCreate(id)
  File "/usr/lib/pymodules/python2.6/sqlobject/main.py", line 1298, in

(Continue reading)

Oleg Broytman | 1 Jun 2012 15:19

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi!

On Fri, Jun 01, 2012 at 10:06:15AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> When I use the new patch on my test case, after adding some extended 
> characters to the pickle column and setting the encoding to utf-8 (see below), 
> i get this error:

   I cannot promise I'll debug this before vacation. I'm busy packing my
sacks. Sorry.

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Oleg Broytman | 21 Jun 2012 09:51

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi! I've come back.

On Fri, Jun 01, 2012 at 10:06:15AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> On Thursday 31 May 2012 16:16:41 Oleg Broytman wrote:
> > Ok, found the problem - str() removes one level of backslashes so I have
> > to double backslashes once again. See the attached patch - it passes all
> > tests.
> 
> When I use the new patch on my test case, after adding some extended 
> characters to the pickle column and setting the encoding to utf-8 (see below), 
> i get this error:
> 
> ------------------------
> $ python pickleTest.py 
>  1/QueryIns:  INSERT INTO server (id, config, name) VALUES (1, 
>
E'\\\\200\\\\002}q\\\\001(U\\\\005otherq\\\\002X\\\\010\\\\000\\\\000\\\\000f\\\\303\\\\266\\\\303\\\\266barq\\\\003U\\\\006remoteq\\\\004\\\\211U\\\\003dnsq\\\\005}q\\\\006(U\\\\006domainq\\\\007U\\\\007foo.barq\\\\010U\\\\010hostnameq\\\\011NuU\\\\010servicesq\\\\012]q\\\\013(U\\\\003webq\\\\014U\\\\010databaseq\\\\015eU\\\\006loggerq\\\\016NU\\\\005portsq\\\\017K\\\\002u.'::bytea, 
> 'foo')

   Very strange. It seems I have to redouble backslashes in my case but
not in yours. Puzzled...

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and

(Continue reading)

Oleg Broytman | 22 Jun 2012 21:42

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Tom, can I see the values for the following configuration parameters:

grep 'standard_conforming_strings\|escape_string_warning\|backslash_quote' /etc/postgresql/9.1/main/postgresql.conf

   Mine are:

#backslash_quote = safe_encoding # on, off, or safe_encoding
#escape_string_warning = on
#standard_conforming_strings = on

   I'm trying to understand the difference in our setup...

Oleg.
--

-- 
     Oleg Broytman            http://phdru.name/            phd <at> phdru.name
           Programmers don't die, they just GOSUB without RETURN.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Tom Coetser | 25 Jun 2012 06:49

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi Oleg,

On Friday 22 June 2012 21:42:28 Oleg Broytman wrote:
> Tom, can I see the values for the following configuration parameters:
> 
> grep 'standard_conforming_strings\|escape_string_warning\|backslash_quote'
> /etc/postgresql/9.1/main/postgresql.conf
> 
>    Mine are:
> 
> #backslash_quote = safe_encoding # on, off, or safe_encoding
> #escape_string_warning = on
> #standard_conforming_strings = on

I am still running PostgreSQL 8.4 :

$ grep 'standard_conforming_strings\|escape_string_warning\|backslash_quote' 
/etc/postgresql/8.4/main/postgresql.conf

#backslash_quote = safe_encoding        # on, off, or safe_encoding
#escape_string_warning = on
#standard_conforming_strings = off

>    I'm trying to understand the difference in our setup...

Could it be the PostgreSQL version?

Thanks,
 Tom

(Continue reading)

Oleg Broytman | 25 Jun 2012 20:50

X-Face

Re: PostgreSQL warnings filling up the logs with PickleCol

Hi!

On Mon, Jun 25, 2012 at 06:49:03AM +0200, Tom Coetser <subs <at> icave.net> wrote:
> On Friday 22 June 2012 21:42:28 Oleg Broytman wrote:
> > Tom, can I see the values for the following configuration parameters:
> > 
> > grep 'standard_conforming_strings\|escape_string_warning\|backslash_quote'
> > /etc/postgresql/9.1/main/postgresql.conf
> > 
> >    Mine are:
> > 
> > #backslash_quote = safe_encoding # on, off, or safe_encoding
> > #escape_string_warning = on
> > #standard_conforming_strings = on
> 
> I am still running PostgreSQL 8.4 :
> 
> $ grep 'standard_conforming_strings\|escape_string_warning\|backslash_quote' 
> /etc/postgresql/8.4/main/postgresql.conf
> 
> #backslash_quote = safe_encoding        # on, off, or safe_encoding
> #escape_string_warning = on
> #standard_conforming_strings = off

   Thanks. Everything is default. Ok.

> >    I'm trying to understand the difference in our setup...
> 
> Could it be the PostgreSQL version?

(Continue reading)

Return

Return to gmane.comp.python.sqlobject.

Project Web Page

Discussion of Python object-relational mapper SQLObject

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane