Martin Weissenboeck | 4 Aug 2012 00:23
Picon

strange response.flash

I have tried: response.flash="Auswählen" (German for "select") and I got "Ausw%C3%A4hlen". Looks like urllib.quote, but why?
And in some other contexts  I got "Auswählen" as expected. I could not find any rule.

I am using the trunk from yesterday.

Then I wrote some test functions. I have written the results after each block:

def flash1():
    redirect (URL('flash1a', args=['hello there']))
   
def flash1a():
    response.flash=request.args[0]
    return dict(text='done')

Result:  hello_there  (with an underscore in place of the space)
------------------------------------------------------

def flash2():
    redirect (URL('flash2a', args=['hello there äöü']))
   
def flash2a():
    response.flash=request.args[0]
    return dict(text='done')

Result: Invalid request. I thought "args" uses urllib.quote?
------------------------------------------------------

def flash3():
    redirect (URL('flash3a', vars=dict(t='hello there äöü')))
   
def flash3a():
    response.flash=request.vars['t']
    return dict(text='done')

Result: hello there äöü  ok!
------------------------------------------------------

import urllib   
   
def flash4():
    response.flash='hello there äöü'
    return dict(text='done')
   
Result:  hello there äöü  ok
------------------------------------------------------

def flash4a():
    response.flash=urllib.quote('hello there äöü')
    return dict(text='done')
       
Result: hello%20there%20%C3%A4%C3%B6%C3%BC  ok
------------------------------------------------------

def flash5():
    redirect (URL('flash5a', args=[urllib.quote('hello there äöü')]))
   
def flash5a():
    response.flash=urllib.unquote(request.args[0])
    return dict(text='done')   

Result: invalid request   
------------------------------------------------------

def flash6():
    return dict(load=LOAD('default', 'flash6a.load'))
   
def flash6a():
    response.flash='hello there äöü'
    return dict()

Result: nothing!!
------------------------------------------------------

def flash7():
    return dict(load=LOAD('default', 'flash7a.load', vars=dict(t='hello there äöü')))
   
def flash7a():
    response.flash=request.vars['t']
    return dict()

Result: nothing!

Some strange results, isn't it?

All functions again if someone wants to try it himself:

def flash1():
    redirect (URL('flash1a', args=['hello there']))
   
def flash1a():
    response.flash=request.args[0]
    return dict(text='done')

def flash2():
    redirect (URL('flash2a', args=['hello there äöü']))
   
def flash2a():
    response.flash=request.args[0]
    return dict(text='done')

def flash3():
    redirect (URL('flash3a', vars=dict(t='hello there äöü')))
   
def flash3a():
    response.flash=request.vars['t']
    return dict(text='done')
   
def flash4():
    response.flash='hello there äöü'
    return dict(text='done')

import urllib   
   
def flash4a():
    response.flash=urllib.quote('hello there äöü')
    return dict(text='done')
       
def flash5():
    redirect (URL('flash5a', args=[urllib.quote('hello there äöü')]))
   
def flash5a():
    response.flash=urllib.unquote(request.args[0])
    return dict(text='done')   
   
def flash6():
    return dict(load=LOAD('default', 'flash6a.load'))
   
def flash6a():
    response.flash='hello there äöü'
    return dict()

def flash7():
    return dict(load=LOAD('default', 'flash7a.load', vars=dict(flash='hello there äöü')))
   
def flash7a():
    response.flash=request.vars.flash
    return dict()

Any answers welcome!
Regards, Martin

--
 
 
 
Anthony | 4 Aug 2012 00:35
Picon

Re: strange response.flash

This issue is not with response.flash but with request.args. The characters allowed in args are fairly restrictive -- here are the regexes used: http://code.google.com/p/web2py/source/browse/gluon/rewrite.py#51http://code.google.com/p/web2py/source/browse/gluon/rewrite.py#575.


In the LOAD() examples, try setting ajax=True and the flash should work.

Anthony

On Friday, August 3, 2012 6:23:53 PM UTC-4, mweissen wrote:
I have tried: response.flash="Auswählen" (German for "select") and I got "Ausw%C3%A4hlen". Looks like urllib.quote, but why?
And in some other contexts  I got "Auswählen" as expected. I could not find any rule.

I am using the trunk from yesterday.

Then I wrote some test functions. I have written the results after each block:

def flash1():
    redirect (URL('flash1a', args=['hello there']))
   
def flash1a():
    response.flash=request.args[0]
    return dict(text='done')

Result:  hello_there  (with an underscore in place of the space)
------------------------------------------------------

def flash2():
    redirect (URL('flash2a', args=['hello there äöü']))
   
def flash2a():
    response.flash=request.args[0]
    return dict(text='done')

Result: Invalid request. I thought "args" uses urllib.quote?
------------------------------------------------------

def flash3():
    redirect (URL('flash3a', vars=dict(t='hello there äöü')))
   
def flash3a():
    response.flash=request.vars['t']
    return dict(text='done')

Result: hello there äöü  ok!
------------------------------------------------------

import urllib   
   
def flash4():
    response.flash='hello there äöü'
    return dict(text='done')
   
Result:  hello there äöü  ok
------------------------------------------------------

def flash4a():
    response.flash=urllib.quote('hello there äöü')
    return dict(text='done')
       
Result: hello%20there%20%C3%A4%C3%B6%C3%BC  ok
------------------------------------------------------

def flash5():
    redirect (URL('flash5a', args=[urllib.quote('hello there äöü')]))
   
def flash5a():
    response.flash=urllib.unquote(request.args[0])
    return dict(text='done')   

Result: invalid request   
------------------------------------------------------

def flash6():
    return dict(load=LOAD('default', 'flash6a.load'))
   
def flash6a():
    response.flash='hello there äöü'
    return dict()

Result: nothing!!
------------------------------------------------------

def flash7():
    return dict(load=LOAD('default', 'flash7a.load', vars=dict(t='hello there äöü')))
   
def flash7a():
    response.flash=request.vars['t']
    return dict()

Result: nothing!

Some strange results, isn't it?

All functions again if someone wants to try it himself:

def flash1():
    redirect (URL('flash1a', args=['hello there']))
   
def flash1a():
    response.flash=request.args[0]
    return dict(text='done')

def flash2():
    redirect (URL('flash2a', args=['hello there äöü']))
   
def flash2a():
    response.flash=request.args[0]
    return dict(text='done')

def flash3():
    redirect (URL('flash3a', vars=dict(t='hello there äöü')))
   
def flash3a():
    response.flash=request.vars['t']
    return dict(text='done')
   
def flash4():
    response.flash='hello there äöü'
    return dict(text='done')

import urllib   
   
def flash4a():
    response.flash=urllib.quote('hello there äöü')
    return dict(text='done')
       
def flash5():
    redirect (URL('flash5a', args=[urllib.quote('hello there äöü')]))
   
def flash5a():
    response.flash=urllib.unquote(request.args[0])
    return dict(text='done')   
   
def flash6():
    return dict(load=LOAD('default', 'flash6a.load'))
   
def flash6a():
    response.flash='hello there äöü'
    return dict()

def flash7():
    return dict(load=LOAD('default', 'flash7a.load', vars=dict(flash='hello there äöü')))
   
def flash7a():
    response.flash=request.vars.flash
    return dict()

Any answers welcome!
Regards, Martin

--
 
 
 
Martin Weissenboeck | 4 Aug 2012 11:53
Picon

Re: Re: strange response.flash

Hi Anthony, thank you!

(1)  Restricted character for args - that is ok!

(2) ajax=True:

def flash6():
    return dict(load=LOAD('default', 'flash6a.load', ajax=True))
   
def flash6a():
    response.flash='hello there äöü'
    return dict()

Same result, nothing to see.
But

def flash6():
    return dict(load=LOAD('default', 'flash6a.load', ajax=True))
   
def flash6a():
    response.flash='hello there'
    return dict()

without umlaut works.

(3) response.flash="Auswählen" (German for "select") writes sometime
"Ausw%C3%A4hlen" and sometimes "Auswählen". Why?

Regards, Martin


2012/8/4 Anthony <abastardi-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
This issue is not with response.flash but with request.args. The characters allowed in args are fairly restrictive -- here are the regexes used: http://code.google.com/p/web2py/source/browse/gluon/rewrite.py#51http://code.google.com/p/web2py/source/browse/gluon/rewrite.py#575.

In the LOAD() examples, try setting ajax=True and the flash should work.

Anthony


--
 
 
 
Anthony | 4 Aug 2012 17:53
Picon

[web2py-dev] Re: [web2py] Re: strange response.flash

I see. I think the problem is only with response.flash within Ajax components. The message is escaped on the server via urllib2.quote, and then decoded in the browser via decodeURIComponent (see source code):


jQuery('.flash').html(decodeURIComponent(flash)).slideDown();

The problem is that if there are any ascii encodings in flash, decodeURIComponent seems to expect a valid URI and throws an error otherwise, which is what is happening when the unicode characters are included. A previous version of web2py.js did the escaping in Javascript on the client side, but with the same effect.

A fix might be to use xmlescape() to do any escaping on the server side (which is effectively the same as the escaping of a regular flash message), and then don't do any escaping or decoding on the client side -- so the above line would change to:

jQuery('.flash').html(flash).slideDown();

To do the server-side escaping, I think we can change line 552 in main.py from:

urllib2.quote(str(response.flash).replace('\n',''))

to:

xmlescape(response.flash).replace('\n','')

(Would also have to import xmlescape from html.py.)

Anthony

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Martin Weissenboeck | 4 Aug 2012 22:41
Picon

Re: Re: strange response.flash

I have tried

xmlescape(response.flash).replace('\n','')
Yes, this solves the problems. Would be nice to have it in trunk.

Thank you!

2012/8/4 Anthony <abastardi-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
I see. I think the problem is only with response.flash within Ajax components. The message is escaped on the server via urllib2.quote, and then decoded in the browser via decodeURIComponent (see source code):

jQuery('.flash').html(decodeURIComponent(flash)).slideDown();

The problem is that if there are any ascii encodings in flash, decodeURIComponent seems to expect a valid URI and throws an error otherwise, which is what is happening when the unicode characters are included. A previous version of web2py.js did the escaping in Javascript on the client side, but with the same effect.

A fix might be to use xmlescape() to do any escaping on the server side (which is effectively the same as the escaping of a regular flash message), and then don't do any escaping or decoding on the client side -- so the above line would change to:

jQuery('.flash').html(flash).slideDown();

To do the server-side escaping, I think we can change line 552 in main.py from:

urllib2.quote(str(response.flash).replace('\n',''))

to:

xmlescape(response.flash).replace('\n','')

(Would also have to import xmlescape from html.py.)

Anthony

--
 
 
 
Vladyslav Kozlovskyy | 5 Aug 2012 01:00
Picon
Gravatar

Re: Re: strange response.flash

I do not understand what the problem with decodeURIComponent()?

your flash6 and flash7 tests return "hello there äöü" for me (in trunk):



But this solution does not work for me:

$ web2py.py -S welcome
...
>>> from html import xmlescape
>>> xmlescape('проба')
'\xd0\xbf\xd1\x80\xd0\xbe\xd0\xb1\xd0\xb0'
>>> import urllib2
>>> urllib2.quote('проба')
'%D0%BF%D1%80%D0%BE%D0%B1%D0%B0'



У сб, 2012-08-04 у 22:41 +0200, Martin Weissenboeck пише:
I have tried
xmlescape(response.flash).replace('\n','')
Yes, this solves the problems. Would be nice to have it in trunk.

Thank you!

2012/8/4 Anthony <abastardi <at> gmail.com>
I see. I think the problem is only with response.flash within Ajax components. The message is escaped on the server via urllib2.quote, and then decoded in the browser via decodeURIComponent (see source code):


jQuery('.flash').html(decodeURIComponent(flash)).slideDown();

The problem is that if there are any ascii encodings in flash, decodeURIComponent seems to expect a valid URI and throws an error otherwise, which is what is happening when the unicode characters are included. A previous version of web2py.js did the escaping in Javascript on the client side, but with the same effect.

A fix might be to use xmlescape() to do any escaping on the server side (which is effectively the same as the escaping of a regular flash message), and then don't do any escaping or decoding on the client side -- so the above line would change to:


jQuery('.flash').html(flash).slideDown();

To do the server-side escaping, I think we can change line 552 in main.py from:


urllib2.quote(str(response.flash).replace('\n',''))


to:


xmlescape(response.flash).replace('\n','')

(Would also have to import xmlescape from html.py.)


Anthony



--
 
 
 

--
 
 
 
Anthony | 5 Aug 2012 03:23
Picon

[web2py-dev] Re: [web2py] Re: strange response.flash

On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:

I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony



--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Anthony | 6 Aug 2012 04:50
Picon

[web2py-dev] Re: [web2py] Re: strange response.flash

Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?


Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony



--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Niphlod | 6 Aug 2012 05:22
Picon

[web2py-dev] Re: [web2py] Re: strange response.flash

Would serialized HTML messages be escaped then ?

What if someone uses response.flash = A("Hello World", _href="#") ?

On Monday, August 6, 2012 4:50:31 AM UTC+2, Anthony wrote:

Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?

Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony



--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Massimo DiPierro | 6 Aug 2012 05:45
Picon

Re: [web2py-dev] [web2py] Re: strange response.flash

I think

    response.flash = A("Hello World", _href="#") 

should be allowed. It was always allowed. This is a backward compatibility issue. Yet I see there is a potential security issue there.
I am not sure what the exact solution should be. Perhaps automatic sanitization of flash messages before response?

    response.flash = XML(str(response.flash), sanitize=True).xml()


On Aug 5, 2012, at 10:22 PM, Niphlod wrote:

Would serialized HTML messages be escaped then ?

What if someone uses response.flash = A("Hello World", _href="#") ?

On Monday, August 6, 2012 4:50:31 AM UTC+2, Anthony wrote:
Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?

Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony





--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Anthony | 6 Aug 2012 06:21
Picon

Re: [web2py-dev] Re: strange response.flash

With xmlescape, HTML helpers should still work fine -- xmlescape does not escape helpers, only the components within the helpers (I think it basically replicates the escaping behavior of web2py views):


>>> xmlescape(A("Hello World", _href="#"))
'<a href="#">Hello World</a>'

However, with xmlescape, raw HTML will now be escaped:

>>> xmlescape('<a href="#">Hello World</a>')
'&lt;a href=&quot;#&quot;&gt;Hello World&lt;/a&gt;'

Of course, that can be avoided as usual with XML():

>>> xmlescape(XML('<a href="#">Hello World</a>'))
'<a href="#">Hello World</a>'

I see this as fixing a security bug rather than a backward compatibility issue. Regular flash messages are escaped, along with everything else in the view -- why not escape Ajax flash messages?

Anthony

On Sunday, August 5, 2012 11:45:21 PM UTC-4, Massimo Di Pierro wrote:
I think

    response.flash = A("Hello World", _href="#") 

should be allowed. It was always allowed. This is a backward compatibility issue. Yet I see there is a potential security issue there.
I am not sure what the exact solution should be. Perhaps automatic sanitization of flash messages before response?

    response.flash = XML(str(response.flash), sanitize=True).xml()


On Aug 5, 2012, at 10:22 PM, Niphlod wrote:

Would serialized HTML messages be escaped then ?

What if someone uses response.flash = A("Hello World", _href="#") ?

On Monday, August 6, 2012 4:50:31 AM UTC+2, Anthony wrote:
Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?

Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony





--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers <at> googlegroups.com
unsubscribe: web2py-developers+unsubscribe <at> googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

--
 
 
 
Massimo DiPierro | 6 Aug 2012 13:29
Picon

Re: [web2py-dev] [web2py] Re: strange response.flash

I had missed this message. I think you are right and this is the right solution. It is in trunk now. I will run more tests later today.

On Aug 5, 2012, at 11:21 PM, Anthony wrote:

With xmlescape, HTML helpers should still work fine -- xmlescape does not escape helpers, only the components within the helpers (I think it basically replicates the escaping behavior of web2py views):

>>> xmlescape(A("Hello World", _href="#"))
'<a href="#">Hello World</a>'

However, with xmlescape, raw HTML will now be escaped:

>>> xmlescape('<a href="#">Hello World</a>')
'&lt;a href=&quot;#&quot;&gt;Hello World&lt;/a&gt;'

Of course, that can be avoided as usual with XML():

>>> xmlescape(XML('<a href="#">Hello World</a>'))
'<a href="#">Hello World</a>'

I see this as fixing a security bug rather than a backward compatibility issue. Regular flash messages are escaped, along with everything else in the view -- why not escape Ajax flash messages?

Anthony

On Sunday, August 5, 2012 11:45:21 PM UTC-4, Massimo Di Pierro wrote:
I think

    response.flash = A("Hello World", _href="#") 

should be allowed. It was always allowed. This is a backward compatibility issue. Yet I see there is a potential security issue there.
I am not sure what the exact solution should be. Perhaps automatic sanitization of flash messages before response?

    response.flash = XML(str(response.flash), sanitize=True).xml()


On Aug 5, 2012, at 10:22 PM, Niphlod wrote:

Would serialized HTML messages be escaped then ?

What if someone uses response.flash = A("Hello World", _href="#") ?

On Monday, August 6, 2012 4:50:31 AM UTC+2, Anthony wrote:
Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?

Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony





--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers <at> googlegroups.com
unsubscribe: web2py-developers+unsubscribe <at> googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 


--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Massimo Di Pierro | 6 Aug 2012 13:59
Picon

Re: [web2py-dev] [web2py] Re: strange response.flash

I think Anthony is right. His suggested fix is now in trunk. Give it a try.

On Sunday, 5 August 2012 23:21:53 UTC-5, Anthony wrote:

With xmlescape, HTML helpers should still work fine -- xmlescape does not escape helpers, only the components within the helpers (I think it basically replicates the escaping behavior of web2py views):

>>> xmlescape(A("Hello World", _href="#"))
'<a href="#">Hello World</a>'

However, with xmlescape, raw HTML will now be escaped:

>>> xmlescape('<a href="#">Hello World</a>')
'&lt;a href=&quot;#&quot;&gt;Hello World&lt;/a&gt;'

Of course, that can be avoided as usual with XML():

>>> xmlescape(XML('<a href="#">Hello World</a>'))
'<a href="#">Hello World</a>'

I see this as fixing a security bug rather than a backward compatibility issue. Regular flash messages are escaped, along with everything else in the view -- why not escape Ajax flash messages?

Anthony

On Sunday, August 5, 2012 11:45:21 PM UTC-4, Massimo Di Pierro wrote:
I think

    response.flash = A("Hello World", _href="#") 

should be allowed. It was always allowed. This is a backward compatibility issue. Yet I see there is a potential security issue there.
I am not sure what the exact solution should be. Perhaps automatic sanitization of flash messages before response?

    response.flash = XML(str(response.flash), sanitize=True).xml()


On Aug 5, 2012, at 10:22 PM, Niphlod wrote:

Would serialized HTML messages be escaped then ?

What if someone uses response.flash = A("Hello World", _href="#") ?

On Monday, August 6, 2012 4:50:31 AM UTC+2, Anthony wrote:
Bump. Should we replace str() with xmlescape() so Ajax flash messages get escaped, just like regular flash messages and everything else in the view?

Anthony

On Saturday, August 4, 2012 9:23:53 PM UTC-4, Anthony wrote:
On Saturday, August 4, 2012 7:00:18 PM UTC-4, dbdeveloper wrote:
I do not understand what the problem with decodeURIComponent()?

When I tried trunk, I think there was a problem with the encoding of my controller file (ANSI instead of UTF-8) -- in that case, I guess urllib2.quote didn't yield the correct output for decodeURIComponent (same problem in the earlier version, when the escaping was done on the client side via the Javascript escape() function). Now it works.

In any case, a remaining issue is that there's still no escaping of potentially dangerous content in the flash message. Everything written to HTML by web2py is typically escaped, including regular flash messages. The only content that isn't getting escaped are flash messages for Ajax components. To be consistent (and safe), we should probably escape those messages as well (you can always put them in an XML() if you don't want them escaped, as with any template content). In main.py, I replaced:

urllib2.quote(str(response.flash).replace('\n',''))

with:

urllib2.quote(xmlescape(response.flash).replace('\n',''))

With that change, the flash message still looks fine (see screenshot below).

Anthony





--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers <at> googlegroups.com
unsubscribe: web2py-developers+unsubscribe <at> googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 
Massimo DiPierro | 6 Aug 2012 16:58
Picon

Re: [web2py-dev] [web2py] Re: strange response.flash

Sorry. I misread your email.

Massimo

On Aug 4, 2012, at 10:53 AM, Anthony wrote:

I see. I think the problem is only with response.flash within Ajax components. The message is escaped on the server via urllib2.quote, and then decoded in the browser via decodeURIComponent (see source code):

jQuery('.flash').html(decodeURIComponent(flash)).slideDown();

The problem is that if there are any ascii encodings in flash, decodeURIComponent seems to expect a valid URI and throws an error otherwise, which is what is happening when the unicode characters are included. A previous version of web2py.js did the escaping in Javascript on the client side, but with the same effect.

A fix might be to use xmlescape() to do any escaping on the server side (which is effectively the same as the escaping of a regular flash message), and then don't do any escaping or decoding on the client side -- so the above line would change to:

jQuery('.flash').html(flash).slideDown();

To do the server-side escaping, I think we can change line 552 in main.py from:

urllib2.quote(str(response.flash).replace('\n',''))

to:

xmlescape(response.flash).replace('\n','')

(Would also have to import xmlescape from html.py.)

Anthony


--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-developers-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
unsubscribe: web2py-developers+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
 
 

Gmane