libcurl development talk. How to program using libcurl, and how to develop libcurl further

unkadoug | 22 Feb 17:22

Connections and multibyte responses

From the 7.24.0 sources, lib/http.c, line 2850:
/* On HTTP 1.1, when connection is not to get closed, but no
Content-Length nor Content-Encoding chunked have been
received, according to RFC2616 section 4.4 point 5, we
assume that the server will close the connection to
signal the end of the document. */

This assumption is invalid. Point 4 of the same section says:

If the message uses the media type "multipart/byteranges", and the
transfer-length is not otherwise specified, then this self-
delimiting media type defines the transfer-length. This media type
MUST NOT be used unless the sender knows that the recipient can parse
it; the presence in a request of a Range header with multiple byte-
range specifiers from a 1.1 client implies that the client can parse
multipart/byteranges responses.

If the curl client sends a range header with mulitple byte-range specifiers, and
gets a multipart/byteranges reply that does not explicitly close the connection,
it improperly awaits a signal:

> GET http://img.timeinc.net/time/daily/2012/1202/307_gs_india_china_0209.jpg HTTP/1.1
> Range: bytes=0-100,400-451
> User-Agent: curl/7.24.0 (i386-unknown-freebsd7.2) libcurl/7.24.0 OpenSSL/0.9.8q zlib/1.2.3 libidn/1.13
> Host: img.timeinc.net
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 206 Partial Content
< Server: Apache
< ETag: "360aaa2-2c35-4b88fab342cc0"
< Accept-Ranges: bytes
< Date: Wed, 22 Feb 2012 09:44:55 GMT
< Content-Type: multipart/byteranges; boundary=4883C05FB1736617
< Last-Modified: Thu, 09 Feb 2012 22:38:19 UTC
< Connection: keep-alive
* no chunk, no close, no size. Assume close to signal end

Doug

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Daniel Stenberg | 22 Feb 22:13

Re: Connections and multibyte responses

On Wed, 22 Feb 2012, unkadoug <at> gmail.com wrote:

> /* On HTTP 1.1, when connection is not to get closed, but no Content-Length 
> nor Content-Encoding chunked have been received, according to RFC2616 
> section 4.4 point 5, we assume that the server will close the connection to 
> signal the end of the document. */
>
> This assumption is invalid.

Yes you're right! But still...

> If the message uses the media type "multipart/byteranges", and the 
> transfer-length is not otherwise specified, then this self- delimiting media 
> type defines the transfer-length. This media type MUST NOT be used unless 
> the sender knows that the recipient can parse it; the presence in a request 
> of a Range header with multiple byte- range specifiers from a 1.1 client 
> implies that the client can parse multipart/byteranges responses.
>
> If the curl client sends a range header with mulitple byte-range specifiers, 
> and gets a multipart/byteranges reply that does not explicitly close the 
> connection, it improperly awaits a signal:

In the HTTPbis work which will eventually end up in a refreshed HTTP 1.1 spec, 
this mentioning of the special (and IMHO very odd and confused) way of 
detecting the end of a body is removed:

Here's the ticket:

   http://trac.tools.ietf.org/wg/httpbis/trac/ticket/90

Here's the new HTTP range section:

   http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-18

I would agree that since RFC2616 did state that, we could indeed add code to 
acknowledge this in 206 responses and detect the end of a body this way but it 
seems to be used extremely rarely ("it is believed that no implementation 
currently uses them" as the ticket says) and won't exist soon. As we've 
survied this far without implementing it, is there really any good reason for 
us to implement it *now*?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

unkadoug | 23 Feb 07:36

Re: Re: Connections and multibyte responses

On , Daniel Stenberg <daniel <at> haxx.se> wrote:
> I would agree that since RFC2616 did state that, we could indeed add code to acknowledge this in 206 responses and detect the end of a body this way but it seems to be used extremely rarely ("it is believed that no implementation currently uses them" as the ticket says) and won't exist soon. As we've survied this far without implementing it, is there really any good reason for us to implement it *now*?

If it's a behavior that will be nonstandard, soon, then nevermind. I added support for it to a proxy, tentatively, but with curl not supporting it, that feature will never get released.

Thank you for your helpful reply.

Doug

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html