Re: HTTP Pipelining Contributions

> FIFO to start, but we should consider expandability.
>
> Can you suggest an interface that would let the client prioritize queued 
> requests?
>
> I can think of a couple off the top of my head:
>
> 1. Allow the client to provide a sort function for requests.  If one is 
> set, then requests are stored in a priority queue using that sort 
> function. If not, use FIFO.
>
> This has the advantage of being simple and expandable, but needs a call 
> through a function pointer for every request.
>
> 2. Add an algorithm parameter (initially defining only FIFO), and have 
> curl implement other algorithms internally.
>
> This can be optimized well, but is a lot harder to extend and could cause 
> bloat inside curl.
>
> (In both of these cases, we can default to FIFO if the new option is not 
> set, so the option can be added after the initial work is done.)
>
> Joe

I was thinking along the lines that a new int "easy" option can be 
introduced for expressing priorities so that the clients can set whatever 
priority they want, internally libcurl would not attempt to interpret the 
value but would simply maintain a queue sorted by handle priority. If the 
client chooses not to set priorities then all handles will have the same 

Re: HTTP Pipelining Contributions

Joe Mason <jmason <at> rim.com> wrote:

> > From: curl-library-bounces <at> cool.haxx.se [curl-library-bounces <at> cool.haxx.se] on
> >  behalf of Daniel Stenberg [daniel <at> haxx.se]

> > > If the first response to a request on a socket is marked as HTTP/1.0, or 
> > > an
> > > older IIS server version is used, or the site is black-listed (see below),
> > > the socket should be characterized as CAN_PIPELINE = false.
> > 
> > This makes the blacklist basically always just grow. Will there be any means
> > of purging old entries or don't you see this as a problem?
> 
> Well, it means that socket can't be used for pipelining for the rest of
> its life, but it's not like the server characteristics are going to change
> while we're connected to it, so that's unavoidable.

How do you intend to deal with HTTP proxies?

If an HTTP proxy is used, the socket isn't directly connected to the
server and may be used for requests to different servers. curl already
does that.

It's also possible that the server supports pipelining while the
proxy does not (or only poorly).

Fabian

-------------------------------------------------------------------

Re: HTTP Pipelining Contributions

On Thu, Jul 26, 2012 at 05:30:26PM +0000, Joe Mason wrote:
> Can they? curl today doesn't expose the number of connections and the
> mapping of curl handles to connections.  The only options I can find

For pipelined connections, this is implicitly controlled by the
application as it adds pipelined easy handles to a multi handle.

> to control how a request is mapped to a connection are
> The only callbacks I can find today that can be used to control how
> requests are assigned to connections are CURLOPT_FRESH_CONNECT and
> CURLOPT_FORBID_REUSE, which force requests to use new connections (and
> disable both pipelining and regular connection reuse).
> 
> I don't see any way to implement this proposal outside curl without
> adding functions to assign requests to connections explicitly, and I
> thought that Daniel was strongly against that.

I'll admit I'm not completely familiar with the existing pipelining
code, but my understanding is that libcurl will pipeline all it can on a
single connection within a multi handle. An application that wants two
connections would use two multi handles. Within each handle, the
application can control (and in some cases, even reorder) requests and
the pipeline depth by controlling when the easy handles are added to the
multi handle. This could become a bit hairy, which is why I suggest
delegating it to a libcurlapp which would only have to be written once,
not for each app.

> What would the interface to curl look like?

I'm thinking of something along the lines of the multi interface, where