gmane.comp.lang.tcl.core

Donal K. Fellows | 7 Oct 17:40

Fixing TIP 234: zlib support

Hi

I've been reviewing TIP#234 today (which describes a feature set that
I'd really like to have in 8.6) and I've seen that it has a number of
odd limitations and unfortunate features. This message is an attempt to
write some of them down. I don't claim to have all the answers. 

   1) Do we really want to publicise the streaming API? Maybe it would be
      fine to just provide a way to attach a streaming gzip/gunzip to a
      channel as a transformation (see [chan push]) and leave it at that.

   2) The API for [zlib gzip] and [zlib gunzip] seems odd too. Why can't
      I control (for gzip) or discover (for gunzip) the metadata? For the
      gzipper, it'd probably be best to express this as a set of options
      (the compression level would be best done the same way) but I'm not
      quite so sure about for gunzip.

   3) Is it necessary to give a buffer size, or is letting Tcl pick one
      itself "good enough"? (After all, we give a buffer growth algorithm
      in the TIP!)

Let me be clear again: I'd *really* like to see zlib support in Tcl in
8.6 (and PNG images too) and I think it would make a good Big Ticket
item for many users. But I'd also like the API to be nice too, and we've
got a little time now (rest of October to be honest) to clear things up...

Donal.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge

(Continue reading)

Pascal Scheffers | 8 Oct 09:03

Re: Fixing TIP 234: zlib support

Hi Donal,

When I started on this project, I expected some feedback from the TCT 
and others on this TIP, which didn't come because it wasn't high on 
anyone else's priorities. It seems that has changed :)

I'll have to get back on this later, though, as this week has 1) A 
conference which needs preparation, 2) A new kitchen being installed in 
our house and 3) A weekend of competitive sailing.

And my existing implementation satisfied my needs (my own TEAPOT).

- Pascal.

Donal K. Fellows wrote:
> Hi
>
> I've been reviewing TIP#234 today (which describes a feature set that
> I'd really like to have in 8.6) and I've seen that it has a number of
> odd limitations and unfortunate features. This message is an attempt to
> write some of them down. I don't claim to have all the answers. 
>
>   1) Do we really want to publicise the streaming API? Maybe it would be
>      fine to just provide a way to attach a streaming gzip/gunzip to a
>      channel as a transformation (see [chan push]) and leave it at that.
>
>   2) The API for [zlib gzip] and [zlib gunzip] seems odd too. Why can't
>      I control (for gzip) or discover (for gunzip) the metadata? For the
>      gzipper, it'd probably be best to express this as a set of options
>      (the compression level would be best done the same way) but I'm not

(Continue reading)

Donal K. Fellows | 8 Oct 10:32

Re: Fixing TIP 234: zlib support

Pascal Scheffers wrote:
> When I started on this project, I expected some feedback from the TCT 
> and others on this TIP, which didn't come because it wasn't high on 
> anyone else's priorities. It seems that has changed :)

I'm embarrassed to say that we collectively left this alone for far too
long. There were always just enough niggling problems for it not to go
in as it was, and it was always easier to work on something else. This
is why having a deadline is a good thing. 

(This TIP makes TIP#244 much easier, since it takes the responsibility
for zlib integration, and that's needed for a Good Out Of Box Experience
on modern machines.)

Donal.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Lars Hellström | 11 Oct 00:29

Re: Fixing TIP 234: zlib support

Donal K. Fellows skrev:
> Hi
> 
> I've been reviewing TIP#234 today (which describes a feature set that
> I'd really like to have in 8.6) and I've seen that it has a number of
> odd limitations and unfortunate features. This message is an attempt to
> write some of them down. I don't claim to have all the answers. 
> 
>    1) Do we really want to publicise the streaming API? Maybe it would be
>       fine to just provide a way to attach a streaming gzip/gunzip to a
>       channel as a transformation (see [chan push]) and leave it at that.

I, for one, would much rather have a stack-on-channel command than a 
"create standalone stream object" command. Usecases for the former are 
clear: read (write) compressed data from (to) socket/file. The only 
usecase I've heard of for the latter is to implement the former.

Conversely, the only part of the stream interface that shouldn't 
immediately be provided by a compressor/decompressor stacked on a 
reflected channel are the partial stream checksum and the distinctions 
between flush and fullflush, which seems like rather esoteric features. 
Could the checksum be made available as an [fconfigure] option, if it 
is needed?

>    2) The API for [zlib gzip] and [zlib gunzip] seems odd too. Why can't
>       I control (for gzip) or discover (for gunzip) the metadata? For the
>       gzipper, it'd probably be best to express this as a set of options
>       (the compression level would be best done the same way) but I'm not
>       quite so sure about for gunzip.
>

(Continue reading)

Pascal Scheffers | 13 Oct 11:21

Re: Fixing TIP 234: zlib support

Hi Lars,

Lars Hellström wrote:
> I, for one, would much rather have a stack-on-channel command than a 
> "create standalone stream object" command. Usecases for the former are 
> clear: read (write) compressed data from (to) socket/file. The only 
> usecase I've heard of for the latter is to implement the former.
True for a lot of use cases. Not true where you're sending compressed 
data via other APIs to other places, storing in a database, for example. 
I really like simple utility commands which take a string I've somehow 
obtained and convert it to something else.

>
> Conversely, the only part of the stream interface that shouldn't 
> immediately be provided by a compressor/decompressor stacked on a 
> reflected channel are the partial stream checksum and the distinctions 
> between flush and fullflush, which seems like rather esoteric features. 
I think they're quite important for streaming interfaces. To quote from 
the zlib manual:
>
> If the parameter flush is set to Z_SYNC_FLUSH 
> <http://www.zlib.net/manual.html#Z_SYNC_FLUSH>, all pending output is 
> flushed to the output buffer and the output is aligned on a byte 
> boundary, so that the decompressor can get all input data available so 
> far. (In particular avail_in 
> <http://www.zlib.net/manual.html#avail_in> is zero after the call if 
> enough output space has been provided before the call.) Flushing may 
> degrade compression for some compression algorithms and so it should 
> be used only when necessary.
>

(Continue reading)

Lars Hellström | 15 Oct 12:27

Re: Fixing TIP 234: zlib support

Pascal Scheffers skrev:
> Hi Lars,
> 
> Lars Hellström wrote:
>> Another issue is integration with the [binary encode] and [binary 
>> decode] ensembles introduced by TIP#317. Putting the string-oriented 
>> commands under [binary] could bring the following changes:
> I am not very fond of [binary], too abstract for my tastes. And 
> compress/deflate and decompress/inflate are the names given to the 
> action everywhere else. I don't like calling them encode/decode even 
> though that is technically just as correct.

I wasn't overly fond of [binary encode] and [binary decode] myself, but 
now that they're there then those are logical places for these 
commands. Why put base64 there but not gzip? Both reversibly transform 
bytearrays to other bytearrays.

As for "are the names given to the action everywhere else": This is not 
true. PDF definitely uses the term "flate" throughout (try grepping for 
/FlateDecode in a collection of PDF files). Deflate is more common 
though, and since it is what this is called in RFC1951 it should 
probably be taken as authorative.

"compress" makes me think of .Z, although [zlib compress] contradicts 
this interpretation.

> I'd really like to have both - I love the ability to do things in more 
> than one way. I like the distinct (utility) command, and I also like the 
> channel stacking.

(Continue reading)

Donal K. Fellows | 15 Oct 12:48

Re: Fixing TIP 234: zlib support

Lars Hellström wrote:
> I wasn't overly fond of [binary encode] and [binary decode] myself, but 
> now that they're there then those are logical places for these 
> commands. Why put base64 there but not gzip? Both reversibly transform 
> bytearrays to other bytearrays.

Good suggestion.

Donal.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Tcl-Core mailing list
Tcl-Core <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tcl-core

Pascal Scheffers | 15 Oct 14:58

Re: Fixing TIP 234: zlib support

Lars Hellström wrote:
>
> I wasn't overly fond of [binary encode] and [binary decode] myself, 
> but now that they're there then those are logical places for these 
> commands. Why put base64 there but not gzip? Both reversibly transform 
> bytearrays to other bytearrays.
I see your point, and don't actually have much of a preference either 
way. Can an extension add encodings/commands to [binary]? My impression 
was that [zlib] would remain an extension, not something always present.

> "compress" makes me think of .Z, although [zlib compress] contradicts 
> this interpretation.
Me too. I don't see a way to change it, though. Is the old compress .Z 
in use much anywhere?

> String command: Yes, please!
> Channel stacking: Yes, please!
> Stream command: Maybe later. (No need to create a new abstraction for 
> what is already covered by channels. If "stream" is ever introduced as 
> a common abstraction in Tcl, it should probably rather focus on 
> datagram style communication, with discrete message units; preferably 
> preserving Tcl_Obj internalRep whenever possible.)
Heh. The funny thing is, my reference implementation implements the 
stream command already. That was easier to implement than the channel 
stacking stuff. I was always convinced channel stacking should be 
implemented, but my priorities were reversed.

So the real question is: Is the streaming command offensive enough to 
warrant removal from the TIP, or do we keep it because it already exists.

(Continue reading)

Pascal Scheffers | 13 Oct 10:55

Re: Fixing TIP 234: zlib support

Hi Donal,

the TIP was written to provide a reasonably Tcl-ish interface to gzip, 
waaaay back in 2004 I saw this as a two stage process. Get basic support 
in this TIP, and add the fancier stuff later. An important motivation 
for this was to be able to use most of the code I'd written then as a 
starting point.

Donal K. Fellows wrote:
>   1) Do we really want to publicise the streaming API? Maybe it would be
>      fine to just provide a way to attach a streaming gzip/gunzip to a
>      channel as a transformation (see [chan push]) and leave it at that.
I am not sure. I always like to have some high level utility functions 
like [gunzip $data], they tend to work for 90% of the uses. For the 
remaining 10%, I might need access to lower-level stuff. Tcl generally 
doesn't provide low-level access, so I don't see a pressing reason to 
supply them. By providing them, and its only a small handful of 
functions provided as sub commands, makes it possible to do everything 
you'll ever need in zip file handling straight from Tcl.

>
>   2) The API for [zlib gzip] and [zlib gunzip] seems odd too. Why can't
>      I control (for gzip) or discover (for gunzip) the metadata? 
Because the 2004 implementation of gzip 1.1.something did not provide 
them. They're easy enough to decode/encode. They could very well be 
fconfigure properties?

>   3) Is it necessary to give a buffer size, or is letting Tcl pick one
>      itself "good enough"? (After all, we give a buffer growth algorithm
>      in the TIP!)

(Continue reading)

Donal K. Fellows | 13 Oct 12:52

Re: Fixing TIP 234: zlib support

Pascal Scheffers wrote:
> the TIP was written to provide a reasonably Tcl-ish interface to gzip, 
> waaaay back in 2004 I saw this as a two stage process. Get basic support 
> in this TIP, and add the fancier stuff later. An important motivation 
> for this was to be able to use most of the code I'd written then as a 
> starting point.

Sure. Understood. (I really should have raised this TIP's issues much
earlier, but I've been busy with TclOO for most of the time since 2004.)

> Donal K. Fellows wrote:
>>   1) Do we really want to publicise the streaming API? Maybe it would be
>>      fine to just provide a way to attach a streaming gzip/gunzip to a
>>      channel as a transformation (see [chan push]) and leave it at that.
> I am not sure. I always like to have some high level utility functions 
> like [gunzip $data], they tend to work for 90% of the uses. For the 
> remaining 10%, I might need access to lower-level stuff. Tcl generally 
> doesn't provide low-level access, so I don't see a pressing reason to 
> supply them. By providing them, and its only a small handful of 
> functions provided as sub commands, makes it possible to do everything 
> you'll ever need in zip file handling straight from Tcl.

I suppose my point is not that we wouldn't have a streaming API, but
rather that we should be able to make it sit with channels. It's the
streaming non-channel API that I'm not sure about. (Many of the extra
features could be done as [fconfigure] options or something like that; I
could imagine doing [fconfigure $gzChan -crc] to get the checksum of the
data that's been processed so far.)

By the way, going for having the optional bits in '-keyword $value' form

(Continue reading)

Pascal Scheffers | 13 Oct 13:39

Re: Fixing TIP 234: zlib support

Donal K. Fellows wrote:
> I suppose my point is not that we wouldn't have a streaming API, but
> rather that we should be able to make it sit with channels. It's the
> streaming non-channel API that I'm not sure about. 
I'm cool with that. I basically didn't know how to implement channel 
stacking (still don't), I only wrote the streaming code and some tests 
for them because zlib/libz provides the API. I've never actually used 
them myself, I could always get away with the in-one-go [gzip inflate 
$data] method.

I can't put my finger on it, but I have a nagging feeling that it would 
be nice to be able to get your hands on the compressed data without 
having to resort to temporary files. I'm not sure that is possible with 
a channel stacking only interface.

> By the way, going for having the optional bits in '-keyword $value' form
> consistently is probably a good idea anyway. It's an idiom that's
> present all over Tcl and Tk, and it seems popular.
Yes, I like that too. I'll add it to the list.
>
>>>   2) The API for [zlib gzip] and [zlib gunzip] seems odd too. Why can't
>>>      I control (for gzip) or discover (for gunzip) the metadata? 
>> Because the 2004 implementation of gzip 1.1.something did not provide 
>> them. They're easy enough to decode/encode. They could very well be 
>> fconfigure properties?
>
> For reading, that'd be great. For writing, I'm not quite so sure; don't
> the values have to be set in the file header? (Take this one as more of
> a "thinking with my mouth open" sort of thing. )
That could be an fconfigure thing too? After all, it is normal to set

(Continue reading)

Andreas Kupries | 14 Oct 19:48

Re: Fixing TIP 234: zlib support


> I can't put my finger on it, but I have a nagging feeling that it would
> be nice to be able to get your hands on the compressed data without
> having to resort to temporary files. I'm not sure that is possible with
> a channel stacking only interface.

Actually you can, simply because you can also put a string based channel
underneath the transformation, through base channel reflection (TIP #219).

--
	Andreas Kupries <andreask@...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Pascal Scheffers | 15 Oct 09:50

Re: Fixing TIP 234: zlib support

Andreas Kupries wrote:
>> I can't put my finger on it, but I have a nagging feeling that it would
>> be nice to be able to get your hands on the compressed data without
>> having to resort to temporary files. I'm not sure that is possible with
>> a channel stacking only interface.
>>     
>
> Actually you can, simply because you can also put a string based channel
> underneath the transformation, through base channel reflection (TIP #219).
>
>   
That is the answer I was looking for. Cool. I'll put that in the docs as 
a hint.

- Pascal.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Return

Return to gmane.comp.lang.tcl.core.

Advertisement

Project Web Page

Tcl Core Team discussions about Tcl development

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane