headers
Mike Melanson | 28 Mar 2012 03:19

The right data for Embedded XZ?

Hi,

I am trying to use Embedded XZ in a project. Is there something special I
should know about how to compress data so that it is suitable for decoding
by Embedded XZ? Is it supposed to work with any valid .xz file? Or is it
supposed to use raw .xz files?

Here is what I have done: Compress a file as:

  xz file

So I have file.xz. I wrote a C program to load that entire file into an
array, call xz_dec_init() and then xz_dec_run(), with all the right xz_buf
parameters filled in. xz_dec_run() always returns XZ_DATA_ERROR, which
xz.h defines as meaning the data is corrupt. I figured that perhaps I'm
not supposed to include the 6-byte magic header (FD 37 7A 58 5A 00) so I
defined xz_buf.in_pos to skip those bytes. This changes the returned error
to XZ_FORMAT_ERROR.

How should I compress the data so that Embedded XZ likes it? Or have I got
the wrong problem?

Thanks,
--
    -Mike Melanson
Permalink | Reply |
headers
g.esp | 28 Mar 2012 08:47
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Mercredi 28 Mars 2012 03:19:09
> Objet: [xz-devel] The right data for Embedded XZ?
> 
> Hi,
> 
> I am trying to use Embedded XZ in a project. Is there something special I
> should know about how to compress data so that it is suitable for decoding
> by Embedded XZ? Is it supposed to work with any valid .xz file? Or is it
> supposed to use raw .xz files?
> 
> Here is what I have done: Compress a file as:
> 
>   xz file
> 
> So I have file.xz. I wrote a C program to load that entire file into an
> array, call xz_dec_init() and then xz_dec_run(), with all the right xz_buf
> parameters filled in. xz_dec_run() always returns XZ_DATA_ERROR, which
> xz.h defines as meaning the data is corrupt. I figured that perhaps I'm
> not supposed to include the 6-byte magic header (FD 37 7A 58 5A 00) so I
> defined xz_buf.in_pos to skip those bytes. This changes the returned error
> to XZ_FORMAT_ERROR.
> 
> How should I compress the data so that Embedded XZ likes it? Or have I got
> the wrong problem?
> 
> Thanks,
(Continue reading)

Permalink | Reply |
headers
Mike Melanson | 28 Mar 2012 09:01

Re: The right data for Embedded XZ?

On 3/27/2012 11:47 PM, g.esp@... wrote:
> I have used xzminidec code unchanged and --check=crc32 has to be used during compression with xz as this is
the onlyt crc supported by xzminidec.
>
> Depending of data to be compressed size, setting dictionary size on compression to the size of
uncompressed data help to minimize memory requirement on decompression.
> To create a floppy image, Lasse advised me to use xz --check=crc32 --lzma2=dict=2MiB,nice=273,depth=512

Thanks for the tips. I'm still having the same difficulty with these 
options, though (xz_dec_run() just returns XZ_DATA_ERROR).

Is xzminidec the same as Embedded XZ that I'm trying to use?

Thanks,
--

-- 
     -Mike Melanson
Permalink | Reply |
headers
g.esp | 28 Mar 2012 17:00
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Mercredi 28 Mars 2012 09:01:24
> Objet: Re: [xz-devel] The right data for Embedded XZ?
> 
> On 3/27/2012 11:47 PM, g.esp@... wrote:
> > I have used xzminidec code unchanged and --check=crc32 has to be
> > used during compression with xz as this is the onlyt crc supported
> > by xzminidec.
> >
> > Depending of data to be compressed size, setting dictionary size on
> > compression to the size of uncompressed data help to minimize
> > memory requirement on decompression.
> > To create a floppy image, Lasse advised me to use xz --check=crc32
> > --lzma2=dict=2MiB,nice=273,depth=512
> 
> Thanks for the tips. I'm still having the same difficulty with these
> options, though (xz_dec_run() just returns XZ_DATA_ERROR).
> 
> Is xzminidec the same as Embedded XZ that I'm trying to use?
> 
> Thanks,
> --
>      -Mike Melanson
> 
> 
Yes, using
cd userspace && make
(Continue reading)

Permalink | Reply |
headers
Mike Melanson | 29 Mar 2012 01:52

Re: The right data for Embedded XZ?

> Yes, using
> cd userspace && make
> should compile xzminidec for you.
>
> I hacked the makefile a bit more to compile against klibc.

Ah, I hadn't seen this sample app before. Unfortunately, building on my
Ubuntu Linux system failed with:

gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86 -DXZ_DEC_POWERPC
-DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
-DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o boottest.o
boottest.c
In file included from ../linux/lib/decompress_unxz.c:235:0,
                 from boottest.c:22:
../linux/lib/xz/xz_dec_lzma2.c: In function ‘xz_dec_lzma2_run’:
/usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining failed in
call to ‘memmove’: redefined extern inline functions are not considered
for inlining
../linux/lib/xz/xz_dec_lzma2.c:884:11: sorry, unimplemented: called from here
make: *** [boottest.o] Error 1

*However!* Studying the source code in that directory demonstrated what
was wrong in my own sample app-- I need to call xz_crc32_init() before the
other functions. I see that's mentioned near the end of xz.h; perhaps it
warrants an earlier mention.

Anyway, I got past that problem. It should be noted that "--check=crc32"
really is necessary for compressed data if Embedded XZ will be chewing on
it. The library returns XZ_OPTIONS_ERROR otherwise, but only on the first
(Continue reading)

Permalink | Reply |
headers
Thorsten Glaser | 29 Mar 2012 15:59
Picon
Gravatar

Re: The right data for Embedded XZ?

Mike Melanson dixit:

>gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86 -DXZ_DEC_POWERPC
     ^^^^^^^^^^
You probably want -std=gnu99 here.

>-DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
>-DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o boottest.o
>boottest.c
>In file included from ../linux/lib/decompress_unxz.c:235:0,
>                 from boottest.c:22:
>../linux/lib/xz/xz_dec_lzma2.c: In function xz_dec_lzma2_run:
>/usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining failed in
>call to memmove: redefined extern inline functions are not considered
>for inlining

Yes well, that will of course break. (Huh. So there i̲s̲ a
legitimate case for that error message. I usually only see
it when compiling mksh with -fwhole-program --combine on
about three differently broken GCC versions. Luckily there
is now LTO which usually works.)

bye,
//mirabilos
--

-- 
> Hi, does anyone sell openbsd stickers by themselves and not packaged
> with other products?
No, the only way I've seen them sold is for $40 with a free OpenBSD CD.
	-- Haroon Khalid and Steve Shockley in gmane.os.openbsd.misc
(Continue reading)

Permalink | Reply |
headers
Lasse Collin | 31 Mar 2012 09:58
Favicon

Re: The right data for Embedded XZ?

On 2012-03-29 Thorsten Glaser wrote:
> Mike Melanson dixit:
> 
> >gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86
>      ^^^^^^^^^^
> You probably want -std=gnu99 here.

gnu89 should work (gnu99 should work too). The Linux kernel is compiled
with gnu89 so XZ Embedded needs to conform to that too.

> >-DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
> >-DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o
> >boottest.o boottest.c
> >In file included from ../linux/lib/decompress_unxz.c:235:0,
> >                 from boottest.c:22:
> >../linux/lib/xz/xz_dec_lzma2.c: In function ‘xz_dec_lzma2_run’:
> >/usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining
> >failed in call to ‘memmove’: redefined extern inline functions are
> >not considered for inlining
> 
> Yes well, that will of course break.

It works for me but I'm not sure why. In boottest.c I want to use the
memmove() and other functions from decompress_unxz.c instead of libc.
Maybe it would be enough to avoid <string.h> in boottest.c and replace
strcmp() with something else.

--

-- 
Lasse Collin  |  IRC: Larhzu  <at>  IRCnet & Freenode
(Continue reading)

Permalink | Reply |
headers
g.esp | 29 Mar 2012 22:43
Picon
Favicon

Re: The right data for Embedded XZ?


----- Mail original -----
> De: "Mike Melanson" <mike@...>
> À: xz-devel@...
> Envoyé: Jeudi 29 Mars 2012 01:52:42
> Objet: Re: [xz-devel] The right data for Embedded XZ?
> 
> > Yes, using
> > cd userspace && make
> > should compile xzminidec for you.
> >
> > I hacked the makefile a bit more to compile against klibc.
> 
> Ah, I hadn't seen this sample app before. Unfortunately, building on
> my
> Ubuntu Linux system failed with:
> 
> gcc -std=gnu89 -I../linux/include/linux -I. -DXZ_DEC_X86
> -DXZ_DEC_POWERPC
> -DXZ_DEC_IA64 -DXZ_DEC_ARM -DXZ_DEC_ARMTHUMB -DXZ_DEC_SPARC
> -DXZ_DEC_ANY_CHECK -ggdb3 -O2 -pedantic -Wall -Wextra -c -o
> boottest.o
> boottest.c
> In file included from ../linux/lib/decompress_unxz.c:235:0,
>                  from boottest.c:22:
> ../linux/lib/xz/xz_dec_lzma2.c: In function ‘xz_dec_lzma2_run’:
> /usr/include/bits/string3.h:56:1: sorry, unimplemented: inlining
> failed in
> call to ‘memmove’: redefined extern inline functions are not
> considered
(Continue reading)

Permalink | Reply |
headers
Lasse Collin | 31 Mar 2012 09:48
Favicon

Re: The right data for Embedded XZ?

On 2012-03-28 Mike Melanson wrote:
> *However!* Studying the source code in that directory demonstrated
> what was wrong in my own sample app-- I need to call xz_crc32_init()
> before the other functions. I see that's mentioned near the end of
> xz.h; perhaps it warrants an earlier mention.

XZ Embedded has been written primarily for the Linux kernel and there
you don't need xz_crc32_init() except in decompress_unxz.c. So in the
Linux context it's better to keep the xz_crc32_init() docs at the end of
xz.h. I'm sorry that you didn't notice this or the existence of
xzminidec earlier. I added a reference to xzminidec.c to README.

> Anyway, I got past that problem. It should be noted that
> "--check=crc32" really is necessary for compressed data if Embedded
> XZ will be chewing on it.

Yes. This is mentioned in linux/Documentation/xz.txt. The existence
of this file is pointed in README.

> The library returns XZ_OPTIONS_ERROR otherwise, but only on the first
> call. If you call xz_dec_run() again, decoding will proceed fine (and
> accurately). I figured this out when I made a mistake in my decode
> loop and didn't terminate on error.

Calling xz_dec_run() again after XZ_OPTIONS_ERROR leads to undefined
behavior in sense that I haven't thought what will happen. liblzma
would keep returning the same error code if one calls lzma_code() again
after an error, but in XZ Embedded I skipped such things to make the
code smaller.
(Continue reading)

Permalink | Reply |

Gmane