What to do about portability with EBCDIC?

[khwilliamson]

With the application of #74, IO::Compress passes all its tests on an EBCDIC platform, except for two recently added ones in 006zip.t: where a zip file was created on an ASCII machine and read on the EBCDIC one. The problem is that the filename field "hello.txt" in the header is in an alien coding (ASCII) that the test doesn't expect.

The question is what to do about this. Originally, I thought just to skip the failing tests on an EBCDIC box, but reading the code and documentation a bit, I think that the goal should be to make compressed files to be completely portable. I'm willing to write a patch, but I need some guidance. I saw that the code does attempt to look at the current locale, but I didn't see how it deals with that. It appears there is a bit that indicates if the text is UTF-8 or not. A problem is that the UTF-8 on EBCDIC boxes is not the same as the UTF-8 on ASCII ones.

If EBCDIC boxes output all their text as ASCII and ASCII-based UTF-8, then we would have complete portability.

What are all the fields that would be affected?

This would be done as the last step in writing and the first step in reading a file, so that the rest of the module would work in native values, so would not have to change. It might be that Encode could hide most of it from this module. There would be effectively no change in behavior from currently when run on an ASCII box

[pmqs]

I assume you are referring to these tests from 06zip.t?

    title "bad datetime";
    # https://github.com/pmqs/IO-Compress/issues/65

    {
        # files/time-zero.zip has the modification time set to zero
        my $file1 = "t/files/time-zero.zip";
        my $u = IO::Uncompress::Unzip->new( $file1)
            or die "Cannot open $file1: $UnzipError";

        isa_ok $u, "IO::Uncompress::Unzip";

        my $name = $u->getHeaderInfo()->{Name};

        my $hdr = $u->getHeaderInfo();
        is $hdr->{Name}, 'hello.txt', "Name is 'hello.txt'";
        is $hdr->{Time}, 0, "Time is zero";
    }


    {
        # files/time-invalid.zip has the modification time set to an invalid date
        my $file1 = "t/files/time-invalid.zip";
        my $u = IO::Uncompress::Unzip->new( $file1)
            or die "Cannot open $file1: $UnzipError";

        isa_ok $u, "IO::Uncompress::Unzip";
        my $name = $u->getHeaderInfo()->{Name};

        my $hdr = $u->getHeaderInfo();
        is $hdr->{Name}, 'hello.txt', "Name is 'hello.txt'";
        is $hdr->{Time}, 0, "Time is zero";
    }

Fix Option 1

If you just want to get the test to pass, the lines is $hdr->{Name}, 'hello.txt', "Name is 'hello.txt'"; aren't needed. They probably got there because I cut-n-pasted an existing test when creating these ones. The purpose of the test is to check the Time field in the zip file, not the filename.

Fix option 2

The alternative fix depends on how how an EBCDIC Perl handles ASCII utf-8.

In a zip file there is a bit in one of data structures called the Language encoding flag (EFS). When set it flags that the filename (and comment) are encoded in ASCII utf-8 (when the bit isn't set the filename encoding can be absolutely anything -- that causes huge problems with interoperability) .

In my code when I see the Language encoding bit set AND the IO::Uncompress::Unzip option efs option is set to true, this code gets executed when reading the filename from the zip file

            eval { $filename = Encode::decode_utf8($filename, 1) }
                or Carp::croak "Zip Filename not UTF-8" ;

I presume on an EBCDIC system that just does the same as an ASCII system and stores the Unicode codepoints internally?

Assuming that is the case, option two is for me to recreate the zip files with the Language encoding bit set and to change the test s to use the efs option.

        my $u = IO::Uncompress::Unzip->new( $file1, efs => 1)
            or die "Cannot open $file1: $UnzipError";

Hmmm, how do you handle the filename test -- is the hello.txt an ascii string or EBCDIC ?

`is $hdr->{Name}, 'hello.txt',

Even if we go for option 1, it is probably worth having a specific test for the filename that checks that an EBCDIC system can read ASCII uitf8 filenames.

[khwilliamson]

I think in the short term that making these two tests TODO wil work.

Longer term is for me to write a routine that translates a SV from EBCDIC to ASCII, and vice versa. That would include changing UTF-EBCDIC to UTF-8. If users of this module create their archives with the bit set, then they become completely portable, I believe.

[pmqs]

@khwilliamson, just looking at the test harness in more detail,

There are a pile of tests that explicitly check the use of utf8 and the efs bit in t/112utf8-zip.t. The code only gets run if Encode is available. Is Encode available with the EBCDIC port?

In fact, there is already a test that is explicitly skipped for UTF-EBCDIC that you gave me a few years ago -- see #32

SKIP: {
    title "unzip: EFS => 1 filename not valid utf8 - language encoding flag set";

    # The name hard-coded into this pre-built file is not illegal UTF-EBCDIC
    skip "ASCII-centric test", 1, unless ord "A" == 65;

    my $filename = "t/files/bad-efs.zip" ;

    eval { my $u = IO::Uncompress::Unzip->new( $filename, efs => 1 )
        or die "Cannot open $filename: $UnzipError" };

    like $@, qr/Zip Filename not UTF-8/,
            "  Zip Filename not UTF-8" ;

}

Also, apart from the last two tests in t/006zip.t that you highlighted, there are a few other tests in that file that do need to check the filename. For example, this test

{
    title "Member & Comment 0";

    my $lex = LexFile->new( my $file1 );

    my $content = 'hello' ;


    my $zip = IO::Compress::Zip->new( $file1,
                    Name => "0", Comment => "0", EFS => 1 );
    isa_ok $zip, "IO::Compress::Zip";

    is $zip->write($content), length($content), "write";

    ok $zip->close(), "closed";



    my $u = IO::Uncompress::Unzip->new( $file1, Append => 1, EFS => 1)
        or die "Cannot open $file1: $UnzipError";

    isa_ok $u, "IO::Uncompress::Unzip";

    my $name = $u->getHeaderInfo()->{Name};

    is $u->getHeaderInfo()->{Name}, "0", "Name is '0'";
}

Now I'm confused about the extent of EBCDIC support.

[khwilliamson]

My thoughts are still evolving on this.

Encode is available, but not fully.

In the past, I've viewed the ASCII world and the EBCDIC world as two different ecosystems, with little communication between them. So a compressed file built in one could be uncompressed successfully from the same one. But So that's why your second example above works now. It's compressed and uncompressed from the same ecosystem.

But now I'm thinking that your module could fairly easily be enhanced so that it works on files created from the opposite ecosystem. I think all that is needed is to standardize on using the ASCII form of UTF-8 for text fields when the efs bit is set.

If someone compresses an EBCDIC text file on either type of box and uncompresses it on the other type, it will still be EBCDIC. Just as I believe if someone compresses and uncompresses a Japanese text file, it still will be Japanese.

Let me know what I haven't addressed here.

[pmqs]

My thoughts are still evolving on this.

Encode is available, but not fully.

In the past, I've viewed the ASCII world and the EBCDIC world as two different ecosystems, with little communication between them. So a compressed file built in one could be uncompressed successfully from the same one. But So that's why your second example above works now. It's compressed and uncompressed from the same ecosystem.

Just to be clear - we are only talking about the filename & comment fields in the zip metadata. The actual compressed payload is not in scope for this discussion.

But now I'm thinking that your module could fairly easily be enhanced so that it works on files created from the opposite ecosystem. I think all that is needed is to standardize on using the ASCII form of UTF-8 for text fields when the efs bit is set.

When I write a filename/comment to the zip file and EFS is set, I use this

    utf8::downgrade($filename, 1)
        or Carp::croak "Wide character in zip filename";

reading the filename triggers this

    eval { $filename = Encode::decode_utf8($filename, 1) }
        or Carp::croak "Zip Filename not UTF-8" ;

At https://perldoc.perl.org/utf8 I see this definition of utf8::downgrade: Converts in-place the internal representation of the string from UTF-8 to the equivalent octet sequence in the native encoding (Latin-1 or EBCDIC).

Not 100% clear what that actually means, but does it mean that for the equivalent filename the byte sequence written to the zip file is different on an ASCII and an EBCDIC system? If it is, that is a problem. We only want the ASCII utf8 byte sequence.

A simple test could be to see what utf8::downgrade actually does on EBCDIC and ASCII systems

Looking at the perlebcdic docs at URL decoding and encoding where it discussed dealing with the ASCII semantic of URLS, the functions used are utf8::unicode_to_native and utf8::native_to_unicode. Perhaps I should be using these to encode/decode filenames?

If someone compresses an EBCDIC text file on either type of box and uncompresses it on the other type, it will still be EBCDIC. Just as I believe if someone compresses and uncompresses a Japanese text file, it still will be Japanese.

Yep - if the round trip uses the same encoding for the filename metadata it will appear to work. So the tests in 112utf8-zip.t create zip files on the fly, triggering utf8::downgrade and then unzip them, triggering Encode::decode_utf8, so the tests may seem to be working on an EBCDIC system, but the byte sequences stored in the zip file metadata may be wrong if you attempt to unzip them on an ASCII system.

A possible enhancement I could do for 112utf8-zip.t is to check that the actual bytes written to the zip file correspond to the ASCII definition of utf8.

[khwilliamson]

I would add a function to core that you would call to convert to/from the two UTF-8 forms. It would optimize to nothing on ASCII platforms. You would check for the existence of the function, and without it, you'd get the existing behavior.

[pmqs]

I would add a function to core that you would call to convert to/from the two UTF-8 forms. It would optimize to nothing on ASCII platforms. You would check for the existence of the function, and without it, you'd get the existing behavior.

I assume this means that older Perl's won't support this feature unless the function gets backported.

You are sure that the existing encode functionality doesn't have what is needed? I don't know enough about the inner working to know either way.

[khwilliamson]

Yes. I suppose I could backport it

Yes Encode does not have this functionality

[pmqs]

Yes. I suppose I could backport it

This is going to be a niche application, so not sure what the demand will be like,

Yes Encode does not have this functionality

Ok, thanks for that.

I've added a todo to get the some of the efs tests to check the byte sequence actually written to the zip file. That will guard against the round-trip encoding trap we discussed yesterday.

Ping me if you need anything

[khwilliamson]

This discussion has persuaded me that the functionality would be better in Encode

[pmqs]

Sounds good. Shout when you need any changes made.