Unzip will not work on some archives

When I run the unzip function, it will fail on certain archives. What really confuses me is that I can extract an archive's contents manually via the gui (right click an archive -> unzip).

Error message is simply "Invalid ZIP file:

C:\users\Desktop\myzipfile.zip"

The zip file is not invalid, and it is not corrupted

[Walter Roberson]

.zip files can contain any of a number of different compression schemes. unzip() only handles some of them.

[Matthew Rademacher]

is there some way I can tell what the compression scheme of an archive is? and is there any other option to unzip (besides what chrisw23 has already mentioned?

[Walter Roberson]

https://superuser.com/questions/120354/analyzing-properties-of-a-zip-file

https://github.com/nlitsme/zipdump

[Matthew Rademacher]

@Walter Roberson, using your suggestions I've learned that my problem is that unzip fails while trying to extract entries created with the deflate64 method. (which lines up with the problem seen a decade ago here: Matlab fails to unzip large files - MATLAB Answers - MATLAB Central (mathworks.com)).

With that knowledge, do I have any options?

Answer 1

[chrisw23]

try to use .net Functionality (Windows OS assumed)

asm = NET.addAssembly("System.IO.Compression");
import System.IO.Compression.ZipFileExtensions.*
ZipFileExtensions.ExtractToFile(...
    ZipFileExtensions.ExtractToDirectory(... 
    % search for system.io.compression.zipfileextensions.extracttodirectory to get the argument list

[Matthew Rademacher]

thanks for the answer chris! this is exactly the type of functionality I think that I need. I'm still having a bit of trouble understanding how to execute the "ZipFileExtensions.ExtractToDirectory(... " function.

I can find documentation on microsoft's website, but not sure how to handle it from matlab's perspective. Do you have any additional guidance or could you help point me to some way to learn about interacting with a .net assembly from matlab?

This is my reference for system.io.compression.zipfileextensions.extracttodirectory

Here's what I'm trying to execute:

asm = NET.addAssembly("System.IO.Compression");
import System.IO.Compression.ZipFileExtensions.*
MyFileName =  'W:\DATA\myfile.zip';
MyDirName = 'W:\DATA\TestDir';
ZipFileExtensions.ExtractToDirectory(MyFileName,MyDirName);

error code is: Unable to resolve the name 'ZipFileExtensions.ExtractToDirectory'.

[chrisw23]

1) don't work with char, use strings because theyare compatible with .net type System.String MyFileName = "W:\DATA\myfile.zip";

2) import System.IO.Compression.* will solve the namespace problem. As an alternative you can use the full namespace because imports just allow to shorten the namespace System.IO.Compression.ZipFileExtensions.ExtractToDirectory(MyFileName,MyDirName);

[Matthew Rademacher]

I still can't get it to work. Same error after trying both of your suggestions. Exact same issue here:

asm = NET.addAssembly("System.IO.Compression");
import System.IO.Compression.ZipFileExtensions.*
MyFileName =  'W:\DATA\myfile.zip';
MyDirName = 'W:\DATA\TestDir';
System.IO.Compression.ZipFileExtensions.ExtractToDirectory(MyFileName,MyDirName);

[chrisw23]

NET.addAssembly("System.IO.Compression");

NET.addAssembly("System.IO.Compression.FileSystem")

import System.IO.*

import System.IO.Compression.*

MyFileName = "W:\DATA\myfile.zip";

MyDirName = "W:\DATA\TestDir";

fStream = File.Open(MyFileName, System.IO.FileMode.Open);

zipArc = ZipArchive (fStream, System.IO.Compression.ZipArchiveMode.Read);

System.IO.Compression.ZipFileExtensions.ExtractToDirectory(zipArc,MyDirName)

fStream.Close

[Matthew Rademacher]

This works! Thanks ChrisW23 you are amazing!

I should have read this documentation on the ZipArchive class

So to summarize why this works:

1. the ExtractToDirectory method requires its first input to be of the "ZipArchive" class
2. ZipArchive class must be created from a FileStream object

[chrisw23]

ya doc is always worth to be read :) ...same for me

[Matthew Rademacher]

the problem with this strategy is that it fails to extract entries compressed via the "deflate64" method (anything above 2GB looks like). so while it works great and the general method can be helpful for other things- looks like I'm still struggling to actualy complete my task in Matlab.

[Walter Roberson]

Using java https://commons.apache.org/proper/commons-compress/apidocs/org/apache/commons/compress/compressors/deflate64/Deflate64CompressorInputStream.html

In paticular https://zenodo.org/record/6609826 is an example of a toolbox for matlab use that incorporates java code for deflate64

[Matthew Rademacher]

%point to files
MyDirName = "W:\DATA\";
MyFileName = "myarchive.zip";
MyDestFile = "mydata.mf4";

%Create a java file object
ZipJavaFile = java.io.File(MyFileName);

%Creata java fileinputstream
JavaFileInputStream = java.io.FileInputStream(ZipJavaFile);
%executing JavaFileInputStream.available yields 631434840

%Create Deflate64CompressorInputStream object
D64InputStream = org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream(JavaFileInputStream);
%executing D64InputStream.available yields 0, may be related to my problem

D64InputStream.read

%fails with error:
%Java exception occurred:
%java.io.IOException: Invalid Deflate64 input
%	at org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:87)
%	at java.io.InputStream.read(InputStream.java:101)
%	at org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:60)
%Caused by: java.lang.IllegalStateException: Illegal LEN / NLEN values
%	at org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.switchToUncompressedState(HuffmanDecoder.java:178)
%	at org.apache.commons.compress.compressors.deflate64.HuffmanDecoder.decode(HuffmanDecoder.java:143)
%	at org.apache.commons.compress.compressors.deflate64.Deflate64CompressorInputStream.read(Deflate64CompressorInputStream.java:85)
%	... 2 more

[Matthew Rademacher]

The code I posted above will successfully create a java Deflate64InputStream object- but I am struggling to actually do anything with it. I am new to both java and File Input Streams, so please forgive my ignorance.

Not sure if this is relevant, but I can read the JavaFileInputStream byte-by-byte, it's only the Deflate64 inputstream that yeilds an error from the "read" method.

[Matthew Rademacher]

I am not a huge fan of this implementation at all but I do have a workaround. I have to install 7zip and execute the 7z.exe executable from command line using the "!" operator. Kind of stinky that you need 7zip installed but I got stuck on the java implementation and need something that works. Here zpath is the path where the 7z.exe file is located (C:\Program Files\7-Zip for me), Archive is the .zip archive compressed with deflate64, and targetpath is the path you want the file to be on.

cmdstring = sprintf('!"%s7z.exe" x %s -o%s -y',zpath,Archive,TargetPath)
eval(cmdstring)