Blayne Chard

@EvenRouault Then it mostly comes down to the compression method used by the zip? whether the compression supports random access With the basic flow of reading the ZIP CD to find the offset to compressed file. then randomly read the file Something like github.com/vasi/lzopfs ?

@EvenRouault Shameless plug for tar files and github.com/linz/cotar Minio has created zipindex to efficiently random read files from a zip file. github.com/minio/zipindex Both options use a precomputed index for the archive but allow random reads from remote locations like s3.

Verifying myself: I am blacha on Keybase.io. aELkcYOBbWMAT1qvdq5mwDdonyCFzI7CmoFj / keybase.io/blacha/sigs/aE…

@IsaacBrodsky @kylebarron2 @rabernat @kurtschwehr @opencholmes This is the problem with zip, it takes too many random reads to find a specific file from the zip CD, cotar can find you a random file in 2-3 reads.

@kylebarron2 @rabernat @kurtschwehr @opencholmes Zip’s central directory isn’t well optimized for arbitrary read requests, so I think cotar is beyond zip. Just comparing the metadata sizes I think tar’s headers are smaller and don’t need to be materialized to an extra single metadata blob at the end like zip.

Surprised again. Cloud optimized tar archive (cotar). Thanks @opencholmes github.com/linz/cotar

@pwramsey @dinomirMT @ThomasG77 tar is just a container format, you could compress the contents of the container, archiving pbf.gz files instead of pbf. Otherwise gzip has random access if you index the file.

@dinomirMT @ThomasG77 Still uncompressed. I've been looking for ways to do byte-offsets into files compressed with DEFLATE and not really found any great solutions.

Dare to dream: is "cloud optimized shape file" a possibility? blog.cleverelephant.ca/2022/04/coshp.…