Recently I had an issue sending JSON via API to our analytics tool. After few days of troubleshooting, I realized what was the problem during the weekend when the file being uploaded was of lower size due to less amount of data during weekends. Obviously, I decided to zip its contents, afterward facing a problem with a corrupted archive.

The process which we use to upload data to our analytics tools is the following:

Query data from datastore (adding some filters, calculations etc.) for the last 24 hours Upload file to GCP Storage (in case of failure, we can manually upload it) Download file from GCP Storage, then send it via HTTP request

A Cron job doing this runs every midnight. As mentioned earlier, one of the entities had troubles with exporting, due to its size.

I decided to compress the file using archive/zip but had an issue with Corrupted archives. On Mac, I would get the following error when trying to open the archive:

This is how I tried to compress my file:

1 func GetZip ( c context . Context , path string ) ([] byte , error ) { 2 client , err := storage . NewClient ( c ) 3 if err != nil { 4 return nil , err 5 } 6 7 appID := appengine . AppID ( c ) 8 bucketH := client . Bucket ( bucketName [ appID ]) 9 10 // Downloading file from bucket (GCP Storage) 11 reader , err := bucketH . Object ( path + "/~.json" ). NewReader ( c ) 12 if err != nil { 13 return nil , err 14 } 15 16 defer reader . Close () 17 18 bts , err := ioutil . ReadAll ( reader ) 19 20 // Fixing JSON issues we had 21 bts = bytes . TrimSuffix ( bts , [] byte ( "," )) 22 content := bytes . NewReader ( bts ) 23 24 buf := new ( bytes . Buffer ) 25 w := zip . NewWriter ( buf ) 26 27 defer w . Close () 28 29 f , err := w . Create ( "report.json" ) 30 if err != nil { 31 return nil , err 32 } 33 34 _ , err = f . Write ([] byte ( "[" )) 35 if err != nil { 36 return nil , err 37 } 38 39 _ , err = io . Copy ( f , content ) 40 if err != nil { 41 return nil , err 42 } 43 44 _ , err = f . Write ([] byte ( "]" )) 45 if err != nil { 46 return nil , err 47 } 48 49 return buf . Bytes (), nil 50 }

Notice the highlighted line. Searching for a solution brought me to test of archive/zip on golang.org, which instead of deferring closing the zip writer, manually closed it at the end. By doing this little change to my code (err := w.Close() instead of defer w.Close()) I was able to fix my issue. At the end, this is what I got (minus the JSON alteration part):

1 func GetZip ( c context . Context , path string ) ([] byte , error ) { 2 client , err := storage . NewClient ( c ) 3 if err != nil { 4 return nil , err 5 } 6 7 appID := appengine . AppID ( c ) 8 bucketH := client . Bucket ( bucketName [ appID ]) 9 10 reader , err := bucketH . Object ( path + "/~.json" ). NewReader ( c ) 11 if err != nil { 12 return nil , err 13 } 14 15 defer reader . Close () 16 17 buf := new ( bytes . Buffer ) 18 w := zip . NewWriter ( buf ) 19 20 f , err := w . Create ( "report.json" ) 21 if err != nil { 22 return nil , err 23 } 24 25 _ , err = io . Copy ( f , reader ) 26 if err != nil { 27 return nil , err 28 } 29 30 err = w . Close () 31 if err != nil { 32 return nil , err 33 } 34 35 return buf . Bytes (), nil 36 }

This occurs because explicitly closing the zip writer “finishes writing the zip file by writing the central directory”. On the other hand, defers occur after the return statement, so call to buf.Bytes() is happening before the call to Close() and therefore doesn’t have the “central directory.”