Ruby: writing binary string (7zip) to file. file gets corrupted

Question

We're building an application on Ruby on Rails 4 and it's going really fluently, except for this pain-giving issue

In our application we get a Multipart Response from a SOAP request. This response comes from an external party and sends us a 7zip compressed file. The response has 2 parts, where the second contains a application/octet-stream.

The parts are of the Mail:part type.

When I save the contents of 'part.body.decoded' to a file, I can't open the file with 7zip, because it says the file is damaged.

When I open the generated file in the hex-editor the first lines seem ok.

p part.body.decoded.encoding.name # gives => "ASCII-8BIT"

File.open("file.7z", "wb") do |f|
 f.write(part.body.decoded)
end

Here are the first HEX lines.

37 7A BC AF 27 1C 00 03 EF F2 1B 17 EA FA 03 00
00 00 00 00 68 00 00 00 00 00 00 00 BD D9 66 08
00 1E 10 88 27 43 3B A7 F5 C7 75 19 5D F6 A6 E0

As a test I created a 7z file myself and read it in a string and wrote it back to a file. This works and gives me a valid 7zip.

Any ideas on how to tackle this?

Update: when I put the contents of part.body.decoded in a string str = part.body.decoded #and put the string p str # I get the string as below (I've deleted the middle section of the string because it has to many characters to post it here)

"7z\xBC\xAF'\x1C\x00\x03\xFCpd\x01\xEA\xFA\x03\x00\x00\x00\x00\x00h\x00\x00\x00\x00\x00\x00\x006E\x91\x1F\x00\x1E\x10\x88'C;\xA7\xF5\xC7u\x19]\xF6\xA6\xE0\xACC\xD9\xA4\xA0:,4\xE9\xA1\xAD%\x05\\a\xFBq\xA4Z\xEC7\xCC\x8C\x91\xBC?\x80H)\xFBa\xB8\xB8u\xBE\xB2\x1C|`\xA7P|\xC3\xA9\x82\xB3\r\n\xA2\x15\xA2\xDAn\xBB\xB9

Well, it logically follows that part.body.decoded does not represent a valid 7z byte stream. Is it Base64 stream instead? Or you need to write the other part first? Or the other party sends you garbage? Or something else. — Sergio Tulentsev
– Sergio Tulentsev, Commented Mar 28, 2017 at 15:27
It's not encoded. 37 7A BC AF 27 1C is officially a 7z header (note 37 7A) (side note: it's scary that I can look at "37 7A" and know what that is). If it was still base 64 encoded that wouldn't be there. The question I would ask the original poster is "how do you know the 7z file coming from the other side is valid?" — Michael Chaney
– Michael Chaney, Commented Mar 28, 2017 at 16:54
@SergioTulentsev, the other party assures me it's a valid 7zip with a xml inside with in this xml Base64 encoded smaller files. I've asked them to email me the original 7zip, but they say they can't. We're not the only party connecting with them, so for now I have to believe them ... — Joris
– Joris, Commented Mar 28, 2017 at 17:42
@MichaelChaney: yeah, presence of null bytes makes it pretty clear that this is no base64. But I needed one more question for the volume. :) — Sergio Tulentsev
– Sergio Tulentsev, Commented Mar 28, 2017 at 17:49
The next question would be "is it complete"? At this point, I would write a parser to show the high level chunks in the 7z file along with a type and length and set it loose on this to see if the file is complete or otherwise corrupt. — Michael Chaney
– Michael Chaney, Commented Mar 28, 2017 at 18:11

Joris · Accepted Answer · 2017-03-30 16:05:14Z

1

Solved :-) The Savon 2 Soap Gem depends on the Mail Gem (max. version 2.5). In the Mail gem there is a bug, which touches the raw binary reponse (it ads line-ends). I've forked the Mail gem and made adaptions (to be found in the higher version of Mail (which Savon doesn't use)) and added 'my' Mail gem to my gemfile. Thnx to all commenters for their tips!

answered Mar 30, 2017 at 16:05

Joris

112 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Ruby: writing binary string (7zip) to file. file gets corrupted

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related