PNG End Bytes Significance?

[13th]

That's pretty awesome actually, even though I don't know what 99% of all this means other than your trying to compress it

[AceInfinity]

That's pretty awesome actually, even though I don't know what 99% of all this means other than your trying to compress it

Not exactly using the term compression here, but I am removing arbitrary data to optimize the file size. No real compression though here. I'll be releasing this application soon however for everyone interested

[BlackChaos]

I finally get it. I was afraid of commenting on what you were doing earlier in leau of sounding totally redundant in regards to the topic.

[AceInfinity]

Well now you know what i'm doing I'm actually rendering a video right now, and in a few minutes it will be up on youtube, and a preview of what I was doing will be available.

[AceInfinity]

Preview:
[yt]http://www.youtube.com/watch?v=3NDhPNPezV4&fmt=100[/yt]

[Omniscient]

It's the streaming endbit which means on a streaming download like an internet connection the file might not load properly.

When you're downloading any file there has to be a method to know it's completely. Unsure what is inside the PNG header bits but it may not provide enough info for a file to know it's ended.

Try your edited PNGs on a remote host and see what happens in a browser.

[AceInfinity]

I'll do some tests right now actually, and see what I can come up with.

Update: Hmm, everything surprisingly actually works fine with the IEND bytes removed. I tested on my own forum uploading the image, and there's no delay in load time, however even on my localhost with apache installed, it seems to load fine. Tested downloading the file on both my localhost, and my website with curl back to my hard disk, and the IEND bytes don't seem to affect any of that either, after the download the IEND still remains out of the picture, but yet my file still acts completely normal... Also tried the right click and save option, but same result. I have yet to see why IEND is an important part of the file, it seems odd to me that I can't replicate any error with it's absence in the byte structure.

More investigation is probably needed with this, otherwise the smallest PNG file possible might be even less than 68 bytes...

Edit:




I know it marks the end of the PNG stream of data and it always appears last as the last 12 bytes in the file, but there's no identification for why it's such an important/critical chunk from what i've read up on PNG files up 'till now. Very strange that they don't document any of that.

I have my application check for these bytes at the beginning of the file because it's the first stream of bytes that define the PNG file format:

Code:

0x137 0x80 0x78 0x71 0x13 0x10 0x26 0x10

Then you have the IHDR and IDAT along with all the other arbitrary junk data in there, and the ICC profiles I still have to learn about (*also junk data however that could be removed), but that last part (IEND) based on my tests, doesn't do anything as far as my eyes have seen

The CRC in my files without it seems to be stable. Each bit of the IEND chunk points towards the fact that the Ancillary bit is 0, along with the Private, Reserved, and Safe-to-copy bits as well, being that they are all uppercase.

Edit: I finally see the importance of IEND now! And it is for data streaming... I will no longer look into removing this sector from the PNG file itself for optimization. I tried uploading my file with the IEND bytes removed, from an image upload site, and it wouldn't read my file. Strange how it worked with direct upload though, and I was seemingly able to view the image also...

I am sure this only happens when the bytes have to be read for streaming in a selection of interpreters however as photoshop and windows live gallery had no problem identifying the file's data (with or) without it. And I was able to view the image on a webserver if it was uploaded directly, and not through some script on a webpage. I would have to look into this further, but I finally came across an event that proved IEND valuable.

[haphazard]

I'm now at a point where i've been able to compress PNG files by a difference of 200 bytes, however there's still text data chunks and a photoshop iCCP profile that i've seen in some of my own personal PNG files that I used for debugging that I don't know what to deal with. If I could remove that photoshop data with all it's bytes and all the meta data associated with it I could potentially reduce the file by another 200 bytes on top of that depending on whether or not it exists within the byte structure.

I could also take out the transparency chunks which is only about 1 byte difference in file optimization so I felt it's not really worth it. Transparency actually doesn't make a big difference in PNG file size after my investigation... Unless you care about getting rid of an extra whopping 1 byte.

Here's what i've got so far:

Code:

(pHYS):
    HEX Signature = [0x00 0x09 0x70 0x48 0x59 0x73 0x00 0x00]
    DEC Value = [0x0 0x9 0x112 0x72 0x89 0x115 0x0 0x0]
        *Total byte capacity = 21
        *Remove 13 more bytes starting after the signature

Code:

(cHRM):
    HEX Signature = [0x00 0x20 0x63 0x48 0x52 0x4D 0x00 0x00]
    DEC Value = [0x0 0x32 0x99 0x72 0x82 0x77 0x0 0x0]
        *Total byte capacity = 44
        *Remove 36 more bytes starting after the signature

Code:

(gAMA):
    HEX Signature = [0x00 0x04 0x67 0x41 0x4D 0x41 0x00 0x00 0xB1 0x8F 0x0B 0xFC 0x61 0x05 0x00 0x00]
    DEC Value = [0x0 0x4 0x103 0x65 0x77 0x65 0x0 0x0 0x177 0x143 0x11 0x252 0x97 0x5 0x0 0x0]
    Length: 1 byte

Code:

(sRGB):
    HEX Signature = [0x01 0x73 0x52 0x47 0x42 0x00 0xAE 0xCE 0x1C 0xE9 0x00 0x00 0x00]
    DEC Value = [0x1 0x115 0x82 0x71 0x66 0x0 0x174 0x206 0x28 0x233 0x0 0x0 0x0]
    Length: 1 byte

Code:

(bKGD):
    HEX Signature = [0x06 0x62 0x4B 0x47 0x44 0x00 0xFF 0x00 0xFF 0x00 0xFF 0xA0 0xBD 0xA7 0x93 0x00 0x00 0x00]
    DEC Value = [0x6 0x98 0x75 0x71 0x68 0x0 0x255 0x0 0x255 0x0 0x255 0x160 0x189 0x167 0x147 0x0 0x0 0x0]
    Length: 6 bytes

Note the mid value bytes for 0xFF 0xFF 0xFF represent the background color in RGB format in hexadecimal form, hence why you now start to see some real values being the 255 max in the decimal version I found. The background image of the file I examined here was white.

Code:

(vpAg):
    HEX Signature = [0x00 0x09 0x76 0x70 0x41 0x67 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x10 0x00 0x5C 0xC6 0xAD 0xC3 0x00 0x00]
    DEC Value = [0x0 0x9 0x118 0x112 0x65 0x103 0x0 0x0 0x0 0x16 0x0 0x0 0x0 0x16 0x0 0x92 0x198 0x173 0x195 0x0 0x0]
    Length: 9 bytes

These are all demos of what i've extracted from my own personal debugging and testing, variable bytes do change but the signatures for each of these arbitrary byte chunks do not, which is why they are identifiable.

Now onto moving how I can modify the critical sectors of the PNG file structure with each of the IDAT chunks. All of what I did so far however, results in no lossy compression. It's all useless data kept in the background that most people wouldn't even realize was there in the first place.

200 bytes is not bragworthy at all...

[Peter L]

He is removing as much as he can without giving up quality. He is deleting it byte by byte so I think it is pretty impressive. A very interesting project here, nothing very practical but you still learn a lot doing things like this.

[AceInfinity]

200 bytes is not bragworthy at all...
slowly getting better

So if you had a file of 1kb, and you could only remove 500 bytes from it, that would be unimpressive just because it's a small amount?

PNG is already a very minimal file format, basically compressed to it's max from the time it's been written into a byte stream to disk. There's not much other than arbitrary data that can be removed, and even at the most that can only be about 300-400 bytes sometimes. Compare the filesize between gif, png, and jpeg and you'll see what I mean about filesize. PNG is usually a smaller file format (and there's a reason for that), along with gif, especially in comparison to jpeg, but PNG supports a better tolerance for transparency (which occupies 1 byte in the PNG file - JUST for transparency) than gif, and jpeg doesn't support transparency at all, yet jpeg can be about 5 times that of a PNG equivilant if you were to convert from jpeg over to PNG structure.

JPEG has SO much useless data, at a lossless and careless level that I would pretty much never even consider using it as a fileformat unless it's absolutely required. PNG basically has a PNG signature, the PNG header (IHDR), arbitrary data, IDAT chunks (the actual image data without formatting) and the end of the byte stream constant called IEND.

It's not much more than that, and there's not much you can do with the IDAT chunks. Before that and after the header is almost all that you can modify in reality without loosing quality, until you get into changing the transparency ore removing that. But it's only a difference of 1 byte, so I don't think it's necessary. If you want a transparent PNG at least you know you're not packing on a lot of bytes.