'extra' lossless compression for camera raw images

I just finished writing a small tool (sgraw) which can losslessly compress
camera raw images, usually between 20-to-60%. (Regulars here might recall I
have mentioned this 'possibility' earlier, I finally put in some hours to
actually build a working demo :-)

http://www.sachingarg.com/compression/sgraw/

That would be savings between 1GB to 3GB for every 5GB of data.

Even when camera has already compressed the raw image (which isn't always
the case), its still possible to squeeze out extra compression as cameras
have to use simple algorithms for speed.

Compression is totally lossless, we get back the exact file we start with
(just like with zip), so both pixel information and meta-data are perfectly
restored.

This is just the first quick hack version (and supports all nikon and fuji
cameras), so it obviously wouldn't make sense to actually use it for backups
etc, but if there is enough interest I can build it further to support other
raw formats, along with improved compression and speed.

Your thoughts on this?

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

"Sachin Garg" <sachingarg@c10n.info> wrote:

>
>I just finished writing a small tool (sgraw) which can losslessly compress
>camera raw images, usually between 20-to-60%. (Regulars here might recall I
>have mentioned this 'possibility' earlier, I finally put in some hours to
>actually build a working demo :-)
>
> http://www.sachingarg.com/compression/sgraw/
>
>That would be savings between 1GB to 3GB for every 5GB of data.
>
>Even when camera has already compressed the raw image (which isn't always
>the case), its still possible to squeeze out extra compression as cameras
>have to use simple algorithms for speed.
>
>Compression is totally lossless, we get back the exact file we start with
>(just like with zip), so both pixel information and meta-data are perfectly
>restored.
>
>This is just the first quick hack version (and supports all nikon and fuji
>cameras), so it obviously wouldn't make sense to actually use it for backups
>etc, but if there is enough interest I can build it further to support other
>raw formats, along with improved compression and speed.
>
>Your thoughts on this?


You should patent it.

"Sachin Garg" <sachingarg@c10n.info> wrote in
news:frp6tm$8dv$1@aioe.org:

>
> I just finished writing a small tool (sgraw) which can losslessly
> compress camera raw images, usually between 20-to-60%. (Regulars
> here might recall I have mentioned this 'possibility' earlier, I
> finally put in some hours to actually build a working demo :-)
>
> http://www.sachingarg.com/compression/sgraw/
>
> That would be savings between 1GB to 3GB for every 5GB of data.
>
Do you have any experience with freeware tools [preferably
commandline] which do Lossy Compression of jpegs?
The tool by JDO works well but it is proprietary and not freeware.
It can regularly wring out 30% savings in byte size and still retain
the "viewability" as a jpg file. Of course quality will be lost
depending on the quality of the input..

The RAW file is already pretty low in file size compared to the PSD and TIFF
file size. Harddrive space is also pretty low in cost too. So, would I use
it? Unfortunately no.




"Sachin Garg" <sachingarg@c10n.info> wrote in message
news:frp6tm$8dv$1@aioe.org...
>
> I just finished writing a small tool (sgraw) which can losslessly compress
> camera raw images, usually between 20-to-60%. (Regulars here might recall
> I have mentioned this 'possibility' earlier, I finally put in some hours
> to actually build a working demo :-)
>
> http://www.sachingarg.com/compression/sgraw/
>
> That would be savings between 1GB to 3GB for every 5GB of data.
>
> Even when camera has already compressed the raw image (which isn't always
> the case), its still possible to squeeze out extra compression as cameras
> have to use simple algorithms for speed.
>
> Compression is totally lossless, we get back the exact file we start with
> (just like with zip), so both pixel information and meta-data are
> perfectly restored.
>
> This is just the first quick hack version (and supports all nikon and fuji
> cameras), so it obviously wouldn't make sense to actually use it for
> backups etc, but if there is enough interest I can build it further to
> support other raw formats, along with improved compression and speed.
>
> Your thoughts on this?
>
> Sachin Garg [India]
> www.sachingarg.com | www.imagecompression.info
>
>

On Wed, 19 Mar 2008 01:28:47 +0530, "Sachin Garg"
<sachingarg@c10n.info> wrote:

>
>I just finished writing a small tool (sgraw) which can losslessly compress
>camera raw images, usually between 20-to-60%. (Regulars here might recall I
>have mentioned this 'possibility' earlier, I finally put in some hours to
>actually build a working demo :-)
>
> http://www.sachingarg.com/compression/sgraw/
>
>That would be savings between 1GB to 3GB for every 5GB of data.
>
>Even when camera has already compressed the raw image (which isn't always
>the case), its still possible to squeeze out extra compression as cameras
>have to use simple algorithms for speed.

How much more can you squeeze out of a compressed raw image? When I
shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
files. I get about 9% further compression. I don't know exactly what
compression algorithm Nikon uses but I doubt it's much more
complicated than a requantization of the data to reduce the bits per
pixel. Nothing near as complicated as what bzip2 is doing.

9% savings isn't worth the effort for me. But 20-60% definitely would
be if you get that with compressed raw.

Steve

Sachin Garg wrote:
> I just finished writing a small tool (sgraw) which can losslessly compress
> camera raw images, usually between 20-to-60%. (Regulars here might recall I
> have mentioned this 'possibility' earlier, I finally put in some hours to
> actually build a working demo :-)
>
> http://www.sachingarg.com/compression/sgraw/
>
> That would be savings between 1GB to 3GB for every 5GB of data.
>
> Even when camera has already compressed the raw image (which isn't always
> the case), its still possible to squeeze out extra compression as cameras
> have to use simple algorithms for speed.
>
> Compression is totally lossless, we get back the exact file we start with
> (just like with zip), so both pixel information and meta-data are perfectly
> restored.
>
> This is just the first quick hack version (and supports all nikon and fuji
> cameras), so it obviously wouldn't make sense to actually use it for backups
> etc, but if there is enough interest I can build it further to support other
> raw formats, along with improved compression and speed.
>
> Your thoughts on this?

For GP use? No biggie. Disk space is cheap.

If it can be reduced to a h/w implementation (by the camera co's) that
is more energy efficient (and faster), then it would be a great thing to
improve in camera storage.

Once you offload from the camera (flash) to disk, the memory is not that
important (to me). Disk space is cheap.

Cheers,
Alan

--
-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
-- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
-- [SI] gallery & rulz: http://www.pbase.com/shootin
-- e-meil: Remove FreeLunch.

"Tony Polson" <docnews2011@googlemail.com> wrote in message
news:0f80u3lcs4dojhkq99glrlca7a6ehr4691@4ax.com...
> "Sachin Garg" <sachingarg@c10n.info> wrote:
>>
>>I just finished writing a small tool (sgraw) which can losslessly compress
>>camera raw images, usually between 20-to-60%. (Regulars here might recall
>>I
>>have mentioned this 'possibility' earlier, I finally put in some hours to
>>actually build a working demo :-)
>>
>> http://www.sachingarg.com/compression/sgraw/
>>
>>That would be savings between 1GB to 3GB for every 5GB of data.
[]
> You should patent it.

Thanks for that tip. Like all research my work is also based on hard work
from many other researchers, I will explore if there are any patentable
parts.

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

"Mike Mills" <efficient@canada.com> wrote in message
news:Xns9A65A9516DCBDefficientcanadacom@194.177.96 .78...
> "Sachin Garg" <sachingarg@c10n.info> wrote in
> news:frp6tm$8dv$1@aioe.org:
>
>>
>> I just finished writing a small tool (sgraw) which can losslessly
>> compress camera raw images, usually between 20-to-60%. (Regulars
>> here might recall I have mentioned this 'possibility' earlier, I
>> finally put in some hours to actually build a working demo :-)
>>
>> http://www.sachingarg.com/compression/sgraw/
>>
>> That would be savings between 1GB to 3GB for every 5GB of data.
>>
> Do you have any experience with freeware tools [preferably
> commandline] which do Lossy Compression of jpegs?

If you want lossy compression on commandline, you can try the opensource
options like "libjpeg" or "libpnm". Even if you don't want to play with
sourcecode, you should be able to find its (free) executables for any
popular platform.

On the other hand, if you want further 'lossless' compression of Jpeg files,
look for "jpegoptim" (also opensource). Many popular image editors like
photoshop and irfanview etc also allow the jpegoptim optimizations in batch
mode (if you dont want to use a commandline tool).

> The tool by JDO works well but it is proprietary and not freeware.
> It can regularly wring out 30% savings in byte size and still retain
> the "viewability" as a jpg file. Of course quality will be lost
> depending on the quality of the input..

You probably already know that moderate jpeg loss is less of a problem when
'viewing', but it hurts when editing or priniting after the fact. Dont do
'too much' of it :-)

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

"Ali" <me@privacy.com> wrote in message
news:1u2dnY24UYDpqX3anZ2dneKdnZydnZ2d@pipex.net...
> "Sachin Garg" <sachingarg@c10n.info> wrote in message
> news:frp6tm$8dv$1@aioe.org...
>>
>> I just finished writing a small tool (sgraw) which can losslessly
>> compress camera raw images, usually between 20-to-60%. (Regulars here
>> might recall I have mentioned this 'possibility' earlier, I finally put
>> in some hours to actually build a working demo :-)
>>
>> http://www.sachingarg.com/compression/sgraw/
>>
>> That would be savings between 1GB to 3GB for every 5GB of data.
>
> The RAW file is already pretty low in file size compared to the PSD and
> TIFF file size.

Yep, that is correct. Similar savings are 'technically possible' on TIFF and
PSD files too, but that would be another story.

> Harddrive space is also pretty low in cost too. So, would I use it?
> Unfortunately no.

ok

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info

"Steve" <steve@example.com> wrote in message
news:mtl0u3pcgmlsd0npmuj0bmbeb1952bdedt@4ax.com...
> On Wed, 19 Mar 2008 01:28:47 +0530, "Sachin Garg"
> <sachingarg@c10n.info> wrote:
>
>>
>>I just finished writing a small tool (sgraw) which can losslessly compress
>>camera raw images, usually between 20-to-60%. (Regulars here might recall
>>I
>>have mentioned this 'possibility' earlier, I finally put in some hours to
>>actually build a working demo :-)
>>
>> http://www.sachingarg.com/compression/sgraw/
>>
>>That would be savings between 1GB to 3GB for every 5GB of data.
>>
>>Even when camera has already compressed the raw image (which isn't always
>>the case), its still possible to squeeze out extra compression as cameras
>>have to use simple algorithms for speed.
>
> How much more can you squeeze out of a compressed raw image? When I
> shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
> files. I get about 9% further compression. I don't know exactly what
> compression algorithm Nikon uses but I doubt it's much more
> complicated than a requantization of the data to reduce the bits per
> pixel. Nothing near as complicated as what bzip2 is doing.

While Nikon does uses the requantization you mention, they also compress
those quantized pixels further using lossless encoding. And yes, its still a
'simple' algorithm not as complicated as bzip2, but bzip2 type methods
(based on bwt) can't perform good on 2D data like images.

> 9% savings isn't worth the effort for me. But 20-60% definitely would
> be if you get that with compressed raw.

You should be able to get atleast 20-30% even on compressed raw files using
sgraw.

If you try the tool on your images, I would love to see what results you
get. In case you get anything less than that it will mean I can further
improve it for your camera's raw files.

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info