Adobe DNG and the 'danger' of writing to NEF, CR2 etc

[dgordon]

Adobe, and their paid cheerleaders, want me to convert all my camera raw files in to DNG format. As you know, they say its better than having my metadata in a separate XML side-car and I really really should never ever write data into my proprietary raw files. Its far to dangerous say Adobe because we don't have all the information about these files to guarantee we aren't writing somewhere we shouldn't and thus potentially corrupting our valuable original images.

How much of the Adobe line is FUD?

I'm hoping people here, who have a much better understanding of how raw files work, can explain if Adobe have a good point, or just want me to use their systems.

I'd rather not convert all my files to DNG format as its another (time consuming) step in my workflow. I'm reasonably happy to have XML side-cars containing my metadata but the best solution is having it embedded in the file (yes, in the way a DNG works!).

I'm using Apple's Aperture to catalogue and process my raw files. If I want to update my files with metadata entered using Aperture I can "write IPTC metadata to masters". I still get a feeling I ought not to do this, have I just read too much Adobe propaganda? What could possibly go wrong?

Thanks

[Phil Harvey]

It is sad if what you say is true.

First of all.  If you are considering converting your RAW files to DNG because you think this is safer, STOP NOW!  This is the most dangerous thing you could do (unless you choose to embed the original RAW image inside the DNG), because proprietary information is often lost in the DNG conversion.  This is a known problem.

The DNG converter must understand the structure of the proprietary information in the original RAW image in order to successfully preserve it in the DNG.  This is the same requirement for writing to the RAW file in the first place.  Unfortunately though, the Adobe DNG converter doesn't understand this structure as well as ExifTool, and information is lost for some RAW file formats.

But even if the Adobe DNG converter didn't corrupt proprietary information, converting to DNG still wouldn't make sense.  The stupid thing is that the DNG converter often converts the proprietary manufacturer information from the RAW image into a different format of proprietary Adobe information in the DNG.  There are parts of the DNG format that are proprietary and undocumented, which is something many people don't realize.  So in this respect, the DNG isn't much better than the original RAW image.  The only advantage is that Adobe specifically states that the proprietary information must be relocatable, which means that it shouldn't be corrupted (any further) by rewriting the DNG.  Here is all that the DNG 1.3 specification has to say about this:

    DNGPrivateData provides a way for camera manufacturers to store private data in the DNG
    file for use by their own raw converters, and to have that data preserved by programs that edit
    DNG files.

    The private data must follow these rules:

    •The private data must start with a null-terminated ASCII string identifying the data. The
    first part of this string must be the manufacturer's name, to avoid conflicts between
    manufacturers.

    • The private data must be self-contained. All offsets within the private data must be offsets
    relative to the start of the private data, and must not point to bytes outside the private data.

    • The private data must be byte-order independent. If a DNG file is converted from a big-
    endian file to a little-endian file, the data must remain valid.

But you are correct that there is always a chance of corrupting information when writing to a RAW file.  In fact, there is always a chance of corrupting information when writing to any file!  The question is, what sort of a risk are you willing to live with?  The safest option in all situations it to always keep backups of your original RAW images, which is what I recommend.  However, having said this, writing metadata to RAW images using ExifTool is probably safer than writing metadata to JPEG images using any other software.

- Phil

[BogdanH]

Hi dgordon,

I've followed DNG development since the beginning. I mean, DNG idea (as presented) isn't bad, IF major camera makers would follow it. According to wiki (http://en.wikipedia.org/wiki/Digital_Negative), first DNG spec came out in 2004 and not much has changed since then: DNG is still expected to be standardized by ISO "anytime soon".
The problem (as I see it) is, camera makers don't have a time to wait endlessly, for specifications to be approved (within "standard") by some organisation: as soon they make new sensor and/or some new data is needed to be stored into file, they must act immediately -and they can do that only with their own file format.
Adobe is selling imaging software and of course, "proprietary" files are difficult to manage (for them). I understand that, I really do -but hey, that's not MY problem.
Funny thing is, it's not Adobe that is spreading FUD about editing metadata inside original raw files. Adobe has just planted a seed... the rest is done by "believers", where many of them don't really know much about metadata.
I have written a short article about this here:
https://exiftool.org/gui/articles/tiff_raw.html
-you don't need to understand it fully. The point being: editing metadata inside DNG or i.e. CR2 is the same thing! It is the same "risk"!
The next thing I read often is, "will you be able to open proprietary raw files in, say, 20,.. 50.. years? -or when camera maker dissapears?". I won't even comment such stupid argumnets.

I don't feel the need to convince you into anything. Use you own head, it's your files, it's your decission -no matter what you do, world won't fall apart 

Bogdan

[dgordon]

Thank you both for those replies, very informative.

I had believed that camera manufacture's raw files were some kind of TIFF variant, but it seems from reading Bogdan's article they are not as different as I thought.

Let's see if I understood (in general)...

A CR2 is a TIFF as far as its own metadata is concerned. I'm not clear on how the image data is different from a TIFF (yet!).

The makernotes section is part of the TIFF specification. I had thought this was where a CR2 parted ways from a TIFF and why people make a fuss about not embedding their own metadata to the file. I though it was an unknown in an unknown place which had to be tip-toed around.

So I'm going to continue to use CR2s and embed my IPTC metadata into them. Naturally I do have backups, I keep a set of 'originals' elsewhere. I want to have my captions and keywords in my files as it allow me move to a different cataloguing application should I wish or need to in the future.

David

[BogdanH]

Hi David,

Yes, if one know how to read metadata from TIFF files, then he can also read metadata from CR2 files. And as I mentioned in article, it is specified, that Exif metadata can be saved into TIFF file.. and it is specified by Exif, that Makernotes section can exist if needed. Exif standard is clear about that:
MakerNote
A tag for manufacturers of Exif/DCF writers to record any desired information. The contents are up to the manufacturer, but this tag shall not be used for any other than its intended purpose.
-so, CR2 has everything by the book.
The problem is, that content of MakerNote, is a "black box". So, if you're not 100% sure what's going in there, then you should not modify MakerNote content. However, those tags we are talking most of the time (DateTime, Copyright, GPS, Location, etc..) are always saved outside MakerNote: they are saved inside i.e. Exif (or Xmp) and here, as said, everything is standardized.
Now, one might ask: "and where does Canon state, that CR2 files are using TIFF specification?". As far I know, nowhere -it's just a fact.
I have started writting Xmp metadata into CR2 files in year 2006 (when I discovered ExifTool).. and now wonder: years later, Canon decided to write metadata into Xmp as well (Rating tag). Another example: In year 2010, Metadata Working Group (MWG) organisation recommended using UTF-8 character set for Exif. And guess what: Canon software allready accept UTF-8 inside Exif (btw. among others, members of MWG are Adobe, Apple, Canon, etc.).

Image data... TIFF can contain any image data: it can be plain RGB image data, it can be jpeg encoded image data,.. and it can be (jpeg lossless compressed) raw image data, or any combination of these. TIFF allows that. The problem (again) is, raw image data encoding mechanism depends on camera manufacturer. Anyway, famous Dave Coffin started with decoding raw images and wrote decoding open source software (DCraw) -which is a starting point for many raw converters we have today.
Going back to topic: when we modify metadata tags content, we don't touch image data itself. That is, as long we change only standardized (Exif, Xmp,..) metadata tags inside CR2, the "danger" is the same as we are modifying metadata inside TIFF (or JPG) file: if we use badly written tool, image can be lost.
For whatever reason, some prefer Xmp sidecar files, some convert to DNG,.. and some write into CR2 -it's a personal decission.

Bogdan