EXIF has TMI - Canon EOS 400D / Digital Rebel XTi and Focus Distance

[Philnick]

I've got one of the first Canon Digital Rebel XTi  (EOS 400D) bodies made - I got it in the fall of 2006 - my camera store let me trade-in the XT (350D) I'd bought a few months before.

I've recently started using Canon's Digital Photo Professional 3.9.2 (available for free download from Canon's European site at http://software.canon-europe.com/products/0010911.asp), since Adobe hasn't published a version of Adobe Camera Raw that supports the G12, which I also have.

I like that DPP has profile-based lens aberration correction - which has started me on a merry chase, since in opening my XTi's RAW files in DPP, I've found that it can't see the focus distance, though it can with the G12's RAW files. I was just about to buy a profile-based lens aberration correction tool when I found that I already have one in DPP! I can simply replace ACR with DPP in my workflow, having it write out 16 bit TIFFs to use in Bridge and Photoshop.

Only problem is that the lens aberration correction screen has a shooting distance slider that the G12's files automatically set, while it comes up at infinity for the XTi's files - and there isn't even any indication what you're setting it to other than an infinity symbol at one end!

I looked at the EXIF for the XTi's RAW files in Adobe Bridge, and saw that it was blank there, while the G12 listed a distance.

At first I assumed that the XTi was so old it wasn't writing the focus distance into EXIF, until I found ExifTool and saw that my XTi was writing both FocusDistanceUpper and FocusDistanceLower.

I assume that FDL and FDU are the outer bounds of the depth of field. (Incidentally, I find that FDL is the larger number, and FDU is the smaller number, which is counter-intuitive. Maybe Canon came up with these terms using an optical bench that put the camera at the top and the subject below it.)

I ran some G12 RAW files through ExifTool and found that it sets FocusDistanceLower at zero for all its files.

Apparently, the presence of a non-zero FDL in the XTi's EXIF is confusing Bridge and DPP!

What I'd like to do is use ExifTool's batch mode to set the EXIF in my four years' worth of XTi RAW files to be compatible with DPP by zeroing out FDL and putting the focus distance into FDU. (Is it possible to "comment out" the original FDL and FDU entries, to preserve them for future use while making them not interfere?)

I found the following formula in Wikipedia for determining the actual focus point from the depth of field boundaries:

subject distance s = 2*Near*Far/(Near+Far)

It calls this function the harmonic mean of those values. Fortunately, it doesn't require figuring out which is which, since they're treated the same in this formula!

What I want to do is set FDL to zero and set FDU to a new value equal to the harmonic mean of the original values:

Set FDU to equal = 2*FDU*FDL / (FDU + FDL)

I set up a spreadsheet to compute this, and found that it usually comes out with a value closer to FDU than to FDL, since FDU is the smaller number.

So the quick-and-dirty solution would be just to set FDL to zero and be done with it, but I'd rather calculate and put the correct value into FDU. Preserving the original FDU and FDL "commented out" for future use - maybe by inventing new tag names for them that would simply be ignored - would be nice, but not necessary.

Any ideas for automating this? I have just under 9,000 RAW files. I'm running Windows XP Pro SP3.

(My old Rebel XT files (EOS 350) are a lost cause, since their EXIF contain no focus distance info at all! Fortunately, I traded the XT in for the XTi only a few months after I got it.)

I hope this isn't too much of a challenge!

[Phil Harvey]

I find that FDL is the larger number, and FDU is the smaller number, which is counter-intuitive.

For most other models this is the other way around.  I'm not sure why the values are swapped for some models.

But the bad news is that the upper/lower distances don't represent the range of distances that is in focus.  They represent the accuracy of the measurement.  There are only about 10 or so steps in the focus distance range, and these two values are the bounding steps.  Adobe just takes the average of these two values to approximate the focus distance.

The G12 is odd in that the "Lower" value is always zero.  DPP seems to totally ignore the "Upper" value for this camera, and sets the slider based on the "Lower" value only.

I looked at the EXIF for the XTi's RAW files in Adobe Bridge, and saw that it was blank there, while the G12 listed a distance.

Yes, but where is Bridge getting this information?  I assume from the FocusDistanceUpper/Lower tags.  So I don't know where you plan to write the values from the XTi.  Have you run some tests and found something that enables the correction in DPP for the XTi?

Applying some formula to two tag values is easy with user-defined tags in ExifTool if this is what you need to do.  And batch processing thousands of files is easy too.  So ExifTool will do everything you need, but I don't know if this idea will work.

- Phil

[Philnick]

The G12 is odd in that the "Lower" value is always zero.  DPP seems to totally ignore the "Upper" value for this camera, and sets the slider based on the "Lower" value only.
. . .
I don't know where you plan to write the values from the XTi.  Have you run some tests and found something that enables the correction in DPP for the XTi?

What I found with my cameras' RAW files was that Adobe shows the value in FDU in my G12 files as the Subject Distance. But it claims not to see anything when FDL is non-zero, as with my XTi's files.

DPP's Info screen doesn't even have a line for distance, but it moves the slider with G12 files and leaves it at infinity for the XTi.

I experimented today with setting FDL to 0 an old XTi file. My hope was that this would make Bridge and DPP use the number in FDU as the distance.

In the unmodified file, FDU was 24.01 m and FDL was 33.95 m.

After zeroing out FDL, Bridge reported a Subject Distance not of 24.01 but 23.6 m!

23.6 isn't the average of 24.01 and 0, nor is it the harmonic mean, which is zero. How did Bridge come up with that value!

Worse, DPP still said "I know NOTHING!"

I averaged the original values and got 26.07. I put that into FDU, leaving FDL at 0, and Bridge updated its Subject Distance to 25.6 m! DPP still knew NOTHING!

I plugged the original values into my spreadsheet, which said that the distance (if those were the focus bounds) is 28.13. I put that into FDU, leaving FDL at 0, and Bridge promptly reported a subject distance of 27.7 m! DPP - well, you know.

Applying some formula to two tag values is easy with user-defined tags in ExifTool if this is what you need to do.  And batch processing thousands of files is easy too.  So ExifTool will do everything you need, but I don't know if this idea will work.

- Phil

Doesn't seem to work! There's obviously some other difference between the G12 files and the XTi files that is blocking DPP from seeing the distance. Anyone have any ideas?

[Philnick]

I remembered that the XTi files have the entry FocusRange=Not Known.

Maybe that's keeping DPP from seeing the value?

I tried deleting that tag. ExifTool replied:

Warning: Can't delete Canon:FocusRange
Nothing to do.

So I tried setting it to the average of FDU and FDL:

Warning: Can't convert Canon:FocusRange (not in PrintConv)
Nothing to do.

Then I decided to set it to "Known" which was carried out by ExifTool without complaint.

I had gone back to a copy of the original of the file, hoping that the two values would be used, but neither program saw them.

So I deleted FDL, making Bridge happier than usual, giving me back the exact FDU instead of a lower number.

DPP still knows nothing.

Reversed which was zero, and neither saw anything.

Upshot:
Regardless of whether FocusRange says Not Known or Known,  
giving them both values works with neither program,
giving them only FDL works with neither program
giving them only FDU works with Bridge, though it's happier with Known

PS In comparing print-outs of the XTi and G12 metadata (thank you ExifTool GUI!) I noticed that the G12 has FocusRange set to Auto, so I set the XTi's file to say the same thing.

FocusRange=Auto behaves exactly the same as FocusRange=Known

Any one have any other ideas?

[BogdanH]

Hi,

I've gone thru some Canon raw samples and as it seems, 400D is "special case" in this regard: content of FDL & FDU seems to be always reversed (even if original lens is being used). And no matter what values are inside FDU & FDL, DPP shows "Shooting distance information < ! >"  (exclamation point) for this camera.
IMO, FocusRange has nothing to do with this issue -it's "Not Known" for 450D too, but inside DPP, focus distance is displayed correctly for 450D (without exclamation point).
I assume, besides the FDL & FDU bug(?), 400D isn't writting some data (correctly), which is needed for DPP to "recognize" focus distance -that's why modifying/swaping FDL & FDU doesn't help.

Bogdan

[Philnick]

Bogdan:

I've mostly been working with older 400D files, firmware v. 1.0.4.

Last week, in preparation to use the CHKD firmware extensions that enable spotmetering and ISO 3200 on the 400D (as well as a dizzying 45 AF patterns - too many to cycle through, unfortunately), I upgraded the camera's firmware with version 1.1.1 from Canon's site.

I'll experiment with a newer shot and see if there's any improvement. Maybe they at least fixed the FDU/FDL swap. I'll see what I can find out.

Phil Olenick

[BogdanH]

Hi Phil Olenick,

All 400D files I could get are taken with fw 1.0.4 -so I'm very curious about your findings.

Bogdan

[Philnick]

Fortunately, I took my XTi out to the park yesterday with the new firmware v. 1.1.1 (and the CHDK hack) and got a nice series of shots in burst mode of a seagull taking off from Jamaica Pond.

As shot, it still had the Upper value (57.09 m) smaller than the Lower value (533.92), so that bug hasn't been fixed.

Neither program could make out the distance.

So swapped the two values.

DPP saw nothing, but the swap resulted in Bridge reporting a Subject Distance of 291.0 m, which is closer to the average of the two (318.54) than their harmonic mean (103.15). (That spreadsheet I constructed is turning out to be useful for diagnostic purposes, at least.)

However, this is the first time that either program saw anything with both FDU and FDL containing non-zero values, so clearly the bug, by swapping the values, stumps Bridge, which doesn't like FDL being the larger of the two.

Then I changed FocusRange from Not Known to Auto and Bridge still reported 291.0

Changed it to Known, and Bridge went mute about the distance. "Known" is apparently not a valid value for FocusRange - it may be a string value, and thus OK with ExifTool, but it's meaningless.

But as long as FocusRange is either Auto (as the G12 says) or Not Known (as the 400D says), swapping the two values so that FDU is the larger of the two is all that's needed to make Bridge happy enough to calculate an in-between value for Subject Distance.

So I went to one of my old files with firmware version 1.0.4 and swapped the numbers in FDU and FDL - it behaved exactly the same way.

However, with neither version of the firmware is there any pleasing DPP so far.

[Phil Harvey]

It is interesting that Bridge cares about the FocusRange.  This isn't stored as a string value, it is a number.   You can see the list of values in the Canon tag name documentation:

Code Select Expand

0 = Manual
1 = Auto
2 = Not Known
3 = Macro
4 = Very Close
5 = Close
6 = Middle Range
7 = Far Range
8 = Pan Focus
9 = Super Macro
10 = Infinity

- Phil

[Philnick]

So my next step is to design a batch process to fix the swapped values so at least Bridge will work with them properly.

After that, I have another idea: Phil, you mentioned that the focus distance upper and lower tags are populated with members of a set of ten numbers that represent ten distance ranges. The table of values for FocusRange you just posted also has ten steps (okay, eight other than Auto and Not Known). Are the threshholds for those eight ranges published? If FocusRange could be set to the value that represents the approximate focus distance, DPP might understand it enough to move its lens aberration distance slider.

What are those eight distances? I could look at what Bridge comes up with as the Subject Distance (SD) and put the corresponding value into FocusRange. I tried to reverse-engineer Bridge's formula, but the two frames I used, one 3 meters away and one nearly 300 meters away, Bridge's distances were 91% and 97% of the average.

It seems simplest to use the FocusRange value just larger than the average of FDL and FDU.

Could you post that table of distances?

What I envision doing, if all this works, is a three- or four-step batch process:

Do a run that tests to see if Model=Canon DIGITAL REBEL XTi (folks in other countries would have to use EOS 400D or even KISS DIGITAL X - whatever ExifTool shows in their files), and if so, to see if "FDU/FDLSwap" exists. If FDU/FDLSwap doesn't exist, it would create that tag and set it to equal the value of FDU.

Do a run that tests to see if FDU/FDLSwap equals FDU and if so, sets FDU to equal the value of FDL.

Do a run that tests to see if FDU and FDL are equal, and if so, sets FDL to equal the value of FDU/FDLSwap. FDU/FDLSwap would then be set to 0 to flag this as a file that's had the swap done, so that the next time you run this routine after a shoot, the values are not swapped back in the already processed files.

If ExifTool can average the two values and do a table lookup to insert the appropriate value into FocusRange, I'd include a fourth step to test whether FR=Not Known and if so, compute the right value for it and load it in.

But first a simple test. If I knew what distances constitute Macro, Very Close, Close, Middle Range, Far Range, Pan Focus, Super Macro, and Infinity, I could manually plug the appropriate value into FocusRange in my two test files and see if DPP's distance slider comes to life and stops saying <!>

Also, I'd appreciate a cheat sheet on how to write those batch process steps - particularly the one that does math and a table lookup!

[Philnick]

Tried putting a few values from the list into FocusRange for the image with the main focus 3 meters away.

Found that the numeric equivalents wouldn't be accepted, only the words, and with the two word values, I should just put in the first word, since the full two word phrase would just lead to an error message.

Tried Close, Very [Close], and Middle [Range]. The others seemed inapplicable.

As usual, Bridge was happy with any reasonable and permissible FocusRange entry, computing the same Subject Distance each time.

Unfortunately also as usual, all DPP said was <!>.

So it'll just be a three-step routine to make all 8,900 XTi RAW files behave in Bridge. I may have to use PTLens to do lens aberration correction for the XTi.

Can I just write a three line batch file and count on DOS to wait for each step to conclude before going on to the next?

Will this be an all day process with that many files?

Is this even worth bothering with? I'm not sure if PTLens - or any other tool, even makes use of the distance information - I do know that moving the distance slider in DPP makes a barely-visible difference.

[BogdanH]

Hi,

..Phil, you mentioned that the focus distance upper and lower tags are populated with members of a set of ten numbers that represent ten distance ranges...

I think, Phil just meant there's a limited set of distances. How many distances set actually contain, depends on lens only (for example, EF-S 17-55 f/2.8 contains a set of 28 possible distances). And as said, FocusRange value for this lens is always "Not Known" from 450D -still, DPP recognizes shooting distance.
IMO, 400D is writting some other tag with "wrong" value (or tag isn't written at all) -because of the same reason (bug?), FDL & FDU have swapped content.
I was hoping firmware v1.1.1 would solve this. Just curious: what's CHDK hack good for?

Bogdan

[Philnick]

Hi,

..Phil, you mentioned that the focus distance upper and lower tags are populated with members of a set of ten numbers that represent ten distance ranges...

I think, Phil just meant there's a limited set of distances. How many distances set actually contain, depends on lens only (for example, EF-S 17-55 f/2.8 contains a set of 28 possible distances). And as said, FocusRange value for this lens is always "Not Known" from 450D -still, DPP recognizes shooting distance.
IMO, 400D is writting some other tag with "wrong" value (or tag isn't written at all) -because of the same reason (bug?), FDL & FDU have swapped content.
I was hoping firmware v1.1.1 would solve this. Just curious: what's CHDK hack good for?

Bogdan

Well, I've given up for the time being on making DPP recognize shooting distance (see my last post before this) and I'll settle for creating a routine to massage my XTi files to swap FDL and FDU so Bridge and Photoshop - and thus Photoshop plugins - will see subject distance.

The Canon Hack Development Kit open-source group at CHDK.wikia.com started out by trying to make the inexpensive Powershots able to write RAW files and then expanded to all kinds of other goodies like expanding the ISO range in both directions, automating time-lapse sequences, motion-sensing shutters like catching lightning bolts, and a general scripting language.

They do this by installing a piece of firmware that looks, when the camera is turned on, to see if the memory card is bootable (they've created a tool called Card Tricks to make them bootable), and if it is a bootable card, it looks for a file in the root directory of the card called Autoexec.bin and loads it in. That file contains extensions to the camera's firmware that hook into the existing firmware to add additional functions.

If the camera is turned on with a card in it that isn't bootable, the extended functions aren't there.

Then they attracted Canon DSLR owners. The 400D / XTi hack is in its early stages of development and doesn't include the scripting language or most of the goodies. What it does include is an expanded ISO range (up to 3200), true spot metering (both of these borrow features from the 40D, which uses the same sensor as the 400D), and a set of 45 AF patterns using the 9 AF points, similar in concept to the 7D's, but gone wild to include not only contiguous clusters but lines, donuts, and even zig-zags. (The sheer number of them to cycle through reduces the usefulness of this feature, however.) Upgrading my XTi's firmware to version 1.1.1 was the first step in implementing that hack.

Most expanded features on a hacked 400D are accessed by going to a basic feature's screen and pressing the Direct Print button - which flashes blue when Autoexec.bin is loaded and to confirm when you access an extended feature. You get spot metering by going to the screen to pick a metering mode (Evaluative, Center-Weighted, or Averaged), highlighting Evaluative and pressing Direct Print, which takes you back to the main feature display with a dot instead one of those icons in the frame. You get a non-standard ISO by highlighting an ISO and pressing Direct Print to cycle through a group of other ISOs on the main feature display between the one you highlighted and the next standard value (except that doing this at 100 also cycles through some lower values as well). Highlight 1600 and you can choose between about five higher ISOs, topping out at 3200, which is the extended ISO on the 40D, and produces results that look pretty much the same as running the 40D at 3200.

To get one of the 45 focus patterns, you hit Focus Frame twice quickly and then Direct Print. Every successive press of Direct Print gets you a different pattern - but with 45 of them and no shortcut to any particular one this gets old fast!

Whatever variant of a basic setting you've chosen is written into the EXIF as that setting's value.

They had briefly implemented Auto ISO on the 400D but found that it was too slow to react, so they took it out.

Want to know how big Autoexec.bin is, which contains all these magic tricks? 14 kilobytes! (This reminds me of the original MS-DOS Flight Simulator, which modeled several kinds of aircraft and contained maps with images of landmarks across the USA - all on one 360K floppy! That programmer must have felt like God.)

It was the discussion board for the XTi hack that led me here. I had posted there asking whether they could make the XTi write focus distance into the EXIF and they responded that it was up to the lenses to relay distance to the camera, and posted a link to a page that listed a few that did and didn't, which showed that my 70-300 IS does, and had a link to a page at Magic Lantern that sourced some of its info to ExifTool.

I've been keeping them updated - in very broad outline - about what I've learned here, and I'll ask them if they can include a hack to fix the swapped FDU/FDL bug, along with including my marker to label files that are correct. The development forum for the 400D hack is at http://chdk.setepontos.com/index.php?topic=3290.0

If we ever figure out what the 400D writes (or omits) that screws up DPP, I'll ask them to fix that as well!

PS I'm unhappy that the new 60D writes such a skimpy set of EXIF data. Only 14 tags, not including FDU, FDL or FR? It looks like it was rushed out the door! I hope they flesh that out soon - that's the camera I want to upgrade to! (I've gotten spoiled by the articulating screen on the G12 - which lets me shoot from my lap to be inconspicuous and also lets me steady the camera by shooting from a tabletop.)

[Philnick]

Bogdan,

Could you attach a textfile dump of the EXIF from a 450D that I could compare side by side with one of my 400D dumps to look for a difference? Maybe that can help me find what's going on with DPP. I attach one of my firmware version 1.1.1 EXIF dumps (with the CHDK hack) from a shot taken a few days ago and dump of the EXIF of one of my version 1.0.4 shots from May.

These are from shots where I've manually swapped the values of FDU and FDL to make Bridge show a Subject Distance.

"Never give up. Never surrender."
                    - Commander Taggart, Galaxy Quest

[Philnick]

Just opened up a pair of my RAW files in Photoshop CS4, one doctored to undo the FDU/FDL swap, and one not.

On the File Info listing, which shows a lot of the EXIF data, neither of them includes anything containing the string "focus": not FocusDistanceUpper or Lower and not FocusRange either.

The doctored one, however, includes the Subject Distance displayed by Bridge, in this format:

 <exif:SubjectDistance>291/1</exif:SubjectDistance>

The undoctored one, with FDU smaller than FDL, does not include that line.

Since I opened the image directly from Photoshop's File Open menu, it was processed by Adobe Camera Raw, but not by Bridge - so it's apparently ACR that does the calculation and supplies it to Bridge and Photoshop.

I'll have Photoshop - and DPP - write TIFFs of the same doctored RAW file and see if their EXIF differ.