CR3 fileindex and directoryindex not available

[Martin B.]

Hi,

I just noticed -fileindex and -directoryindex return nothing for my CR3 files (Canon G5Xii and R6), but they return the correct information for CR2 files.

This is for exiftool 12.60. I checked the version history up to 12.65, and there's no mention of these tags being added.

Is there a known solution?
Is this because the information isn't in the CR3 files? I doubt it, since both cameras can display this information.

I'm willing to spend time to try to find the location of this information in CR3 files, if that can help, so I'd appreciate pointers if you have any.

Thanks!

[Phil Harvey]

If the information is there, you should be able to find it in the -U output.  Let me know if you discover anything.  I should be able to help when I get some time if you upload a picture with a known file and directory index (preferably some unique number for both to make it easier to find... definitely not 100 for the directory index because this is too common).

- Phil

[Martin B.]

Thanks Phil,

I thought I had found the fileindex using the -U output. It was not obvious and I resorted to correlate in Excel each "Canon Camera Info Unknown" tag with the numerical part of the file name, on a number of consecutive files. I found that one combination of two adjacent tags correlates 100% with the file name, but it doesn't wrap at 9999, it isn't the same as the numerical part of the file name, and I cannot find a directory index nearby.

So I think I may have found the shutter count, which is still a very useful value to decode. I'm validating this over my whole catalog of R6 files. I'll have more to say tomorrow.

[Martin B.]

So, I couldn't find a file or directory index for the Canon R6 CR3 files, but I found the ShutterCount.

• The "Canon Camera Info Unknown 0x0af1" and "Canon Camera Info Unknown 0x0af2" tags are part of the shutter count for the Canon R6.
• 0x0af2 is the high byte, and 0x0af1 is the low byte.
• I have not found a single instance of 0x0af0, 0x0af3, or 0x0af4 being other than zero in over 18,000 files, so I suspect one of these is also part of the ShutterCount, because I doubt this counter would be designed to wrap to zero after 65535.
• These tags are valid for the CR3 and JPG files (but not MP4) produced by my R6.
• The same tags are always zero for my Canon G5Xii.

The rest of this message just provides details and explains how I found and validated the above statements.

I found this tag by correlating all "Canon Camera Info Unknown" tags (there are 4608 such tags) against the file name, over 13 files. I used the Excel CORREL function (it took a while and I have a fast desktop). The summary of this investigation is:

• 3703 tags yielded a #DIV/0! error, probably because their values didn't change over the 13 files
• only one tag had a correlation of 100%, tag 0x0af1, indicating this tag is a perfect predictor of the file name
• other tags with high but imperfect correlation were 0x0b85 (97%), 0x0b83 (92%), and 0x0b81 (91%). They happen to be adjacent; perhaps they related to the time the photos were taken. Or could they be double-byte encoding of the file index??
• I would never have found this tag without this brute-force correlation approach because its values were different from the file numbers (and values above 128 happen to be listed as negative decimal numbers by ExifTool).
• I was lucky that, for my sequence of files, 0x0af1 didn't wrap around 0xFF, because this tag is the low byte of the counter. Had it wrapped, correlation would not have been 100%.

In my photo collection, this counter ranges from 1 to 18,395, increasing monotonically with time and without a single duplicate.

These tags work for the CR3 and JPG formats of my R6, and the counter increments continuously, regardless of the image format.

This number is not the same as the file index, because the file index wraps at 9999, or can be reset when changing memory cards, etc. For instance, a recent photo I took is named _MBL6521.CR3 and its counter is 18384 (0x47D0). If this counter was the file index, it would be 6521 (0x1979).

This counter advances by 1 for each photo file, regardless of mechanical or electronic shutter.

For video files (MP4): Exiftool doesn't report these tags. The file name advances with each video file, but the counter doesn't change. For instance, taking a video and then a photo increments the counter by only one, even though two files are produced.

Switching to another card doesn't change the behavior of the counter; it increments as if I hadn't changed cards (my camera is set to continuous file numbering).

Saving to two cards at the same time (e.g. RAW on one card, JPG on the other, or JPG on both cards), does the expected thing: the counter is the same for the twin photos on each card.

Creating a second directory (e.g. Canon101) on an existing card from the camera has no influence on the counter (counter and file number increment, as expected).

Finally, manually resetting the counter (which automatically creates a new directory) from the camera has no influence on the counter (it increments but the file number resets to 0001, as expected).

My R6 currently has firmware 1.8.2. The first picture was on firmware 1.2, and I've upgraded frequently since then. I haven't noticed any change in behavior of this counter over the sequence of files, so I don't think this is a factor. I have never done a full reset of my camera.

All data was extracted with ExifTool 12.60.

I can provide sample files, in CR3 and JPG format. I can also provide the output of exiftool -U -w %f_%e.txt for a large number of files. Just let me know what would be useful.

I can also provide the Excel files I used to correlate and validate the tags. This correlation method could be used to find other tags, provided one has a reasonable estimate of the tag values for a representative number of files.

If someone has an R6 with a shutter count higher than 65535 could provide sample files, it would be nice to validate my bullet number 3 above.

The same tags are always zero for my G5Xii. Perhaps other tags are the file index or shutter counter, but I haven't investigated this yet. I will.

[Martin B.]

Given the above information, is it possible to add this tag to Exiftool?
There are references to ShutterCount in the documentation for the Canon tags, but Exiftool doesn't (yet) support it on the Canon R6.

Thanks!

[Phil Harvey]

Sorry for the delay in responding.  I will add your ShutterCount discovery to ExifTool 12.68.

Thanks!!

- Phil

[Martin B.]

Thanks Phil.

Now that I have implemented this manual, brute-force, method, I'll try it again with my Canon G5X ii. The only thing I know now is it isn't the same tag as in the R6 (even though they are both using CR3). I don't even know if the shutter count or file number are available for the G5X ii.

Would it be useful to automate this method for other tags and models? Essentially, it requires taking multiple pictures while varying a single parameter, and correlating all unknown tags against values of this parameter. A 100% correlation indicates which tag it is. I think it's possible to write a Perl script and feed it with:

• the list of unknown tags
• a list of pairs containing (image file, value of parameter for this file)

Provided enough pairs are provided to calculate a significant correlation, the script would list the top 5 unknown tags along with their correlation with the provided parameter values.

If it's useful, I'll do it and make it available.

[Phil Harvey]

One problem is that ImageNumber, FileNumber and ShutterCount all increment once per image.  I already have a script that looks for bytes that increment between pictures (and much more).  The problem is that it is time consuming and difficult to do this for all camera models.

- Phil

[Martin B.]

I'm proposing to generalize the concept, going beyond incrementing bytes between pictures, and use regression over a table like this (assuming we're looking for a tag that stores the f-stop):

Known value	File
1.4	img_0033.cr3
2.8	img_0034.cr3
4	img_0036.cr3
5.6	img_0042.cr3
8	img_0061.cr3
8	img_0062.cr3
8	img_0063.cr3
8	img_0064.cr3
11	img_0108.cr3
22	img_0035.cr3
16	img_0037.cr3

and get a result like this:

Tag	Correlation (%)
0x0ad1	100
0x0b16	100
0x0c11	98
0x0756	96
0x0af9	83

What I don't know is if enough people would use it and hand you the results, freeing you from doing it yourself.

[Phil Harvey]

Interesting.  My version doesn't go that far.  What you have proposed could be very interesting and potentially useful, but for things like F-stop you would need to look for non-linear correlations (in this case, logarithmic).  Plus, you would have to check all possible storage formats (IEEE float, fixed point, integer, rational, 1/2/4/8-byte, little/big-endian).  The number of possibilities is huge.  I think the project may be more complicated than you expect, and running the correlations on this many variables with a number of files will take a significant number of CPU cycles (I'm thinking hours of processing).

- Phil