Found: Canon G5X-ii (and some Canon R6) fileindex, directoryindex, shuttercount

[Martin B.]

Following up on my previous post regarding the Canon R6 shutter count (CR3 fileindex and directoryindex not available), I used the same technique, and some new ones, to find more tags for these two cameras.

Here's the summary (the tags I recommend adding to Exiftool are in blue):

	|	Canon R6 CR3	|	Canon R6 JPG	|	Canon G5X mark II CR3	|	Canon G5X mark II JPG	|
	|		|		|		|		|
Shutter count	|	Canon Camera Info Unknown 0x0af2 + 0x0af1	|	Same as CR3	|	Canon Camera Info Unknown 0x0a96 + 0x0a95	|	Canon Camera Info Unknown 0x0294 + 0x0293	|
File index	|	not found	|	not found	|	not found	|	Canon Camera Info Unknown 0x0b2e + 0x0b2d (V+1)	|
Directory index	|	not investigated	|	not investigated	|	not found	|	Canon Camera Info Unknown 0x0b22 + 0x0b21	|
Directory index	|		|		|		|	Canon Camera Info Unknown 0x0b3a + 0x0b39 (V-1)	|

In the above table:

• Tags linked by a plus (+) sign indicate that the first tag is the high byte and the second is the low byte of the tag value.
  
• "not found" means I searched for the tag and didn't find any match, whereas "not investigated" means I didn't try to correlate this tag with known values. Thus, I believe the "not found" tags don't exist in the file, while I'm not sure if the "not investigated" tags are present or not in the file. The tags I examined are those given by exiftool -U -w out/%f_%e.txt "-Canon_CameraInfoUnknown_*" "-Canon_shotinfo_*" "-Canon_fileinfo_*" "-Canon_CMP1_*" "-Canon_IAD1_*" .
  I was looking to meet at least one of the following criteria:
  
  • correlate 100% with the known value;
  • a constant difference between the combinations of the next and current tags (in a high/low byte fashion) and the known values;
  • or for the tag values to have at least the same range (the value of max minus min) as the known values and for tag values to be proportional to the known values.
• A string in parentheses next to a tag indicates how to get the desired value from the tag value. For example, to get the Canon G5Xii directory value from the Canon Camera Info Unknown 0x0b3a+0x0b39 tag in the JPG file, one needs to take the tag value and subtract 1 from it. Thus, a tag value of 106 corresponds to a directory value of 106-1=105. In this particular case, it is simpler to use the Canon Camera Info Unknown 0x0b22+0x0b21 tag, because it is equal to the directory value, but I'm publishing any alternate tags here for two reasons: to eliminate them from other searches (they are no longer "unknown"), and to facilitate searches for other tags (the file index is often near the directory index).
  
• To identify candidate tags, I searched with Exiftool 12.60 on Windows 10, over 10 to 33 JPG or CR3 files. Once a candidate tag was found, the validation was done over more files, which did eliminate some candidates.
  

Details for the Canon G5X mark II
The shutter count behaves the same way for this camera as for the R6:

• advances by 1 for each photo file, regardless of mechanical or electronic shutter
• does not advance for regular video files
• advances for time lapse video files
• creating a new directory or resetting the counter from the menu doesn't affect this shutter count

For JPG files:
As indicated by the (V+1) note, the file index is one less than the file name, e.g. file IMG_0001.JPG will have a fileindex tag value of 0000.

Strangely, the directory index occurs in two locations, with a difference of one (1) between them. I recommend using the Canon Camera Info Unknown 0x0b22 + 0x0b21 tag because it matches the folder name on the memory card and in the camera.

For CR3 files:
The shutter count is in two different tags for CR3 and JPG files (whereas the R6 used the same tag regardless of the file type).
I validated the shutter count over about 4400 files; this is all I've got in my library. This means I cannot validate what happens when the shutter count crosses a 16-bit boundary. I assume it does the same thing as for the Canon R6.

For MP4 files:
I couldn't find any shutter count or file index in MP4 files by correlating the file name with any of the tags generated by the following command:
exiftool -U -ext mp4 -w out/%f_%e.txt "-Canon_shotinfo_*" "-Canon_fileinfo_*" .

Details for the Canon R6
No new tags, but I eliminated some possibilities ("not found"), even after more investigation (see below). I also repeated here the top left table entry on the shutter count from my other post.

In addition, I also verified that the shutter count increases when doing timelapse movies. Even though the output is a video file, the shutter count increases by the number of images in the resulting video file. This makes sense, since those are really a series of photos assembled as a video file. Regular video files do not affect the shutter count.

Because the G5X mark II has file index information in the JPG files, I also searched in JPG files for the R6, but I found no 100% correlation. I did find a couple of tags that somewhat correlated, but there were enough exceptions (e.g. the tag value decreased for a few files while the file number increased) or just no correlation when combining two bytes (because all counters I searched can go beyond 255, thus requiring two bytes) to rule out these tags. I think they are time related because the non-monotonicity tended to occur when I paused between two sets of photos.

Thoughts
I find it odd that the file-index and directory-index values are not in the CR3 files. They appeared in the CR2 files of all my previous Canon models, and they appear in the JPG files of the G5X-ii. I spent days correlating and searching most "unknown" tags in CR3 files and found nothing. Could some tags be mislabelled (Color Temp, Vignetting, etc.)? Could some tags still be unreported? I don't know.

I'm not sure it's possible, but it would have helped my search if I had known in advance which unknown tag could be the high byte of which other tag. Some tags seemed to be reported as signed bytes, whereas others were obviously larger (at least 16 bits).

I understand (really well now  ) that it takes a lot of effort to decipher image formats and find new tags. I hope this helps.

And thanks again to Phil for a great piece of software!

[Phil Harvey]

Good job!  It took me a while to read through your post and verify your findings with my samples, but I've done that now and I'll add your G5XmarkII discoveries in the next release (12.70).

Usually integers in the maker notes are stored in the ExifByteOrder, but this isn't always the case.

I took a quick look in the R6 CR3's for the DirectoryIndex, but didn't see it.  This is odd because the CameraInfo record exists in the CR3.

- Phil

[Martin B.]

Hi Phil,

I see the new CameraInfoG5XII tags are documented for version 12.70. Thanks!

Note that the documentation (https://exiftool.org/TagNames/Canon.html#CameraInfoG5XII) says "CameraInfo tags for the EOS R6.", which is probably just a copy-paste oversight.

One question, though, do the tags also work for the CR3 files? I only found them in the JPG files for this camera.

[Phil Harvey]

Thanks.  I'll fix that description (yeah, cut-n-paste error).

In my sample G5XII CR3 file the CameraInfo block is mostly all zeros.  I was hoping this was from a pre-production version and this would be fixed in production, but maybe not.

Code Select Expand

> exiftool ../pics/CanonPowerShotG5X_MarkII.* -fileindex -directoryindex
======== ../pics/CanonPowerShotG5X_MarkII.cr3
File Index                      : 1
Directory Index                 : 0
======== ../pics/CanonPowerShotG5X_MarkII.jpg
File Index                      : 80
Directory Index                 : 100
    2 image files read
- Phil

[Martin B.]

I just downloaded and tried exiftool 12.70 on my files, and this is what I get:

Code Select Expand

> exiftool.exe -fileindex -directoryindex IMG_0001.*
======== IMG_0001.CR3
File Index                      : 1
Directory Index                 : 0
======== IMG_0001.JPG
File Index                      : 1
Directory Index                 : 100
Both images were generated at the same time (camera set to RAW+JPG), so I'd expect them to have the same indexes. All my CR3 images have a fileindex of 1 and directory index of 0, while all my JPG images have different file indexes and (when appropriate) different directory indexes. The file indexes match the file names.

Like I wrote earlier, I really tried correlating all tags, for several days, and it looks like the Canon R6 and G5Xii CR3 files have no file and directory indexes.

Also, I tried the -shuttercount option on my G5Xii files, and the value is zero for all my CR3 files, but it seems OK for my JPG files. Did you notice the tags for these two file formats are in different places?

I can send you my sample images if you'd like production versions (I took them with the lens cap on).

Martin

[Phil Harvey]

Hi Martin,

They shouldn't be in separate places in the two types of files, but for some reason most of the CameraInfo is filled with zeros for the CR3 (take a look at the -v3 or -v4 output).

I didn't look at ShutterCount though, and I'm heading out for a week-long holiday now so I won't be able to look at this until I get back.

- Phil

[Martin B.]

Hi Phil,

I compared a pair of JPG and CR3 files from my G5XmII, specifically their CanonCameraInfoG5XII values from exiftool -v -htmlDump0.

I noticed this, and I think a solution is near:

• There's a block of zeros about 2048 bytes long at the beginning of CanonCameraInfoG5XII in the CR3 file.
• The difference between the offsets of the shuttercount tag in the JPG and CR3 files is also about 2048 (it's 2050). See my original post, above: 0x0a96 - 0x0294 = 2050
• If you skip the initial block of zeros in the CR3 file, the contents of CanonCameraInfoG5XII is very similar in the JPG and CR3 files.
• More specifically, at the same position (relative to the shuttercount value, 20 lines down in the htmlDump), I found the directory index (0x60) in the CR3 file. I wasn't able to find this tag in my initial search (listed as "not found" in my post).

Could it be that, for the CR3 files, there's an initial block of about 2048 bytes that isn't in the JPG files? Of perhaps the actual start of CameraInfo is about 2048 bytes later in the CR3 files?

For what it's worth, I searched the shuttercount value of this photo (4373 = 0x1115, which appears as "15 11" in the htmlDump) in both the JPG and CR3 files, to make sure there weren't multiple copies of the CameraInfo block. I searched the output of exiftool -v5 -htmlDump0, and found one instance of "15 11" in the JPG file, and 89 instances of it in the CR3 file. Of these 89, only one had a zero on either side, which I find is a necessary criteria because I can't believe a shutter counter would be limited to 16 bits. So I'm pretty sure there's only one copy of the shutter counter in the JPG and CR3 files.

I made my files available here: https://1drv.ms/f/s!AmOoLCII70USzlOj2ZQzn4kDXzfB
Feel free to copy them, add them to your test cases, whatever you find useful.

[Phil Harvey]

Sorry for the delay, I haven't forgotten about this but it will take a while before I have time to look at this in detail.  I'll post back here when I have done this.

- Phil

[Martin B.]

No problem! Just let me know how I can help.
I don't mean to bug you; I'm just motivated to improve the decoding of CR3 tags. Like you, I find it odd that a lot of the CameraInfo is filled with zeros.

Enjoy the holidays!

Martin

[Martin B.]

Hi Phil,

Again, no pressure; I can see you're busy fielding requests and improving ExifTool...

To summarize the issue: starting with release 12.70 ExifTool can report the shuttercount, fileindex and directory index of JPG files from the Canon G5X mark II. However, it reports incorrect values for CR3 files from the came camera. I think the data for the CR3 files is at a different location.

I'm willing to help if I can. I know enough Perl to use Komodo and step though ExtractInfo / ProcessMOV / ProcessCMT3, but that gives me visibility, not understanding  .

Would it help if I annotated output from exiftool -htmlDump to show similarities and differences between JPG and CR3 files from the Canon G5X mark II?

Thanks,

Martin

[Phil Harvey]

Hi Martin,

Sorry.  I did lose track of this post.  Thanks for reminding me.

I'm impressed that your browser can handle these large -v5 -htmldump output files -- mine can't.

I can find the ShutterCount at 2050 bytes later in the CR3, but I don't find the FileIndex/DirectoryIndex.

Also, I don't see anything in the structure of the data itself that allows me to identify when/why this 2050 offset is required.

So I will do a brute-force test of the file type and add 2050 bytes for the offset of the CR3 ShutterCount.  But this doesn't work for the FileIndex/DirectoryIndex with your sample files or my test files.

- Phil

[Martin B.]

Thanks Phil!

I don't see anything in the structure of the data itself that allows me to identify when/why this 2050 offset is required.

Hmmm, The offset is the same in all the file pairs I've checked, so it doesn't look like it's rounding up to some address.

...add 2050 bytes for the offset of the CR3 ShutterCount.  But this doesn't work for the FileIndex/DirectoryIndex with your sample files

I checked (manually on one file), and I agree with you.

Well, at least we'll have nailed the ShutterCount, and we know this inexplicable offset exists!



I'd like to continue exploring the similarities and differences in CanonCameraInfoG5XII between the JPG and CR3 files, because they are very similar once the CR3 address is increased by 2048. Is there a way to get a printout similar to -htmlDump[OFFSET], but without the HTML formatting? Something like:

Code Select Expand

0ab0   00 00 00 00 00 00 00 64  00 ca 02 64 00 ca 02 64
0ac0   80 ca 02 02 01 01 01 01  00 00 00 38 3f 00 00 27
0ad0   f7 05 00 7d 8a 00 00 00  00 00 00 00 00 00 00 00For a given group, I'd find the proper offset in each file with -v -htmlDump[OFFSET], and then get a text-only printout that would allow me to search and compare with Emacs, BeyondCompare, or even Perl.

Thanks again,

Martin

PS.

I'm impressed that your browser can handle these large -v5 -htmldump output files -- mine can't.

I'm using Firefox 122 on Windows 10. Firefox consumes 1.8 GB of memory when I load such a file. It takes a few seconds to show the initial page, then scrolling page per page is quick. Arbitrary scrolling (end of document, using the scroll bar, etc.) sends the one CPU core usage to 100%, and it takes a couple of seconds to refresh, so I suspect Firefox prefetches and computes the next few pages down the document. Anyways it works, and now we know EXifTool can also be used as a browser benchmark

[Phil Harvey]

Hi Martin,

I would think that exiftool -v5 will give you want you want in non-html form.

But personally I like to use the -U option for things like this.

- Phil

[Martin B.]

Thanks, I think the -U option is indeed what I need.
I'll report here what I find.

Martin