Bulk update GPS lat/long tags to fix camera's incorrect use of rational64s type

[JWilson]

Hello team ExifTool. I'm looking for assistance in creating a batch process in Windows that will allow me to "fix" thousands of JPG images spread across hundreds of sub-folders (all under one root, mind you) whereby the GPS latitude and longitude tags have been created with a rational64s (signed) data type, as opposed to the standard rational64u (un-signed) data type. The net effect is that photo management applications like Mylio are not able to properly recognize the embedded GPS information.

This issue was eloquently described in this blog post by Peter Rukavina:

https://ruk.ca/content/mysterious-case-why-photos-and-preview-couldnt-see-gps-data-embedded-exif-my-jpegs

Unfortunately the instructions he provided are for Apple, not Windows. Furthermore it's not clear whether the example shell script he created will recurse through all sub-folders of the target folder. In my case, I have far too many folders in my library to fix each one manually.

Any help would be appreciated!

Jason

[StarGeek]

https://ruk.ca/content/mysterious-case-why-photos-and-preview-couldnt-see-gps-data-embedded-exif-my-jpegs

Yeah, you really, really don't want to do what he lists on that page.  It's Common Mistake #3.  Never loop exiftool if you can avoid it.  And you usually can due to exiftool's power batch processing ability.  This is because exiftool's biggest performance hit is it's startup time and when you run it separately for thousands of files, it's going to take a long time.

Just to be clear, exiftool is correctly extracting the GPS coordinates?  Also, run this command on one of the files
exiftool -g1 -a -s -GPSL* /path/to/file.jpg
That will give all GPS Latitude and Longitude tag (Coordinates + Reference directions, plus maybe a couple others).  The reason for running this command is to see if the GPS coordinates are just in the GPS group or in both the GPS and XMP groups.

If exiftool is properly extracting the gps data, then try this command on some test files
exiftool -r -TagsFromFile @ -GPS* /path/to/files/
This command will rewrite the GPS Coordinates for every file it can.  The -r (-recurse) option tells it to recurse into subdirectories.  If the first command showed that there were coordinates in the XMP group as well, you can add -XMP:GPS* to the command somewhere after the @ sign.

This command creates backup files.  Add -Overwrite_Original to suppress the creation of backup files.

[JWilson]

First off, thank you StarGeek for your quick and detailed response. Greatly appreciated!

ExifTool is able to extract the GPS coordinates, presumably because it is somewhat forgiving of the datatype with which the lat/long are stored with. My issue is with Mylio not being able to read the GPS coordinates (and presumably Apple too), which is not a slight against them since they are following the spec and the BlackBerry Key2 camera module is not. Here is the output from one of the offending files:

C:\>exiftool -g1 -a -s -GPSL* blackberrykey2.jpg

Code Select Expand

---- GPS ----
GPSLatitudeRef                  : North
GPSLatitude                     : 43 deg 40' 7.30"
GPSLongitudeRef                 : West
GPSLongitude                    : 79 deg 18' 48.94"
---- Composite ----
GPSLatitude                     : 43 deg 40' 7.30" N
GPSLongitude                    : 79 deg 18' 48.94" W

Here is the full "verbose 3" listing of the GPS section extract from the same file:

Code Select Expand

  | 7)  GPSInfo (SubDirectory) -->
  |     - Tag 0x8825 (4 bytes, int32u[1]):
  |         0072: 00 00 03 0f                                     [....]
  | + [GPS directory with 9 entries]
  | | 0)  GPSLatitudeRef = N
  | |     - Tag 0x0001 (2 bytes, string[2]):
  | |         0325: 4e 00                                           [N.]
  | | 1)  GPSLatitude = 43 40 7.3 (43/1 40/1 730/100)
  | |     - Tag 0x0002 (24 bytes, rational64s[3]):
  | |         038d: 00 00 00 2b 00 00 00 01 00 00 00 28 00 00 00 01 [...+.......(....]
  | |         039d: 00 00 02 da 00 00 00 64                         [.......d]
  | | 2)  GPSLongitudeRef = W
  | |     - Tag 0x0003 (2 bytes, string[2]):
  | |         033d: 57 00                                           [W.]
  | | 3)  GPSLongitude = 79 18 48.94 (79/1 18/1 4894/100)
  | |     - Tag 0x0004 (24 bytes, rational64s[3]):
  | |         03a5: 00 00 00 4f 00 00 00 01 00 00 00 12 00 00 00 01 [...O............]
  | |         03b5: 00 00 13 1e 00 00 00 64                         [.......d]
  | | 4)  GPSAltitudeRef = 0
  | |     - Tag 0x0005 (1 bytes, int8u[1]):
  | |         0355: 00                                              [.]
  | | 5)  GPSAltitude = 45 (45/1)
  | |     - Tag 0x0006 (8 bytes, rational64u[1]):
  | |         03bd: 00 00 00 2d 00 00 00 01                         [...-....]
  | | 6)  GPSTimeStamp = 14 23 8 (14/1 23/1 8/1)
  | |     - Tag 0x0007 (24 bytes, rational64u[3]):
  | |         03c5: 00 00 00 0e 00 00 00 01 00 00 00 17 00 00 00 01 [................]
  | |         03d5: 00 00 00 08 00 00 00 01                         [........]
  | | 7)  GPSProcessingMethod = ASCII
  | |     - Tag 0x001b (14 bytes, string[14]):
  | |         03dd: 41 53 43 49 49 00 00 00 46 55 53 45 44 00       [ASCII...FUSED.]
  | | 8)  GPSDateStamp = 2020:08:22
  | |     - Tag 0x001d (11 bytes, string[11] read as undef[11]):
  | |         03eb: 32 30 32 30 3a 30 38 3a 32 32 00                [2020:08:22.]


Clearly the lat/long are using the rational64s signed datatype, not rational64u which I understand to be the proper spec.

Prior to running your suggested corrective command line, I copied the image file into a "exiftooltest" subfolder off of the root of C. I then created an "exiftoolsubfolder" under that one with another copy of the same image file (to test the recursion). Here is the command line I used:

C:\>exiftool -r -TagsFromFile @ -GPS* c:\exiftooltest\*.jpg
Warning: [minor] Entries in IFD0 were out of sequence. Fixed. - c:/exiftooltest/BlackBerryKey2_1.jpg
    1 image files updated

For the moment, let's ignore the warning (which was also reported in the earlier verbose output). As shown, ExifTool only found and fixed the JPG file in the top level folder that I targeted, so it didn't appear to recurse into the subfolder. I re-ran the verbose query on the modified image and got these results:

Code Select Expand

  | 10) GPSInfo (SubDirectory) -->
  |     - Tag 0x8825 (4 bytes, int32u[1]):
  |         0096: 00 00 03 14                                     [....]
  | + [GPS directory with 9 entries]
  | | 0)  GPSLatitudeRef = N
  | |     - Tag 0x0001 (2 bytes, string[2]):
  | |         032a: 4e 00                                           [N.]
  | | 1)  GPSLatitude = 43 40 7.3 (43/1 40/1 73/10)
  | |     - Tag 0x0002 (24 bytes, rational64u[3]):
  | |         0392: 00 00 00 2b 00 00 00 01 00 00 00 28 00 00 00 01 [...+.......(....]
  | |         03a2: 00 00 00 49 00 00 00 0a                         [...I....]
  | | 2)  GPSLongitudeRef = W
  | |     - Tag 0x0003 (2 bytes, string[2]):
  | |         0342: 57 00                                           [W.]
  | | 3)  GPSLongitude = 79 18 48.94 (79/1 18/1 2447/50)
  | |     - Tag 0x0004 (24 bytes, rational64u[3]):
  | |         03aa: 00 00 00 4f 00 00 00 01 00 00 00 12 00 00 00 01 [...O............]
  | |         03ba: 00 00 09 8f 00 00 00 32                         [.......2]
  | | 4)  GPSAltitudeRef = 0
  | |     - Tag 0x0005 (1 bytes, int8u[1]):
  | |         035a: 00                                              [.]
  | | 5)  GPSAltitude = 45 (45/1)
  | |     - Tag 0x0006 (8 bytes, rational64u[1]):
  | |         03c2: 00 00 00 2d 00 00 00 01                         [...-....]
  | | 6)  GPSTimeStamp = 14 23 8 (14/1 23/1 8/1)
  | |     - Tag 0x0007 (24 bytes, rational64u[3]):
  | |         03ca: 00 00 00 0e 00 00 00 01 00 00 00 17 00 00 00 01 [................]
  | |         03da: 00 00 00 08 00 00 00 01                         [........]
  | | 7)  GPSProcessingMethod = ASCIIASCII
  | |     - Tag 0x001b (13 bytes, undef[13]):
  | |         03e2: 41 53 43 49 49 00 00 00 41 53 43 49 49          [ASCII...ASCII]
  | | 8)  GPSDateStamp = 2020:08:22
  | |     - Tag 0x001d (11 bytes, string[11] read as undef[11]):
  | |         03f0: 32 30 32 30 3a 30 38 3a 32 32 00                [2020:08:22.]

Notice that the lat/long values appear to vary slightly from the original.

The good news is that when I then proceeded to import the modified file into Mylio, it was able to recognize the GPS (hooray!):

<see attached image "fixed_gps_in_mylio.png">

However when comparing the actual coordinates of the image reported in Mylio against those shown on the phone itself, there is a clear difference, presumably owing to the conversion/fix performed by ExifTool(?).

Mylio: 43.668335, -79.313667
Google Maps: 43.668700, -79.313600 (these are the coordinates that Google Photos used to launch Maps from the properties dialog for the photo)

Indeed, looking at the actual placement of the photo on the Mylio map (above) versus Google Maps on the device (below) there is a more pronounced displacement -- and the Google Maps placement is the correct one (the photo was taken in the park, not on the adjacent street).

<see attached image "device_gps_in_google_maps.png">

Any thoughts? Is there some additional rounding or precision qualifier need on the ExifTool command line? Any idea why the recursion didn't work?

Regards,

Jason Wilson

PS: Just in case the inline images referenced above don't embed properly, I'm attaching them to the entry, along with the original/modified photo.

[JWilson]

Argh. The inline images didn't embed properly. They seem to be base64 encoded when pasted into the editor, so given their size I hit the maximum character limit of the post. At least I was able to add them as attachments.

Jason

[StarGeek]

I didn't look over your whole post carefully yet, but try adding the -n (printConv) option to see if that helps with the precision.

For further details, see this post.

[JWilson]

Shaka, when the walls fell:(

Nope, that "-n" parameter didn't appear to make a difference, StarGeek. Here's the full command line I used:

C:\>exiftool -r -n -TagsFromFile @ -GPS* c:\exiftooltest\*.jpg

Code Select Expand

Warning: [minor] Entries in IFD0 were out of sequence. Fixed. - c:/exiftooltest/BlackBerryKey2.jpg
    1 image files updated

Unfortunately, the output coordinates reported by Mylio after deleting and re-importing the image were the same (43.668335, -79.313667). Is the position of the parameter on the command line significant (it doesn't appear to be based on the help file)?

Also, still no love with the subfolder recursion as the image file in the subfolder was again unaffected.

Jason

[StarGeek]

Got a second to look a bit closer, but still don't have time to read all of it, but here's one correction

C:\>exiftool -r -TagsFromFile @ -GPS* c:\exiftooltest\*.jpg
Warning: [minor] Entries in IFD0 were out of sequence. Fixed. - c:/exiftooltest/BlackBerryKey2_1.jpg
    1 image files updated

For the moment, let's ignore the warning (which was also reported in the earlier verbose output). As shown, ExifTool only found and fixed the JPG file in the top level folder that I targeted, so it didn't appear to recurse into the subfolder.

See Common Mistake #2.  Drop the *.jpg and use the -ext (extension) option, e.g. -ext jpg

[JWilson]

Thanks StarGeek, it appears that my mistakes have been mostly common ones -- and here I thought I was exceptional;)

After applying the extension parameter per your suggestion, ExifTool now successfully recurses through the subfolders:

C:\>exiftool -r -n -ext jpg -TagsFromFile @ -GPS* c:\exiftooltest\

Code Select Expand

Warning: [minor] Entries in IFD0 were out of sequence. Fixed. - c:/exiftooltest/BlackBerryKey2.jpg
Warning: [minor] Entries in IFD0 were out of sequence. Fixed. - c:/exiftooltest/exiftoolsubfolder/BlackBerryKey2_subfolder.jpg
    2 directories scanned
    2 image files updated

Now if only there was a mechanism to retain the precision of the original GPS lat/long? When I look back at my photos in the years to come, I'd hate to think that the events of the last few years of my life actually took place ten or twelve metres offset from where they are being shown:)

The next challenge after this one will be to figure out how to set a conditional on the above updates. I have a number of other devices that are generating legitimate GPS coordinates (with the correct data types), so there's no sense in me redundantly re-writing their EXIF data. Is there a parameter/expression that I can use that will restrict the updates to just the files with "Model = BBF100-2", or potentially a delimited list of models just in case other phones/cameras that I have used exhibit the same flawed EXIF data (if the latter is not possible, I guess I could always run ExifTool multiple times)?

Jason

[StarGeek]

Just to be clear, BlackBerryKey2_1.jpg_original (original removed in output below) is the original file and BlackBerryKey2_1.jpg (renamed to edited) is the one where exiftool fixed the data?

The coordinates are the same in both files

Code Select Expand

C:\>exiftool -g1 -a -s -n -GPSLatitude -GPSLongitude Y:\!temp\ccccc\BlackBerryKey2_1-edited.jpg Y:\!temp\ccccc\BlackBerryKey2_1.jpg -n
======== Y:/!temp/ccccc/BlackBerryKey2_1-edited.jpg
---- GPS ----
GPSLatitude                     : 43.6686944444444
GPSLongitude                    : 79.3135944444444
---- Composite ----
GPSLatitude                     : 43.6686944444444
GPSLongitude                    : -79.3135944444444
======== Y:/!temp/ccccc/BlackBerryKey2_1.jpg
---- GPS ----
GPSLatitude                     : 43.6686944444444
GPSLongitude                    : 79.3135944444444
---- Composite ----
GPSLatitude                     : 43.6686944444444
GPSLongitude                    : -79.3135944444444
    2 image files read

Looking at the -v3 output, the number are the same, except in the exiftool version, the fractions have been simplified.
Original
  | | 1)  GPSLatitude = 43 40 7.3 (43/1 40/1 730/100)
  | |     - Tag 0x0002 (24 bytes, rational64s[3]):
  | | 3)  GPSLongitude = 79 18 48.94 (79/1 18/1 4894/100)
  | |     - Tag 0x0004 (24 bytes, rational64s[3]):
vs
  | | 1)  GPSLatitude = 43 40 7.3 (43/1 40/1 73/10)
  | |     - Tag 0x0002 (24 bytes, rational64u[3]):
  | | 3)  GPSLongitude = 79 18 48.94 (79/1 18/1 2447/50)
  | |     - Tag 0x0004 (24 bytes, rational64u[3]):

730/100 simplifies to 73/10 and 4894/100 simplifies to 2447/50.

The problem appears to be Mylio.

[StarGeek]

Is there a parameter/expression that I can use that will restrict the updates to just the files with "Model = BBF100-2", or potentially a delimited list of models just in case other phones/cameras that I have used exhibit the same flawed EXIF data (if the latter is not possible, I guess I could always run ExifTool multiple times)?

You can use something like
exiftool -if "$Model eq 'BBF100-2' " <rest of command>
or if you have experience with regex
exiftool -if "$Model=~/BBF100-2/"  <rest of command>

In the first case, the model must exactly match the string.  The second case will match a substring.  One thing to be aware of is that it's fairly common for some data to be written with spaces (often trailing spaces) that may not be obvious.  Usually in older cameras, but I don't think it happens in the Model tag that much.

You can use -if multiple times.  See the -if option for details.

[JWilson]

StarGeek, you are certainly living up to your moniker!

You are absolutely correct about the before/after GPS coordinates. I tried a few more conversions of a random sampling of other images and the lat/long are indeed identical according to ExifTool when comparing the original and the modified file. After importing them into Mylio however it is reporting them to slightly different. Maybe some kind of rounding error? I will follow up with their support folks, who have been fantastic BTW. I'll be sure to report back with whatever resolution I come to with Mylio.

Thanks too for the guidance regarding the conditional expression to limit the impact of the conversion. It worked perfectly!

All the best!

Jason

[JWilson]

I'm back asking for one more act of charity...

Here's the final command line that I arrived at for my PC sandbox library:

exiftool -if "$Model eq 'BBF100-2' " -r -n -ext jpg -ext jpeg -TagsFromFile @ -GPS* -Overwrite_Original <target folder path>

However for the "real deal" I will need to invoke this against my media library folder that is running on a Mac. When I paste the above command line into the Terminal window of the Mac (less the "<target folder path>" placeholder, of course), I type a trailing space then drag my actual target folder via Finder into the Terminal window. At that point if I hit the Return key, I get the following error response:

Code Select Expand

zsh: no matches found: -GPS*

Lastly, presuming the above can be rectified, is there any special formatting required for Mac folders/paths that include space characters so that they will be interpreted properly by ExifTool? For example, the test folder that I'm using on my Mac is located on an external drive in a folder called "My Test ExifTool Repair Folder". This is how it appears when I drag it to the Terminal window:

/Volumes/MediaDrive/MediaLibary/My\ Test\ ExifTool\ Repair \Folder

Will that be recognized properly (given the literal space chars still in that path description) or will it need to be enclosed in quotes or such?

Jason   

PS: I looked for a Mac equivalent of Phil's "Getting started: Command-line ExifTool in Windows" post, but couldn't seem to track one down. Perhaps a sticky post with the key syntax or formatting differences between Windows/Mac might be useful?

[StarGeek]

However for the "real deal" I will need to invoke this against my media library folder that is running on a Mac.
...
At that point if I hit the Return key, I get the following error response:

Code Select Expand

zsh: no matches found: -GPS*

There are two major differences for running on bash/zsh/etc.  The first is that any asterisk on the command line will be replaced with the a directory file listing.  So in this case, zsh is trying to replace -gps* with a list of all files starting with -gps.  To avoid this, put quotes around it.  Or, new to me, escape it with a backslash according to this StackOverflow question.

Second, which you haven't come across yet, is that Windows CMD and Mac/Linux Terminal/bash/etc quote things differently.  On Mac/Linux, a dollar sign inside of double quotes interprets that as a bash variable.  So any single/double quotes need to be swapped.

So you want to use
'-GPS*'
and
-if '$Model eq "BBF100-2" '

Another problem you might run across is that Mac has a setting which will try to "help" you by changing any normal single/double quotes into fancy/curly quotes ' ' " "  These won't work on the command line.

is there any special formatting required for Mac folders/paths that include space characters so that they will be interpreted properly by ExifTool?
...
This is how it appears when I drag it to the Terminal window:

/Volumes/MediaDrive/MediaLibary/My\ Test\ ExifTool\ Repair \Folder

Will that be recognized properly (given the literal space chars still in that path description) or will it need to be enclosed in quotes or such?

If  you look closely, you'll see that each space has a backslash before it, which isn't part of the original directory path.  This escapes any space to show that it's part of the same argument.  The other option would be to remove the backslashes and put quotes around it.  These would be identical paths
/Volumes/MediaDrive/MediaLibary/My\ Test\ ExifTool\ Repair \Folder
'/Volumes/MediaDrive/MediaLibary/My Test ExifTool Repair Folder'

[JWilson]

Success!!!!!!

Thank you StarGeek for your quick and detailed answers to all my newbie questions.

And thanks too to Phil, for creating such a powerful beast of a utility.

Jason