Batch Process for Google Photos Takeout

[ptsch]

there are literally hundreds of forum entries on this topic everywhere around the internet, but i am unable to find any that specifically address what i'm trying to achieve.

tl;dr, i did a google takeout, and now i have all the jpgs/mp4s/movs but the exif data in a json file for each image. and all of them are also filed in additional subdirectories. i don't wanna use immich-go. can someone share a command i can use to just merge all these different files with their metadata that's in the json file and one giant sweep?

the meta data for the images looks like this:

Code Select Expand

{
  "title": "20240802_111303.jpg",
  "description": "",
  "imageViews": "1",
  "creationTime": {
    "timestamp": "1722645794",
    "formatted": "Aug 3, 2024, 12:43:14 AM UTC"
  },
  "photoTakenTime": {
    "timestamp": "1722615183",
    "formatted": "Aug 2, 2024, 4:13:03 PM UTC"
  },
  "geoData": {
    "latitude": 0.0,
    "longitude": 0.0,
    "altitude": 0.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "url": "https://photos.google.com/photo/AF1QipOPmvlmpwl41FlJwY16ke33OQVv8_cm5M4s-FKA",
  "googlePhotosOrigin": {
    "mobileUpload": {
      "deviceFolder": {
        "localFolderName": ""
      },
      "deviceType": "ANDROID_PHONE"
    }
  }
}
i have also attached a sample image and json file in case it's needed.

[StarGeek]

The tl;dr, the data in your files show that you do not need to do anything.

I'm going to split this into two posts. The first detailing why you probably don't need to do much, if anything, and the second detailing the commands to use.

tl;dr, i did a google takeout, and now i have all the jpgs/mp4s/movs but the exif data in a json file for each image.

Use exiftool and check the files. You'll find that they have the same data as when they were uploaded. Google DOES NOT remove any metadata from the files.

Look at that JSON. The only useable data listed there is the "photoTakenTime". Notice how it is in UTC. The GPS data is 0,0, and I doubt you took the photo in the Atlatic Ocean off the coast of Africa so you don't want to use that.

Now look at the date/time tags in your file.

Code Select Expand

C:\>exiftool -time:all --system:all -G1 -a -s Y:\!temp\x\y\20240802_111303.jpg
[IFD0]          ModifyDate                      : 2024:08:02 11:13:03
[ExifIFD]       DateTimeOriginal                : 2024:08:02 11:13:03
[ExifIFD]       CreateDate                      : 2024:08:02 11:13:03
[ExifIFD]       OffsetTime                      : -05:00
[ExifIFD]       OffsetTimeOriginal              : -05:00
[ExifIFD]       SubSecTime                      : 058
[ExifIFD]       SubSecTimeOriginal              : 058
[ExifIFD]       SubSecTimeDigitized             : 058
[Samsung]       TimeStamp                       : 2024:08:02 09:13:03.058-07:00
[Composite]     SubSecCreateDate                : 2024:08:02 11:13:03.058
[Composite]     SubSecDateTimeOriginal          : 2024:08:02 11:13:03.058-05:00
[Composite]     SubSecModifyDate                : 2024:08:02 11:13:03.058-05:00
Notice that the date/time embedded in the file is "2024:08:02 11:13:03"  The filename is also this value without the extra formatting.

Your example file shows that if you try to "fix" your files, you'll be overwriting the correct date/time of "2024:08:02 11:13:03" with the UTC time of "2024:08:02 16:13:03", destroying the correct local time.

Now, it might be that some of your images don't have an embedded date like this. Images that you may have downloaded from social media (Facebook/Insta/WhatsApp) and screenshots rarely have such embedded data. Also, you may have corrected or added data on the Google photos website. As Google has not changed the data in your files, this would be data that would be in the JSON file and you might want to embed that.

One more thing you might want to do is set the file system time stamps to be the correct time, as these will currently reflect the time that you downloaded the file. Immich is supposed to use the DateTimeOriginal value first, but I've heard that sometimes it uses the FileModifyDate instead.
 

[StarGeek]

For the following commands, if you are on Mac/Linux, change the double quotes in to single quotes. If on Windows, use CMD, not PowerShell.

If you are on Linux, then remove the parts that copy to FileCreateDate, as Linux doesn't have that tag.

Some of these commands will create backup files. To suppress the creation of backup files, add the -overwrite_original option. To recurse into subdirectories, add the -r (-recurse) option.

I would suggest doing this. First, copy the embedded timestamp to the file system timestamps with this.
exiftool -efile2 MissingDate.txt -api QuickTimeUTC --ext JSON "-FileCreateDate<CreateDate" "-FileModifyDate<CreateDate" "-FileCreateDate<DateTimeOriginal" "-FileModifyDate<DateTimeOriginal" /path/to/files/

What this command will do is copy either the DateTimeOriginal or CreateDate from the file to the file system time stamp. Additionally, it will give you a list of the files that fail this process. The files that fail will be the ones that don't have a DateTimeOriginal or CreateDate tag. Every file that isn't on this list is already correct and doesn't need any further processing.

You now have a list of files that actually do need fixing in the MissingDate.txt. Take a look at that list and see how many of them have filenames that have all 14 numbers that would be the full date/time. If you have files like that, then use this command. It will only look at files that are listed in MissingDate.txt
exiftool -wm cg -Efile4 CopyFromJSON.txt -@ MissingDate.txt -if "length(${Basename;tr/0-9//cd}) == 14" "-AllDates<Filename" "-FileCreateDate<Filename" "-FileModifyDate<Filename" /path/to/files/

The files that had the date/time as the filename are now fixed. CopyFromJSON.txt is the list of files that have failed all the previous commands. These are the only files you need to copy  data from the JSON files. Note that these files will end up with time stamps that are UTC and not the original time of the images.

Edit: Removed the slash after the %d because it is included already as part of the %d
exiftool -efile2 FailedFiles.txt -d %s -@ CopyFromJSON.txt -tagsFromFile "%d%F.json" "-AllDates<PhotoTakenTimeTimestamp" "-FileCreateDate<PhotoTakenTimeTimestamp" "-FileModifyDate<PhotoTakenTimeTimestamp" /path/to/files/

There may still be files that fail here due to the way Google names the JSON files and they will be listed in FailedFiles.txt.  The JSON files created by Google have a maximum filename length of about 30 characters, so if the image files had a name longer then that, some more commands will be needed, and I'll have to dig through the archives for the fixes.

[ptsch]

wow. well this is a reply and a half 😏 wow.

Use exiftool and check the files. You'll find that they have the same data as when they were uploaded. Google DOES NOT remove any metadata from the files.

ok, so the reason i came here is, i did a batch upload, and immich showed a few thousand images under "today". and when i right-click -> properties on many of the images, i looked at the dates under general tab, which were not correct. but going to the details tab, like the exif tool probably would, there are correct dates on many, but not all, of the photos. so thanks for pointing that out

I doubt you took the photo in the Atlatic Ocean off the coast of Africa

bold of you to assume, sir 😌. but just by coincidence, you are right.

Images that you may have downloaded from social media (Facebook/Insta/WhatsApp)

this could likely be the case. and in that case, i am hoping to batch process something with your tool, so that like mentioned, i don't have a few thousand photos under today. maybe that will solve the problem entirely. so now i will begin to go through your post detailing specific operations.

[ptsch]

hmm.. it didn't work.

[ptsch]

sorry, couldn't help but troll your signature

so the first two commands neatly do what they're supposed to and generate a txt file each time with listen files. the third command, however, fails with the following result:

Code Select Expand

C:\Users\ptsch\Desktop\exiftool>exiftool -efile2 FailedFiles.txt -d %s -@ CopyFromJSON.txt -tagsFromFile "%d/%F.json" "-AllDates<PhotoTakenTimeTimestamp" "-FileCreateDate<PhotoTakenTimeTimestamp" FileModifyDate<PhotoTakenTimeTimestamp" C:\Users\ptsch\Desktop\exiftool
The filename, directory name, or volume label syntax is incorrect.
i have attached the sample image i was trying to make this work with. as you can see, it does not have a "created date" in the details of the image. likely because it was a media file sent via social media.

[StarGeek]

wow. well this is a reply and a half 😏 wow.

Yes, I get a bit irrationally angry when I see this urban legend perpetuated. A lot of that anger is directed at myself for not double-checking things when I first answered this question in 2017.

It also shows how complex metadata is and how there isn't a simple solution.

ok, so the reason i came here is, i did a batch upload, and immich showed a few thousand images under "today". and when i right-click -> properties on many of the images, i looked at the dates under general tab, which were not correct.

I really need to install and test out immich and see what data it reads. My quick searches seem to indicate that it should be reading the correct metadata that is embedded in the file, but I keep reading posts like this which indicate that it is reading the file system time stamps, not the embedded EXIF ones.

so now i will begin to go through your post detailing specific operations.

Part of the reason I made this post is to be the start of documenting a procedure for dealing correctly with the whole Google Takeout thing. You forced me to start re-thinking things and adding in some of the newer features of exiftool that weren't around in my original post. Hmm... I think that I could maybe work up a batch file that would cover all of this.

[StarGeek]

sorry, couldn't help but troll your signature

so the first two commands neatly do what they're supposed to and generate a txt file each time with listen files. the third command, however, fails with the following result:

Ah, my mistake. I forgot a quote and minus sign. Change this
"-FileCreateDate<PhotoTakenTimeTimestamp" FileModifyDate<PhotoTakenTimeTimestamp"
into this
"-FileCreateDate<PhotoTakenTimeTimestamp" "-FileModifyDate<PhotoTakenTimeTimestamp"

[ptsch]

now i'm getting this:

Code Select Expand

C:\Users\ptsch\Desktop\exiftool>exiftool -efile2 FailedFiles.txt -d %s -@ CopyFromJSON.txt -tagsFromFile "%d/%F.json" "-AllDates<PhotoTakenTimeTimestamp" "-FileCreateDate<PhotoTakenTimeTimestamp" "-FileModifyDate<PhotoTakenTimeTimestamp" C:\Users\ptsch\Desktop\exiftool
Warning: Error opening file - C:/Users/ptsch/Desktop/exiftool/IMG_0131.JPG.json
Warning: Error opening file - C:/Users/ptsch/Desktop/exiftool/IMG_0131.JPG.json
Warning: Error opening file - C:/Users/ptsch/Desktop/exiftool/IMG_0131.JPG.json
    1 directories scanned
    0 image files updated
    3 image files unchanged
i assume it's because the command doesn't take into account that the json file is actually suffixed with "supplemental-metadata" before the .json extension. and i'm not bright enough to figure out how to syntax it correctly when the source json file doesn't exactly match the destination jpg file :/

[StarGeek]

*sigh* Of course Google has changed things. "supplemental-metadata" isn't something I've seen before. Is there a space or something before the source filename?

In this part (changing this because the slash shouldn't be there)
-tagsFromFile "%d%F.json"
The %d stands for the directory (including trailing slash) and %F is the filename (including extension)

To prefix (edit: oops, suffix) "supplemental-metadata", you would put it directly before after the %F
-tagsFromFile "%d%F.supplemental-metadata.json"

This should look for a JSON file called
C:/Users/ptsch/Desktop/exiftool/supplemental-metadataIMG_0131.JPG.json

[ptsch]

*sigh* Of course Google has changed things.

and after that, they cancel them 

Is there a space or something before the source filename?

no space, but the whole original file name is in the json file name

so the image is actually called "IMG_0131.JPG.supplemental-metadata.json" so you need to append it like
-tagsFromFile "%d%F.supplemental-metadata.json" aka the supplemental-metadata AFTER the %F, not forgetting the .

now it changed the file, and imported exactly the "data taken" from the metadata into the file. good job  i will do some additional testing to see if i can just wholesale point this at the takeout directory and say "fire!"

[ptsch]

Note that these files will end up with time stamps that are UTC and not the original time of the images.

is there a simple way to not have that be the case? or is that as complicated as usb over ip on windows?