Request for Input on Another Google Photos "Takeout" Instance

[Phil Harvey]

DatePhotoTakenTimestamp?  Where is this from?

- Phil

[StarGeek]

That's from the Google Takeout JSON file.

One thing I forgot to mention is that the time stamps in the JSON file are in UTC.  So they may be off by the time zone difference. 

[elopsentorml]

I guess I've gone blind.  I've looked at multiple posts about this in the forums and it sure looks like this should work:

Code Select Expand

exiftool -r -tagsfromfile "%d/%f.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"

That and the %F variant just don't seem to do anything ("0 image files updated").  As far as I can tell, at least some of the .json files have the correct field populated, but nothing gets moved to the associated .jpg file:

Code Select Expand

D:\GooglePhotos\Wallpaper>exiftool -s cats_black_black_511583_1440x2560.jpg
ExifToolVersion                 : 11.77
FileName                        : cats_black_black_511583_1440x2560.jpg
Directory                       : .
FileSize                        : 868 kB
FileModifyDate                  : 2019:11:29 11:52:33-08:00
FileAccessDate                  : 2019:11:29 11:52:33-08:00
FileCreateDate                  : 2019:11:29 10:38:38-08:00
FilePermissions                 : rw-rw-rw-
FileType                        : JPEG
FileTypeExtension               : jpg
MIMEType                        : image/jpeg
JFIFVersion                     : 1.02
ExifByteOrder                   : Big-endian (Motorola, MM)
ImageDescription                :
XResolution                     : 300
YResolution                     : 300
ResolutionUnit                  : inches
YCbCrPositioning                : Centered
GPSVersionID                    : 2.3.0.0
GPSLatitudeRef                  : North
GPSLongitudeRef                 : East
GPSAltitude                     : 0 m
CurrentIPTCDigest               : 14636aa63c1f98be7f3c11fad46b0219
Caption-Abstract                :
ApplicationRecordVersion        : 4
XMPToolkit                      : Image::ExifTool 11.77
Description                     :
ImageWidth                      : 1440
ImageHeight                     : 2560
EncodingProcess                 : Baseline DCT, Huffman coding
BitsPerSample                   : 8
ColorComponents                 : 3
YCbCrSubSampling                : YCbCr4:4:4 (1 1)
ImageSize                       : 1440x2560
Megapixels                      : 3.7
GPSLatitude                     : 0 deg 0' 0.00" N
GPSLongitude                    : 0 deg 0' 0.00" E
GPSPosition                     : 0 deg 0' 0.00" N, 0 deg 0' 0.00" E

Code Select Expand

D:\GooglePhotos\Wallpaper>exiftool -s cats_black_black_511583_1440x2560.jpg.json
ExifToolVersion                 : 11.77
FileName                        : cats_black_black_511583_1440x2560.jpg.json
Directory                       : .
FileSize                        : 685 bytes
FileModifyDate                  : 2018:06:14 16:58:58-07:00
FileAccessDate                  : 2019:11:29 10:38:38-08:00
FileCreateDate                  : 2019:11:29 10:38:38-08:00
FilePermissions                 : rw-rw-rw-
FileType                        : JSON
FileTypeExtension               : json
MIMEType                        : application/json
CreationTimeFormatted           : Jun 3, 2018, 5:54:02 PM UTC
CreationTimeTimestamp           : 1528048442
Description                     :
GeoDataAltitude                 : 0.0
GeoDataLatitude                 : 0.0
GeoDataLatitudeSpan             : 0.0
GeoDataLongitude                : 0.0
GeoDataLongitudeSpan            : 0.0
GeoDataExifAltitude             : 0.0
GeoDataExifLatitude             : 0.0
GeoDataExifLatitudeSpan         : 0.0
GeoDataExifLongitude            : 0.0
GeoDataExifLongitudeSpan        : 0.0
ImageViews                      : 0
ModificationTimeFormatted       : Jun 14, 2018, 10:58:58 PM UTC
ModificationTimeTimestamp       : 1529017138
PhotoTakenTimeFormatted         : Jun 3, 2018, 5:51:29 PM UTC
PhotoTakenTimeTimestamp         : 1528048289
Title                           : Cats_Black_Black_511583_1440x2560.jpg

I've been staring at the command and I can't see any messed up spaces, quotes, dashes, or misspellings.  Am I missing something obvious?

[elopsentorml]

I ran the following stripped down command against that single file I showed, above:

Code Select Expand

exiftool -tagsfromfile "%d/%F.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s -overwrite_original D:\GooglePhotos\Wallpaper\Cats_Black_Black_511583_1440x2560.jpg

That worked.  As far as I can see, the only differences between that command and my earlier one are 1) the -r to recurse, 2) the -ext for the file extensions, and 3) the change from the whole directory to the specific file.

Stargeek:  I've been wondering about the UTC stuff, but I'm putting that off until I get the whole process roughed in.

[elopsentorml]

Is there a limit to the path length of files to be processed?  I can't get that DateTimeOriginal command to recurse on these remaining files.  But, looking at them, their fully qualified names are darn long.  I'm wondering if that's the problem.

[StarGeek]

Windows does have a path length limit, though I've only had problems with it when I try to specify the long path on the command line.  I haven't checked to see if exiftool can recurse properly into a long path.

To clarify the difference between %F and %f.  %F will look for a file name with the extension.  In your example of "cats_black_black_511583_1440x2560.jpg", then %F will reference "cats_black_black_511583_1440x2560.jpg.json" and %f will reference "cats_black_black_511583_1440x2560.json".

Another thing to take note of is that it's useless to include BMAP images.  BMAPs can't contain any metadata, so nothing will get copied to a BMAP.

Otherwise, I don't see anything wrong with the command.

[elopsentorml]

I had hoped to sort of brute force my way through this since I'm not a photographer (I just wanted to get all my metadata back into my Google Photos photos).  But, obviously, I'm going to have to spend more time and do this right.  So, I did some more experimentation and read some more of the various files around here.  By doing that, I found out about the .BMP files, too  .  I also noted that ExifTool will only process the files it recognizes, so I removed the -ext parameters from my commands since they were making the commands too long to see.

As to why I was having trouble with that command, I'm not sure.  I found a few things, but I'm not sure if they make sense.  First, by moving my remaining files out of their subdirectories and into a single folder (thus shortening the path), when I re-ran my various commands, new things happened.  Similarly, if I didn't recurse from the root, but just worked in one folder, more files processed.  No all files, just some.  Most notably, I could fold the "-DateTimeOriginal<PhotoTakenTimeTimestamp" bit into the rest of my merge command and it didn't fail.  Also, removing those -ext parameters one at a time gave me more files processed too.  Maybe it stopped doing later activities when it tried to do things that weren't allowed on a particular file type earlier in the string.  I also found that I had some .png files and ExifTool didn't like adding my tags to those (it made the file non-standard or something).  I also found that several of my .jpg files were actually .mov files in disguise, so ExifTool didn't like that.  All in all, I managed to get my unprocessed files down to a "mere" 440 or so jpg files.  At that point, I couldn't come up with any reason for things not working and there were just too many to look at.  I'm going to think about this some more and see if I can approach this from  a different direction.

[StarGeek]

I also found that I had some .png files and ExifTool didn't like adding my tags to those (it made the file non-standard or something).

What version of exiftool are you using? 

PNG files at one time didn't have a standard for EXIF group tags (which includes DateTimeOriginal), though that has changed in recent years.  Exiftool would create the group, but it was non-standard since there wasn't one.  I believe it gave a warning along the lines of "Creating non-standard EXIF in PNG".  A more recent version of exiftool should write the EXIF data to the proper location.

Any time you see a "Warning: [minor]" it shouldn't be that big of a problem. It's just that there's some sort of problem with the file, but not such that exiftool can't deal with it. Sometimes exiftool will fix the problem if it is updating the file.

Also, regarding PNG files, while exiftool has excellent metadata support for PNG files, very few other programs do.  Any data added to a PNG file is unlikely to be read by a lot of programs.

[elopsentorml]

I'm using the Windows Executable version 11.77 on Windows 10 Pro x64.  I want to emphasize that I'm not complaining about the program.  It's an amazing piece of work.  I'm sure all the problems I'm having are from me doing things wrong (well, that and the way Google gives us our photos and metadata back via Takeout).  Of course, it doesn't help that my photos go back to my scanning days more than 30 years ago and I've used all kinds of software to try to maintain them.

Regarding the .png files, I ran the commands on them again, and you're right:

Code Select Expand

D:\GooglePhotos>exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -overwrite_original -F -ext png "D:\GooglePhotos"
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2018-01-26/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/[redacted]/2016-04-10.png
Warning: No writable tags set from D:/GooglePhotos/[redacted].png
1410 directories scanned
    5 image files updated
    6 image files unchanged

D:\GooglePhotos>exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original -ext png "D:\GooglePhotos"
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/1994-06-30/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/1994-06-30/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2005-09-01/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2018-01-26/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
1410 directories scanned
   11 image files updated

I've got the -F option on the "fix bad files" command.  Should I also have it on the long "merge" command?

[StarGeek]

For the No writable tags set warnings, at best you can take a look at the JSON files and try to determine what data might be missing from there.  But it's probably that Google Takeout didn't have that data to begin with.

For the Creating non-standard IPTC in PNG warning, you only have to be aware that most other programs aren't going to be able to read the data, but then, as I mentioned, most programs don't read metadata in PNG files anyway.

I wouldn't put the -F option on a general bulk command.  Personally, I'd only use it on files where there is an actual problem with some other program, Lightroom/Photoshop for example, has a problem reading the file. Use it as a Scalpel instead of a Sledgehammer would be the metaphor I'd use. A lot of cameras write their MakerNotes badly and exiftool reports it as such, but the important data is still usually readable by programs that need to read it.

[elopsentorml]

Thanks.  I started with a fresh copy of my data and renamed some of the folders so that they weren't as long and didn't have odd characters in them.  Then I re-ran some of my commands.  When I ran my first "merge" command, it only worked for a very short time (tens of thousands of files, so it should go for many minutes) and just stopped.  There's no error message and no processing summary:

Code Select Expand

D:\GooglePhotos>exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"
Warning: Error opening file - D:/GooglePhotos/[redacted]/dining room(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/FOYER(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/GARAGE(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/KITCHEN(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/laundry room(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/master bedroom(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/upstairs bathroom(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Close-up of Termite Damage in Sill (and Awl Hol.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Electric Subpanel with Neutral and Ground Conne.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Electric Subpanel with Neutral and Ground Separ.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/rear left(1).jpg.json

D:\GooglePhotos>

I'm assuming it hit something it couldn't handle and silently failed.  Is there some way to see where it stopped processing (assuming I'm correct about that)?

EDIT:  I ran the %f version of the command and it finished normally (i.e., it gave me the summary at the end telling me what was processed).

[Phil Harvey]

If ExifTool terminated due to an unmanaged error, then there my be an _exiftool_tmp file laying around somewhere, which would be the file it was writing at the time.  If you find this file then it will tell you how far the processing got.  However, it would be very very uncommon for this to happen (ie. it has never happened to me).

- Phil

[elopsentorml]

I've split the "ExifTool stopped on one file" thing off to its own thread:

https://exiftool.org/forum/index.php?topic=10636.0

I searched for a _exiftool_tmp file, but found nothing.  Sorry.

[elopsentorml]

I think I've got this (at least roughly).  I'm down to a handful or so of photos which did not process.  It looks like the problem with those is that the file name is something like:

"filename(something).ext" (with those parenthesis)

but the associated .json file is something like:

"filename.ext(something).json"

IOW, Google Takeout misnamed the .json file by putting the "(something)" in the wrong place (well, perhaps I'm too harsh -- they just didn't follow the same naming convention as files without those parenthesis).  So, my ExifTool command wasn't finding the associated .json files.  Since there's only a handful of these, I can manually take care of them.

Manual Pre-processing
But, back to the beginning.  First, I copied my Google Photos Takeout tree over to D:\GooglePhotos (so that's why all my folder specifications are for that, and, since this is a copy of my data, that's why I used the -overwrite flag in my commands).  After that, even though I've got over 1400 folders in that data, I renamed many of those folders to shorten the paths and clean up their character sets.  I basically looked for any folders with non-standard characters (such as punctuation and # (I have no idea why Google created folder names with a pound symbol)) in them and renamed them to something simpler.  Then I did my ExifTool commands:

Fix bad tags:  Over the decades, a lot of non-standard stuff has ended up in my metadata.  So, I ran the following on all my files to clear out that non-standard stuff.  So far, I haven't noticed any downside to doing this (but, it's early times still):

Code Select Expand

exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -overwrite_original -F "D:\GooglePhotos"

Merge essential .json file tags with corresponding photo files:  This is for those .json files that include the photo extension in their name (i.e., like "filename.jpg.json").  Note that initially I was using "-DateTimeOriginal<PhotoTakenTimeTimestamp" for that.  But many of my .json file had negative versions of that (denoting seconds before 1970) and the current version of ExifTool fails silently on those.  So, I had to switch to "-DateTimeOriginal<PhotoTakenTimeFormatted" with a format string matching what was in my .json files:

Code Select Expand

exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeFormatted" -d "%b %d, %Y, %I:%M:%S %p UTC" "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"

Merge essential .json file tags with corresponding photo files, version 2:  This time, it's for those .json files without the photo extension built in (i.e., filename.json):

Code Select Expand

exiftool -r -wm cg -tagsfromfile "%d/%f.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeFormatted" -d "%b %d, %Y, %I:%M:%S %p UTC" "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"

Move photo files to Year-based folders:  I decided to put the processed photo files in another directory ("D:\Result") and in subfolders for the year they were taken.  Since Google has made oodles of duplicate photos, I used the command which appends a numerical suffix to each copy:

Code Select Expand

exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<DateTimeOriginal" overwrite_original "D:\GooglePhotos"

Move photo files to Year-based folders, version 2:  After the prior move, I still had many files left over where there was simply no DateTimeOriginal dates available.  So, this time the move is based on CreateDate:

Code Select Expand

exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<CreateDate" overwrite_original "D:\GooglePhotos"

Manual Post-Processing:  Searching "D:\GooglePhotos" in Windows 10 File Explorer on *.* and adding a kind:pictures tag showed me what I had left.  Many were .bmp files which ExifTool doesn't do things with.  So, I manually copied them to their own folder in "D:\Result".  Then there were some .jpg files whose names started with MVC.  I think those were some kind of movie files.  So, I moved them to their own folder in Result.  What was left are those misnamed filepairs that I'm going to manually apply specific versions of my commands to.

Remaining:
Besides the handful of misnamed filepairs I've got left, the one remaining issue is the lack of people/place tagging in my photos.  I don't see any easy way around this.  Google Takeout "handled" it by simply duplicating each photo with multiple such tags into folders named with those tags.  All I've done here is take all those duplicate copies and dump them together into their Year folder.  I'm not sure if I'll ever find a solution to the tagging loss.  And, of course, I've got to delete all those duplicate photos.