Sorry for another of these Google Photos "Takeout" posts, but I've been going over everything I can find here and was hoping someone could confirm that the ExifTool commands I'm going to use look reasonable. As a note, right now, all my photo "Takeout" jpg/json files are under "D:\GooglePhotos" on my Windows 10 machine.
First, while doing some experimentation with these, I saw many (426) problems with tags that shouldn't be in my jpg files (all I have are jpgs). I ran a command I found here to find those bad files (sorry, I didn't keep track of the originator):
exiftool -r -Exif:StripOffsets -Exif:Compression -Exif:RowsPerStrip -Exif:StripByteCounts -Exif:Software -if '$EXIF:StripOffsets or $Exif:RowsPerStrip or $Exif:StripByteCounts' -ext jpg -ext jpeg -ext tif -ext bmp -ext jpg_original -ext jpeg_original -ext tif_original -ext bmp_original "D:\GooglePhotos"EDIT: Corrected wrong apostrophes and added other file types, above.
As I mentioned, that gives me a lot of problem files spread all over the place. If I run the command found here:
https://exiftool.org/faq.html#Q20
to fix "unsafe" data against the whole "D:\GooglePhotos" directory:
exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -ext jpg -ext jpeg -ext tif -ext bmp -ext jpg_original -ext jpeg_original -ext tif_original -ext bmp_original "D:\GooglePhotos"EDIT: Added -r (recurse flag) and -ext stuff (to restrict to just photo files) to above command. BTW, trying to run this command in Powershell results in an error at the "@". Maybe that's fixable, but I just made sure to run it in a standard cmd shell.
will that strip out stuff that Google added (like people/place tags) or just tags that ExifTool knows don't belong in jpgs? Also, does that command recurse through all the sub-directories? Is there a better way to do this? Perhaps some kind of piping?
Second, once the bad tags are gone, I need to merge the json data into the corresponding jpgs. Looking at what's in one of those jsons, I get:
D:\GooglePhotos\Purchase Stuff> exiftool -s img_20191023_114419.jpg.json
ExifToolVersion : 11.72
FileName : img_20191023_114419.jpg.json
Directory : .
FileSize : 871 bytes
FileModifyDate : 2019:10:31 15:44:12-07:00
FileAccessDate : 2019:11:28 09:21:30-08:00
FileCreateDate : 2019:11:28 09:21:30-08:00
FilePermissions : rw-rw-rw-
FileType : JSON
FileTypeExtension : json
MIMEType : application/json
CreationTimeFormatted : Oct 25, 2019, 1:59:20 AM UTC
CreationTimeTimestamp : 1571968760
Description : [redacted]
GeoDataAltitude : [redacted]
GeoDataLatitude : [redacted]
GeoDataLatitudeSpan : 0.0
GeoDataLongitude : [redacted]
GeoDataLongitudeSpan : 0.0
GeoDataExifAltitude : [redacted]
GeoDataExifLatitude : [redacted]
GeoDataExifLatitudeSpan : 0.0
GeoDataExifLongitude : [redacted]
GeoDataExifLongitudeSpan : 0.0
ImageViews : 0
ModificationTimeFormatted : Oct 31, 2019, 9:44:13 PM UTC
ModificationTimeTimestamp : 1572558253
PhotoTakenTimeFormatted : Oct 23, 2019, 6:44:19 PM UTC
PhotoTakenTimeTimestamp : 1571856259
Title : IMG_20191023_114419.jpgAnd, looking here:
https://stackoverflow.com/questions/42024255/bulk-join-json-with-jpg-from-google-takeout
it looks like I'll need to merge in the Description and GPS stuff from the json files. From other comments on this forum, I think I need to merge in the photo taken time. Assuming I can do all of that at once, this is what I've come up with (once with a %F for json files with jpg exensions and once with a %f for those without):
exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -ext jpg -ext jpeg -ext tif -ext bmp -ext jpg_original -ext jpeg_original -ext tif_original -ext bmp_original-overwrite_original "D:\GooglePhotos"EDIT: Removed "-DateTimeOriginal<PhotoTakenTimeTimestamp", added -ext stuff for other file types (not sure how those ???_original ones will work) and added -wm cg (create new tags but don't edit existing ones) from above command.
The above seems to have done something (and that's where I noticed all the bad file tags). But, when I ran the %f version for json files without the jpg extension, I got a boatload of "Error Opening File" messages against various json files. Right now, I'm not too worried about those because it's possible my earlier activities messed something up.
Anyway, the final thing I need to do is re-organize these files and eliminate duplicates (since, apparently, when I added things like people/places tags in Google Photos it decided to give me duplicate photos in each tag-based folder). I'm having trouble deciding on an organizational structure, but I guess I'll just go with putting all the photos in their Year taken folder. From things I've found around here, that looks like:
exiftool -r -d %Y/%%f%%-c.%%e "-FileName<DateTimeOriginal" -ext jpg -ext jpeg -ext tif -ext bmp -ext jpg_original -ext jpeg_original -ext tif_original -ext bmp_original "D:\GooglePhotos"But, I'm not sure what that structure will look like. A Year folder with all the files named after when they were taken with any duplicates including a suffix before the extension? If so, hopefully each jpg will contain all people/place tags and I'll be able to manually delete all but one of any duplicates. Unless there's a better way to do that?
Again, I'm sorry for bringing up another "Takeout" set of problems. But, I'd really appreciate any input. Thanks.
Quote from: elopsentorml on November 28, 2019, 06:53:38 PM
As I mentioned, that gives me 426 problem files spread all over the place. If I run the command found here:
https://exiftool.org/faq.html#Q20
to fix "unsafe" data against the whole "D:\GooglePhotos" directory:
exiftool -all= -tagsfromfile @ -all:all -unsafe -icc_profile "D:\GooglePhotos"
will that strip out stuff that Google added (like people/place tags) or just tags that ExifTool knows don't belong in jpgs?
It's been a while since I've tested Google takeout, but at that time Google did not alter the metadata of the files. Anything extra will only appear in the JSON file. That command will keep all tags that exiftool has definitions for. Unless you created you're own tags and added them to the files to begin with, it's most likely that nothing will be lost except possibly data that was corrupted in some way.
QuoteAlso, does that command recurse through all the sub-directories?
Add the
-r (recurse) option (https://exiftool.org/exiftool_pod.html#r-.--recurse) to recurse into subdirectories. You might also want to include the
-ext (extension) option (https://exiftool.org/exiftool_pod.html#ext-EXT---ext-EXT--extension) (i.e.
-ext jpg) to prevent exiftool from trying to process the json files.
QuoteAnd, looking here:
https://stackoverflow.com/questions/42024255/bulk-join-json-with-jpg-from-google-takeout
Hmmm... coming up on 2 years since I posted that. I probably should double check it.
Quoteit looks like I'll need to merge in the Description and GPS stuff from the json files. From other comments on this forum, I think I need to merge in the photo taken time.
You only will need to merge if the data has been changed in Google website. Things like time stamps when the photo was taken probably don't need to be changed. Also, something I failed to mention in that answer, you can't directly copy from "PhotoTakenTimeTimestamp" into a image tags such as
DateTimeOriginal. See FAQ #5 (https://exiftool.org/faq.html#Q5) for details on how exiftool needs the time stamp to be formatted. To translate to something exiftool can copy, the format needs to be set with the
-d (dateFormat) option (https://exiftool.org/exiftool_pod.html#d-FMT--dateFormat) (see Common Date Format Codes (https://exiftool.org/filename.html#codes) for the list of formatting codes).
Quote(once with a %F for json files with jpg exensions and once with a %f for those without):
...
But, when I ran the %f version for json files without the jpg extension, I got a boatload of "Error Opening File" messages against various json files.
That probably isn't something to worry about. That just means that most of your files don't have json files that don't include the ".jpg".
Quoteexiftool -r -tagsfromfile "%d/%F.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -ext jpg -overwrite_original "D:\GooglePhotos"
Except for the previously mentioned time stamp problem, that probably should be ok.
QuoteFrom things I've found around here, that looks like:
exiftool -r -d %Y/%%f%%-c.%%e "-FileName<DateTimeOriginal" -ext jpg "D:\GooglePhotos"
But, I'm not sure what that structure will look like. A Year folder with all the files named after when they were taken with any duplicates including a suffix before the extension? If so, hopefully each jpg will contain all people/place tags and I'll be able to manually delete all but one of any duplicates.
That command will separate the files by year (the
%Y, see the above link for date formatting codes). It keeps the original file name and extension (
%f and
%e, with the % signs doubled because it's in a
-d formatting string), but will add a counter number if it encounters a duplicate file name. See
-w (textout) option (https://exiftool.org/exiftool_pod.html#w-EXT-or-FMT--textOut) for more details on the
%f,
%c, and
%e codes.
Quotehopefully each jpg will contain all people/place tags and I'll be able to manually delete all but one of any duplicates
That looks like it might be a problem. None of that data appears in the JSON output you listed and, as I mentioned, Google didn't change any data in the original file at the time I tested.
QuoteAgain, I'm sorry for bringing up another "Takeout" set of problems. But, I'd really appreciate any input. Thanks.
Never a problem. It gives me a reminder to double check things to see if anything has changed. Plus
Thanks for the reply. I'll make those changes you mentioned and see what happens. Thankfully, if I break anything, I can just start over with the original data Google sent me.
Digging around in these Takeout files and sampling some more of them, it seems like the json files with GPS data are associated with jpgs that have it. I don't know what I was looking at where that didn't seem to be the case. So, it might be that the only thing I need to merge from the json to the jpg is the Description. Also, it looks like the jpgs that have people/place tags are old enough that I must have added them there with earlier software (and there's nothing in the jsons). So, I don't know if I'll be able to recover those. Well, I suppose I could somehow parse the folder names (which are people/place tags) and write them into the jpgs someplace. This is getting to be a bigger mess every time I look at it.
Nice of Google to give us our data back all in pieces with no way (or only a very convoluted way) to get it all back together. I don't know how Joe-Bag-of-Donuts is supposed to do this.
One additional suggestion would be to add -wm cg to your command (see -wm (writemode) option (https://exiftool.org/exiftool_pod.html#wm-MODE--writeMode)). This will tell exiftool to write only new information and not to overwrite any data already in the file.
Aargh. The forum timed me out while I was responding, so I lost everything. I'll try to recreate it:
1. I've edited my original post with your suggestions. Hopefully, this thread will help future Google Photos refugees.
2. After running through those commands, it looks like they worked (at a cursory level, so far).
3. Since most of the files have been moved out of the original Takeout locations, I now see that I've got some .jpeg files left behind (I thought they were all .jpg files). I'll modify the commands and go back to handle those. EDIT: and .tif and .bmp.
4. There are also some ".jpg_original" files left behind. I'm guessing those were the originals of files I edited on Google Photos and the edited versions are what got moved. I'll have to decide what to do with those. EDIT: I added -ext stuff for the ???_original types in all my commands, above. I'm not sure if those will work with the merge one. I'm pretty sure it will in the %f version.
5. The files got moved into their Year folders and the duplicates are there with their suffixes. I still don't know how I'm going to get the people/places tags into those (since they're only reflected in the names of the Takeout folders they came from). It'd be a shame to lose all the work I put into tagging those photos in Google Photos. I'll have to think about that some more.
6. I also still have to go through an delete all the duplicates in those folders. But, this whole process is looking good.
7. EDIT: Several of the old Takeout folders have photo files left in them. I'm going to have to examine them to figure out what happened.
Again, thanks for the help.
Quote from: elopsentorml on November 28, 2019, 11:36:36 PM
4. There are also some ".jpg_original" files left behind. I'm guessing those were the originals of files I edited on Google Photos and the edited versions are what got moved. I'll have to decide what to do with those.
Those are the backup files exiftool creates when it processes files. From the docs (https://exiftool.org/exiftool_pod.html#DESCRIPTION):
By default the original files are preserved with _original appended to their names -- be sure to verify that the new files are OK before erasing the originals.
Quote5. The files got moved into their Year folders and the duplicates are there with their suffixes. I still don't know how I'm going to get the people/places tags into those (since they're only reflected in the names of the Takeout folders they came from). It'd be a shame to lose all the work I put into tagging those photos in Google Photos. I'll have to think about that some more.
It might be possible to copy part of the directory names into the files. Though it might be difficult once they've been moved. There is the
Directory tag which can be altered using the Advanced formatting feature (https://exiftool.org/exiftool_pod.html#Advanced-formatting-feature) in order to extract the needed parts of the path.
I've got -overwrite_original on the "merge" command, but forgot to add it to the others. I assume it's valid for all the commands. I'm fine with exiftool overwriting the originals in this case because I've got all the original data from Takeout stored away someplace else. I'll change the commands and try the whole process over. Thanks.
The Directory tag sound promising. But, I think the big issue would be having to search for duplicates of each file and add the directory information as a tag to one "original" version for each of the duplicates.
EDIT: Do we lose the ability to Modify posts after they're a day old? I was keeping my OP up-to-date, but now I no longer have a "Modify" button on that one.
The maximum time for allowing edits of posts in this forum is configurable, and is currently set to 12 hours.
- Phil
OK. I'll add my current set of commands here, then. Note that 1) everything is under "D:\GooglePhotos", 2) so far it looks like I have .jpg, .jpeg, .tif and .bmp files, and 3) I have the original data stored elsewhere so I don't care if files are overwritten:
exiftool -r -Exif:StripOffsets -Exif:Compression -Exif:RowsPerStrip -Exif:StripByteCounts -Exif:Software -if '$EXIF:StripOffsets or $Exif:RowsPerStrip or $Exif:StripByteCounts' -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"- Merge .json file (with photo extension substring) information with photo files:
exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"- Merge .json file (without photo extension substring) information with photo files:
exiftool -r -wm cg -tagsfromfile "%d/%f.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"- Move photo files to Year-based folders:
exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<DateTimeOriginal" -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"
Well, I expected some photos to not get moved over to their "final" positions. But, after the:
exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<DateTimeOriginal" -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"
to move everything under their Year folder (out of the way), I ended up with "1304 image files unchanged." Here's one of them:
ExifTool Version Number : 11.77
File Name : Cats_Black_Black_511583_1440x2560.jpg
Directory : D:/GooglePhotos/Wallpaper
File Size : 868 kB
File Modification Date/Time : 2019:11:29 08:04:28-08:00
File Access Date/Time : 2019:11:29 08:04:28-08:00
File Creation Date/Time : 2019:11:29 07:39:35-08:00
File Permissions : rw-rw-rw-
File Type : JPEG
File Type Extension : jpg
MIME Type : image/jpeg
JFIF Version : 1.02
Exif Byte Order : Big-endian (Motorola, MM)
Image Description :
X Resolution : 300
Y Resolution : 300
Resolution Unit : inches
Y Cb Cr Positioning : Centered
GPS Version ID : 2.3.0.0
GPS Latitude Ref : North
GPS Longitude Ref : East
GPS Altitude : 0 m
Current IPTC Digest : 14636aa63c1f98be7f3c11fad46b0219
Caption-Abstract :
Application Record Version : 4
XMP Toolkit : Image::ExifTool 11.77
Description :
Image Width : 1440
Image Height : 2560
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:4:4 (1 1)
Image Size : 1440x2560
Megapixels : 3.7
GPS Latitude : 0 deg 0' 0.00" N
GPS Longitude : 0 deg 0' 0.00" E
GPS Position : 0 deg 0' 0.00" N, 0 deg 0' 0.00" E
There's a File Creation Date/Time that looks correct, but no Date/Time Original. I assume that's the problem. I also assume I could add something like:
"-FileName<FileCreationDateTime"
in to the reorganizing command (probably before the "-FileName<DateTimeOriginal" option) and that would handle it. I'll give it a try and see.
Quote from: elopsentorml on November 29, 2019, 12:24:12 PM
There's a File Creation Date/Time that looks correct, but no Date/Time Original. I assume that's the problem. I also assume I could add something like:
"-FileName<CreationDateTime"
Add the
-s option when extracting to see the actual tag names. "CreationDateTime" is not a valid tag name.
- Phil
FileCreateDate seems to have caught most of them (I have to confirm). Still have 107 left, but I'm getting there. I'll check out the -s option, too.
EDIT: Oops. I've got to slow down with these edits. Yes, the FileCreateDate seems to have worked (I've got to look more in depth, though). But, it looks like one of my earlier commands set the create date to today. I'll have to research some more.
If you actually modify the file with exiftool, then the filesystem date/times will get set to the time you modified the file unless you use the -P option when writing.
- Phil
I'm not sure which of my activities changed those times, but they're definitely not the right field for me to use. It looks like I'll have to go back to one of my earlier ideas and merge the Photo Taken Times in with something like "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s. So:
exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s" "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"
Does that look reasonable?
EDIT: Nope. The command I listed, above, fails with: "The filename, directory name, or volume label syntax is incorrect." But, if I remove the "-DateTimeOriginal..." stuff, that command runs and works. If I run a separate command before that just to handle the DateTimeOriginal:
exiftool -r -tagsfromfile "%d/%f.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"
it runs fine, but apparently does nothing. Still trying to figure that out.
DatePhotoTakenTimestamp? Where is this from?
- Phil
That's from the Google Takeout JSON file.
One thing I forgot to mention is that the time stamps in the JSON file are in UTC. So they may be off by the time zone difference.
I guess I've gone blind. I've looked at multiple posts about this in the forums and it sure looks like this should work:
exiftool -r -tagsfromfile "%d/%f.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s -ext jpg -ext jpeg -ext tif -ext bmp -overwrite_original "D:\GooglePhotos"
That and the %F variant just don't seem to do anything ("0 image files updated"). As far as I can tell, at least some of the .json files have the correct field populated, but nothing gets moved to the associated .jpg file:
D:\GooglePhotos\Wallpaper>exiftool -s cats_black_black_511583_1440x2560.jpg
ExifToolVersion : 11.77
FileName : cats_black_black_511583_1440x2560.jpg
Directory : .
FileSize : 868 kB
FileModifyDate : 2019:11:29 11:52:33-08:00
FileAccessDate : 2019:11:29 11:52:33-08:00
FileCreateDate : 2019:11:29 10:38:38-08:00
FilePermissions : rw-rw-rw-
FileType : JPEG
FileTypeExtension : jpg
MIMEType : image/jpeg
JFIFVersion : 1.02
ExifByteOrder : Big-endian (Motorola, MM)
ImageDescription :
XResolution : 300
YResolution : 300
ResolutionUnit : inches
YCbCrPositioning : Centered
GPSVersionID : 2.3.0.0
GPSLatitudeRef : North
GPSLongitudeRef : East
GPSAltitude : 0 m
CurrentIPTCDigest : 14636aa63c1f98be7f3c11fad46b0219
Caption-Abstract :
ApplicationRecordVersion : 4
XMPToolkit : Image::ExifTool 11.77
Description :
ImageWidth : 1440
ImageHeight : 2560
EncodingProcess : Baseline DCT, Huffman coding
BitsPerSample : 8
ColorComponents : 3
YCbCrSubSampling : YCbCr4:4:4 (1 1)
ImageSize : 1440x2560
Megapixels : 3.7
GPSLatitude : 0 deg 0' 0.00" N
GPSLongitude : 0 deg 0' 0.00" E
GPSPosition : 0 deg 0' 0.00" N, 0 deg 0' 0.00" E
D:\GooglePhotos\Wallpaper>exiftool -s cats_black_black_511583_1440x2560.jpg.json
ExifToolVersion : 11.77
FileName : cats_black_black_511583_1440x2560.jpg.json
Directory : .
FileSize : 685 bytes
FileModifyDate : 2018:06:14 16:58:58-07:00
FileAccessDate : 2019:11:29 10:38:38-08:00
FileCreateDate : 2019:11:29 10:38:38-08:00
FilePermissions : rw-rw-rw-
FileType : JSON
FileTypeExtension : json
MIMEType : application/json
CreationTimeFormatted : Jun 3, 2018, 5:54:02 PM UTC
CreationTimeTimestamp : 1528048442
Description :
GeoDataAltitude : 0.0
GeoDataLatitude : 0.0
GeoDataLatitudeSpan : 0.0
GeoDataLongitude : 0.0
GeoDataLongitudeSpan : 0.0
GeoDataExifAltitude : 0.0
GeoDataExifLatitude : 0.0
GeoDataExifLatitudeSpan : 0.0
GeoDataExifLongitude : 0.0
GeoDataExifLongitudeSpan : 0.0
ImageViews : 0
ModificationTimeFormatted : Jun 14, 2018, 10:58:58 PM UTC
ModificationTimeTimestamp : 1529017138
PhotoTakenTimeFormatted : Jun 3, 2018, 5:51:29 PM UTC
PhotoTakenTimeTimestamp : 1528048289
Title : Cats_Black_Black_511583_1440x2560.jpg
I've been staring at the command and I can't see any messed up spaces, quotes, dashes, or misspellings. Am I missing something obvious?
I ran the following stripped down command against that single file I showed, above:
exiftool -tagsfromfile "%d/%F.json" "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s -overwrite_original D:\GooglePhotos\Wallpaper\Cats_Black_Black_511583_1440x2560.jpg
That worked. As far as I can see, the only differences between that command and my earlier one are 1) the -r to recurse, 2) the -ext for the file extensions, and 3) the change from the whole directory to the specific file.
Stargeek: I've been wondering about the UTC stuff, but I'm putting that off until I get the whole process roughed in.
Is there a limit to the path length of files to be processed? I can't get that DateTimeOriginal command to recurse on these remaining files. But, looking at them, their fully qualified names are darn long. I'm wondering if that's the problem.
Windows does have a path length limit, though I've only had problems with it when I try to specify the long path on the command line. I haven't checked to see if exiftool can recurse properly into a long path.
To clarify the difference between %F and %f. %F will look for a file name with the extension. In your example of "cats_black_black_511583_1440x2560.jpg", then %F will reference "cats_black_black_511583_1440x2560.jpg.json" and %f will reference "cats_black_black_511583_1440x2560.json".
Another thing to take note of is that it's useless to include BMAP images. BMAPs can't contain any metadata, so nothing will get copied to a BMAP.
Otherwise, I don't see anything wrong with the command.
I had hoped to sort of brute force my way through this since I'm not a photographer (I just wanted to get all my metadata back into my Google Photos photos). But, obviously, I'm going to have to spend more time and do this right. So, I did some more experimentation and read some more of the various files around here. By doing that, I found out about the .BMP files, too :) . I also noted that ExifTool will only process the files it recognizes, so I removed the -ext parameters from my commands since they were making the commands too long to see.
As to why I was having trouble with that command, I'm not sure. I found a few things, but I'm not sure if they make sense. First, by moving my remaining files out of their subdirectories and into a single folder (thus shortening the path), when I re-ran my various commands, new things happened. Similarly, if I didn't recurse from the root, but just worked in one folder, more files processed. No all files, just some. Most notably, I could fold the "-DateTimeOriginal<PhotoTakenTimeTimestamp" bit into the rest of my merge command and it didn't fail. Also, removing those -ext parameters one at a time gave me more files processed too. Maybe it stopped doing later activities when it tried to do things that weren't allowed on a particular file type earlier in the string. I also found that I had some .png files and ExifTool didn't like adding my tags to those (it made the file non-standard or something). I also found that several of my .jpg files were actually .mov files in disguise, so ExifTool didn't like that. All in all, I managed to get my unprocessed files down to a "mere" 440 or so jpg files. At that point, I couldn't come up with any reason for things not working and there were just too many to look at. I'm going to think about this some more and see if I can approach this from a different direction.
Quote from: elopsentorml on November 29, 2019, 10:42:43 PMI also found that I had some .png files and ExifTool didn't like adding my tags to those (it made the file non-standard or something).
What version of exiftool are you using?
PNG files at one time didn't have a standard for EXIF group tags (which includes
DateTimeOriginal), though that has changed in recent years. Exiftool would create the group, but it was non-standard since there wasn't one. I believe it gave a warning along the lines of "Creating non-standard EXIF in PNG". A more recent version of exiftool should write the EXIF data to the proper location.
Any time you see a "Warning: [minor]" it shouldn't be that big of a problem. It's just that there's some sort of problem with the file, but not such that exiftool can't deal with it. Sometimes exiftool will fix the problem if it is updating the file.
Also, regarding PNG files, while exiftool has excellent metadata support for PNG files, very few other programs do. Any data added to a PNG file is unlikely to be read by a lot of programs.
I'm using the Windows Executable version 11.77 on Windows 10 Pro x64. I want to emphasize that I'm not complaining about the program. It's an amazing piece of work. I'm sure all the problems I'm having are from me doing things wrong (well, that and the way Google gives us our photos and metadata back via Takeout). Of course, it doesn't help that my photos go back to my scanning days more than 30 years ago and I've used all kinds of software to try to maintain them.
Regarding the .png files, I ran the commands on them again, and you're right:
D:\GooglePhotos>exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -overwrite_original -F -ext png "D:\GooglePhotos"
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2004-12-16/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/2018-01-26/[redacted].png
Warning: No writable tags set from D:/GooglePhotos/[redacted]/2016-04-10.png
Warning: No writable tags set from D:/GooglePhotos/[redacted].png
1410 directories scanned
5 image files updated
6 image files unchanged
D:\GooglePhotos>exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original -ext png "D:\GooglePhotos"
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/1994-06-30/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/1994-06-30/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2004-12-16/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2005-09-01/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/2018-01-26/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
Warning: [minor] Creating non-standard IPTC in PNG - D:/GooglePhotos/[redacted].png
1410 directories scanned
11 image files updated
I've got the -F option on the "fix bad files" command. Should I also have it on the long "merge" command?
For the No writable tags set warnings, at best you can take a look at the JSON files and try to determine what data might be missing from there. But it's probably that Google Takeout didn't have that data to begin with.
For the Creating non-standard IPTC in PNG warning, you only have to be aware that most other programs aren't going to be able to read the data, but then, as I mentioned, most programs don't read metadata in PNG files anyway.
I wouldn't put the -F option on a general bulk command. Personally, I'd only use it on files where there is an actual problem with some other program, Lightroom/Photoshop for example, has a problem reading the file. Use it as a Scalpel instead of a Sledgehammer would be the metaphor I'd use. A lot of cameras write their MakerNotes badly and exiftool reports it as such, but the important data is still usually readable by programs that need to read it.
Thanks. I started with a fresh copy of my data and renamed some of the folders so that they weren't as long and didn't have odd characters in them. Then I re-ran some of my commands. When I ran my first "merge" command, it only worked for a very short time (tens of thousands of files, so it should go for many minutes) and just stopped. There's no error message and no processing summary:
D:\GooglePhotos>exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"
Warning: Error opening file - D:/GooglePhotos/[redacted]/dining room(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/FOYER(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/GARAGE(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/KITCHEN(1).JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/laundry room(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/master bedroom(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/upstairs bathroom(1).jpg.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Close-up of Termite Damage in Sill (and Awl Hol.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Electric Subpanel with Neutral and Ground Conne.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/Electric Subpanel with Neutral and Ground Separ.JPG.json
Warning: Error opening file - D:/GooglePhotos/[redacted]/rear left(1).jpg.json
D:\GooglePhotos>
I'm assuming it hit something it couldn't handle and silently failed. Is there some way to see where it stopped processing (assuming I'm correct about that)?
EDIT: I ran the %f version of the command and it finished normally (i.e., it gave me the summary at the end telling me what was processed).
If ExifTool terminated due to an unmanaged error, then there my be an _exiftool_tmp file laying around somewhere, which would be the file it was writing at the time. If you find this file then it will tell you how far the processing got. However, it would be very very uncommon for this to happen (ie. it has never happened to me).
- Phil
I've split the "ExifTool stopped on one file" thing off to its own thread:
https://exiftool.org/forum/index.php?topic=10636.0
I searched for a _exiftool_tmp file, but found nothing. Sorry.
I think I've got this (at least roughly). I'm down to a handful or so of photos which did not process. It looks like the problem with those is that the file name is something like:
"filename(something).ext" (with those parenthesis)
but the associated .json file is something like:
"filename.ext(something).json"
IOW, Google Takeout misnamed the .json file by putting the "(something)" in the wrong place (well, perhaps I'm too harsh -- they just didn't follow the same naming convention as files without those parenthesis). So, my ExifTool command wasn't finding the associated .json files. Since there's only a handful of these, I can manually take care of them.
Manual Pre-processingBut, back to the beginning. First, I copied my Google Photos Takeout tree over to D:\GooglePhotos (so that's why all my folder specifications are for that, and, since this is a copy of my data, that's why I used the -overwrite flag in my commands). After that, even though I've got over 1400 folders in that data, I renamed many of those folders to shorten the paths and clean up their character sets. I basically looked for any folders with non-standard characters (such as punctuation and # (I have no idea why Google created folder names with a pound symbol)) in them and renamed them to something simpler. Then I did my ExifTool commands:
Fix bad tags: Over the decades, a lot of non-standard stuff has ended up in my metadata. So, I ran the following on all my files to clear out that non-standard stuff. So far, I haven't noticed any downside to doing this (but, it's early times still):
exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -overwrite_original -F "D:\GooglePhotos"Merge essential .json file tags with corresponding photo files: This is for those .json files that include the photo extension in their name (i.e., like "filename.jpg.json"). Note that initially I was using "-DateTimeOriginal<PhotoTakenTimeTimestamp" for that. But many of my .json file had negative versions of that (denoting seconds before 1970) and the current version of ExifTool fails silently on those. So, I had to switch to "-DateTimeOriginal<PhotoTakenTimeFormatted" with a format string matching what was in my .json files:
exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeFormatted" -d "%b %d, %Y, %I:%M:%S %p UTC" "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"Merge essential .json file tags with corresponding photo files, version 2: This time, it's for those .json files without the photo extension built in (i.e., filename.json):
exiftool -r -wm cg -tagsfromfile "%d/%f.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeFormatted" -d "%b %d, %Y, %I:%M:%S %p UTC" "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"Move photo files to Year-based folders: I decided to put the processed photo files in another directory ("D:\Result") and in subfolders for the year they were taken. Since Google has made oodles of duplicate photos, I used the command which appends a numerical suffix to each copy:
exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<DateTimeOriginal" overwrite_original "D:\GooglePhotos"Move photo files to Year-based folders, version 2: After the prior move, I still had many files left over where there was simply no DateTimeOriginal dates available. So, this time the move is based on CreateDate:
exiftool -r -d "D:\Result\%Y/%%f%%-c.%%e" "-FileName<CreateDate" overwrite_original "D:\GooglePhotos"Manual Post-Processing: Searching "D:\GooglePhotos" in Windows 10 File Explorer on *.* and adding a kind:pictures tag showed me what I had left. Many were .bmp files which ExifTool doesn't do things with. So, I manually copied them to their own folder in "D:\Result". Then there were some .jpg files whose names started with MVC. I think those were some kind of movie files. So, I moved them to their own folder in Result. What was left are those misnamed filepairs that I'm going to manually apply specific versions of my commands to.
Remaining:Besides the handful of misnamed filepairs I've got left, the one remaining issue is the lack of people/place tagging in my photos. I don't see any easy way around this. Google Takeout "handled" it by simply duplicating each photo with multiple such tags into folders named with those tags. All I've done here is take all those duplicate copies and dump them together into their Year folder. I'm not sure if I'll ever find a solution to the tagging loss. And, of course, I've got to delete all those duplicate photos.