Combine flickr JSON files with flickr photo files

Started by kkaiser, January 27, 2020, 08:17:14 PM

Previous topic - Next topic

kkaiser


I've decided to leave Flickr.
I asked for a complete download of my data.
I've watched the EXIFTools videos.
I don't understand how to do what I want....

I downloaded and unzipped all 229 photo files they sent me, each containing between 400 and 500 photos.
Putting them in a folder called 'flickr'. I opened all twelve of the zipped, EXIF (JSON) data sent by Flickr putting them inside a folder called JSON, inside the flickr folder used above.

I'd like to import all to my Apple Mac Photo application.

Frankly I'm afraid to run a command for fear of goofing it up.  So I thought I'd reach out and see who would be willing to help me.

The goal of course is to add the EXIF data to the appropriate photo, most (maybe all) of them are .jpg, .png and .mov. I'd like to add my name to the EXIF data and adjust the photo name to its original (if possible). I don't want any user comments from flickr added, if there even are any.

The photos files is  /Volumes/D2AF4/flickr  each folder in it is named "data-download-1" up to "data-download-229". Each photo is named like this "38948687244_3293974258_o.jpg".

The 12 json data folders are in /Volumes/D2AF4/flickr/JSON each folder is named "72157712783885213_01c246fd5da2_part1" up to "72157712783885213_01c246fd5da2_part12" and each file in them have names like "photo_42906270441.json".

How does EXIFTools know which of the JSON files the information it needs is in to put in any given photos? Can this whole process be run with one or two EXIFTool steps? Can it figure out how to move from photo to photo? How does it know what kind of tags are supposed to be in each photos, there were many different cameras used.

Please, don't leave me hanging if you know what my command should look like please give me an example. I'll pull a couple of photos into a test folder and give it a try.

Thank You....

StarGeek

The most important question is:
Did you add/change any data through the Flickr website?

If you didn't change or add anything on Flickr, then you have no need to copy the json data to the images.  Flickr does not remove any data from the files when you upload/download your images.  The json files won't have any data that's not already in the file.

If you did change data on Flickr, you need to figure out what it's called in the json file and where you want to put it in the original image.  Also, you might want to think about what the data is.  For example, if it's simple data, like the copyright, it would just be easier to write it as new data rather than try and copy it from the json files.

To figure out what the tags in the json file are, run this one one of the json files that you know has data you want to copy from
exiftool -g1 -a -s /path/to/file.json
Look for the data you want to copy and what the name of the tag is called.  After that, post back here and we can advise you further on what to do.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

kkaiser

I'm not sure where I got the idea that flickr pulled the EXIF data from the photos and stored it separately. But its great news that they didn't, because I've made no changes to any of the photos I put on flickr.
So the only issue I really have then is how to fix the file name for example dsc00928_49124570127_o.jpg is NOT what it was originally named. I do see that the dsc00928 part of this name is the original name.
Can I use EXIFTool to fix this or do I need to run it through a pearl script or something?

StarGeek

Quote from: kkaiser on January 28, 2020, 12:23:31 AM
I'm not sure where I got the idea that flickr pulled the EXIF data from the photos and stored it separately.

It reads the metadata (using ExifTool) but doesn't remove it.

QuoteSo the only issue I really have then is how to fix the file name for example dsc00928_49124570127_o.jpg is NOT what it was originally named. I do see that the dsc00928 part of this name is the original name.

So basically, you want to remove everything after the first underscore? Try this:
exiftool '-Testname<${Filename;s/_.*//}.%e' /path/to/files/

This command just lists what changes it would make.  Test it out and make sure it's what you want.  If it looks good, change Testname to Filename.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

kkaiser

StarGeek your suggested command worked great. The vast majority of the photos are now better named, the errors will be easy enough to correct.

That was so much help I'm hoping you'll indulge me with one more thing. I'd like to sort the 15k photos using the 'EXIF Create Date' into folders labeled by year. Can EXIFTool actually do this? It would almost certainly have to be able to create a folder, or I would in advance.

Phil Harvey

ExifTool will create output folders.  I think you want a command something like this:

exiftool -d DSTDIR/%Y "-directory<createdate" SRCDIR

Where SRCDIR and DSTDIR are the source and destination folder names.  The folders named by year will be created in DSTDIR.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

StarGeek

I'm too slow, but I'm posting anyway.

Yes, that's one of the most common uses of exiftool.  See the -d (dateFormat) optionRenaming Examples for some examples, Common Date Format Codes for the date codes, and search here on the forums for more examples.

But the basic command would be this, replacing the /path/to/* to the actual paths you need to use:
exiftool "-Directory</path/to/TargetDir/$Exif:CreateDate" -d %Y /path/to/SourceDir

As you can see by the differences between my command and Phil's, there are multiple ways to do it.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

kkaiser

Phil Harvey & StarGeek you guys are awesome thank you so much. I obviously have a lot to learn about EXIFTool but I can see so much use for it. I'll be using it for all kinds of things...
Just to be clear, both solutions work...

kkaiser

It took over 12 hours to process all my photos only to learn that 50% didn't have the CreateDate but many have 'Profile Date Time' or 'Date/Time Original'. Seems I need to add the 'CreateDate' to the EXIF based on the value of one of these other two.
A new post in this section shows this command which I've modified to what I know, but that's not much yet. So can I add the CreateDate, based on one of the other two, and still have the image moved to the appropriate folder base on the year? Or maybe I have to do it for one 'datetimeoriginal' first then 'profiledatetime' after. Maybe I don't really need the CreateDate at all and can just use one/both of these other dates in my command?  My current command looks like this;


exiftool -d /Volumes/TimeMachine/photos-by-year/%Y "-directory< datetimeoriginal" /Volumes/TimeMachine/flickr


Your opinions are greatly valued.

Phil Harvey

You shouldn't use ProfileDateTime, it is the time the ICC profile was made and has nothing to do with the photo.

Usually, the fallback is to use FileModifyDate if nothing else is available.  So typically, one would do this:

... "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" ...

Which will use DateTimeOriginal if available, otherwise CreateDate, and finally FileModifyDate.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

kkaiser

Thats good advice Phil, so I created the command;
exiftool -addtagsfromfile @ -datetimeoriginal-= "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" FILESorDIRS

The output:
% exiftool -addtagsfromfile @ -datetimeoriginal-= "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" /Volumes/TimeMachine/flickr/data-download-4/dcp.jpg
Error: '2020:01:22 10:24:32-06:00/dcp.jpg' already exists - /Volumes/TimeMachine/flickr/data-download-4/dcp.jpg
    0 image files updated
    1 files weren't updated due to errors


I'm just not sure I'm doing this correctly. There was no change made to the file that I can see.



Phil Harvey

ExifTool won't overwrite existing files.

Also, why did you add -datetimeoriginal-= ?

- Phil

Edit:  Note that it isn't a good idea be writing a directory name containing colons. You forgot the -d option.
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

kkaiser

I'm lost again, the added code was a mistake, now I have this;

exiftool -addtagsfromfile @  "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" FILESorDIRS

But I don't know where the -d goes.

Phil Harvey

exiftool -d /Volumes/TimeMachine/photos-by-year/%Y "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" /Volumes/TimeMachine/flickr
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

kkaiser

Running this;
% exiftool -d /Volumes/TimeMachine/photos-by-year/%Y "-directory<filemodifydate" "-directory<createdate" "-directory<datetimeoriginal" /Volumes/TimeMachine/flickr/
Produces only this;

    1 directories scanned
    0 image files read


but there are 226 folders in flickr so I'm still doing something wrong.