News:

2023-03-15 Major improvements to the new Geolocation feature

Main Menu

Write exif data from Flickr JSON files

Started by andreasmb, April 16, 2020, 09:40:13 PM

Previous topic - Next topic

andreasmb

Hi there!

My question is very similar to the user kkaiser in this thread - https://exiftool.org/forum/index.php?topic=10780.0, but unlike that thread, I want to write some of the metadata in the flickr JSON files to the image's Exif data.

Long story short – I'm trying to leave Flickr. And there's a lot of metadata that I've added to Flickr over the years that I'd very much like to keep – tags, album names, titles, and descriptions. This metadata is unfortunately in separate JSON files. I'd like to some of the fields in the JSON file and write it to the appropriate Exif tags on the image, so that I can integrate my Flickr photos into the rest of my collection in Lightroom.

After spending my day googling for various solutions, Exiftool seems to be the most expedient and powerful solution for this. I watched a couple of videos, read some documentation, but it seems like what I want to do is not beginners stuff. So I'm hoping for advice about which commands to look into and how many steps it should take. (Also – I don't want to mess up the 15k images that I have.) I've done some programming in the past, but apparently not enough to find my way to the right commands.

The first challenge is mapping the JSON file to the corresponding image (or MP4/MOV video). This seems like the most difficult step, since the file names are not the same. But they share a unique ID. The JSON file has this file naming convention:
photo_89015302.json (The number is the image ID).

The image file naming convention is:
mySpecialTitle_89015302_o.jpg ("mySpecialTitle" could be anything, including strings of random numbers)

Second challenge is the metadata mapping. I'd like to transfer the JSON data to the Exif data like this:
"Tags" -> Keywords
"Albums" -> Keywords (I don't think there's a clean way to make Flickr albums into Lightroom 'Sets', so I'll keep them as keywords and re-make the sets in Lightroom later)
"Name" -> Object Name
"Description" -> Caption-Abstract


Then, lastly, I'd like to rename the files, so that "date taken" is added a prefix with this format - YYYY-MM-DD. (This is something I can probably figure out myself, but I'd still appreciate any pointers.)

The time that this would save me is immeasurable, so I would be in your debt if I could make any progress on this.

StarGeek

Quote from: andreasmb on April 16, 2020, 09:40:13 PMBut they share a unique ID. The JSON file has this file naming convention:
photo_89015302.json (The number is the image ID).

The image file naming convention is:
mySpecialTitle_89015302_o.jpg ("mySpecialTitle" could be anything, including strings of random numbers)

Does the jpeg file always have two characters after the ID number?  And are the json files in the same directory as the image files?  If so, you would be able to map it in a command like this:
exiftool -TagsFromFile %dphoto_%-8.2f.json "-IMAGETAG<JSONTAG" /path/to/files

The %-8.2f part take the base name (mySpecialTitle_89015302_o), strip away the two trailing characters and take the last 8 characters from the end (89015302).  This is prefixed by the directory %d, the string photo_, and have the .json extension added.

The "-IMAGETAG<JSONTAG" is where you would remap the JSONTAG name into the tag in the image file.

QuoteSecond challenge is the metadata mapping. I'd like to transfer the JSON data to the Exif data like this:
"Tags" -> Keywords
"Albums" -> Keywords (I don't think there's a clean way to make Flickr albums into Lightroom 'Sets', so I'll keep them as keywords and re-make the sets in Lightroom later)
"Name" -> Object Name
"Description" -> Caption-Abstract

Run this on one of the json files
exiftool -s file.json

Exiftool will output the names of the tags in the json file.  You would decide which tag you want to copy and where you want to copy it.  You seem to have the basic mapping planned out, you just need to verify the names of the json tags.

I don't have any Flickr json files on hand at the moment, but I'll try find time to get some tomorrow so I can advise further.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

andreasmb

Wow, StarGeek, thank you for the reply! I'll start experimenting with the commands you suggested. In the meantime, here's some answers to your questions –

Quote from: StarGeek on April 16, 2020, 10:19:49 PM
Does the jpeg file always have two characters after the ID number? 
Yes, there's always a "_o" at the end of every media file name.

Quote from: StarGeek on April 16, 2020, 10:19:49 PM
And are the json files in the same directory as the image files? 

No, but I could move them all into the same folder. It would be a bit of manual work, but it's not unsurmountable.

The images are batched into separate folders, like this:

  • data-download-1
  • data-download-2
  • data-download-3
  • etc... (up to 40)

...and the json files for all the media files are also batched into a smaller set of separate folders, but there's only two:


  • data-download 72157713880626132_375a3c67e881_part1
  • data-download 72157713880626132_375a3c67e881_part2

Here's an example of what one of those JSON files look like –

https://www.dropbox.com/s/xrfczynami7b2hb/2020-04%20flickr%20JSON%20example%20-%20photo_57298148.json?dl=0

Hopefully that's helpful. A tricky thing with the with JSON files is that I don't know how to reference nested fields. When running exiftool on the JSON file, I get a comma-separated list of values called "Albums Title" and "Tags Tag", which is good, but I think exiftool doesn't allow for spaces in the tag titles, so I'm not sure how I can refer to them. Tags.Tag? TagsTag?

StarGeek

Quote from: andreasmb on April 16, 2020, 11:10:52 PM
...and the json files for all the media files are also batched into a smaller set of separate folders, but there's only two:


  • data-download 72157713880626132_375a3c67e881_part1
  • data-download 72157713880626132_375a3c67e881_part2

If you can drop them all into the same directory without any filename collisions, then you could replace the %d in the above command with the path to that directory, e.g.
exiftool -TagsFromFile /path/to/jsonfiles/photo_%-8.2f.json "-IMAGETAG<JSONTAG" /path/to/imagefiles/
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

andreasmb

#4
That seems like it should do the trick!

However, I'm getting an error when trying to read from the JSON file.

Warning: Error opening file - /json/photo_57298148.json

Here's a picture of file structure and the test command:
https://www.dropbox.com/s/t322vo23j18hipu/2020-04%20exiftool%20test%20output.png?dl=0[/s]

Edit: I had an extraneous forward slash in front of my folder path, which caused the issue.

andreasmb

Everything seems to be working well! Thanks a million @StarGeek!

One minor hiccup seems to be that copying values from two places ("TagsTag" and "AlbumsTitle") and writing to the same field ("keywords") doesn't work. One overwrites the other, so I'm left with the values from either "TagsTag" or "AlbumsTitle". Ideally, there's the equivalent of a concat() function that lets me join or "add to" an existing exif field.

Is it possible to "add to" an exif tag? 

Phil Harvey

Quote from: andreasmb on April 17, 2020, 02:26:55 AM
One minor hiccup seems to be that copying values from two places ("TagsTag" and "AlbumsTitle") and writing to the same field ("keywords") doesn't work.

Use -addTagsFromFile instead of -tagsFromFile to do this.  (For more information, read Note 5 of the -tagsFromFile section in the exiftool application documentation.)

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

andreasmb

Brilliant... just brilliant. I'm definitely donating to the upkeep of this excellent piece of software!

nmerchant

Hi there,

The above post is very helpful as I am intending to leave flickr and would like to retain the metadata.
My challenge is mapping the album title and description.
Flickr provides a separate Album.json file which has an array of albums. Each album has an array of photo IDs
Here is an example:
{
   "albums": [
      {
         "photo_count": "163",
         "id": "72177720295550429",
         "url": "https://www.flickr.com/photos/xxxx/albums/72177720295550429",
         "title": "Album # 1",
         "description": "Album # 1 Description",
         "view_count": "0",
         "created": "1640670226",
         "last_updated": "1640670975",
         "cover_photo": "https://www.flickr.com/photos/xxx/0",
         "photos": [
            "51781521571",
            "51782399990",
            "51781775853",
            "51781519701",
            ]
               {
         "photo_count": "19",
         "id": "72157720030487061",
         "url": "https://www.flickr.com/photos/xxxx/albums/72157720030487061",
         "title": "Album # 2",
         "description": "Album # 2 Description",
         "view_count": "0",
         "created": "1634433341",
         "last_updated": "1634433359",
         "cover_photo": "https://www.flickr.com/photos/xxxx/0",
         "photos": [
            "51597322671",
            "51598223380",
            "51597322791",
            "51597989364",
            
         ]
      },

Every photo has a corresponding JSON file such as photo_51597989364.json.
I would like to store album title and description from album.json to  each of the photo json file by looking at the ID or the file name

Here is the photo_51597989364.json example
{
   "id": "51597989364",
   "name": "",
   "description": "",
   "count_views": "0",
   "count_faves": "0",
   "count_comments": "0",
   "date_taken": "2007-08-12 00:06:50",
   "count_tags": "0",
   "count_notes": "0",
   "rotation": 0,
   "date_imported": "2007-08-12 00:06:50",
   "license": "All Rights Reserved",
   "geo": [

   ],
   "groups": [

   ],
   "album[
                Album # 2
   ],
   "tags": [

   ],
   "people": [

   ],
   "notes": [
],
   
}

StarGeek

Quote from: nmerchant on August 07, 2022, 04:25:40 PMI would like to store album title and description from album.json to  each of the photo json file by looking at the ID or the file name

Exiftool doesn't have the ability to edit JSON files.  You'll have to use a program such as JQ or similar program.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

nmerchant

I see. Is it possible to migrate album title and album description from album.json below to an image file. Image file has the naming convention photo_51781525411_o.jpg. The 51781525411 is the image ID which can be parsed in the album.json file

Phil Harvey

You would need to parse the JSON file with something other than ExifTool to associate the metadata with a specific file using the ID.  But if your JSON were structured in the same way as the ExifTool -j output then you could do this (ie. with a "SourceFile" entry instead of "ID" to identify the file, and labels that use ExifTool tag names).

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

nmerchant

thanks Phil.

How do I make sure that JSON is structured in the same way as the ExifTool -j output ? and can you give me an example ?

Below is the sample of album.json file. The Photo ID under photos array is part of the image filename photo_51781521571_o.jpg. These images are stored in different folder organized by Flickr as data-download-1, data-download-2 and so on...

{
   "albums": [
      {
         "photo_count": "163",
         "id": "72177720295550429",
         "url": "https://www.flickr.com/photos/xxxx/albums/72177720295550429",
         "title": "Album # 1",
         "description": "Album # 1 Description",
         "view_count": "0",
         "created": "1640670226",
         "last_updated": "1640670975",
         "cover_photo": "https://www.flickr.com/photos/xxx/0",
         "photos": [
            "51781521571",
            "51782399990",
            "51781775853",
            "51781519701",
            ]
               {
         "photo_count": "19",
         "id": "72157720030487061",
         "url": "https://www.flickr.com/photos/xxxx/albums/72157720030487061",
         "title": "Album # 2",
         "description": "Album # 2 Description",
         "view_count": "0",
         "created": "1634433341",
         "last_updated": "1634433359",
         "cover_photo": "https://www.flickr.com/photos/xxxx/0",
         "photos": [
            "51597322671",
            "51598223380",
            "51597322791",
            "51597989364",
           
         ]
      },

Phil Harvey

Run ExifTool with the -j option on a file to see what the -j output looks like. :)

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).