Organising folders of photos: changing filenames and folder name

[Isonomia]

For my sins I am using windows 10, picasa to face tag photos, but I intend to convert the face tags for use other software, and at present the target is digikam. I normally use a Canon camera with Zoombrowser to import images and for displaying images. As such my folder structure reflects what is easily done with these:
(to avoid confusion, I'm calling the text I add to folder and file names the "annotation")

My folder structure is

Pictures -> Year -> YYYY_MM_DD - annotation -> IMG_1234 - annotation.EXT

E.g. for a picture taken today of icicles in the frosty garden it may be either:-

Pictures -> 2023 -> 2023_01_17 -> IMG_1234.jpg
Pictures -> 2023 -> 2023_01_17 - frosty garden -> IMG_1234 -icicles.jpg

But if taken with another camera it could be:

Pictures -> 2023 -> 2023_01_17 -> p1234567.png
Pictures -> 2023 -> 2023_01_17 - frosty garden -> p1234567 icicles.png


Yes the folder 2023 is redundant, but there would be too many subfolders (i.e. YYYY_MM_DD) if they all went into one folder. By using the year, I just have to change the targer folder for import once a year.

PROBLEM

This is the problem: when I run out of space I back up my photos and keep some on the PC. But, as I work on them in the working folder on my PC I add the description text to what was a unique identifier for the folder and image. That means, I end up having different names on the backup and working directory on my computer. Which means that as I backup and restore to work on images, I keep getting duplicate folders and images.

OLD FAMILY PHOTOS

The problem has really come to a head when trying to digitise old family photos. Because these go through different stages of processing (Scanning/photo, cleaning up, adding meta data) there are multiple copies. To make it worse, the same photo appears in different family albums, and of course, I seldom know the date the photo was taken. So, the file structure that has worked so well, is falling apart, and with the issue of restoring backups, and retaking the photo, I now have eight copies of some old photos. So I'm having a major rethink.

REAL SOLUTION

The proper solution would be for windows to display the filename and a "annotation" on the folders and images in the folder view. So, if I restore a folder back onto the PC, it recognises that the image: "IMG_1234" and "IMG_1234 - icicles" are the same thing. However I have checked windows and there is no such option in windows. AND OF COURSE - IT DOES NOT HELP THAT WINDOWS USES DIFFERNT METADATA FIELDS AND OVERWRITES THEM!! But my main problem is adding a description to the folders in a way I can restore them without creating new versions due to the added description. So even if I could get an annotation in windows it would be a start.

EXPLORER ALTERNATIVES

I looked to see if there is any alternative to windows explorer that does this, and I could not obviously see any.

EXIFTOOL

Although I have used Exiftool for the odd thing, such as converting Picasa metadata to a more universal form, I struggle with the simplest command line. So, a command that changes all my photos is going to be very daunting!

Checking I found a comment from Phil that suggests the following code can change the filename:

Code Select Expand

"exiftool.exe" "-testname<$CreateDate - ${model;} - %f.%e" -d "%Y-%m-%d_%H.%M.%S" -api QuickTimeUTC .
So, I think exiftool could help me!

I am thinking that it might be possible, to detect the "comment" part of each folder/file name (the bit after the first space) and to remove it from the name and store it temporarily in a metafield. If I do this on both the backup and the working folder, I can then combine the pair. I can then restore the original folder and file names by adding back the annotation from the meta field.

QUESTION 0

Do folders have metadata? If not, is there a way to create a "folder-annotation" txt file?

QUESTION 1

Which metadata field should I use to store the "annotation" part of the file name.

I was thinking of using an existing field. This way I could display and even perhaps edit it.

Trying to find this "one metafield to rule them all" I discovered that:

Windows Folder Title Changes:
   XMP:Title **(Displayed Title)
   XMP:Description
   IPTC:Caption-Abstract
   Exif:XPTitle
   Exif:ImageDescription

Windows folder subject changes
   EXIF:XPSubject **(Displayed Subject)

Picasa Caption Changes:
   IPTC:Caption-Abstract **(Display Caption)
   XMP:Description

I have added a few comments in Picasa. So, my thinking is that the one that is "safest" for my use is EXIF:XPSubject ... or should I invent ANOTHER tag?

QUESTION 2

What are the command(s) to go through each folder and each file stripping off the text after the first space and placing it into EXIF:XPSubject and then changing the filename?

QUESTION 3

How do I reverse the above?

QUESTION 4

Is this really the way to do it? I need the folders dated so that they are in order and I can find the right one. But I also need to display a description so - again so I can find the right folder. Windows only displays the folder name, so the two have to be in the file name. I can't think of a way around a dual purpose folder/file name. Surely after 50 years of home PCs, there has got to be a better way!

[StarGeek]

Do folders have metadata? If not, is there a way to create a "folder-annotation" txt file?

Technically, yes.  But not really in a useful way.  Windows has Alternate Data Streams (ADS) and Mac has XAttr and MDItem attributes which can be applied to directories, though I'm not sure if these can be applied to directories.  On Windows, almost nothing can use ADS and they are often used by malware to hide.

So in practice, not really, at least not without a lot of work.

Which metadata field should I use to store the "annotation" part of the file name.

Trying to find this "one metafield to rule them all" I discovered that:

See this post for a list of what tags Windows reads to fill the Properties window.

The IPTC Photo Metadata Standard is the best thing to look at to figure out the purpose of various tags.

I have added a few comments in Picasa. So, my thinking is that the one that is "safest" for my use is EXIF:XPSubject ... or should I invent ANOTHER tag?

All the EXIF:XP* tags are pretty much Windows only, as almost no programs outside of Windows will read them.  They are not a good choice, IMO.  And, also IMO, inventing your own tags is almost never a good idea, as nothing other than exiftool or any code you write will ever be able to read them.

The best thing to use (other than exiftool ), is a Digital Asset Management (DAM) program.  These programs will index you images, write metadata to the correct locations, and help you find duplicates.  Picasa is such a program, though IMO, it is a very poor one.  Pretty to look at but I found that it will destroy some metadata in files, most notably in my case the Nikon Makernotes.

Adobe Bridge, while not a full fledged DAM, will allow you to write metadata to files in the proper places.  It is free to use.  More powerful programs are DigiKam (free), Lightroom, and IMatch (both paid).  I personally use IMatch.  It's expensive but very powerful.  DigiKam is a good place to start and you can decide later if you want to use something else.  Because all of these programs write metadata to the correct locations, you can move between them freely.

Going back to your question about where to store your annotations, three locations might be appropriate. Title is "A shorthand reference for the digital image. Title provides a short human readable name which can be a text and/or numeric reference". The tags to write for this would be XMP:Title and/or IPTC:ObjectName.

Headline is "A brief synopsis of the caption".  Basically a short version of Description. The tags to write for this would be XMP:Headline and/or IPTC:Headline.  By default, if you do not give a group name (i.e. -Headline=Text), exiftool will only write to IPTC:Headline. 

Adobe breaks with the standard by using the Title field as the short description, which should be Headline.

The third tag, which is less commonly used, would be Event (XMP:Event). Though that may not really be appropriate given some of your examples.

You should remember that this is your data.  You don't have to be locked into the standards and can use any tag in any way you want.  If you decide to change things at a later, you can always use exiftool to move things around in bulk.

What are the command(s) to go through each folder and each file stripping off the text after the first space and placing it into EXIF:XPSubject and then changing the filename?

A bit more detail is needed.  Do you mean the text in the directory name (2023_01_17 - frosty garden) or the filename (IMG_1234 -icicles.jpg).  Do you want to include the leading dash?  Have you been consistent with using the dash (p1234567 icicles.png)?

The basic command would be something like this. 
exiftool -r -overwrite_original "-Filename<${Filename;s/(^[^ ]+).*/$1/}" "-XPSubject<${Filename;s/^[^ ]+ -?([^\.]+).*/$1/}" /path/to/files/

"-Filename<${Filename;s/(^[^ ]+).*/$1/}.%e" takes the filename, captures everything up to the first space and removes the rest.  %e is used to keep the extension.

"-XPSubject<${Filename;s/^[^ ]+ -?([^\.]+).*/$1/}" takes everything up to the first space and removes it and then strips the extension.  It also drops the dash.  Remove the -? if you want to keep the dash.

Because you are renaming and editing in the same command, the -overwrite_original option is needed to avoid creating a duplicate of the original with the edits. See Notes on the "Writing "FileName" and "Directory" tags" page.

How do I reverse the above?

exiftool "-Filename<%f -$XPSubject.%e" /path/to/files/

Is this really the way to do it? I need the folders dated so that they are in order and I can find the right one. But I also need to display a description so - again so I can find the right folder. Windows only displays the folder name, so the two have to be in the file name. I can't think of a way around a dual purpose folder/file name. Surely after 50 years of home PCs, there has got to be a better way!

That really is something you'll need to figure out what works best for you.  Personally, I don't use Windows to search or display anything.  To find images, I use IMatch and keywords for the pictures.  This allows me to tag images with multiple properties.  For example, I mostly take pictures at comic conventions.  Using the keywords and IMatch allows me to tag large groups of cosplayers with all the costumes their wearing as well as the names of the people.  Putting all that into the name is unwieldy to say the least.

Also, I have Windows search disabled.  It's a resource hog and doesn't do a good job IMO.  If I need to search by filenames, Everything Search is an extremely powerful and better search option.

[Isonomia]

Thanks! That is an amazingly detailed response. I also love the regex ... which is far superior to the approach I was heading toward (!!NOT TESTED!!):-

Code Select Expand

FORWARD FILES:
Set the description from the "annotation" [IMG_1234 Annotation => Description=Annotation}
    exiftool '-Description<${FileName;s/^[^\s]*\s//;s/\.[^.]*$//}' dir

Rename file removing annotation: [IMG_1234 Annotation.ext -> IMG_1234.ext]
    exiftool -filename<${FileName;s/\s[^\.]*//} dir
   
REVERSE FILES:
   
Rename file adding in Description [IMG_1234.jpg -> IMG_1234 Description.jpg]
    exiftool '-filename<%f ${Description;}.%e' dir

But with your help I think I can change the file names. However, I am really stuck on directories. I think I can move all the files to a new directory which excludes the annotation part of the directory name using something like this:

Code Select Expand

Rename dir removing annotation part, after first space of name(but keeps old directory!)
    exiftool -Directory<${Directory;s/\s[^\.]*//} dir
That will leave a lot of empty directories ... but an empty directory uses a lot less space that two directories that are identical except for an extra comment in the directory title!

But if I am going to restore the original name, I need to store it somehow and I cannot think how to store the old name or annotation.

My first idea was to work out a way to create a file in each directory (say meta.txt) and store the name in a meta data field of each file meta.txt" in each directory. That way I read the metadata using Exiftool and then move the files back. But I then found txt files don't have metadata.

I know I can write output to a file using ">" appended to the end. So, I could create a txt file in each directory called: "meta.txt". I assume I can find a way to use "-r" and then write the directory name to each meta.txt. I only want to do this once in each directory, so I assume I will have to find a way to use "EndDir()" somehow. But I don't know how to do this recursively, writing the output to each meta.txt in each directory and not just the one meta.txt in the directory where I execute the command.

As for reversing the process of reading the data from each txt file and then changing the directory name, I am totally out of my depth!

ADDENDUM

I finally discovered the "-w" directive. Then after a bit of trial and error I finally managed to store the (old) directory name in a file. So, I think I might have a way to do it, albeit, it needs a lot of testing before I go anywhere near my photos.

In theory ... this command:-

   exiftool -r -W! %dmeta.txt -Directory DIR

Produces a file "meta.txt" in each directory with the contents (For a directory named "DIR/2022_01_01" of:

   "Directory                       : DIR/2022_01_01"

I can move all the files in a directory DIR and its subdirectors to another one where the name removes the annotation part (which is anything after the space after YYYY_MM_DD) with this

    exiftool -Directory<${Directory;s/\s[^\.]*//} DIR

And to get back to where I start I can use this command:

   exiftool -r -ext jpg "-Directory<=meta.txt
   
which reads the file meta.txt and then moves each jpg back to the original directory.

[Isonomia]

I've almost managed it, but I've hit a roadblock.

I can create a meta.txt file with the directory name thus:

Code Select Expand

exiftool -r -s3 -W! %dmeta.txt -Directory DIR
I can then move my files and the meta.txt to the new directories thus:

Code Select Expand

exiftool -r -ext jpg "-Directory<${Directory;s/\s[^\.]*//}" DIR
exiftool -r -ext "*" -if "$filename=~/meta.txt/" "-Directory<${Directory;s/\s[^\.]*//}" DIR
And then this command ought to move all the files back to the directory in meta.txt

Code Select Expand

exiftool -r -ext "*" "-Directory<=%dmeta.txt"
However, there is a newline at the end, and this makes it unusuable. And, I cannot see any way to remove the newline from either the output using -w or from the input using "<=". And if I don't have -s3, it is worse.

[Phil Harvey]

Try using -b instead of -s3 to the command where you are writing meta.txt.

Also, there could be some optimizations that might be made, but I don't have time right now to read through this whole post to be sure I understand.   For example, in this command

Code Select Expand

exiftool -r -ext "*" -if "$filename=~/meta.txt/" "-Directory<${Directory;s/\s[^\.]*//}" DIR
why are you using -ext "*" and not -ext txt ?

- Phil

[Isonomia]

Thanks Phil, you are a star!!! I've finally got something that works. Not perfect, but on my test directory it does the basics which I need, even if I end up with empty directories.

I will detail the complete solution so far.

The reason I am doing this is to restore the directory and files to a "bare bones" form without annotation. Directories are just date and images are just ID. If I also do this to the backup, I can combine the two without creating duplicate directories of images. At the end I want to get back to the files and directories with annotations by adding back the annotation to the image and file name

First "DIR" is either something like C:\Users\Phil\Pictures\tst\ ... which means at the end they are written to this exact directory, or you can CD C:\Users\Phil\Pictures and then DIR is tst (or whatever you name the main directory with the relevant photos. You can then repeat the steps by CD to a backup drive where you have a directory with the same directory structure again same named (tst) and by copying that over the "tst" at  C:\Users\Phil\Pictures\ and skipping any duplicates, the reverse process should add in all the missing folders which can be restored to their names with annotations as found on the backup drive.

Second, file naming is

<ID><SEP><ANNOTATION><EXT>

Where:
<ID> is a unique string excluding any separator
<SEP> is one or a combination of space and "-"
<ANNOTATION> is a string which may include separators but must not include "."
<EXT> is an optional .ext (obviously missing with directories)

To store files whose name has an annotation in XPSubject, rename pictures removing annotation & then restore


TO CLEAR XPSubject

Code Select Expand

exiftool -r -P -overwrite_original -XPSubject= DIR
This is a precaution as if there is a file name clash the existing text in XPSubject will not be changed and it will later be incorporated in the file name.

TO WRITE FILE ANNOTATION TO XPSubject CHANGE FILENAME TO REMOVE SEP & ANNOTATION (for files with extensions jpg, bmp & gif)

Code Select Expand

exiftool -r -P -overwrite_original -ext jpg -ext bmp -ext gif "-Filename<${Filename;s/[\s][^\.]*//}" "-XPSubject<${FileName;s/\.[^\.]*$//;s/^[^\s]*[\s-]*//}" DIR
s/\..*$// finds the first dot and replaces everything from the dot to the end with ""
s/^[^\s\.]*[\s-]*// looks for a block without space or "." and then it finds any additional spaces or "-" afterwards, and replaces them with ""
"s/[\s][^\.]*//" looks for a space and then for a block of anything except "." and removes them. Removes from first space to first "."

TO CHANGE FILENAME BACK TO ADD " - ANNOTATION"

Code Select Expand

exiftool -r -P -ext jpg -ext bmp -ext gif "-Filename<%f - $XPSubject.%e" DIR
To store current directory name in meta.txt, rename directories and the restore

To store directory name in meta.tx

Code Select Expand

exiftool -r -b -W! %dmeta.txt -Directory DIR
To move files to a new directory without the annotation in directory name (just jpg)

Code Select Expand

exiftool -r -P -ext jpg "-Directory<${Directory;s/[\s-]+[^\.]*//}" DIR
To move the meta.txt to the new directory

Code Select Expand

exiftool -r -ext "txt" -if "$filename=~/meta.txt/" "-Directory<${Directory;s/[\s-]+[^\.]*//}" DIR
To get back to where I start(ish) (Moves all files back to directory as given in meta.txt)

Code Select Expand

exiftool -r -P -ext "*" "-Directory<=%dmeta.txt" DIR

NAME CLASHES
If when renaming files or moving files to a new directory, there is already a file of that name, exiftool does not move the file. So if I had "IMG_1234" and "IMG_1234 edited", the above attempts to rename them both as IMG_1234 and fails and so the two remain where they were.

ISSUES

    • I think I am rewriting meta.txt for each file .. I see no way to stop processing after the first file in each directory (-if Endfile() stops processing before the first file is processed)
    • The annotation should not contain a ".". If it does, it treats everything up to the first "." as annotation. It ignores everything after the first dot.
    • Ideally I would check the contents of the annotation in XPSubject and Meta.txt to ensure they are fit for filenames. So, if you edit XPSubject or meta.txt, you should not add in a "." or any other characters not permitted in filename.
    • When I add the annotation, I should first remove any annotation already present, so this: "-Filename<%f - $XPSubject.%e" should be something like "-Filename<${Filename;s/\.[^\.]*$//} - $XPSubject.%e"  ... but that doesn't work
    • At the end I'm left with a lot of empty directories (or with files not moved)
    • To be consistent I shouldn't be storing the whole directory name, but instead I should just store the annotation text in meta.txt. I should then rebuild the directory name from the current name plus the annotation text. But I can find no way to work on the output using -W nor the input when reading from a file using <=.
JUST A THOUGHT
Lying in bed, I was thinking how exiftool could work on directories. And, the best way is not to work on the directory that exiftool is currently in, but to work on subdirectories as if they are files. So, this could be achieved using a "-ext /" meaning the files being selected are directories.

Then, any tags written to the "current file" (i.e a directory), except those that have a meaning (creation date) would be written to a file within the sub directory called .meta.exif (I assume that's the ending). And likewise any read. That would allow directories to be treated like other files and have "it's metadata" read and written.

[StarGeek]

However, I am really stuck on directories. I think I can move all the files to a new directory which excludes the annotation part of the directory name using something like this:

You can include directory paths when you do the file rename.  Example #6 on the Writing "FileName" and "Directory" tags page shows the common example of including directory paths as part of the date string for renaming and moving files.  But it can also be directly set
exiftool "-Filename=c:\pictures\new\directory\%f%-c.%e" /path/to/files/

That will leave a lot of empty directories

If you have Perl installed, I use this one liner to recursively remove empty directories
perl -MFile::Find -e"finddepth(sub{rmdir},'/path/to/files/')".

I also use this Windows program which can remove empty directories with a GUI.  It also has some options treat directories as empty if they contain only certain files.   For example, I have it set to remove a directory if it contains only a desktop.ini or Thumbs.db file and nothing else.

But if I am going to restore the original name, I need to store it somehow and I cannot think how to store the old name or annotation.

The IPTC standard linked above gives IPTC:ObjectName and XMP:Title as two options for this.  Adobe uses XMP:PreservedFileName for this.

[Phil Harvey]

• I think I am rewriting meta.txt for each file .. I see no way to stop processing after the first file in each directory (-if Endfile() stops processing before the first file is processed)

Add -if "EndDir()" to your command.  This stops processing the directory after the first file.

- Phil