Selective and Recursive File Copy

[JohnF]

I have published a large collection of images which are saved in a hierarch of folders and subfolders.  The images are all ranked. I would like to filter this large collection and make a copy based on image ranking.  I have found the following from a post dating back to Aug 22, 2011.

-----
Here is an example command that assumes the tag name is "Rating", and copies all files from directory c:\source\directory with a rating of 2 or higher to another directory:

exiftool "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory"
-----

I have modified this to include recursion with -r

exiftool -r "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory"

which recursively operates on on the folders and subfolders specified as the source BUT all files are output to the singular output folder.  I wish to have the same folder structure mirrored under the specified output folder (assuming the any given output folder is not empty). 

I've looked far and wide across the forum but cannot seem to find the solution.  I'm sure it must be something quite obvious.  We're essentially trying to perform and xcopy using rating as a filter.

[StarGeek]

You have to use the directory %d variable as detailed under the -w (-TextOut) option.

The easiest thing to do would be to CD to your source directory "c:\source\directory".  Then you just have to add the %d to the output name. In this case, the dot is the current directory.

exiftool -r . -if "$rating >= 2" -o "c:\destination\directory\%d"

If you don't CD to the directory, you have to edit the %d as shown under the Advanced features section of the above link.

In your example of "c:\source\directory", you would want to remove the three top levels of the directory path, so your command would become

exiftool -r "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory\%:3d"

Make sure and test these command first in case I've made a mistake.

[JohnF]

Hello StarGeek,

this is exactly the help I needed. Thanks for providing all three solutions to build out my understanding. In the end, It's easy to use full address - I simply build the commandline in Word, then paste in into the CMD window making your third approach just what I'll routinely use.

[JohnF]

Hello StarGeek...

One additional filtering case... this time based on keywords (often via face recognition) that I've added to the file in Lightroom that come through as "Tags" when looking at the Details tab on the File Properties in Windows.  Instead of using:

-if "$rating >= 2"

What syntax should I insert?

[StarGeek]

See this post to translate Windows Properties into actual tag names.

In this case, Subject is probably what you want to search on, maybe Keywords, but if it is Lightroom that added them, then Subject is the tag.

The usual way to filter on Subject would be to use Perl Regular Expressions (RegEx), which is a very complex subject on its own.  An example would be
-if "$Subject=~/John Smith/"

But if you wanted to search for more than one, or if you wanted a case insensitive search, it can be much more complex.  It would be easier if you have some specific examples to work with.

[JohnF]

Hi StarGeek,

Yes, I'm on a PC with Windows 10.

Perfect... This works in just the way I had hoped.  With your answer as a template I also tried using

-if "$Keywords=~/John Smith/" which for my testing worked identically. 

Is there any reason that you suggested the use of $Subject ?

So, with such great success, can you give me a tip where we have an "AND" function, requiring two keyword entries?  Something like...  -if "$Keywords=~/John Smith/ && /Karen Carpenter/"

 

[Phil Harvey]

Is there any reason that you suggested the use of $Subject ?

Subject is the newer XMP version of the older IPTC Keywords tag.

Something like...  -if "$Keywords=~/John Smith/ && /Karen Carpenter/"

Close.  Try this:

-if "$Keywords=~/John Smith/ && $Keywords=~/Karen Carpenter/"

- Phil

[StarGeek]

Is there any reason that you suggested the use of $Subject ?

As Phil says it is newer. If you're writing data with LightRoom, then this is where it will save it.  The Keywords tag is part of the much older IPTC standard.  Lightroom will also write Keywords but only if it or any other IPTC tag already exists.

To further complicate things, if you are writing the tags through the Windows Property window, Windows will also write the XPKeywords tag, but that is basically a Windows only tag and can (should IMO) be ignored.

[JohnF]

Hello StarGeek and Phil,

many thanks for your help...

I've been able to create a small program which creates the needed input strings for Exiftool which allows me and more importantly others to sort through a large photo library based on existing image Ratings and Keywords outside of the Lightroom world where the original exists. 

I've also been able to incorporate both AND and OR logic and the option to preserve or flatten the existing folder structure for filtered output.  The current image library has nearly 4000 entries, so this is a big help.

I do have a follow-on question.  To make this all a bit more robust with regards to the entry of Keywords, can you suggest a method to "scan" the library and create a comprehensive listing of keywords that I can then add to my program?

This has been a great experience.  See the program GUI in the attached.

John

[Phil Harvey]

Hi John,

On Mac/Linux this command will give you a sorted list of keywords (from XMP:Subject):

exiftool DIR -subject -b -sep "\n" -sep "\n" | sort | uniq

(yes, two -sep options are necessary)

...but I don't know if the sort and uniq utilities are available with Windows.

- Phil

[JohnF]

Thanks Phil,

Is there someway that the output can be written to a file?

[JohnF]

Hi Phil,

The command sequence you provided works for me in Windows except for the | uniq command.  I get a full listing.  I'll either see if there is a windows equivalent, or as I'm calling this programmatically, I can remove duplicates it program which is invoking Exiftool.  Hurray!


However, when I call the above programmatically, then the listing goes straight to the CMD window, hence the request in the previous post to learn if there is a way to get the output to go to file. 

Here's what I see when I try to call and read the output directly.  The procedure does not complete with an error on the line...

nCharaters = AvailableProgramOutput(Program_code)

that indicates the specified program (ExifTools, referenced via Program_code) is not valid. This suggests to me that after the execution of the RunProgram command that calls Exiftools does not establish a connection that allows a continuing connection that would support reading back the data - consistent with the fact that the data simply streams by in the CMD window.

I'm happy to try to bring data in directly to the calling program or indirectly via writing to a file, just not sure how either might be done.

Here's the relevant procedure code (written in PureBasic - which has a nice IDE and compiler which creates an executable exe that will run elsewhere without any supporting dlls or other runtime environment.)

Code Select Expand

Procedure Find_Keywords(EventType.i)
  Exiftool$ = "exiftool"
  Workingdirectory$=""
  Commandline$ = " -r " + Chr(34) + Input_Directory$ + Chr(34) + " -subject -b -sep " + Chr(34) + "\n" + Chr(34) + " -sep " + Chr(34) + "\n" + Chr(34) + " | sort"
  Program_code = RunProgram(Exiftool$,Commandline$,Workingdirectory$,#PB_Program_Read)
  List_Counter=0
  nCharacters = AvailableProgramOutput(Program_code)
  While nCharacters
    List_Counter+List_Counter+1
    Keyword_List.s(List_Counter)= ReadProgramString(Program_code)
    nCharacters = AvailableProgramOutput(Program_code)
  Wend
  SetGadgetText(Text_0,Exiftool$+Commandline$+"  "+Str(Input_Level))
EndProcedure

[Phil Harvey]

Hi John,

To write the output out a file, do this:

exiftool DIR -subject -b -sep "\n" -sep "\n" | sort | uniq > out.txt

- Phil

[StarGeek]

The command sequence you provided works for me in Windows except for the | uniq command.  I get a full listing.  I'll either see if there is a windows equivalent, or as I'm calling this programmatically, I can remove duplicates it program which is invoking Exiftool.

There are Windows ports of Linux commands, which is useful as the Linux commands are very powerful and it's easy to search StackExchange to figure out how to do specific things.  I use the MSys2 ports.

However, when I call the above programmatically, then the listing goes straight to the CMD window, hence the request in the previous post to learn if there is a way to get the output to go to file.

Phil's answer uses the command lines file redirection ability to pipe the output into a file.  This is often unavailable when calling from a program.  In such cases, there should be a way to capture the output what your programming language's system call.

[Phil Harvey]

Phil's answer uses the command lines file redirection ability to pipe the output into a file.  This is often unavailable when calling from a program.

This is true since it requires that the command is executed by a shell processor.  But I think it should work here because the pipe to sort (| sort) works.

- Phil