Selective and Recursive File Copy

[JohnF]

I have published a large collection of images which are saved in a hierarch of folders and subfolders.  The images are all ranked. I would like to filter this large collection and make a copy based on image ranking.  I have found the following from a post dating back to Aug 22, 2011.

-----
Here is an example command that assumes the tag name is "Rating", and copies all files from directory c:\source\directory with a rating of 2 or higher to another directory:

exiftool "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory"
-----

I have modified this to include recursion with -r

exiftool -r "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory"

which recursively operates on on the folders and subfolders specified as the source BUT all files are output to the singular output folder.  I wish to have the same folder structure mirrored under the specified output folder (assuming the any given output folder is not empty). 

I've looked far and wide across the forum but cannot seem to find the solution.  I'm sure it must be something quite obvious.  We're essentially trying to perform and xcopy using rating as a filter.

[StarGeek]

You have to use the directory %d variable as detailed under the -w (-TextOut) option.

The easiest thing to do would be to CD to your source directory "c:\source\directory".  Then you just have to add the %d to the output name. In this case, the dot is the current directory.

exiftool -r . -if "$rating >= 2" -o "c:\destination\directory\%d"

If you don't CD to the directory, you have to edit the %d as shown under the Advanced features section of the above link.

In your example of "c:\source\directory", you would want to remove the three top levels of the directory path, so your command would become

exiftool -r "c:\source\directory" -if "$rating >= 2" -o "c:\destination\directory\%:3d"

Make sure and test these command first in case I've made a mistake.

[JohnF]

Hello StarGeek,

this is exactly the help I needed. Thanks for providing all three solutions to build out my understanding. In the end, It's easy to use full address - I simply build the commandline in Word, then paste in into the CMD window making your third approach just what I'll routinely use.

[JohnF]

Hello StarGeek...

One additional filtering case... this time based on keywords (often via face recognition) that I've added to the file in Lightroom that come through as "Tags" when looking at the Details tab on the File Properties in Windows.  Instead of using:

-if "$rating >= 2"

What syntax should I insert?

[StarGeek]

See this post to translate Windows Properties into actual tag names.

In this case, Subject is probably what you want to search on, maybe Keywords, but if it is Lightroom that added them, then Subject is the tag.

The usual way to filter on Subject would be to use Perl Regular Expressions (RegEx), which is a very complex subject on its own.  An example would be
-if "$Subject=~/John Smith/"

But if you wanted to search for more than one, or if you wanted a case insensitive search, it can be much more complex.  It would be easier if you have some specific examples to work with.

[JohnF]

Hi StarGeek,

Yes, I'm on a PC with Windows 10.

Perfect... This works in just the way I had hoped.  With your answer as a template I also tried using

-if "$Keywords=~/John Smith/" which for my testing worked identically. 

Is there any reason that you suggested the use of $Subject ?

So, with such great success, can you give me a tip where we have an "AND" function, requiring two keyword entries?  Something like...  -if "$Keywords=~/John Smith/ && /Karen Carpenter/"

 

[Phil Harvey]

Is there any reason that you suggested the use of $Subject ?

Subject is the newer XMP version of the older IPTC Keywords tag.

Something like...  -if "$Keywords=~/John Smith/ && /Karen Carpenter/"

Close.  Try this:

-if "$Keywords=~/John Smith/ && $Keywords=~/Karen Carpenter/"

- Phil

[StarGeek]

Is there any reason that you suggested the use of $Subject ?

As Phil says it is newer. If you're writing data with LightRoom, then this is where it will save it.  The Keywords tag is part of the much older IPTC standard.  Lightroom will also write Keywords but only if it or any other IPTC tag already exists.

To further complicate things, if you are writing the tags through the Windows Property window, Windows will also write the XPKeywords tag, but that is basically a Windows only tag and can (should IMO) be ignored.

[JohnF]

Hello StarGeek and Phil,

many thanks for your help...

I've been able to create a small program which creates the needed input strings for Exiftool which allows me and more importantly others to sort through a large photo library based on existing image Ratings and Keywords outside of the Lightroom world where the original exists. 

I've also been able to incorporate both AND and OR logic and the option to preserve or flatten the existing folder structure for filtered output.  The current image library has nearly 4000 entries, so this is a big help.

I do have a follow-on question.  To make this all a bit more robust with regards to the entry of Keywords, can you suggest a method to "scan" the library and create a comprehensive listing of keywords that I can then add to my program?

This has been a great experience.  See the program GUI in the attached.

John

[Phil Harvey]

Hi John,

On Mac/Linux this command will give you a sorted list of keywords (from XMP:Subject):

exiftool DIR -subject -b -sep "\n" -sep "\n" | sort | uniq

(yes, two -sep options are necessary)

...but I don't know if the sort and uniq utilities are available with Windows.

- Phil

[JohnF]

Thanks Phil,

Is there someway that the output can be written to a file?

[JohnF]

Hi Phil,

The command sequence you provided works for me in Windows except for the | uniq command.  I get a full listing.  I'll either see if there is a windows equivalent, or as I'm calling this programmatically, I can remove duplicates it program which is invoking Exiftool.  Hurray!


However, when I call the above programmatically, then the listing goes straight to the CMD window, hence the request in the previous post to learn if there is a way to get the output to go to file. 

Here's what I see when I try to call and read the output directly.  The procedure does not complete with an error on the line...

nCharaters = AvailableProgramOutput(Program_code)

that indicates the specified program (ExifTools, referenced via Program_code) is not valid. This suggests to me that after the execution of the RunProgram command that calls Exiftools does not establish a connection that allows a continuing connection that would support reading back the data - consistent with the fact that the data simply streams by in the CMD window.

I'm happy to try to bring data in directly to the calling program or indirectly via writing to a file, just not sure how either might be done.

Here's the relevant procedure code (written in PureBasic - which has a nice IDE and compiler which creates an executable exe that will run elsewhere without any supporting dlls or other runtime environment.)

Code Select Expand

Procedure Find_Keywords(EventType.i)
  Exiftool$ = "exiftool"
  Workingdirectory$=""
  Commandline$ = " -r " + Chr(34) + Input_Directory$ + Chr(34) + " -subject -b -sep " + Chr(34) + "\n" + Chr(34) + " -sep " + Chr(34) + "\n" + Chr(34) + " | sort"
  Program_code = RunProgram(Exiftool$,Commandline$,Workingdirectory$,#PB_Program_Read)
  List_Counter=0
  nCharacters = AvailableProgramOutput(Program_code)
  While nCharacters
    List_Counter+List_Counter+1
    Keyword_List.s(List_Counter)= ReadProgramString(Program_code)
    nCharacters = AvailableProgramOutput(Program_code)
  Wend
  SetGadgetText(Text_0,Exiftool$+Commandline$+"  "+Str(Input_Level))
EndProcedure

[Phil Harvey]

Hi John,

To write the output out a file, do this:

exiftool DIR -subject -b -sep "\n" -sep "\n" | sort | uniq > out.txt

- Phil

[StarGeek]

The command sequence you provided works for me in Windows except for the | uniq command.  I get a full listing.  I'll either see if there is a windows equivalent, or as I'm calling this programmatically, I can remove duplicates it program which is invoking Exiftool.

There are Windows ports of Linux commands, which is useful as the Linux commands are very powerful and it's easy to search StackExchange to figure out how to do specific things.  I use the MSys2 ports.

However, when I call the above programmatically, then the listing goes straight to the CMD window, hence the request in the previous post to learn if there is a way to get the output to go to file.

Phil's answer uses the command lines file redirection ability to pipe the output into a file.  This is often unavailable when calling from a program.  In such cases, there should be a way to capture the output what your programming language's system call.

[Phil Harvey]

Phil's answer uses the command lines file redirection ability to pipe the output into a file.  This is often unavailable when calling from a program.

This is true since it requires that the command is executed by a shell processor.  But I think it should work here because the pipe to sort (| sort) works.

- Phil

[JohnF]

Hello Phil and StarGeek,

Once again - your suggestions got me the answers needed. 

The syntax to recursively find, sort and list unique subject entries in Windows is

exiftool -r DIR -subject -b -sep "\n" -sep "\n" | sort /unique > out.txt

Because of the piping this creates, I'm required to execute cmd from my calling program and pass the run string

/c exiftool -r DIR -subject -b -sep "\n" -sep "\n" | sort /unique > out.txt

I also ensure that calling program statement is set to wait for execution to complete.  Then I launch notepad, specifying to load out.txt.

Attached is a screenshot of the final program which allows me to search and copy all specified images. 

[Phil Harvey]

sort /unique

Good to know.

- Phil

[JohnF]

Hello Phil and StarGeek,

With good success in creating the search and copy program in Windows, I'm now working to get achieve the same on macOS (10.13.6 - High Sierra).  I'm currently working from the Terminal window and using this command syntax

exiftool -r "/Volumes/Data/NBF_Photo_Library/" –if "$Keywords=~/John/" –o "/Volumes/Data/2-Star"

In the Terminal window you can see that each image is read, so the first portion of the command is working well.  The problem is that no files are written to the output directory.  I also noticed at the end of the processing I get the following which seems to indicate that each entry beyond the source directory is also treated as a source file. Any suggestions?

Thanks,
John

Code Select Expand

Error: File not found - –if
Error: File not found -  >= 0
Error: File not found - –o
  263 directories scanned
3507 image files read
    3 files could not be read


[Phil Harvey]

You need to use the standard hyphen (not some probably-pasted special hyphen-looking Unicode character).

Also, read my signature.

- Phil

[JohnF]

yes, I just noticed the single quote comment. I'll but both together and give it a whirl.

[JohnF]

Ah... I should NEVER try to compose the command line string in WORD which leads to non-standard characters. And, of course, if I could only read just a bit further and pick up on footer, then things would go much smoother.

I'm up and running now!

[JohnF]

Hi Phil and StarGeek,

Now that I can at least get a command to run in Terminal, I've returned to calling it programmatically.

I have two cases where I call exiftool.  Below are the code snippets for the Windows implementation, which work as expected. 

The first case (creating a comprehensive list of keywords) includes sorting, reducing the list to unique entries, and outputting to a file.  Because this is piped it required that I execute it by first calling "cmd /c".  Here's the working Windows code.

Code Select Expand

  Workingdirectory$=""
  Commandline$ = " /c exiftool -r " + Chr(34) + Input_Directory$ + Chr(34) + " -Keywords -b -sep " + Chr(34) + "\n" + Chr(34) + " -sep " + Chr(34) + "\n" + Chr(34) + " | sort /unique > " + Chr(34) + Output_Directory$ + "Keyword Listing.txt" + Chr(34)
  Result = RunProgram("cmd",Commandline$,Workingdirectory$,#PB_Program_Hide|#PB_Program_Wait)

The second case (selectively copying images to an output folder) was achieved by invoking exiftool directly. Here's the working Windows code:

Code Select Expand

  Exiftool$ = "exiftool"
  Workingdirectory$=""
  Commandline$ = " -r " + Chr(34) + Input_Directory$ + Chr(34) + " -if " + Chr(34) + "$rating >="+Rating$ + Keyword_Filter$ + Chr(34) + " -o " + Chr(34) + Output_Directory$ + Output_String$ + Chr(34)
  Result=RunProgram(Exiftool$,Commandline$,Workingdirectory$,#PB_Program_Hide | #PB_Program_Wait)

Both work just as they should in Windows.

So I'm 100% new to the mac world.  I've scoured the internet, looking also for python equivalent examples, etc. and seen some posts on this site too.  I've believe I've accounted for special characters and have substituted with Char(39) to use a single quote where needed (though not reflected in the original working Windows posted above).  It seems that my difficulties are more fundamental and that exiftool is never really even called.  I've experimented with using either "open", "sh" or "bash" in the same fashion as "cmd /c" but as of yet, no luck.  As mentioned in my previous post, when directly entering a good command string in to a Terminal window, things are good.

On the Mac, how do I call exiftool with all the associated syntax for these two cases - one to mimic using "cmd /c" as the principle program called with exiftool and its syntax passed as the argument and the other to call exiftool and its argument directly?  As you can see my calling program (PureBasic) is flexible - it allows me to specify the command, command line string, and working directory. In addition, I can stipulate if I wish to have it execute hidden, to hold execution, etc..

An answer to either would be great, both would be fantastic.  Do you have a tutorial on this?

Many, many thanks!
John

[Phil Harvey]

Hi John,

On Mac, your command should probably start with this to run exiftool:

/usr/local/bin/exiftool ...

Because the PATH may not include /usr/local/bin in whatever environment is used when you execute a command from PureBasic.  If this doesn't work, you could resort to:

/usr/bin/perl /usr/local/bin/exiftool ...

But this shouldn't be necessary.

- Phil

[JohnF]

Hi Phil,

I've added the path so that I'm now calling /usr/local/bin/exiftool.  (I've also checked that exiftool can be found there - and it is).  Regrettably, this does not solve seem to work either.

I've also reached out to the PureBasic community to see what I can learn there.

It seems that Mac world has additional complexity.  I'm guessing it's just a matter of finding the right calling approach and syntax - but getting there is not intuitive.

What I can say, is that adding GUI wrapper, as I've now nicely achieved in Windows, means I can now post my 3500 image library for my extended family, and they can search for exactly the pictures they want to see, all thanks to exiftool, without every having to worry about syntax and command line operation.  The problem is our extended family is divided between Windows and Mac, so I'm trying to address both.

I would think that there are many exiftool users that would put GUI wrappers in place - which would mean easy access to examples as to how this can be accomplished.  It would be a good area to add to the forum.

Any additional thoughts would be appreciated!

Best,
John

[Phil Harvey]

Hi John,

I doubt this is an ExifTool-specific problem.  I think you need to figure out how to execute any command-line utility on a Mac from PureBasic.  Your post to the PureBasic community is the way to go.

There are many programming resources linked from the ExifTool home page, but nothing specific to PureBasic.

- Phil

[JohnF]

Hi Phil,

I'm in full agreement.  Last night I created my own little command-line program which pops up a GUI and displays the command-line arguments passed to it.  With this I began to test PureBasic's actually support for calling any external .exe.  What I found is just what you've suggested - the problems I'm running in to are do to defects in PureBasic..  I'll be reaching back to their develop team with a bug/enhancement report. 

I'm delighted that I have things working well in the Windows world, and hopefully in the future this will be true for MacOS as well.

As a follow-up I plan to put a set of "How-To" notes together that I believe would be a help to others who wish  to create a simple GUI to invoke Exiftool.  I'll pass it on to you.  It's the least I can do as thanks for your support and quick replies!

Best,
John

PS.  Your tip that the full path is required to execute any .exe in MacOS gave me an important key towards meaningful troubleshooting on the Mac side. Thanks!