Strange performance metrics running on directory vs files

[sn0cr4sh]

I am trying to find the absolute fastest way to extract XMP files from RAW files, and have been experimenting with a couple of different methods. I'm a little confused on the results, and wondering if I'm not using the right command.  My testing is against 500 RAW files, output to XMP files.  This is done on OSX 10.10

Method 1: Exiftool run against the directory as one command
exiftool -o %d%f.xmp dir
It takes about 3:20 to complete

Method 2: ExifTool run on each individual file from a shell script
I wrote a script that took each individual file and instructed ExifTool to output the XMP using -b.  It was done in a loop, piping the output to an SH script. So its one command per file. That script was run through the Terminal
exiftool -xmp -b "$f" > "$xmpfile" >>myscript.sh
sh myscript.sh
This took 2:20 to complete.  A full minute faster.

Is there some kind of performance boost that I am getting by running it through a shell script like this?  Or am I using the wrong command in Method 1?

[Phil Harvey]

Method 1 is generating XMP from all available tags.

Method 2 is extracting only the XMP block.

Try using the same technique for both methods.

- Phil

Edit:  I tried your first command on my Mid 2011 iMac (2.7 GHz Intel Core i5), and it averaged 5 raw files per second on a sample of 250 raw files of various types, which is about twice what you are getting.  If I add the -fast2 option to omit the makernote information, this doubles to 10/sec.

[sn0cr4sh]

Ok, I think I understand. I wasn't comparing the same things.

If what I want is to dump ALL available tags into an XMP file, then Method 1 is one way to do it.  But, I can also do it with -tagsfromfile from my understanding.  I ran this command:

exiftool -tagsfromfile *.cr2 -o %d%f.xmp

This process took about 10 seconds, total.  The XMP files look like they have all the tags I was expecting - though in a different format. I tried the -X option, but I think its already in RDF format.

So would this method be the best for what I want to accomplish?

*Edit:  By the way, when I run that command, it skips the very first file in the directory.  I've tried it now a few times on different groups of files.  Each time it skips the first source file. Am I doing this incorrectly, or could that be a bug?  I'm on 9.94.

[Phil Harvey]

If what I want is to dump ALL available tags into an XMP file, then Method 1 is one way to do it.  But, I can also do it with -tagsfromfile from my understanding.

Yes.  The -o option is equivalent to using -tagsFromFile when writing to a file of a different format...

exiftool -tagsfromfile *.cr2 -o %d%f.xmp

...but that command won't do what you expect.  You shouldn't use a wildcard in the -tagsFromFile argument.  This is why you have the funny problem of the first file being skipped (because it is the -tagsFromFile argument, and the rest are taken as source files).

See examples 11 and 13 on the Metadata Sidecar Files page for exactly what you are trying to do.  The only difference between these two commands is that 11 will fail if the output XMP file already exists, while 13 will add the new metadata to an existing XMP file.

Also, as I mentioned, you can add the -fast2 option to speed things up significantly, but the MakerNote information won't be copied.

- Phil

[sn0cr4sh]

Ah, I forgot about that page. I kept looking in the Reading / Writing section.  Got it now.  I figured something was wrong when it dumped all those XMPs in 10 seconds. It certainly looked like it was working. 

Example 13 puts it back in that 3:20 range, and then -fast2 cut the time in half to about 1:40.  So yeah, that's a big change.