Slow file copy with "exiftool -directory" on some servers

[Matt Q]

First, a bit of background: we use the command-line exiftool under MacOSX as part of a shell script to organize and update the metadata on batches of images for a digital library.

Recently, execution of the shell script has slowed down to a crawl.  Execution time has ballooned from 15-20 minutes to 6-8 hours.  I added some timestamps to the script log and found that the choke point occurs when the script executes this command:


exiftool -directory=$AutoTemp/NewMasters/ -ext .tif -preserve $SourceDirectory -r -v1


Now for the strange stuff: this ONLY occurs when the destination $AutoTemp is on a cifs: or smb: shared drive.  Checking the Activity Monitor, I can see that the data transfer rate is about 1mb/sec.  To a network drive connected via afp: it will run at full speed.  Later in the script there is a cp command that executes at full speed on all three types of network mounts.

I suspect the problem is with the server configuration rather than with exiftool itself, but I need more information about what exactly is going on so that I can tell the server admins what needs to be changed in their configuration.

[Phil Harvey]

When you set the directory of a file with ExifTool, first it tries to move the file by renaming (the standard C "rename" library call").  If this call fails (which is expected when attempting to move files across filesystems), then ExifTool reverts to copying the file in 64kB blocks to the new directory.  If the copy is successful, then the original file is deleted.

- Phil

[Matt Q]

That makes sense.  Do you know if there is an easy way to boost the block size?  That may provide a workaround if we can't figure out how to get the SMB and CIFS connections to run more efficiently.

[Phil Harvey]

You can tweak this by changing the buffer size at line 1634 in lib/Image/ExifTool/Writer.pl:

        while (read EXIFTOOL_SFN_IN, $buff, 65536) {

Let me know if you get any significant performance gains by doing this.

- Phil

[Matt Q]

Thanks for the quick response!  Sadly, it doesn't seem to make a difference.  I tested 1 MB, 10 MB and even 30 MB (the images are around 25 MB each) and the run time barely changed.  For some reason the CIFS and SMB servers are handling the write requests at around 1 MB / sec, no matter how big the buffer.