Importing image files from memory cards

[Andreas Spindler]

Hi, this is my first post here. 

Today I tried to write a script that would import image files from an SD card to a preview directory. The solution finally was

    function import() {
                local dcimdir="$1" previewdir="${2:-.}"
                exiftool -r -P '-FileName<DateTimeOriginal' \
                    -o dummy/ \
                    -d "$previewdir/%Y-%m-%d/%%f%%-c.%%le" \
                    "$dcimdir/" || panic
    }

Usage:

    import /cygdrive/f/DCIM "/cygdrive/h/foto/Speicherkarten/"

I post this here to share and validate this code. Is there a better way to do it? Have I understood the interface?

What I discovered so far is that the dummy-directory is necessary to let exiftool copy files. Without -o it would delete the original files on the card simply because no tag is set that names metadata. "Directory", "FileName" and "FileModifyDate" are the only tags that do not refer to metadata, and since I'm using only "FileName", exiftool deletes the source files.

In the above function exiftool derives the filename from "DateTimeOriginal". What this actually means is: it applies the format string passed to -d to the filename. "DateTimeOriginal" is just a data structure and by itself cannot be a filename. Right? But since the format-string contains a directory exiftool also sets the directory component. If there would be an additional "Directory" tag exiftool might do something else... will it use the "Directory" directory and just the "FileName" filename? I don't know.

However, since the "FileName" tag has precedence over -o the dummy-directory is never used. On the other hand it is required because otherwise exiftool runs in "delete mode". Right?

Ok. This seems to be the idiom - without alternatives - to rename a file into a different directory. It is explained here (https://exiftool.org/filename.html) as example 5 and here http://cpanforum.com/threads/6699.

While I developed the command-line  I was lucky that exiftool hadn't already removed files from my SD-card. Alas, there's no --dry-run option so I had a hard time figuring the command-line out. Simply because I ^C'd it before it was done (my 600X Thinkpad has only USB 1.1 and is slow) I prevented exiftool from killing precious photos from the Leonard Cohen Open-air in Wiesbaden last friday during my experiments...

A --copy boolean option would be nice to make things explicit, rather than having to hack a dummy-directory into the command-line that is then never used anyway. And, exiftool should  generally fail when multiple target directories are specified on the command-line, rather than relying on the precedence rules. A global --relax boolean option could then make it less pedantic (restoring the current behaviors).

For humans all the rules exiftool applies are difficult: the correlations between "Directory", "FileName" and real metadata tags in conjunction with 4 precedence rules. You are allowed to specify multipe target directories on the command-line. It should not allow that. I think people will rather try to google some examples that happen to work already. Well, Phil, you've a higher view on the use cases for exiftool. From my perspective the mentioned options would have helped a lot in this case.

Thanks for your attention,
Andreas

[Phil Harvey]

Hi Andreas,

Thanks for your comments.

I post this here to share and validate this code. Is there a better way to do it? Have I understood the interface?

This looks good.  You seem to have a good grasp of the command.

In the above function exiftool derives the filename from "DateTimeOriginal". What this actually means is: it applies the format string passed to -d to the filename. "DateTimeOriginal" is just a data structure and by itself cannot be a filename. Right?

DateTimeOriginal is a string, which can be assigned to a file name.  As can any tag, which are all ultimately converted to strings.  The -d option just affects how the conversion is done.

But since the format-string contains a directory exiftool also sets the directory component. If there would be an additional "Directory" tag exiftool might do something else... will it use the "Directory" directory and just the "FileName" filename? I don't know.

This gets a bit complicated, but is explained in the last example on the Writing "FileName" and "Directory" tags page.

However, since the "FileName" tag has precedence over -o the dummy-directory is never used. On the other hand it is required because otherwise exiftool runs in "delete mode". Right?

Not exactly.  ExifTool always preserves the original file by default.  There is only one tricky exception to this rule, and you have found it.  Normally writing the FileName tag simply causes the file to be renamed.  On many filesystems you can rename a file to move it to a new directory.  The original file is not deleted, it is just moved to a different location.  So this is what exiftool does.

The problem occurs on filesystems where you can't rename the file as specified to the new directory.  In this case, to maintain the same behaviour as when the file can be renamed, it is copied to the new location and deleted from the original directory (only if the copy was successful).  This is the only case where exiftool will delete a file without specifying the -overwrite_original option.  But it is necessary to maintain a consistent behaviour.  Essentially, from a user's perspective, this is the same as moving the file.

Simply because I ^C'd it before it was done (my 600X Thinkpad has only USB 1.1 and is slow) I prevented exiftool from killing precious photos from the Leonard Cohen Open-air in Wiesbaden last friday during my experiments...

ExifTool will never kill a file.  It will only delete the file from the memory card if it was successfully copied to the hard disk.  So your files would still be there.

A --copy boolean option would be nice to make things explicit, rather than having to hack a dummy-directory into the command-line that is then never used anyway.

I try to prevent adding new options because nobody has the time to read through the existing options anyway.

And, exiftool should  generally fail when multiple target directories are specified on the command-line, rather than relying on the precedence rules.

The precedence rules come in very handy when performing some (admittedly uncommon) operations.

A global --relax boolean option could then make it less pedantic (restoring the current behaviors).

Another unnecessary option.

For humans all the rules exiftool applies are difficult: the correlations between "Directory", "FileName" and real metadata tags in conjunction with 4 precedence rules. You are allowed to specify multipe target directories on the command-line. It should not allow that.

Here is an example of why it should:

exiftool '-directory<lensid' '-filename<Unknown/$focallength/%f.%e' FILE ...

This command sorts images by LensID and FocalLength.  In the case where the image doesn't contain a LensID, it placees it in an "Unknown" category.  There are many valid reasons like this why someone would want to set multiple target directory names.

- Phil

[Andreas Spindler]

Thanks for your answer, Phil.

DateTimeOriginal is a string, which can be assigned to a file name.  As can any tag, which are all ultimately converted to strings.  The -d option just affects how the conversion is done.

Yes, but the string usually contains characters valid only under UNIX. UNIX prohibits only the null character in filenames, since it's the C string terminator. Windows disallows ':', for example, which may occur in time strings. Therefore I think -d is always required.

ExifTool will never kill a file.  It will only delete the file from the memory card if it was successfully copied to the hard disk.  So your files would still be there.

I was just experimenting yet. Once I'd understood that it moves files I immediately turned the write lock switch on the SD card on. That exiftool deletes files on the SD card makes no sense to me.

I try to prevent adding new options because nobody has the time to read through the existing options anyway.

Interesting.

It might be true for quick hacks. But once you discover how powerful and unique exiftool is you'll have the time...

The precedence rules come in very handy when performing some (admittedly uncommon) operations.
...
exiftool '-directory<lensid' '-filename<Unknown/$focallength/%f.%e' FILE ...

Thanks, I wasn't aware of that. The regular scenario in which exiftool is used seems to be: go into the image directory, then run exiftool there to move/rename/retag images. But when source/target directories are on different branches you usually do not want that files are moved.

Unrelated directories occur with the find utility. find is UNIX' way of doing directory traversal. It is used quite often, because most UNIX programs simply do not understand directory structures and expect pan-ready filenames. find can select files subtly by searching the file's name and type, permissions, the directory where files are stored and the file system type. It is perfect for feeding images into exiftool. I wonder how find and exiftool fit together?

For example, one could use find to preview image files shot in the last 2 days on the big screen of a laptop:

    
    function import_jpegs_shot_recently() {
        local indir="$1" previewdir="${2:-$TMPDIR/preview$$}"
        echo "The temporary directory is '$previewdir'"
        find "$indir" -mtime +2 -type f \( -iname '*.jpg' -o -iname '*.jpeg' \) -print0 | \
            xargs -0 \
                exiftool -P '-FileName<DateTimeOriginal' \
                    -o dummy/ \
                    -d "$previewdir/%m-%d %H.%M/%%f.%%le"
    }

The intent is to let exiftool rename the files by precise shooting time (e.g. day, hour, minute) into a temporary directory. Of course this command is not limited to memory cards; it can shake recently edited files out of NAS directories, or the "My Documents" hairball of files under Windows etc.

If you forget to use the -o dummy/ idiom, however, some Jpegs disappear from the source directory. Consider RAW+JPEG shooting mode: the RAWs would suddenly become orphans on the card. A --dry-run option would be handy to develop such a command.

Note that the above function requires GNU find and xargs. It is efficient because it will use the maximum number of file-names it can pass to exiftool in a shell command on this system, so exiftool is not called for every file. Finally, another advantage is that multiple exiftools can be called parallely, making use of multicore CPUs. This means find+xargs+exiftool may process a huge number of files much faster than exiftool alone. See the -P switch on the xargs man-page.

Thanks again, Andreas