Renaming images and sidecar together, but using names depending on camera model

[Tillomar]

Hi,

I need some help, please.

I try to rename my images to a new name schema. (I will feed them to Lightroom afterwards.) Currently, I have to rename all 'old' images, but later on, I will do this whenever I copy new images from the camera to the storage. So I need a batch to convert all existing files, and then maybe another batch for regular renaming of new files within a single folder.

Picking the right pieces from the manual and some postings here, I came up with:

exiftool -r -P -tagsfromfile "%%d%%f.nef" "-FileName<${CreateDate}_D300-${SerialNumber}-${ShutterCount}_DSC-%%4.4f.%%le" -d %%Y%%m%%d_%%H%%M%%S -ext xmp .
exiftool -r -P -tagsfromfile "%%d%%f.nef" "-FileName<${CreateDate}_D300-${SerialNumber}-${ShutterCount}_DSC-%%4.4f.%%le" -d %%Y%%m%%d_%%H%%M%%S -ext jpg .
exiftool -if "$model =~ /NIKON D300/i" -r -P "-FileName<${CreateDate}_D300-${SerialNumber}-${ShutterCount}_DSC-%%4.4f.%%le" -d %%Y%%m%%d_%%H%%M%%S -ext nef .

The schema "invented" is:
- first a CreateDate field
- second a camera identification field -- here: camera make + camera serial + shutter count
- last the tag "DSC-" followed by the original image number given by the camera

These calls should rename all Nikon raw images, and jpgs and xmp sidecar files along with them if they exist. This works correctly if all files are from the D300 camera. But I wish to insert the "D300..." name tag only if the images really came from that camera; the third call for renaming the raw file does that correctly using a condition. But I'm unable to do that check on the raw image when renaming the accompanying files -- because the check will address the file to rename, not the origin of the tags when there is a different file specified by -tagsfromfile.

But this isn't the end of the story already, because there are more requirements:

1. As my wife, my daughter and I (is it "I" or "me"??) use different cameras, I would like to add an appropriate camera identification tag to each file -- as shown above for the Nikon. Later, I could use different batches with hard-coded tags for different camera models if neccessary -- that would just be a little inconvenient --, but it's no solution for the reorganization batch.

2. It might be possible that later there are two different cameras of the same model; so it seems wise to include the camera model with the camera serial number to form a unique key; the camera model is then for the human reader, mostly. But... there might be cameras without a serial number!?

3. I want to use the shutter count from my Nikon cameras, because it gives me a sequence even if the time/date setting or the image numbering gets disturbed (has happend already when I lend my camera to the neigbour...). But none of the other cameras in the familiy provide a shutter count so far. So the renaming should take care of different camera makes and use appropriate name schemas. 

These points 1 to 3 together require me to distigush the image's camera, then chose the appropriate exiftool commands for building the desired camera identification tag which may differ between cameras (makes). 

4. If the are "lone" files to rename, without the "master" raw file to provide the exif information, then the renaming should be done based on the exif information within these files. (This would be standard behavior for all cameras which do not produce raw files.)

5. But if a file has exif information embedded AND a "master" raw file with conflicting exif information, then this file should not be renamed. (I would rename those files manually after checking what's wrong with the embedded information.) For raw files I would always expect the exif information (if present) to be correct.

6. If those batches accidentally run into files already renamed, they shouldn't rename them once more. More specific, only files conforming to the respective camera's naming scheme should be renamed; but for exiftool to work recursivly I have to specify a starting folder, so I cannot use a file mask (other then specifying the file extension).

I would very much appreciate some help on this as it seems rather complicated, and I'm no scripting guru. I'm on Windows (7), so it would be nice for any hint to keep in mind what's available on Windows.

Thanks,
Tillomar

[Alan Clifford]

Have a look at

-@ ARGFILE                       Read command-line arguments from file

in the documentation.  Then you can put all the arguments in an arg file rather than on the command line itself.  You could put in multiple "ifs" to detect the cameras.

The serial number might give you rather long file names so it might be a bit of a non-starter

Code Select Expand

exiftool -*serial* -model DSCF2056.JPG
Internal Serial Number          : FC  A5622285     592D32353433 2009:09:28 C7833022BDFE
Camera Model Name               : FinePix F200EXR


Take your programming a bit at a time.  The conditional stuff for sidecar files could be done logically in perl but looks a bit complicated for the command line.  Leave that until last.

[Phil Harvey]

Hi Tillomar,

Alan gives some good advice.  I'll add my two cents.

If you can identify a camera uniquely based on any metadata, you can give it a custom name with a user-defined tag, with a config file something like this:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Composite' => {
        MyID => {
            Desire => {
                0 => 'Model',
                1 => 'SerialNumber',
                2 => 'FirmwareVersion',
                3 => 'ShutterCount',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                return undef unless $val[0];
                if ($val[0] =~ /D300/ and $val[3]) {
                    return "MyD300_$val[3]" if $val[1] eq '12345';
                    return "WifesD300_$val[3]" if $val[1] eq '67890';
                    return "OtherD300_$val[3]";
                }
                return "HS50-1" if $val[0] =~ /HS50/ and $val[2] and $val[2] =~ /1\.01/;
                return "HS50-2" if $val[0] =~ /HS50/ and $val[2] and $val[2] =~ /1\.20/;
                # ...
            },
        },
    },
);
1;  #end

And an argfile something like this:

Code Select Expand

-if
length($filename) < 13
-r
-P
-tagsfromfile
%d%f.nef
-FileName<${CreateDate}_${MyID}_DSC-%4.4f.%le
-d
%Y%m%d_%H%M%S
-ext
xmp
-ext
jpg
-execute
-if
length($filename) < 13
-r
-P
-FileName<${CreateDate}_${MyID}_DSC-%4.4f.%le
-d
%Y%m%d_%H%M%S
-ext
nef
-execute
-if
length($filename) < 13
-r
-P
-tagsfromfile
%d%f.jpg
-FileName<${CreateDate}_${MyID}_DSC-%4.4f.%le
-d
%Y%m%d_%H%M%S
-ext
xmp
-execute
-if
length($filename) < 13
-r
-P
-FileName<${CreateDate}_${MyID}_DSC-%4.4f.%le
-d
%Y%m%d_%H%M%S
-ext
jpg

Here I am only renaming files with names less than 13 characters to exclude files that have already been renamed.  Unfortunately the renaming for each file type has to be split into 2 separate commands (separated by -execute) to ensure that the file we are getting the metadata from gets renamed last.

This addresses many of your requirements, but doesn't deal with the problem of conflicting metadata for example.  However, I'm hoping that my example may give you a place to start.

- Phil

[Tillomar]

Phil, Allan, thanks for your comments.

The example serial number given by Allan is indeed unfit for makeing the filename; but if I include it into a user defined tag as Phil suggests, I won't have to use it verbatim or even just in parts, it could simply be part of the base data for selecting a camera "name".

About this problem:

But I'm unable to do that check on the raw image when renaming the accompanying files -- because the check will address the file to rename, not the origin of the tags when there is a different file specified by -tagsfromfile.

Do I understand your solution right:

The configuration file defines a user-defined tag, which will be calculated on-the-fly whenever a file is accessed for a tag. So, when I access the NEF file through -tagsfromfile for this tag, it may either have a special value as calculated by the tag's definition -- in which case this value would be used for renaming -- or it won't be there if any of the preconditions stated in the definition is not met, in which case the renaming will not happen, leaving the file untouched?

The analysis of the external parameter file shows that my own (modified) rename commands (which you compressed into two executes using multiple -ext) were complemented with a rename taking care of xmp sidecar files which could not be renamed before because of a missing NEF file, but with a same-named jpg file available; and another rename command which will finaly rename any remaining jpg file as long as they are providing the compound tag.

As far as I can see, this will indeed do everything except the conflict resolution. Maybe it's best to check if this is necessary at all by reading all relevant exif info into one big text file, and then doing some sorting and comparing...

What I do not understand satisfactorily is the code defining the compound tag. As advised in the config file from the source package, I took into Exif.pm and README, and got steamrolled. So I beg your apology for asking some questions which may possibly answered easily from the documentation...

1. Could this:

        MyID => {
            Desire => {
                0 => 'Model',
                1 => 'SerialNumber',
                2 => 'FirmwareVersion',
                3 => 'ShutterCount',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                return undef unless $val[0];
                ...


have been replaced by

        MyID => {
            Require => {
                0 => 'Model',
            }
            Desire => {
                1 => 'SerialNumber',
                2 => 'FirmwareVersion',
                3 => 'ShutterCount',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                # return undef unless $val[0]; -- not necessary anymore
                ...


...or is this a different semantic?

What is the meaning of the closing "i" in the -if expression:

"$model =~ /NIKON D300/i"  ?

This code (found in the example config)

ValueConv => '$val[0] =~ /(.*)\./ ? $1 : $val[0]',

looks vaguely like an assignment from a C style conditional expression, but also bear some similarities to the expression above. When is it an assignment, and when is it an relational expression?

This kind of expression

/    ...some text...    /

seems to be a string. If so, the expression from the config:

/(.*)\./

may be a RE, and the following $1 may refer to the first RE group. It looks for the longest string which is closed by a dot... but where does it look?

If I want to do some more formatting on the tags when they are included into my compound tag, how do I have to use this syntax:

return "MyD300_$val[3]" if $val[1] eq '12345';

together with this (just some examples):

ValueConv => 'sqrt(24*24+36*36) / ($val * 1750)',
PrintConv => 'sprintf("%.3f mm",$val)',  ?

From the style of it, it seem as if some code (would be PERL, then) is included from a configuration file and then executed as part of the tool.
Do I have to learn PERL to understand this?

Thanks again,
Tillomar

[Tillomar]

And another question, coming up while I collect information to distinguish different cameras:
In Nikon cameras, there seem to be two (different?) "SerialNumber" tags -- 0x001d and 0x00a0. What's the difference, and is there a way to access these tags through their ID?

Tnx
Tillomar

[Phil Harvey]

or it won't be there if any of the preconditions stated in the definition is not met, in which case the renaming will not happen, leaving the file untouched?

Right.

1. Could this:

        MyID => {
            Desire => {
                0 => 'Model',
                1 => 'SerialNumber',
                2 => 'FirmwareVersion',
                3 => 'ShutterCount',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                return undef unless $val[0];
                ...


have been replaced by

        MyID => {
            Require => {
                0 => 'Model',
            }
            Desire => {
                1 => 'SerialNumber',
                2 => 'FirmwareVersion',
                3 => 'ShutterCount',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                # return undef unless $val[0]; -- not necessary anymore
                ...

Good point.  Yes.  You understand well.

What is the meaning of the closing "i" in the -if expression:

"$model =~ /NIKON D300/i"  ?

i = Case-insensitivity.

This code (found in the example config)

ValueConv => '$val[0] =~ /(.*)\./ ? $1 : $val[0]',

looks vaguely like an assignment from a C style conditional expression, but also bear some similarities to the expression above. When is it an assignment, and when is it an relational expression?

It is indeed a conditional expression.  With no "return", the value of the last expression is returned.  In this case, the "$1" is returned if the match is a success, otherwise the original value is returned.  Google "Perl regular expressions" if you want to know what "$1" represents (and the "i" for case insensitivity too).

Do I have to learn PERL to understand this?

Yes.

- Phil

[Tillomar]

Ok, I'm getting closer...

This is my code to build the CameraID tag for the D300's:

Code Select Expand


    'Image::ExifTool::Composite' => {
        CameraID => {
            Require => {
                0 => 'Model',
            },
            Desire => {
                1 => 'SerialNumber',
                2 => 'ShutterCount',
                3 => 'FirmwareRevision',
            },
            # remove characters that are illegal in Windows file names
            ValueConv => q{
                if ($val[0] =~ /NIKON D300/ and $val[1] and $val[2]) {
                    $val[2] = sprintf("%06.0f",$val[2]);
                    if ($val[1] eq '4148781') {
                        return "D300-MYNAME-$val[2]";
                    }
                    if ($val[1] eq '4158022') {
                        return "D300-MYWIFE-$val[2]";
                    }
                    return "D300-$val[1]-$val[2]";                   
                }


It works, but is this the way the formatting of the ShutterCount value should be done? Redefining $val[2] seems a bit awkward.

This code:

Code Select Expand


                if ($val[0] =~ /Canon DIGITAL IXUS 970 IS/) {
                    return "IX970-MYWIFE" if $val[3] =~ /1\.00 rev 3\.00/i;
                    return "IX970-UNKNOWN";
                }


should recognize my wife's Canon Ixus based on it's firmware revision, but insists on stating "IX970-UNKNOWN" even when exiftool shows FirmwareRevision : 1.00 rev 3.00. I also tried the RegEx with \s to match the whitespace, but without success. What am I doing wrong?

Thanks,
Tillomar

[Phil Harvey]

Hi Tillomar,

It works, but is this the way the formatting of the ShutterCount value should be done? Redefining $val[2] seems a bit awkward.

Instead of redefining $val[2], you should use a local variable:

  my $var = sprintf("%06.0f",$val[2]);

Then use $var in your expressions.

insists on stating "IX970-UNKNOWN" even when exiftool shows FirmwareRevision : 1.00 rev 3.00. I also tried the RegEx with \s to match the whitespace, but without success. What am I doing wrong?

The ValueConv $val[] variables are the numerical values, the same as you see with the -n option.

- Phil

P.S.  You can delete the "remove characters" comment that I accidentally left in my example.

[Tillomar]

Hi Phil,

Instead of redefining $val[2], you should use a local variable:

my $var = sprintf("%06.0f",$val[2]);

Then use $var in your expressions.

I see... I already tried that, but did't know to use the keyword my.

The ValueConv $val[] variables are the numerical values, the same as you see with the -n option.

I now use return "IX970-MYWIFE" if $val[3] eq 16777984;, and that works.
But I also compare $val[0] against some strings, and that works also... so when do I have to expect strings, and when do I have to expect numerical values? (...if there is an easy way to answer that.

Thanks,
Tillomar

[Phil Harvey]

Hi Tilomar,

I now use return "IX970-MYWIFE" if $val[3] eq 16777984;, and that works.
But I also compare $val[0] against some strings, and that works also... so when do I have to expect strings, and when do I have to expect numerical values? (...if there is an easy way to answer that.

If you want to compare the strings, use the $prt[] array instead of the $val[] array and you don't have to worry.  The only way to determine if the -n value is different is to try it (or look at the ExifTool code), because this depends entirely on how the metadata is stored.

One Perl pointer:  "eq" is a string comparison, which will work OK here because you probably don't have to worry about leading zeros, but since you are comparing numbers you could also use "==" (the numerical comparison).

- Phil

[Tillomar]

Thanks Phil,

I modified my config accordingly and added some other commands to the parameter file, and then did a lot of testing. It works like a charm, and I've already renamed about 35000 images... So this is the right time to say: Thank you, Phil, very well done -- it's really, really helpful!

But... (as always )
There has been some manual rework, though, -- which I expected -- and reviewing the real-live results, I realize that it would be a good thing to get more control over which files I want to ignore; currently, there is only the condition that a file name's length must be shorter then a certain constant (which is ok for excluding a file from the next run...). Is there a way to implement more detailed analysis of a file -- possibly not only regarding the file name, but perhaps other attributes?

For the filename, I would like to do some (multiple) pattern matching, and on each match I can then decide -- perhaps through looking on what has been captured -- if the file should be ignored. I suppose this should be possible by implementing another compound tag -- which could, depending on the implementation, end up being being 'undef' -- but I don't know how to use this to ignore the file. Could this tag deliver 'undef' (=ignore) or "" (go on) which can then be used in the renaming without doing harm? Or could this be done much simpler within the -if conditional?

Another problem that I encounterd is that the original NEF (...) file not always has the same name, so using -tagsfromfile %d%f.nef won't help -- for example, a file has some extension to the filename: DSC_0013062_extr.jpg (which would be an automatically extracted preview image from the NEF). Is there an easy way to build possible tag source file names from the file to rename?

Thanks,
Tillomar

[Phil Harvey]

Hi Tillomar,

Is there a way to implement more detailed analysis of a file -- possibly not only regarding the file name, but perhaps other attributes?

Absolutely.  The -if expression may access any information in the file and you may use the full power of Perl to do whatever logic you want.  If the expression returns false (empty string, "0" or undef), then the file is not processed.

Another problem that I encounterd is that the original NEF (...) file not always has the same name, so using -tagsfromfile %d%f.nef won't help -- for example, a file has some extension to the filename: DSC_0013062_extr.jpg (which would be an automatically extracted preview image from the NEF). Is there an easy way to build possible tag source file names from the file to rename?

ExifTool's file name format string is very flexible.  Read the -w section of the application documentation for details.

- Phil

[Tillomar]

Hi Phil,

as a solution for the first problem I've started with

Code Select Expand

$filename =~ /\d{8}_\d{6}.*\./ ? undef : $filename

which works well.
If I want to do multiple checks -- is there a way to do it sequentially rather then recursive?
Example for what I mean with recursive:

Code Select Expand

$filename =~ /\d{8}_\d{6}.*\./ ? undef : $filename =~ /^(DSC|IMG|PICT?)[-_]\d{4,8}\./i ? : $filename =~ /^\d{12}\./

This would be very difficult to read, so I would like to write it as a sequence of if (...) return ....

For the second problem -- building a more sophisticated filename to address the file from which I want to extract tag values -- I haven't found a solution so far. The main reason is that the argument of -tagsfromfile is a format string, which accepts only print format fields. But I need to analyze the filename of the file in work to build a master file name. Example:

Those are the master files:

DSC_0063.raw
DSC_00020397.raw
DSC_0261.tif

Those are the files which are looking for masters:

DSC_0063_extr.jpg
DSC_0063_extr_crop.jpg
DSC_00020397ex.jpg
DSC_00020397ex.tif
DSC_00020397ex_klein.jpg
DSC_0261_mail.jpg

So I would like to take the filename, extract certain parts of a standard filename schema -- while ignoring other parts which were obviously added later --, and then build a new filename to be used for -tagsfromfile.
I recognize that I would have to issue one exiftool run per try, but that's ok with me.

Like this:

Code Select Expand


# first: try for NEF
if ($filename =~ /([a-z]{2,3}(?:[a-z]|[-_])\d{4,8}).*\./i  ) {
    return $1.raw
}
return undef

#second: try for TIF
if ($filename =~ /([a-z]{2,3}(?:[a-z]|[-_])\d{4,8}).*\./i  ) {
    return $1.tif
}
return undef


(This code isn't actually tested, it's just an example.)

Anything I can do here?

So long,
Tillomar

[Phil Harvey]

Hi Tillomar,

I would like to write it as a sequence of if (...) return ....

Yes.  You can return a value (ie. return undef) from anywhere in the expression.

For the second problem -- building a more sophisticated filename to address the file from which I want to extract tag values -- I haven't found a solution so far. The main reason is that the argument of -tagsfromfile is a format string, which accepts only print format fields. But I need to analyze the filename of the file in work to build a master file name.

You can do arbitrary Perl manipulations on any tag in 2 ways:

1) using the advanced formatting feature in the -tagsFromFile expression.

2) by creating a user-defined Composite tag (see the sample config file for examples).

- Phil