News:

2023-03-15 Major improvements to the new Geolocation feature

Main Menu

Question to NoDups

Started by herb, July 11, 2018, 10:11:12 AM

Previous topic - Next topic

herb

Hello Phil,

in order to avoid duplicate entries in listtype tags I did some tests with NoDups feature.
In forum I have seen that all NoDups-examples use separator ##.
exiftool -sep "##" "-keywords<${keywords;NoDups}" DirOrFile
During my tests I have seen that using e.g. separator ? or | will lead to unexpected results.

Ok, ## is a good separator, but it cannot be sent when -stay_open is used.

So my question is: Which characters must not be used as separator for NoDups feature.

Thanks for your help in advance.

Best regards
Herb

Phil Harvey

Hi Herb,

Good point.  I'll fix this in version 11.07.  I should have been quoting these special characters in the regular expression.

Thanks for pointing this out.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#2
Hi again,

This also raises the issue of a big oversight on my part.  You should be able to start an argument with "#' in the argfile.  ExifTool 11.07 will allow you to add "##" as an argument like this:

#[NOT_COMMENT]##

So if you have any argument that starts with "#", just replace this with "#[NOT_COMMENT]#" in the argfile to avoid it being interpreted as a comment.


- Phil

Edit: I just realized the you can in fact add an argument of "##" by putting a space before it.  But then there is no mechanism to have an argument start with a space.  I will re-think this.
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

OK, second try.

ExifTool 11.07 will add a feature allowing an argfile line to begin with "#[CSTR]".  The rest of the line is taken as a standard C string.  As well as allowing spaces at the start of a line, this new feature also allows newlines to be embedded.  For example:

#[CSTR] this argument has a leading space

#[CSTR]this\nargument\nhas\nmultiple\lines

This also provides a mechanism to have an empty argument if necessary (which wasn't allowed before):

#[CSTR]

As well as allowing "##" as an argument, which was the initial impetus:

#[CSTR]##

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

Hello Phil,

thanks for the new version 11.07 and also many thanks for all the corrections.

A very short test showed that e.g. character "," is working as separator for NoDups feature.
The new feature #[CSTR] is also working properly when I start Exiftool from Dos-Box and call an argument file with -@ option.

But when I send a line starting with #[CSTR] via pipe to Exiftool - using the -stay_open feature -  this line is still ignored by Exiftool.
Or is there a misunderstanding ony my side?

Best regards
Herb

Phil Harvey

#5
Hi Herb,

Ooops, you're right.  I have different code to filter arguments from the stay_open file.  The new feature needs to be applied in two different places.  :(

I'll fix this in the next release.

Thanks for letting me know.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

Hello Phil,

thanks for your quick reply.

You created #[CSTR] feature and now we have a very easy possibility to enter e.g. NEWLINE into a textstring.

Please allow to think loud about the output of such a NEWLINE.
As far as I know Exiftool changes - in general - all binary characters to a dot inside the output.
Would it be helpful to modiy this feature and change NEWLINE to "\n" instead of a dot?

Best regards
Herb

Phil Harvey

Hi Herb,

A change like that would require a new command-line option (otherwise it wouldn't be backward compatible), but I really try to avoid that if possible.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

Hello Phil,

thanks for your reply.
I agree Exiftool should always be backward compatible and I also agree not to introduce a new commandline option.
But what do you think about using an userParam - called NEWLINE?

Best regards
Herb

StarGeek

Quote from: herb on July 28, 2018, 04:41:43 AM
Would it be helpful to modiy this feature and change NEWLINE to "\n" instead of a dot?

This can be done without adding new command options.  Add the following subroutine to your .exiftool_config file, somewhere before the main definitions (i.e. before %Image::ExifTool...)
# Replace NewLines (and other characters) with their escape sequences
sub RepNL {
($_)=@_;
my %replace = (
"\n" => "\\n", # line feed 0x0A
"\r" => "\\r", # carriage return 0x0D
"\t" => "\\t", # tab 0x09
);
s/(@{[join '|', map { quotemeta($_) } keys %replace]})/$replace{$1}/g;
}


You can now use the RepNL (rename as you desire for ease of use) as an advanced formatting option. 

Examples:
C:\>exiftool -Echo "Set Description and Caption with whitespace control characters" -P -overwrite_original -E -description="NL:&#x0a; CR:&#x0d; Tab:&#x09;" -Caption-Abstract="NL:&#x0a; CR:&#x0d; Tab:&#x09;" -E y:\!temp\Test3.jpg
Set Description and Caption with whitespace control characters
    1 image files updated

C:\>exiftool -g1 -a -s -Description -Caption-Abstract y:\!temp\Test3.jpg
---- XMP-dc ----
Description                     : NL:. CR:. Tab:.
---- IPTC ----
Caption-Abstract                : NL:. CR:. Tab:.

C:\>exiftool -g1 -a -s -echo "Global Replace NLs" -api "Filter=RepNL($_)" -Description -Caption-Abstract y:\!temp\Test3.jpg
Global Replace NLs
---- XMP-dc ----
Description                     : NL:\n CR:\r Tab:\t
---- IPTC ----
Caption-Abstract                : NL:\n CR:\r Tab:\t

C:\>exiftool -g1 -a -s -echo "Using -p, selectively replace"  -p "${Caption-Abstract;RepNL($_)}" y:\!temp\Test3.jpg
Using -p, selectively replace
NL:\n CR:\r Tab:\t


Other non-printable characters can be added if you desire.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

herb

#10
Hello StarGeek, hello Phil,

@StarGeek:
Thanks for your hints and thanks for your Perl code; it could be very helpful.
But your design will only work when I ask explicitely for a tag; e.g.: -Description in your example.
Or am I wrong?
Will it also work when I display all tags with -all:all because of the -api filter=... option?

Edit:
In meantime I did some more tests and now I also got it working with filter defined inside config file.


@Phil:
Please allow some additional questions to #[CSTR] feature:
(1) I thought \n is system depending: so \n == h'0A0D on Windows systems.
     A short test showed that also on Windows only h'0A will be used.

     Edit: Please forget (1): I mixed NEWLINE and LINEFEED.
(2) Using #[CSTR] feature I also think on Unicode characters, strings as values of listtype tags and of course also on structures.
     In a short test I entered Chinese characters to IPTC and XMP listtype tags and I have seen no restriction.
     But just to be sure: are there any restrictions?
     In combination with structures I think on possible problems because of escaping some characters.

Thanks again for your help in advance.
Best regards
Herb

Phil Harvey

Very smart StarGeek.

The #[CSTR] should work fine for inputting structures and Unicode characters.  The only difference is that it unescapes characters as if it was a double-quoted Perl string (no not actually a C string because Perl adds a few extra features).  See section 2.3.2 here.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

Hello Phil, hello StarGeek,

thanks again for your detail hints.

I see, I have to learn more about Perl.
But at the moment it looks like that I have only to escape backslash.

Best regards
Herb

Phil Harvey

Hi Herb,

That reminds me.  (For now) you must also escape "$" and "@" symbols.  I'll fix this in the next release.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

herb

Hello Phil,

thanks; you are really great!

Best regards
Herb