-overwrite_original failing in script file

[camner]

I am trying to create a zsh (on MacOS) script file to execute some ExifTool commands while passing the filename as a command-line parameter.  Here's the code:

Code Select Expand

FILE_OR_DIR="$1"

#Echo "$FILE_OR_DIR"

Exiftool \
'-IPTC:DateCreated < EXIF:DateTimeOriginal' \
'-IPTC:TimeCreated < EXIF:DateTimeOriginal' \
'-IPTC:TimeCreated < EXIF:DateTimeOriginal' \
'-XMP-exif:all < EXIF:all' \
'-XMP-exifEX:all < EXIF:all' \
'-XMP-tiff:all < EXIF:all' \
'-overwrite_original_in_place -X -m -p -P' \
"$FILE_OR_DIR"I then execute this by invoking the name of the file in Terminal and dragging the file or directories I want to process through the end of the command line. This works almost perfectly, but ExifTool seems to ignore the -overwrite_original_in_place and creates the filename_original file.

If I just copy the commands into Terminal, things work if I omit the single quotes around  -overwrite_original_in_place -X -m -p -P.  But if I omit the single quotes in the script file, I get a "no such file or directory" error.

I feel I must be missing something silly, but I can't pass all this out.

[StarGeek]

When you put single quotes around
'-overwrite_original_in_place -X -m -p -P'
that forces the script to treat this as one single option. The spaces become included in it. The result is that exiftool things your looking for a tag called this (spaces included)
-overwrite_original_in_place -X -m -p -P

This will fail because there isn't a tag with that name.

Additional problems, the -X (-xmlFormat) option does nothing when it is part of a write operation. It only takes effect when it is outputting data.

The -p (-printFormat) option is the same way, except that it requires a second argument to follow it. In this case, you have -P as that argument, so the -p option basicaly eats the -P and you do not get the -P (-preserve) option as part of the command.

But if I omit the single quotes in the script file, I get a "no such file or directory" error.

Is this the exact output? Because I can't find this phrase in the source code. If exiftool can't find a file or directory, it returns Error: File not found

Code Select Expand

C:\>exiftool -G1 -a -s fakefile.jpg
Error: File not found - fakefile.jpg

C:\>exiftool -G1 -a -s ./fakefile/
Error: File not found - ./fakefile/
I do not use a Mac, so I can't help with why the script is reading things the way it does. Maybe first put it all on one line and add the line feeds after you get that working?

Code Select Expand

Exiftool '-IPTC:DateCreated < EXIF:DateTimeOriginal' '-IPTC:TimeCreated < EXIF:DateTimeOriginal' '-IPTC:TimeCreated < EXIF:DateTimeOriginal' '-XMP-exif:all < EXIF:all' '-XMP-exifEX:all < EXIF:all' '-XMP-tiff:all < EXIF:all' -overwrite_original_in_place -m -P "$FILE_OR_DIR"
Also, is the Mac case insensitive for executables? I'm pretty sure it's case sensitive for files, and it would seem weird if it wasn't the case for commands. In other words, are you sure you saved the program as Exiftool and not exiftool?

[camner]

@STARGEEK:
Thanks for the for the quick and complete reply. I appreciate your taking the time to point out some inherent issues with my code.

Maybe first put it all on one line and add the line feeds after you get that working?

I took your suggestion and did exactly that. It worked! So, I am wondering whether there might have been an extra space someplace after a \ resulting in the space, rather than the newline character, being escaped.

I have now run into another issue. After executing the file, I get this warning message:
Warning: IPTCDigest is not current. XMP may be out of sync
I have taken the time to read some forum posts here about this issue. I think I understand the cause of the warning, which is that I have modified some IPTC group variables, which could then cause a lack of sync between the IPTC group variables and XMP variables.  I modified my script to include all of the commands in the exif2xmp.args file, which I think should result in the IPTC and XMP variables being in sync, but I still get that warning message. A forum post I read talked about needing to reset IPTCDigest, but I am not sure how to do that.

I read your reply in this post to use this code:

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest= /path/to/files/but I don't understand why one wants the filepath in the IPTCDigest.  Can't one remove by just doing

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest=
Thanks again for your help!

Also, is the Mac case insensitive for executables? I'm pretty sure it's case sensitive for files, and it would seem weird if it wasn't the case for commands. In other words, are you sure you saved the program as Exiftool and not exiftool?

By default, Mac volumes are formatted APFS, which is case insensitive as to filenames, though when a volume is formatted, one can specify a file system with file names being case sensitive, but the general advice seems to be that unless one has a good reason for doing that, it's best not to do this.

As far as I know, shell scripts are case sensitive (variables, etc.), as MacOS is a flavor of Unix, and Unix scripts are case sensitive.

[StarGeek]

I have now run into another issue. After executing the file, I get this warning message:
Warning: IPTCDigest is not current. XMP may be out of sync
I have taken the time to read some forum posts here about this issue. I think I understand the cause of the warning, which is that I have modified some IPTC group variables, which could then cause a lack of sync between the IPTC group variables and XMP variables.

Not quite. It just means that the current IPTC data does not match the data from when the IPTCDigest was last computed. I try to explain it here and give my opinion on it. I do need to rewrite it and make it more complete. Basically, as long as you are properly keeping the data in sync, you can ignore it or reset it with -ITPCDigest=new.

I modified my script to include all of the commands in the exif2xmp.args file, which I think should result in the IPTC and XMP variables being in sync, but I still get that warning message. A forum post I read talked about needing to reset IPTCDigest, but I am not sure how to do that.

Any reason you wouldn't just use the exif2xmp.args file? It will do the IPTCDigest reset for you.

I read your reply in this post to use this code:

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest= /path/to/files/but I don't understand why one wants the filepath in the IPTCDigest.

This isn't setting it to /path/to/files/, it is removing the tag. Notice there is a space between the two. /path/to/files/ is literally what it says. You replace it with the actual path to the directories and files you want to process. It's my generic standard term to fill in for what files/directories you want to process. Phil usually uses simply DIR

Can't one remove by just doing

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest=

My cheeky response...

Code Select Expand

C:\>exiftool -if "$IPTCDigest" -IPTCDigest=
No file specified
Phil's coded response

Code Select Expand

C:\>exiftool -if "$IPTCDigest" -IPTCDigest= DIR
Error: File not found - DIR
You were meant to enter any valid directory name, not "DIR" literally.

[camner]

I modified my script to include all of the commands in the exif2xmp.args file, which I think should result in the IPTC and XMP variables being in sync, but I still get that warning message. A forum post I read talked about needing to reset IPTCDigest, but I am not sure how to do that.

Any reason you wouldn't just use the exif2xmp.args file? It will do the IPTCDigest reset for you.

Well, I don't how to use the exif2xmp.args file within a MacOS shell script! It's probably simple, but I couldn't find the info in the ExifTool docs. (Maybe ExifTool has built in a way to pull in those commands?)

I read your reply in this post to use this code:

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest= /path/to/files/but I don't understand why one wants the filepath in the IPTCDigest.

This isn't setting it to /path/to/files/, it is removing the tag. Notice there is a space between the two. /path/to/files/ is literally what it says. You replace it with the actual path to the directories and files you want to process. It's my generic standard term to fill in for what files/directories you want to process. Phil usually uses simply DIR

Can't one remove by just doing

Code Select Expand

exiftool -if "$IPTCDigest" -IPTCDigest=

My cheeky response...

Code Select Expand

C:\>exiftool -if "$IPTCDigest" -IPTCDigest=
No file specified

Ha ha, yes!

In all seriousness, I can see how to put the path for a single file there (just use my $FILE_OR_DIR variable.  But if I'm processing an entire directory, $FILE_OR_DIR will contain the path to the directory.  Will ExifTool know to substitute the individual file paths as it iterates across the files in the directory?

[StarGeek]

Well, I don't how to use the exif2xmp.args file within a MacOS shell script! It's probably simple, but I couldn't find the info in the ExifTool docs. (Maybe ExifTool has built in a way to pull in those commands?)

It's really simple. Save the exif2xmp.args into the same directory as exiftool and use the -@ (Argfile) option

To copy EXIF tags to XMP in the same file, it would be
exiftool -@ exif2xmp.args file.jpg
or from another file
exiftool -TagsFromFile source.jpg -@ exif2xmp.args target.jpg

Though I do now see that I was incorrect about this resetting the IPTCDigest. You have to use either the xmp2iptc.args or the iptc2xmp.args file to automatically do this. Otherwise, you need to include -IPTCDigest=new.

But the best thing to do, if you can, i.e. none of your programs only work with ITPC data, is to drop IPTC completely. It's an old and outdated standard.

Will ExifTool know to substitute the individual file paths as it iterates across the files in the directory?

Yes. If you give exiftool a directory, it will process every file in that directory that it can. Since you are doing a write operation, this will be limited to only files it can write to, i.e. files  such as AVI or Docx will be skipped. It will not process files in subdirectories unless the -r (-recurse) option is used.

You can pass any number of individual files and directories up to the limits of the command line (appx 8k characters on Windows, 100k+ on Mac/Linux). These can be mixed and match, so you could pass several individual file names and several directory names and exiftool will process them all.

[camner]

Well, I don't how to use the exif2xmp.args file within a MacOS shell script! It's probably simple, but I couldn't find the info in the ExifTool docs. (Maybe ExifTool has built in a way to pull in those commands?)

It's really simple. Save the exif2xmp.args into the same directory as exiftool and use the -@ (Argfile) option

To copy EXIF tags to XMP in the same file, it would be
exiftool -@ exif2xmp.args file.jpg
or from another file
exiftool -TagsFromFile source.jpg -@ exif2xmp.args target.jpg

Though I do now see that I was incorrect about this resetting the IPTCDigest. You have to use either the xmp2iptc.args or the iptc2xmp.args file to automatically do this. Otherwise, you need to include -IPTCDigest=new.

But the best thing to do, if you can, i.e. none of your programs only work with ITPC data, is to drop IPTC completely. It's an old and outdated standard.

Will ExifTool know to substitute the individual file paths as it iterates across the files in the directory?

Yes. If you give exiftool a directory, it will process every file in that directory that it can. Since you are doing a write operation, this will be limited to only files it can write to, i.e. files  such as AVI or Docx will be skipped. It will not process files in subdirectories unless the -r (-recurse) option is used.

You can pass any number of individual files and directories up to the limits of the command line (appx 8k characters on Windows, 100k+ on Mac/Linux). These can be mixed and match, so you could pass several individual file names and several directory names and exiftool will process them all.

Thanks again for an extensive reply. You write very clearly and completely, and I appreciate the time you took to do this.

As far as IPTC is concerned, I would love to just drop it, as you have suggested, but with the apps I use, I don't know if that's possible. For example, I use Lightroom extensively. I also use Radiant Photo and, less often, Topaz photo AI. My workflow involves importing photos directly into Lightroom and then sending them to Radiant or Topaz directly from Lightroom after I enter metadata such as keywords, Title, Caption, etc. During the export process, Lightroom creates a TIFF file and sends it on to Radiant or Topaz.  Lightroom, of course, has its own catalog database and it is not necessary to write out the data in the database to the files. If, however, I write out the metadata before exporting to Radiant or Topaz, upon examining the metadata tags I see extensive use of XMP, but almost no IPTC tags. But, the TIFF file handed  off to Radiant or Topaz DOES have a lot of IPTC tags. When Radiant edits the photo and sends the TIFF file back to Lightroom, those IPTC tags are preserved. But, if I write out the metadata from the Lightroom catalog back to the TIFF file, almost all the IPTC tags disappear! So, for unknown reasons, Lightroom, when creating a TIFF export file, creates IPTC tags. But then, if one manually writes out the metadata from the LR catalog to the resultant TIFF file, the IPTC tags are erased. 

All this can lead to some wonkiness.  Lightroom does not place any limit on the number of characters allowed in the Title field.  But, the IPTC standard says that the Title field should contain no more than 64 characters. So, if I examine the metadata in an exported TIFF file, the XMP-DC tag could contain more the 64 characters, but the IPTC-Object Name tag will truncate that Title.  That doesn't seem to cause any difficulties, but after I run the script we've been talking about (which is essentially exif2xmp.args) and I read the metadata back into Lightroom, the Title field is truncated, even though when I examine the metadata, the XMP-DC tag is still there with its full character count. So, for some reason, when the metadata is read back in to Lightroom, Lightroom seems to read it from the IPTC-Object Name tag rather than the XMP-DC tag!  Why, who knows? Adobe doesn't document which tags it reads to determine dates and other metadata, and obviously uses algorithms to determine what tags to prefer if tags that contain the same information in different metadata groups conflict.

I haven't yet figured out how to work around the issue with truncated titles. There is probably a way.

My need for a script is not with photos taken with a digital camera but with scanned photos. The dates are all wrong, of course, since the scanning company only knows the date scanned, not the date taken. I have manually entered the dates using Adobe Bridge (it's much faster there than using Lightroom) into EXIF Date Time Original, and then wanted to use the script to populate the other date fields. I just have to figure out a way not to have longer titles truncated when the metadata is read back into Lightroom.

[wywh]

IPTC standard says that the Title field should contain no more than 64 characters

"You can add -m (-ignoreMinorErrors) option the write IPTC data of any length.  I have yet to encounter a program that holds to the spec's maximum tag length.  They all just ignore it."

https://exiftool.org/forum/index.php?topic=15007.msg80782#msg80782

BTW I tested this and the latest Lightroom Classic and Bridge prefer XMP-dc:Title over IPTC:ObjectName.

- Matti

[StarGeek]

Doing a quick check just now, Topaz will copy over most, if not all, metadata. It will only change/add three tags that I can see, EXIF:Software, XMP:CreatorTool, and XMP:HistorySoftwareAgent. Basically, just tags that are for the name of the editing software.

Photo AI uses exiftool behind the scenes to deal with the metadata. Interestingly, in the past they just used the old executable. Checking right now, they've rolled exiftool into a couple of DLL files, called exiftool.dll and texiftool.dll.

[camner]

@wywh & @stargeek:
Thanks both for your contributions. I am going to investigate this further. I understand what each of you is saying, and certainly defer to your expertise and experience, which are far greater than mine.

However, I know that with my scanned images, something has caused all of the IPTC-Object Name tags To be truncated at 64 characters and when I read the metadata from those files back into Lightroom, despite the presence of an XMP-DC Tag with the untruncated version, Lightroom displays in its interface the truncated version. I don't know how to square that with the fact that Lightroom prioritizes reading from XMP tags over IPTC tags. I will do some more testing and see if I can't determine precisely what step caused this issue.

[StarGeek]

However, I know that with my scanned images, something has caused all of the IPTC-Object Name tags To be truncated at 64 characters and when I read the metadata from those files back into Lightroom, despite the presence of an XMP-DC Tag with the untruncated version, Lightroom displays in its interface the truncated version. I don't know how to square that with the fact that Lightroom prioritizes reading from XMP tags over IPTC tags. I will do some more testing and see if I can't determine precisely what step caused this issue.

See my link above about the IPTCDigest. tl;dr If the IPTCDigest exists and is out of sync, then Lightroom will ignore the XMP data.

Try removing the IPTCDigest and see what LR will do in that case. That's something I don't know the answer to and I'm curious as to the result.

[wywh]

If the IPTCDigest exists and is out of sync, then Lightroom will ignore the XMP data.

Try removing the IPTCDigest and see what LR will do in that case. That's something I don't know the answer to and I'm curious as to the result.

You are right. I tested this with the latest Adobe Lightroom Classic 13.5.1 and Adobe Bridge 2024 14.1.2.300.

If a .jpg has warning (and 86 characters in IPTC:ObjectName)...

Code Select Expand

exiftool -a -G1 -s -Warning -CurrentIPTCDigest -IPTCDigest -XMP-dc:Title -IPTC:ObjectName .
======== ./xmp_iptc_test.jpg
[ExifTool]      Warning                         : IPTCDigest is not current. XMP may be out of sync
[File]          CurrentIPTCDigest               : 766dce8c2aa3b7ead8c064e375fafd2f
[Photoshop]     IPTCDigest                      : a44c6491b26004f85a9ab6516deb2a53
[XMP-dc]        Title                           : XMP-dc:Title
[IPTC]          ObjectName                      : IPTC:ObjectName 1234567890123456789012345678901234567890123456789012345678901234567890
...Lightroom and Adobe Bridge display IPTC:ObjectName (with all 86 characters) and ignore XMP-dc:Title.

If I either sync the digests or remove the out-of-sync digest with -Photoshop:IPTCDigest=new or -Photoshop:IPTCDigest= which results as...

Code Select Expand

exiftool -a -G1 -s -Warning -CurrentIPTCDigest -IPTCDigest -XMP-dc:Title -IPTC:ObjectName .                           
======== ./xmp_iptc_test_new.jpg
[File]          CurrentIPTCDigest               : 766dce8c2aa3b7ead8c064e375fafd2f
[Photoshop]     IPTCDigest                      : 766dce8c2aa3b7ead8c064e375fafd2f
[XMP-dc]        Title                           : XMP-dc:Title
[IPTC]          ObjectName                      : IPTC:ObjectName 1234567890123456789012345678901234567890123456789012345678901234567890
======== ./xmp_iptc_test_delete.jpg
[File]          CurrentIPTCDigest               : 766dce8c2aa3b7ead8c064e375fafd2f
[XMP-dc]        Title                           : XMP-dc:Title
[IPTC]          ObjectName                      : IPTC:ObjectName 1234567890123456789012345678901234567890123456789012345678901234567890
...Lightroom and Bridge display XMP-dc:Title and ignore IPTC:ObjectName. BTW macOS Sonoma Photos.app behaves the same.

- Matti

[camner]

OK, I've done some more digging, and I know what step causes IPTC-Object Name to be truncated, but I have yet to figure out what sequence of steps led to the truncated version appearing in Lightroom when metadata is manually read back into Lightroom after being changed in an external app. My testing thus far is with a new digital photo, not an old negative scanned by an external scanning service. I will do this test next.

Here are the steps I took and their results, which are reproducible:

• Import a digital photo into Lightroom-at this point, there are no IPTC tags in the image file
• In Lightroom, add a title with more than 64 characters-of course, at this point the image file itself has not been altered, as Lightroom stores metadata in the catalog only unless one chooses to manually write out the metadata to the image file
• In Lightroom, export the file to an external app, such as Radiant Photo-in this step, Lightroom creates a new file to hand off to Radiant Photo. Examining this file, there are a number of IPTC tags, including Object Name. Object Name is NOT truncated!
• Edit the photo in Radiant Photo and save the file back to Lightroom-Radiant Photo adds some tags, but does not alter the IPTC block
• Back in Lightroom, manually choose to read the metadata from the file back into Lightroom-after this step, the Title field in Lightroom displays the full length of the title.
• Run my ExifTool script (see below for the script code)-after the script runs, IPTC-Object Name is now truncated to 64 characters, so it appears thatExifTool is the culprit! [This puzzles me since the script doesn't reference Object Name...I should test this again to make sure I haven't made a stupid mistake]
• Back in Lightroom, manually choose to read the metadata from the file back into Lightroom-the result here depends on whether the script code line that resets the IPTC Digest is present or absent. Present, I get no warning about my IPTC Digest being out of date, and Lightroom continues to display the full length of the Title. If I omit that line, I get the morning, and Lightroom reads the truncated version from IPTC-Object Name and displays it in the Title field

Given that my workflow involves using an extra little app for almost all photos and given that Lightroom creates an IPTC block when exporting the file, I can't realistically do without the IPTC block. I guess my question is why ExifTool truncates IPTC-Object Name...

Code Select Expand

#!/bin/zsh

FILE_OR_DIR="$1"

#Echo "$FILE_OR_DIR"

exiftool \
'-IPTC:DateCreated < EXIF:DateTimeOriginal' \
'-IPTC:TimeCreated < EXIF:DateTimeOriginal' \
'-IPTC:TimeCreated < EXIF:DateTimeOriginal' \
'-XMP-exif:all < EXIF:all' \
'-XMP-exifEX:all < EXIF:all' \
'-XMP-tiff:all < EXIF:all' \
'-XMP-dc:Description < EXIF:ImageDescription' \
'-XMP-photoshop:DateCreated < EXIF:DateTimeOriginal' \
'-XMP-photoshop:DateCreated < Composite:SubSecDateTimeOriginal' \
'-XMP-xmp:CreateDate < EXIF:CreateDate' \
'-XMP-xmp:CreateDate < Composite:SubSecCreateDate' \
'-XMP-xmp:ModifyDate < EXIF:ModifyDate' \
'-XMP-xmp:ModifyDate < Composite:SubSecModifyDate' \
'-XMP-xmp:CreatorTool < EXIF:Software' \
'-XMP-dc:Rights < EXIF:Copyright' \
'-XMP-dc:Creator < EXIF:Artist' \
'-Composite:Flash < EXIF:Flash' \
'-XMP:GPSLatitude < Composite:GPSLatitude' \
'-XMP:GPSLongitude < Composite:GPSLongitude' \
'-XMP:GPSDateTime < Composite:GPSDateTime' \
'-AllDates < EXIF:DateTimeOriginal' \
'-IPTCDigest=new' \
-overwrite_original_in_place  -m -P -progress \
"$FILE_OR_DIR"

[StarGeek]

I guess my question is why ExifTool truncates IPTC-Object Name...

Because the standard says that it is limited to a maximum of 64 characters. See the IPTC tags page, which details the maximum length of each IPTC tag.

As @WYWH said, you can override this with the -m (-ignoreMinorErrors) option.

[camner]

 

I guess my question is why ExifTool truncates IPTC-Object Name...

Because the standard says that it is limited to a maximum of 64 characters. See the IPTC tags page, which details the maximum length of each IPTC tag.

As @WYWH said, you can override this with the -m (-ignoreMinorErrors) option.

My script uses the -m option...it's in the 2nd to last line of the code in post #12