Cannot seem to keep original file modification dates for all files when updating

[Infobleep]

Hi there

When I run the following ExifTool 8.41 commands, in BASH on my Snow Leopard Mac, some of my files are having their file modification dates changed, even though I state keep the originals. This is only occurring when I am copying EXIF meta-data to IPTC fields. However when I create the EXIF meta-data using similar commands it keeps the original dates. As I use Time Machine to backup I am able to go back to the files before I do the update and I have run this several times always with the same issue. I've not noted whether it is the same files that are having their modification time changed each time but could do.

This issue doesn't occur when I run a Windows DOS version of the script, although that is a slightly earlier version of ExifTool being used.

The part of the code in question is:
exiftool '-iptc:by-line<EXIF:Artist' -L -overwrite_original_in_place -P -r "toamend"
set v-
# *******************************
# *******************************
# ** Contact - set phase**
set v+
exiftool '-IPTC:Contact=my name and e-mail address in here' -L -overwrite_original_in_place -P -r "toamend"
set v-
# *******************************
# *******************************
# ** copyrightNotice - copy from EXIF Copyright **
set v+
exiftool '-IPTC:copyrightNotice<EXIF:Copyright' -L -overwrite_original_in_place -P -r -k "toamend"

I have another question which I think is best placed in a separate thread so I will do that.

Thank you for your help and Happy Christmas to everyone

Tim

[Phil Harvey]

If ExifTool has a problem setting the date/time it will send an "Error setting file time" warning to stderr.  Do you see this?

I have seen similar problems in Windows due to locking of files by antivirus software, but I didn't think this type of problem would be possible on a Mac.

I have no other ideas.  If it always happens with the same files, see if you can reproduce the problem with a single command.

- Phil

[Infobleep]

Hi Phil

Thank you for your reply. I never saw the problem myself in Windows, despite having anti-virus software installed. Of course I don't have such software installed in Snow Leopard so that can't be the issue.

I don't see the message "Error setting file time". In a test ran just one command on 113 files, the first one of the three, and it caused the same problem. However when I ran each command separately on just 6 of the 113 files, some of which had previously had their file modification time changed, it didn't cause a problem. So may be it is related to the shear number of files being processed and the speed at which ExifTool is trying to processing them vers the speed at which Snow Lepoard is doing whatever it does to the files. Possible whilst trying to replace the original files with the temporary created files or some locking of the files by BASH. My programming skills are not advanced enough to know or figure out anything more in depth about this. What is odd is the fact that so far it has only occurred when copying the EXIF fields to the IPTC fields.

Kind regards

Tim

[Infobleep]

Hi there

I also need to add that today it also changed the modification date when I was adding some JPEG comments, which will then get copied to the EXIF and IPTC comment fields. Again it didn't do it to all files but this time I wasn't working with large amounts of files either. I know this will work in Windows DOS without such an issue.

I don't think I provide much more information than this, unless there is a way of me tracking the processes that are going on when the files get updated.

Kind regards

Tim

[Phil Harvey]

I will do some tests when I get a chance to see if I can reproduce this problem.

- Phil

[Infobleep]

Thanks Phil

Kind regards

Tim

[Phil Harvey]

Wow, tricky problem.  I can make it happen by running a command similar to the one you used, but not consistently.  The funny thing is that I inserted some code to read back the date/time after setting it to that of the original, and I always returns the correct time.  Only later when I look at the directory, some files have the wrong (current) times.  I can run the exact nsame command again on the same 1500 files and often see no problems.  The problem happens less than about half the time, but when it does there are usually many files affected (about 10-30).  The affected files tend to be bunched in groups, but oddly again not sequential in the order they were processed.  This seems to be some weird Perl and/or OS X quirk, and not something that ExifTool is doing wrong, but my hope is to find a way to patch this behaviour to avoid the problem.  Right now I'm guessing either some funny caching bug, or perhaps an interference with Spotlight.  I'll keep you posted if I discover anything else.

Also, it only seems to happen with -overwrite_original_in_place.  I haven't yet been able to make it happen when using -overwrite_original instead.

- Phil

[Phil Harvey]

Curiouser and curiouser.  I added the current time to my debugging print statement, and the incorrect file timestamps don't match up with the time the file was closed by exiftool.  Instead, the times are from 5 to 8 seconds later.  How odd is that?  In this amount of time exiftool has processed another 100 files or more.

- Phil

[Phil Harvey]

More news:  Today I try to reproduce the problem using the exact same command as yesterday with the exact same set of 1500 files, but now I can't get it to misbehave at all.  I have run the command about 10 times now with no problems.  Very annoying. 

For the record, the command is:

exiftool tmp -overwrite_original_in_place "-iptc:keywords<exif:model" -P

In the "tmp" directory I have 1500 JPEG images from a variety of camera models.

- Phil

[Phil Harvey]

Status update:

I spent a couple more days working on this, but am a bit frustrated because the problem is intermittent.  Here is what I have learned so far:

• happens randomly and intermittently to groups of files in large batch jobs
• tried open/close file again before utime: NO EFFECT  (utime is the routine used to set the file date/time)
• tried open/close directory before utime: NO EFFECT
• tried disabling Spotlight indexing on the volume: NO EFFECT
• tried writing to a different pre-existing file in the last stage: NO EFFECT (but can't make problem occur if the file isn't pre-existing!)
• checked to see if any files were being left open: NO
• can't reproduce this with an isolated test script
• happens maybe once every 3000 files or so, but affects groups of 10-40 (non-sequential) files when it happens
• affected files have FileModifyDate set to a time 5-11 seconds after file was closed (this phenomenon is observed intermittently for random files even without the -P option)
• immediate read-back of file's timestamp yields correct result (must change after this, presumably 5-11 seconds later)
• seems to happen more often when I'm connected to the internet(not sure about this)

- Phil

[Phil Harvey]

I've done more testing with this.  I have run the command in Linux and Mac OS X Tiger on 50,000 files each, with no problems at all.

And you say that you don't see the problem in Windows either.

It may be safe to say that the problem doesn't exist on these platforms, or with these versions of Perl.  The Perl versions were: Tiger 5.8.6, Linux 5.8.8 and Windows 5.8.7 (the version that ships with the ExifTool executable).  Hmmm....  Could be a pattern?  I'll repeat the Tiger tests with Perl 5.10 and report back.

- Phil

[Phil Harvey]

Well, I ran this test for many, many hours on OS X Tiger and Perl 5.10.0 (the same version I was running on Snow Leopard), and not a problem.

So it is the OS version (or something else machine related), and not the Perl version that is the common factor.

- Phil

[Infobleep]

Hi Phil

Thank you for looking into this. I'm amazed at your dedication to find a solution. Perhaps I'd be the same if I knew more about the technical side, which I find interesting. I wondered myself whether it was a Snow Leopard issue when I was coming across it and it appears from what you are saying it is. I was finding every time I ran the command random dates were changing but I probably didn't run it as often as you did to get a period of dates not changing.

Is their any chance you can raise such an issue with Apple, assuming you enough info to give to them on the problem? Or do you have to find a work around yourself?

Kind regards

Tim

[Phil Harvey]

Hi Tim,

If I could generate a simple script that to demonstrate this problem then I could submit a bug report to Apple.  However, so far I have failed to reproduce the problem with a simple script that generates the files itself.

But I'll keep working at it.  It is a slow process though.  (My testing on Tiger takes about 8 hours of running time to go through 50,000 files since the machine isn't very fast.)  I just hope I solve the problem before I wear out my hard disks...

- Phil

[Infobleep]

Hi Tim,

If I could generate a simple script that to demonstrate this problem then I could submit a bug report to Apple.  However, so far I have failed to reproduce the problem with a simple script that generates the files itself.

But I'll keep working at it.  It is a slow process though.  (My testing on Tiger takes about 8 hours of running time to go through 50,000 files since the machine isn't very fast.)  I just hope I solve the problem before I wear out my hard disks...

- Phil

That's a lot of files Phil. And I thought I had a lot of photos.....

Good luck and I of course hope you resolve it sometime. In the mean time I'm using Windows implementation to process the photos. However Windows is installed on my Mac as a Guest OS using VirtualBox and that has a load of problems in itself, which I still need to resolve. The biggest being it's rather slow speed.

Kind regards

Tim