"file already exists" error (WINDOWS)

[Uli]

• on a fast W7 64bit, or a slow XP 32bit machine
• folder having about 250 pairs of sony ARW/JPG files and a few single JPG images
• installed last official, or latest beta of geosetter and latest exiftool
• running any combination of create/overwrite file, or keep or set time for the JPGs and ARWs

I always get the error "temporary file already exists" for a few files (W7) or many files (XP).

Running the same operation multiple times different files are affected.

This happens to JPG files, and to sidecar files.

I switched of the virus scanner, no difference.

I used a set of files with no sidecar files to let exiftool generate the sidecar files.

I used another set of files with sidecar files to let exiftool update the sidecar files.

I got the feeling that the results are depending on the disk activities caused by other processes. In other words the issue is highly dynamic. I had a run with ARW files only, and got no errors. But having the ARW/JPG pairs in one run I got errors on the XMPs.

Related topic: 405, and some on the geosetter forum.

[Phil Harvey]

Is this command updating the same sidecar file for both ARW and JPG images?  I'm just wondering if the command creates the same temporary file for 2 different images in the same command.  If so, perhaps some sort of delay in the system updating the directory cache could cause this problem.  You say that you got not errors when running on ARW images only?  Is this always the case?  What about JPG only?

- Phil

[Uli]

I switched of sidecar files for JPGs, and activated them for ARW.

Removing the ARWs and then doing a run for the JPGs, some of them are renamed to ...temp...

That's just what the error message says, the files can not be renamed to the final name. I tried to do that manually, and that worked.

I am not too sure if I have seen the issue by only having ARWs. I made many runs and it worked fine.

If my theory is correct that it is a dynamic problem, then writing to a small text file minimizes the possibility to get an access to the sidecar file by another process while exiftool is processing it.

From the days I wrote some SW I know that there is an OS operation to request a temporary file name/handle generated by the system. This name/handle should be unique and not accessable by other processes.

[Phil Harvey]

This renaming error is most likely due to another process locking the file and preventing exiftool from renaming it, as you suspect.  Virus scanners have been known to do this, but you have apparently ruled out this possibility by disabling yours.  I'm convinced there must be some other software running on both your systems which is causing this problem.  What about Windows file indexing?  Whatever it is though, it probably isn't running in a default Windows installation or else more people would be reporting this problem.

- Phil

[Uli]

I have a quesiton to the internal sequence of execution. Is it true that exiftools is generating a temp file, then is processing this file, closing this file and then (!) renaming it by using an OS (shell) level rename command? If yes, then that's the point where  another process might jump in.

An idea from my side: why not having a mode where the temp file is beeing created and NOT renamed. This would leave it up to the user to do some post processing, e.g. doing the renaming later via a script. Having this feature we could easily find out if it is really the renaming step which fails (on MS OS).

EDIT 1:
OBSERVATION: I monitored visually (WINDOWS explorer, holding F5) the content of the folder containing the images. I recognized one to three temp files at the same time. If one file has been processed and renamed, the next one is beeing opened while two others are still processed. File size is zero.

EDIT 2:
Indexing is disabled, too.

[Phil Harvey]

I have a quesiton to the internal sequence of execution. Is it true that exiftools is generating a temp file, then is processing this file, closing this file and then (!) renaming it by using an OS (shell) level rename command? If yes, then that's the point where  another process might jump in.

Yes, with the exception that the rename is done using the Perl library "rename" function.

An idea from my side: why not having a mode where the temp file is beeing created and NOT renamed. This would leave it up to the user to do some post processing, e.g. doing the renaming later via a script. Having this feature we could easily find out if it is really the renaming step which fails (on MS OS).

No rename is done if the -o option is used.

EDIT 1:
OBSERVATION: I monitored visually (WINDOWS explorer, holding F5) the content of the folder containing the images. I recognized one to three temp files at the same time. If one file has been processed and renamed, the next one is beeing opened while two others are still processed. File size is zero.

Odd.  Something is being cached somehow.  Either that or there is a system delay with the rename operation.

EDIT 2:
Indexing is disabled, too.

Oh well, but it is good to rule this out as the source of the problem.

- Phil

[Uli]

Need to come back to

An idea from my side: why not having a mode where the temp file is beeing created and NOT renamed. This would leave it up to the user to do some post processing, e.g. doing the renaming later via a script. Having this feature we could easily find out if it is really the renaming step which fails (on MS OS).

I guess the -o switchs behaviour like

• create temp file
• if successful delete source file
• rename temp file to original source file name

instead of

• rename source file to *original
• create new file from *original

In both cases a rename has to be done.

My idea

• create a new file from source (append e.g.  *_updated)

In this case the original file will not be touched, and the the new file can be kept open until the last processing step has been performed.

Does this make sense?

Question: how can I get the parameters the exiftools is receiving from the geosetter tool? That would be helpful for deeper analysis.

[Phil Harvey]

The normal behaviour is (FILE is the original file specified on the command line):

1) create TEMP
2) write changes to TEMP
3) close TEMP
4) erase TEMP and abort if any errors writing or closing file
5) rename FILE to FILE_original unless FILE_original already exists or -overwrite_original is used
6) rename TEMP to FILE

With -o, the behaviour is:

1) abort if OUTFILE already exists
2) create specified OUTFILE
3) write changes to OUTFILE
4) close OUTFILE
5) erase OUTFILE and abort if there were any errors
6) erase FILE if -overwrite_original is used

- Phil

Edit: Is there any way in Geosetter to show the ExifTool command line or a console log?

[Uli]

Edit: Is there any way in Geosetter to show the ExifTool command line or a console log?

No external logging, only in memory, and you can only look on a per-file basis.

geosetter is writing *.arg files, but they were deleted on the fly. One arg file per image.

By-the-way: also this *.arg files are created many at once. It is NOT an effect of the OS. I tried to drop some to my text editor, and could read the content. Seems that geosetter starts multiple instances of the exiftool simultaneously. Does this ring a bell?

Edit: WINDOWS task manager showed about 10 exiftool instances!

Edit: Re-test ARW files only, initially no sidecar files, 2nd run with sidecar files. With and without -o option. No issues.

[Phil Harvey]

10 exiftool instances?  Hmmm.  Could they be interfering with each other?  You should check with Friedemann to see if this is possible.  Maybe this is a Geosetter issue.  If it processes the sidecar of FILE.arw and FILE.jpg simultaneously, it would be a problem.

- Phil

[Uli]

Sidecar files are not a problem because I can also run geosetter on JPGs only (actually I deleted the ARW files) and the problem stays. Only the number of errors is different. Sometimes only one single file fails, BUT for every run it is a different one.

I seems the forums don't like me. I tried to get access to the geosetter forum twice, but no response 

[Uli]

Someone made an entry in the geosetter forum today, telling the same story.

http://www.geosetter.de/mantis/view.php?id=973

http://www.geosetter.de/mantis/view.php?id=405

[MOL]

10 exiftool instances?

That sounds more like ExifTool is called 10 times by Geosetter, but the instances are never freed. Which would be a bug in Geosetter, of course. The only other scenario I know of where ExifTool stays in memory is after a program crash. You have to kill it in the Task Manager in this case.

Uwe

[Uli]

That sounds more like ExifTool is called 10 times by Geosetter, but the instances are never freed. Which would be a bug in Geosetter, of course. The only other scenario I know of where ExifTool stays in memory is after a program crash. You have to kill it in the Task Manager in this case.

Uwe

That's also my guess: geosetter is starting many exiftool processes, and might get confused which temp file is related to which original file.

Task Manager: no hanging processes. It's really a massiv start of exiftool instances which are immediately disapearing after a second. Can not tell how much exactly. It's fluctuating between 3 and 10 instances. That's also visible in the number of ARG files (located in the temp folder).

Please, because I can not get access to the geosetter forum, can someone place a cross link to http://www.geosetter.de/mantis/view.php?id=973

[Phil Harvey]

I've just released ExifTool 8.99.  Try this version to see if it helps.

This version includes a patch that retries the renaming up to 50 times at 0.01 sec intervals (on Windows only).

- Phil