Feature Request: Agressive Repair Option

[StoneMonkeyMark]

Phil,
I know you have said in other threads about this tool not being a repair tool, but I would like to suggest a rationale and design that may be acceptable.
I have been finally getting around to tidying up all my photos and getting them on the cloud.
And in doing so I have been using your tool again for the tags and file renaming etc.
But with the warnings that show up I have become obsessed with cleaning them all up.
I think most people who use your tool are likely to be 'collectors and organisers' and as such like things tidy, clean and repaired.

To do this has been a long process with me trying to find the right ways to clean up each warning.
I decided no warnings would even be better than keeping the original (unusable) data.
So my objective was to repair where possible and to just remove when repair was not possible.

It would seem possible that you could have a mode that would do a full set of tags reads (and trial write) to get all the -warnings and -errors, (actually writing them out to get the full set seems a waste of time and file writes).
You may have repair/rewrite more than once per file it seems too.
If there are any warnings not fixed by your rewrite you can then just delete the offending tag/data at the lowest part for the tag tree.

And if this could have a 'what I would touch/change without doing it' mode then it can be up to the user if they want to go ahead.

Some reasons why I found this time consuming.
1. -if "-warning eq '[minor] message...' has not always had a match. Not sure why and rather than work it out I just pushed on with other tests.
2. Once repaired for one reason there can be other warnings behind the first one so iteration is required.
3. The multi level tags can catch you out when working with warnings.
4. This is a list of the sorts of things I was doing. Each run would take a while. I know I could have used file lists perhaps but see 2.
  -if "-warning eq '[minor] Fixed incorrect URI for xmlns:MicrosoftPhoto'" "-DateAcquired<$DateAcquired -overwrite_original_in_place -r "DIR"
  -if "-warning eq 'Duplicate UserComment tag in IFD1'" -IFD1:All= -overwrite_original_in_place -r "DIR"
  -if "defined $warning and defined $MakerNotes" "-MakerNotes<$MakerNotes" -overwrite_original_in_place -r "DIR"
  -if "-warning eq 'Bad MakerNotes directory' or -warning eq 'Unrecognized MakerNotes' or -warning eq 'Invalid CanonCameraSettings data'" -MakerNotes= -overwrite_original_in_place -r "DIR"

The first in that list is as case of finding the warning but then not being able to find the easiest way to rewrite it out again cleanly. I had to find a tag that was present and use it as XMP:all<XMP:All did not do the job. Files had only some of About, DateAcquired, DateTime, or CreateDate as an example.

I know this is not really a direction of interest, but thanks for listening.

Is there a way to get the list of files with errors without doing a trial write out?
So far I am using
  -if "$error" -p "$error $directory\$filename" "DIR" -q -q -f > errors.txt

Thanks as always for writing such a densely powerful tool.
MarkB

PS for others reading this, here are some other threads I found on this when searching:

How can I detect images with the shifted MakerNote problem?  (Read 166 times)
https://exiftool.org/forum/index.php/topic,6919.0.html

Repair Maker Notes  (Read 2723 times)
https://exiftool.org/forum/index.php/topic,2732.0.html

Overlapping MakerNotes - any chance to fix?  (Read 2068 times)
https://exiftool.org/forum/index.php/topic,1686.0.html

[Phil Harvey]

It is likely that most of the errors you are mentioning are in the maker notes.  If so, then I don't think you will ever be able to achieve the tidy and clean solution that you desire.  Keep in mind:

1. Makernote errors are rather subjective because there is no published specification.

2. ExifTool can't fix all makernote problems.  Even if it fixed all of the problems we know about now, we may find other problems in the future as we learn more about the specific makernote structure.

It would seem possible that you could have a mode that would do a full set of tags reads (and trial write) to get all the -warnings and -errors, (actually writing them out to get the full set seems a waste of time and file writes).

I think this may be possible.  I'm not saying that I'll add this feature, but it may also satisfy other requests for a "dry run" mode.

Some reasons why I found this time consuming.
1. -if "-warning eq '[minor] message...' has not always had a match. Not sure why and rather than work it out I just pushed on with other tests.

You mean?: -if "$warning eq ... ("$" instead of "-" before "warning")

Yes.  Checking for specific warnings like this is problematic, mainly because there may be other warnings that hide the one you are looking for as you found out.

The first in that list is as case of finding the warning but then not being able to find the easiest way to rewrite it out again cleanly. I had to find a tag that was present and use it as XMP:all<XMP:All did not do the job. Files had only some of About, DateAcquired, DateTime, or CreateDate as an example.

I don't think I understand your problem exactly, but when you rewrote XMP with "XMP:all<XMP:all", you are allowing ExifTool to move the tags to the preferred location in XMP.  Instead you may want "all:all<XMP:all", which preserves the specific location.

Is there a way to get the list of files with errors without doing a trial write out?

Not really.  You could possibly do a write to a dummy file (-o /dev/null) and pipe stderr to a file: 2> errors.txt, but I haven't tried this myself.

- Phil

[StoneMonkeyMark]

Phil,

Thanks for the reply.

>If so, then I don't think you will ever be able to achieve the tidy and clean solution that you desire

I know the Makernotes are subjective and also any errors you know about and some you might be able to fix now could change in the future. I am OK with that in that the original idea for this is an *aggressive* clean up. IE accept any fixes you can do now, delete them if you cant and accept that running again in the future may show new things.

Yes a 'deep' "dry run" is what we need to get a list of all the files that are needed to be processed. Perhaps you don't need to report all the 'hiding' warnings if the request to rewrite it would perform what improvements you can do all at the same time.
Perhaps this is already possible now and rather than try to repair the warnings one at a time I should be trying to get the tool to read all the metadata and write it all out again for any file with any warning. Is that possible as I could not find out how to do it without finding a tag to write in each group.

>I don't think I understand your problem exactly, but when you rewrote XMP with "XMP:all<XMP:all", you are allowing ExifTool to move the tags to the preferred location in XMP.  Instead you may want "all:all<XMP:all", which preserves the specific location.

I need to read up more about 'preferred location' as I did not understand that bit.
But regardless, my issue is that when I tried to do the "XMP:all<XMP:all" it would say that it was fixed but not write out the file. Is there an easy way to force the writing out?

>You mean?: -if "$warning eq ... ("$" instead of "-" before "warning")

Drat I knew you would spot that as soon as I sent this. I was cut and pasting from my ideas file rather than the real lines I was using. Sorry.

>Not really.  You could possibly do a write to a dummy file (-o /dev/null) and pipe stderr to a file: 2> errors.txt, but I haven't tried this myself.

OK I understand. Perhaps there is another way to do this efficently. Can I supply a long list of filenames (5000 say) from an external file? That way I could get the subset of files and just be processing the 5000 each time rather than searching the full set of 100k.

MarkB