"JPEG EOI marker not found" in Samsung S8 / S9 panos: In search of a batch fix

Started by robertklink, April 04, 2021, 09:52:41 PM

Previous topic - Next topic

robertklink

Reference March 17 2019 post by urbacher: "How to fix a file giving "JPEG EOI marker not found" error.

The problem in a nutshell is: Samsung Galaxy S8 & S9 (at least) phone photos taken in Panorama mode all have this error, and the impact is that Exiftool won't write to them.

I have thousands of S9 photos, with hundreds of pano mode photos mixed among them having the EOI marker error. Puts a big hole in my Exiftool work flow. (And thousands of other S8 & S9 users must have the same problem.)

StarGeek suggested in the previous post that it can be fixed by doing a 0 degree rotation in IrfanView without any degradation - I confirmed that it works, and I have not noticed any degradation. XnViewMP does also. With IrfanView I don't see a batch mode for that particular operation, but XnViewMP does. Hence I'm starting to see a means to my desired batch fix end via XnViewMP.

Now to get to my specific issues:

1. How to identify the specific offending files:

Option 1 - IMAGE SIZE: The Exif data doesn't have any direct indication of being in panorama mode. I reckon I could do a test on the order of "IF Exifwidth + Exifheight gt Value, it must be a pano" and hence can be "presumed guilty". My hesitation on that is that it will keep passing even after it's been fixed, and subject to getting processed again and again

Option 2 - IF I CAN'T WRITE TO IT, IT'S BAD: I could try to write "I'm good" into (pick a spare) tag xyz in all the S9's files, relying on exiftool to bounce off the files with the EOI error. (Do not use Overwrite_Original.) Then make a 2nd pass to find and list the files where tag xyz ne "I'm good". Restore originals.

Option 3 - USE "VALIDATE": Initially my logical preference is to base it on the "Warning  : JPEG format error" (among other things) that gets thrown onscreen (/STDOUT?) by "exiftool -validate -warning -a <Dir>". Question on that is: How to recognize and act on it, where "act" is to generate a list (Text or CSV file?) of the offending files. I have only the barest inkling of how to do that and the inkling is telling me "It's Complicated". Hmm, maybe Option 2?

2. How to list / package the offending files to feed them into an XnViewMP batch.

Haven't really given that much thought yet, beyond "If a have a list, there's gonna be a way". Hopefully without having to flatten my existing subfolder structure.

Looking for sage advice / better ideas











StarGeek

Quote from: robertklink on April 04, 2021, 09:52:41 PM
Reference March 17 2019 post by urbacher: "How to fix a file giving "JPEG EOI marker not found" error.

Original thread

QuoteStarGeek suggested in the previous post that it can be fixed by doing a 0 degree rotation in IrfanView without any degradation - I confirmed that it works, and I have not noticed any degradation. XnViewMP does also. With IrfanView I don't see a batch mode for that particular operation,

Open up one of the files
Hit T or File Menu -> Thumbnails, this will open up Irfanview's thumbnail browser, optionally you can use the Options Menu -> Load thumbs from all subfolders to show all images in all subdirectories.
Select images to process.  I always just hit Cntr+A to select all images.
Hit Shift+J or File Menu -> JPG Lossless Operations -> Lossless rotation with selected files...
The lossless rotation window will pop up and can be run on all selected files.

I've run this operation multiple times over images to test them.  Once you run it the first time, any additional times you run it over the same file produces the exact same file.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

robertklink

My 2019 folder (example) has 9806 files, 59 folders, 56 GB. Guessing that maybe 500 of those files have the EOI issue.

Are you suggesting that I go ahead and batch rotate the whole lot?

StarGeek

That's quite a lot.  I probably wouldn't do it all at once.  At the very least, a limited number at a time, with a proper backup and verification that nothing went wrong.  I've done large batches with Irfanview before but don't really feel comfortable recommending a large batch from a camera that I haven't previously tested.

Can you share one of the problem files?  Maybe I can figure out a way to narrow the number needing processing down a bit.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

robertklink

Attached one of the problem files.

Here's the "validate" results on it, with the ": JPEG format error" being the critical one:
C:\Users\rober>exiftool -validate -warning -a "D:\Folders Synced from NAS\P_Picasa(P)\Pictures\Pictures by Year\2018\2018-01-16 to -31 Bob's New Zealand N Island\20180119_171821.jpg"
Validate                        : 12 Warnings (3 minor)
Warning                         : [minor] Boolean value for XMP-acdsee:Tagged should be capitalized
Warning                         : [minor] IPTC TimeCreated too short (6 bytes; should be 11)
Warning                         : [minor] Unknown APP9 segment
Warning                         : JPEG format error
Warning                         : ExifIFD tag 0xa434 LensModel requires ExifVersion 0230 or higher
Warning                         : ExifIFD tag 0xa430 OwnerName requires ExifVersion 0230 or higher
Warning                         : Missing required JPEG ExifIFD tag 0x9101 ComponentsConfiguration
Warning                         : Missing required JPEG ExifIFD tag 0xa000 FlashpixVersion
Warning                         : Missing required JPEG GPS tag 0x0000 GPSVersionID
Warning                         : Missing required JPEG IFD1 tag 0x011a XResolution
Warning                         : Missing required JPEG IFD1 tag 0x011b YResolution
Warning                         : Missing required JPEG IFD1 tag 0x0128 ResolutionUnit


For reference, here's more typical "validate" results on some ostensibly good non-pano files:
======== D:/Folders Synced from NAS/P_Picasa(P)/Pictures/Pictures by Year/2018/2018-01-16 to -31 Bob's New Zealand N Island/20180130_133206.jpg
Validate                        : 6 Warnings (2 minor)
Warning                         : [minor] Boolean value for XMP-acdsee:Tagged should be capitalized
Warning                         : [minor] Unknown APP5 segment
Warning                         : ExifIFD tag 0xa430 OwnerName requires ExifVersion 0230 or higher
Warning                         : Missing required JPEG IFD1 tag 0x011a XResolution
Warning                         : Missing required JPEG IFD1 tag 0x011b YResolution
Warning                         : Missing required JPEG IFD1 tag 0x0128 ResolutionUnit
======== D:/Folders Synced from NAS/P_Picasa(P)/Pictures/Pictures by Year/2018/2018-01-16 to -31 Bob's New Zealand N Island/20180130_133215.jpg
Validate                        : 8 Warnings (3 minor)
Warning                         : [minor] Boolean value for XMP-acdsee:Tagged should be capitalized
Warning                         : [minor] IPTC TimeCreated too short (6 bytes; should be 11)
Warning                         : [minor] Unknown APP5 segment
Warning                         : ExifIFD tag 0xa434 LensModel requires ExifVersion 0230 or higher
Warning                         : ExifIFD tag 0xa430 OwnerName requires ExifVersion 0230 or higher
Warning                         : Missing required JPEG IFD1 tag 0x011a XResolution
Warning                         : Missing required JPEG IFD1 tag 0x011b YResolution
Warning                         : Missing required JPEG IFD1 tag 0x0128 ResolutionUnit


In the search for an example "bad file" that was small enough to attach, I incidentally discovered that not all pano images have the ": JPEG format error." I also noticed that larger "bad" ones have more (e.g. 15) of the "Warning  : [minor] Unknown APPxx segment"  errors.

I did go ahead and IrfanView batch lossless rotate 9256 files last night. It took about an hour and seems to have come out fine - EXCEPT for operator's error of leaving the "Apply original EXIF date/time to new file" checked. That's a bother when I want to sync (FreeFileSync) my working copy of these files back into my master set. (The latter being under watch by Picasa.) I'll be re-setting those working files from the master set and re-doing the rotate, followed by my unrelated exiftool batch jobs.

I am a bit concerned about possible unintended consequences of IrfanView's "Transformation (sets Orientation tag always to top-left)" proclamation. Never gave the orientation flag much thought before, but starting to wonder if it could possibly mess with my face tagging in Picasa under some circumstances, i.e. face tag position info is for pic with orientation ne top left?

StarGeek

Quote from: robertklink on April 05, 2021, 03:00:45 PM
I am a bit concerned about possible unintended consequences of IrfanView's "Transformation (sets Orientation tag always to top-left)" proclamation. Never gave the orientation flag much thought before, but starting to wonder if it could possibly mess with my face tagging in Picasa under some circumstances, i.e. face tag position info is for pic with orientation ne top left?

Ah, yes, if you have files that have a different orientation and they either get rotated or otherwise set differently, then the regions will be out of places.  There is the Rotate_Regions.config file which will allow regions to be rotated.  Or you could batch copy the Orientation tag from the master set.

I tend to forget about such things because using the lossless rotation is usually the first thing I do with any image before dealing with the face regions.

For what it's worth, here's an extremely messy command which will list all the files that have the JPEG format error warning.  It's based upon this post from Phil.  I've done only limited testing, as I only have the one example you gave.
exiftool -validate -if "${Warning;my $flag=0;for (my $i=0; ;++$i){ my $tagKey='Warning';$tagKey.=\" ($i)\" if $i ;my $val = $self->GetValue($tagKey); if ($val=~/JPEG format error/) {$flag=1} ; last unless defined $val}; $_=$flag }" -filename /path/to/files/

To search for your other problem images, just replace the JPEG format error in the command with the text of the other error you are looking for.  In the case of the [minor] text, either drop it or escape the brackets with a backslash, e.g. \[minor\]
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

robertklink

Your "JPEG format error finder" works. It's just finished working it's way through my 2018 folder, finding 515 problem files out of 12841 total, scattered through the entire year. (That was a "big trip year" with me & my wife snapping like crazy.) Given the scattershot sprinkling, I'll just go ahead and do the entire set of files.

I'm think that your suggestion to batch copy the orientation tag from the master set will work nicely. My proposed method is to have exiftool run CSV output files of -filename and -orientation from each set, match the filename rows up in a spreadsheet (should be identical), then delete all columns but the "to be fixed" SourceFile and "master" Orientation. Then convert back to CSV and run it back in Exiftool to write.

Of course I'll do that after performing the lossless zero rotation. And in the future, I'll be doing the rotation on all my new files before I start any face tagging.

OBTW: I learned that this problem goes back at least to my Samsung S7 circa 2017.

Thanks much.

robertklink

It finally dawned on me that I can just run the output of -Orientation to CSV from my working set before I do the lossless rotation, then run the CSV back into the working set after the lossless rotation to restore the original value. Easy peasy. Just gonna take some time.