News:

2023-03-15 Major improvements to the new Geolocation feature

Main Menu

ExifTool Dies on One Specific Photo

Started by elopsentorml, November 30, 2019, 01:06:01 PM

Previous topic - Next topic

elopsentorml

I've split this off of my Google Takeout thread because it seems specific to one photo.  I ran a "merge" command to copy some information from my Google Takeout .json folders to their associated photo folders:

exiftool -r -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "D:\GooglePhotos"

After a very short time (seconds), that process failed without any kind of message.  It just stopped.  There was no summary at the end of things.  Just a new command prompt.  I ran a validate command against everything (similar to this):

exiftool -validate -warning -error -a test.jpg

and piped it into a text file.  That showed just warnings.  No errors.

Then I ran my initial "merge" command but with the -verbose option.  I took the last file where it stopped and looked at it:

[ExifTool]      ExifToolVersion                 : 11.77
[File]          FileName                        : forestport 001.jpg
[File]          Directory                       : D:/1939-01-01
[File]          FileSize                        : 62 kB
[File]          FileModifyDate                  : 2019:11:30 08:01:51-08:00
[File]          FileAccessDate                  : 2019:11:30 08:01:51-08:00
[File]          FileCreateDate                  : 2019:11:29 20:42:45-08:00
[File]          FilePermissions                 : rw-rw-rw-
[File]          FileType                        : JPEG
[File]          FileTypeExtension               : jpg
[File]          MIMEType                        : image/jpeg
[File]          ExifByteOrder                   : Little-endian (Intel, II)
[File]          CurrentIPTCDigest               : 3b6de139c6e8a694079289c325b3e831
[File]          ImageWidth                      : 640
[File]          ImageHeight                     : 520
[File]          EncodingProcess                 : Baseline DCT, Huffman coding
[File]          BitsPerSample                   : 8
[File]          ColorComponents                 : 3
[File]          YCbCrSubSampling                : YCbCr4:2:0 (2 2)
[JFIF]          JFIFVersion                     : 1.02
[JFIF]          ResolutionUnit                  : inches
[JFIF]          XResolution                     : 200
[JFIF]          YResolution                     : 200
[EXIF]          ImageDescription                : redacted Clan in Forestport
[EXIF]          XResolution                     : 72
[EXIF]          YResolution                     : 72
[EXIF]          ResolutionUnit                  : inches
[EXIF]          Software                        : Adobe Photoshop Elements 4.0
[EXIF]          ModifyDate                      : 2005:10:17 09:32:08
[EXIF]          YCbCrPositioning                : Centered
[IPTC]          ApplicationRecordVersion        : 2
[IPTC]          Keywords                        : redacted, Portraits
[IPTC]          Caption-Abstract                : redacted Clan in Forestport
[XMP]           XMPToolkit                      : Image::ExifTool 11.77
[XMP]           Description                     : redacted Clan in Forestport
[XMP]           Subject                         : redacted, Portraits
[XMP]           NativeDigest                    : 36864,40960,40961,37121,37122,40962,40963,37510,40964,36867,36868,33434,33437,34850,34852,34855,34856,37377,37378,37379,37380,37381,37382,37383,37384,37385,37386,37396,41483,41484,41486,41487,41488,41492,41493,41495,41728,41729,41730,41985,41986,41987,41988,41989,41990,41991,41992,41993,41994,41995,41996,42016,0,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,22,23,24,25,26,27,28,30;D41D8CD98F00B204E9800998ECF8427E
[XMP]           NativeDigest                    : 256,257,258,259,262,274,277,284,530,531,282,283,296,301,318,319,529,532,306,270,271,272,305,315,33432;7187616F350C7C8B8750E27A55214AC4
[XMP]           CreatorTool                     : Adobe Photoshop Elements 4.0
[XMP]           ModifyDate                      : 2005:10:17 09:32:08-05:00
[Composite]     ImageSize                       : 640x520
[Composite]     Megapixels                      : 0.333


[ExifTool]      ExifToolVersion                 : 11.77
[File]          FileName                        : forestport 001.jpg.json
[File]          Directory                       : D:/1939-01-01
[File]          FileSize                        : 771 bytes
[File]          FileModifyDate                  : 2019:04:05 14:19:20-07:00
[File]          FileAccessDate                  : 2019:11:29 20:42:45-08:00
[File]          FileCreateDate                  : 2019:11:29 20:42:45-08:00
[File]          FilePermissions                 : rw-rw-rw-
[File]          FileType                        : JSON
[File]          FileTypeExtension               : json
[File]          MIMEType                        : application/json
[JSON]          CreationTimeFormatted           : Jan 3, 2018, 12:33:42 AM UTC
[JSON]          CreationTimeTimestamp           : 1514939622
[JSON]          Description                     : Sometime around 1939 in Forestport, NY: . redacted Clan in Forestport (I just guessed at 1939 for a date)
[JSON]          GeoDataAltitude                 : 0.0
[JSON]          GeoDataLatitude                 : 0.0
[JSON]          GeoDataLatitudeSpan             : 0.0
[JSON]          GeoDataLongitude                : 0.0
[JSON]          GeoDataLongitudeSpan            : 0.0
[JSON]          GeoDataExifAltitude             : 0.0
[JSON]          GeoDataExifLatitude             : 0.0
[JSON]          GeoDataExifLatitudeSpan         : 0.0
[JSON]          GeoDataExifLongitude            : 0.0
[JSON]          GeoDataExifLongitudeSpan        : 0.0
[JSON]          ImageViews                      : 0
[JSON]          ModificationTimeFormatted       : Apr 5, 2019, 8:19:21 PM UTC
[JSON]          ModificationTimeTimestamp       : 1554495561
[JSON]          PhotoTakenTimeFormatted         : Jan 1, 1939, 5:00:00 PM UTC
[JSON]          PhotoTakenTimeTimestamp         : -978246000
[JSON]          Title                           : forestport 001.jpg


The one thing that looks off to me is that the PhotoTakenTimeTimestamp in the .json file is negative and that's what I'm using to merge a Created Date into the .jpg file.  But, I don't know if that's bad or not.

I ran that "validate" command against the files and it came back "OK":

D:\1939-01-01>exiftool -validate -warning -error -a "d:\1939-01-01"
======== d:/1939-01-01/forestport 001.jpg
Validate                        : OK
======== d:/1939-01-01/forestport 001.jpg.json
Validate                        : OK
    1 directories scanned
    2 image files read


Then, I ran that initial "merge" command against them and it once again just stopped:

D:\1939-01-01>exiftool -verbose -wm cg -tagsfromfile "%d/%F.json" "-Caption-Abstract<Description" "-ImageDescription<Description" -Description "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s "-GPSAltitude<GeoDataAltitude" "-GPSLatitude<GeoDataLatitude" "-GPSLatitudeRef<GeoDataLatitude" "-GPSLongitude<GeoDataLongitude" "-GPSLongitudeRef<GeoDataLongitude" -overwrite_original "d:\1939-01-01"
======== d:/1939-01-01/forestport 001.jpg
Setting new values from d:/1939-01-01/forestport 001.jpg.json

D:\1939-01-01>


I have no idea why ExifTool is failing on this, but I thought I'd pass it on.

Windows Executable version 11.77 of ExifTool on Windows 10 Pro x64

elopsentorml

And, just in case another sample helps, I ran into another one:

[ExifTool]      ExifToolVersion                 : 11.77
[File]          FileName                        : redacted at Birth.jpg.json
[File]          Directory                       : D:/GooglePhotos/1959-02-03
[File]          FileSize                        : 711 bytes
[File]          FileModifyDate                  : 2019:04:05 14:19:14-07:00
[File]          FileAccessDate                  : 2019:11:29 20:42:45-08:00
[File]          FileCreateDate                  : 2019:11:29 20:42:45-08:00
[File]          FilePermissions                 : rw-rw-rw-
[File]          FileType                        : JSON
[File]          FileTypeExtension               : json
[File]          MIMEType                        : application/json
[JSON]          CreationTimeFormatted           : Dec 30, 2017, 3:59:34 PM UTC
[JSON]          CreationTimeTimestamp           : 1514649574
[JSON]          Description                     : redacted
[JSON]          GeoDataAltitude                 : 0.0
[JSON]          GeoDataLatitude                 : 0.0
[JSON]          GeoDataLatitudeSpan             : 0.0
[JSON]          GeoDataLongitude                : 0.0
[JSON]          GeoDataLongitudeSpan            : 0.0
[JSON]          GeoDataExifAltitude             : 0.0
[JSON]          GeoDataExifLatitude             : 0.0
[JSON]          GeoDataExifLatitudeSpan         : 0.0
[JSON]          GeoDataExifLongitude            : 0.0
[JSON]          GeoDataExifLongitudeSpan        : 0.0
[JSON]          ImageViews                      : 0
[JSON]          ModificationTimeFormatted       : Apr 5, 2019, 8:19:15 PM UTC
[JSON]          ModificationTimeTimestamp       : 1554495555
[JSON]          PhotoTakenTimeFormatted         : Feb 3, 1959, 5:00:00 PM UTC
[JSON]          PhotoTakenTimeTimestamp         : -344242800
[JSON]          Title                           : redacted at Birth.jpg


[ExifTool]      ExifToolVersion                 : 11.77
[File]          FileName                        : redacted at Birth.jpg.json
[File]          Directory                       : D:/GooglePhotos/1959-02-03
[File]          FileSize                        : 711 bytes
[File]          FileModifyDate                  : 2019:04:05 14:19:14-07:00
[File]          FileAccessDate                  : 2019:11:29 20:42:45-08:00
[File]          FileCreateDate                  : 2019:11:29 20:42:45-08:00
[File]          FilePermissions                 : rw-rw-rw-
[File]          FileType                        : JSON
[File]          FileTypeExtension               : json
[File]          MIMEType                        : application/json
[JSON]          CreationTimeFormatted           : Dec 30, 2017, 3:59:34 PM UTC
[JSON]          CreationTimeTimestamp           : 1514649574
[JSON]          Description                     : redacted
[JSON]          GeoDataAltitude                 : 0.0
[JSON]          GeoDataLatitude                 : 0.0
[JSON]          GeoDataLatitudeSpan             : 0.0
[JSON]          GeoDataLongitude                : 0.0
[JSON]          GeoDataLongitudeSpan            : 0.0
[JSON]          GeoDataExifAltitude             : 0.0
[JSON]          GeoDataExifLatitude             : 0.0
[JSON]          GeoDataExifLatitudeSpan         : 0.0
[JSON]          GeoDataExifLongitude            : 0.0
[JSON]          GeoDataExifLongitudeSpan        : 0.0
[JSON]          ImageViews                      : 0
[JSON]          ModificationTimeFormatted       : Apr 5, 2019, 8:19:15 PM UTC
[JSON]          ModificationTimeTimestamp       : 1554495555
[JSON]          PhotoTakenTimeFormatted         : Feb 3, 1959, 5:00:00 PM UTC
[JSON]          PhotoTakenTimeTimestamp         : -344242800
[JSON]          Title                           : redacted at Birth.jpg


Again, the PhotoTakenTimeTimestamp is negative.  I'm assuming the validate command and the

exiftool -r -all= -tagsfromfile @ -all:all -unsafe -icc_profile -overwrite_original -F "D:\GooglePhotos"

command I did didn't catch it (if it's actually a problem) because its not a standard tag.
Windows Executable version 11.77 of ExifTool on Windows 10 Pro x64

StarGeek

That date is definately a problem since the time stamp in the JSON PhotoTakenTimeTimestamp is in Epoch time which starts at 00:00:00 UTC on 1 January 1970.  I didn't even realize that it could be counted prior to that date until I read the wiki page.

I'm guessing that the perl time routines that exiftool uses can't deal with a negative Epoch time stamp.  For time stamps before 1970, try using -d "%b %d, %Y, %I:%M:%S %p" and the JSON PhotoTakenTimeFormatted instead.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

elopsentorml

Do I need to report this somewhere as a bug?  According to that Wikipedia link:

QuoteThe Unix epoch is the time 00:00:00 UTC on 1 January 1970.[2] There is a problem with this definition, in that UTC did not exist in its current form until 1972; this issue is discussed below. For brevity, the remainder of this section uses ISO 8601 date and time format, in which the Unix epoch is 1970-01-01T00:00:00Z.

The Unix time number is zero at the Unix epoch, and increases by exactly 86400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12677 days after the epoch, is represented by the Unix time number 12677 × 86400 = 1095292800. This can be extended backwards from the epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4472 days before the epoch, is represented by the Unix time number −4472 × 86400 = −386380800.

So, all photos taken/modified before 1970 with the unformatted date/time code will cause this problem.

I was leery of using the formatted field because I'm not sure if all the software and services I've used over the last several decades have formatted those numbers the same way.  Also, I can't find the FAQ talking about date/time codes (I know I've seen it before).  Can you point me in the right direction?

EDIT:  Hopefully the following is good for those format codes:

https://linux.die.net/man/3/strftime
Windows Executable version 11.77 of ExifTool on Windows 10 Pro x64

elopsentorml

I'm running the command with the formatted string now.  Just about everything popped up a warning about "garbage at end of string in strptime:  UTC in ExifIFD:DateTimeOriginal (PrintConvInv)."  It looks like the date didn't get copied over.  Perhaps because of the "UTC" after the actual formatted string in the .json file?
Windows Executable version 11.77 of ExifTool on Windows 10 Pro x64

StarGeek

Quote from: elopsentorml on November 30, 2019, 01:54:09 PMJust about everything popped up a warning about "garbage at end of string in strptime:  UTC in ExifIFD:DateTimeOriginal (PrintConvInv)."

Oops, sorry, I didn't copy/paste the complete line from my notes. Use
-d "%b %d, %Y, %I:%M:%S %p UTC"
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

elopsentorml

Windows Executable version 11.77 of ExifTool on Windows 10 Pro x64

Phil Harvey

I don't have time right now to read this in detail, but if you haven't already provided the files necessary to reproduce this problem, could you please email them to me along with the command you used (philharvey66 at gmail.com).

Thanks.

It sounds like I may have to put in some protection against the standard time routines crashing on odd input values.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

StarGeek

Here's a quicky json file I made from a previous Google Takeout of mine.  I just changed the date to match one elopsentorml's examples.  Any image file can be used.

{
  "title": "test1.Jpg",
  "description": "XMP-tiff:ImageDescription",
  "imageViews": "0",
  "creationTime": {
    "timestamp": "1548714348",
    "formatted": "Jan 28, 2019, 10:25:48 PM UTC"
  },
  "modificationTime": {
    "timestamp": "1554915781",
    "formatted": "Apr 10, 2019, 5:03:01 PM UTC"
  },
  "geoData": {
    "latitude": 5.346782999999999,
    "longitude": -3.985286,
    "altitude": 2222.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 5.346782999999999,
    "longitude": -3.985286,
    "altitude": 2222.0,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "photoTakenTime": {
    "timestamp": "-978246000",
    "formatted": "Jan 1, 1939, 5:00:00 PM UTC"
  }
}


The command to duplicate would be as follows.  I can confirm that on my end exiftool just exits without any output
exiftool -TagsFromFile test1.Jpg.json "-DateTimeOriginal<PhotoTakenTimeTimestamp" -d %s test1.Jpg
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

Thanks StarGeek,

It seems that this problem may be specific to the Windows libraries because it works on the Mac and sets DateTimeOriginal to "1939:01:01 12:00:00".

I'll try this on Windows when I get a chance, but it looks like this will be a Windows-only patch.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

Thanks for this report.

OK.  I can reproduce this on Windows.  Ouch.  The Perl interpreter dies when calling the strptime function with this message:



This is particularly egregious because I was already protecting the strptime call inside an eval statement (which would normally guard against any sort of error in the function from affecting ExifTool).  I will put in a specific patch to protect against negative epoch times in Windows, and ExifTool 11.78 should give you this warning instead of crashing:

        Can't convert negative epoch time in ExifIFD:DateTimeOriginal (PrintConvInv)

I really hope there aren't other argument combinations that cause strptime to crash as well.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).