Issue extracting GPS tracks - was a problem with GPSdate in random files

[sidneyd]

When extracting GPS information from folders with a large number of files the GPS datestamp becomes 01 January 2000 for all images in that directory.  All images in the directory have a GPS location and date set to 2013.  When running the command on smaller directory sizes of 32 to 64 files it seems to work correctly.

System type: Windows 10 Pro x64, current build 1903, 32GB RAM
Exiftool version 11.74

I am using a batch script to extract GPS tracks from a number of directories, where the variable %%A is the directory name and !logname! is the output gpx filename.  The script issues the following exiftool command,

"C:\Program Files (x86)\Geosetter\tools\exiftool.exe" -r -if "$gpsdatetime" -fileOrder gpsdatetime -p "C:\Program Files (x86)\Geosetter\tools\gpx.fmt" %%A > !LogName!"


If I execute the same from the command prompt manually specifying a directory and output for directories more than 128 files, then the same issue appears.

D:\PIC-PROCESSING>"C:\Program Files (x86)\Geosetter\tools\exiftool.exe" -r -if "$gpsdatetime" -fileOrder gpsdatetime -p "C:\Program Files (x86)\Geosetter\tools\gpx.fmt" DRN128* > DRN128manual.gpx
    3 directories scanned
    2 files failed condition
  128 image files read


When the directory size has a smaller number of files the output gpx tracks file is fine, but when there is a large number of files  then the date output in the file is always 1 January 2000 and the GPS data seems to get corrupted in some cases also.

To simulate the problem, I have taken the directory with 707 files and created a few copies of that directory,  then have reduced the number of files in each directory according to the name of the directory (for example DRN032 contains 32 files,  DRN064 contains 64 files, DRN128 contains 128 files and so on...).  I then ran the batch script and below is the header output from the file, the first image DSC_23714.nef is always the same image, but you can see that things get messed up with more than 64 files in a directory. 

I am using the standard gpx.fmt file.

GPS date from output for DRN032 - 32 files:

Code Select Expand


<?xml version="1.0" encoding="utf-8"?>
<gpx version="1.0"
creator="ExifTool 11.74"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<trk>
<number>1</number>
<trkseg>
<trkpt lat="64.1170533299997" lon="-21.25345">
  <ele>156</ele>
  <time>2013:03:31 07:09:43Z</time>
</trkpt>


GPS date from output for DRN128 - 128 files:

Code Select Expand


<?xml version="1.0" encoding="utf-8"?>
<gpx version="1.0"
creator="ExifTool 11.74"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<trk>
<number>1</number>
<trkseg>
<trkpt lat="64.265945" lon="-21.1174566666667">
  <ele>112</ele>
  <time>2000:01:01 10:51:12.16Z</time>
</trkpt>


GPS date from output for DRN256 - 256 files:

Code Select Expand


<?xml version="1.0" encoding="utf-8"?>
<gpx version="1.0"
creator="ExifTool 11.74"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<trk>
<number>1</number>
<trkseg>
<trkpt lat="64.265945" lon="-21.1174566666667">
  <ele>112</ele>
  <time>2000:01:01 10:51:12.16Z</time>
</trkpt>


GPS date from output for DRN707 - 707 files:

Code Select Expand


<?xml version="1.0" encoding="utf-8"?>
<gpx version="1.0"
creator="ExifTool 11.74"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<trk>
<number>1</number>
<trkseg>
<trkpt lat="64.265945" lon="-21.1174566666667">
  <ele>112</ele>
  <time>2000:01:01 10:51:12.16Z</time>
</trkpt>


I have attached four output files along with the batch script and gpx.fmt file used.

I suspect that when there is a large number of files, that data is leaking somewhere and being overwritten.

[Phil Harvey]

Wow, this is very odd.  I can't see how anything could ever get overwritten.  Memory problems like that can't happen in Perl.

I tried this on my Mac here with 4000 files in a directory and it worked perfectly.

Note that the lat/lon is also different between the 2013 and the 2000 entries.  Try adding $filename to the gpx.fmt to see which files have the year 2000, then check the metadata of these files specifically.  I really think you have some files in your collection with the year 2000.  And of course they will come first in the output because you are sorting by GPSDateTime.

- Phil

[sidneyd]

Hi Phil,  thanks for the very quick reply.

Interestingly I have found that randomly a few images (about 1 in 200-300) have GPS co-ordinates set, but their gpsdate is this default epoch date of 1 January 2000, while all the other dates align to 2013.

Making the modification to gpx.fmt, I was able to find the filename of 1 image  with 2000 GPSdate.

Code Select Expand


<trkpt lat="64.2657283333333" lon="-21.1171233333333">
  <ele>112</ele>
  <time>2000:01:01 11:01:58.58Z</time>
  <info>DSC_23876.nef</info>
</trkpt>


Which is confirmed by running exiftool command against this file and following is the output.

Code Select Expand


"C:\Program Files (x86)\Geosetter\tools\exiftool.exe"  DRN707_FromDRN058\DSC_23837.nef

ExifTool Version Number         : 11.74
File Name                       : DSC_23837.nef
Directory                       : DRN707_FromDRN058
File Size                       : 42 MB
File Modification Date/Time     : 2013:03:31 11:51:11+01:00
File Access Date/Time           : 2019:11:04 13:17:12+00:00
File Creation Date/Time         : 2019:11:04 07:19:58+00:00
Modify Date                     : 2013:03:31 11:51:11
....
Create Date                     : 2013:03:31 11:51:11.80
Date/Time Original              : 2013:03:31 11:51:11.80
Modify Date                     : 2013:03:31 11:51:11.80
...
[b]GPS Date/Time                   : 2000:01:01 10:51:12.16Z[/b]
GPS Latitude                    : 64 deg 15' 57.40" N
GPS Longitude                   : 21 deg 7' 2.84" W
Date/Time Created               : 2013:03:31 11:51:11+00:00
...
GPS Position                    : 64 deg 15' 57.40" N, 21 deg 7' 2.84" W


Could you assist in advising for some command line options which would look for images with GPS co-ordinates set and GPS date = 1 January 2000; then set the GPS date to the date taken from the image

[Phil Harvey]

Something like this may allow you to do what you ask:

exiftool "-gpsdatetime<datetimeoriginal" if "$gpsdatetime =~ /^2000:01:01/" DIR

- Phil

[sidneyd]

Hi Phil, this was helpful, but something is still not working as one would expect to set the gpsdatetime

When I used the command as indicated then while one file should have been updated, the gpsdatetime was not changed.


exiftool.exe" -if "$gpsdatetime =~ /^2000:01:01/" -overwrite_original "-gpsdatetime<datetimeoriginal" .
    1 directories scanned
    2 files failed condition
    1 image files updated

or I even I tried with the force flag

exiftool.exe" -if "$gpsdatetime =~ /^2000:01:01/" -overwrite_original -F "-gpsdatetime<datetimeoriginal" .
    1 directories scanned
    2 files failed condition
    1 image files updated


I then used exiftool to check the three files, and as you can see the gpsdatetime has not been updated,  am I missing something?

exiftool.exe" -filename -gpsdatetime -datetimeoriginal .

Code Select Expand


======== ./DSC_23714_ValidGPSdate.nef
File Name                       : DSC_23714_ValidGPSdate.nef
GPS Date/Time                   : 2013:03:31 07:09:43Z
Date/Time Original              : 2013:03:31 07:09:43
======== ./DSC_23876_EpochGPSdate.nef
File Name                       : DSC_23876_EpochGPSdate.nef
GPS Date/Time                   : 2000:01:01 11:01:58.58Z
Date/Time Original              : 2013:03:31 12:01:58
======== ./_DSC4534_noGPS.nef
File Name                       : _DSC4534_noGPS.nef
Date/Time Original              : 2019:03:05 22:25:41
    1 directories scanned
    3 image files read


[Phil Harvey]

Can you show me the output of this command for the 3 files after you ran the exiftool command I gave?:

exiftool -filename -filemodifydate -gpsdatetime -datetimeoriginal -G1 -a .

[sidneyd]

Hi Phil,

here is the output from exiftool -filename -filemodifydate -gpsdatetime -datetimeoriginal -G1 -a .

Code Select Expand


======== ./DSC_23714_ValidGPSdate.nef
[System]        File Name                       : DSC_23714_ValidGPSdate.nef
[System]        File Modification Date/Time     : 2019:11:04 17:40:38+00:00
[XMP-exif]      GPS Date/Time                   : 2013:03:31 07:09:43
[Composite]     GPS Date/Time                   : 2013:03:31 07:09:43Z
[XMP-exif]      Date/Time Original              : 2013:03:31 07:09:43
[ExifIFD]       Date/Time Original              : 2013:03:31 07:09:43
======== ./DSC_23876_EpochGPSdate.nef
[System]        File Name                       : DSC_23876_EpochGPSdate.nef
[System]        File Modification Date/Time     : 2019:11:04 18:08:57+00:00
[XMP-exif]      GPS Date/Time                   : 2013:03:31 12:01:58
[Composite]     GPS Date/Time                   : 2000:01:01 11:01:58.58Z
[XMP-exif]      Date/Time Original              : 2013:03:31 12:01:58
[ExifIFD]       Date/Time Original              : 2013:03:31 12:01:58
======== ./_DSC4534_noGPS.nef
[System]        File Name                       : _DSC4534_noGPS.nef
[System]        File Modification Date/Time     : 2019:03:06 04:25:42+00:00
[XMP-exif]      Date/Time Original              : 2019:03:05 22:25:41-06:00
[ExifIFD]       Date/Time Original              : 2019:03:05 22:25:41
    1 directories scanned


[Phil Harvey]

There's your answer.  Ah, right.  GPSDateTime is a Composite tag.  You need to write GPSDateStamp and GPSTimeStamp (the tags from which Composite:GPSDateTime is derived) instead of GPSDateTime:

exiftool -if "$gpsdatetime =~ /^2000:01:01/" -overwrite_original -F "-gpsdatestamp<datetimeoriginal" "-gpstimestamp<datetimeoriginal" .

- Phil

[sidneyd]

Hi Phil, thanks this proved really helpful  - yes it was a composite tag

For peoples reference looking at this thread at a later date:

I used the following to locate inconsistencies in my GPS data and thought I should reply to confirm this as it could prove helpful for others who may stumble upon this thread from Google as a way to validate their GPS data.

Echo Reporting No GPSdata
exiftool.exe -r -if "not $gpslatitude"       -filename  -gpsdatetime . > GPS_no_gps.log
Echo Reporting No GPSdate
exiftool.exe -r -if "not $gpsdatetime"        -filename  -gpsdatetime . > GPS_no_gpsdate.log
Echo Reporting GPSdate set to Epoch 1 January 2000
exiftool.exe -r -if "$gpsdatetime =~ /^2000:01:01/"    -filename  -gpsdatetime . > GPS_gpsdate_settoepoc.log


I then can process files in a given directory using the following command to overwrite  the Epoch GPS data.

exiftool.exe -v0 -r -if "$gpsdatetime =~ /^2000:01:01/"  -overwrite_original -preserve -F "-gpsdatestamp<datetimeoriginal" "-gpstimestamp<datetimeoriginal"  .


Again Phil thanks very much for your help and guidance.

[StarGeek]

I then can process files in a given directory using the following command to overwrite  the Epoch GPS data.
exiftool.exe -v0 -r -if "$gpsdatetime =~ /^2000:01:01/"  -overwrite_original -preserve -F "-gpsdatestamp<datetimeoriginal" "-gpstimestamp<datetimeoriginal"  .

One thing to point out.  GPS timestamps are supposed to be in UTC and this command does not compensate for this.  To get an accurate setting you would need to either add the timezone
exiftool.exe -v0 -r -if "$gpsdatetime =~ /^2000:01:01/"  -overwrite_original -preserve -F "-gpsdatestamp<${datetimeoriginal}-07:00" "-gpstimestamp<${datetimeoriginal}-07:00"  .
or if the timezone has already been set in EXIF:OffsetTimeOriginal, then this could be used
exiftool.exe -v0 -r -if "$gpsdatetime =~ /^2000:01:01/"  -overwrite_original -preserve -F "-gpsdatestamp<SubSecDateTimeOriginal" "-gpstimestamp<SubSecDateTimeOriginal"  .

[sidneyd]

Thanks for the advice StarGeek, yes I do work to set all of my timestamps to UTC.