Set alldates from filename but ignore some numbers in the name

[wumpus]

My scanner names files IMG_YYYYMMDD_9999 (where IMG is text of my choosing, YYYY is 4digit year, MM is 2digit month, DD is 2digit day of the scan date and 9999 is an incrementing number starting at 0001). The only control I have over this format is the beginning text.

The command exiftool -v "-alldates<filename" IMG_20170911_0001.tif fails as exiftool seems to interpret the incrementing number as the time. It doesn't actually error but it set the time to 00:01:00, which I don't want.

How do I get exiftool to ignore the incrementing number in the filename and just set the time to zeroes? I've tried several different formats in the "-d" option but I can't figure it out.

Also, is there a way to set the timezone used by alldates in this same command? I've tried -d"%Y%m%d_....-05:00" which I really didn't think was going to work, and it didn't, but I'm grabbing at straws here.

[StarGeek]

This gets complicated so if you don't want the technical details, skip to the tl;dr at the bottom.

The command exiftool -v "-alldates<filename" IMG_20170911_0001.tif fails as exiftool seems to interpret the incrementing number as the time. It doesn't actually error but it set the time to 00:01:00, which I don't want.

Yes, the date/time stamps pretty much require a time.  Well, it's a bit more nuanced then that.  You can force exiftool to write it without but you probably won't find a program that can read the dates, at least, I have yet to find one.

How do I get exiftool to ignore the incrementing number in the filename and just set the time to zeroes? I've tried several different formats in the "-d" option but I can't figure it out.

You'll have to use advanced formatting to grab the first 8 digits (YearMonthDay) and then pad zeroes.  For that you could use something like ${filename;s/_(\d{8})_.*/$1/} 000000.

Also, is there a way to set the timezone used by alldates in this same command? I've tried -d"%Y%m%d_....-05:00" which I really didn't think was going to work, and it didn't, but I'm grabbing at straws here.

This is where things get complicated and you run into the differences between the types of metadata.  AllDates is a shortcut for three specific tags, DateTimeOriginal, CreateDate, and ModifyDate.  When writing these three tags, exiftool will give priority to writing them in the EXIF block.  But they also can exist in the XMP block, but exiftool won't write the dates there unless the tags already exist there or you are explicit in writing those tags.

The trouble is that timestamp tags in EXIF don't allow for timezones.  There is an unofficial tag (TimeZoneOffset) that's been around for a while and recently (last year?) three official timezone tags (OffsetTimeDigitized, OffsetTimeOriginal, and OffsetTime).  I'm not sure how well any of them are supported by the various programs out there.  But the fact that they are separate tags means they need to be assigned a value separately.

On the other hand, the XMP tags do allow for timezones in the timestamps.  But since XMP is a newer standard, some older programs won't read the data (though I think Lightroom, for example, will give the XMP data priority).

So you may have to test what programs you use to see what data they read.  But an overall command to write all this info, just to cover all the bases:

tl;dr
exiftool "-AllDates<${filename;s/_(\d{8})_.*/$1/} 00:00:00" "-xmp:AllDates<${filename;s/_(\d{8})_.*/$1/} 00:00:00+05:00" OffsetTimeDigitized=+05:00 -OffsetTimeOriginal=+05:00 -OffsetTime=+05:00 FileOrDir

Wrapup:
First, an explanation of the various bits of the above command.  The first AllDates writes only to the EXIF tags.  The second one will write to the XMP tags and includes the timezone.  This was a cool discovery for me, as I didn't realize I could specify XMP with AllDates.  The next three write to all three of the timezone tags.  I didn't include the unofficial EXIF timezone tag as that has to be written differently and can include two different offsets (don't ask, but if you have to, read the notes on the EXIF tags page).

Now, that makes for a long and messy looking command line.  If you want, you can clean it up by creating your own shortcut to do this all in one shot.  You have to create a config file (see the example config file for instructions).  Find the line %Image::ExifTool::UserDefined::Shortcuts  = ( and add this line right after it.

Code Select Expand

    MyAllDates => ['EXIF:DateTimeOriginal','EXIF:CreateDate','EXIF:ModifyDate','XMP:DateTimeOriginal','XMP:CreateDate','XMP:ModifyDate','OffsetTimeDigitized','OffsetTimeOriginal','OffsetTime'],

If done correctly, your command can now be reduced to
exiftool "-MyAllDates<${filename;s/_(\d{8})_.*/$1/} 00:00:00+05:00" FileOrDir

Example output:

Code Select Expand

C:\>exiftool -g1 -a -s -myalldates "Y:\X_Drive\!temp\IMG_20170911_0001.jpg"

C:\>exiftool -P -overwrite_original "-MyAllDates<${filename;s/_(\d{8})_.*/$1/} 00:00:00+05:00" "Y:\X_Drive\!temp\IMG_20170911_0001.jpg"
    1 image files updated

C:\>exiftool -g1 -a -s -myalldates "Y:\X_Drive\!temp\IMG_20170911_0001.jpg"
---- IFD0 ----
ModifyDate                      : 2017:09:11 00:00:00
---- ExifIFD ----
DateTimeOriginal                : 2017:09:11 00:00:00
CreateDate                      : 2017:09:11 00:00:00
OffsetTime                      : +05:00
OffsetTimeOriginal              : +05:00
OffsetTimeDigitized             : +05:00
---- XMP-exif ----
DateTimeOriginal                : 2017:09:11 00:00:00+05:00
---- XMP-xmp ----
CreateDate                      : 2017:09:11 00:00:00+05:00
ModifyDate                      : 2017:09:11 00:00:00+05:00

Yes, your post made me decide to work up this shortcut and add it to my own config file.  So I learned something new and created something new.  I'm a happy camper!

[wumpus]

Wow. You weren't kidding.

It's gonna take me a bit to grok this. But none the less, thank you very much!

I was using alldates as a quick example. I'm actually interested in setting DateTime, DateTimeOriginal and DateTimeDigitized (initially to the same value, then I'll modify the first two after I determine the actual date of the photo). I didn't realize EXIF dates didn't support TZ info. GeoSetter complains about not having TZ in my dates and adds it. I'll have to research what it's actually setting.

My env:
Win10 x64 Pro
Canon 9000F Mark II
TIFF image files
ExifTool v10.61
Exif Pilot v5.1.0
GeoSetter v3.4.53

[StarGeek]

I was using alldates as a quick example. I'm actually interested in setting DateTime, DateTimeOriginal and DateTimeDigitized

One thing to take note of is that what the EXIF spec calls DateTime, exiftool calls ModifyDate.  The same for DateTimeDigitized, which is CreateDate in exiftool.  So AllDates is actually what you want.

If it makes a difference, the official EXIF timezone each refer to a different timestamp.  OffsetTime is time zone for ModifyDate, OffsetTimeOriginal is time zone for DateTimeOriginal, and OffsetTimeDigitized is time zone for CreateDate.

GeoSetter complains about not having TZ in my dates and adds it. I'll have to research what it's actually setting.

GeoSetter is great software.  It'll set everything correctly.  As for the timezone, it will set the unofficial EXIF timezone tag, TimeZoneOffset.  It hasn't been updated to set the official ones as of Beta 3.4.53, though.

But then, if you're using GeoSetter, then you shouldn't have to worry about setting the timezone because it will set the timezone properly when you geolocate the image.  It will also read the XMP data (which can hold the timezone) and copy the timestamps to the appropriate spots in EXIF.

[wumpus]

WOOHOO! Works like a champ. Thank you. Thank you.

I tried to get fancy and add TimeZoneOffset to my shortcut in support of GeoSetter's usage:

Code Select Expand


MyAllDateTimes => ['EXIF:DateTimeOriginal','EXIF:CreateDate','EXIF:ModifyDate','XMP:DateTimeOriginal','XMP:CreateDate','XMP:ModifyDate','OffsetTimeDigitized','OffsetTimeOriginal','OffsetTime','TimeZoneOffset'],


But it did not add TimeZoneOffset. I didn't get any errors either so I guess it's just ignored 'cause it's not a standard?

EDIT: Nevermind. I just re-read your reply above about the difference to set the unsupported TimeZoneOffset. muhbad.

[wumpus]

On a somewhat related note.

This works great to set the desired metadata in my original .tif files. So I set my dates, times and timezone offsets using exiftool; then I set the GPS coords with GeoSetter. Life's good ... except:

Now I want to batch convert some/all TIFFs to JPEGs for sharing/posting. When I run my new TIFFs through ImageMagick's command line interpreter to convert TIFF to JPEG, I get errors and it strips out the unknown tags.

Code Select Expand


C:\TEMP\test>convert -quality 95 IMG_20170911_0002.tif IMG_20170911_0002.jpg
convert: Unknown field with tag 36880 (0x9010) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/925.
convert: Unknown field with tag 36881 (0x9011) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/925.
convert: Unknown field with tag 36882 (0x9012) encountered. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/925.

C:\TEMP\test>convert --version
Version: ImageMagick 7.0.6-0 Q16 x64 2017-06-11 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2015 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Visual C++: 180040629
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib cairo flif freetype jng jp2 jpeg lcms lqr openexr pangocairo png ps rsvg tiff webp xml zlib


Are you aware of a batch tool to convert TIFF to JPEG that won't strip out the tags?

[StarGeek]

That's annoying.  If you check the hex codes on those "Unknown fields" (see EXIF tag names), you'll see that those are the three official timezone tags.

There is a slightly newer version of ImageMagick (7.0.7) but I wouldn't be too hopeful.  The next step I guess would be bring it up on their forums, but again, I doubt there would be any speed on getting it done.

On the other hand, maybe take a look at Irfanview.  While mostly a image viewer, it can convert files and can do it through the command line.  Even better, it looks like it can take wildcards.  My quick test showed that it copies the EXIF block without editing as well as IPTC and XMP.

i_view64.exe C:\PathToImages\*.tif /convert=C:\PathToImages\*.Jpg /jpgq=95

[wumpus]

Vera nice. I will definitely give it a try.

There is a slightly newer version of ImageMagick (7.0.7) but I wouldn't be too hopeful.  The next step I guess would be bring it up on their forums, but again, I doubt there would be any speed on getting it done.

Submitted: https://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=32679

My next observation

This is all so I can archive (TIFF) our really old family photos and upload (JPEG) some of them to Google Photos and easily share them.

An extremely sub-optimal thing about Google Photos. If the timestamp on your photo is before 4:00A, it lists it as the previous day in the "Photos" view. But if you examine the particular pic you'll see that the date and time are actually correct. They're just interpreting them wrong. I found that it's been reported via their support forum since last year (https://support.google.com/photos/forum/AAAAzDQC9KEavU6UtmXDsU?hl=en), but they haven't fixed it. 

Hopefully this will be the last off-topic post I put in this thread. 

[StarGeek]

But it did not add TimeZoneOffset. I didn't get any errors either so I guess it's just ignored 'cause it's not a standard?

EDIT: Nevermind. I just re-read your reply above about the difference to set the unsupported TimeZoneOffset. muhbad.

Yeah.  Since this thread has become pretty extensive, here's some of the details, just in case someone in the future wants to know.

For the official timestamps, exiftool wants an actual full timezone.  +5, +5:0,  isn't good enough. It wants the plus/minus, hours, and minutes.

TimeZoneOffset on the other hand, is weird.  It requires an integer.  Or two.  Or more.  Only the first two have definitions.  The first number corresponds to OffsetTimeOriginal and the second to OffsetTime.

But as I said, it has to be an integer.  Which breaks down if you're in an unusual time zone.  For example, from that page, India uses IST, which would be +5:30 and you can't represent that with an integer.

So when you tell exiftool to copy +5:00 to TimeZoneOffset, it's doesn't do the conversion, so it didn't get written.  You could write it separately if so desired.

[StarGeek]

Vera nice. I will definitely give it a try.

I may have to step back from the Irfanview recommendation.

While playing with the conversion, I discovered that when it copies the EXIF block, it doesn't update some of the internal stuff in the block and exiftool comes back with a Error: Error reading StripOffsets data in IFD0 error when I tried to write new data to the image.

But the error seems to be easily fixed with the command from FAQ 20 without any apparent loss of data.

I found that it's been reported via their support forum since last year (https://support.google.com/photos/forum/AAAAzDQC9KEavU6UtmXDsU?hl=en), but they haven't fixed it.

Yeah, Google has always been iffy on fixing things like that, in my experience.  I was very surprised when I recently figured out that Google Photos can now read metadata from PNG files, after years of no changes.

Hopefully this will be the last off-topic post I put in this thread. 

It's all good, useful info, off topic or not.  Again, I'm thinking of when it might help out somebody finding this thread at a later date.

[wumpus]

While playing with the conversion, I discovered that when it (Irfanview) copies the EXIF block, it doesn't update some of the internal stuff in the block and exiftool comes back with a Error: Error reading StripOffsets data in IFD0 error when I tried to write new data to the image.

Paint.net does the same thing. I open a .tif and save it as a .jpg and I get the same error from exiftool later.

[StarGeek]

Another option might be nconvert, which is the command line version of XnView.  It appears to write the EXIF block correctly and doesn't delete the EXIF timezone tags.  But the docs are... poor to say the least.

[wumpus]

Another option might be nconvert, which is the command line version of XnView.  It appears to write the EXIF block correctly and doesn't delete the EXIF timezone tags.  But the docs are... poor to say the least.

The latest version of nconvert uses LibJPEG 6b - http://www.ijg.org whose latest version is 9b of 17-Jan-2016 

But it seems to work (assuming -comp_ratio is what I think it is):

Code Select Expand


C:\TEMP\test>nconvert -out jpeg -comp_ratio 95 IMG_20170911_0001.tif
** NCONVERT v7.10 (c) 1991-2017 Pierre-E Gougelet (Sep  5 2017/11:04:24) **
        Version for Windows NT/9x/2000/Xp/Vista/7  (All rights reserved)
** This is freeware software (for non-commercial use)

Over...
Over...
Over...
Over...
Over...
Over...
Over...
Over...
Conversion of IMG_20170911_0001.tif into IMG_20170911_0001_1.jpg OK


and all the tags are intact.

Maybe I'll write something in Perl. Give me a chance to give back... sorta (since I'll just use other's modules).

[StarGeek]

The latest version of nconvert uses LibJPEG 6b - http://www.ijg.org whose latest version is 9b of 17-Jan-2016 

For what it's worth, according to Wikipedia, the later releases just introduced incompatible format extensions.  And my copy of Covert ver 7.05 (convert -list format) says it's using Joint Photographic Experts Group JFIF format (62), which I assume is 6.2.