Date and time metadata problem

[LisaTedeschi]

Hello to everyone,
this is my first post in this forum so please excuse me for eventual errors.
I am currently working with a big set of images from camera traps. I want to organize them using the R package CamtrapR (https://cran.r-project.org/web/packages/camtrapR/index.html) but some images have a metadata problem.

1) First, R sends this message:

Error in if (all(intable$DateTimeOriginal == "-")) { :
  missing value where TRUE/FALSE needed
In addition: Warning message:
In system(command.tmp, intern = TRUE) :
  running command 'exiftool -q -f -t -r -Directory -FileName -EXIF:DateTimeOriginal -HierarchicalSubject -ext JPG ""' had status 1

2) So I opened ExifPro to take a look to the images, and they have the correct date and time but for some reason we cannot read them (as you can see in the "Img1" that I attached).
3) I took a look at all the metadata with ExifTool (as you can see in the "Img2" that I attached): on the left side there is one of the "problematic" images, and on the right side a normal one. I noticed that in the left image the original date and time metadata is in the comment section, and in general the metadata are messed up.
4) The images were taken with a Reconyx camera: knowing they can have problems writing the metadata, I tried the 'fixDateTimeOriginal' function implemented in the CamtrapR package on R but it didn't work.
5) I tried the 'fixHyperfire' function { Tobler, Mathias (2015). Camera Base Version 1.7 http://www.atrium-biodiversity.org/tools/camerabase/files/CameraBaseDoc1.7.pdf } but it didn't work.

I am looking everywhere but I am running out of ideas. Anyone has experienced the same problem?

Best regards,
Lisa Tedeschi

[Phil Harvey]

In system(command.tmp, intern = TRUE) :
  running command 'exiftool -q -f -t -r -Directory -FileName -EXIF:DateTimeOriginal -HierarchicalSubject -ext JPG ""' had status 1

There is no file name in the command (empty string "").

- Phil

[StarGeek]

Looking at the data listed in the second image, it appears that the first image has been processed by some other program which has removed a lot of the EXIF data and placed it in the Comment.  Adobe Photoshop is a name that appears there, but I would think it was unlikely that Adobe would do this.

I would suggest that you be careful because the Comment is relatively fragile, as there are a lot of programs that would overwrite this tag without regards to what was previously there. Make sure you have a backup until it's fixed.

As for fixing the date, test out this command on a copy of a file to see if it works.
exiftool -P "-DateTimeOriginal<${Comment;s/Dat: (\d{4}-\d\d-\d\d) (\d\d)(:\d\d:\d\d) ([AP]M).*/$1 $2$3 $4/;s/ (\d\d)(?=:\d\d:\d\d PM)/${1} +12 /ei}" FILE

Replace FILE with the name of the test file.

I'm not sure what the rest of the data in comment is.  Brt maybe brightness?  Shp, Sharpness?  Ill, Illumination or flash(?) since it seems to be set to OFF. 

[LisaTedeschi]

Dear Dr. Harvey, thanks for your quick answer. Yes, actually the working directory I set was wrong. Now it's working.

Dear StarGeek, thanks for answering. Your code worked! I put *.JPG at the end, to have the modification in all the pictures of the folder.
Now I have two other problems:
1) the code didn't work for all the pictures; here's the error message in R:

C7 : 1499 images
Warning: Could not read Exif DateTimeOriginal tag of 210 image(s) at station C7 . Will omit them. Consider checking for corrupted Exif metadata.

2) seems like it didn't discriminate between a.m. and p.m.. If a picture was taken at 01:00 p.m., now the time displayed is 01:00.

Thanks a lot for your help and your time and sorry for the naive questions.

Best regards,
Lisa Tedeschi

[StarGeek]

2) seems like it didn't discriminate between a.m. and p.m.. If a picture was taken at 01:00 p.m., now the time displayed is 01:00.

Might be a spacing issue.  Hard to tell exactly with a screen shot.  For one of these PM pictures, can you run this command:
exiftool -comment FILE |clip

This will output the results of exiftool directly to the clipboard.  Then you can just paste the results here and the command can be refined.

As for the problem with 210 files it couldn't read, if you could find one of them, could you post it?

[LisaTedeschi]

Dear StarGeek, thanks again for your quick answer.
Here I will post all the screenshot: Img3 is the output of R, Img4 and Img5 shows a picture where the code had effect (IMG_0004) and one where the code didn't have effect (IMG_0010), Img6 is the "comment" section of ExifTool of both imagenes. Img7 is me trying to run the last command you wrote (and right now it's not working).

Best regards,
Lisa Tedeschi

[StarGeek]

I'm sorry if I wasn't clear.  Screen shots don't necessarily give me the info I need.  the addition of |clip copies the output directly onto the clipboard, so you can come here and just rightclick->paste the results.  That way I can copy the results and test things here.

Img7 was working.  There is no output from exiftool because it ends up on the clipboard.

[LisaTedeschi]

Dear StarGeek, you were perfectly clear, it's just that I am very naive with this. Anyway, here you have the output of a picture with the PM issue (IMG_0015), and below the output of a picture where the code didn't work (IMG_0009).

Comment                         : Trg: M 5/5.Evt: 3.Dat: 2017-10-25 1:01:47 PM.MP:  1.Tmp: 30 C.Bat: 8870.Ill: Off.Exp: 000A,320C,0.Ctr: 159.Brt: 95.Shp: 64.Sat: 191.PIR: 25.Nam: IMG_0015.JPG.Tot: 0000D65E.SN:  RC31AA10001644.Ver: 1.2.0.20070917.Rst: 0,1.Pwu: 0018,001F,002B,3124,3143,32D0,32D0.Lht: 002E,0084,01EB,0580.Sce: 1,4F.Seg: 0,34,6D,59,6B,52,34,3B,3E,4B,48,5B,6D,67,5F,3B,51,4C,39.Agc: 4F,000A,000A,0000,0108,2100.Cfg: 0003004619050A01000503E8003CFFFFFFFFFFFFFFFFFFFFFFFF00000005000000000000000000000000C040003E808031453030203220202020202020202020000000005C027EE046A5.

Comment                         : Trg: M 4/5.Evt: 2.Dat: 2017-10-25 12:56:34 PM.MP:  1.Tmp: 30 C.Bat: 8870.Ill: Off.Exp: 0019,21A3,1.Ctr: 159.Brt: 95.Shp: 64.Sat: 191.PIR: 25.Nam: IMG_0009.JPG.Tot: 0000D658.SN:  RC31AA10001644.Ver: 1.2.0.20070917.Rst: 0,1.Pwu: 0018,001F,002B,1F47,1F4B,2268,2268.Lht: 0035,0098,0233,0659.Sce: 1,58.Seg: 0,1A,AF,1A,1B,3B,33,36,5C,76,88,91,7D,AF,A8,36,42,40,3F.Agc: 58,0019,0016,0000,0108,2100.Cfg: 0003004619050A01000503E8003CFFFFFFFFFFFFFFFFFFFFFFFF00000005000000000000000000000000C040003E808031453030203220202020202020202020000000005C027EE046A5.

Best regards,
Lisa Tedeschi

[StarGeek]

Ah, I see the problem with the PM.  I assumed that there would be 2 digits for the hour, but instead of 01, it's just 1.

Just needs a slight tweak for that
exiftool -P "-DateTimeOriginal<${Comment;s/Dat: (\d{4}-\d\d-\d\d) (\d\d?)(:\d\d:\d\d) ([AP]M).*/$1 $2$3 $4/;s/ (\d\d?)(?=:\d\d:\d\d PM)/${1} +12 /ei}" FILE

If you have backups elsewhere, you can add the Overwrite_Original option to the command to avoid creating the "_original" backup files.  You can also use exiftool -delete_original! DIR to remove the "_Original" files.

For IMG_0009.jpg, it seems likely that the EXIF block is damaged.  Run this command and copy/paste the output here
exiftool -validate -warning -a IMG_0009.jpg

If you wish to try the next step while waiting for a response, take a look at FAQ #20.   Running the command in FAQ 20 might fix the 210 images by itself or it might also take running the above command afterwards as well.  Again, make sure you test on a copy first.

[LisaTedeschi]

Dear StarGeek, the code for the PM problem worked.

For IMG_0009 (and the others like that) I tried the code of the FAQ#20 (and also the one from FAQ#15) but it seems they are not working (idk if I'm missing something). Here there's the output of IMG_0009:

Validate                        : 3 Warnings (1 minor)
Warning                         : [minor] Hour out of range for EXIF:DateTimeOriginal
Warning                         : Missing required JPEG ExifIFD tag 0xa002 ExifImageWidth
Warning                         : Missing required JPEG ExifIFD tag 0xa003 ExifImageHeight

Best regards,
Lisa Tedeschi

[StarGeek]

Hour out of range for EXIF:DateTimeOriginal

Ah, that's the likely problem.  It has something incorrect in the hour spot.  What is the output of
exiftool -DateTimeOriginal -comment IMG_0009.jpg

I'm mostly interested in the hour, but want to look at the comment as well to see if the data there can be used to fix the hour or if that's where the hour went wrong in the first place.

[LisaTedeschi]

Dear StarGeek, here is the output.

Date/Time Original              : 2017:10:25 24:56:34
Comment                         : Trg: M 4/5.Evt: 2.Dat: 2017-10-25 12:56:34 PM.MP:  1.Tmp: 30 C.Bat: 8870.Ill: Off.Exp: 0019,21A3,1.Ctr: 159.Brt: 95.Shp: 64.Sat: 191.PIR: 25.Nam: IMG_0009.JPG.Tot: 0000D658.SN:  RC31AA10001644.Ver: 1.2.0.20070917.Rst: 0,1.Pwu: 0018,001F,002B,1F47,1F4B,2268,2268.Lht: 0035,0098,0233,0659.Sce: 1,58.Seg: 0,1A,AF,1A,1B,3B,33,36,5C,76,88,91,7D,AF,A8,36,42,40,3F.Agc: 58,0019,0016,0000,0108,2100.Cfg: 0003004619050A01000503E8003CFFFFFFFFFFFFFFFFFFFFFFFF00000005000000000000000000000000C040003E808031453030203220202020202020202020000000005C027EE046A5.

Best regards,
Lisa Tedeschi

[StarGeek]

I see the problem now.  In my original command, I didn't account for the fact that 12 AM was midnight (00 hour) and 12 PM was noon (12 hour).  So this mess up the time for your images taken at those times.  That complicates things. 

I'm trying to be very careful with fixing this as my previous command messed up the time of any file that was taken at 12 midnight.  If you know you don't have any of those, then there's less to worry about.  The object is to shift the time of all the images taken at 12 AM and 12 PM backwards by 12 hours, so that 12 AM becomes 00 instead of 12 and 12 PM becomes 12 instead of 24.

Try this to correct the times
exiftool -if "$comment=~/Dat: \d{4}-\d\d-\d\d 12/" -DateTimeOriginal-=12

[LisaTedeschi]

Dear StarGeek, again, it's working!
I tried it also within all the folder (and within other folders too) but I noticed it fixs only one image.
Is there a way to use this to fix all the images in a folder that were not fixed by the command exiftool -P "-DateTimeOriginal<${Comment;s/Dat: (\d{4}-\d\d-\d\d) (\d\d?)(:\d\d:\d\d) ([AP]M).*/$1 $2$3 $4/;s/ (\d\d?)(?=:\d\d:\d\d PM)/${1} +12 /ei}" FILE ?
Because just in this folder I have 210 pictures taken at 12 AM or 12 PM, and more and more folders.

Best regards,
Lisa Tedeschi

[StarGeek]

Ok, I think I got it

Try
exiftool -if "$Comment=~/Dat: \d{4}-\d\d-\d\d \d\d?:\d\d:\d\d [AP]M/" "-DateTimeOriginal<${Comment;m/Dat: (\d{4}-\d\d-\d\d )(\d\d?)(:\d\d:\d\d) ([AP]M)/;$_=$1.(($2%12)+(($4 eq 'AM') ? ' 00' : ' 12')).$3}"

This can be used on any of the files.  It will:
* Only run on files that contain the "Dat:" data in comment.
* extract the date from comment
* If the hour is 12, it is changed to 0
* If it is PM, then 12 is added to the hour.  This means that 12 AM is changed to 0 (midnight), but if it is 12 PM, it will end up as 12 (noon).  All other AM numbers will be unchanged and all PM numbers will have be increased by 12, ending up as the correct time.

This should work correctly and I'm adding this bit to my bag o' tricks for future use.

Edit: slight tweak to guarantee the space between the date and the time