Hi,
I've been helping a family member transfer old videos from tape to Apple's Photo app. When importing these videos we found out that while the filename contains the date and time the video was taken, the metadata tag used by the Photos app contained the import date. To fix this, ExifTool looks like a great option.
I've used the following command on a folder containing the videos:
exiftool "-allDates<filename" .
Since we had only limited space, after some testing we decided to use the -overwrite option. This turned out to be a horrible mistake, as only after 1600 family video files were altered, I found out all videos turn into random color blocks (I've attached a screenshot) accompanied by white noise after the 10 second mark. (I ran only test on small files, less than 10 seconds long so never encountered this problem).
I'm wondering what I did wrong and hoping there's some way to repair this. Any help in recovering 20 years worth of family videos is greatly appreciated. I have very little knowledge of video files and metadata so I honestly have no idea where to even start looking. If any specific details help, please let me know so I can provide them.
The files are .mov files and were stored on an external hard drive. After running exiftool, no error messages were displayed.
Are all 1600 corrupted? Have you viewed any off the corrupted files in their uncorrupted form? I ask this as you need to determine if they were uncorrupted before using exiftool.
Quote from: Alan Clifford on June 10, 2018, 03:42:24 AM
Are all 1600 corrupted? Have you viewed any off the corrupted files in their uncorrupted form? I ask this as you need to determine if they were uncorrupted before using exiftool.
Hi Alan,
As far as I can tell, all 1600 files are affected. There are some videos that are no longer than 10 seconds that display no symptoms. I have not found a video longer than 10 seconds that is not corrupted after the 10 second mark.
We still have the _original files from the first try (before the drive started to fill up and we used the -overwrite option). Those videos, including the ones over 10 seconds long, are fine.
Send me a (small) sample with this problem and I'll see what I can do. My email is philharvey66 at gmail.com
This should be fixable as long as the videos were OK before you ran ExifTool, and you didn't alter them with anything else afterwards, provided that they weren't corrupted due to a failing hard disk.
- Phil
Quote from: Phil Harvey on June 10, 2018, 10:23:49 AM
This should be fixable as long as the videos were OK before you ran ExifTool, and you didn't alter them with anything else afterwards, provided that they weren't corrupted due to a failing hard disk.
That's great news, I've sent you an e-mail.
Thanks!
Great. I'll look at it as soon as I can, but I may not get a chance to do it until tomorrow.
- Phil
I got the files and will be able to fix this problem.
The reason that this happens is because the video data in these files is split into two separate blocks. I have never seen this before, and was under the impression that it must be in a single block. So the offsets into the second block are getting messed up when writing with ExifTool. This will take a fairly substantial re-write of ExifTool's QuickTime writing algorithms to avoid this problem, and some additional work to be able to fix your damaged videos, but I should be able to accomplish this within a few days.
I'm assuming you're on a Mac... What app did you use to create the original videos?
I'll release a new version of ExifTool with this problem fixed as soon as possible, along with a utility for you to repair your damaged files.
Thanks for reporting this, and for providing samples so I could reproduce the problem!
- Phil
Edit: In the mean time I have released ExifTool 11.01 which will avoid adding tags to videos with multiple mdat blocks.
OK. I have prepared an ExifTool 11.02 pre-release (https://exiftool.org/Image-ExifTool-11.02p.tar.gz) that fixes the corrupted MOV you sent me. This should fix all your videos if they are corrupted in the same way.
Here are the instructions:
1. Unpack the tarball on your Desktop
2. Run this command
~/Desktop/Image-ExifTool-11.02/exiftool -api FixCorruptedMOV -xmp= FILE
where FILE is the name of a corrupted MOV file, or a directory containing corrupted MOV files.
Let me know how it goes (fingers crossed).
- Phil
BTW, The original command you should have used to change the times was this:
exiftool "time:all<filename" -wm w .
Doing this will change all existing writable date/time tags according to the file name. The command you used only changed/added common EXIF date/time tags. It was the adding of new tags in a MOV with multiple 'mdat' blocks that resulted in the corruption. The command above doesn't add new tags, so if you had done this we wouldn't have discovered this bug in ExifTool. I'm glad to have found the bug, but from your point of view things would have been much less painful if we hadn't. Sorry for that.
- Phil
Thanks again for taking the time to look at the problem and coming up with a solution this quickly. Good news: after running ExifTool using the command you posted, all videos are back to normal!
We did encounter some errors:
$ ~/Desktop/Image-ExifTool-11.02/exiftool -api FixCorruptedMOV -xmp= /Volumes/Extern/Filmpjes/
Error: Not a valid MOV - /Volumes/Extern/Filmpjes/._2002-05-22 22_19_57-1.mov_original.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-23 13_03_58.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-23 17_49_34.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-24 21_31_58.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-25 01_03_27.mov
Error: Not a valid MOV - /Volumes/Extern/Filmpjes/._2002-05-25 01_03_27.mov_original.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-07-27 11_38_00.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-08-09 13_23_31.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-08-16 13_47_13.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2005-05-21 14_37_39.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2005-07-11 13_36_51.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2006-08-09 17_28_23.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2007-09-21 20_50_43-1.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2008-10-12 16_52_05.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2009-04-08 21_05_56.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2009-05-10 12_39_27.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_37_42.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_52_14.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_53_18.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_57_55.mov
1 directories scanned
1509 image files updated
2 image files unchanged
20 files weren't updated due to errorsWe checked the listed videos, but they seemed to be fine. Only after arriving back home, I realized these might actually have been hidden files instead of building a path to a file incorrectly (e.g. there was a file called "2002-05-23 13_03_58.mov" that was fixed, but I didn't check if "._2002-05-23 13_03_58.mov" exists. If it helps, I could check if these files actually exist.
Quote from: Phil Harvey on June 11, 2018, 07:42:45 AM
The reason that this happens is because the video data in these files is split into two separate blocks. I have never seen this before, and was under the impression that it must be in a single block. So the offsets into the second block are getting messed up when writing with ExifTool. This will take a fairly substantial re-write of ExifTool's QuickTime writing algorithms to avoid this problem, and some additional work to be able to fix your damaged videos, but I should be able to accomplish this within a few days.
I'm assuming you're on a Mac... What app did you use to create the original videos?
I didn't do it personally, but iMovie was used to import the videos from a camera using miniDV tapes connected to the laptop.
Quote from: Phil Harvey on June 11, 2018, 10:41:25 AM
BTW, The original command you should have used to change the times was this:
exiftool "time:all<filename" -wm w .
Doing this will change all existing writable date/time tags according to the file name. The command you used only changed/added common EXIF date/time tags. It was the adding of new tags in a MOV with multiple 'mdat' blocks that resulted in the corruption. The command above doesn't add new tags, so if you had done this we wouldn't have discovered this bug in ExifTool. I'm glad to have found the bug, but from your point of view things would have been much less painful if we hadn't. Sorry for that.
Out of curiousity I just tried this command on one of the original test files I have available, but got the following output:
$ exiftool "time:all<filename" -wm w .
File not found: time:all<filenameAdding a dash didn't help either:
$ exiftool "-time:all<filename" -wm w .
Warning: No writable tags set from ./2008-05-01 16_48_37-1.mov
1 directories scanned
0 image files updated
1 image files unchangedAdding the "write only existing tags" option to the command I used seems to do what you described:
$ exiftool "-allDates<filename" -wm w .
1 directories scanned
1 image files updatedSome more background information: Apple's Photos app uses the
Quicktime:CreationDate tag. According to your documentation, ExifTool can't write this tag, but I found a script that can. Since that script reads the
CreateDate tag, I wanted to use ExifTool to use the filename as the
CreateDate tag.
This is the script I used:
https://github.com/HayoBaan/matchDateTime (https://github.com/HayoBaan/matchDateTime)
Quote from: dgltcg on June 11, 2018, 06:21:41 PM
This is the script I used:
https://github.com/HayoBaan/matchDateTime (https://github.com/HayoBaan/matchDateTime)
HayoBaan is a mod here. He created that script back in 2016 (https://exiftool.org/forum/index.php/topic,6938.msg34848.html#msg34848) and keeps it up to date.
Quote from: dgltcg on June 11, 2018, 06:21:41 PM
Thanks again for taking the time to look at the problem and coming up with a solution this quickly. Good news: after running ExifTool using the command you posted, all videos are back to normal!
Great!
QuoteWe did encounter some errors:
These are all with the "._" files the MacOS creates on non-mac filesystems to store the extended attributes. They really shouldn't be processed anyway.
QuoteOnly after arriving back home, I realized these might actually have been hidden files
Yes, they were. ExifTool currently doesn't ignore filenames starting with "._", but maybe it should.
QuoteOut of curiousity I just tried this command on one of the original test files I have available, but got the following output:
$ exiftool "time:all<filename" -wm w .
File not found: time:all<filename
My mistake. I forgot a "-":
exiftool "-time:all<filename" -wm w .QuoteAdding a dash didn't help either:
$ exiftool "-time:all<filename" -wm w .
Warning: No writable tags set from ./2008-05-01 16_48_37-1.mov
1 directories scanned
0 image files updated
1 image files unchanged
OK, then maybe none of the existing date/time tags are writable. What is the output of this command?:
exiftool -G1 -a time:all FILEQuoteAdding the "write only existing tags" option to the command I used seems to do what you described:
$ exiftool "-allDates<filename" -wm w .
1 directories scanned
1 image files updated
Odd. I need to figure out how this happened. AllDates should be a subset of Time:All.
QuoteThis is the script I used:
https://github.com/HayoBaan/matchDateTime (https://github.com/HayoBaan/matchDateTime)
Excellent. That is probably a better solution.
- Phil
Quote from: Phil Harvey on June 11, 2018, 07:42:58 PM
OK, then maybe none of the existing date/time tags are writable. What is the output of this command?:
exiftool -G1 -a time:all FILE
QuoteAdding the "write only existing tags" option to the command I used seems to do what you described:
$ exiftool "-allDates<filename" -wm w .
1 directories scanned
1 image files updated
Odd. I need to figure out how this happened. AllDates should be a subset of Time:All.
$ exiftool "-time:all<filename" -wm w 2002-05-30\ 12_32_41.mov
Warning: No writable tags set from 2002-05-30 12_32_41.mov
0 image files updated
1 image files unchanged
$ exiftool -G1 -a -time:all 2002-05-30\ 12_32_41.mov
[System] File Modification Date/Time : 2018:06:03 11:20:50+02:00
[System] File Access Date/Time : 2018:06:12 17:39:13+02:00
[System] File Inode Change Date/Time : 2018:06:12 17:38:43+02:00
[QuickTime] Create Date : 2018:06:03 09:20:50
[QuickTime] Modify Date : 2018:06:03 09:21:01
[Track1] Track Create Date : 2018:06:03 09:20:50
[Track1] Track Modify Date : 2018:06:03 09:21:01
[Track1] Media Create Date : 2018:06:03 09:20:50
[Track1] Media Modify Date : 2018:06:03 09:21:01
[Track2] Track Create Date : 2018:06:03 09:20:50
[Track2] Track Modify Date : 2018:06:03 09:21:01
[Track2] Media Create Date : 2018:06:03 09:20:50
[Track2] Media Modify Date : 2018:06:03 09:21:01
[Track3] Track Create Date : 2018:06:03 09:20:50
[Track3] Track Modify Date : 2018:06:03 09:21:01
[Track3] Media Create Date : 2018:06:03 09:20:50
[Track3] Media Modify Date : 2018:06:03 09:21:01
[QuickTime] Creation Date (nld-NL) : 2018:06:03 09:59:18+02:00
[QuickTime] Creation Date : 2018:06:03 09:59:18+02:00
$ exiftool "-allDates<filename" -wm w 2002-05-30\ 12_32_41.mov
1 image files updated
$ exiftool -G1 -a -time:all 2002-05-30\ 12_32_41.mov
[System] File Modification Date/Time : 2018:06:12 17:40:01+02:00
[System] File Access Date/Time : 2018:06:12 17:40:04+02:00
[System] File Inode Change Date/Time : 2018:06:12 17:40:01+02:00
[QuickTime] Create Date : 2002:05:30 12:32:41
[QuickTime] Modify Date : 2002:05:30 12:32:41
[Track1] Track Create Date : 2018:06:03 09:20:50
[Track1] Track Modify Date : 2018:06:03 09:21:01
[Track1] Media Create Date : 2018:06:03 09:20:50
[Track1] Media Modify Date : 2018:06:03 09:21:01
[Track2] Track Create Date : 2018:06:03 09:20:50
[Track2] Track Modify Date : 2018:06:03 09:21:01
[Track2] Media Create Date : 2018:06:03 09:20:50
[Track2] Media Modify Date : 2018:06:03 09:21:01
[Track3] Track Create Date : 2018:06:03 09:20:50
[Track3] Track Modify Date : 2018:06:03 09:21:01
[Track3] Media Create Date : 2018:06:03 09:20:50
[Track3] Media Modify Date : 2018:06:03 09:21:01
[QuickTime] Creation Date (nld-NL) : 2018:06:03 09:59:18+02:00
[QuickTime] Creation Date : 2018:06:03 09:59:18+02:00The file I used is the (renamed) original file that is also in the zip file I sent you earlier, in case you want to test this yourself.
Ah, right. Thanks. Rookie mistake (and I've fallen for this one before). "-time:all<filename" only tries to write the Time:FileName tag:
A destination group and/or tag name of "All" or "*" writes
to the same family 1 group and/or tag name as the source.
But Time:FileName doesn't exist, so this doesn't do anything.
This syntax makes sense when using wildcards in the source, but maybe I should change this when the source is a single tag.
- Phil