Video files corrupted after changing dates

[dgltcg]

Hi,

I've been helping a family member transfer old videos from tape to Apple's Photo app. When importing these videos we found out that while the filename contains the date and time the video was taken, the metadata tag used by the Photos app contained the import date. To fix this, ExifTool looks like a great option.

I've used the following command on a folder containing the videos:

Code Select Expand

exiftool "-allDates<filename" .

Since we had only limited space, after some testing we decided to use the -overwrite option. This turned out to be a horrible mistake, as only after 1600 family video files were altered, I found out all videos turn into random color blocks (I've attached a screenshot) accompanied by white noise after the 10 second mark. (I ran only test on small files, less than 10 seconds long so never encountered this problem).

I'm wondering what I did wrong and hoping there's some way to repair this. Any help in recovering 20 years worth of family videos is greatly appreciated. I have very little knowledge of video files and metadata so I honestly have no idea where to even start looking. If any specific details help, please let me know so I can provide them.

The files are .mov files and were stored on an external hard drive. After running exiftool, no error messages were displayed.

[Alan Clifford]

Are all 1600 corrupted?  Have you viewed any off the corrupted files in their uncorrupted form? I ask this as you need to determine if they were uncorrupted before using exiftool.

[dgltcg]

Are all 1600 corrupted?  Have you viewed any off the corrupted files in their uncorrupted form? I ask this as you need to determine if they were uncorrupted before using exiftool.

Hi Alan,

As far as I can tell, all 1600 files are affected. There are some videos that are no longer than 10 seconds that display no symptoms. I have not found a video longer than 10 seconds that is not corrupted after the 10 second mark.

We still have the _original files from the first try (before the drive started to fill up and we used the -overwrite option). Those videos, including the ones over 10 seconds long, are fine.

[Phil Harvey]

Send me a (small) sample with this problem and I'll see what I can do.  My email is philharvey66 at gmail.com

This should be fixable as long as the videos were OK before you ran ExifTool, and you didn't alter them with anything else afterwards, provided that they weren't corrupted due to a failing hard disk.

- Phil

[dgltcg]

This should be fixable as long as the videos were OK before you ran ExifTool, and you didn't alter them with anything else afterwards, provided that they weren't corrupted due to a failing hard disk.

That's great news, I've sent you an e-mail.

Thanks!

[Phil Harvey]

Great.  I'll look at it as soon as I can, but I may not get a chance to do it until tomorrow.

- Phil

[Phil Harvey]

I got the files and will be able to fix this problem.

The reason that this happens is because the video data in these files is split into two separate blocks.  I have never seen this before, and was under the impression that it must be in a single block.  So the offsets into the second block are getting messed up when writing with ExifTool.   This will take a fairly substantial re-write of ExifTool's QuickTime writing algorithms to avoid this problem, and some additional work to be able to fix your damaged videos, but I should be able to accomplish this within a few days.

I'm assuming you're on a Mac...  What app did you use to create the original videos?

I'll release a new version of ExifTool with this problem fixed as soon as possible, along with a utility for you to repair your damaged files.

Thanks for reporting this, and for providing samples so I could reproduce the problem!

- Phil

Edit: In the mean time I have released ExifTool 11.01 which will avoid adding tags to videos with multiple mdat blocks.

[Phil Harvey]

OK.  I have prepared an ExifTool 11.02 pre-release that fixes the corrupted MOV you sent me.  This should fix all your videos if they are corrupted in the same way.

Here are the instructions:

1. Unpack the tarball on your Desktop

2. Run this command

~/Desktop/Image-ExifTool-11.02/exiftool -api FixCorruptedMOV -xmp= FILE

where FILE is the name of a corrupted MOV file, or a directory containing corrupted MOV files.

Let me know how it goes (fingers crossed).

- Phil

[Phil Harvey]

BTW, The original command you should have used to change the times was this:

exiftool "time:all<filename" -wm w .

Doing this will change all existing writable date/time tags according to the file name.  The command you used only changed/added common EXIF date/time tags.  It was the adding of new tags in a MOV with multiple 'mdat' blocks that resulted in the corruption.  The command above doesn't add new tags, so if you had done this we wouldn't have discovered this bug in ExifTool.  I'm glad to have found the bug, but from your point of view things would have been much less painful if we hadn't.  Sorry for that.

- Phil

[dgltcg]

Thanks again for taking the time to look at the problem and coming up with a solution this quickly. Good news: after running ExifTool using the command you posted, all videos are back to normal!

We did encounter some errors:

Code Select Expand


$ ~/Desktop/Image-ExifTool-11.02/exiftool -api FixCorruptedMOV -xmp= /Volumes/Extern/Filmpjes/
Error: Not a valid MOV - /Volumes/Extern/Filmpjes/._2002-05-22 22_19_57-1.mov_original.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-23 13_03_58.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-23 17_49_34.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-24 21_31_58.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-05-25 01_03_27.mov
Error: Not a valid MOV - /Volumes/Extern/Filmpjes/._2002-05-25 01_03_27.mov_original.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-07-27 11_38_00.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-08-09 13_23_31.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2002-08-16 13_47_13.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2005-05-21 14_37_39.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2005-07-11 13_36_51.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2006-08-09 17_28_23.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2007-09-21 20_50_43-1.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2008-10-12 16_52_05.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2009-04-08 21_05_56.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2009-05-10 12_39_27.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_37_42.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_52_14.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_53_18.mov
Error: File not found - /Volumes/Extern/Filmpjes/._2010-06-29 19_57_55.mov
    1 directories scanned
1509 image files updated
    2 image files unchanged
   20 files weren't updated due to errors

We checked the listed videos, but they seemed to be fine. Only after arriving back home, I realized these might actually have been hidden files instead of building a path to a file incorrectly (e.g. there was a file called "2002-05-23 13_03_58.mov" that was fixed, but I didn't check if "._2002-05-23 13_03_58.mov" exists. If it helps, I could check if these files actually exist.

The reason that this happens is because the video data in these files is split into two separate blocks.  I have never seen this before, and was under the impression that it must be in a single block.  So the offsets into the second block are getting messed up when writing with ExifTool.   This will take a fairly substantial re-write of ExifTool's QuickTime writing algorithms to avoid this problem, and some additional work to be able to fix your damaged videos, but I should be able to accomplish this within a few days.

I'm assuming you're on a Mac...  What app did you use to create the original videos?

I didn't do it personally, but iMovie was used to import the videos from a camera using miniDV tapes connected to the laptop.

BTW, The original command you should have used to change the times was this:

exiftool "time:all<filename" -wm w .

Doing this will change all existing writable date/time tags according to the file name.  The command you used only changed/added common EXIF date/time tags.  It was the adding of new tags in a MOV with multiple 'mdat' blocks that resulted in the corruption.  The command above doesn't add new tags, so if you had done this we wouldn't have discovered this bug in ExifTool.  I'm glad to have found the bug, but from your point of view things would have been much less painful if we hadn't.  Sorry for that.

Out of curiousity I just tried this command on one of the original test files I have available, but got the following output:

Code Select Expand

$ exiftool "time:all<filename" -wm w .
File not found: time:all<filename

Adding a dash didn't help either:

Code Select Expand

$ exiftool "-time:all<filename" -wm w .
Warning: No writable tags set from ./2008-05-01 16_48_37-1.mov
    1 directories scanned
    0 image files updated
    1 image files unchanged

Adding the "write only existing tags" option to the command I used seems to do what you described:

Code Select Expand

$ exiftool "-allDates<filename" -wm w .
    1 directories scanned
    1 image files updated

Some more background information: Apple's Photos app uses the Quicktime:CreationDate tag. According to your documentation, ExifTool can't write this tag, but I found a script that can. Since that script reads the CreateDate tag, I wanted to use ExifTool to use the filename as the CreateDate tag.

This is the script I used:

https://github.com/HayoBaan/matchDateTime

[StarGeek]

This is the script I used:

https://github.com/HayoBaan/matchDateTime

HayoBaan is a mod here.  He created that script back in 2016 and keeps it up to date.

[Phil Harvey]

Thanks again for taking the time to look at the problem and coming up with a solution this quickly. Good news: after running ExifTool using the command you posted, all videos are back to normal!

Great!

We did encounter some errors:

These are all with the "._" files the MacOS creates on non-mac filesystems to store the extended attributes.  They really shouldn't be processed anyway.

Only after arriving back home, I realized these might actually have been hidden files

Yes, they were.  ExifTool currently doesn't ignore filenames starting with "._", but maybe it should.

Out of curiousity I just tried this command on one of the original test files I have available, but got the following output:

Code Select Expand

$ exiftool "time:all<filename" -wm w .
File not found: time:all<filename

My mistake.  I forgot a "-":

exiftool "-time:all<filename" -wm w .

Adding a dash didn't help either:

Code Select Expand

$ exiftool "-time:all<filename" -wm w .
Warning: No writable tags set from ./2008-05-01 16_48_37-1.mov
    1 directories scanned
    0 image files updated
    1 image files unchanged

OK, then maybe none of the existing date/time tags are writable.  What is the output of this command?:

exiftool -G1 -a time:all FILE

Adding the "write only existing tags" option to the command I used seems to do what you described:

Code Select Expand

$ exiftool "-allDates<filename" -wm w .
    1 directories scanned
    1 image files updated

Odd.  I need to figure out how this happened.  AllDates should be a subset of Time:All.

This is the script I used:

https://github.com/HayoBaan/matchDateTime

Excellent.  That is probably a better solution.

- Phil

[dgltcg]

OK, then maybe none of the existing date/time tags are writable.  What is the output of this command?:

exiftool -G1 -a time:all FILE

Adding the "write only existing tags" option to the command I used seems to do what you described:

Code Select Expand

$ exiftool "-allDates<filename" -wm w .
    1 directories scanned
    1 image files updated

Odd.  I need to figure out how this happened.  AllDates should be a subset of Time:All.

$ exiftool "-time:all<filename" -wm w 2002-05-30\ 12_32_41.mov
Warning: No writable tags set from 2002-05-30 12_32_41.mov
    0 image files updated
    1 image files unchanged

$ exiftool -G1 -a -time:all 2002-05-30\ 12_32_41.mov
[System]        File Modification Date/Time     : 2018:06:03 11:20:50+02:00
[System]        File Access Date/Time           : 2018:06:12 17:39:13+02:00
[System]        File Inode Change Date/Time     : 2018:06:12 17:38:43+02:00
[QuickTime]     Create Date                     : 2018:06:03 09:20:50
[QuickTime]     Modify Date                     : 2018:06:03 09:21:01
[Track1]        Track Create Date               : 2018:06:03 09:20:50
[Track1]        Track Modify Date               : 2018:06:03 09:21:01
[Track1]        Media Create Date               : 2018:06:03 09:20:50
[Track1]        Media Modify Date               : 2018:06:03 09:21:01
[Track2]        Track Create Date               : 2018:06:03 09:20:50
[Track2]        Track Modify Date               : 2018:06:03 09:21:01
[Track2]        Media Create Date               : 2018:06:03 09:20:50
[Track2]        Media Modify Date               : 2018:06:03 09:21:01
[Track3]        Track Create Date               : 2018:06:03 09:20:50
[Track3]        Track Modify Date               : 2018:06:03 09:21:01
[Track3]        Media Create Date               : 2018:06:03 09:20:50
[Track3]        Media Modify Date               : 2018:06:03 09:21:01
[QuickTime]     Creation Date (nld-NL)          : 2018:06:03 09:59:18+02:00
[QuickTime]     Creation Date                   : 2018:06:03 09:59:18+02:00

$ exiftool "-allDates<filename" -wm w 2002-05-30\ 12_32_41.mov
    1 image files updated

$ exiftool -G1 -a -time:all 2002-05-30\ 12_32_41.mov
[System]        File Modification Date/Time     : 2018:06:12 17:40:01+02:00
[System]        File Access Date/Time           : 2018:06:12 17:40:04+02:00
[System]        File Inode Change Date/Time     : 2018:06:12 17:40:01+02:00
[QuickTime]     Create Date                     : 2002:05:30 12:32:41
[QuickTime]     Modify Date                     : 2002:05:30 12:32:41
[Track1]        Track Create Date               : 2018:06:03 09:20:50
[Track1]        Track Modify Date               : 2018:06:03 09:21:01
[Track1]        Media Create Date               : 2018:06:03 09:20:50
[Track1]        Media Modify Date               : 2018:06:03 09:21:01
[Track2]        Track Create Date               : 2018:06:03 09:20:50
[Track2]        Track Modify Date               : 2018:06:03 09:21:01
[Track2]        Media Create Date               : 2018:06:03 09:20:50
[Track2]        Media Modify Date               : 2018:06:03 09:21:01
[Track3]        Track Create Date               : 2018:06:03 09:20:50
[Track3]        Track Modify Date               : 2018:06:03 09:21:01
[Track3]        Media Create Date               : 2018:06:03 09:20:50
[Track3]        Media Modify Date               : 2018:06:03 09:21:01
[QuickTime]     Creation Date (nld-NL)          : 2018:06:03 09:59:18+02:00
[QuickTime]     Creation Date                   : 2018:06:03 09:59:18+02:00

The file I used is the (renamed) original file that is also in the zip file I sent you earlier, in case you want to test this yourself.

[Phil Harvey]

Ah, right.  Thanks.  Rookie mistake (and I've fallen for this one before).  "-time:all<filename" only tries to write the Time:FileName tag:

            A destination group and/or tag name of "All" or "*" writes
            to the same family 1 group and/or tag name as the source.

But Time:FileName doesn't exist, so this doesn't do anything.

This syntax makes sense when using wildcards in the source, but maybe I should change this when the source is a single tag.

- Phil