Reading tags from large AVI files

BrianR · September 21, 2011, 12:09:48 AM

Hi Phil,

I'm writing metadata to AVI files via Adobe Bridge. When I read the tags via exiftool, I can see the new tags with smaller files (200-300MB) but not with large files (1.5GB). I know the data is there as it is read just fine by Bridge.

The follow tags are missing in the output of the large file (the rest of the tags match the smaller files):

XMP Toolkit
Date Created
Metadata Date
Instance ID
Document ID
Original Document ID
History Action
History Instance ID
History When
History Changed
Hierarchical Subject
Subject

System specs:

OS: Windows 7 SP1 64bit
ExifTool Version: 8.60
Command line: exiftool (no options)

Let me know if you need more info.

Thanks,
Brian

Phil Harvey · September 21, 2011, 07:27:28 AM

Hi Brian,

This will happen if you are using the -fast option since Bridge stores the metadata at the end of the file and exiftool will stop processing when it hits the movie data with this option.

- Phil

Edit: Wait. This would also occur for small files. Could you post the output from the -v3 option here so I can see what is going on?

BrianR · September 21, 2011, 11:05:53 PM

Output requested:

$ exiftool -v3 video.avi
ExifToolVersion = 8.60
FileName = video.avi
Directory = .
FileSize = 1569132090
FileModifyDate = 1316575556
FilePermissions = 33216
FileType = AVI
MIMEType = video/x-msvideo
RIFF 'LIST_hdrl' chunk (65080 bytes of data):
Hdrl (SubDirectory) -->
- Tag 'LIST_hdrl' (65080 bytes):
0000: 61 76 69 68 38 00 00 00 56 82 00 00 34 1d 3a 00 [avih8...V...4.:.]
0010: 00 02 00 00 10 08 00 00 41 30 00 00 00 00 00 00 [........A0......]
0020: 02 00 00 00 c8 d4 01 00 d0 02 00 00 e0 01 00 00 [................]
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0040: 4c 49 53 54 74 7e 00 00 73 74 72 6c 73 74 72 68 [LISTt~..strlstrh]
[snip 65000 bytes]
+ [Hdrl directory, 65080 bytes]
| AVIHeader (SubDirectory) -->
| - Tag 'avih' (56 bytes):
| 0008: 56 82 00 00 34 1d 3a 00 00 02 00 00 10 08 00 00 [V...4.:.........]
| 0018: 41 30 00 00 00 00 00 00 02 00 00 00 c8 d4 01 00 [A0..............]
| 0028: d0 02 00 00 e0 01 00 00 00 00 00 00 00 00 00 00 [................]
| 0038: 00 00 00 00 00 00 00 00 [........]
| + [BinaryData directory, 56 bytes]
| | FrameRate = 33366
| | - Tag 0x0000 (4 bytes, int32u[1]):
| | 0020: 56 82 00 00 [V...]
| | MaxDataRate = 3808564
| | - Tag 0x0001 (4 bytes, int32u[1]):
| | 0024: 34 1d 3a 00 [4.:.]
| | FrameCount = 12353
| | - Tag 0x0004 (4 bytes, int32u[1]):
| | 0030: 41 30 00 00 [A0..]
| | StreamCount = 2
| | - Tag 0x0006 (4 bytes, int32u[1]):
| | 0038: 02 00 00 00 [....]
| | ImageWidth = 720
| | - Tag 0x0008 (4 bytes, int32u[1]):
| | 0040: d0 02 00 00 [....]
| | ImageHeight = 480
| | - Tag 0x0009 (4 bytes, int32u[1]):
| | 0044: e0 01 00 00 [....]
| Stream (SubDirectory) -->
| - Tag 'LIST_strl' (32368 bytes):
| 004c: 73 74 72 68 38 00 00 00 76 69 64 73 64 76 73 64 [strh8...vidsdvsd]
| 005c: 00 00 00 00 00 00 00 00 00 00 00 00 e9 03 00 00 [................]
| 006c: 30 75 00 00 00 00 00 00 41 30 00 00 c8 d4 01 00 [0u......A0......]
| 007c: 00 00 00 00 00 00 00 00 00 00 00 00 d0 02 e0 01 [................]
| 008c: 73 74 72 66 28 00 00 00 28 00 00 00 d0 02 00 00 [strf(...(.......]
| [snip 32288 bytes]
| + [Stream directory, 32368 bytes]
| | StreamHeader (SubDirectory) -->
| | - Tag 'strh' (56 bytes):
| | 0054: 76 69 64 73 64 76 73 64 00 00 00 00 00 00 00 00 [vidsdvsd........]
| | 0064: 00 00 00 00 e9 03 00 00 30 75 00 00 00 00 00 00 [........0u......]
| | 0074: 41 30 00 00 c8 d4 01 00 00 00 00 00 00 00 00 00 [A0..............]
| | 0084: 00 00 00 00 d0 02 e0 01 [........]
| | + [BinaryData directory, 56 bytes]
| | | StreamType = vids
| | | - Tag 0x0000 (4 bytes, string[4]):
| | | 006c: 76 69 64 73 [vids]
| | | VideoCodec = dvsd
| | | - Tag 0x0001 (4 bytes, string[4]):
| | | 0070: 64 76 73 64 [dvsd]
| | | VideoFrameRate = 0.03336666667
| | | - Tag 0x0005 (8 bytes, rational64u[1]):
| | | 0080: e9 03 00 00 30 75 00 00 [....0u..]
| | | VideoFrameCount = 12353
| | | - Tag 0x0008 (4 bytes, int32u[1]):
| | | 008c: 41 30 00 00 [A0..]
| | | Quality = 0
| | | - Tag 0x000a (4 bytes, int32u[1]):
| | | 0094: 00 00 00 00 [....]
| | | SampleSize = 0
| | | - Tag 0x000b (4 bytes, int32u[1]):
| | | 0098: 00 00 00 00 [....]
| | VideoFormat (SubDirectory) -->
| | - Tag 'strf' (40 bytes):
| | 0094: 28 00 00 00 d0 02 00 00 e0 01 00 00 01 00 18 00 [(...............]
| | 00a4: 64 76 73 64 c0 d4 01 00 00 00 00 00 00 00 00 00 [dvsd............]
| | 00b4: 00 00 00 00 00 00 00 00 [........]
| | + [BinaryData directory, 40 bytes]
| | | ImageWidth = 720
| | | - Tag 0x0004 (4 bytes, int32u[1]):
| | | 00b0: d0 02 00 00 [....]
| | | ImageHeight = 480
| | | - Tag 0x0008 (4 bytes, int32s[1]):
| | | 00b4: e0 01 00 00 [....]
| | | Planes = 1
| | | - Tag 0x000c (2 bytes, int16u[1]):
| | | 00b8: 01 00 [..]
| | | BitDepth = 24
| | | - Tag 0x000e (2 bytes, int16u[1]):
| | | 00ba: 18 00 [..]
| | | Compression = 1685288548
| | | - Tag 0x0010 (4 bytes, int32u[1]):
| | | 00bc: 64 76 73 64 [dvsd]
| | | ImageLength = 120000
| | | - Tag 0x0014 (4 bytes, int32u[1]):
| | | 00c0: c0 d4 01 00 [....]
| | | PixelsPerMeterX = 0
| | | - Tag 0x0018 (4 bytes, int32u[1]):
| | | 00c4: 00 00 00 00 [....]
| | | PixelsPerMeterY = 0
| | | - Tag 0x001c (4 bytes, int32u[1]):
| | | 00c8: 00 00 00 00 [....]
| | | NumColors = 0
| | | - Tag 0x0020 (4 bytes, int32u[1]):
| | | 00cc: 00 00 00 00 [....]
| | | NumImportantColors = 0
| | | - Tag 0x0024 (4 bytes, int32u[1]):
| | | 00d0: 00 00 00 00 [....]
| | RIFF_Stream_indx = ..00db.~..p..~....<~...~[~..
| | - Tag 'indx' (32248 bytes):
| | 00c4: 04 00 00 00 04 00 00 00 30 30 64 62 00 00 00 00 [........00db....]
| | 00d4: 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 [................]
| | 00e4: 00 7e 00 00 bc 0f 00 00 00 70 7f 1e 00 00 00 00 [.~.......p......]
| | 00f4: 00 7e 00 00 bc 0f 00 00 00 ac fc 3c 00 00 00 00 [.~.........<....]
| | 0104: 00 7e 00 00 bc 0f 00 00 00 08 7e 5b 00 00 00 00 [.~........~[....]
| | [snip 32168 bytes]
| Stream (SubDirectory) -->
| - Tag 'LIST_strl' (32356 bytes):
| 7ec8: 73 74 72 68 38 00 00 00 61 75 64 73 00 00 00 00 [strh8...auds....]
| 7ed8: 00 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 [................]
| 7ee8: 00 ee 02 00 00 00 00 00 74 e3 2d 01 10 19 00 00 [........t.-.....]
| 7ef8: 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 [................]
| 7f08: 73 74 72 66 12 00 00 00 01 00 02 00 80 bb 00 00 [strf............]
| [snip 32276 bytes]
| + [Stream directory, 32356 bytes]
| | StreamHeader (SubDirectory) -->
| | - Tag 'strh' (56 bytes):
| | 7ed0: 61 75 64 73 00 00 00 00 00 00 00 00 00 00 00 00 [auds............]
| | 7ee0: 00 00 00 00 04 00 00 00 00 ee 02 00 00 00 00 00 [................]
| | 7ef0: 74 e3 2d 01 10 19 00 00 00 00 00 00 04 00 00 00 [t.-.............]
| | 7f00: 00 00 00 00 00 00 00 00 [........]
| | + [BinaryData directory, 56 bytes]
| | | StreamType = auds
| | | - Tag 0x0000 (4 bytes, string[4]):
| | | 7ee8: 61 75 64 73 [auds]
| | | AudioCodec =
| | | - Tag 0x0001 (4 bytes, string[4]):
| | | 7eec: 00 00 00 00 [....]
| | | AudioSampleRate = 2.083333333e-05
| | | - Tag 0x0005 (8 bytes, rational64u[1]):
| | | 7efc: 04 00 00 00 00 ee 02 00 [........]
| | | AudioSampleCount = 19784564
| | | - Tag 0x0008 (4 bytes, int32u[1]):
| | | 7f08: 74 e3 2d 01 [t.-.]
| | | Quality = 0
| | | - Tag 0x000a (4 bytes, int32u[1]):
| | | 7f10: 00 00 00 00 [....]
| | | SampleSize = 4
| | | - Tag 0x000b (4 bytes, int32u[1]):
| | | 7f14: 04 00 00 00 [....]
| | AudioFormat (SubDirectory) -->
| | - Tag 'strf' (18 bytes):
| | 7f10: 01 00 02 00 80 bb 00 00 00 ee 02 00 04 00 10 00 [................]
| | 7f20: 00 00 [..]
| | + [BinaryData directory, 18 bytes]
| | | Encoding = 1
| | | - Tag 0x0000 (2 bytes, int16u[1]):
| | | 7f28: 01 00 [..]
| | | NumChannels = 2
| | | - Tag 0x0001 (2 bytes, int16u[1]):
| | | 7f2a: 02 00 [..]
| | | SampleRate = 48000
| | | - Tag 0x0002 (4 bytes, int32u[1]):
| | | 7f2c: 80 bb 00 00 [....]
| | | AvgBytesPerSec = 192000
| | | - Tag 0x0004 (4 bytes, int32u[1]):
| | | 7f30: 00 ee 02 00 [....]
| | | BitsPerSample = 16
| | | - Tag 0x0007 (2 bytes, int16u[1]):
| | | 7f36: 10 00 [..]
| | RIFF_Stream_JUNK =
| | - Tag 'JUNK' (2 bytes):
| | 7f2a: 00 00 [..]
| | RIFF_Stream_indx = ..01wb~.~,pb.o.~,pb..<~.pb$n[~...
| | - Tag 'indx' (32248 bytes):
| | 7f34: 04 00 00 00 04 00 00 00 30 31 77 62 00 00 00 00 [........01wb....]
| | 7f44: 00 00 00 00 00 00 00 00 00 7e 01 00 00 00 00 00 [.........~......]
| | 7f54: 00 7e 00 00 2c 70 62 00 00 8c 6f 1e 00 00 00 00 [.~..,pb...o.....]
| | 7f64: 00 7e 00 00 2c 70 62 00 00 c8 ec 3c 00 00 00 00 [.~..,pb....<....]
| | 7f74: 00 7e 00 00 2e 70 62 00 00 24 6e 5b 00 00 00 00 [.~...pb..$n[....]
| | [snip 32168 bytes]
| OpenDML (SubDirectory) -->
| - Tag 'LIST_odml' (256 bytes):
| fd38: 64 6d 6c 68 f8 00 00 00 41 30 00 00 00 00 00 00 [dmlh....A0......]
| fd48: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| fd58: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| fd68: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| fd78: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| [snip 176 bytes]
| + [OpenDML directory, 256 bytes]
| | ExtendedAVIHeader (SubDirectory) -->
| | - Tag 'dmlh' (248 bytes):
| | fd40: 41 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [A0..............]
| | fd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| | fd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| | fd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| | fd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
| | [snip 168 bytes]
| | + [BinaryData directory, 248 bytes]
| | | TotalFrameCount = 12353
| | | - Tag 0x0000 (4 bytes, int32u[1]):
| | | fd58: 41 30 00 00 [A0..]
RIFF 'JUNK' chunk (412 bytes of data):
TextJunk =
- Tag 'JUNK' (412 bytes):
0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
[snip 332 bytes]
RIFF 'LIST_movi' chunk (1069257728 bytes of data):
RIFF 'idx1' chunk (269816 bytes of data):
RIFF 'RIFF' chunk (499538994 bytes of data):
Out of memory during "large" request for 268439552 bytes, total sbrk() is 273969152 bytes at /usr/lib/perl5/site_perl/5.10/File/RandomAccess.pm line 187.

Phil Harvey · September 22, 2011, 07:41:13 AM

Interesting. This is not what I expected. Here the verbose option is killing us because ExifTool is trying to load the data for the "RIFF" chunk, which it wouldn't normally try to do. However, the "RIFF" chunk should only appear at the start of an AVI file, so something funny is going on here. I will see what I can do about generating a 1.5GB AVI so I can test this myself.

- Phil

Phil Harvey · September 22, 2011, 07:00:31 PM

Darn,

I just spent the last hour joining together AVI files until I got one that was 1.35 GB, but my AVI doesn't have the same problem:

> exiftool /Volumes/Time\ Machine\ Backups/SplitFuse\ Fused\ Movie.avi -v3
ExifToolVersion = 8.64
FileName = SplitFuse Fused Movie.avi
Directory = /Volumes/Time Machine Backups
FileSize = 1359289008
[...]
| | | AvgBytesPerSec = 44100
| | | - Tag 0x0004 (4 bytes, int32u[1]):
| | | 0130: 44 ac 00 00 [D...]
| | | BitsPerSample = 16
| | | - Tag 0x0007 (2 bytes, int16u[1]):
| | | 0136: 10 00 [..]
RIFF 'LIST_movi' chunk (1354420864 bytes of data):
RIFF 'idx1' chunk (4867808 bytes of data):

And that's the end of the file. No additional RIFF chunk like you are seeing. When I add XMP with Adobe Bridge, exiftool reads it fine.

But it is interesting to me that the "movi" chunk in my 1.35 GB file is about the full size of the file, while in your 1.5 GB file your "movi" chunk is only about 1 GB. I think the file structure may be the difference.

Would it be possible for you to host the file so I can download it? If so, email a URL to me (philharvey66 at gmail.com).

Otherwise it may be difficult for me to reproduce this problem.

- Phil

Edit: HOLD ON before you upload your AVI. I have another idea. It looks like your AVI file may be a simple concatenation of 2 other AVI files (that would explain the extra RIFF chunk)... I will run some more tests by generating files like this.

Phil Harvey · September 22, 2011, 08:48:27 PM

I was able to reproduce the -v3 problem by concatenating 2 AVI videos, but I could still read metadata written by Bridge. However, I think this is likely the problem, and the Bridge metadata may be easily hidden from ExifTool if it was written inside the 2nd RIFF chunk (which didn't appear to be the case for my tests, but could well be in your file).

So I have a solution that I will implement in ExifTool 8.65 which may solve this problem (fingers crossed).

In case you can test this out before the official release, I have prepared a 8.65 pre-release that you can try.

- Phil

BrianR · September 22, 2011, 10:01:12 PM

Excellent! Your fix in the pre-release did the trick.

Now that I think of it, the files were actually split off from a larger AVI file that was captured straight from tape using Sony Vegas. I'm thinking that Vegas inserted the additional chunks during capture (scene markers?). Looking at the original AVI that was captured I see many more RIFF chunks. I used a different program (DVdate) to split out the scenes since it does a better job than Vegas so maybe the extra chucks in the split out files are what Vegas thought were different scenes. The smaller files just didn't happen to have any.

Anyway, thank you for taking the time to resolve this. Much appreciated!

Brian

Phil Harvey · September 23, 2011, 07:04:37 AM

Great. Glad this solved the problem.

Thanks for the report and the follow up.

- Phil

ExifTool Forum

News:

Reading tags from large AVI files

BrianR

Phil Harvey

BrianR

Phil Harvey

Phil Harvey

Phil Harvey

BrianR

Phil Harvey