I'm trying to understand why exifTool is going so slow and if there is anything I can do to boost the performance. I am running this on a linux server with quite a bit of cpu and ram.
[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 2015-NAS-08665_S001.mov ArchiveName DIRTY_OFF_AIR_PRACTICE_POCONO_31JUL2015_SC
Warning: Tag 'ArchiveName' does not exist
1 image files updated
real 8m46.379s
user 0m29.755s
sys 3m28.377s
My config file looks like this:
%Image::ExifTool::UserDefined::Options = (
LargeFileSupport => 1,
);
%Image::ExifTool::UserDefined = (
'Image::ExifTool::XMP::Main' => {
NASCAR => {
SubDirectory => {
TagTable => 'Image::ExifTool::UserDefined::NASCAR',
},
},
},
);
%Image::ExifTool::UserDefined::NASCAR = (
GROUPS => { 0 => 'XMP', 1 => 'XMP-NASCAR', 2 => 'Other' },
NAMESPACE => { 'NASCAR' => 'NASCARCustomMetadata' },
WRITABLE => 'string', Archive_Name => {},
);
%Image::ExifTool::UserDefined = (
'Image::ExifTool::XMP::Main' => {
xmpDM => {
SubDirectory => {
TagTable => 'Image::ExifTool::UserDefined::xmpDM',
},
},
},
);
%Image::ExifTool::UserDefined::NASCAR = (
GROUPS => { 0 => 'XMP', 1 => 'XMP-xmpDM', 2 => 'Other' },
WRITABLE => 'string', Tape_Name => {},
);
Thanks for your help
- Jason
Hi Jason,
How large is this .mov file, and how long does it take to just do a straight 'cp' of it?
- Phil
The copy does take a long time too:
real 5m16.197s
The file is 45Gigs. Also I just realized it actually is failing to add the metadata. The same exact command works on a smaller file. Not sure whats different here.
Seen below, the same command works on the tiny 5.5M file but seems to do nothing to the 45G file.
[reachengine@cltp-re-srv9 embedMeta]$ ls -lah /stornext/LevelsBeyond/ReachEngine/Video/2015/07/10/16/1920-NAS-00119_AA801801-EDIT_PROXY.mov
-rw-r--r-- 1 reachengine reachengine 5.5M Aug 5 13:17 /stornext/LevelsBeyond/ReachEngine/Video/2015/07/10/16/1920-NAS-00119_AA801801-EDIT_PROXY.mov
[reachengine@cltp-re-srv9 embedMeta]$ ls -lah /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
-rw-r--r-- 1 reachengine reachengine 45G Aug 5 13:09 /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
I'm running exifTool through a script with these options btw:
#!/bin/bash
fileToEmbed=${1}
exifConfig=${2}
field1=${3}
value1=${4}
field2=${5}
value2=${6}
/reachengine/utilities/exiftool -config $exifConfig -$field1="$value1" -$field2="$value2" $fileToEmbed -overwrite_original
That's odd. If ExifTool says the file was updated, then something should have been changed. Try using -v3 when writing to see if that gives any more information (or maybe a pause in this output will show where the time is being spent).
I don't have any MOV files so large, so it may be difficult for me to reproduce this here.
You didn't tell me how long a straight 'cp' of the 45GB file takes on your system.
- Phil
The straight copy took about 8 minutes.
When I run the command with the -v3 it sits here for a majority of the time
[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 'myCmdTest1' archive_name 'arch1'
Writing XMP-xmpDM:TapeName
Tag 'archive_name' does not exist
Warning: Tag 'archive_name' does not exist
======== /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
Rewriting /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov...
FileType = MOV
FileTypeExtension = MOV
MIMEType = video/quicktime
Editing tags in: MOV Movie UserData XMP
Creating tags in: MOV Movie UserData XMP
Then the rest of the output is
Rewriting Movie
Rewriting MovieHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting UserData
Rewriting XMP
- XMP-xmpDM:TapeName = '2015-NAS-08665_S001.mov'
+ XMP-xmpDM:TapeName = 'myCmdTest1'
1 image files updated
real 3m32.599s
user 0m26.851s
sys 2m2.522s
Is it not possible to update the headers without having to rewrite the entire movie?
This one went way faster for some reason.
The final result still doesnt get any updated data though
[reachengine@cltp-re-srv9 embedMeta]$ exiftool /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
ExifTool Version Number : 9.97
File Name : 2015-NAS-08665_S001.mov
Directory : /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15
File Size : 45812 MB
File Modification Date/Time : 2015:08:05 13:46:13-04:00
File Access Date/Time : 2015:08:05 13:42:41-04:00
File Inode Change Date/Time : 2015:08:05 13:46:13-04:00
File Permissions : rw-r--r--
File Type : MOV
File Type Extension : mov
MIME Type : video/quicktime
Major Brand : Apple QuickTime (.MOV/QT)
Minor Version : 2005.3.0
Compatible Brands : qt
Warning : End of processing at large atom (LargeFileSupport not enabled)
Quote from: intuitive on August 05, 2015, 01:48:41 PM
The straight copy took about 8 minutes.
Well, there you go. You can't get faster than that. (your 3m time must have been due to the disk cache, but that doesn't help for the first time the file is copied)
QuoteIs it not possible to update the headers without having to rewrite the entire movie?
No. ExifTool always rewrites the entire file.
Quote
Warning : End of processing at large atom (LargeFileSupport not enabled)
Try adding
-api LargeFileSupport to the command.
- Phil
I added the LargeFIleSupport to the cmd
/reachengine/utilities/exiftool -config $exifConfig -$field1="$value1" -$field2="$value2" $fileToEmbed -v3 -api LargeFileSupport -overwrite_original
Still produces the same results, no metadata added to the file. But also didnt see any error
[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 'myCmdTest1' archive_name 'arch1'
Writing XMP-xmpDM:TapeName
Tag 'archive_name' does not exist
Warning: Tag 'archive_name' does not exist
======== /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
Rewriting /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov...
FileType = MOV
FileTypeExtension = MOV
MIMEType = video/quicktime
Editing tags in: MOV Movie UserData XMP
Creating tags in: MOV Movie UserData XMP
Rewriting Movie
Rewriting MovieHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting Track
Rewriting TrackHeader
Rewriting Media
Rewriting MediaHeader
Rewriting UserData
Rewriting XMP
- XMP-xmpDM:TapeName = 'myCmdTest1'
+ XMP-xmpDM:TapeName = 'myCmdTest1'
1 image files updated
real 4m51.501s
user 0m26.491s
sys 2m24.549s
No. The warning was when reading back the metadata.