Why is this going so slow

Started by intuitive, August 05, 2015, 01:14:54 PM

Previous topic - Next topic

intuitive

I'm trying to understand why exifTool is going so slow and if there is anything I can do to boost the performance. I am running this on a linux server with quite a bit of cpu and ram.


[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 2015-NAS-08665_S001.mov ArchiveName DIRTY_OFF_AIR_PRACTICE_POCONO_31JUL2015_SC
Warning: Tag 'ArchiveName' does not exist
    1 image files updated

real   8m46.379s
user   0m29.755s
sys   3m28.377s



My config file looks like this:
%Image::ExifTool::UserDefined::Options = (
LargeFileSupport => 1,
);

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::XMP::Main' => {
        NASCAR => {
            SubDirectory => {
                TagTable => 'Image::ExifTool::UserDefined::NASCAR',
            },
        },
    },
);
%Image::ExifTool::UserDefined::NASCAR = (
    GROUPS => { 0 => 'XMP', 1 => 'XMP-NASCAR', 2 => 'Other' },
    NAMESPACE => { 'NASCAR' => 'NASCARCustomMetadata' },
    WRITABLE => 'string', Archive_Name => {},
);


%Image::ExifTool::UserDefined = (
    'Image::ExifTool::XMP::Main' => {
        xmpDM => {
            SubDirectory => {
                TagTable => 'Image::ExifTool::UserDefined::xmpDM',
            },
        },
    },
);
%Image::ExifTool::UserDefined::NASCAR = (
    GROUPS => { 0 => 'XMP', 1 => 'XMP-xmpDM', 2 => 'Other' },
    WRITABLE => 'string', Tape_Name => {},
);


Thanks for your help

- Jason

Phil Harvey

Hi Jason,

How large is this .mov file, and how long does it take to just do a straight 'cp' of it?

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

intuitive

The copy does take a long time too:
real   5m16.197s

The file is 45Gigs. Also I just realized it actually is failing to add the metadata. The same exact command works on a smaller file. Not sure whats different here.

Seen below, the same command works on the tiny 5.5M file but seems to do nothing to the 45G file.


[reachengine@cltp-re-srv9 embedMeta]$ ls -lah /stornext/LevelsBeyond/ReachEngine/Video/2015/07/10/16/1920-NAS-00119_AA801801-EDIT_PROXY.mov
-rw-r--r-- 1 reachengine reachengine 5.5M Aug  5 13:17 /stornext/LevelsBeyond/ReachEngine/Video/2015/07/10/16/1920-NAS-00119_AA801801-EDIT_PROXY.mov

[reachengine@cltp-re-srv9 embedMeta]$ ls -lah /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
-rw-r--r-- 1 reachengine reachengine 45G Aug  5 13:09 /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov


I'm running exifTool through a script with these options btw:

#!/bin/bash

fileToEmbed=${1}
exifConfig=${2}
field1=${3}
value1=${4}
field2=${5}
value2=${6}

/reachengine/utilities/exiftool -config $exifConfig -$field1="$value1" -$field2="$value2" $fileToEmbed -overwrite_original


Phil Harvey

That's odd.  If ExifTool says the file was updated, then something should have been changed.  Try using -v3 when writing to see if that gives any more information (or maybe a pause in this output will show where the time is being spent).

I don't have any MOV files so large, so it may be difficult for me to reproduce this here.

You didn't tell me how long a straight 'cp' of the 45GB file takes on your system.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

intuitive

The straight copy took about 8 minutes.

When I run the command with the -v3 it sits here for a majority of the time


[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 'myCmdTest1' archive_name 'arch1'
Writing XMP-xmpDM:TapeName
Tag 'archive_name' does not exist
Warning: Tag 'archive_name' does not exist
======== /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
Rewriting /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov...
  FileType = MOV
  FileTypeExtension = MOV
  MIMEType = video/quicktime
  Editing tags in: MOV Movie UserData XMP
  Creating tags in: MOV Movie UserData XMP


Then the rest of the output is


  Rewriting Movie
  Rewriting MovieHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting UserData
  Rewriting XMP
    - XMP-xmpDM:TapeName = '2015-NAS-08665_S001.mov'
    + XMP-xmpDM:TapeName = 'myCmdTest1'
    1 image files updated

real   3m32.599s
user   0m26.851s
sys   2m2.522s


Is it not possible to update the headers without having to rewrite the entire movie?

This one went way faster for some reason.

The final result still doesnt get any updated data though


[reachengine@cltp-re-srv9 embedMeta]$ exiftool /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
ExifTool Version Number         : 9.97
File Name                       : 2015-NAS-08665_S001.mov
Directory                       : /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15
File Size                       : 45812 MB
File Modification Date/Time     : 2015:08:05 13:46:13-04:00
File Access Date/Time           : 2015:08:05 13:42:41-04:00
File Inode Change Date/Time     : 2015:08:05 13:46:13-04:00
File Permissions                : rw-r--r--
File Type                       : MOV
File Type Extension             : mov
MIME Type                       : video/quicktime
Major Brand                     : Apple QuickTime (.MOV/QT)
Minor Version                   : 2005.3.0
Compatible Brands               : qt
Warning                         : End of processing at large atom (LargeFileSupport not enabled)

Phil Harvey

Quote from: intuitive on August 05, 2015, 01:48:41 PM
The straight copy took about 8 minutes.

Well, there you go.  You can't get faster than that.  (your 3m time must have been due to the disk cache, but that doesn't help for the first time the file is copied)

QuoteIs it not possible to update the headers without having to rewrite the entire movie?

No.  ExifTool always rewrites the entire file.

Quote
Warning                         : End of processing at large atom (LargeFileSupport not enabled)


Try adding -api LargeFileSupport to the command.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

intuitive

I added the LargeFIleSupport to the cmd


/reachengine/utilities/exiftool -config $exifConfig -$field1="$value1" -$field2="$value2" $fileToEmbed -v3 -api LargeFileSupport -overwrite_original


Still produces the same results, no metadata added to the file. But also didnt see any error


[reachengine@cltp-re-srv9 embedMeta]$ time ./embedExif.sh /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov exifToolConfig.cfg TapeName 'myCmdTest1' archive_name 'arch1'
Writing XMP-xmpDM:TapeName
Tag 'archive_name' does not exist
Warning: Tag 'archive_name' does not exist
======== /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov
Rewriting /stornext/Y/ReachEngine/Mezzanines/2015/07/31/15/2015-NAS-08665_S001.mov...
  FileType = MOV
  FileTypeExtension = MOV
  MIMEType = video/quicktime
  Editing tags in: MOV Movie UserData XMP
  Creating tags in: MOV Movie UserData XMP
  Rewriting Movie
  Rewriting MovieHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting Track
  Rewriting TrackHeader
  Rewriting Media
  Rewriting MediaHeader
  Rewriting UserData
  Rewriting XMP
    - XMP-xmpDM:TapeName = 'myCmdTest1'
    + XMP-xmpDM:TapeName = 'myCmdTest1'
    1 image files updated

real   4m51.501s
user   0m26.491s
sys   2m24.549s

Phil Harvey

No.  The warning was when reading back the metadata. 
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).