Performance issue stay open windows (please help)

[sebutzu]

I am using a wrapper in C# to perform some file metadata extraction using exiftool.exe. I am using also the ImageDataMD5 feature.
My arguments when starting the -stay_open process are:

Code Select Expand

public const string Arguments = "-fast -charset filename=UTF8 -m -q -q -stay_open True -@ - -common_args -d \"%Y.%m.%d %H:%M:%S\" -c \"%.8f\" -t -s -G:0:1 -a -all -use mwg -api largefilesupport=1 -api requesttags=imagedatamd5 -sep \"||\"";
I am trying to process 600GB of files (JPG, MOV, MP4) but it is painfully slow.
Any ideas on what parameters I can change to make it faster? Am I doing something wrong? I think it takes more than 1 hour to process the files (meaning extract metadata). Shouldn't it be possible to "read" 600 GB faster than 1 hour?
Any ideas are welcomed.

P.S. I tried to do it multi-threaded - meaning starting more than 1 exiftool.exe application, but it seems to be even slower (probably everything is disk bound).
P.S. I am using Windows 10, and the files are on a HDD (not SSD).

[Phil Harvey]

Yes.  Calculating the MD5 will be slow.  This is the whole idea of a security checksum, and is the reason these aren't calculated unless requested.

In fact, people have gone to using SHA checksums instead because they are stronger (hence even slower).

- Phil

[sebutzu]

It seems that even without md5 it is still slow. Any parameters that can improve speed? Or parameters I use that makes it slower? Thanks a lot for the greatest tool ever! I am just wondering if you ever compared the file operations speed in perl vs C++ or C#. Somehow the CPU is almost on 0 percent so it seems the disk operations are slow. I want more performance. I would like to help (I am a programmer) but I am not familiar with perl. Any chance of switching to a more mainstream programming language?

[StarGeek]

It seems that even without md5 it is still slow. Any parameters that can improve speed? Or parameters I use that makes it slower?

Exiftool should be just as fast as copying the file once it has moved past the start up time.  If you're working with large files, then the limiting factor will be the disk speed.  Your comment about the CPU on 0 supports this.

I am just wondering if you ever compared the file operations speed in perl vs C++ or C#.

See Why Write ExifTool in Perl?.