We have a directory with 10k files in. Running EXIFTool on this directory takes a long time (understandable).
Would it be possible to add a paginator so we can pass in a sort, limit and offset so that it only runs on a selection of the files
For example
exiftool -sort name -limit 1000 -offset 500 /my/big/directory
Would output data for 1000 files starting a file 500 when ordered by name
The cost of adding ExifTool options is that the (already lengthy) documentation gets bloated... eventually it will get so long that nobody ever reads it. [We may have already passed that point.]
I think adding 2 options for this likely-very-seldom-used (perhaps even only-used-by-you) feature is too high of a cost. Instead, doing something like this works in Mac/Linux:
ls /my/big/directory/* | sed -n '500,1500p;1501q' | exiftool -@ -
But there may be easier ways to do this.
- Phil
The ability to sort can be found in the -FileOrder option (https://exiftool.org/exiftool_pod.html#fileOrder-NUM---TAG).
Quote from: Phil Harvey on October 13, 2022, 10:43:37 AMThe cost of adding ExifTool options is that the (already lengthy) documentation gets bloated... eventually it will get so long that nobody ever reads it. [We may have already passed that point.]
I think adding 2 options for this likely-very-seldom-used (perhaps even only-used-by-you) feature is too high of a cost. Instead, doing something like this works in Mac/Linux:
ls /my/big/directory/* | sed -n '500,1500p;1501q' | exiftool -@ -
But there may be easier ways to do this.
- Phil
This is actually what are currently doing - but the issue we have is that ExifTool still runs its command on the 10k file, and we then just filter the output. So its extremely wasteful.
Could one param do it all?
-limit 1,500 (records 1 to 500)
-limit 500,500 (records 500 to 1000)
Similar to MYSQL LIMIT https://www.guru99.com/limit.html
Quote from: StarGeek on October 13, 2022, 11:07:12 AMThe ability to sort can be found in the -FileOrder option (https://exiftool.org/exiftool_pod.html#fileOrder-NUM---TAG).
Thanks!
Apologies Phil, we arent doign what you proposed actually.
We are runing exiftool on a directory and then piping that through jq which will indeed cause it to run on all the files.
exiftool my/directory/* | jq []
I will try your approach and report back, that looks like it would work just fine (providing I can sort the files before run sed on them)
Quote from: laenglish on October 14, 2022, 06:30:29 AM(providing I can sort the files before run sed on them)
ls /my/big/directory/* | sort | sed -n '500,1500p;1501q' | exiftool -@ -- Phil