About exiftool stay_open using multi-process

Started by jishi92, November 09, 2022, 03:42:51 AM

Previous topic - Next topic

jishi92

Hi guys:
I use exiftool command line in Go.
I have used `stay_open` feature of exiftool to get a speed up compared to invoking one exiftool instance for each image.
However, the processing queue will accumulate if the speed of requests is greater than the speed of processing.Then the response-time will increase a lot.
I have too many images need to process.
Does the `stay_open` pattern support concurrent multi-process processing?
If yes, can you provide a demo code.
I would appreciate some guidance on where to look.

Phil Harvey

All you need to do is create multiple instances of the exiftool application and send the commands to them in parallel.  You already have the code to do this if you have created one.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jishi92

Thanks Phil, I already  have  the code and test it done. It's successful.

jishi92

Hey Phil:

I have another question. Does the `stay_open` pattern support input []byte data?
If I have []byte data from internet and now I have to save it a file and "exiftool -stay_open True -@ - filename -execute ".
I try write the file to ramdisk but it's slower than write it to disk immediately.
I don't know why that is.
Did I understand something wrong?

Phil Harvey

Are you asking if you can send the image file directly to ExifTool rather than read it off a disk?  I don't know of a way to do this when using -stay_open.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jishi92

Thank you Phil, I have process my images using multi-process processing in my code.
But I don't know is it concurrent safe?
I don't find any error in my concurrent test.
I want to confirm my result.

Phil Harvey

Each thread must be running its own copy of the exiftool application.  If you do this, then the only possible conflict is if you try to process the same files simultaneously.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

jishi92

A file will be process in a thread and a thread will process many files.
So it's safe.
Thanks very much.