Stream data to STDIN of exiftool process with "-stay_open", "true" "-@", "-"

Started by Anonan, June 16, 2021, 11:25:40 AM

Previous topic - Next topic

Anonan

It's not a problem to stream a data to STDIN of the program without "-stay_open=true -@ -" — just spawn the exiftool process with "-", and then pipe ReadableStream of a file to STDIN of the process.
It works OK. (It only requires to unpipe after receiving the response on "data" event since sometime the program take only the first part of a file and then is closed)

But wait, I can't do this with "-stay_open=true -@ -"?

I tried to spawn a process, then write the command, then write (pipe) a file. It's not working.
It prints:

"FileSize": "64 KiB",
"Error": "Unknown file type"



const exiftool = spawn("exiftool", ["-stay_open", "True", "-@", "-"]);
exiftool.stdin.write("-json\n-struct\n-b\n-\n-execute\n");
fs.createReadStream("image.jpeg").pipe(exiftool.stdin);


I want to read a file one time, but stream it to multiple processes for handling.

Anonan

Well, it looks impossible do with only STDIN of the program. (To use STDIN for commands and for data input)

But as a workaround I can create a temp file for commands, so it not so bad.

Phil Harvey

It is -stay_open true, not -stay_open=true.  But your spawn is correct.

If you are piping arguments from STDIN, then you can't also pipe a file that way.  Maybe use a temporary file for the arguments.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Anonan

But I can pipe only one file.

I get the empty result for the second file and the follow files.

"Error": "File is empty"


To simplify the example I just use pauses.


const argumentsFile = fs.createWriteStream("ARGFILE.txt");
const exiftool = spawn("exiftool", ["-stay_open", "true", "-@", "ARGFILE.txt"]);
let fileStream;
exiftool.stdout.on("data", data => {
    console.log(data.toString("utf8"));
    fileStream.unpipe(exiftool.stdin);
});


// It works
fileStream = fs.createReadStream("file1.mp4");
fileStream.pipe(exiftool.stdin);
argumentsFile.write("-json\n-\n-execute\n");

await sleep(1000);

// This does not works as expected
fileStream = fs.createReadStream("file1.mp4");
fileStream.pipe(exiftool.stdin);
argumentsFile.write("-json\n-\n-execute\n");


await sleep(100);
argumentsFile.write("-stay_open\nfalse\n");


function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

The process exits with code 1.

Anonan

Wait, is it technically possible to do?

How the program should understand when I stopped to write the first file, and when I started to write the second one?

If I write the entire file with exiftool.stdin.write(file); I need to call also exiftool.stdin.end();.
But after that I can't write more to stdin.

Phil Harvey

Good point.  You would need to reset STDIN somehow after each file for this to work.  I don't know how to do this, and it would probably need some special support in the ExifTool side.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

mceachen

I'd love to have exiftool support reading input from stdin, if possible! Windows temp file creation can take seconds (!!)


Solution 1: accept a new option in lieu of FILE. It could be -srcfile -, or could be a reserved filename (if that's easier to implement), or a new switch (like -base64src). Then base-64 encode the source file. As base64 doesn't include dash, you can append lines not starting with a dash to the source file buffer until you see /^-execute\n/.

Here's what an example run would look like (if we use the first suggestion above):

$ exiftool -stay_open True -@ -
-srcfile -
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR4nGP6bwwAAjoBNT578agA
AAAASUVORK5CYII=

-execute

File Size                       : 68 bytes
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 1
Image Height                    : 1
Bit Depth                       : 8
Color Type                      : Grayscale with Alpha
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Image Size                      : 1x1
Megapixels                      : 0.000001


Solution 2: Make people encode source files using something like mime multi-part encoding, and use something off the shelf (like https://metacpan.org/pod/Email::MIME#subparts).

That would look like:

$ exiftool -stay_open True -@ -
-boundary=c9ed4aae-df5a-11eb-889b-33674266c02d

--c9ed4aae-df5a-11eb-889b-33674266c02d
Content-Type: image/png; charset="UTF-8"
Content-Transfer-Encoding: base64

iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR4nGP6bwwAAjoBNT578agA
AAAASUVORK5CYII=

--c9ed4aae-df5a-11eb-889b-33674266c02d

-execute

File Size                       : 68 bytes
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 1
Image Height                    : 1
Bit Depth                       : 8
Color Type                      : Grayscale with Alpha
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Image Size                      : 1x1
Megapixels                      : 0.000001


(I realize either solution will be a bit involved to implement, as a number of File* attributes won't be relevant (like FileName, File Modification Date/Time, File Access Date/Time, File Permissions, ...)

Phil Harvey

Quote from: mceachen on July 07, 2021, 03:40:07 PM
I'd love to have exiftool support reading input from stdin, if possible! Windows temp file creation can take seconds (!!)

I think this would still be faster than encoding/decoding in Base64.  Decoding a large file from Base64 in Perl would be very slow.

Quote(I realize either solution will be a bit involved to implement, as a number of File* attributes won't be relevant (like FileName, File Modification Date/Time, File Access Date/Time, File Permissions, ...)

This is already the case when reading a single file from stdin, which is already supported.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Przemek

exiftool drops "{ready}" marker each time which means we can use transform stream to detect and terminate stream in next pipe without terminating exiftool process itself but it would spin up separate process each time.

const fs = require('fs');
const util = require('util');
const stream = require('stream');
const spawn = require('child_process').spawn;
const temp = require('temp-dir');

const pipeline = util.promisify(stream.pipeline);

const marker = '{ready}';
const terminate = ['-stay_open', 'false'].join('\n');
const action = ['-json', '-JpgFromRaw', '-b', '-', '-execute'].join('\n');

function timeout(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

function stripReadyEnd(done = false) {
    const trans = new stream.Transform({
        transform(chunk, encoding, callback) {
            if (encoding === 'buffer') {
                if (chunk.includes(marker)) {
                    chunk = chunk.slice(0, chunk.indexOf(marker));
                    done = true;
                }
                callback(null, chunk);
                if (done) {
                    trans.end();
                }
            }
        }
    });
    return trans;
}

function streamToExif(target, act, close = false, args = temp + '/args.txt') {
    const argsFile = fs.createWriteStream(args);
    const exiftool = spawn('exiftool', ['-stay_open', 'true', '-@', args]);
    exiftool.stdout.on('data', data => {
        const id = data.toString();
        const isReady = id.substring(id.length - 8);
        if (isReady.includes(marker) && close) {
            argsFile.write(terminate + '\n');
        }
    });
    fs.createReadStream(target).pipe(exiftool.stdin);
    argsFile.write(act + '\n');
    return exiftool.stdout;
}

(async () => {

    await pipeline(streamToExif('nikon_d850_01.nef', action), stripReadyEnd(), fs.createWriteStream('nikon_d850_01.json'));
    await pipeline(streamToExif('panasonic_s1r_01.rw2', action), stripReadyEnd(), fs.createWriteStream('panasonic_s1r_01.json'));
    await pipeline(streamToExif('canon_eos_1d_x_mark_iii_01.cr3', action), stripReadyEnd(), fs.createWriteStream('canon_eos_1d_x_mark_iii_01.json'));

    await timeout(1000);
    await pipeline(streamToExif('nikon_z7_ii_01.nef', action), stripReadyEnd(), fs.createWriteStream('nikon_z7_ii_01.json'));

    await timeout(1000);
    await pipeline(streamToExif('sony_a7r_iii_01.arw', action, true), stripReadyEnd(), fs.createWriteStream('sony_a7r_iii_01.json'));

})();