exiftool terminates during batch processing mode

[ugh_bough]

Hi,

Setup:

the following stripped-down version of my script starts exiftool in "-stay_open True" mode, then walks a folder of jpegs and dummy-writes each jpg, and eventually stops exiftool using the "-stay_open False" command.

Code Select Expand


#!/bin/bash

# controls for exiftool's batch processing mode
ETWORKSPACE="/tmp/etWorkspace"
ETCMDFILE="$ETWORKSPACE/commands"
ETOUTPUTFILE="$ETWORKSPACE/output"
ETWRITETESTFILE="$ETWORKSPACE/writeTest.jpg"
ETPID=-1 # to be replaced with pid of exiftool

# start exiftool
mkdir -p "$ETWORKSPACE"
rm -f    "$ETCMDFILE"
mkfifo   "$ETCMDFILE"
rm -f    "$ETOUTPUTFILE"
mkfifo   "$ETOUTPUTFILE"
exiftool -stay_open True -@ "$ETCMDFILE" > "$ETOUTPUTFILE" 2> "$ETOUTPUTFILE" &
ETPID="$!" # needed to close exiftool
read -t .1 < "$ETOUTPUTFILE" # unblock pipe => unblock exiftool

# dummy write files from folder
PHOTODIR="$HOME/photos"
cd "$PHOTODIR"
find "." -iname "*.jpg" | cut -c 3- | while read photo
do
echo $PHOTODIR/$photo

# dummy write a file
rm -f "$ETWRITETESTFILE"
{
  echo -artist=dummy
  echo -o
  echo $ETWRITETESTFILE
  echo $PHOTODIR/$photo
  echo -execute
} >> "$ETCMDFILE"

# read output
while [ 1 ]
do
  read out < "$ETOUTPUTFILE"
  echo "$out"
  [ "$out" == "{ready}" ] && break
done
done

{
echo "-stay_open"
echo "False"
} >> "$ETCMDFILE"
wait $ETPID



I observe that exiftool frequently stops already during the loop instead of waiting for the "-stay_open False" command. In particular, exiftool does not terminate each time, and if it does then it does on a different file each time.

I have encountered this behaviour on two different machines (both Ubuntu 16.04 x64) for two versions of exiftool (10.10 and 10.25), and both local and over an ssh-connection -- if the last one matters. Moreover, in order to verify that exiftool stops early, at the same time I run

Code Select Expand


while [ 1 ]
do
et="`ps aux | grep exiftool | grep -v grep`"
[ "$et" == "" ] && echo stopped
[ "$et" != "" ] && echo "$et"
done


in another shell: I have a feeling that this kind of 'monitoring' makes exiftool stop even earlier.

Question:

Since exiftool stops randomly, I think that my use of pipes to control/read exiftool is somehow flawed -- but I can't see the mistake...

What am I doing wrong? What can I do to find out? (e.g. debugging options for exiftool)

Thanks in advance! 
ugh_bough

[Phil Harvey]

I can reproduce the problem with your script on OS X.

Somehow I think the fifos are causing a stall.  Also, there could be a problem redirecting STDERR to the same output file.  The following script works for me without stalling:

Code Select Expand

#!/bin/bash

# controls for exiftool's batch processing mode
ETWORKSPACE="/tmp/etWorkspace"
ETCMDFILE="$ETWORKSPACE/commands"
ETOUTPUTFILE="$ETWORKSPACE/output"
ETWRITETESTFILE="$ETWORKSPACE/writeTest.jpg"
ETPID=-1 # to be replaced with pid of exiftool

# start exiftool
mkdir -p "$ETWORKSPACE"
rm -f    "$ETCMDFILE"
touch    "$ETCMDFILE"
rm -f    "$ETOUTPUTFILE"
touch    "$ETOUTPUTFILE"
exec 3<  "$ETOUTPUTFILE"
exiftool -stay_open True -@ "$ETCMDFILE" > "$ETOUTPUTFILE" &
ETPID="$!" # needed to close exiftool

# dummy write files from folder
PHOTODIR="$HOME/photos"
cd "$PHOTODIR"
find "." -iname "*.jpg" | cut -c 3- | while read photo
do
echo $PHOTODIR/$photo

# dummy write a file
rm -f "$ETWRITETESTFILE"
{
  echo -artist=dummy
  echo -o
  echo $ETWRITETESTFILE
  echo $PHOTODIR/$photo
  echo -execute
} >> "$ETCMDFILE"

# read output
while [ 1 ]
do
  read out <&3
  if [ "x$out" == "x" ]; then
   sleep 0.01
  else
   echo "$out"
   [ "$out" == "{ready}" ] && break
  fi
done
done

exec 3<&-

{
echo "-stay_open"
echo "False"
} >> "$ETCMDFILE"
wait $ETPID

- Phil

[ugh_bough]

Dear Mr. Harvey,

thank you (again) for your instant feedback !

To 'pipe' both stdout and stderr to the same file the following seems to work fine (no early terminations so far)

Code Select Expand


exiftool -stay_open True -@ "$ETCMDFILE" 2>&1 > "$ETOUTPUTFILE" &


Now I can happily proceed to process photos of my trip around lake huron and to toronto

Tschüss from Germany
ugh_bough

[Phil Harvey]

OK, so getting rid of the fifos fixed the problem?

I hope you had a good trip.  You came close to me -- I don't live far from Toronto.

- Phil

[ugh_bough]

OK, so getting rid of the fifos fixed the problem?

Seems like, very much. Had the "no-fifo-2>&1" version of the script running for tens of minutes without problem, whereas the original version couldn't even make it for a minute.

I hope you had a good trip.  You came close to me -- I don't live far from Toronto.

- Phil

That's why I mentioned it (I was curious about the url and found out about Queen's University in Kingston.)

The trip was great... sooo many nice experiences. E.g. we liked hanging out in Killbear Provincial Park near Parry Sound (not very far from you) -- during mid of may we were virtually alone there, had a pleasant sleep on bare rock right by the lake, and awoke to three deer passing by calmly in ~10-15m distance

ugh_bough

[Phil Harvey]

Great.

I've been through Parry Sound a number of times, but have never been to Killbear Park.

- Phil

[ugh_bough]

Sorry, don't want to turn your bug report forum into a trip chat, so this will be the last off topic post 

sleep + deer: https://www.google.de/maps/place/45%C2%B020'50.0%22N+80%C2%B013'17.8%22W/@45.3472407,-80.2238057,467m/data=!3m2!1e3!4b1!4m5!3m4!1s0x0:0x0!8m2!3d45.347237!4d-80.221617

sleep + water: https://www.google.de/maps/place/45%C2%B020'44.6%22N+80%C2%B013'02.2%22W/@45.346528,-80.2218355,932m/data=!3m1!1e3!4m5!3m4!1s0x0:0x0!8m2!3d45.345717!4d-80.217266

Rocks are uneven but not rough, i.e. a comfortable position for a nap should not be hard to find. Especially when the sun is shining all the rock gets warm, i.e. no blanket required.

The park streches east over the entire peninsula. Tip of peninsula also has some nice rock, though no sleep possible there.

Greetings to Canada
    ugh_bough

EDIT: Please remove post if you feel that it should'nt be here in the first place

[Phil Harvey]

Chatting about trips is fine here.

I've been to Killarney a couple of times (also on Georgian Bay), and had great naps on the rocks there too:



- Phil

[ugh_bough]

WHOA   8^O

Killbear Park's rocks are somewhat tiny in comparison, yet here is a pretty good impression (sorry, couldn't figure out how to include images in the post):

https://www.google.ca/maps/place/Killbear+Provincial+Park/@45.347915,-80.2002572,3a,75y,129.61h,64.17t/data=!3m8!1e1!3m6!1s-UfkgWNXn100%2FVCub6QlbWoI%2FAAAAAAACN4w%2F-OxWxzokyko80MfM4UoEAg6PeOoIuKQegCLIB!2e4!3e11!6s%2F%2Flh5.googleusercontent.com%2F-UfkgWNXn100%2FVCub6QlbWoI%2FAAAAAAACN4w%2F-OxWxzokyko80MfM4UoEAg6PeOoIuKQegCLIB%2Fw172-h86-n-k-no%2F!7i10776!8i5388!4m5!3m4!1s0x4d2bbd619d3e2477:0x4b05eea1796c8494!8m2!3d45.3540924!4d-80.1921647!6m1!1e1

ugh_bough

[Phil Harvey]

Lovely.  I'll have to drop by there the next time I'm passing through.

- Phil