inotifywait / Exiftool commands - can you see any problems and anyway to improve

[j038lack]

Hi everyone,
I'm a newbie to Exiftool and shell scripts and am looking to use a wifi enabled camera and a synology box to store photos that are organized and renamed by the Exiftool. To create a cloud based database for users to retrieve photos and the data/sound files with them.

I have created the two scripts running at start up to achieve the above task. The first script renames the .jpg to date/time.jpg and moves to new tag created folders. This works pretty good but doesn't overwrite a .jpg if it already exists and just leave a jpg in the transfer folder unprocessed.

Code Select Expand


#!/bin/sh
#
# rename jpegs by date/time and move to CustomerId / Job Type folder
#

case "$1" in
start|"")
inotifywait -ms --format '%w%f' -e create /volume1/Ricoh/transferPics | while read line
do
# rename jpegs by date/time and move to CustomerId / Job Type folder
perl /lib/Image-ExifTool-9.28/exiftool -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line"
# above line doesnt over
done
;;
stop)
killall inotifywait
;;
*)
echo "Usage: testexifdata2.sh [start|stop]" >&2
exit 3
;;
esac

The Second script waits for files to be moved to the new watched folder. Then creates a .txt file of all Ricoh RMETA data and a .wav file of the embedded sound. Then it check if the transfer folder is empty and runs recursive scan and creates a index type .csv file database of all EXIF able files (may need to make multi-dated backups). This script works mostly some bugs if transfer folder does not empty, database will not be created. and sometimes misses file if i don't run recursively.

Code Select Expand


#!/bin/sh
#
# extract the data from jpegs into jpg, txt, wav, files and create a csv file of all of the database
#

case "$1" in   # create a startup case
start|"")
inotifywait -mr --format '%w' -e move /volume1/RicohDemo/testspot | while read line
do # watch for files moved to and from folder recruisivly
# create a txt file from all RMETA data
perl /lib/Image-ExifTool-9.28/exiftool -m -r -RMETA:all -ext .jpg '/volume1/RicohDemo/testspot' -w %d%f_tag.txt
# extract the sound file
perl /lib/Image-ExifTool-9.28/exiftool -m -r -soundfile -b -w wav -ext .jpg '/volume1/RicohDemo/testspot'
# Run if Transfer folder is empty confriming all files are moved
if find /volume1/RicohDemo/transferPics -maxdepth 0 -empty | read
then
#create a CSV file database of all files saved
perl /lib/Image-ExifTool-9.28/exiftool -m -csv -r '/volume1/RicohDemo/testspot' > '/volume1/RicohDemo/testspot/testdatabase.csv'
fi
done
;;
stop)  # create a stop case
killall inotifywait
;;
*)  # create a Usage case
echo "Usage: testexifdata2.sh [start|stop]" >&2
exit 3
;;
esac

I'm looking to learn how to improve the code and maybe combined the two scripts if possible.
Be easy on me show me how to improve, not tell me what i have done wrong.
Any help would be great

Thank
J03

[j038lack]

So i did a little digging and have decided to use only one Inotifywait command and join the two scripts together.

As a result it seems fine. but it is recrusively scanning files already changed and could cause problem as the database grows.

code below-

Code Select Expand


#!/bin/sh
#
# rename jpegs by date/time and move to CustomerId / Job Type folder
#

case "$1" in
start|"")
inotifywait -ms --format '%w%f' -e create /volume1/Ricoh/transferPics | while read line
do
# rename jpegs by date/time and move to CustomerId / Job Type folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line"
sleep 3
# create a txt file from all RMETA data
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -r -RMETA:all -ext .jpg '/volume1/RicohDemo/testspot' -w %d%f_tag.txt
# extract the sound file
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -r -soundfile -b -w wav -ext .jpg '/volume1/RicohDemo/testspot'
# Run if Transfer folder is empty confriming all files are moved
if find /volume1/RicohDemo/transferPics -maxdepth 0 -empty | read
then
#create a CSV file database of all files saved
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -csv -r '/volume1/RicohDemo/testspot' > '/volume1/RicohDemo/testspot.csv'
fi
#done
done
;;
stop)
killall inotifywait
;;
*)
echo "Usage: testexifdata2.sh [start|stop]" >&2
exit 3
;;
esac

I'm looking for a little help storing the directory name and filename as a string so i can be more specific when running the other exiftool commands instead of running everything recursively - see below

perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line" - Is there a way to store these changes as a string to recall later

So I can run this:

perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -RMETA:all '$newDIR/$newFilename' -w %d%f_tag.txt

Any help would be great
J03

[Phil Harvey]

There is no way in ExifTool to store the directory and recall it later.  What you could do is produce the .txt file in the original directory, then move it then the image file (assuming .jpg) to the destination:

Code Select Expand

exiftool -q -q -m -RMETA:all -w %d%f_tag.txt "$line"
exiftool -q -q '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" -srcfile %d%f_tag.txt "$line"
exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line"

- Phil

[j038lack]

Thanks Phil,

I completely over look the -scrfile command. With a few correction I did accomplish what I was looking for.

With the pointers you gave me I realized i had to save a the files to Temp Directory due to the inotifywait -e create command as any file created including the temp files in the same folder would cause the inotifywait to run, causing errors and slowing the code down. Also had to change ... highlighted in orange.

exiftool -q -q '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" -srcfile %d%f_tag.txt "$line"

to .txt or .wav depending on the file type I was saving.

So now the improved code waits for a file to be created in a FTP folder. Then extracts the RMETA txt file and sound file in a Temp directory from the JPG. It then renames the newly created temp files to Date/Time file name format and moves to a directory created from the source JPG Exif tags. Where the Source JPG will be renamed and move to also.

The script then ends by checking if the FTP folder is empty and creates csv database file of all the files in the photo database folder. This new script is now added to my synology startup and is working good so far

New Code:

Code Select Expand


# rename jpegs by date/time and move to CustomerId / Job Type folder
#

case "$1" in
start|"")
inotifywait -ms --format '%w%f' -e create /volume1/Ricoh/transferPics | while read line
do
# create a txt file from all RMETA data and a sound file
sleep 3
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -r -soundfile -b -w /volume1/Ricoh/temp/%f.wav "$line"
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -RMETA:all -w /volume1/Ricoh/temp/%f_tag.txt "$line"
# rename RMETA txt and wav file by date/time and move to CustomerId / Job Type folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.wav "-filename<datetimeoriginal" -srcfile /volume1/Ricoh/temp/%f.wav "$line"
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S_tag.txt "-filename<datetimeoriginal" -srcfile /volume1/Ricoh/temp/%f_tag.txt "$line"
# rename jpegs by date/time and move to CustomerId / Job Type folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory</volume1/Ricoh/testspot/$CustomerID/$JobType' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line"
#create a CSV file database of all files saved
find /volume1/Ricoh/transferPics  -maxdepth 0 -empty -exec perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -csv -r '/volume1/Ricoh/testspot' > '/volume1/Ricoh/testspot/testdatabase.csv' \;
done
;;
stop)
killall inotifywait
;;
*)
echo "Usage: testexifdata2.sh [start|stop]" >&2
exit 3
;;
esac

On a side note after extracting -RMETA:all to a .txt file does anyone know a simple way to read the .txt file and record the the custom TAGS that could then be used as a string so the code would not be so static.

Thanks Again
J03

[j038lack]

Ok to make the script a little less Static, I had to switch to a Bash script as sh did not allow for Arrays.

As you can see below, I use Exiftool to extract all the RMETA data to a .xml file then used grep and sed to rip apart the xml file into only the Tag names and stored them in a Array to be used when moving the files to TAG created folders even if i didn't know what the TAG names are.

perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -RMETA:all -X "$line" > ${temp_folder}'filename.xml' # extract RMETA data.
RMETA_array=($(cat ${temp_folder}filename.xml | grep -o '<RMETA:.*>' | sed 's/<RMETA:/$/' | sed 's/>.*//')) # find only <RMETA: txt lines | then remove <RMETA:  | then remove everything after >       then put the TAG names in a Array.

Also made some minor adjustments to make the script a little more friendly to change for multiple users. By defining some config strings that can be changed easily.

And one final note that haunted me for a week. I couldn't get the script to run in the backgroud it would hang open on the While loop.

Easy solution for those playing  ----- Added the & after the done in the While Loop ( done& )

See final code below:

Code Select Expand


#!/opt/bin/bash
#
# rename jpegs by date/time and move to CustomerId / Job Type folder
#

case "$1" in
start|"")
#echo "started"
# Configuration setting for FTP transfer, Temp, and PhotoDataBase folders
temp_folder='/volume1/homes/UserCamera/temp/'
ftp_folder='/volume1/homes/UserCamera/TransferPics'
photodb='/volume1/UserAccount/PhotoDataBase/'
inotifywait -ms --format '%w%f' -e create ${ftp_folder} | while read line
do
#echo $line "create a txt file from all RMETA data and a sound file"
# also extract a XML file from RMETA
sleep 3
# extract Sound file and stored in Temp folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -soundfile -b -w ${temp_folder}%f.wav "$line"
# extract RMETA data txt file and stored in Temp Folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -RMETA:all -w ${temp_folder}%f_tag.txt "$line"
# extract RMETA data xml file and stored in Temp Folder
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -RMETA:all -X "$line" > ${temp_folder}'filename.xml'
RMETA_array=($(cat ${temp_folder}filename.xml | grep -o '<RMETA:.*>' | sed 's/<RMETA:/$/' | sed 's/>.*//'))
#echo "rename RMETA txt and wav file by date/time and move to 1st RMETA / 2nd RMETA folder"
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory<'${photodb}''${RMETA_array[0]}'/'${RMETA_array[1]}'' -d %Y-%m-%d_%H%M%S.wav "-filename<datetimeoriginal" -srcfile ${temp_folder}%f.wav "$line"
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory<'${photodb}''${RMETA_array[0]}'/'${RMETA_array[1]}'' -d %Y-%m-%d_%H%M%S_tag.txt "-filename<datetimeoriginal" -srcfile ${temp_folder}%f_tag.txt "$line"
#echo "rename jpegs by date/time and move to CustomerId / Job Type folder"
perl /lib/Image-ExifTool-9.28/exiftool -q -q -m '-directory<'${photodb}''${RMETA_array[0]}'/'${RMETA_array[1]}'' -d %Y-%m-%d_%H%M%S.%%e "-filename<datetimeoriginal" "$line"
#echo "files moved"
#echo "Run if Transfer folder is empty confriming all files are moved"
find ${ftp_folder}  -maxdepth 0 -empty -exec perl /lib/Image-ExifTool-9.28/exiftool -q -q -m -csv -r '${photodb}' > ${photodb}'testdatabase.csv' \;
#echo "removed temp xml file"
rm ${temp_folder}'filename.xml'
#echo "updated database"
done&
;;
stop)
killall inotifywait
;;
*)
echo "Usage: testexifdata2.sh [start|stop]" >&2
                exit 3
;;
esac



Thanks Phil for the help
J03

PS.    I still got to send you more JPG files with Audio embedded so you can figure out how extract the sound from all Custom Fields the ricoh camera has.

[Phil Harvey]

PS.    I still got to send you more JPG files with Audio embedded so you can figure out how extract the sound from all Custom Fields the ricoh camera has.

Yes, please do.

- Phil

[j038lack]

Yes, please do.
- Phil

Hi Phil,
I Sent the test Pics to you.

Thanks
J03

[Phil Harvey]

Thanks.  I'll download those tomorrow and let you know how it goes extracting the audio information.

- Phil

[Phil Harvey]

I got the files, and have been able to extract these sound files.  ExifTool 9.32 will be able to extract all of the sound files from an image with this command:

exiftool -soundfile -b -a -W %d%f%c.wav DIR

- Phil