Main Menu

Ubuntu Automation

Started by polayer2015, August 04, 2017, 10:17:26 AM

Previous topic - Next topic

polayer2015

I would like to automate my photo processing on Ubuntu 16.04.  My photos are upload via photosync on IOS to a qnap network folder.  Folder has been permanently mapped and is accessible by exiftool.  From there I would like to automate the renaming of each photo into a folder structure of Year/Month/Date.    I can do this separately via CLI, but would like to automate the process.  So far I have:

This renames the files based on the parameters I want to use.
exiftool -d %Y%m%d_%H%M%%-c.%%e "-filename<CreateDate" DIR

This places my photos into a folder structure of Year/Month/Date located at /home/myusraccount/2017. 
exiftool -d %Y/%B/%d "-directory<datetimeoriginal" *.jpg

I would like to merge the first two steps and move the files into my master photo collection located at /media/Multimedia/My Pictures.  This folder also uses the  Year/Month/Date structure to store photos.  I would like for this to take place automatically once new photos are uploaded from photosync via my iphone or new images are added from my DSLR.  Thanks in advance for any help with my setup.






Phil Harvey

I'm not clear.  You want to merge these 2 commands?  If you can use CreateDate for both, this would work:

exiftool -d %Y/%B/%d/%Y%m%d_%H%M%%-c.%%e "-filename<CreateDate" DIR

To do this automatically you could add this as a cron job, but whatever copies the images into the source directory must first copy to a temporary file then rename so that exiftool doesn't pick up partially-written files.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

polayer2015

Thanks for the quick reply.  This worked fantastic!  How can I set the destination folder that the files are moved to.  With this cli, the files are moved to /home/myusraccount/2017.  I would like the final destination to be /media/Multimedia/My Media/My Pictures.  This would require files to be copied into preexisting folders with the directory structure of year/month/date.  Thanks in advance.

Phil Harvey

Since we specified a relative directory, the destination directory will be relative to the current working directory, so you could either first "cd" to "/media/Multimedia/My Media/My Pictures", or alternatively specify an absolute directory like this:

exiftool -d "/media/Multimedia/My Media/My Pictures/%Y/%B/%d/%Y%m%d_%H%M%%-c.%%e" "-filename<CreateDate" DIR

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

polayer2015

Thanks!  Once again this worked perfectly.  Just reviewed my work flow and had a few additional questions.  My wife and I will use the IOS app photosync to upload photos/videos to a network folder (/media/Multimedia/My Media/My Pictures/Need to sort).  I wanted see if exiftool could scan this folder recursively (/media/Multimedia/My Media/My Pictures/Need to sort/My Phone and /media/Multimedia/My Media/My Pictures/Need to sort/Wifes Phone).  The next question is how to handle photos and videos.  Photos are stored /media/Multimedia/My Media/My Pictures and videos are stored in /media/Multimedia/My Media/My Pictures/My Videos in a Year/Month/Day structure.  Sorry for complicating the process, but this would make my digital life so much easier if I could automate these steps.

Phil Harvey

Again, I'm not clear on what you are asking, but just add the -r option to process the source directory recursively.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

polayer2015

Thanks for the help so far!

Ran this:
exiftool -r -d "/media/Multimedia/My Media/My Pictures/%Y/%B/%d/%Y%m%d%%-c.%%e" "-filename<CreateDate" "/media/Multimedia/My Media/My Pictures/Need to Sort"

Got this:
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0551.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0441.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0440.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0439.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0437.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0356.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0309.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0308.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0306.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0297.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0273.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0255.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0252.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0196.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0194.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0159.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0160.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0156.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0155.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0146.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0133.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0562.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0476.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0438.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0484.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0550.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0372.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0370.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0364.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0363.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0342.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0331.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0311.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0314.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0310.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0307.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0287.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0254.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0256.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0239.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0222.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0215.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0211.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0192.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0189.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0182.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0166.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0165.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0157.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0148.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0145.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0315.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0369.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0573.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0362.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0359.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0324.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0242.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0290.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0253.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0361.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0143.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0371.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0487.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0225.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0483.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0330.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0303.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0296.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0514.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0320.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0300.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0367.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0243.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0323.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0542.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0301.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0227.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0150.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0144.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0135.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0100.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0007.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0006.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0564.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0137.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0283.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0183.JPG
Warning: No writable tags set from /media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0184.JPG
    5 directories scanned
   37 image files updated
   89 image files unchanged




Seems like I have 89 images that are in limbo. Files are no longer in directory and were not moved to final destination.  Not sure how to address the writable tag issue.

To clarify my second request, I would like to separate .JPG's  and .MOV's .  Photosync (IOS App) uploads my photos and videos into a permanently mounted network folder located at /media/Multimedia/My Media/My Pictures/Need to sort/My Phone and /media/Multimedia/My Media/My Pictures/Need to sort/Wifes Phone.  From there I would like exiftool to recursively scan the folders, rename, and move to specific folders. Photos are stored in /media/Multimedia/My Media/My Pictures and videos are stored at /media/Multimedia/My Media/My Pictures/My Videos.  Both locations have year/month/date directory structures.  The goal is to run a scheduled exiftool command that can distinguish between .JPG/.MOV and place the files in the appropriate directory.  Hopefully this clarifies my request.

Hayo Baan

Regarding the non-writable tag issue, check to see if the images indeed have a CreateDate. They probably don't so you'll need to use another tag instead.

To select only JPG files add -ext JPG, similarly for MOV. That should easily solve the split of images and videos. To split between your phone and your wife's is a bit more of a challenge. But is doable too as long as you have good way to make the distinction. Is there perhaps a unique tag of part of the filename?
Hayo Baan – Photography
Web: www.hayobaan.nl

polayer2015

Quote from: Hayo Baan on August 05, 2017, 01:20:01 PM
Regarding the non-writable tag issue, check to see if the images indeed have a CreateDate. They probably don't so you'll need to use another tag instead.

Running this
exiftool -time:all -s '/media/Multimedia/My Media/My Pictures/Need to Sort/iPhone/Camera Roll/IMG_0551.JPG'

Gave me this
FileModifyDate                  : 2017:08:05 06:12:20-05:00
FileAccessDate                  : 2017:08:05 07:33:03-05:00
FileInodeChangeDate             : 2017:08:05 06:12:20-05:00


I ran a few of the files and they all have similar information.  These files were created in mid June, July, and August.  Not sure why information was not recorded on wife's iphone.  Would referencing information from this thread fix the 89 files that have no writable tags?  Specifically, adding something like this to the command in case "-filename<CreateDate" creates the No writable tags set from issue?

exiftool -timecreated<datetimeoriginal -datecreated<datetimeoriginal file

Thanks for any advice or help!



Hayo Baan

#9
It looks like all/most metadata of those files was stripped. This could happen if the photos were edited. In that case your use of datetime original won't help either. You either have to use e.g. FileModifyDate or use some other means to determine the correct date.

Using a non existing tag is what causes the exiftool error (and yes, the error message is kinda misleading in this case since it's actually a tag that is being read that is missing)

Edit: correction
Hayo Baan – Photography
Web: www.hayobaan.nl

polayer2015

#10
Quote from: Hayo Baan on August 06, 2017, 03:24:09 AM
It looks like all/most metadata of those files was stripped. This could happen if the photos were edited. In that case your use of datetime original won't help either. You either have to use e.g. FileModifyDate or use some other means to determine the correct date.

Using a non existing tag is what causes the exiftool tag (and yes, the error message is kinda misleading in this case since it's actually a tag that is being read that is missing)

I think I found the culprit.  It looks like a combination of Photosync (IOS app) and the social media apps my wife is using.  Photosync and the social media apps appear to be modifying/deleting/or not creating exif data.  I will need to test further and possibly contact the developer to see why exif data is being stripped during modification/transfer.   

One final question.  I would like to create a directory structure of year/month/date for my files.  I'm currently using %Y/%B/%d, which works perfect.  Is there an argument that can create a directory structure like this:
2017
01 - January
(Then folders by date as needed 01...31)
02 - February
03 - March
...12 - December


Hayo Baan

With some perl trickery the directory structure you're after can be achieved:

-fileName'<${datetimeoriginal;
my @months=qw(January February March April May June July August September October November December);
my $months=join("|", @months);
my %monthNr; for (my $i=0; $i<@months; ++$i) { $monthNr{$months[$i]} = $i+1; }
s/($months)/sprintf("%02d",$monthNr{$1}) . " - $1"/e;}' -d "%Y/%B/%d"


Note: If you use -testName instead of -fileName you can see the resulting filename without actually moving the file. Very helpful :)
Hayo Baan – Photography
Web: www.hayobaan.nl

Phil Harvey

I don't have much time to think about what Hayo was trying to do, so I may be off base here, but I thought that creating the directory structure you want should be simple:

exiftool "-directory<datetimeoriginal" -d "%Y/%m - %B/%d" DIR

See here for more information.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Hayo Baan

Quote from: Phil Harvey on August 06, 2017, 03:30:42 PM
exiftool "-directory<datetimeoriginal" -d "%Y/%m - %B/%d" DIR

Doh ::) I couldn't find the code for the month number in the manual (how I overlooked that in the man page, I don't know...) so I made up this piece of perl trickery. But %m should do the trick too ;D
Hayo Baan – Photography
Web: www.hayobaan.nl

polayer2015

Quote from: Phil Harvey on August 06, 2017, 03:30:42 PM
I don't have much time to think about what Hayo was trying to do, so I may be off base here, but I thought that creating the directory structure you want should be simple:

exiftool "-directory<datetimeoriginal" -d "%Y/%m - %B/%d" DIR

See here for more information.

- Phil

Thank you Phil and Hayo!  Your support has been amazing and I'm finally able to organize my digital life!  In case anyone wants to know, here is the code I tested.

exiftool -r -d "/home/jim/Desktop/Test%Y/%m - %B/%d/%Y%m%d%%-c.%%e" "-filename<CreateDate" "/media/Multimedia/My Media/My Pictures/Need to Sort"

This renames my JPG's/Mov's and places them into folders by year/month/date.  Numbers are added to the months to create logical ordering inside the year folder.  Thanks once again!   :)