Request for Step by Step info to add XMP shotdate data in specified format

[RokkoRokko]

Hi Phil

I am absolutely new to Exiftool to the point its very confusing for me

However my requirement is very simple

I have terrabytes of MPEG files I have shot with my HDV camera that have XMP sidecar files.

My request is for a step by step "How to" regarding batch renaming my files in the following format using the XMP "shotdate" tag.

I wish my new filenames to appear in the xmp shotdate date-time format as follows with the dashes and spaces as shown preceeding the original filename of each file....

2015-02-25  16-12 47 - original Filename

As I am an absolute newbie I am hoping your step by step will also include how to insert the request into Exiftool and how to select the directories I wish to rename.

Can you also include a step by step to time-shift the shotdate info if I have to to compensate for daylight savings errors or camera mis-dating errors.

each directory also includes the xmp files so I am also hoping for info to both ensure the original xmp files are still linking to the renamed files afterwards

Thank you in advance - I have been looking for a solution like this for a very long time.

Regards

RokkoRokko

[Phil Harvey]

Hi RokkoRokko,

This is a bit tricky because the information is in a sidecar file, which requires you use the somewhat advanced -srcfile option.  The command would look like this:

exiftool "-filename<shotdate" -d "%Y-%m-%d %H-%M %S - %%f.mpg" -ext xmp -srcfile %d%f.mpg DIR

where DIR is the name of the directory containing the files (with quotes around it if it contains spaces).  Here I assume that your MPEG files have an extension ".mpg".  Also, you can add the -r option to the command to recursively process subdirectories.

After this, you would use this command to also rename the XMP files:

exiftool "-filename<shotdate" -d "%Y-%m-%d %H-%M %S - %%f.xmp" -ext xmp DIR

See this page for more information about renaming files.

To shift the ShotDate between a range of dates, you could do this:

exiftool -shotdate-=1 -if "$shotdate ge '2014:03:12' and $shotdate le '2014:06:30'" -ext xmp DIR

Which shifts ShotDate back by 1 hour between March 12 and June 30, 2014.  "ge" means "greater than or equal", and "le" means "less than or equal".  The above quoting is for Windows.  Swap the single and double quotes if you are on Mac or Linux.

- Phil

P.S.  I might recommend keeping a verbose log of the rename commands so you have a record of what happened in case something goes wrong.  To do this, add a -v to the command and pipe the output to a text file, like this:

exiftool "-filename<shotdate" -d "%Y-%m-%d %H-%M %S - %%f.mpg" -ext xmp -srcfile %d%f.mpg -v DIR >> out.txt

[RokkoRokko]

Thank you for your exceptionally quick reply Phil

I have been searching for this for some time as I archive so many files and really don't want to have to "Figure out" dates / times for each.

This will give me a unified format for all my files regardless of their origin, now.\

I am a newbie (only found Exiftool a few hours before I mailed you) and have no real idea yet how to operate it but your lines of code seem nice and tidy.

I have copied a few files to experiment on.

I'll let you know how I get along.

[RokkoRokko]

Hi again Phil,

I am definitely suffering from "newbie-itus" and have realized the potential here for my ideal solution to be including

1/ end time and shot length
2/ as well as with ExifTool being a "One Stop Shop" for all my recorded formats rather that use other renamers for other types.


As earlier

"2015-02-25  16-12 47 - original Filename " is the ideal I had envisioned for my new file names.

However, after reading your time shift line I wondered if the formula could be expanded to...

Shotdate (as above) - "time at end of recording" - "[shot length]" -  "original Filename"

...Where the shot length info is added to the shot date to get a time at end of recording.


as in example....

2015-02-25  16-12 47 - 17-24 52 - [01-12 05] - original Filename


That would be awesome if realistic.


My other file types are for recordings to SD cards which don't have a shotdate metadata tag but recording start time is the "date created"  value and end of recording time is "Date Modified" value

Can Exiftool do that also as in a formula to create the same result as above...

2015-02-25  16-12 47 - 17-24 52 - [01-12 05] - original Filename

Man, if it could do that, dream come true as thats all the data I need in my filenames.


Additionally....

Maybe what I am trying to achieve is useful to others on this forum


If anyone else is interested my archiving retrieval system is based on....

My media files archived to.....

"Date progressive" External drive storage with particular drive letters assigned to each Hard Drive volume

ie "04 - 2004-2005" [R]

The solution I am hoping ExifTool can provide is all my files on each drive names as above and thus sortable into progressive Date time order


The next step for me is my Logging system in a Microsoft Word Framesfile system which allows me to select time location of recordings..

1/ - I have the location in the Word system for the Files I am wanting to log - IE March 2004

2/ - I go to the drive, ensure the driveletter [R] is correctly assigned then select the March 2004 files....

3/ - then rightclick drag them to the position in the word logging file under "March 2004"...

4/ - Then select "Create Hyperlink here"

and all the links are pasted but in a block with the full paths.

5/ - From here I select the part of the path I wish to remove and "copy"

6/ - I then select the whole hyperlink block and...

7/ -  go to "replace" in Word and paste the unwanted part into the "Find what:" box and

8/  - Got to the "replace with:" box and put in "^p" (without quotes)

9/ - then  hit "replace all"

This leaves my files list in an ordered list in my word file.

10 / Any other parts I don't want, such as the extension, I can remove from the list by copying the extension into the "Find what:" box and then leave nothing in the "Replace with:" box and, once again, hit "Replace all"


After this exercise I now have a "ListLink" system where, as long as drive with driveletter "R" is connected as driveletter "R" I can hit any of my files in that list and they will play direct from the hard drive.


From here I can make notes under each link describing the shot.

and simply repeat for all other files on all other drives.

It's an awesomely powerful and simple system.

Hope thats of value to you guys out there.

[Phil Harvey]

Doing the date/time calculations you mention is possible using user-defined tags, but a bit more involved so I don't have time to think about this right now.  Tomorrow I should have more time.

- Phil

[RokkoRokko]

Awesome Phil

I shall look forward to that as it will not only make my goal achievable, but also simple by only needing to useExifTool

Regards

Rokkorokko

[Phil Harvey]

To do this, I need to know what tag you want to use for the "shot length" you mentioned.  Do your MPEG files have Duration information?  If so, I can use that.  If not, can I use the difference between DateCreated and DateModified as with the files that don't have a ShotDate?  Also, what are the ExifTool tag names for these tags (extract with the -s option to see the tag names)?  I don't think that you mean DateModified (do you mean ModifyDate perhaps?).

Also, do all files have XMP sidecars, or are there some without?  If some don't have sidecars, what tags may be used from these?

- Phil

[RokkoRokko]

Hi Phil,

The "drag file to ExifTool" function is awesome... It returned the following info...


For the ".mpeg" files  (with XMP sidecar files)

The Exiftool tag produced is

"Shot Date" = Recording start time
"Duration" = Length


Length is also available in standard 'Windows explorer" columns

(note Windows explorer column Headers "Date Created" and "Date Modified" have no relevance here as they refer to the time the file was digitised as these ".Mpeg" files are digitised, rather than ingested, from a tape based camera)


For the ".Mp4" files (without XMP sidecar files) (From recordings to SD Card media)

The Exiftool tags produced are

"File Creation Date/time" = Start time
"File Modification Date/Time" = recording finish time
"Duration" = Length


The other file type I work with, once again digitised from a tape based camera are ".avi" files

However the date / time of recording was added to the filename only

In ExifTool summary the only information relevant is

"File Name" which has the date and time of the recording start already in my chosen format but has the additional original tape name info at the end

"Duration" = length


The current format of the files is...

1996-12-01   15-28 20  DV A-02

(With 3 spaces between date and time as opposed to my preferred 2 spaces)   

If it's possible to provide a formula that would correct the format to match the others by

- resetting date and time part of filename to be 2 space between date and time
- add the length to the "time" part to give the 'finish' time 
- Show the length
- leave the remaining data as original (which can vary)


as in, using the above example,

1996-12-01  15-28 20 - 16-40 25 - [01-12 05] - DV A-02

that would be awesome


This would allow consistency and availability of relevant information in all my formats in my archive


The remaining request is I am, being a newbie, a bit unsure on how to operate Exiftool with the formulas you are supplying...

IE I might wish to rename files on drive 04 driveletter "O" in a folder on the main drive called 1996

or, operate renames in a subfolder called "March" in the "1996" folder

What would be the steps to do this?

If possible I would prefer to use standardised formulas for each type of operation (ie 1 each for the .Mpeg, .Mp4 and .avi files) and direct Exiftool to each folder Then apply the selected standardized formula.



Thank you for your much appreciated efforts, Phil, I am very excited about the potential ExifTool offers.


Regards

RokkoRokko

[Phil Harvey]

Hi RokkoRokko,

I think I amost have enough information to provide you with a Composite user-defined tag and a command to do all of this.  The only thing I am not clear about is exactly what information needs to come from the XMP file?  Or is it only from the video files?

(I don't have time to create the user-defined tag now, so this would have to wait until tomorrow anyway.)

- Phil

[RokkoRokko]

Hi Phil

The only XMP data required, as sourced from the sidecar ".xmp" file,  is in the case of the ".MPEG" files only and that is "Shot Date" as written with the space between shot and date

the other information....

"File Creation Date/time" = Start time
"File Modification Date/Time" = recording finish time
"Duration" = Length

seems to be available when any video file is dropped onto ExifTool, even without an xmp file

Regards

RokkoRokko

[Phil Harvey]

This is a bit tricky because you can't rename both a file and its sidecar from metadata taken from both (because after the first one is renamed, exiftool won't be able to use information from this file to rename the other).

However, I have a plan that will work.

But first, an explanation:  The "File Modification Date/Time" (tag name FileModifyDate) and "File Creation Date/time" (tag name FileCreateDate) you mentioned are not stored in the file itself, but in the filesystem.  As such, they are easily changed by just touching the files with other software, so you must be careful.

To avoid the problem of ShotDate being in a sidecar file, we should first move it over the FileCreateDate of the associated MPEG, with this command (type this into a cmd.exe window):

1. exiftool -tagsfromfile %d%f.xmp "-filecreatedate<shotdate" -r O:/1996/March

But I'm not sure you want to do this.  The problem is that if FileCreateDate is wrong for these files, then FileModifyDate may be as well, and in this case the calculated duration (the difference between these two times) will be wrong.

Once the FileCreateDate and FileModifyDate information is correct for the movie files, you can run this command first to rename the sidecar files:

2. exiftool -config newname.config -tagsfromfile %d%f.mpeg "-filename<$newname.xmp" -ext xmp -r O:/1996/March

Then use this command to rename the video files:

3. exiftool -config newname.config "-filename<$newname.%e" -ext mpeg -ext mp4 -ext avi -r O:/1996/March

In all of these commands I have used "O:/1996/March" as an example directory, and added the -r option to also process subdirectories.  If you want, you can do multiple directories at the same time by adding more of them to the commands.

The last two commands assume that a config file called "newname.config" is in the current directory.  (This config file is attached to this post, and does all the calculations necessary to generate the NewName tag in the format you described.)  Alternatively, you can remove the "-config newname.config" from these commands and activate the config file as described in the comments of the sample config file.

The third command above renames .mpeg, .mp4 and .avi files, but you can change the -ext options to process any files you want.

Be sure to run some trials on a test directory to be sure this does what you want before unleashing it on your videos.

Also, I would recommend adding a verbose (-v) option to these commands, and piping the output to a text file so you will have a record of what ExifTool has done in case you ever need to un-do it.  For example:

exiftool -config newname.config "-filename<$newname.%e" -ext mpeg -ext mp4 -ext avi -r -v O:/1996/March >> out.txt

Finally, be sure to use the most recent version of ExifTool (9.87) to do this, since there have been some bug fixes in recent versions regarding the filesystem date/times in Windows.

Good luck.

- Phil

[RokkoRokko]

Hi Phil

That all makes sense, I think, haha

I do have ExifTool 9.87 version.

However, I am a bit confused, dam "newbie-itus" as I am unsure of what Exiftool process you have offered relates to what result.


I suspect I may have confused you with all the info so re-clarifying...


My footage is all archive, never to be changed apart from the renaming as any editing from them only uses the files as source without ever changing them as changes are saved in a brand new edited file.

What I am asking from you here will be a final solution which will set up my files in archive for later editing (and viewing) access.

Also, as the 3 types (.MPEG, .mp4, .avi) are each completely different situations to each other they would require a separate ExifTool formula / process, for each  of...

1/  - The ".MPEG" - and possible another to deal with the sidecar files
2/  - The ".mp4" files
3/  - The ".avi" files


1/ - ".MPEG" files  - (the ONLY ones with with XMP sidecars).....

the filesystem create and modify dates are completely irrelevant as they only relate to the time digitised which is often way later than actual recording create date and, as far as I know, the XMP metadata doesn't record any 'time recording stopped' type of info

This means the only way to calculate an "end of recording time" is to add the Shot Date time to the Length time to get that correct end time for the file.

To update the filesystem create and modify times could have use, I guess but not really needed as the new filenames from this process will have the info "hard wired" into the filename, which is the purpose.


And also the XMP filename is identical to the MPEG filename apart from the extension.


An idea, but there's probably a good reason for this not to work that you have thought of but..

-  if it was set up that Exiftool grabbed the "shot date" xmp info,
- added that to the MPEG filename along with the(shot date plus length info to get recording end time),
- added the length and then remaining original filename as per my result model...
- THEN it takes that resulting WHOLE NEW FILENAME and uses THAT to rename the associated XMP file, would that not work?

as in result...

2015-02-25  16-12 47 - 17-24 52 - [01-12 05] - original Filename.MPEG

2015-02-25  16-12 47 - 17-24 52 - [01-12 05] - original Filename.xmp 

however, if the xmp rename part is too difficult, or the link gets broken to the original, its not a big deal as I will still keep the .xmp files and will know what .xmp relates to what file by the retention of the original filename.

I wouldn't use the other xmp data much and if I needed to and link was broken, I could easily manually drop the .xmp onto exiftool to view that data anyway, so no real issue.


2/ - ".mp4" files - 

Which formula applies to the .MP4" files that sources the new name from the filestystem time create, modify and duration? (which are correct as relate to time each file created with the modify time being the finish time - as these files were created at record time and not from any earlier recordings on tape)


3/ - ".avi" Files

The ".mp4" formula / process wouldn't work with the avi's as these were digitised from tape, without any XMP data, making the filessystem create and modify times irrelevant (again they only are digitise times)
- There is no other info source to apply except the date time in the actual filename which is the equivalent of the XMP "shot date" time so that formula could only be calculated by adding the filename date and time to the filesystem length info to get the new stop time.

as in

an avi file existing in this format

"1996-12-01   15-28 20  DV A-02"

Which the filesystem states duration as 01-12 15...


The only way this can work as there is no data other than the original filename to work with... is to add the

01-12 15 duration to the  "15-28 20" part of the original filename to get  16-40 35, followed by including the length in brackets before the remainder of the filename

So the new file name would simply be the old filename with the new stop time and then length inserted in the middle after the original date time part.

So the result would look like

1996-12-01  15-28 20 - 16-40 35 - [01-12 15] - DV A-02




I can see this is complicated and apologize if I am being a pain, but once sorted,  this will be a dream come true.

Your time and effort is truly appreciated.

Cheers

Rokkorokko

[Phil Harvey]

Hi RokkoRokko,

Try generating the NewName tag to see how it works:

exiftool -config newname.config -newname O:/1996/March

This won't change anything, and will tell you what the new name will be (minus the extension).

I tried to handle all of your logic with the NewName tag.  But looking at it again I may need to handle the case where the duration should be taken from the file name.

But it would be useful to get some feedback to see if things are looking like they might work.

- Phil

[RokkoRokko]

Cheers Phil,

You have awesome patience and it is appreciated.
I just amended the above and noticed you replied while I did that so may have missed the amendment (Just trying to make my stuff clearer)

I'll have a play with what you have offered so far

Regards

RokkoRokko

[Phil Harvey]

Hi RokkoRokko,

I have read through your explanation, and think that what I have done should work (although I have added an additional check -- see the end of this post).  There isn't a case where the duration needs to be determined from the file name.

My footage is all archive, never to be changed apart from the renaming as any editing from them only uses the files as source without ever changing them as changes are saved in a brand new edited file.

Yes.  The steps I outline only change the FileName, and the FileCreateDate for the files with an XMP sidecar.  The file itself is not changed.  Both of these are stored in the filesystem, not the file.

Also, as the 3 types (.MPEG, .mp4, .avi) are each completely different situations to each other they would require a separate ExifTool formula / process, for each  of...

Yes.  If you take a look at newname.config, you will see logic in there that does this:

1) Parse existing file name and take start time from file name if it contains a time.

2) Otherwise, take the start time from FileCreateDate (which you have already set from ShotDate in the XMP sidecar for those videos which didn't have FileCreateDate set properly).

3) Calculate the end time from the start time plus the Duration if this tag exists in the video.  Otherwise, take the end time from FileModifyDate and calculate the duration from end time minus start time.

4) Generate NewName based on the start time, end time, duration, and the original file name (with the time removed if the time was taken from the file name).

I think this should work for all file types.

I have attached a new version of the config file that does some validity checking on the duration (must be less than 100 hours).

- Phil