Rename raw image and darktable XMP

[mkl]

I tried some lines from this thread (without luck) and first replied there but noticed it didn't bump the topic (due to it being too old?). So, here's a separate thread for my .ARW/.ARW.xmp pairs.

Thanks to you, works great with these lines:

exiftool -V -d "%Y-%m-%d_%Hh%Mm%S_%f" -tagsfromfile %f.CR2 '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %-.4f.CR2 '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename .

I'm up to a huge move/clean task and would love for this work with my .ARW/.ARW.xmp pairs. I don't have any JPEGs.

Following the above example, I did:

Code Select Expand

exiftool -r -P -progress -ext arw -d "/outputdir/%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e" -tagsfromfile %f.ARW '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %-.4f.ARW '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename "/inputdir/"

And that gave me the following for every file:

Code Select Expand

Warning: Error opening file - DSC09589.ARW
Warning: Error opening file - DSC0.ARW
Nothing changed in /inputdir/DSC09589.ARW

Clues?

Unless exiftool has a better way of handling paired files since this was suggested, the more complete solution would be to use multiple statements to process files in normal ascending sequence, and to process dependent paired files first so we can find the original raw file with tags.

I also wonder if there's a better way to handle darktable .xmp files now. I was used to the "automatic" support for Adobe's standard compliant file.xmp pattern and I realize that darktable's file.ext.xmp is the deviation. But I would love for file.ext.xmp to be supported. If I don't find a solution here I'll probably just throw away the .ext.xmp and start from scratch.

[Phil Harvey]

Your post to the old thread did bump the thread, btw.

I don't have time to read through the old thread right now though.

- Phil

[StarGeek]

Your problem is the -tagsfromfile %f.ARW part.

You are telling exiftool to look for the ARW file in the current directory, but you are processing files in the "/inputdir/" subdirectory, i.e. you are processing files ./inputdir/file.arw and ./inputdir/file.xmp, but telling exiftool to read data from ./file.arw

Try adding the %d directory variable (see the -w (-TextOut) option for details on the percent variables)
-tagsfromfile %d%f.ARW

[mkl]

Your problem is the -tagsfromfile %f.ARW part.

You are telling exiftool to look for the ARW file in the current directory, but you are processing files in the "/inputdir/" subdirectory, i.e. you are processing files ./inputdir/file.arw and ./inputdir/file.xmp, but telling exiftool to read data from ./file.arw

Try adding the %d directory variable (see the -w (-TextOut) option for details on the percent variables)
-tagsfromfile %d%f.ARW

Thanks! This is clearly way beyond my skill level. I tried using the current directory as in this example:

Thanks to you, works great with these lines:


exiftool -V -d "%Y-%m-%d_%Hh%Mm%S_%f" -tagsfromfile %f.CR2 '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %-.4f.CR2 '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename .

Here's what I tried:
/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e -tagsfromfile %f.ARW '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %-.4f.ARW '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename .
======== ./DSC00163.ARW.xmp [1/6]
Warning: Error opening file - DSC00163.ARW.ARW
Warning: Error opening file - DSC00163.ARW
Nothing changed in ./DSC00163.ARW.xmp


Did not go well. Tried adding %d%f like so:

/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e -tagsfromfile %d%f.ARW '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %-.4f.ARW '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename .
======== ./DSC00015.ARW.xmp [1/10]
Warning: Error opening file - ./DSC00015.ARW.ARW
'./DSC00015.ARW.xmp' --> './2018/20180510/20180510_095434_00.xmp.xmp'


Neither went well. Seems like there's room for errors. darktable also uses a 01 extension for duplicates (filename_01.ARW.xmp) which further complicates things. Perhaps I'll have to start from scratch with my edits after renaming...

Would love for this just to work like it does with darktable .XMPs and not just proprietary software (Lightroom/ARW). But yeah, I understand if there's reluctance considering darktable is not confirming to XMP naming standards.

[mkl]

Your post to the old thread did bump the thread, btw.

Sorry, saw now it was in the Newbies category... 

[mkl]

Would it perhaps be easier to somehow use other *nix CLI tools to find the pairs and then run ExifTool on matching files/pairs and not on entire dirs using ExifTool directly?

[StarGeek]

Can you show some exact examples of the file names? I seem to be missing something. But you will get errors you have two different -tagsfromfile options. One of them will always fail, which would be normal and can be ignored.

One error I do see is that you are doubling up on the extension. You have a %e in your date format string
-d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e
And then again here
'-FileName<${DateTimeOriginal}.%e'

Use either
-d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e '-FileName<DateTimeOriginal'
or
-d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-FileName<${DateTimeOriginal}%+.2c.%e'

[mkl]

Thanks again StarGeek!

Use either
-d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e '-FileName<DateTimeOriginal'

I did that by issuing:

Code Select Expand

/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S%%+.2c.%%e '-FileName<${DateTimeOriginal}.%e' -tagsfromfile %d%f.ARW -tagsfromfile %-.4f.ARW '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename . in a directory containing files with the following filename formatting:
DSC01893.ARW
DSC01893.ARW.xmp
DSC01894.ARW
DSC01894.ARW.xmp
DSC01895.ARW
DSC01895.ARW.xmp
DSC01896.ARW
DSC01896.ARW.xmp

The output was:

Code Select Expand

======== ./DSC01896.ARW.xmp [1/8]
Warning: Error opening file - ./DSC01896.ARW.ARW
'./DSC01896.ARW.xmp' --> './2021/20210725/20210725_055845_00.xmp.xmp'
======== ./DSC01896.ARW [2/8]
Warning: Error opening file - DSC0.ARW
'./DSC01896.ARW' --> './2021/20210725/20210725_055845_00.ARW.ARW'
Warning: [minor] Oversized SubIFD StripByteCounts (48674304 bytes, but expected 42590016) - ./DSC01896.ARW
======== ./DSC01895.ARW.xmp [3/8]
Warning: Error opening file - ./DSC01895.ARW.ARW
'./DSC01895.ARW.xmp' --> './2021/20210725/20210725_055800_00.xmp.xmp'
======== ./DSC01895.ARW [4/8]
Warning: Error opening file - DSC0.ARW
'./DSC01895.ARW' --> './2021/20210725/20210725_055800_00.ARW.ARW'
Warning: [minor] Oversized SubIFD StripByteCounts (48674304 bytes, but expected 42590016) - ./DSC01895.ARW
======== ./DSC01894.ARW.xmp [5/8]
Warning: Error opening file - ./DSC01894.ARW.ARW
'./DSC01894.ARW.xmp' --> './2021/20210725/20210725_055736_00.xmp.xmp'
======== ./DSC01894.ARW [6/8]
Warning: Error opening file - DSC0.ARW
'./DSC01894.ARW' --> './2021/20210725/20210725_055736_00.ARW.ARW'
Warning: [minor] Oversized SubIFD StripByteCounts (48674304 bytes, but expected 42590016) - ./DSC01894.ARW
======== ./DSC01893.ARW.xmp [7/8]
Warning: Error opening file - ./DSC01893.ARW.ARW
'./DSC01893.ARW.xmp' --> './2021/20210725/20210725_055710_00.xmp.xmp'
======== ./DSC01893.ARW [8/8]
Warning: Error opening file - DSC0.ARW
'./DSC01893.ARW' --> './2021/20210725/20210725_055710_00.ARW.ARW'
Warning: [minor] Oversized SubIFD StripByteCounts (48674304 bytes, but expected 42590016) - ./DSC01893.ARW
    1 directories scanned
    1 directories created
    4 image files created
    4 image files updated

The results was that four files stayed in the current dir i ran the command from:
DSC01893.ARW
DSC01894.ARW
DSC01895.ARW
DSC01896.ARW

The /2021/20210725/ dir was also created with four .ARW.ARW/.xmp.xmp pairs. example:
20210725_055710_00.ARW.ARW and 20210725_055710_00.xmp.xmp

-d ./%Y/%Y%m%d/%Y%m%d_%H%M%S'-FileName<${DateTimeOriginal}%+.2c.%e'

Code Select Expand

/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S'-FileName<${DateTimeOriginal}%+.2c.%e' -tagsfromfile %d%f.ARW -tagsfromfile %-.4f.ARW '-FileName<${DateTimeOriginal}.%e' -fileOrder -filename . created:

Code Select Expand


drwxrwx---+ 3 m  m         3 Mar 16 14:57 2021
-rwxrwx---+ 1 m  m  48884028 Jul 25  2021 DSC01809.ARW
-rwxrwx---+ 1 m  m  48887296 Jul 25  2021 DSC01809.ARW_original
-rwxrwx---+ 1 m  m  48881550 Jul 25  2021 DSC01810.ARW
-rwxrwx---+ 1 m  m  48883200 Jul 25  2021 DSC01810.ARW_original
-rwxrwx---+ 1 m  m  48895050 Jul 25  2021 DSC01811.ARW
-rwxrwx---+ 1 m  m  48899584 Jul 25  2021 DSC01811.ARW_original
In the 2021 subdir there was three files named:

Code Select Expand

20210725_002135-FileName<${DateTimeOriginal}Sun Jul 25 00:21:35 CEST 2021.2c.25.xmp
20210725_002200-FileName<${DateTimeOriginal}Sun Jul 25 00:22:00 CEST 2021.2c.25.xmp
20210725_002210-FileName<${DateTimeOriginal}Sun Jul 25 00:22:10 CEST 2021.2c.25.xmp

[StarGeek]

Ok, I see I made a couple mistakes in my post.

But this works here
exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S -tagsfromfile %d%f.ARW '-Filename<${DateTimeOriginal}%+.2c.%e' -tagsfromfile %-.4f.ARW '-Filename<${DateTimeOriginal}%+.2c.arw.%e'  -fileOrder -filename .

The %c and %e have been moved out of the date format string because of the .arw.xmp naming scheme.

Example

Code Select Expand

Y:\!temp\x\y>exiftool -a -s -s3 -r -ext xmp -ext arw -dummy .
======== ./DSC01893.ARW
======== ./DSC01893.arw.xmp
======== ./DSC01894.ARW
======== ./DSC01894.arw.xmp
    4 directories scanned
    4 image files read

Y:\!temp\x\y>exiftool -P -progress -ext arw  -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S -tagsfromfile %d%f.ARW "-Filename<${DateTimeOriginal}%+.2c.%e" -tagsfromfile %-.4f.ARW "-Filename<${DateTimeOriginal}%+.2c.arw.%e"  -fileOrder -filename .
======== ./DSC01894.arw.xmp [1/4]
Warning: Error opening file - ./DSC01894.arw.ARW
'./DSC01894.arw.xmp' --> './2018/20180317/20180317_125752_00.arw.xmp'
======== ./DSC01894.ARW [2/4]
Warning: Error opening file - DSC0.ARW
'./DSC01894.ARW' --> './2018/20180317/20180317_125752_00.ARW'
======== ./DSC01893.arw.xmp [3/4]
Warning: Error opening file - ./DSC01893.arw.ARW
'./DSC01893.arw.xmp' --> './2018/20180317/20180317_125752_01.arw.xmp'
======== ./DSC01893.ARW [4/4]
Warning: Error opening file - DSC0.ARW
'./DSC01893.ARW' --> './2018/20180317/20180317_125752_01.ARW'
    1 directories scanned
    1 directories created
    4 image files updated

Y:\!temp\x\y>exiftool -a -s -s3 -r -ext xmp -ext arw -dummy .
======== ./2018/20180317/20180317_125752_00.ARW
======== ./2018/20180317/20180317_125752_00.arw.xmp
======== ./2018/20180317/20180317_125752_01.ARW
======== ./2018/20180317/20180317_125752_01.arw.xmp
    6 directories scanned
    4 image files read

[mkl]

Ok, I see I made a couple mistakes in my post.

But this works here
exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S -tagsfromfile %d%f.ARW '-Filename<${DateTimeOriginal}%+.2c.%e' -tagsfromfile %-.4f.ARW '-Filename<${DateTimeOriginal}%+.2c.arw.%e'  -fileOrder -filename .

Thanks again, I'm so thankful! This worked well (despite the error messages) until I had one of those duplicate files... When you want to have several different edits with one raw file in darktable, you use the "duplicate" function.

If you press "duplicate" for DSC02055.ARW (and DSC02055.ARW.xmp), you will now also have a DSC02055_01.ARW.xmp (for the duplicate). Press duplicate again and there will be a DSC02055_02.ARW.xmp. This, I guess, further complicates the rename task.

mdrmike also hinted about a possible "burst problem".

But Wait ...

After using this solution a couple times, I realized reversing input by filename to rename, and using %C to ensure unique filenames (such as in the OP {DateTimeOriginal}%-.2nc and in Phil Harveys suggestion); the result can cause camera "bursts" to be numbered in reverse chronological order.

Is it out of the question to "natively" support renaming of raw files and darktable XMP pairs (and duplicates)?

[StarGeek]

If you press "duplicate" for DSC02055.ARW (and DSC02055.ARW.xmp), you will now also have a DSC02055_01.ARW.xmp (for the duplicate). Press duplicate again and there will be a DSC02055_02.ARW.xmp. This, I guess, further complicates the rename task.

You could try adding another entry to subtract those extra three characters
-tagsfromfile %-.7f.ARW '-Filename<${DateTimeOriginal}%+.2c.arw.%e'

Is it out of the question to "natively" support renaming of raw files and darktable XMP pairs (and duplicates)?

The final post in the original thread ran a command separately for each fill type, with the ARW rename running last.

But thinking about it, the whole thing might be moot. Run exiftool on an XMP file and see if it has the correct DateTimeOriginal value. There's no reason to use -TagsFromFile if the XMP file has the correct date/time already.

I changed -fileOrder -filename into -fileOrder filename so the file order isn't reversed, but that may not even be necessary depending upon the file system.
exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-Filename<${DateTimeOriginal}%+.2c.%e' -fileOrder filename .

[mkl]

I changed -fileOrder -filename into -fileOrder filename so the file order isn't reversed, but that may not even be necessary depending upon the file system.
exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-Filename<${DateTimeOriginal}%+.2c.%e' -fileOrder filename .

I ran this on a test dir containing 7 files:

Code Select Expand

├── DSC00678.ARW
├── DSC00678.ARW.xmp
├── DSC00679.ARW
├── DSC00679.ARW.xmp
├── DSC02055_01.ARW.xmp
├── DSC02055.ARW
└── DSC02055.ARW.xmp

1 directory, 7 files
Notice that DSC02055.ARW has two .xmp files. DSC02055.ARW.xmp and DSC02055_01.ARW.xmp with a _01 suffix, which was created using the duplicate function in darktable.

DSC02055.ARW.xmp should become YYYYMMDD_HHMMSS_SEQUENCENUMBERPERSECOND.xmp and the duplicate .xmp (DSC02055_01.ARW.xmp) should be renamed YYYYMMDD_HHMMSS_SEQUENCENUMBERPERSECOND_01.xmp.

This was the output of ExifTool:

Code Select Expand

/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-Filename<${DateTimeOriginal}%+.2c.%e' -fileOrder filename .
======== ./DSC00678.ARW [1/7]
'./DSC00678.ARW' --> './2021/20210617/20210617_223111_00.ARW'
======== ./DSC00678.ARW.xmp [2/7]
'./DSC00678.ARW.xmp' --> './2021/20210617/20210617_223111_00.xmp'
======== ./DSC00679.ARW [3/7]
'./DSC00679.ARW' --> './2021/20210617/20210617_223230_00.ARW'
======== ./DSC00679.ARW.xmp [4/7]
'./DSC00679.ARW.xmp' --> './2021/20210617/20210617_223230_00.xmp'
======== ./DSC02055.ARW [5/7]
'./DSC02055.ARW' --> './2021/20210803/20210803_053337_00.ARW'
======== ./DSC02055.ARW.xmp [6/7]
Warning: Masks_history is not a structure! [x15] - ./DSC02055.ARW.xmp
'./DSC02055.ARW.xmp' --> './2021/20210803/20210803_053337_00.xmp'
======== ./DSC02055_01.ARW.xmp [7/7]
Warning: Masks_history is not a structure! [x15] - ./DSC02055_01.ARW.xmp
'./DSC02055_01.ARW.xmp' --> './2021/20210803/20210803_053337_01.xmp'
    1 directories scanned
    2 directories created
    7 image files updated

This was the result according to tree:

Code Select Expand

tree
.
└── 2021
    ├── 20210617
    │   ├── 20210617_223111_00.ARW
    │   ├── 20210617_223111_00.xmp
    │   ├── 20210617_223230_00.ARW
    │   └── 20210617_223230_00.xmp
    └── 20210803
        ├── 20210803_053337_00.ARW
        ├── 20210803_053337_00.xmp
        └── 20210803_053337_01.xmp

4 directories, 7 files

We're close! DSC02055_01.ARW.xmp became 20210803_053337_01.xmp when it should have become 20210803_053337_00_01.xmp.

Sorry for digging this hole deeper and deeper! I realize this was not an easy challenge.

There's no reason to use -TagsFromFile if the XMP file has the correct date/time already

Without knowing anything about ExifTool (compared to you), it seems that using filenames (not metadata) to extract logic would be the safer way.

EDIT: Made lots of edits!

[mkl]

Without knowing anything about ExifTool (compared to you), it seems that using filenames (not metadata) to extract logic would be the safer way.

Following up on this I created a stress test. What happens if we have multiple shots per second and darktable .xmp duplicates for those files?

I shot:

Code Select Expand

DSC04229.ARW
DSC04231.ARW
DSC04232.ARW
DSC04233.ARW
DSC04229.ARW was taken 2025:03:17 10:40:07.

DSC04231.ARW, DSC04232.ARW and DSC04233.ARW were all taken on the same second, 2025:03:17 10:45:16.

I opened the files in darktable and that created an .xmp file for every raw file.

For DSC04232.ARW I also created two duplicates in darktable which created DSC04232_01.ARW.xmp and DSC04232_02.ARW.xmp.

Here's tree output before running ExifTool:

Code Select Expand

tree
.
├── DSC04229.ARW
├── DSC04229.ARW.xmp
├── DSC04231.ARW
├── DSC04231.ARW.xmp
├── DSC04232.ARW
├── DSC04232.ARW.xmp
├── DSC04232_01.ARW.xmp
├── DSC04232_02.ARW.xmp
├── DSC04233.ARW
└── DSC04233.ARW.xmp

1 directory, 10 files

I now ran:

exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-Filename<${DateTimeOriginal}%+.2c.%e' -fileOrder filename .

This was the ExifTool output:

Code Select Expand

/mnt/data01/data01/bin/Image-ExifTool-13.25/exiftool -P -progress -ext arw -ext xmp -d ./%Y/%Y%m%d/%Y%m%d_%H%M%S '-Filename<${DateTimeOriginal}%+.2c.%e' -fileOrder filename .
======== ./DSC04229.ARW [1/10]
'./DSC04229.ARW' --> './2025/20250317/20250317_104007_00.ARW'
======== ./DSC04229.ARW.xmp [2/10]
'./DSC04229.ARW.xmp' --> './2025/20250317/20250317_104007_00.xmp'
======== ./DSC04231.ARW [3/10]
'./DSC04231.ARW' --> './2025/20250317/20250317_104516_00.ARW'
======== ./DSC04231.ARW.xmp [4/10]
'./DSC04231.ARW.xmp' --> './2025/20250317/20250317_104516_00.xmp'
======== ./DSC04232.ARW [5/10]
'./DSC04232.ARW' --> './2025/20250317/20250317_104516_01.ARW'
======== ./DSC04232.ARW.xmp [6/10]
Warning: History is not a structure! [x160] - ./DSC04232.ARW.xmp
'./DSC04232.ARW.xmp' --> './2025/20250317/20250317_104516_01.xmp'
======== ./DSC04232_01.ARW.xmp [7/10]
Warning: History is not a structure! [x160] - ./DSC04232_01.ARW.xmp
'./DSC04232_01.ARW.xmp' --> './2025/20250317/20250317_104516_02.xmp'
======== ./DSC04232_02.ARW.xmp [8/10]
Warning: History is not a structure! [x160] - ./DSC04232_02.ARW.xmp
'./DSC04232_02.ARW.xmp' --> './2025/20250317/20250317_104516_03.xmp'
======== ./DSC04233.ARW [9/10]
'./DSC04233.ARW' --> './2025/20250317/20250317_104516_02.ARW'
======== ./DSC04233.ARW.xmp [10/10]
'./DSC04233.ARW.xmp' --> './2025/20250317/20250317_104516_04.xmp'
    1 directories scanned
    1 directories created
   10 image files updated

tree output after running ExifTool:

Code Select Expand

tree
.
└── 2025
    └── 20250317
        ├── 20250317_104007_00.ARW
        ├── 20250317_104007_00.xmp
        ├── 20250317_104516_00.ARW
        ├── 20250317_104516_00.xmp
        ├── 20250317_104516_01.ARW
        ├── 20250317_104516_01.xmp
        ├── 20250317_104516_02.ARW
        ├── 20250317_104516_02.xmp
        ├── 20250317_104516_03.xmp
        └── 20250317_104516_04.xmp

3 directories, 10 files

The result should have been:

Code Select Expand

tree
.
└── 2025
    └── 20250317
        ├── 20250317_104007_00.ARW
        ├── 20250317_104007_00.ARW.xmp
        ├── 20250317_104516_00.ARW
        ├── 20250317_104516_00.ARW.xmp
        ├── 20250317_104516_01.ARW
        ├── 20250317_104516_01.ARW.xmp
        ├── 20250317_104516_01_01.ARW.xmp
        ├── 20250317_104516_01_02.ARW.xmp
        ├── 20250317_104516_02.ARW
        └── 20250317_104516_02.ARW.xmp

3 directories, 10 files

EDIT: corrected filename typos.

[mkl]

Here's a working solution using darktable with a Lua script.