CSV export: Added tags, missing tags simplified and not in order with -f

[Andi]

The documentation says:

When exporting specific tags, the CSV columns are arranged in the same order as the specified tags provided the column headings exactly match the specified tag names, otherwise the columns are sorted in alphabetical order.

I would expect that, at least along with

Code Select Expand

-api MissingTagValue=- -f, exactly defined tags that are missing should not be simplified and reordered, but this happens:

Code Select Expand

exiftool-5.22 -XMP:Creator -XMP:CreatorCountry -IPTC:DateCreated -IPTC:TimeCreated -XMP:Title -XMP:Description -EXIF:ExposureTime -csv -api MissingTagValue=- -f -G -sep ', ' FILES

First three lines of the result:

Code Select Expand

ExposureTime -csv -api MissingTagValue=- -f -G -sep ', ' 01/1/
SourceFile,Creator,CreatorCountry,Description,EXIF:ExposureTime,ExposureTime,IPTC:DateCreated,IPTC:TimeCreated,Title,XMP:Creator,XMP:Title
01/1/_01-1-03.tif,-,-,-,0.4,-,1959:01:01,00:34:36,-,Andi,Title
01/1/01-1-03.tif,-,-,-,0.4,-,1959:01:01,00:34:36,-,-,Title

Without -f, missing columns for all images are omit, as expected.

I copied 01-1-03.tif to _01-1-03.tif and have set

Code Select Expand

XMP:Creator on the copy to get this result. You see that the order of missing tags is wrong; this seems to be a fallback solution that should not happen because I have explicitly specified the directory in the command:

Code Select Expand

-XMP:Creator, and not

Code Select Expand

-Creator. How can I force exiftool to not fall back if

Code Select Expand

-XMP:Creator is missing? This would imply as well that all tags are in correct order.
With -f, an additional ExposureTime column is added. I want to prohibit this, too.

With -f the columns included in the export should not depend on the image data!

[Phil Harvey]

Could you perhaps try this with a version of ExifTool that is less than 15 years old?  You seem to be using version 5.22.  The current version is 11.99.

- Phil

[StarGeek]

I'm not sure what the 5.22 is but I doubt it's the version since it looks like the CSV option was added in version 8.51.  I had the same thought and had to double check.

[Phil Harvey]

Right.  OK.   Yes, I can reproduce this with version 11.99.

Things get very tricky when using -f to output non-existing tags.  In general, ExifTool can't know what group they are in, so it leaves the group name out.  (eg. It doesn't know the tag groups unless it can read the tag).

But the order change is a bit unexpected.  I'll have to dig to figure that one out.

- Phil

[Andi]

Strangely, exiftool-5.22 -ver shows me "10.28"

I have updated it now to v11.99, and I can confirm that the behavior is the same.

I would accept if exiftool was slower with a complete scan of the directories. Somehow it must be possible to know if an explicit tag exists or not, else the semantics of -f and -G seems incorrect to me.

I want to be sure that my CSV table contains the tags as given in the command so that I can import the data into a spreadsheet editor and the columns are already in the order I wish to edit them, including the missing tags.

Thanks for investigating, Phil.

[Andi]

The JSON output with

Code Select Expand

exiftool -XMP:Creator -XMP:CreatorCountry -IPTC:DateCreated -IPTC:TimeCreated -XMP:Title -XMP:Description -EXIF:ExposureTime -j -api MissingTagValue=- -G -sep ', ' -f *03.tif

is

Code Select Expand

[{
  "SourceFile": "01-1-03.tif",
  "Unknown:Creator": "-",
  "Unknown:CreatorCountry": "-",
  "IPTC:DateCreated": "1959:01:01",
  "IPTC:TimeCreated": "00:34:36",
  "XMP:Title": "Title",
  "Unknown:Description": "-",
  "EXIF:ExposureTime": 0.4
},
{
  "SourceFile": "_01-1-03.tif",
  "XMP:Creator": "Andi",
  "Unknown:CreatorCountry": "-",
  "IPTC:DateCreated": "1959:01:01",
  "IPTC:TimeCreated": "00:34:36",
  "XMP:Title": "Title",
  "Unknown:Description": "-",
  "EXIF:ExposureTime": 0.4
}]

The dirs of missing tags are wrong (why not just using the expected names?), but no simplified tags were added and the order is correct.

Again the ouput with -csv:

Code Select Expand

SourceFile,Creator,CreatorCountry,Description,EXIF:ExposureTime,IPTC:DateCreated,IPTC:TimeCreated,XMP:Creator,XMP:Title
01-1-03.tif,-,-,-,0.4,1959:01:01,00:34:36,-,Title
_01-1-03.tif,-,-,-,0.4,1959:01:01,00:34:36,Andi,Title

[Phil Harvey]

The dirs of missing tags are wrong (why not just using the expected names?),

They are not wrong.  The family 0 group is unknown if the tag isn't extracted.  You specified a group when requesting the tag, but ExifTool doesn't know which family it belongs to.

If you want all the headings filled in properly, I suggest you generate a file containing all of the tags and add that file name to your command.

- Phil

[Andi]

If -csv generated a result that fits to the JSON result (with Unknown:... or KnownDir:...) as column names, but without added columns and with correct order, I would be satisfied.

[Phil Harvey]

OK.  ExifTool 12.00 will add the "Unknown" to the CSV headings with unknown tags when -G is used.

- Phil

[Andi]

This is my current solution using https://stedolan.github.io/jq/ including -f:

Code Select Expand


# From https://stackoverflow.com/a/17841619/6307827
# $1: joiner
# $... args
# Example: join_by "," "${array[@]}"
joinby() { local d=$1; shift; print "$1"; shift; printf "%s" "${@/#/$d}"; }

# Export all tags as requested (names and order) to CSV with a bug fix and another fix: Always use the requested group name instead of "Unknown" if the tag was not found.
# The resulting column names and their order are determined and do not depend on the existing tags in the given images.
#
# $@ Args are the same as exiftool with the following exceptions:
# - All tags must be given with group name, i. e. contain ":", and must not contain CSV format characters , " \n
# - Any of the arguments -csv -f -g -struct are disallowed because internally, "-f -json" is used and then transformed to CSV
# - G[NUM] has no effect because tag names are always printed exactly as given in the arguments. It is recommended to use the number fitting to the requested tags
#
# Example:
# exiftool_exportcsv -XMP:HierarchicalSubject -XMP:Creator -XMP:CreatorCountry -IPTC:DateCreated -IPTC:TimeCreated -XMP:Title -XMP:Description -EXIF:ExposureTime -api MissingTagValue=- -sep ', ' -G *.tif
exiftool_exportcsv() {

# Filter out tags
declare -a tags=("$@")
declare -i i=0
for tag in "${tags[@]}"; do
if [[ "$tag" != "-"* ]] || [[ "$tag" != *":"* ]]; then
unset tags[$i]
else
tags[$i]="${tags[i]:1}"
fi
(( i++ ))
done
tags=('SourceFile' "${tags[@]}")

joinby ',' "${tags[@]}"; echo ""
exiftool -f -j "$@" | jq -r '.[] | map(.) | @csv'
}