There are 19 integer tags in QuickTime ItemList as of ExifTool 12.03. I will split them into 3 groups: broken (2), probably broken (13), and probably unbroken (4):
Broken
The data-type flag for the following tags in ItemList should be 0x15. Yet, as far as I can tell, ExifTool should still read and write them as unsigned integers. This is what iTunes writes; this is what iTunes reads. But this is also in contravention to Apple's own QuickTime File Format Specification (https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/Metadata/Metadata.html#//apple_ref/doc/uid/TP40000939-CH1-SW35) for data atoms under the item list hierarchy. At any rate, if I'm right about this, then ExifTool's int8u, int8s, int16u, etc., formats are unsuitable for these and the other ItemList integer tags. (They may still be good for other QuickTime tables, but I can't tell.)
tmpo BeatsPerMinute
cpil Compilation
Probably broken
The data-type flag for the following tags should probably be 0x15 as well. I'm not a hundred percent sure, though, because I have no Apple-written samples containing them. The only evidence I have is: this is what AtomicParsley writes; this is what it reads; and AtomicParsley strives to comply with Apple. Moreover, AtomicParsley is open source, and its source code seems to take these to be both 0x15 and unsigned.
rtng Rating
hdvd HDVideo
pcst Podcast
pgap PlayGap
stik MediaType
tves TVEpisode
tvsn TVSeason
cnID AppleStoreCatalogID
geID GenreID
akID AppleStoreAccountType
atID AlbumTitleID
plID PlayListID
sfID AppleStoreCountry
Probably unbroken
The data-type flag for the following tags is probably good. It's already 0x15. But now it is the format that may be broken. However, I'm not sure, and for the same reason. At any rate, all I can say is there is an outright type and length mismatch between ExifTool and AtomicParsley here. Type: none, signed, unsigned? Length: 8 or 16 bits? Please have a look at this table:
ID Name ExifTool AtomicParsley
©mvi MovementNumber x15 int16s uint8_t
©mvc MovementCount x15 int16s uint8_t
shwm ShowMovement x15 int8s binary (x0)
itnu iTunesU x15 int8s ?
Would be nice if a good soul could tell us just how newer versions of the iTunes media player, as well as the iTunes Store, read and write those, and the probably-broken ones too.
(By the way, iTunesU means university (https://en.wikipedia.org/wiki/ITunes#iTunes_U) lectures.)
I'll be happy to share code, output, and screenshots if you need more details.
Thank you.
Thanks for looking into this.
It is a bit troubling that the QuickTime flags don't agree with the datatype being read by AtomicParsley. I would like to keep ExifTool consistent with the specification (ie. read 0x15 as signed, and 0x16 as unsigned). Is it likely that any of the signed values that AtomicParsley writes as unsigned will use the upper bit?
This needs more consideration. I will spend some time re-reading the specifications and looking at more sample files, but it may be a while before I have time for this.
- Phil
Quote from: Phil Harvey on August 08, 2020, 08:02:04 AM
This needs more consideration. I will spend some time re-reading the specifications and looking at more sample files, but it may be a while before I have time for this.
No problem. Thanks. :) Do you believe you'll be able to include the other bug fixes we talked about since 12.03 was released, though?
Yes. I've made the other changes already.
- Phil
Quote from: Phil Harvey on August 08, 2020, 09:28:52 AM
Yes. I've made the other changes already.
Thanks a lot. :)
Quote from: Phil Harvey on August 08, 2020, 08:02:04 AM
Is it likely that any of the signed values that AtomicParsley writes as unsigned will use the upper bit?
I don't know. AtomicParsley (0.9.6) is happy to write -1 but reads it as unsigned:
$ declare n=-1
$ AtomicParsley file.m4a --tracknum=$n --disk=$n --bpm=$n --TVSeasonNum=$n --TVEpisodeNum=$n --cnID=$n --geID=$n --overWrite
Updating metadata...
$ AtomicParsley file.m4a -t
Atom "trkn" contains: 65535
Atom "disk" contains: 65535
Atom "tmpo" contains: 65535
Atom "tvsn" contains: 65535
Atom "tves" contains: 65535
Atom "cnID" contains: 4294967295
Atom "geID" contains: 4294967295
Apple iTunes (screenshot (https://i.imgur.com/MjF5iue.png)) also reads a presumably negative
tmpo as unsigned.
Quote from: Phil Harvey on August 08, 2020, 08:02:04 AM
It is a bit troubling that the QuickTime flags don't agree with the datatype being read. I would like to keep ExifTool consistent with the specification (ie. read 0x15 as signed, and 0x16 as unsigned).
Well, ExifTool 12.03 reports, say,
tmpo as both 0x15 (signed) and int16u (unsigned) on its
-v3 analysis of AtomicParsley's output:
| | | + [ItemList directory]
| | | | TrackNumber = ..
| | | | - Tag 'trkn', Type='data', Flags=0x0 (8 bytes, undef):
| | | | 133e1: 00 00 ff ff 00 00 00 00 [........]
| | | | DiskNumber = ..
| | | | - Tag 'disk', Type='data', Flags=0x0 (6 bytes, undef):
| | | | 13401: 00 00 ff ff 00 00 [......]
| | | | BeatsPerMinute = 65535
| | | | - Tag 'tmpo', Type='data', Flags=0x15 (2 bytes, int16u):
| | | | 1341f: ff ff [..]
| | | | TVSeason = 65535
| | | | - Tag 'tvsn', Type='data', Flags=0x15 (4 bytes, int32u):
| | | | 13439: 00 00 ff ff [....]
| | | | TVEpisode = 65535
| | | | - Tag 'tves', Type='data', Flags=0x15 (4 bytes, int32u):
| | | | 13455: 00 00 ff ff [....]
| | | | AppleStoreCatalogID = 4294967295
| | | | - Tag 'cnID', Type='data', Flags=0x15 (4 bytes, int32u):
| | | | 13471: ff ff ff ff [....]
| | | | GenreID = 4294967295
| | | | - Tag 'geID', Type='data', Flags=0x15 (4 bytes, int32u):
| | | | 1348d: ff ff ff ff
ExifTool seems to ignore the 0x15 flag, and reads those presumably signed integers as unsigned, just like iTunes and AtomicParsley do. So, in a sense, it's already inconsistent with the specification.
$ exiftool -ItemList:all file.m4a
Track Number : 65535
Disk Number : 65535
Beats Per Minute : 65535
TV Season : 65535
TV Episode : 65535
Apple Store Catalog ID : 4294967295
Genre ID : Unknown (4294967295)
Quote from: Phil Harvey on August 08, 2020, 08:02:04 AM
I would like to keep ExifTool consistent with the specification (ie. read 0x15 as signed, and 0x16 as unsigned).
Except there is no specification for particular item-list tags in MP4/M4A files. As you know, MPEG-4 ≠ QuickTime.
In the introduction to the QuickTime specification, Apple says:
QuoteImportant: The QuickTime File Format has been used as the basis of the MPEG-4 standard and the JPEG-2000 standard, developed by the International Organization for Standardization (ISO). Although these file types have similar structures and contain many functionally identical elements, they are distinct file types.
Warning: Do not use this specification to interpret a file that conforms to a different specification, however similar. (here (https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFPreface/qtffPreface.html#//apple_ref/doc/uid/TP40000939-CH202-TPXREF101)).
So, my guess is Apple was the first to violate its own QuickTime specification. But then again, MPEG-4 is not QuickTime.
As you said somewhere, this is a mess.
Quote from: dae65 on August 08, 2020, 11:51:45 AM
ExifTool seems to ignore the 0x15 flag, and reads those presumably signed integers as unsigned, just like iTunes and AtomicParsley do. So, in a sense, it's already inconsistent with the specification.
You're right. I didn't remember doing this, but the Format in ExifTool's code overrides the flags in the Quicktime data definition. I guess we now know why. But this means that I can't simply change it to 'int16s' for BeatsPerMinute because ExifTool uses the same Format for writing and reading. I may need to split this into two separate format variables, as was necessary for EXIF as well.
- Phil
Quote from: Phil Harvey on August 08, 2020, 03:06:16 PMBut this means that I can't simply change it to 'int16s' for BeatsPerMinute because ExifTool uses the same Format for writing and reading.
Don't worry. For the time being I'm going to work with user-defined int16s Compilation and BeatsPerMinute tags for the sake of the 0x15 flag, and write no negative values to them. The other tags are seldom written to.
Looking forward to downloading 12.04! Hope you won't put it off just because of this particular issue.
Thank you. :)
I haven't yet searched through the (painful) ISO14496 specification (I don't have it on this computer), but I have gone through my samples and verified that the flags are 0x15 for all of the tags I found from your list. So I will go ahead and make this change for the upcoming release (hopefully within a couple of hours). I will write flags 0x15 and read as unsigned for the tags you listed. I didn't have any samples handy which contained your "probably broken" tags, so won't yet change the size of these.
- Phil
Thanks, Phil. :) Just to share more findings with you, I've just concluded that not only with iTunes and AtomicParsley, but this also happens with Mp3tag (https://www.mp3tag.de/en/) (3.01 for Windows), and puddletag (http://docs.puddletag.net/) (1.2 for Linux, using Mutagen 1.36, a Python library that reads metadata).
They too write 0x15 but read as unsigned. Here is a screenshot (https://i.imgur.com/AM46E2P.png) from Mp3tag, and here is another screenshot (https://i.imgur.com/D4C8OkF.png) from puddletag.
And a hex dump from Mp3tag's output:
00012c2f 00 00 00 1a 74 6d 70 6f 00 00 00 12 64 61 74 61 |....tmpo....data|
00012c3f 00 00 00 15 00 00 00 00 00 01 |..........|
00012c49
00012c49 00 00 00 19 63 70 69 6c 00 00 00 11 64 61 74 61 |....cpil....data|
00012c59 00 00 00 15 00 00 00 00 01 |.........|
00012c62
00012c62 00 00 00 19 70 63 73 74 00 00 00 11 64 61 74 61 |....pcst....data|
00012c72 00 00 00 15 00 00 00 00 01 |.........|
00012c7b
00012d4c 00 00 00 1c 74 76 73 6e 00 00 00 14 64 61 74 61 |....tvsn....data|
00012d5c 00 00 00 15 00 00 00 00 00 00 00 01 |............|
00012d68
00012d68 00 00 00 1c 74 76 65 73 00 00 00 14 64 61 74 61 |....tves....data|
00012d78 00 00 00 15 00 00 00 00 00 00 00 01 |............|
00012d84
Puddletag (Mutagen) complains if I try to write a negative value to tmpo:
File "/usr/lib/python2.7/dist-packages/mutagen/mp4/__init__.py", line 684, in __render_tempo
"invalid 16 bit integers: %r" % value)
mutagen.mp4.MP4MetadataValueError: invalid 16 bit integers: [-1]
but is happy to write the 0x15 flag anyway:
00012c35 00 00 00 1a 74 6d 70 6f 00 00 00 12 64 61 74 61 |....tmpo....data|
00012c45 00 00 00 15 00 00 00 00 00 01 |..........|
00012c4f
It seems a bug has become a de facto standard. :D
Edit: Mp3tag can write some of the "probably broken" and "probably unbroken" tags, but it's obviously not iTunes. Let me know if you're willing to trust it.
Thanks for this.
ExifTool 12.04 is now available.
I would be interested to see how Mp3tag writes your "probably unbroken" tags. Sorry, it was these, not the "probably broken" tags for which I don't have samples.
- Phil
Quote from: Phil Harvey on August 10, 2020, 10:23:41 AM
ExifTool 12.04 is now available.
Great! :) Thanks for having fixed those issues. ExifTool 12.04 is now the best MOV / MP4 / M4A tag editor there is.
Quote from: Phil Harvey on August 10, 2020, 10:23:41 AM
I would be interested to see how Mp3tag writes your "probably unbroken" tags.
Sure. Mp3tag 3.01 reads and writes
©mvi,
©mvc, and
shwm as ExifTool 12.04 does, in terms of both flags and formats. See below. AtomicParsley is alone in writing them as int8 and binary. Also, Mp3tag and AtomicParsley don't support
itnu.
ID Name Flag ExifTool Mp3tag AtomicParsley
©mvi MovementNumber x15 int16 int16 int8
©mvc MovementCount x15 int16 int16 int8
shwm ShowMovement x15 int8 int8 binary (x0)
itnu iTunesU x15 int8 - -Here is a hex dump of those tags as written by Mp3tag:
00012bf1 00 00 00 1a a9 6d 76 69 00 00 00 12 64 61 74 61 |.....mvi....data|
00012c01 00 00 00 15 00 00 00 00 00 01 |..........|
00012c0b
00012c0b 00 00 00 1a a9 6d 76 63 00 00 00 12 64 61 74 61 |.....mvc....data|
00012c1b 00 00 00 15 00 00 00 00 00 01 |..........|
00012c25
00012c25 00 00 00 19 73 68 77 6d 00 00 00 11 64 61 74 61 |....shwm....data|
00012c35 00 00 00 15 00 00 00 00 01 |.........|
00012c3eRegardless of the 0x15 flag, Mp3tag follows suit and reads presumably negative integers as unsigned (screenshot (https://i.imgur.com/7TknR3A.png)), provided they're written by other software. Unexpectedly, though, Mp3tag is itself able to write and then read negative values (screenshot (https://i.imgur.com/R9OjsiK.png)) by making use of the following trick: it stores positive integers (https://i.imgur.com/qWD6g64.png) as everybody else does, but negative integers it actually writes under the
ilst.---- hierarchy (aka iTunesInfo (https://exiftool.org/TagNames/QuickTime.html#iTunesInfo) subdirectory) instead of directly under the item list:
$ exiftool -G5 file.m4a | grep iTunesInfo
[MOV-Movie-UserData-Meta-ItemList-iTunesInfo] SHOWMOVEMENT: -1
[MOV-Movie-UserData-Meta-ItemList-iTunesInfo] MOVEMENTTOTAL: -1
[MOV-Movie-UserData-Meta-ItemList-iTunesInfo] MOVEMENT: -1
Great fun. :D Please let me know if you ever make sense of what ISO 14496 part 14 has to say about 0x15!
Great. I'm glad we've arrived at an acceptable solution. I will take a look at the ISO specification when I get a chance to see if I can find a justification for this behaviour.
- Phil
Thanks. Just so that we know, MP4v2 (https://code.google.com/archive/p/mp4v2/), a C library, behaves in the same way. It writes negative integers:
$ declare n=-1
$ mp4tags -tempo=$n -hdvideo=$n -contentid=$n -genreid=$n -episode=$n -season=$n -playlistid=$n -podcast=$n -artistid=$n -composerid=$n file.m4a
Reads them as unsigned:
$ mp4info file.m4a
mp4info version -r
file.m4a:
Track Type Info
1 audio MPEG-4 AAC LC, 224.040 secs, 156 kbps, 44100 Hz
BPM: 65535
HD Video: yes
TV Episode: 4294967295
TV Season: 4294967295
Podcast: yes
Content ID: 4294967295
Artist ID: 4294967295
Playlist ID: 18446744073709551615
Genre ID: 4294967295
Composer ID: 4294967295
But writes the 0x15 flag:
0042759f 00 00 00 1a 74 6d 70 6f 00 00 00 12 64 61 74 61 |....tmpo....data|
004275af 00 00 00 15 00 00 00 00 ff ff |..........|
004275b9
004275b9 00 00 00 1c 74 76 73 6e 00 00 00 14 64 61 74 61 |....tvsn....data|
004275c9 00 00 00 15 00 00 00 00 ff ff ff ff |............|
004275d5
004275d5 00 00 00 1c 74 76 65 73 00 00 00 14 64 61 74 61 |....tves....data|
004275e5 00 00 00 15 00 00 00 00 ff ff ff ff |............|
004275f1
004275f1 00 00 00 19 70 63 73 74 00 00 00 11 64 61 74 61 |....pcst....data|
00427601 00 00 00 15 00 00 00 00 ff |.........|
0042760a
0042760a 00 00 00 19 68 64 76 64 00 00 00 11 64 61 74 61 |....hdvd....data|
0042761a 00 00 00 15 00 00 00 00 ff |.........|
00427623
00427623 00 00 00 1c 63 6e 49 44 00 00 00 14 64 61 74 61 |....cnID....data|
00427633 00 00 00 15 00 00 00 00 ff ff ff ff |............|
0042763f
0042763f 00 00 00 1c 61 74 49 44 00 00 00 14 64 61 74 61 |....atID....data|
0042764f 00 00 00 15 00 00 00 00 ff ff ff ff |............|
0042765b
0042765b 00 00 00 20 70 6c 49 44 00 00 00 18 64 61 74 61 |... plID....data|
0042766b 00 00 00 15 00 00 00 00 ff ff ff ff ff ff ff ff |................|
0042767b
0042767b 00 00 00 1c 67 65 49 44 00 00 00 14 64 61 74 61 |....geID....data|
0042768b 00 00 00 15 00 00 00 00 ff ff ff ff |............|
00427697
00427697 00 00 00 1c 63 6d 49 44 00 00 00 14 64 61 74 61 |....cmID....data|
004276a7 00 00 00 15 00 00 00 00 ff ff ff ff |............|
004276b3
Best,
d
Thanks. Seems pretty universal.