Find Portraits (or photos with face close to lens) by size of FaceTag box

[josjonkeren]

Hi,

I have used exiftool for a while now, and used it to organize my photos collection in years/months/days, I have tagged with geolocations etc., all very nice.
Thank you Phil.

I have tagged all faces (well, as far as you can get -- this takes a LOT of time, even after running multiple face-detection programs)... and have tagged them with Windows Live Photo Gallery (WLPG). Yes, I know, it's very old, but it's the only software that lets me work with my photos easily, and it works fast. I also have tried Win10 Photos app, Digikam, Acdsee and others, but that is all inferior software compared to what the very old WLPG can do.
All face tags are in XMP, in the image file, in the Microsoft People Tags Schema (MPReg:PersonDisplayName etc).

Question:
Is there a way that I can find photos, using exiftool, that have a settable "minimum size face tag box"?
So -- scan for "large face tag boxes"?
As you know, after having tagged people's faces, the people in the back of the pictures have smaller face "rectangles" than the people closer to the camera.
So what I would like to do is: find all pictures of person name "QQ" , with a face-tag rectangle box "larger than XX by YY pixels", or "larger than ZZ% of the picture frame".
That way, I can easily find closeups of people, or portrait photos, rather than seeing ALL pictures, even those where this person is present, but very far in the background.

I think you understand what I mean.

I can imagine that this would take lots of time to run, especially when the list should be sorted somehow -- but maybe some form of: find all pictures of person Q, with face tag rectangles larger than ZZ, and export the found face-tag rectangles to a CSV file.
If that would work, then I could use the CSV data, and sort by face-tag size, and find the corresponding images that way.

I have searched this forum, but have not found something like this.
Thank you!
Jos Jonkeren

[Phil Harvey]

This should certainly be possible. Can you provide a sample image so I can see how you have tagged the faces?  My email is philharvey66 at gmail.com

- Phil

[StarGeek]

I was thinking on this after I read this on my phone (now back on my computer).  I'm assuming the region tags are XMP-MP, since I believe that's what Windows Live Photo Gallery wrote.  Since the region tags data is basically saved as a percent of the total image (to allow for resizing), it shouldn't be too hard.  It would be more complicated than could be done directly on the command line, I would think.  You would also have to take into account the possibility of multiple regions.

Specifying a specific region name and only listing that would be harder.  The easiest would be a simple dump of names and the percentage of the picture.  But that wouldn't necessarily lend itself to an simple csv file, as the names and percents would be in a single column.  The resulting file would be something like:

Sourcefile	FacePercent
File.jpg	John Smith: 25%, Jane Doe: 20%
File2.jpg	Lex Luthor: 30%, Clark Kent: 20%, Lois Lane: 5%

Just my thoughts while I was in a waiting room.

[StarGeek]

Digging through code I haven't touched in four years and looking for an image which had at least two MP regions (I've ditched MP regions due to its weakness to change), here's some more notes, assuming that it is a XMP-MP region.

Example Structure
RegionInfoMP                    : {Regions=[{PersonDisplayName=John Smith,Rectangle=0.666407263|, 0|, 0.065629053|, 0.086030365},{PersonDisplayName=Jane Doe,Rectangle=0.564843214|, 0.628427558|, 0.080476081|, 0.154619669}]}

Flattened tags
RegionPersonDisplayName         : John Smith##Jane Doe
RegionRectangle                 : 0.666407263, 0, 0.065629053, 0.086030365##0.564843214, 0.628427558, 0.080476081, 0.154619669
Edit: Added -sep ## to clarify listing

Each Rectangle consists of X, Y, W, H, where X and Y are the upper left corner of the rectangle (unlike MWG rectagle, which is centered, IIRC) and W and H are the width and height of the rectangle.  All values are decimal.

So, someone correct me if I'm wrong, the basic formula to get a area percent would be
(ImageWidth*RegionRectangleW*ImageHeight*RegionRectangleH)/(ImageWidth*ImageHeight)*100

Still thinking on this, might be able to come up with something.

Edit: RegionRectangle isn't eight separate values, it's two strings of four digits.

[StarGeek]

Here's a basic config based upon what I said in the previous post.

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Composite' => {
        RegionPercentage => {
Require => {
                0 => 'RegionPersonDisplayName',
                1 => 'RegionRectangle',
2 => 'ImageWidth',
3 => 'ImageHeight',
},
ValueConv => q{
my @Names = ref $val[0] eq 'ARRAY' ? @{$val[0]} : ( $val[0] );
my @Regions = @{$val[1]};
my @PercentList;
my $ImageSize = $val[2]*$val[3];
# Loop through regions
foreach my $i (0 .. $#Names) {
# Split the Region string into separate parts, x & w are not used
my ($x,$y,$w,$h) = split /, /, $Regions[$i];
my $Percent = sprintf("%.2f",   ($ImageSize*$w*$h)/$ImageSize*100);
push @PercentList, $Names[$i].': '.$Percent;
}
return @PercentList ? \@PercentList :undef;
},
},
},
);
#------------------------------------------------------------------------------
1;  #end

It will list each region and the percentage of the picture it covers, assuming my math in the previous post was correct.  Using the above data, with the original image dimensions of 1280x802, the output was
Region Percentage               : John Smith: 0.56, Jane Doe: 1.24

Yes, my test image had some very small regions.

[josjonkeren]

Hello Phil,
I have sent Phil two jpg files, one with one face tagged; fairly large "box" and one with two faces tagged. 

But it seems that might be not neccessary, as @Stargeek also already answered, even with an example config script! Wow thanks.
I am still to try out what you posted; will do that later today

The data is indeed in XMP-MP, I found these two tags:

Code Select Expand

[XMP-MP] RegionRectangle        : 0.212500, 0.301042, 0.161719, 0.216667, 0.573712, 0.311611, 0.163410, 0.218009
[XMP-MP] RegionPersonDisplayName: Name1, Name2

@Stargeek, I have looked at your code, and I do not understand the part where it splits -- it seems to me you are splitting on comma, but above you wrote to split on ##.
Am I wrong?

Code Select Expand

# Split the Region string into separate parts, x & w are not used
my ($x,$y,$w,$h) = split /, /, $Regions[$i];

If there are more people in a pic, the only issue is, I guess, how to get all the same person's tags in one "column", to be sorted.
Because how I read the config script is that it dumps all peoples tags, in order of appearance in the file. Am I correct in this?

By the way, wouldn't this be a good "feature" for photo editing / archiving software, to be able to "see only closeup photos" or a similar named option? Or, sort by "face size" in the pictures. That would be really easy if all the image data is stored in a database anyway; the software could just run one query and show results. 
Maybe I am the only person who thought of this :-).

Thanks again.

[Phil Harvey]

@StarGeek:  You were right about the MP regions, although one of the pictures did have MWG regions as well:

Code Select Expand

> exiftool tmp -'*region*' -G1
======== tmp/20071020-173620-Nederland-Hellouw-Hellouw.JPG
[XMP-mwg-rs]    Region Name                     : Ulrike Nagel
[XMP-mwg-rs]    Region Type                     : Face
[XMP-mwg-rs]    Region Area X                   : 0.391847
[XMP-mwg-rs]    Region Area Y                   : 0.558063
[XMP-mwg-rs]    Region Area W                   : 0.36394
[XMP-mwg-rs]    Region Area H                   : 0.54591
[XMP-mwg-rs]    Region Area Unit                : normalized
[XMP-MP]        Region Rectangle                : 0.209877, 0.285108, 0.36394, 0.54591
[XMP-MP]        Region Person Display Name      : Ulrike Nagel
======== tmp/20071227-124219-Duitsland-Dresden-Dresden.JPG
[XMP-MP]        Region Rectangle                : 0.212500, 0.301042, 0.161719, 0.216667, 0.573712, 0.311611, 0.163410, 0.218009
[XMP-MP]        Region Person Display Name      : Ulrike Nagel, Susanne Nagel
    2 image files read
> exiftool tmp -'*region*' -G1 -struct
======== tmp/20071020-173620-Nederland-Hellouw-Hellouw.JPG
[XMP-mwg-rs]    Region Info                     : {RegionList=[{Area={H=0.54591,Unit=normalized,W=0.36394,X=0.391847,Y=0.558063},Name=Ulrike Nagel,Type=Face}]}
[XMP-MP]        Region Info MP                  : {Regions=[{PersonDisplayName=Ulrike Nagel,Rectangle=0.209877|, 0.285108|, 0.36394|, 0.54591}]}
======== tmp/20071227-124219-Duitsland-Dresden-Dresden.JPG
[XMP-MP]        Region Info MP                  : {Regions=[{PersonDisplayName=Ulrike Nagel,Rectangle=0.212500|, 0.301042|, 0.161719|, 0.216667},{PersonDisplayName=Susanne Nagel,Rectangle=0.573712|, 0.311611|, 0.163410|, 0.218009}]}

I tested out your config file, and it works with a slight modification for the case where there was only one region:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Composite' => {
        RegionPercentage => {
            Require => {
                0 => 'RegionPersonDisplayName',
                1 => 'RegionRectangle',
                2 => 'ImageWidth',
                3 => 'ImageHeight',
            },
            ValueConv => q{
                my @Names   = ref $val[0] eq 'ARRAY' ? @{$val[0]} : ( $val[0] );
                my @Regions = ref $val[1] eq 'ARRAY' ? @{$val[1]} : ( $val[1] );   
                my @PercentList;
                my $ImageSize = $val[2]*$val[3];
                # Loop through regions
                foreach my $i (0 .. $#Names) {
                    # Split the Region string into separate parts, x & w are not used
                    my ($x,$y,$w,$h) = split /, /, $Regions[$i];
                    my $Percent = sprintf("%.2f",   ($ImageSize*$w*$h)/$ImageSize*100);
                    push @PercentList, $Names[$i].': '.$Percent;
                }
                return @PercentList ? \@PercentList :undef;
            },
        },
    },
);
#------------------------------------------------------------------------------
1;  #end

- Phil

[Phil Harvey]

With another modification to the config file, I can generate a sorted CSV file based on person name with a command like this:

exiftool -config my.config -regionpercentage -b -sep "\n" -sep "\n" DIR | sort > out.csv

This should work unless a person or filename contains a comma, and produces an output file like this:

Code Select Expand

Susanne Nagel,03.56,tmp/20071227-124219-Duitsland-Dresden-Dresden.JPG
Ulrike Nagel,03.50,tmp/20071227-124219-Duitsland-Dresden-Dresden.JPG
Ulrike Nagel,19.87,tmp/20071020-173620-Nederland-Hellouw-Hellouw.JPG

The last entry for each person name will be the largest image of that person (in percent of picture size).

Here is the config file to do this:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Composite' => {
        RegionPercentage => {
            Require => {
                0 => 'RegionPersonDisplayName',
                1 => 'RegionRectangle',
                2 => 'ImageWidth',
                3 => 'ImageHeight',
            },
            RawConv => q{
                my @Names   = ref $val[0] eq 'ARRAY' ? @{$val[0]} : ( $val[0] );
                my @Regions = ref $val[1] eq 'ARRAY' ? @{$val[1]} : ( $val[1] );
                my @PercentList;
                my $ImageSize = $val[2]*$val[3];
                # Loop through regions
                foreach my $i (0 .. $#Names) {
                    # Split the Region string into separate parts, x & w are not used
                    my ($x,$y,$w,$h) = split /, /, $Regions[$i];
                    my $Percent = sprintf("%05.2f",   ($ImageSize*$w*$h)/$ImageSize*100);
                    push @PercentList, "$Names[$i],$Percent,$$self{FILENAME}";
                }
                return @PercentList ? \@PercentList :undef;
            },
        },
    },
);
#------------------------------------------------------------------------------
1;  #end

- Phil

[olball]

With another modification to the config file, ..

This config file can further simplified. There is no need for calculation of the image size from image width and height. The width and height in the region are already normalized to image dimension. So the image size can be reduced from the fraction.

So the line
my $Percent = sprintf("%05.2f", ($ImageSize*$w*$h)/$ImageSize*100);
can be shortened to
my $Percent = sprintf("%05.2f", $w*$h*100);

and all references to the image dimension can be removed from the tag.

[Phil Harvey]

Right.  Very good:

Code Select Expand

%Image::ExifTool::UserDefined = (
    'Image::ExifTool::Composite' => {
        RegionPercentage => {
            Require => {
                0 => 'RegionPersonDisplayName',
                1 => 'RegionRectangle',
                2 => 'Directory',
                3 => 'FileName',
            },
            RawConv => q{
                my @Names   = ref $val[0] eq 'ARRAY' ? @{$val[0]} : ( $val[0] );
                my @Regions = ref $val[1] eq 'ARRAY' ? @{$val[1]} : ( $val[1] );
                my @PercentList;
                # Loop through regions
                foreach my $i (0 .. $#Names) {
                    # Split the Region string into separate parts, x & w are not used
                    my ($x,$y,$w,$h) = split /, /, $Regions[$i];
                    my $Percent = sprintf("%05.2f", $w*$h*100);
                    push @PercentList, "$Names[$i],$Percent,$val[2]/$val[3]";
                }
                return @PercentList ? \@PercentList :undef;
            },
        },
    },
);
1;  #end

Edit: Also, I have cleaned it up a bit to avoid using an internal ExifTool member variable.

[StarGeek]

But it seems that might be not neccessary, as @Stargeek also already answered, even with an example config script! Wow thanks.

The puzzle got into my head and I wanted to try my hand at it.  I knew that it would take me much longer than Phil to work it out so I rushed it a bit.

Code Select Expand

[XMP-MP] RegionRectangle        : 0.212500, 0.301042, 0.161719, 0.216667, 0.573712, 0.311611, 0.163410, 0.218009
[XMP-MP] RegionPersonDisplayName: Name1, Name2

@Stargeek, I have looked at your code, and I do not understand the part where it splits -- it seems to me you are splitting on comma, but above you wrote to split on ##.
Am I wrong?

Code Select Expand

# Split the Region string into separate parts, x & w are not used
my ($x,$y,$w,$h) = split /, /, $Regions[$i];

It's complicated and threw me for a bit until I took a closer look at my old code for previous region configs.  The output which included the ## was an edit where I added -sep ## to the command.  That was to clarify that while RegionRectangle is a list type tag, it's not a list of 8 separate entries, it's a list of two separate entries.  That isn't obvious at first without using the -sep options because ",(space)" is the default separator and it's also used as a delimiter in the RegionRectangle tag.

Because how I read the config script is that it dumps all peoples tags, in order of appearance in the file. Am I correct in this?

Yes, this is the problem I was trying to think through in my previous posts and couldn't think of a way around it, but it looks like Phil came up with a solution there, though I haven't read his post in detail yet.  I need to so I can learn how to do things better.

By the way, wouldn't this be a good "feature" for photo editing / archiving software, to be able to "see only closeup photos" or a similar named option? Or, sort by "face size" in the pictures. That would be really easy if all the image data is stored in a database anyway; the software could just run one query and show results.
Maybe I am the only person who thought of this :-).

I don't believe many people bother using face regions that much.  So most programmers wouldn't bother trying to add something like that.  But now that idea is getting in my head and I think that a very basic ability to do so could be done by copying the results of this config to (my favorite) the HierarchicalSubject tag.  Round the percent to the nearest 5/10% to organize a bit better.  The end result would be something like

Code Select Expand

Face Regions
├───John Smith
│   ├───90%
│   ├───80%
│   ├───70%
│   └───60%
├───Jane Doe
│   ├───90%
│   ├───80%
│   ├───70%
│   └───60%
├───Joe Schmo
│   ├───50%

This could be done inline using advanced formatting based upon the config file.  But it would require use of a DAM that displayed HierarchicalSubject in a useful way.

This config file can further simplified.

And if I had actually written out the formula on paper properly, it would have been obvious that it could have been simplified.  But these days I hardly ever touch a pencil.  I am amused as my own fallibility in this case.

[josjonkeren]

Thank you Phil and Stargeek.
Nice to see how "eager" you are to get a 'problem' solved, haha!
I never expected an answer at all, and I am pleasantly surprized you both answered so quickly and in-depth.

I ran the script Phil wrote, by using

Code Select Expand

exiftool.exe -config personen.config -regionpercentage -b -sep "\n" -sep "\n" -r .
(So without the pipe through sort, and to a file, just to see what the output would be (without waiting a loooong time while it went through 55.400 images...))

Turns out, this looks fine!
Only thing is, some warnings get printed.

Code Select Expand

Warning: [minor] Unrecognized MakerNotes - ./Camera-album/IMG_20191009_140241.jpg
Warning: [minor] Unrecognized MakerNotes - ./Camera-album/IMG_20191009_143804.jpg
Warning: [minor] Unrecognized MakerNotes - ./Camera-album/IMG_20191009_151052.jpg

Any switch to suppress warnings?
I guess I'll try -q (or -q -q?)

Edit: -q -q suppresses warnings. A single -q does not seem to make a difference.

Thank you again.

[StarGeek]

Also take a look at the -m (ignoreMinorErrors) option.