DJI H20T thermal radiometric jpeg file

[Jonathan McGillivray]

DJI have released a new camera with an integrated 640 x 512 radiometric thermal camera. The thermal camera does not seem to be based on a Flir core.

I have been trying to get the raw temperature information from the radiometric jpeg but they seem to be storing the temperature information in a different format to the Flir R-JPEG images. Their spec sheet just says that the image format is R-JPEG (16 bit).

Any help or guidance with how to extract the temperature information using ExifTool would be greatly appreciated. I have attached a sample image.

[Phil Harvey]

There are a number of new APP segments in this file.  There is 650 kB of data in APP3 segments that could be the raw thermal data.  There is another 32 kB in APP5 that could potentially be something like this too.  APP4 is smaller, and possibly contains some setting or calibration information.  APP6 contains this text:

Code Select Expand

DTAT
{
"points":[
{"x":292,"y":228,"id":0},
{"x":399,"y":178,"id":1},
{"x":489,"y":276,"id":2},
{"x":290,"y":388,"id":3},
{"x":106,"y":341,"id":4}],
"rects":[
{"x":66,"y":75,"width":169,"height":179,"id":0},
{"x":465,"y":56,"width":104,"height":164,"id":1}],
"code":0
}

Lots of fun stuff to try to decode.  You can see it yourself with the -v3 option or with -htmldump

- Phil

[Jonathan McGillivray]

Thanks for the quick response.

You are correct. The APP3 segment seems to be the raw sensor values. I've had a friend look at the image in python and he's managed to convert the APP3 segments from byte arrays into 16-bit unsigned integer pixels. See attached .tiff image generated from the APP3 segment data; However, values range from 18073 - 21222. I'm guessing the calibration data is required to convert the raw sensor values to actual temperature readings.

Can't seem to find out what encoding to use to decode the APP4 segment. Any ideas?

APP6 contains information stored from adding 'features' to the image in their software. For example, users can add points or rectangles to get temperature measurements from the image within their software. This information is subsequently stored in the image file under this segment.

[Phil Harvey]

Some of the APP4 data may be 2-byte integers.  Other parts look like maybe 4-byte floats, but I don't have time to look into this in any depth right now.

- Phil

[mangulo]

Hi all,  ... I would like to seek help from the university to decoding APP4 segment.

[Jonathan McGillivray]

This is the code in python. Not sure how one could do it using ExifTools yet

Code Select Expand


from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
import tifffile as tiff

im = Image.open("DJI_20200513193136_0004_THRM.JPG")

# concatenate APP3 chunks of data
a = im.applist[3][1]
for i in range(4, 14):
    a += im.applist[i][1]

# create image from bytes
img = Image.frombytes('I;16L', (640, 512), a)

temps = np.array(img)

# still need to apply formula to image to convert raw sensor values to temps

tiff.imsave('DJI_20200513193136_0004_THRM.tiff', temps)
np.savetxt('temps.csv', temps, delimiter=',')

plt.imshow(temps.astype(np.uint), cmap="inferno")
plt.show()


[musteresel]

I'm working on this atm.

When zeroing the APP4 segment and then opening the image with the DJI Thermal Analysis Tool gives the error message

Code Select Expand


>>>> ir_image_temp_info_get
ERROR: magic header adjust failed-7


When zeroing the APP5 segment, then that tool gives this error message:

Code Select Expand


>>>> ir_image_temp_info_get
ERROR: curve lut monotonicity adjust failed-8


So the APP5 segment is apparently also needed for decoding; it's a series of ever increasing values (unsigned 16 bit, with wrap around)

The APP4 segment is (in my images) mostly zeros, but information seemingly spread out at random.  Do you know what kind of format that could be?

[Jonathan McGillivray]

It seems like a compensation is done to the images for Flat Field Corrections to prevent a sudden change in temperature values between FFCs. My guess is that maybe this compensation is stored within the APP5 segment?

I've been trying to compare the APP4 data between two different cameras. See attached spreadsheet showing the APP4 data in hex for 2 different cameras and multiple images from the same camera. It seems like all the images start with the same header. If I zero any of the values within this header no temperature information is displayed within the software. I then started changing other values within the APP4 segment. This caused the temperatures within the software to change which shows that these values are definitely used to convert the raw sensor values to temperature values. I highlighted these values in green within the spreadsheet. I'm not sure how one would go about converting these values from binary/hex to actual decimal or integer values without knowing what you're looking for. Hoping that Phil will be able to add some input once he's got time to look at it.

[musteresel]

I've had some success with reverse engineering the DJI Thermal Analysis Tool:

- It is an electron app
- This means it has an app.asar file; which contains a js file with the code which runs the application
- That code uses node-ffi (and ref, ref-struct, ref-array) libraries to access a dll: dji_ir_measure.dll (which uses another dll which I think contains the actual code)

This allowed me to get out which functions are in that DLL, and also on what kind of data structures they operate.  I was able to assemble a C header file with these structures (member names come from the JS code, structure names had been obfuscated) and typedefs for functions pointers for (most of) the available functions:

Code Select Expand


#include <stdint.h>

#pragma pack(1)
struct A
{
  uint16_t raw_width;
  uint16_t raw_height;
  uint32_t raw_size;
  uint32_t header_size;
  uint32_t curvelut_size;
};

#pragma pack(1)
struct S
{
#pragma pack(1)
  struct correct_params_t
  {
    uint16_t distance;
    uint16_t humidity; //hunidity??
    uint16_t emiss;
    int16_t reflection;
    int16_t ambient_temp;
  } correct_params;
};


#pragma pack(1)
struct N
{
  uint32_t app_version;
  char dll_version[16];
#pragma pack(1)
  struct device_info_t
  {
    char serial_number[32];
    uint64_t timestamp;
#pragma pack(1)
    struct version_t
    {
      char sys[16];
      char ps[16];
      uint32_t pl;
      uint32_t hardware;
    } version;
  } device_info;
  struct A raw_info;
  struct S temp_param;
  char padd[10]; // no idea why this is neccessary? it's not in the JS code...
};



// before using any function with this, probably there needs to be a
// function called which fills a struct A!
#pragma pack(1)
struct L
{
  char * product_name; // "Plug\0" : "IR_CAM\0"
  uint16_t *raw; // pointer to array?
  uint16_t *raw_header; // pointer to fresh allocated array?
  int16_t *curve_lut; // pointer to fresh allocated array?
  uint16_t width; // these are set!
  uint16_t height; // this, too
  uint16_t debug; // set to 0; increase!
};

#pragma pack(1)
struct ColorBar
  {
    uint8_t auto_enable;
    uint8_t temp_unit;
    uint8_t res2;
    uint8_t res3;
    int16_t high;
    int16_t low;
  };


#pragma pack(1)
struct H
{
#pragma pack(1)
  struct image_processing_t
  {
    uint8_t format_input;
    uint8_t format_output;
    uint8_t brightness;
    uint8_t contrast;
    uint8_t sharpness;
    uint8_t pseudo_color;
    uint8_t res1;
    uint8_t res2;
  } image_processing;
#pragma pack(1)
  struct isotherm_t
  {
    uint8_t enable;
    uint8_t res1;
    uint8_t res2;
    uint8_t res3;
    int16_t high;
    int16_t low;
  } isotherm;
#pragma pack(1)
  struct roi_t
  {
    uint8_t roi_mode;
    uint8_t res1;
    uint16_t start_x;
    uint16_t start_y;
    uint16_t size_width;
    uint16_t size_height;
  } roi;
  struct ColorBar color_bar;
};


#pragma pack(1)
struct D
{
  struct ColorBar color_bar;
};

#pragma pack(1)
struct M
{
  uint8_t pseudo_color_r[2560];
  uint8_t pseudo_color_g[2560];
  uint8_t pseudo_color_b[2560];
};


typedef char type_c;
typedef char type_e;
typedef char type_a;


#pragma pack()
struct S_small
{
  int16_t lut_temp_ptr;
  uint16_t lut_y16_ptr;
  uint32_t lut_length;
};


typedef int (*dji_ir_image_product_info_get)
(
char const *,
struct A * // type_c * ??
);

typedef int (*dji_ir_image_temp_info_get)
(
struct L*,
struct N *
);

typedef int (*dji_ir_image_temp_measurement)
(
struct L*, /* "rawInfo" */
struct S* /* nullable, "tempInfo" */,
int16_t * /* fresh allocated; width * height * 2 (bc. 16 bit!), "temperData" */
);

typedef int (*dji_ir_image_isp_process)
(
struct L*,
struct S * /* nullable*/,
struct H *,
struct D *, // type_a *,
uint8_t * /* out buffer */
);

typedef int (*dji_get_pseudo_color_data_rgb)
(
struct M * /* most probably! was type_g */
);

typedef int (*dji_ir_refresh_temp_discrete_lut)
(
struct L*,
struct S* /* nullable*/,
struct S_small * /* nullable*/
);

typedef void (*dji_ir_refresh_temp_discrete_lut_free)(void);

typedef int (*dji_ir_save_data)
(char *, uint32_t, char*, uint32_t);

typedef int (*dji_ir_read_data)
(char *, uint32_t, char*, uint32_t *);


This allows me to link against the DLL and get the temperature data, for example:

Code Select Expand


void temp_measurement_example(dji_ir_image_temp_measurement fn) // supply pointer to function loaded from DLL
{
  int const width = 640;
  int const height = 512;
  char const o[] = "IR_CAM\0";
  char *raw, *raw_header, *curve_lut;
  size_t raw_size, raw_header_size, curve_lut_size;
  ReadJpgFile("file.jpg",
              &raw, &raw_size,
              &raw_header, &raw_header_size,
              &curve_lut, &curve_lut_size);
  struct L h;
  h.product_name = o;
  h.raw = raw;  // this is the APP3 segments concatenated
  h.raw_header = raw_header; // this is the APP4 segment
  h.curve_lut = curve_lut; // this is the APP5 segment
  h.width = width;
  h.height = height;
  h.debug = 0;
  int16_t * data = malloc(2 * width * height);
  int ret = fn(&h, NULL, data); // call to the DLL function

  // now we have the temperature values in data! In (°C/10)


  int16_t max = INT16_MIN;
  int16_t min = INT16_MAX;
  for (int i = 0; i < width * height; ++i)
    {
      if (max < data[i]) max = data[i];
      if (min > data[i]) min = data[i];
    }
  printf("min = %d\nmax = %d\n", min, max);
  printf("TopLeft = %d\nTopRight = %d\nBottomRight = %d\nBottomLeft = %d",
         data[0],
         data[width - 1],
         data[width * height - 1],
         data[width * (height - 1)]
         );
}



It doesn't end here...

Being an electron app, it is possible to INJECT code into the DJI Thermal Tool; which makes it possible to observe exactly how and when functions from the DLL are called:

1. Unpack app.asar

Code Select Expand

npx asar extract app.asar extracted
2. Modify extracted/app/dist/js/app.dc53e99d.js file to include any logging or similar
3. Pack a new app.asar

Code Select Expand

npx asar pack extracted app.asar
4. Run DJI Tool with injected code Perhaps

Code Select Expand

set  ELECTRON_ENABLE_LOGGING=true (as per https://stackoverflow.com/a/40929681/1116364) to be able to use

Code Select Expand

console.log("WOW") in the injected code

What's really nice is that the DLL by default seems to print quite a bit debugging information; here's what I get when opening one of my files:

Code Select Expand


>>>> ir_image_temp_measurement
temp head : [0x55AA][0x0612]
dev  head : [0xAA55][0x0612]
K1          -130.443
K2          26.1648
K3          758.386
K4          0
KF          79.5847
B1          0
B2          -575.543
Distance    5
Humidity    70
Emiss       100
Reflection  230
D1          21
D2          -945
KJ          100
DB          0
KK          261


Searching for the HEX values of Distance, Humidity, ... shows that they are at offsets in the APP4 segment:

- 0x44 for distance
- 0x46 for humidity
- 0x48 for Emiss
- 0x4a for reflection
- 0x4c for D1 (what ever that is)

Code Select Expand


00000030: 0000 0000 5b2b 9f42 0000 0000 bee2 0fc4  ....[+.B........
00000040: 0000 0000 0500 4600 6400 e600 1500 0000  ......F.d.......
00000050: 4ffc ffff 6400 0000 0501 0100 0300 0000  O...d...........


I don't know if these offsets are constant, though.  APP4 does not seem to use IFD tags, or I'm just not used to decoding them yet?

However, the values show that there are quite a few parameters to extract.

*edit*

Looking at your spreadsheet shows where most of these values seem to be. The question remains: Are these offsets constant?
Also the last two bytes of the APP4 segment change relatively "freely", this could be a checksum or similar?

Linking these posts (which are related to FLIR raw values decoding) because the parameters here might be similar, or possibly even the same with the extra parameters to include distance and external optics transmission in the calculation:

- https://exiftool.org/forum/index.php/topic,4898.msg23944.html#msg23944
- https://exiftool.org/forum/index.php/topic,4898.msg23972.html#msg23972

[Phil Harvey]

Wow, that is some excellent detective work!!

The offsets for Distance, etc are fairly low numbers (0x44 is near the start of the segment), which is great.  If they were variable, one would expect to see a size word stored in the segment before 0x44 with a value value of less than or equal to 0x44.  But I don't see anything before 0x44 that looks like size/offset value or a version number.  So my guess is that the offsets are constant for these parameters.

- Phil

[FRANKOOL]

Congratulations to everyone for your effort. Is there a relatively simple way to convert thermal files from dji to tiff with the temperature information compensated by the FFC?
In app4 the raw values ​​of the differentials multiplied by 0.005, which is the thermal sensitivity of the camera, coincide with the values ​​obtained in the DJI software. With this and a value from raw pixel to centigrade we could convert to real degrees, but this step is the one that we do not know how to do. Greetings and thank you very much for the effort!

[hairyape1]

DJI have been very short sighted in effectively encrypting their data so that only their software can be used to view the data. Being able to view the raw data in a more useful format eg tiff would allow so many more possibilities in terms of analysis etc.
Ive been in discussions with DJI daily for the last couple of months to try and resolve this and they are not being very helpful :/
This is all way beyond my level of understanding, but great to see that you are making some progress.
Thank you all for your efforts so far. If DJI every give me any actually useful information Ill post it here.
Cheers
Fingers crossed we can overcome this one way or another.

[hairyape1]

If it helps to determine whether offsets/calibration data is standard from one sensor to another... here are a selection of example images captured with my h20t. All captured on the same day within a few minutes of each other.
Very happy to go and collect more data if anything specific would be of use.

[Jonathan McGillivray]

That's some really good reverse engineering @musteresel! Thanks for sharing all your work. Based on the info you provided I've created a wrapper to use the .dll within python using ctypes. Anyone who's not too comfortable with C (like myself) can use this in the meantime to batch convert the DJI R-JPEG images to 16 bit tiff images.

It would be great to still try and figure out what formula DJI is applying to do the temperature conversion within their .dlls. I tried to take a quick look at the values provided with Electron logging enabled and I couldn't pick up a correlation between Flir's calibration values and the values printed through the logging. My values are as follows:

Code Select Expand


K1          -98.3823
K2          25.5608
K3          655.766
K4          0
KF          80.438
B1          0
B2          -463.6
Distance    5
Humidity    70
Emiss       100
Reflection  230
D1          21
D2          -945
KJ          100
DB          0
KK          204


My guess is that maybe K1, K2, K3, K4, KF, B1, B2 are the values used within the formula. I'll try take a look into it in more detail when I get some more time. If anyone else with experience in calibration of thermal sensors has got any more insight it would be appreciated.

Python Wrapper for DLLs

Code Select Expand


from ctypes import (
    Structure,
    c_uint64,
    c_uint32,
    c_uint16,
    c_int16,
    c_uint8,
    c_int,
    c_char,
    c_char_p,
    POINTER,
    util,
    CDLL
)

import sys

class A(Structure):
    _pack_ = 1
    _fields_ = [
        ("raw_width", c_uint16),
        ("raw_height", c_uint16),
        ("raw_size", c_uint32),
        ("header_size", c_uint32),
        ("curvelut_size", c_uint32)
    ]


class correct_params_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("distance", c_uint16),
        ("humidity", c_uint16),
        ("emiss", c_uint16),
        ("reflection", c_int16),
        ("ambient_temp", c_int16),
    ]


class S(Structure):
    _pack_ = 1
    _fields_ = [
        ("correct_params", correct_params_t),
    ]


class version_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("sys", c_char * 16),
        ("ps", c_char * 16),
        ("pl", c_uint32),
        ("hardware", c_uint32),

    ]


class device_info_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("serial_number", c_char * 32),
        ("timestamp", c_uint64),
        ("version", version_t)
    ]


class N(Structure):
    _pack_ = 1
    _fields_ = [
        ("app_version", c_uint32),
        ("dll_version", c_char * 16),
        ("device_info", device_info_t),
        ("raw_info", A),
        ("temp_param", S)
    ]


class L(Structure):
    _pack_ = 1
    _fields_ = [
        ("product_name", c_char_p), # "Plug\0" : "IR_CAM\0"
        ("raw", POINTER(c_uint16)),
        ("raw_header", POINTER(c_uint16)),
        ("curve_lut", POINTER(c_int16)),
        ("width", c_uint16),
        ("height", c_uint16),
        ("debug", c_uint16) # set to 0
    ]

class ColorBar(Structure):
    _pack_ = 1
    _fields_ = [
        ("auto_enable", c_uint8),
        ("temp_unit", c_uint8),
        ("res2", c_uint8),
        ("res3", c_uint8),
        ("high", c_int16),
        ("low", c_int16)
    ]

class image_processing_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("format_input", c_uint8),
        ("format_output", c_uint8),
        ("brightness", c_uint8),
        ("contrast", c_uint8),
        ("sharpness", c_uint8),
        ("pseudo_color", c_uint8),
        ("res1", c_uint8),
        ("res2", c_uint8)
    ]

class isotherm_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("enable", c_uint8),
        ("res1", c_uint8),
        ("res2", c_uint8),
        ("res3", c_uint8),
        ("high", c_int16),
        ("low", c_int16)
    ]

class roi_t(Structure):
    _pack_ = 1
    _fields_ = [
        ("roi_mode", c_uint8),
        ("res1", c_uint8),
        ("start_x", c_uint16),
        ("start_y", c_uint16),
        ("size_width", c_uint16),
        ("size_height", c_uint16),
    ]

class H(Structure):
    _pack_ = 1
    _fields_ = [
        ("image_processing", image_processing_t),
        ("isotherm", isotherm_t),
        ("roi", roi_t),
        ("color_bar", ColorBar)
    ]

class D(Structure):
    _pack_ = 1
    _fields_ = [
        ("color_bar", ColorBar),
    ]

class M(Structure):
    _pack_ = 1
    _fields_ = [
        ("pseudo_color_r", c_uint8 * 2560),
        ("pseudo_color_g[2560]", c_uint8 * 2560),
        ("pseudo_color_b[2560]", c_uint8 * 2560),
    ]
class S_small(Structure):
    _fields_ = [
        ("lut_temp_ptr", c_int16),
        ("lut_y16_ptr", c_uint16),
        ("lut_length", c_uint32),
    ]

mylib_path = util.find_library("./dji_ir_measure")
if not mylib_path:
    print("Unable to find the specified library.")
    sys.exit()

try:
    mylib = CDLL(mylib_path)
except OSError:
    print("Unable to load the system C library")
    sys.exit()

dji_ir_image_temp_measurement = mylib.dji_ir_image_temp_measurement
dji_ir_image_temp_measurement.argtypes = [POINTER(L), POINTER(S), POINTER(c_int16)]
dji_ir_image_temp_measurement.restype = c_int


You can then use the above code to convert the R-JPEG images as follows:

Code Select Expand


from PIL import Image
from ctypes import (
    c_uint16,
    c_int16,
    c_char_p,
    POINTER,
    cast,
    create_string_buffer,
    byref
)
import numpy as np
import tifffile as tiff

from dji_ir_measure_wrapper import dji_ir_image_temp_measurement, L, S, correct_params_t

def read_image(filepath):
    im = Image.open(filepath)
    a3, a4, a5 = None, None, None
    for app in im.applist:
        if app[0] == 'APP3':
            a3 = a3 + app[1] if a3 else app[1]
        elif app[0] == 'APP4':
            a4 = a4 + app[1] if a4 else app[1]
        elif app[0] == 'APP5':
            a5 = a5 + app[1] if a5 else app[1]
    width, height = im.size
    return {'raw': a3, 'raw_header': a4, 'curve_lut': a5, 'width': width, 'height': height}

def create_L_instance(image):
    instance = L()
    instance.product_name = cast(create_string_buffer(b"IR_CAM"),
                          c_char_p)
    instance.raw = cast(image['raw'], POINTER(c_uint16))
    instance.raw_header = cast(image['raw_header'], POINTER(c_uint16))
    instance.curve_lut = cast(image['curve_lut'], POINTER(c_int16))
    instance.width = image['width']
    instance.height = image['height']
    instance.debug = 0
    return instance

def create_params_instance(settings):
    instance = S()
    params = correct_params_t(**settings)
    instance.correct_params = params
    return instance

image = read_image('DJI_20200513193308_0030_THRM.JPG')
settings = {'distance': 20, 'humidity': 70, 'emiss': 100, 'reflection': 230}

data = np.zeros(image['width'] * image['height'], dtype=np.int16)
data_ptr = data.ctypes.data_as(POINTER(c_int16))

h = create_L_instance(image)
params = create_params_instance(settings)
ret = dji_ir_image_temp_measurement(byref(h), params, data_ptr) # // call to the DLL function

max_temp = np.nanmax(data) / 10
min_temp = np.nanmin(data) / 10
print("Max: {}, Min: {}".format(max_temp, min_temp))

data = np.reshape(data, (image['height'], image['width'], 1))
tiff.imsave('image.tiff', data) #temperature values in data are °C/10


Note that the above script needs to be run from a 32bit python version with libdji_irprocessing.dll and dji_ir_measure.dll in the same directory.

[hairyape1]

Thanks Jonathan,

Just having a go with python to convert some images this end well to be honest not, me but a friend is helping out.... Im good with the actual thermal data but less experienced with python etc. Will report back how we get on.

Are any of the calibration values constant from one sensor to the next? Im wondering if they use some constant values and only vary a few depending on specific sensor variation from one sample to the next. When I was involved in calibrating optris sensors we would have some fixed values and then some sensor specific values and some lens specific.

Cheers