dashcam GPS latitude/longitude wrong

[Martin314]

I'm trying to extract GPS info from an MP4 file created by a dash cam using the -ee option.  The date/time, speed, and track are all correct.  However, the latitude and longitude are wrong.

At first I thought it was a simple scaling issue, so I divided the latitude and longitude as reported by the dashcam viewer software (which looks correct) by the values reported by exiftool.  But when I use these values to multiply subsequent GPS positions, the results don't match.  Instead, I would need to constantly adjust the divisor by some unknown amount.

I read a couple of other threads on dashcam GPS here, one in which it was mention that there was a bug fix. I'm using version 11.86 which has that fix.  The newer versions seem to have slightly different GPS values, but both are still way off.  To give an example of scale, to get a value close to the correct one I need to multiple the reported latitude by a value close to 9.65, and the longitude by a value close to 4.58.  But the required multiplier appears to change on the order of 0.0001 per second.  This means that the calculated coordinates are skewing away from the correct location by over 20 feet per second!

Is it possible that exiftool is extracting the value wrong?  Or are dashcams known for using difficultly obfuscated coordinate mappings?

[Martin314]

I'm wondering if maybe it's just a matter of endian order or wrong data type (e.g. different length floating point or something) since everything else seems to be correct and the Lat/Lon do seem to track, just wrong.

[StarGeek]

Can you provide a sample?

[Martin314]

What format?  I was thinking it would be ideal if I could just strip out the metadata, so I don't have a couple hundred megabyte file.  Otherwise, I'll have to make a special test where I just record a few seconds to make the file a manageable size.

[StarGeek]

Phil would have to see the actual file to figure out what's happening.  Take a short video with some movement so it can get different gps coordinates.  Dropbox or Google drive would be a good place to upload the file temporarily.  Or Phil may ask you to email it to him.

[Martin314]

Well I have made some progress... using the verbose option I was able to see the exact bytes of data being interpreted, and comparing GPS samples in which either the latitude or longitude didn't change between samples I was able to identify the exact 4 bytes for each.

exiftool is treating both as a 32-bit float (IEEE-754) which it then interprets as DDDMM.MMMM (it says this at line 1204 of QuickTimeStream.pl).

I haven't figured out exactly how these values are *actually* encoded.  Whether by coincidence or not, it appears (for the limited samples I have looked at) that the difference between two latitudes in binary is exactly 1 binary value per 0.00001 decimal degree, but for longitude it is closer to 0.00012.  This is treating lat and lon as 32-bit floats encoded Little Endian, e.g. LSB first.  What would be more helpful is if I had widely varying GPS coordinates to examine (mine are all within a few miles of each other), otherwise I can work out some weird correspondence which has a low error rate for my samples (for instance, subtracting off a huge number) but most likely this would be totally wrong and would be obvious for GPS samples far apart.

For what it's worth, the dashcam I have is the Rexing V1 (3rd generation)... it has the freeGPS header, and gets parsed in the last else of  ProcessFreeGPS().  Just to see what it would do, I tried forcing it to be parsed by all the prior clauses and none of them did as good.

[Phil Harvey]

If you send me a sample I will add support for this.  If you can upload a full video somewhere I'll download it.  Then email me the link (philharvey66 at gmail.com).  If you make a short video, be sure you are moving, and it is best if you go around a corner so I can decode things like GPSTrack and GPSSpeed.

- Phil

[Martin314]

Thanks Phil!  I'm not sure providing a file is necessary at this time.

I found FAQ #3 at https://dashcamviewer.com/faq/ which mentions that there is a "secret" encoding used for Latitude and Longitude.  Everything is know except for that part - although it might be useful to detect whether or not that encoding is in use (if detectable at all - there could also be a command line switch provided).

I have been mapping binary values to actual latitude/longitudes.  So far I know the representation of 0 for each, and various ranges in which the numbers seem to do different things.  I can share this here or by e-mail if you think this will help decode the scheme being used.

[Phil Harvey]

If I had a sample I should be able to find something unique in the metadata that I could use to determine whether or not this scaling should be applied.

But are you saying the factor is different for different files?  If so, I may be able to figure this out too, but I will need more samples in this case.

- Phil

[Martin314]

Well as all other fields are properly decoded, my main concern is decoding the latitude/longitude.  Even if it's just a special "use proprietary encoding" flag I would be fine with that.

I am attaching a small CSV file mapping around 200 raw values logged in the file versus the actual latitude and longitude (as reported by the "player program" which knows how to decode them) for those values.  There are three columns - the first is the raw 32-bit value in the file, the second is what latitude this represents when it's in the latitude field, and the third is what longitude this value represents when it's in the longitude field.  I haven't provided the value for both latitude and longitude for all values, as in many cases the value would be out of range anyway.   I have probed the entire range in the hopes that some clue could be obtained at the extreme values.

There is just a narrow region of 32-bit values that are actually valid GPS coordinates, and the same value doesn't correspond to the same latitude and longitude - for some reason the encoding is different.  I have found a few patterns - for instance, up near the top of the range there are some encodings which in which the output value is obtained by merely taking the negative value of the exponent (in IEEE-754 format) e.g. so I *think* they are floats, but in other regions the mapping makes no sense to me.   Care to take a look?

If you need any more data points, if you provide me with a list of input values I can tell you what latitude or longitude each value represents.

[Phil Harvey]

What I really need is the original video

[Martin314]

So I guess I'm on my own for the actual decoding of latitude/longitude?

[Phil Harvey]

I think were living on different planes of existence.

I'm offering to help, but I need a sample.

[Martin314]

I uploaded a sample of 200 raw values and the corresponding latitude/longitude where applicable.  That should be enough (or a start) to decoding lat/on.    Everything besides the wonky lat/lon (aside from possible some way to identify if the wonky scheme is in use) is identical to what exiftool expects, and is properly decoded.  So I think the only thing you would need an actual video file for is to figure out that identification.   I don't have a short video file yet, I haven't had a chance to make one - it will require that I specifically find a spot where I can stop the automatic recording and then record a short segment. (Really more the remembering to do this when I'm driving, which isn't every day.)  So if these raw values aren't enough to take a crack at lat/lon then I guess I'm on my own for that part. 

Anyway, I think I may have the longitude cracked, or at least a formula that is pretty close - probably because the range of values doesn't happen to straddle a floating point boundary.  Specifically, it seems that:
Actual Longitude = (Raw_Longitude - 1474560) * 0.000122070312

I have no idea what the significance of those two constants might be...

For latitude the distance between adjacent raw values doesn't correspond to a constant offset across the 0-90 degree range.  Understanding that might lead to a slightly more accurate longitude value also.

[Phil Harvey]

I have never seen a funny scaling like this in GPS coordinates.  Looking at your CSV file doesn't help much because the values are so disjoint, and I can't see how they increment as they would when an object is moving.

What I have seen in the past are:

1. binary values that are xor'd with some random value before being stored

2. binary values stored as 4-byte or 8-byte IEEE floating point, in either DD.DDDDD or DDMM.MMMMM format

3. 4 or 8 byte integer values stored as DD.DDDDD * 1000000 or DDMM.MMMMM * 1000000, and possibly scaled by 256/360 or some other degrees->byte conversion.

- Phil

[Martin314]

Thanks for taking a look.  I noticed the same thing (points not close enough together) when I looked more closely at longitude and realized that I wasn't seeing any patterns pop out in the bit patterns.  I have some more data points that are sequences of 6 or 8 numbers in a row at that point around 0 and 180 degrees longitude.  I'll keep adding data points from latitude and upload a new file when I have a nice collection.

[Martin314]

Attached is a more detailed mapping for Latitude values only.  (390 rows)
I have tried to include at least 16 adjacent values in a row for new test cases, as well as staggering out samples across the entire latitude range.

If there are any specific ranges where additional data points would be helpful, please let me know! 

[obetz]

Actual Longitude = (Raw_Longitude - 1474560) * 0.000122070312

This is obviously a scaling with powers of two:

0.000122070312 = 1/8192

8192 = 2**13

1474560 / 8192 = 180

So your formula can simplified to:

Actual Longitude = Raw_Longitude  / 8192 - 180

Oliver

[Martin314]

Thanks Oliver, good catch on seeing the 8192.    It seems I made a transcription error though between my notes and the post which I didn't find until I actually tried to use the formula.

1474560 is the raw value range between 0 and 180, so +1 in raw value represents 1/8192th of a degree.  I divided 180 / 1474560, if I had done it the other way around I would have noticed 8192.

0x4509732E = 0 Longitude

So the formula actually should be:
Actual Longitude = (Raw Longitude - 1158247214) / 8192

The zero point unfortunately isn't evenly divisble by 8192.

This relationship holds for values as least as low as 0x45000000 and as at least as high as 0x45800000, although it falls apart somewhere outside that range.  Fortunately we don't care about this as valid longitudes are between +0 and +180 (since E/W is a separate field).

[obetz]

0x4509732E = 0 Longitude

A huge number containing two large primes: 1158247214 = 2*9679*59833, not looking good in any common number base.

Your wrong number looked better <g>.

Oliver

[Martin314]

The longitude seems pretty easy to translate, but any ideas on the latitude?  If not, I'll try to find some time this weekend to crack it.

[Martin314]

It seems the GPS player for PC interprets out of range values differently than the Mac version.  On the Mac version I was getting ridiculously huge numbers (e.g. 0xFF000000 = 56713729510357210881534969900898975744.00000000 degrees) whereas on the PC if either latitude or longitude are out of range an incorrect value is shown for both, usually fairly small (< 2 degrees for latitude and < 20 degrees for longitude from what I've seen) but it does vary a bit.

Anyway, I discovered another interesting data point - a single increment of the raw value does NOT equate to the same difference in latitude values; it's dependent on the latitude value!  Additionally, I can't get any value to hit 0 degrees or 180 degrees exactly.

Code Select Expand

Hex      Dec            Longitude    difference from next
44E5E65C 1,155,917,404  -2.333330    N/A
44E5E65D 1,155,917,405  -179.999954  -0.00006100000002
44E5E65E 1,155,917,406  -179.999893  -0.00006099999999
44E5E65F 1,155,917,407  -179.999832  -0.00006099999999
44E5E660 1,155,917,408  -179.999771  N/A
...
4509732D 1,158,247,213  -0.00014     -0.00013
4509732E 1,158,247,214  -0.00001     0.0001
4509732F 1,158,247,215  -0.00011     0.00012
...
451FF32B 1,159,721,771  -179.999619  0.000122
451FF32C 1,159,721,772  -179.999741  0.000122
451FF32D 1,159,721,773  -179.999863  0.0001355
451FF32E 1,159,721,774  -179.9999985 N/A
451FF32F 1,159,721,775  -2.333330    N/A

Strange that there are almost two orders of magnitude in how much each increment changes the longitude going from -180 to 0, but then going back to -180 again it's negligible.  Because of this the difference between -180 to 0 is 2,329,809 while from 0 back to -180 it's only 1,474,560. However, it doesn't seem the dashcam actually uses this second range to encode any values; there is just some kind of weird quasi-symmetry going on in the decoder. 

I'm not sure where the sign of the longitude is stored, either, but I haven't been able to get a positive value manipulating just the 4 bytes holding the longitude.  I'm using a hex editor to patch values in, as I can't seem to find a way for `exiftool -ee` to write data back.

[Phil Harvey]

This is very interesting.  I'm glad to see someone else taking a crack at this because I just don't have the time myself to work on it.

- Phil

[Martin314]

I think I have found an algorithm for finding the longitude when it is between 0 and -180 (180 degrees west).

First, the latitude and longitude are both stored as 32-bit values in big endian format.  So looking at the byte order in the file with a hex editor if I see 78 56 34 12 then this yields a raw value of 0x12345678.

Now, if the raw value is of the form 0x44XXXXXX, then our divisor is 16384, our subtrahend is 0x44E5E65D and our base is -180.
But, if the raw value is of the form 0x45XXXXXX, then our divisor is 8192, our subtrahend is 0x45000000 and our base is -75.59938.

We then subtract our subtrahend from our raw value, then divide by our divisor to get the offset number of degrees, which we add to our base to get the longitude value.

It may be possible to simplify this algorithm further.

At this time I don't know how to represent positive longitude values (or negative latitudes).  I can't get any samples to test it out with.  There may be a bit somewhere else; I have experimented with a few locations but haven't been able to get a positive longitude or negative latitude to come out yet.  exiftool with the -ee option shows a speed and track reference, but no E/W/N/S for the latitude longitude.  For all I know the manufacturer designed this to only work on the north/west hemisphere!

I'll come up with some test cases the verify the above algorithm and then work on the latitude next... it appears to have a similar phenomena where part of the range has half the resolution, but in this case it seems that adjacent values simply translate to the same latitude.

[Phil Harvey]

I think you should be looking at the values as IEEE 4-byte floats.  The 0x44 and 0x45 at the start are the exponent part

- Phil