Hello.
The attached config file (geoloc.config) defines the Composite tag GeoLocation that converts GPS coordinates to locality, city, state and country using the bigdatacloud.com free service.
It requires "curl" (https://curl.se) and "jq" (https://jqlang.github.io/jq/) to be installed in your PATH.
exiftool -config geoloc.config -gpslongitude -gpslatitude -geolocation image.jpg
GPS Longitude : -77.116753
GPS Latitude : +38.861903
Geo Location : Arlington, Washington, US-VA, US
This config file has been tested with ExifTool 12.77 on macOS and Linux.
Christian W.
Hi Christian,
Thanks.
What are your thoughts on bigdatacloud's longevity in maintaining this free geolocation service?
This feature is something that could fairly easily be built into ExifTool, but I have 2 reservations:
1. I don't like the idea of ExifTool accessing the internet connection for anything. It is so refreshing to me to have an app that doesn't do this.
2. I worry about code maintenance with having to find another free service once this one shuts down or starts charging.
- Phil
You probably can skip the use of jq by letting exiftool do the parsing.
I totally forgot that had previously uploaded a config that used cURL to get time zone information from Google maps. You can look at the code I used to do this in this post (https://exiftool.org/forum/index.php?topic=11026.0).
I was interested in following this up but it appears that free usage of BigDataCloud's geolocation API is only available in certain limited cases:
Web browser:
Quote1. You are allowed to make free and unlimited API calls from a client's web browser using their current coordinates...
Mobile device:
Quote2. You are allowed to make unlimited free API calls from client mobile devices such as iOS, Android or IoT...
And a warning:
Quote"IMPORTANT! Any user detected abusing the service by implementing it on a server-side or a client-side along with coordinates obtained elsewhere will be blacklisted from all of our free API Services."
I suspect that "Coordinates obtained elsewhere" implies that the use of their API with multiple different lat and lon coordinates (for example from image metadata) falls outside their fair usage policy :(
:(
I wonder how large the geolocation database is, and whether there is a freely-available version? Adding a database lookup would be fairly easy to add. Then there would be no external dependencies.
- Phil
Edit: I've found a couple of creative commons geolocation databases. One with all Cities greater than population 500 is only 11 MB (from here (https://download.geonames.org/export/)). Maybe I'll look into adding the ability to geolocate the nearest town. There is also one for geographic features, but this is 300+ MB.
Edit2: I've been playing around with that database. I can get all of the information I need to geolocate to the nearest city of 500 people or more including full state/province and country name from a database of 4.2 MB. That would double the download size of ExifTool, so I won't include it by default, but I could think of providing this as an add-on of some type, perhaps with a new -geolocate option or something.
Edit3: Unfortunately this feature would support only english names.
- Phil
Hello Phil,
Quote from: Phil Harvey on March 02, 2024, 10:32:39 PMWhat are your thoughts on bigdatacloud's longevity in maintaining this free geolocation service?
I believe that as with most online services, the longevity of the service is short.
Quote1. I don't like the idea of ExifTool accessing the internet connection for anything. It is so refreshing to me to have an app that doesn't do this.
2. I worry about code maintenance with having to find another free service once this one shuts down or starts charging.
I agree with you. ExifTool should not connect to the Internet. Code maintenance will be a unnecessary burden.
I'm exploring other services that allow reverse geocoding, like Open Street Maps, their database is open source.
Christian W.
Quote from: StarGeek on March 03, 2024, 02:24:47 AMYou probably can skip the use of jq by letting exiftool do the parsing.
That would be a possibility. I feel more comfortable using jq, so the Composite tag code is simpler.
Hello Hubert.
Quote from: Hubert on March 03, 2024, 05:28:55 AMI suspect that "Coordinates obtained elsewhere" implies that the use of their API with multiple different lat and lon coordinates (for example from image metadata) falls outside their fair usage policy
Thanks for sharing bigdatacloud's fair use policy. I didn't read it when I implemented the config file. It seems that they offer the reverse geocoding service API in exchange of real coordinates so they can map GPS coordinates to IP addresses.
I'm exploring Open Street Maps reverse geocoding. I need to check their policies to see if our use case is allowed.
Christian W.
Quote from: Phil Harvey on March 03, 2024, 07:16:52 AMAdding a database lookup would be fairly easy to add. Then there would be no external dependencies.
This is a great idea! The database could be separately downloaded by the users that need geolocation.
I've been exploring OpenStreetMap data and found that the JSON returned is not always consistent, for example: with some coordinates the JSON returns the City, State and Country fields and with other coordinates the JSON returns the Town, State and Country fields . This happens even with coordinates within the same country.
It looks like OpenStreetMap's reverse geocoding is a mess and more code is needed to be able to use it in a composite tag.
I'll continue to look into options for doing reverse geocoding online, but I think Phil's idea of using a local database is the best solution.
I've got a new API Geolocation option working and will include this new feature in version 12.78:
> exiftool a.jpg -api geolocation "-gpsL*" "-geolocation*" -e
GPS Latitude Ref : South
GPS Latitude : 44.000000
GPS Longitude Ref : West
GPS Longitude : 75.000000
Geolocation City : Quellón
Geolocation Region : Los Lagos Region
Geolocation Country : Chile
Geolocation Time Zone : America/Santiago
Geolocation Population : 30000
Geolocation Distance : 117.7
Geolocation GPS : -43.119,-73.614
Using only cities with population of 1000 or greater, I've got the zipped database size down to 1.7 MB, which I think is small enough to include in the standard ExifTool distribution.
- Phil
Quote from: Phil Harvey on March 04, 2024, 09:22:28 AMI've got a new API Geolocation option working and will include this new feature in version 12.78:
Hello Phil. This is great news!
Will the database be replaceable by the user?
Thanks,
Christian W.
Hi Christian,
I'm having to use a custom database file to get the size down. But I would be able to generate alternate versions if the demand arose.
I'm also adding a new write-only Geolocate tag that allows you to do this:
> exiftool a.jpg -xmp:all
> exiftool a.jpg -geolocate=44,-72
1 image files updated
> exiftool a.jpg -xmp:all
XMP Toolkit : Image::ExifTool 12.78
Country Code : US
City : Haverhill
Country : United States
State : New Hampshire
- Phil
I have been looking for a 'database file' to use in ExifToolGui. I found these URL's. Maybe they can be an option to use in ExifTool:
http://download.geonames.org/export/dump/cities500.zip (http://download.geonames.org/export/dump/cities500.zip)
http://download.geonames.org/export/dump/cities1000.zip (http://download.geonames.org/export/dump/cities1000.zip)
http://download.geonames.org/export/dump/cities5000.zip (http://download.geonames.org/export/dump/cities5000.zip)
http://download.geonames.org/export/dump/cities15000.zip (http://download.geonames.org/export/dump/cities15000.zip)
(Where the nbr stands for 'cities with population greater than')
I decided not to use them because they are fairly large, but in stead give the option to use the online api's:
https://geocode.maps.co (https://geocode.maps.co) or https://overpass-api.de/api
(https://overpass-api.de/api)
But if Exiftool would provide this functionality I would certainly add it to ExifToolGui. Is there an option that I can 'play' with 12.78 to test?
I've just released version 12.78 with the new Geolocation feature. Have fun with it! :)
- Phil
Quote from: Phil Harvey on March 05, 2024, 03:31:45 PM12.78 with the new Geolocation feature
Great! Which database does it use?
- Matti
Quote from: Phil Harvey on March 05, 2024, 03:31:45 PMI've just released version 12.78 with the new Geolocation feature. Have fun with it! :)
- Phil
I'm playing and having fun. Thanks.
I do have some questions, but since they're more ExifToolGui I will post it there.
Hi Matti,
Quote from: wywh on March 06, 2024, 05:20:02 AMGreat! Which database does it use?
The >1000-population city database from geonames.org (https://download.geonames.org/export/) that FrankB mentioned, but I have reformatted the database to reduce the file size.
I've already added another feature to allow user-defined cities to be added, and am using that myself to geolocate pictures from my last trip because we stayed in some very small towns that aren't in the database. The user-defined city feature will appear in 12.79. I may also provide the geonames.org >500-population database as an optional download.
- Phil
Edit: I've uploaded the >500-population database (https://exiftool.org/Geolocation500.zip) in case someone wants to use it. Just unzip it and either replace lib/Image/ExifTool/Geolocation.dat with this new file, or set $Image::ExifTool::Geolocation::databaseFile in your ExifTool config file to the path name of the database file.
I have posted something in the ExifToolGui forum. Because I think it applies mostly to GUI.
https://exiftool.org/forum/index.php?topic=15799.0 (https://exiftool.org/forum/index.php?topic=15799.0)
Quote from: Phil Harvey on March 06, 2024, 09:49:48 AMThe >1000-population city database from geonames.org (https://download.geonames.org/export/) that FrankB mentioned, but I have reformatted the database to reduce the file size.
Thanks for the info. I compared city center GPS coordinates that 'exiftool -api geolocation=44.03411,-72.06292' gives (44.033,-72.065) and what the cities1000.txt (https://download.geonames.org/export/dump/cities1000.zip) gives (44.03451,-72.06398) and wondered if some another database was used because they are slightly different.
I wonder who has determined all those city centers because many seem to be just casually thrown there ???
- Matti
Hi Matti,
Yes, I packed the coordinates into 2-byte integers, so there was some rounding.
- Phil
Is there a definition to the Geolocation.dat format so that it could be possible to make our own from some other database?
Also, for the Windows executable, maybe check the executable's directory for a replacement Geolocation.dat?
The format specification is in the comments of Geolocation.pm (https://github.com/exiftool/exiftool/blob/master/lib/Image/ExifTool/Geolocation.pm), and I describe how to use an alternate database at the bottom of the Geolocation instructions here (https://exiftool.org/geotag.html#Geolocation). However the next release will allow user-defined cities and give an example of how to import a CSV-format database.
- Phil
It would have been cool if I could use this feature to automatically set the default time zone when geotagging from a GPX file, but unfortunately it is a chicken-and-egg problem because the location needs to be known to determine the time zone, and the time zone needs to be known to look up the location in the GPX track. :( Darn.
However, I will enhance the -geotag feature in ExifTool 12.79 to allow the Geolocation tags to be set at the same time as the GPS tags.
- Phil