[weather@mockturtl] Applet Bug: GeoJS.IO dependancy causes "Service Error" from some IPs

[BillyCroan]

Applet version/Build date

Version 3.2.13 (2023-04-28 02:24:56)

Cinnamon version

5.6.8

Distribution

Debian 12

Graphics hardware and driver used

Intel Corporation Alder Lake-UP3 GT2 [Iris Xe Graphics]

Applet name and maintainer

weather@mockturtl @Gr3q

What happened?

Panel shows "Service Error" instead of weather. It wasintermittant at for about a day now completely constant.

I thought maybe my API key was saturated, but my usage should be around 10% of whats allowed, and I logged into OWM and looked around and there was no indicaton that my key was at fault. Same error occurs if I reset settings to defaults (don't use my key).

In melange, isee a error 500 when calling URL. I added the DEBUG file to try to get more out of it. I'd sure like to know the full URL it's calling, so that I could verify the 500 in curl and maybe report it to OWM not here if I knew it wasn't an applet problem

Other information

error t=2024-01-06T05:33:46Z [weather@mockturtl#15]: Retrying in the next 105 seconds...
info t=2024-01-06T05:33:48Z [weather@mockturtl#15]: Refreshing in progress, refresh skipped.
info t=2024-01-06T05:33:49Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,

<title>500 Internal Server Error</title>

500 Internal Server Error

---

openresty

error t=2024-01-06T05:33:49Z [weather@mockturtl#15]: Retrying in the next 105 seconds...
info t=2024-01-06T05:33:52Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,

<title>500 Internal Server Error</title>

500 Internal Server Error

---

openresty

error t=2024-01-06T05:33:53Z [weather@mockturtl#15]: Retrying in the next 120 seconds...
info t=2024-01-06T05:33:53Z [weather@mockturtl#15]: App is currently refreshing, refresh skipped in main loop
info t=2024-01-06T05:35:53Z [weather@mockturtl#15]: Error calling URL: Internal Server Error,

<title>500 Internal Server Error</title>

500 Internal Server Error

---

openresty

error t=2024-01-06T05:35:53Z [weather@mockturtl#15]: Retrying in the next 135 seconds...
info t=2024-01-06T05:35:53Z [weather@mockturtl#15]: App is currently refreshing, refresh skipped in main loop

[BillyCroan]

I see the same 500 int server error no matter which 'Data Service' I try to use. Interesting. Maybe its related to location services if that fires before the weather service?

[BillyCroan]

ah HA! Check this out: https://get.geojs.io/ So geojs.io gave up the ghost. Why not use gelclue2?
/usr/libexec/geoclue-2.0/demos/where-am-i
It uses nearby wifi networks to nail down position. Better than geoip because if you're on a vpn geoip will get it wrong.
Alternately the user should be able to specify the coordinates somehow to bypass geojs.io
or perhaps there could be multiple location providers in a dropdown or preference order.

[BillyCroan]

also, i noticed in melange when I switched on 'manual location':
nfo t=2024-01-06T05:54:24Z [weather@mockturtl#15]: Error calling URL: Not Found,

<title>File Not Found</title>

File not found: API no longer accessible via this URL

Using the URL /search/ and /reverse/ (with slashes) is no longer supported. Please use URLs as given in the documentation.

Examples how to change the URL:

You use: https://nominatim.openstreetmap.org/search/?q=Berlin
Change to: https://nominatim.openstreetmap.org/search?q=Berlin

You use: https://nominatim.openstreetmap.org/search/US/Texas/Huston
Change to: https://nominatim.openstreetmap.org/search?q=Huston, Texas, US

See github issue #3134 for more details.

error t=2024-01-06T05:54:24Z [weather@mockturtl#15]: Retrying in the next 225 seconds...

This sounds like evnen the nominatim api at OSM has changed.

I have had some success with "https://ipinfo.io/$public_ip/json" before.

But I also would like to suggest something I've pondered about. Do we really need to look up location more than one time ever for a given network? i.e. once we know the gps coordinate of my house and my gateway is 192.168.0.1 with mac address aa.bb.dd.ee.dd.ff, we probably don't ever need to look that up again.

That gets trickier with the use of mobile hotspots I suppose, but most networks aren't on roaming hotspots, so it seems silly to query every time just in case my house moved. There is also a standard way to detect that a network is a mobile hotspot. I've seen Android 'know' when the wifi its using is another phone's hotspot vs a stationary wifi network.

To lessen the load on geoip services, I would suggest we identify the discrete network we're on, and count the number of times a geoop service has given back the same answer for that network, and after 5 or 6 we save the 'persistant coordintes' of that network, and don't query for it again.

We could probably also identify the ASN we're on, and classify that ASN as a mobile ASN or a stationary ASN.

Another hiccup i've seen... My IPv6 service comes from Hurriane Electric's TunnelBroker service in my city's downtown. It should be local enough for the same weather. but it ip geolocates 250 miles away! I'm not trying to vpn to another place. but IPv6 tunnelbrokers 'emerge' to the public web in a different location sometimes. So geoclue might be a smarter way to determine location. And consider the possibility that a client's v4 and v6 addresses could geolocate to different locations.

[Gr3q]

Hi,

You should update the applet first because it's out of date.

I'll look into geoclue, but I don't know why https://get.geojs.io/ doesn't work for you

[BillyCroan]

I saw the problem on 3.3.1 as well, but i think its mostly on geojs.io's shoulders, other than it would be nice to have a little explanation of how "saved locations" works. i.e. if there is any saved location, does the applet just use that and no longer tries to geolocate? If so I think the Manual Location slider should cause the saved locations list to grey out when Manual location is turned off.

The way I got it working was:

1. I added a location in 'saved locations' with my lat/lon from google maps, and city, country, timezone filled out.
2. I didn't know what to put for 'Search Entry'. That field could use a tool-tip or note to explain it. So I just put the word "Home"
3. I switched Manual Location on
4. In the 'Location' box, I typed "Home". which is odd, because its tool tip implies its supposed to be a globally unique city name or address, and says nothing about checking against the saved locations list below (Another opportunity to update a tooltip)

re: geojs.io. it consistently 500's from my laptop at home over ipv6, and succeeds over ipv4. my v6 works everywhere else, its tunnelbroker from hurricane electric. From my phone on carrier data, geojs.io succeeds on ipv6. I'll track down the v6 error with geojs.io. Maybe they blocked Hurricane Electric. others are reporting similar problems: https://app.gitter.im/#/room/#jloh_geojs:gitter.im

GeoJS.io no longer maintained:
The contact info on geojs.io is bouncing. The response from the remote server was:
550 5.1.1 <contact@geojs.io>: Recipient address rejected: User unknown in virtual mailbox table
Also, their own status page is broken.... https://status.geojs.io/ So they might not be around much longer in case you want to start looking for an alternative.

re: GeoClue. Another benefit is that it multiple available methods of geolocation, not just ip geoloation. It will use gps receivers if available, and it will use cell tower data via 3/4g and gprs if available. network gps servers (NMEA from trucking and maritime environments). And if no direct devices provide gps coordinates, it can use the names and mac addresses of nearby wifi access points to figure out the location.

GeoClue is maintained by the gnome folks though it works fine in kde, cinnamon, whatever.
And GeoClue has pre-existin tooling to let the user turn on/off 'location services' all at once, or per app which is nice from a privacy respecting perspective.

[Gr3q]

the location entry box is for either searching for a location, or the contents match the "Search Entry" field on any of the Saved location it's going to use that.

When the applet is not breaking you can use the arrows to switch between them (or to your saved location), maybe I need to make sure they are accessible even when the applet is in an error state.

I looked into geoclue (meaning I implemented it). It only provides lat/long - which is enough - but the online solutions give me the timezone, city and country as well. So I will decide on which order I'm going to try it (or geoclue and online at the same time and only fail if I don't get anything)

[momodesuka]

Applet version/Build date
Version 3.3.1（2023-10-31 02:58:49）
Cinnamon version
6.0.0

I have encountered the same issue. got Service Error when I use VPN. if turn off the VPN, it works well!

[BillyCroan]

the location entry box is for either searching for a location, or the contents match the "Search Entry" field on any of the Saved location it's going to use that.

@Gr3q Can we get text stating that on the config window. even a tool-tip note would help. I don't think I'm an idiot but it was not obvious to me that the "Location" and "Search entry" were connected to each-other.

It might also help if the whole "Saved Locations" section was hidden unless you turn on 'Manual Location'

@endlessbeloved I suppose it comes down to a question of intent. Does the user want to see the weather of where their computer physically is at the moment, or where "their IP is".

I know for me, "where the computer physically is" is my intent. GeoGlue2 accomplishes that. Though it doesn't provide timezone and city. I'm sure there's other resources to connect a GPS coordinate to timezone and city though. Probably even ones that don't require network traffic.

Wait a sec. computers know their own timezone typically. cat /etc/timezone in Linux.

City is trickier and I'm not sure why the City name would be needed. But in many countries, there is a lot of areas that are not IN or part of a city. I live in city X, and a metropolitan region that goes by the name of its largest city, city Y. I tell locals I live in X, and anyone else I live in Y.

I have friends who live outside any city, 50-100 miles away from the nearest metro area, far enough to have different weather for sure. They don't have a city. "Nearest City" would be a better way to solicit city, but know it could be far enough to have different weather.

I would imagine that fetching the forecast is not based on a city name, but on GPS coordinates right? i.e. what dot is closest to my dot, read that forecast. That avoids the messiness of multiple city names Istanbul/Constantinople and dialect/accent/locale differences in how a city's name might be represented in text..

[Gr3q]

@Gr3q Can we get text stating that on the config window. even a tool-tip note would help. I don't think I'm an idiot but it was not obvious to me that the "Location" and "Search entry" were connected to each-other.

I should definitely do that, it is indeed unclear. The locations section is also a stock component so I did not have many options explaining but a text description should help.

It might also help if the whole "Saved Locations" section was hidden unless you turn on 'Manual Location'

You can technically switch to your saved locations by using the arrows on the applet, this is why it's not dynamically hidden. But I guess I think about it because those controls are not in the config.

@endlessbeloved I suppose it comes down to a question of intent. Does the user want to see the weather of where their computer physically is at the moment, or where "their IP is".

On automatic location the user's real coordinates are always preferred, just like when you use your phone. I simply didn't check if GJS has the capability to expose that before. When I have time to figure out to use it reliable and get all the error messages from I will make a release that uses it then falls back to geoIP services.

Wait a sec. computers know their own timezone typically. cat /etc/timezone in Linux.

I know your set timezone. It's not that same as knowing a set of coordinates' timezone, especially because some users can use more than 1 instance of the applet (using the applet not for their own location, but for some other location)

City is trickier and I'm not sure why the City name would be needed. But in many countries, there is a lot of areas that are not IN or part of a city. I live in city X, and a metropolitan region that goes by the name of its largest city, city Y. I tell locals I live in X, and anyone else I live in Y.

I have friends who live outside any city, 50-100 miles away from the nearest metro area, far enough to have different weather for sure. They don't have a city. "Nearest City" would be a better way to solicit city, but know it could be far enough to have different weather.

I'm not debating the usefulness/accuracy of the city/country info at all. Simply it was in the applet from the start so I have to keep it supported. There is a fallback where it displays the lat/long instead of that info so it will not break, just saying.

I would imagine that fetching the forecast is not based on a city name, but on GPS coordinates right? i.e. what dot is closest to my dot, read that forecast. That avoids the messiness of multiple city names Istanbul/Constantinople and dialect/accent/locale differences in how a city's name might be represented in text..

You are right the applet requests weather info purely on GPS coordinates. On auto, I get it from geoip services (currently), on manual, addresses get converted to coordinates via nominatim

[BillyCroan]

I thought worth sharing some gnome code I just came across. There's GeoClue and GeoCode apparently.

GeoClue is what I was blabbing about before and it can get your location by gps coordinate and an accuracy in meters.

GeoCode looks like it can lookup (given a GPS coordinate) what the closest city is to that gps coordinate without calling out to a web service.

https://discourse.gnome.org/t/commandline-tools-for-geocode/15051/2?u=jrj

I guess I am assuming that when possible and reliable, you'd like to reduce dependencies on web services. I know I would. Now if only gnome provided a libweather that predicted the weather without using any web services, that would be a trick!

[Gr3q]

I'm using geoclue as the primary source in the next version, but if you can recommend a better geoip service I can implement that as well.

These are already implemented:

• ipapi.com - this is what I started with, it had issues in certain countries
• geojs.io - this is what I use now
• geoiplookup.io - this is also implemented but I've never used it.

My problem is that I have no way to test (or I just don't know how) which one has the best coverage/reliability.

Technically I could call all of them at the same time or let the user choose between them...but out of the 2 I would prefer the former.