Offline

[dicer]

Change-Id: I6371df8e092905e2585dfd30ffa93cdf491e4f6f

Added support for searching bssids in an offline database. Just like @n76 Local-GSM-Backend.
I integrated it into your backend because it kinda makes sense ;)
If you want me too, I can also publish this as my own backend.

[dicer]

One thing I just remembered: I have excessive debug logging in my code. Should I remove that or is it ok?

[n76]

This makes it visible by the standard Android launcher. If you leave this off it will only be accessible via the settings menu in NLP (which is what my latest changes to my GSM location back end do.

[mar-v-in]

👍 Once you add the meta-data BACKEND_SETTINGS_ACTIVITY (as you did) and mark the activity as exported using android:exported="true" (which you missed out) it will show up in the UnifiedNlp settings in the overflow menu next to the backend. There is no reason to show a feature in the launcher, the user merely uses.

[mar-v-in]

Nice work, thank you. Would be nice if you remove unnecessary debug logging, as on some devices logging causes excessive load.
Also, I'd like you to move Settings + PrefsFragment to the root package as it's not directly related to the local code.

[n76]

I now have a version of this based on the work of @dicer which:

1. Removes the backend from the launcher (access through NLP settings only).
2. Moves settings and PrefsFragment to the root package
3. Fixes the settings and backend so that you can change back and forth between on line and local lookups.
4. Adds a lat/lon bounding box option to the script that generates the database.

But I don't know the work flow well enough to submit it to you. I could issue pull requests to @dicer and if he likes them have him submit them to you. Or I could clone your repository and apply all of the changes (both @dicer and mine) and then give you a pull request. Or there is probably a work flow I don't know about that makes more sense.

[dicer]

Oh wow. Nice. Didn't know that switching between online and offline didn't work. I was pretty sure it did ;)

One thing I noticed in my tests so far: We really need the VerifyingWifiLocationCalculator @mar-v-in mentioned. Had several occasions where a mobile hotspot was traveling with me, but it had a fixed location in the database. This really badly screws up your location and I think this definitely needs to be fixed before we can merge this.

I wanted to respond earlier, but I'm super busy until the end of the year. So I can't really look into this. Maybe around the holidays but I can't promise anything.

[n76]

@dicer let me give you a pull request for what I have now.

The UI showed a switch between on and offline but logcat showed that it was staying on offline. I found that out when I messed up my data collection option and it resulted in the 7December dump not having any of the WiFi APs in my neighborhood. I tried switching to online and it failed to work for two reasons:

1. Because the back end did not actually change. Once fixed I found
2. OpenWLANMap is refusing my connection.

For what it is worth, the refused connection also happens to me if I use the base OpenWlanMapNlpBackend without your changes. I tried swapping out the libwlocate files and that resulted in a crash because the newer libwlocate attempts to do every type of location lookup and could not create a GPS listener in the thread environment we provide. I've put a bug report in on the the base microG version of this backend. If I get time, I will investigate this further. In the mean time the Apple Wifi backend works for me.

For what it is worth, the OpenWLANMap project only issues a new CSV dump every couple of weeks and the dump does not contain data from contributions that are not marked for sharing. The most recent settings you have for privacy are applied to all of your previous contributions which is what messed me up. There is no FAQ that lists that but I had a email exchange with them to understand this.

The MiniOpenWLANMap@Android app will collect and report hotspots with a "_nomap" suffix in the SSID. I believe that will cause the hotspot to be removed from their database dump.

For APs that don't have a "_nomap" suffix in their name, I think that if more than one AP is visible that if an AP is more than a plausible distance from the average position it should be removed from the calculation and the average location be recalculated. This should fix the one roving AP situation. It won't help though if, for example, all the APs on a cruise ship are in the database. Not sure what the solution to that would be other than to strongly suggest to the operator that they use the "_nomap" SSID naming convention supported by Google, Mozilla and OpenWLANMap.

[dicer]

I merged the changes by @n76 and fixed one commit. Also added a preference to en-/disable debug logging.

@mar-v-in I checked how the apple backend does the calculation. I like the idea of the classes, but the verifying seems to be a bit too much I think. Does it really help that much?
I'd also like to give the user the option to also resolve on only one found AP. That might prove helpful in rural locations where there is only one AP at a time. So I'll add a setting for this soon.

Before I do so I wanted to ask if we/I can merge the VerifyingWifiLocationCalculator into the API? I don't want to copy most of the code. I would make the database for caching optional (thus also disabling verifying) and add an option to allow classes of size 1 even if unverified.

What do you think?

[mar-v-in]

The idea of verifying WiFi locations came up as I wanted to provide a way to locate with a single visible WiFi without locating completely wrong, as I think it's better to be not located at all then being located wrong.

The verification now assures that the location of the WiFi used has been correct lately. It proofed to work very well for me (it happens quiet often, that the single visible WiFi was part of a class before).

However the need to verify depends on the data quality and up-to-dateness. It would be very cool if you could test if it works out fine for you to locate with a single WiFi (eg by tracking your movement through the city and afterwards checking whether there are unnatural jumps in your route)

[mar-v-in]

There is one feature I'd like to add to the verifying locater that is weighting locations. This would make it extremely generic and powerful as it could use signal quality for triangulation and thus provide high accuracy locations from WiFi's and cells. Together with an optional verification process this would become an ultimate utility class for locating based on what-ever-kind signal towers (even in-house locating using Bluetooth LE tokens) and I could imagine it to be put in the API code. Another interesting part that should be put in the API by the way is a version independent framework for retrieving visible WiFis and cell towers as this code is used in several backends (and should be of high quality to assure low battery usage)

[dicer]

Sorry I haven't worked on this for a while. Will start doing so in the next couple of days. Are the weighted locations in the API already?

[ghost]

Any news on this?

[dicer]

I have been using this all the time, but my tests show that it really needs some algorithm to sort out wrong data. Currently a single mobile hotspot will throw off the position by kilometers. And there are quite a few of those around. I haven't found the time to implement this and was hoping it would get solved in the API (some helper method). Until I or someone else solves this, I can't recommend to merge this just yet.

[ghost]

I appreciate the explanation. Is this a problem that is dealt with in a similar backend such as openBmap?

[dicer]

Yes, several backends implement their own way to filter bad data. That's why I didn't want to implement another but was rather hoping this would get moved to the api. Well, never followed this up nor did it myself. Anyone interested? My time is a bit limited at the moment.

[n76]

The way my standalone WiFi backend deals with it has several layers. First, it is sampling from the GPS and if the believed location of the AP is too far from the GPS location then the data for the AP is marked as unreliable (actually "moved and not yet stable"). Second, I borrowed the code @mar-v-in that groups APs based on distance to one another and choses the largest group, so a single outlier won't throw off the estimate. Finally, I require at least two APs to believe the location.

So far with about 6 months of use I haven't noticed a false location from my WiFi backend.

[dicer]

I remember the group-based algorithm. Should be fairly easy to refactor that into the API, right? We can define the parameter list pretty widely. I have been thinking a lot about this the last couple of months and would also like to take things like wifi strength into account. No need to implement that right away, but a comprehensive parameter list will allow us to implement this later. How about passing a data object which can be extended later?

[TjrGithub]

I'd love to use this backend offline. Is it being worked on?

[dicer]

I gave up on this a while ago since the data quality was just too bad. Several other backends have their own code for dropping obviously wrong locations, instead of putting them into the calculation. I always hoped an algorithm would be added to the API, but it never was.
If you want to, you can try and improve the location calculation. @n76 has some good code to only take values into consideration that make sense.

I for myself lost interest also because OWM is more or less defunct as it was split into several subprojects and not a lot of people are scanning anymore.
I'd rather live with less than with completely false data (and this happened LOTS of times) so I'm using https://github.com/n76/wifi_backend now