using Cisco's OpenDNS/Umbrella 1 Million to gauge

[jawz101]

http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

Take a look at my rationale on this Adaway commit:

AdAway/adaway.github.io@fe5414a

In addition to registration lookups, I also find it helpful to see if a DNS lookup is actively being used. If it's not in circulation then there is not much reason to block

[spirillen]

step 1 - http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

Downloaded the daily logs from August 1st, 2020 to October 13th, 2020 top 1 million DNS lookups for all of Cisco's OpenDNS/Umbrella users.

If an entry showed on any one of those days I kept it. Otherwise it was scrapped. The premise being if a DNS lookup is not popular enough to show on a massive DNS log file of that many user devices it is not worth trying to block it.

I would say, to make this real useful you should know if any at all have had a lookup for any of the domains. Some of them can/will be hidden lookup ie. through proxying or CNAME(DNAME)

What would be more interesting would be to collect Newly Observed Domain Tracking for classification.

How ever I do follow your though of cutting the list(s) to what is relevant for users.

Limitation to most used, well personally I disagree as the worst stuff is outside the top 1m.

[jawz101]

I would say, to make this real useful you should know if any at all have had a lookup for any of the domains.

I'm not sure I understand. The 1 million list is the most popular 1 million DNS lookups by all of Cisco's Umbrella customers.

I do like the idea of newly observed domain tracking. I wish I had the knowledge or tools to pull down their list each day and track the changes over time. If newly added domains appear for more than, say, 7 days- put them in a separate bucket for review.

Also, if I had some way to do statistical analysis of the other characteristics of the domain such as the naming of them (e.g. phrases such as loca*, *lytic, pixel, tag) and their whois registration (country, registrar, etc.) info to weight those that bear a resemblance to tracking.

[spirillen]

I would say, to make this real useful you should know if any at all have had a lookup for any of the domains.

I'm not sure I understand. The 1 million list is the most popular 1 million DNS lookups by all of Cisco's Umbrella customers.

If you would like to clear out data, that have not been used for x time, you would need some way to collect data about which rule have been applied to any users. Just clear out rules based on a third party's service is to me a bad idea. (MY view on why)

This would be data collecting and therefore anonymized and require you build some tool and not only a list.

I do like the idea of newly observed domain tracking

This can easily be setup, but it will requires than at bigger number of users would be using my DNS servers to be any useful collector. Posted a couple of examples here https://github.com/mypdns/NOD

Also, if I had some way to do statistical analysis.

What are you using to collect/find the information's you are writing your rules from? at first hands it sounds like some regex that needs to be written

[jawz101]

What are you using to collect/find the information's you are writing your rules from? at first hands it sounds like some regex that needs to be written

I collect DNS lookups mainly from my devices' NetGuard logs. Download crappy app, run it a while, pick and choose the questionable entries, block, see if it breaks the app, add to the list if it still works. There were a few times when I tried pulling in entries from other sources but I want it to be a list of things I've actually seen and know is just from mobile apps (that is, I try to limit including purely web page domains. There are a million blocklists for those).

I use quite a bit of regex but I'm talking about something a bit fancier. Something along the lines of taking a list with columns for domain name, registrar, certficate info (owner, etc.) and manually marking those I know are ads and trackers. Then, have some sort of statistical/machine learning model that characterizes the ad/trackers I've picked and creates its own decision on how it would classify a larger list of domains as good or bad.

I'm trying to avoid saying "machine learning" :P

[spirillen]

I'm trying to avoid saying "machine learning" :P

And how did that one goes 😆

Sidenote:
I follow you, you might be interesting in the Central SQL we are building, also in the yet hidden project https://www.mypdns.org/project/view/15/ | https://github.com/matrix-rocks which is al about categorizing domains for other to extract. And your thought of how to approach this sounds good to my ears.

[funilrys]

Reopening, because including such a dataset as a testing mode or comparison mode may be interesting in the future.

[spirillen]

Hey @jawz101

As I'm re-reading this issue, I'm thinking, Could you use UHBW: https://github.com/Ultimate-Hosts-Blacklist/whitelist/tree/script in conjunction with the top 1mil list?

[jawz101]

How so? Like to see if a blocklist is overblocking?

[spirillen]

reverse engineering 😏 if you like I can help make it spinning, it will likely be a bit clumsy and end up using grep in a for loop

#!/usr/bin/env bash

HOSTSFILE="/etc/hosts"
TOP1MILL="top1mil"
TESTFILE="test.file"


for l in "$TOP1MILL"
do
    grep "$l" "$HOSTSFILE" >> "$TESTFILE"
done

conda activate pyfunceble4

pyfunceble -a f "$TESTFILE"

rm "$TESTFILE"

Or something like that

[jawz101]

I do not understand

[spirillen]

You are extracting matches between the top1mill and source file and generates a new file to test and distribute

# A foor loop to extract lines from the top1million files
for l in "$TOP1MILL"
do
    grep "$l" "$HOSTSFILE" >> "$TESTFILE"
done

With the grep command you are reading lines and making a match to your own source (hosts) file, then for each match between the source (hosts) file and the top1million file will be outputted to a new file, this one could now be tested with pyfunceble and only contains records from the top1mill list. This also means you shouldn't need to do a test with PyFunceble, as the top1mill list only contains ACTIVE & VALID records.