-
Notifications
You must be signed in to change notification settings - Fork 8
Home
ngadmini edited this page Jun 14, 2023
·
143 revisions
- we use passwordless for ssh so that we don't have to hit password over and over again
- we use nano editor with the following profile in ~/.nanorc:
ngadimin@Q4OS:~/Documents/rpz_workdir$ less ~/.nanorc
set casesensitive
set tabsize 3
set titlecolor red,yellowstarting grab_http.sh v10.0 at Thu 2023-06-01 23:27:16 WIB
PREPARING TASKs: check availability configuration file isOK
PREPARING TASKs: check grab_http.sh is executed by non-root privileges isOK
PREPARING TASKs: check required debian-packages in local-host: 172.30.252.251 isOK
PREPARING TASKs: check script-pack's property in local-host: 172.30.252.251 isOK
PREPARING TASKs: check availability of sources-urls (as listed in grab_urls)
url_0: dsi.ut-capitole.fr/..?../adult.tar.gz isUP
url_1: dsi.ut-capitole.fr/..?../gambling.tar.gz isUP
url_2: dsi.ut-capitole.fr/..?../phishing.tar.gz isUP
url_3: dsi.ut-capitole.fr/..?../publicite.tar.gz isUP
url_4: dsi.ut-capitole.fr/..?../redirector.tar.gz isUP
url_5: dsi.ut-capitole.fr/..?../vpn.tar.gz isUP
url_6: blocklistproject.github.io/..?../porn-nl.txt isUP
url_7: trustpositif.kominfo.go.id/..?../domains_isp isUP
url_8: pgl.yoyo.org/..?../serverlist.php?hostformat=nohtml isUP
url_9: v.firebog.net/..?../Easylist.txt isUP
url_10: v.firebog.net/..?../AdguardDNS.txt isUP
url_11: s3.amazonaws.com/..?../simple_ad.txt isUP
url_12: raw.githubusercontent.com/..?../hostnames.txt isUP
url_13: urlhaus.abuse.ch/..?../rpz isUP
url_14: v.firebog.net/..?../Prigent-Malware.txt isUP
url_15: blocklistproject.github.io/..?../phishing-nl.txt isUP
url_16: blocklistproject.github.io/..?../malware-nl.txt isUP
url_17: blocklistproject.github.io/..?../ransomware-nl.txt isUP
url_18: raw.githubusercontent.com/..?../malware isUP
url_19: gist.githubusercontent.com/..?../raw isUP
------------------------------
All URLs of sources-url(s) isUP.
PERFORMING TASKs: initiating CATEGORY of domains
ADULT: dsi.ut-capitole.fr/..?../adult.tar.gz DONE
GAMBLING: dsi.ut-capitole.fr/..?../gambling.tar.gz DONE
PHISHING: dsi.ut-capitole.fr/..?../phishing.tar.gz DONE
PUBLICITE: dsi.ut-capitole.fr/..?../publicite.tar.gz DONE
REDIRECTOR: dsi.ut-capitole.fr/..?../redirector.tar.gz DONE
VPN: dsi.ut-capitole.fr/..?../vpn.tar.gz DONE
initiating: adult ipv4 malware publicite redirector trust+ (6 CATEGORIES)
processing TRUST+ CATEGORY with (2) additional sources-url(s)
grab_1: dsi.ut-capitole.fr/..?../gambling.tar.gz DONE
grab_7: trustpositif.kominfo.go.id/..?../domains_isp DONE
reducing: porn domains and move it's to ADULT CATEGORY DONE
fixing: bads, duplicates and false entries at TRUST+ DONE
moving: IP-address entries into IPV4 CATEGORY DONE
acquired: 914.946 entries.
processing ADULT CATEGORY with (2) additional sources-url(s)
grab_0: dsi.ut-capitole.fr/..?../adult.tar.gz DONE
grab_6: blocklistproject.github.io/..?../porn-nl.txt DONE
fixing: bads, duplicates and false entries at ADULT DONE
moving: IP-address entries into IPV4 CATEGORY DONE
acquired: 5.056.700 entries.
processing REDIRECTOR CATEGORY with (2) additional sources-url(s)
grab_4: dsi.ut-capitole.fr/..?../redirector.tar.gz DONE
grab_5: dsi.ut-capitole.fr/..?../vpn.tar.gz DONE
fixing: bads, duplicates and false entries at REDIRECTOR DONE
moving: IP-address entries into IPV4 CATEGORY DONE
acquired: 108.713 entries.
processing PUBLICITE CATEGORY with (5) additional sources-url(s)
grab_3: dsi.ut-capitole.fr/..?../publicite.tar.gz DONE
grab_8: pgl.yoyo.org/..?../serverlist.php?hostformat=nohtml DONE
grab_9: v.firebog.net/..?../Easylist.txt DONE
grab_10: v.firebog.net/..?../AdguardDNS.txt DONE
grab_11: s3.amazonaws.com/..?../simple_ad.txt DONE
fixing: bads, duplicates and false entries at PUBLICITE DONE
moving: IP-address entries into IPV4 CATEGORY DONE
acquired: 56.293 entries.
processing MALWARE CATEGORY with (8) additional sources-url(s)
grab_2: dsi.ut-capitole.fr/..?../phishing.tar.gz DONE
grab_12: raw.githubusercontent.com/..?../hostnames.txt DONE
grab_13: urlhaus.abuse.ch/..?../rpz DONE
grab_14: v.firebog.net/..?../Prigent-Malware.txt DONE
grab_15: blocklistproject.github.io/..?../phishing-nl.txt DONE
grab_16: blocklistproject.github.io/..?../malware-nl.txt DONE
grab_17: blocklistproject.github.io/..?../ransomware-nl.txt DONE
grab_18: raw.githubusercontent.com/..?../malware DONE
fixing: bads, duplicates and false entries at MALWARE DONE
moving: IP-address entries into IPV4 CATEGORY DONE
acquired: 867.965 entries.
processing IPV4 CATEGORY with (0) additional sources-url(s)
fixing: bads, duplicates and false entries at IPV4 DONE
acquired: 29.201 entries.
processing sources-urls (6 CATEGORIES) in summary:
adult: 5.056.700 entries
ipv4: 29.201 entries
malware: 867.965 entries
publicite: 56.293 entries
redirector: 108.713 entries
trust+: 922.130 entries
TOTAL: 7.041.002 entries
disk-usage: 163,214 Megabytes
[DONE] completed IN 2m:48s:840ms
[WARN] there are still remaining :
- duplicate entries across CATEGORIES
- sub-domains and it's parent-domain across CATEGORIES
- ipv4 that can be reduced to a CIDR-block
[HINT] consider continue to next TASKs
you have 4 options to proceed to the next task:
1) pruning ipv4, duplicate & sub-domain entries across CATEGORIES
2) option [1] and rewriting ipv4 & domain entries to RPZ format-entry [db.* files]
3) option [2] and incrementing serial zones [rpz.* files]
4) option [3] and syncronizing latest [rpz.* and db.*] files to rpz.warnet-ersa.net
WARNING: remote-host rpz.warnet-ersa.net will REBOOT due to low memory
use 'shutdown -c' at remote-host: rpz.warnet-ersa.net to abort
^C) use (Ctrl+C) to quit
please hit: (1|2|3|4) to continue OR (Ctrl+C) to quit
4
[4'th] TASK options chosen
CONTINUED to :
1) pruning ipv4, duplicate & sub-domain entries across CATEGORIES
2) rewriting ipv4 & domain entries to RPZ format-entry [db.* files]
3) incrementing serial zone-files [rpz.*]
4) syncronizing latest [rpz.* and db.*] files to 'rpz.warnet-ersa.net'
WARNING: remote-host rpz.warnet-ersa.net will REBOOT due to low memory
use 'shutdown -c' at remote-host: rpz.warnet-ersa.net to abort
PERFORMING TASKs: based on 4'th options ...
[1'st] TASKs:
starting grab_duplic.sh v10.0 at Thu 2023-06-01 23:30:12 WIB
[INFO] FOUND 6 CATEGORIES: adult ipv4 malware publicite redirector trust+
[PRUNE] duplicate entries across CATEGORIES
pruning duplicate entries based on ADULT
STEP 0.1 = pruning duplicates ipv4 entries SKIP
STEP 0.2 = pruning duplicates malware entries DONE
STEP 0.3 = pruning duplicates publicite entries DONE
STEP 0.4 = pruning duplicates redirector entries DONE
STEP 0.5 = pruning duplicates trust+ entries DONE
pruning duplicate entries based on IPV4 do nothing
pruning duplicate entries based on MALWARE
STEP 2.3 = pruning duplicates publicite entries DONE
STEP 2.4 = pruning duplicates redirector entries DONE
STEP 2.5 = pruning duplicates trust+ entries DONE
pruning duplicate entries based on PUBLICITE
STEP 3.4 = pruning duplicates redirector entries DONE
STEP 3.5 = pruning duplicates trust+ entries DONE
pruning duplicate entries based on REDIRECTOR
STEP 4.5 = pruning duplicates trust+ entries DONE
pruning duplicate entries based on TRUST+ do nothing
[PRUNE] IPV4 and sub-domains if parent domain exist across CATEGORIES DONE
[PRUNE] invalid Top Level Domains across CATEGORIES DONE
[PRUNE] turn-back pruned entries to related CATEGORIES
turn-back pruned sub-domains to ADULT category : 5.007.843 entries
turn-back pruned ipv4-addresses to IPV4 category : 17.420 entries
turn-back pruned sub-domains to MALWARE category : 348.279 entries
turn-back pruned sub-domains to PUBLICITE category : 45.308 entries
turn-back pruned sub-domains to REDIRECTOR category : 107.050 entries
turn-back pruned sub-domains to TRUST+ category : 826.721 entries
TOTAL : 6.352.621 entries
disk-usage : 150,095 Megabytes
[DONE] completed IN 1m:13s:555ms
[2'nd] TASKs:
starting grab_build.sh v10.0 at Thu 2023-06-01 23:31:25 WIB
PREPARING TASKs: check availability configuration file isOK
[INFO] splitting adult CATEGORY to 715.407 entries/sub-category AND
splitting trust+ CATEGORY to 413.361 entries/sub-category:
txt.adultaa txt.adultab txt.adultac txt.adultad txt.adultae txt.adultaf txt.adultag
txt.trust+aa txt.trust+ab
[INFO] rewriting all domains to RPZ format-entry
rewriting txt.adultaa to db.adultaa : 1.430.814 entries
rewriting txt.adultab to db.adultab : 1.430.814 entries
rewriting txt.adultac to db.adultac : 1.430.814 entries
rewriting txt.adultad to db.adultad : 1.430.814 entries
rewriting txt.adultae to db.adultae : 1.430.814 entries
rewriting txt.adultaf to db.adultaf : 1.430.814 entries
rewriting txt.adultag to db.adultag : 1.430.802 entries
rewriting txt.ipv4 to db.ipv4 : 17.420 entries
rewriting txt.malware to db.malware : 696.558 entries
rewriting txt.publicite to db.publicite : 90.616 entries
rewriting txt.redirector to db.redirector : 214.100 entries
rewriting txt.trust+aa to db.trust+aa : 826.722 entries
rewriting txt.trust+ab to db.trust+ab : 826.720 entries
TOTAL : 12.687.874 entries
disk-usage : 408,941 Megabytes
[DONE] completed IN 0m:8s:609ms
[3'th] TASKs:
starting grab_cereal.sh v10.0 at Thu 2023-06-01 23:31:33 WIB
[INFO] check availability configuration file isOK
[INFO] check availability zone-files: FOUND 13 zone-files isOK
[INFO] incrementing serial of zone-files:
rpz.adultaa rpz.adultab rpz.adultac rpz.adultad rpz.adultae rpz.adultaf rpz.adultag
rpz.ipv4 rpz.malware rpz.publicite rpz.redirector rpz.trust+aa rpz.trust+ab
[INFO] all serial of zone-files incremented to 2023060106
[DONE] completed IN 0m:0s:366ms
[4'th] TASKs:
starting grab_rsync.sh v10.0 at Thu 2023-06-01 23:31:34 WIB
[INFO] check availability configuration file isOK
[INFO] check availability: RPZ-dBase and zone-files in local-host: 172.30.252.251 isOK
[INFO] check availability remote-host: rpz.warnet-ersa.net isUP
[INFO] check passwordless ssh compatibility: rpz.warnet-ersa.net isOK
[INFO] check availability: /etc/bind/zones-rpz in rpz.warnet-ersa.net isOK
[INFO] check required debian-packages in rpz.warnet-ersa.net isOK
[INFO] archiving stale RPZ-dBase in rpz.warnet-ersa.net:/home/rpz-20230601-233135.tar.gz DONE
[INFO] find and remove old RPZ-dBase archive in rpz.warnet-ersa.net:/home DONE
[INFO] syncronizing the latest RPZ-dBase to rpz.warnet-ersa.net:/etc/bind/zones-rpz DONE
[WARN] remote-host: rpz.warnet-ersa.net has scheduled to reboot at 11:36:49 PM WIB
[HINT] use 'shutdown -c' at host: rpz.warnet-ersa.net to abort
[DONE] completed IN 0m:15s:425ms
TASKs based on 4'th options: DONE
bye!