Skip to content

Commit 4e2658e

Browse files
NisitayNisitay
committed
Updated README
1 parent 5f6379f commit 4e2658e

File tree

4 files changed

+252
-67
lines changed

4 files changed

+252
-67
lines changed

‎README.md

Lines changed: 246 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,279 @@
1-
# pyp0f
1+
<h1 align="center">pyp0f</h1>
2+
<p align="center">
3+
<em>Native implementation of <strong>p0f v3</strong> in typed Python 3.</em>
4+
</p>
25

3-
Native implementation of ``p0f`` v3 in typed Python 3.
6+
---
47

5-
``pyp0f`` is able to accurately guess the source OS or user application of a given packet with passive fingerprinting.
8+
`pyp0f` is able to accurately guess the source OS or user application of a given packet with passive fingerprinting, as well as impersonate packets so that `p0f` will think it has been sent by a specific OS.
69

7-
#### Motivation
8-
- ``pyp0f`` is platform independent, while p0f can be cumbersome to run on some platforms (such as Windows).
9-
- ``pyp0f`` is mainly used as a library, as opposed to p0f which runs on a seperate process and you query the results using an API.
10-
- p0f depends on full packet flow details, while ``pyp0f`` attempts to use as little information as possible. For example, you can easily fingerprint one packet from a session without knowing the session history.
10+
## Motivation
11+
- `pyp0f` is platform independent (using [Scapy](https://scapy.net)), while `p0f` can be cumbersome to run on some platforms (such as Windows).
12+
- The implementation and concepts behind `p0f` are very sophisticated, but the tool is written in C which makes it harder to understand and extend.
13+
Performance is expected to be slower in Python, but `pyp0f` still performs well enough (see [Performance benchmarks](#performance-benchmarks))
14+
- `p0f` heavily depends on full packet flow details, while `pyp0f` attempts to use as little information as possible. For example, you may be able to fingerprint a SYN+ACK packet from a session without having the matching SYN packet.
15+
- `pyp0f` aims to be highly configurable and used as a library, without limiting its effectiveness to one packet format/library, as opposed to `p0f` which runs on a seperate process and you query the results using an API.
1116

1217
## Installation
1318

14-
```shell
15-
pip install pyp0f
19+
```console
20+
$ pip install pyp0f
1621
```
1722

1823
## Features
19-
- MTU fingerprinting
20-
- TCP fingerprinting
21-
- HTTP fingerprinting
24+
- [X] Full p0f fingerprinting (MTU, TCP, HTTP)
25+
- [X] p0f spoof - impersonation (MTU, TCP)
26+
- [ ] Flow tracking (In progress)
27+
- [ ] TCP uptime detection (In progress)
28+
- [ ] NAT detection (In progress)
2229

23-
## TODO
24-
- Flow tracking
25-
- TCP uptime detection
26-
- p0f tool loop
27-
- Impersonation tool
28-
- NAT detection
30+
## Docs
31+
* [Database configuration](#database-configuration)
32+
* [Fingerprinting](#fingerprinting)
33+
* [Impersonation](#impersonation)
34+
* [Real world examples](#real-world-examples)
35+
* [Sniff connection attempts](#sniff-connection-attempts)
36+
* [Block connections by certain OS fingerprint](#block-connections-by-certain-os-fingerprint)
37+
* [Spoof p0f with impersonation](#spoof-p0f-with-impersonation)
38+
* [Performance benchmarks](#performance-benchmarks)
2939

30-
## Usage
31-
pyp0f accepts SYN, SYN+ACK and HTTP packets. If the packet is invalid for fingerprint, ``pyp0f.exceptions.PacketError`` is raised.
40+
## Database Configuration
41+
Before using `pyp0f`, make sure to load the p0f signatures database.
3242

33-
### Database
34-
Before fingerprinting, make sure to load the p0f signatures database.
35-
36-
By default, the included (v3.09b) database will be loaded. However, you can specify a custom database path to
37-
parse.
43+
By default, the included (p0f v3.09b) database will be loaded. However, you can specify a custom database path to parse.
3844

3945
```python
4046
from pyp0f.database import DATABASE
4147

42-
DATABASE.load()
43-
# or DATABASE.load("path/to/database/file/p0f.fp")
48+
DATABASE.load() # or DATABASE.load("custom/database/file/p0f.fp")
49+
```
50+
51+
## Fingerprinting
52+
`pyp0f` accepts SYN, SYN+ACK and HTTP packets. Invalid packets raise `pyp0f.exceptions.PacketError`.
53+
54+
`pyp0f` makes sure to copy the packet before using it, to not modify the original accidentally.
55+
56+
Each fingerprint function returns a custom result instance which includes some informative fields that are typed appropriately.
57+
58+
<details markdown="1">
59+
<summary>MTU fingerprinting example</summary>
4460

45-
print(len(DATABASE)) # 322
61+
```python
62+
from scapy.layers.inet import IP, TCP
63+
from pyp0f.fingerprint import fingerprint_mtu
64+
from pyp0f.fingerprint.results import MTUResult
65+
66+
google_packet = IP() / TCP(options=[("MSS", 1430)])
67+
result: MTUResult = fingerprint_mtu(google_packet)
68+
print(result.packet_signature) # MTUPacketSignature(mtu=1470)
69+
print(result.match)
70+
# MTURecord(
71+
# label=MTULabel(name="Google"),
72+
# signature=MTUSignature(mtu=1470),
73+
# raw_signature="1470",
74+
# line_number=67,
75+
# )
4676
```
4777

48-
### Fingerprinting
78+
</details>
79+
80+
<details markdown="1">
81+
<summary>TCP fingerprinting example</summary>
4982

50-
pyp0f has 3 main functions:
5183
```python
52-
from pyp0f.fingerprint import fingerprint_mtu, fingerprint_tcp, fingerprint_http
84+
from scapy.layers.inet import IP, TCP
85+
from pyp0f.fingerprint import fingerprint_tcp
86+
from pyp0f.fingerprint.results import TCPResult
87+
88+
linux_packet = IP(tos=0x10, flags=0x02, ttl=58) / TCP(
89+
seq=1,
90+
window=29200,
91+
options=[("MSS", 1460), ("SAckOK", b""), ("Timestamp", (177816630, 0)), ("NOP", None), ("WScale", 7)],
92+
)
93+
result: TCPResult = fingerprint_tcp(linux_packet)
94+
print(result.distance) # 6
95+
print(result.match)
96+
# TCPMatch(
97+
# type=<TCPMatchType.EXACT: 1>,
98+
# record=TCPRecord(
99+
# label=Label(name='Linux', is_generic=False, os_class='unix', flavor='3.11 and newer', sys=()),
100+
# signature=TCPSignature(
101+
# ip_version=-1,
102+
# ip_options_length=0,
103+
# ttl=64,
104+
# is_bad_ttl=False,
105+
# window=WindowSignature(type=<WindowType.MSS: 4>, size=20, scale=7),
106+
# options=OptionsSignature(
107+
# layout=[<TCPOption.MSS: 2>, <TCPOption.SACKOK: 4>, <TCPOption.TS: 8>, <TCPOption.NOP: 1>, <TCPOption.WS: 3>],
108+
# mss=-1,
109+
# eol_padding_length=0
110+
# ),
111+
# payload_class=0,
112+
# quirks=<Quirk.NZ_ID|DF: 6>
113+
# ),
114+
# raw_signature='*:64:0:*:mss*20,7:mss,sok,ts,nop,ws:df,id+:0',
115+
# line_number=97
116+
# )
117+
# )
53118
```
54119

55-
Each fingerprint function returns a custom result object which includes some informative fields that are typed appropriately, such as:
56-
- The parsed packet
57-
- The calculated packet signature
58-
- The matched record, if any
120+
</details>
59121

60-
#### Examples
122+
<details markdown="1">
123+
<summary>HTTP fingerprinting example</summary>
61124

62125
```python
63-
from scapy.layers.inet import IP
126+
from pyp0f.fingerprint import fingerprint_http
127+
from pyp0f.fingerprint.results import HTTPResult
64128

65-
from pyp0f.fingerprint import fingerprint_mtu, fingerprint_tcp, fingerprint_http
129+
apache_payload = b"HTTP/1.1 200 OK\r\nDate: Fri, 10 Jun 2011 13:27:01 GMT\r\nServer: Apache\r\nLast-Modified: Thu, 09 Jun 2011 17:25:43 GMT\r\nExpires: Mon, 13 Jun 2011 17:25:43 GMT\r\nETag: 963D6BC0ED128283945AF1FB57899C9F3ABF50B3\r\nCache-Control: max-age=272921,public,no-transform,must-revalidate\r\nContent-Length: 491\r\nConnection: close\r\nContent-Type: application/ocsp-response\r\n\r\n"
130+
result: HTTPResult = fingerprint_http(apache_payload)
131+
print(result.dishonest) # False
132+
print(result.match)
133+
# HTTPRecord(
134+
# label=Label(
135+
# name="Apache",
136+
# is_generic=False,
137+
# os_class="!",
138+
# flavor="2.x",
139+
# sys=("@unix", "Windows"),
140+
# ),
141+
# signature=HTTPSignature(
142+
# version=1,
143+
# headers=[
144+
# SignatureHeader(
145+
# name=b"Date", lower_name=b"date", is_optional=False, value=None
146+
# ),
147+
# SignatureHeader(
148+
# name=b"Server", lower_name=b"server", is_optional=False, value=None
149+
# ),
150+
# ...
151+
# ],
152+
# expected_software=b"Apache",
153+
# absent_headers={b"keep-alive"},
154+
# ...
155+
# ),
156+
# raw_signature="1:Date,Server,?Last-Modified,?Accept-Ranges=[bytes],?Content-Length,?Connection=[close],?Transfer-Encoding=[chunked],Content-Type:Keep-Alive:Apache",
157+
# line_number=883,
158+
# )
159+
```
66160

67-
packet = IP(b'...')
68-
mtu_result = fingerprint_mtu(packet)
69-
tcp_result = fingerprint_tcp(packet)
70-
http_result = fingerprint_http(packet)
161+
</details>
71162

72-
print(mtu_result.match.label.name) # Ethernet or modem
73-
print(tcp_result.match.record.label.dump()) # s:win:Windows:7 or 8
74-
print(http_result.match.label.dump()) # s:!:nginx:1.x
163+
## Impersonation
164+
`pyp0f` provides functionality to modify Scapy packets so that `p0f` will think it has been sent by a specific OS.
165+
166+
Each impersonation method must be provided with a record label or signature to impersonate.
167+
168+
<details markdown="1">
169+
<summary>MTU impersonation example</summary>
170+
171+
```python
172+
from scapy.layers.inet import IP, TCP
173+
from pyp0f.impersonate import impersonate_mtu
174+
from pyp0f.fingerprint import fingerprint_mtu
175+
176+
impersonated_packet = impersonate_mtu(
177+
IP() / TCP(),
178+
raw_label="generic tunnel or VPN", # impersonate using a label
179+
raw_signature="1300", # or using a signature
180+
)
181+
result = fingerprint_mtu(impersonated_packet) # MTUResult for "generic tunnel or VPN"
75182
```
76183

184+
</details>
185+
186+
187+
<details markdown="1">
188+
<summary>TCP impersonation example</summary>
189+
190+
```python
191+
from scapy.layers.inet import IP, TCP
192+
from pyp0f.impersonate import impersonate_tcp
193+
from pyp0f.fingerprint import fingerprint_tcp
194+
195+
impersonated_packet = impersonate_tcp(
196+
IP() / TCP(),
197+
raw_label="s:unix:OpenVMS:7.x", # impersonate using a label
198+
raw_signature="4:64:0:1460:61440,0:mss,nop,ws::0", # or using a signature
199+
)
200+
result = fingerprint_tcp(impersonated_packet) # TCPResult for "s:unix:OpenVMS:7.x"
201+
```
202+
203+
</details>
204+
205+
## Real World Examples
206+
`pyp0f` can be used in real world scenarios, whether its to passivly fingerprint remote hosts,
207+
or to deceive remote `p0f`.
208+
209+
### Sniff connection attempts
210+
Using `scapy` to sniff incoming packets, we passively fingerprint remote hosts attempting to connect to our HTTP server.
211+
212+
See example [source code](./examples/fingerprint/scapy-sniff.py)
213+
214+
### Block connections by certain OS fingerprint
215+
With `pyp0f.fingerprint` we can block certain OS users attempting to connect to our HTTP server (in this example - Windows).
216+
217+
We use `pydivert` to capture incoming packets before they enter the network stack, fingerprint them, and drop every connection attempt to our server from Windows hosts.
218+
219+
See example [source code](./examples/fingerprint/block-os.py)
220+
221+
### Spoof p0f with impersonation
222+
With `pyp0f.impersonate` we can spoof running p0f by impersonating a certain OS.
223+
224+
In this case, I used Windows 10 with Ethernet, and impersonated Linux 2.2.x-3.x (barebone) with Google MTU.
225+
226+
[Browser leaks (TCP/IP Fingerprint section)](https://browserleaks.com/ip):
227+
<div align="center">
228+
<img src="./docs/images/browserleaks-normal.PNG">
229+
<img src="./docs/images/browserleaks-spoof.PNG">
230+
</div>
231+
232+
Local Ubuntu VM running p0f:
233+
<div align="center">
234+
<img src="./docs/images/p0f-normal.PNG">
235+
<img src="./docs/images/p0f-spoof.PNG">
236+
</div>
237+
238+
We use `pydivert` to capture packets before they leave the network stack, create a new packet that impersonates an OS, and finally re-inject the impersonated packet back to the network stack to spoof p0f on the other end.
239+
240+
See example [source code](./examples/impersonate/spoof-p0f.py)
241+
242+
## Performance benchmarks
243+
`pyp0f` includes a file to benchmark the main methods it provides - fingerprint, impersonate.
244+
The latest results were ran using an i5 7th gen.
245+
246+
```console
247+
Performance benchmark: Load Database
248+
Ran 1000 iterations in 9.76s
249+
Average time for one iteration of Load Database: 9ms
250+
-----------------------------------
251+
Performance benchmark: MTU Fingerprint (25 Packets)
252+
Ran 1000 iterations in 19.808s
253+
Average time for one iteration of MTU Fingerprint (25 Packets): 19ms
254+
-----------------------------------
255+
Performance benchmark: TCP Fingerprint (153 Packets)
256+
Ran 1000 iterations in 227.91s
257+
Average time for one iteration of TCP Fingerprint (153 Packets): 227ms
258+
-----------------------------------
259+
Performance benchmark: HTTP Fingerprint (3 Packets)
260+
Ran 1000 iterations in 0.15791s
261+
Average time for one iteration of HTTP Fingerprint (3 Packets): 0ms
262+
-----------------------------------
263+
Performance benchmark: MTU Impersonation (25 Signatures)
264+
Ran 1000 iterations in 4.5696s
265+
Average time for one iteration of MTU Impersonation (25 Signatures): 4ms
266+
-----------------------------------
267+
Performance benchmark: TCP Impersonation (153 Signatures)
268+
Ran 1000 iterations in 103.97s
269+
Average time for one iteration of TCP Impersonation (153 Signatures): 103ms
270+
```
271+
272+
See benchmark [source code](./scripts/benchmark.py)
273+
274+
## Sources
275+
- [p0f source code](https://github.com/p0f/p0f)
276+
- [Scapy docs & source code](https://scapy.net)
277+
77278
## Authors
78279
- **Itay Margolin** - [Nisitay](https://github.com/Nisitay)

‎docs/images/p0f-normal.PNG

13.4 KB
Loading

‎docs/images/p0f-spoof.PNG

11.7 KB
Loading

‎examples/fingerprint/scapy-sniff.py

Lines changed: 6 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -21,30 +21,17 @@ def handle_packet(packet: ScapyPacket) -> None:
2121

2222
# SYN/SYN+ACK packet, fingerprint
2323
if flags in (TCPFlag.SYN, TCPFlag.SYN | TCPFlag.ACK):
24-
try:
25-
mtu_result = fingerprint_mtu(packet)
26-
print(
27-
f"MTU fingerprint match: {mtu_result.match.label.dump() if mtu_result.match is not None else '???'}"
28-
)
29-
except PacketError as e:
30-
print(e)
31-
32-
try:
33-
tcp_result = fingerprint_tcp(packet)
34-
print(
35-
f"TCP fingerprint match: {tcp_result.match.record.label.dump() if tcp_result.match is not None else '???'}"
36-
)
37-
except PacketError as e:
38-
print(e)
24+
mtu_result = fingerprint_mtu(packet)
25+
tcp_result = fingerprint_tcp(packet)
26+
print(f"MTU fingerprint match: {mtu_result.match}")
27+
print(f"TCP fingerprint match: {tcp_result.match}")
3928

4029
payload = packet[ScapyTCP].payload
4130

4231
if payload:
4332
try:
4433
http_result = fingerprint_http(bytes(payload))
45-
print(
46-
f"HTTP fingerprint match: {http_result.match.label.dump() if http_result.match is not None else '???'}"
47-
)
34+
print(f"HTTP fingerprint match: {http_result.match}")
4835
except PacketError:
4936
print("Not an HTTP payload, skipping fingerprint")
5037

@@ -53,7 +40,4 @@ def handle_packet(packet: ScapyPacket) -> None:
5340
scapy_config.layers.filter([ScapyIPv4, ScapyIPv6, ScapyTCP])
5441

5542
# HTTP server was ran with 'python -m http.server 8080'
56-
sniff(
57-
filter="ip and tcp dst port 8080",
58-
prn=handle_packet,
59-
)
43+
sniff(filter="ip and tcp dst port 8080", prn=handle_packet)

0 commit comments

Comments
 (0)