curl -> curl-impersonate

DvaMishkiLapa · Apr 17, 2023 · ef91aea · ef91aea
1 parent b0f059e
commit ef91aea
Show file tree

Hide file tree

Showing 2 changed files with 31 additions and 10 deletions.
diff --git a/README.md b/README.md
@@ -5,18 +5,36 @@ Micro-library for data output from [sploitus.com](https://sploitus.com/).
 - [sploitus-assistant](#sploitus-assistant)
   - [1. Dependencies](#1-dependencies)
   - [2. Implementation details](#2-implementation-details)
-  - [3. Why curl?](#3-why-curl)
+  - [3. Quick guide](#3-quick-guide)
+  - [4. Why `curl-impersonate`?](#4-why-curl-impersonate)
 
 ## 1. Dependencies
 
-- [curl](https://curl.se/);
+- [curl-impersonate](https://github.com/lwthiker/curl-impersonate);
 - [Python 3.10 or newer](https://www.python.org/).
 
 ## 2. Implementation details
 
-Micro-library allows you to retrieve some targets from [sploitus.com](https://sploitus.com/). The query is done via **curl** using [threading](https://docs.python.org/3/library/threading.html).
+Micro-library allows you to retrieve some targets from [sploitus.com](https://sploitus.com/).
+The query is done via `curl-impersonate` using [threading](https://docs.python.org/3/library/threading.html).
 
-## 3. Why curl?
+## 3. Quick guide
+
+Lib parameters:
+
+- **`targets`**: targets for information as a `list` of `str`;
+- **`headers`**: headers for `http` request as a `dict`;
+- **`curl_cmd`**: version/wrapper used `curl-impersonate` (default: `curl_ff109`) **[*]**;
+- **`targets_type`**: info type for targets (`exploits` or `tools`);
+- **`sort`**: sort results (`default`, `date` or `score`);
+- **`title`**: hide or show titles (default: `False`);
+- **`offset`**: results offset (default: `0`);
+- **`sploitus_url`**: URL sploitus (default: `https://sploitus.com`);
+- **`semaphore`**: number of simultaneously running curl processes (default: `4`).
+
+**[*]** - After installing curl-impersonate, one way will be to specify the script or binary used in this parameter.
+
+## 4. Why `curl-impersonate`?
 
 **Sploitus** changed something in his work in 2022-2023. Most likely started working with [CloudFlare](https://www.cloudflare.com/).
 You can tell from other projects that apparently worked before:
@@ -25,6 +43,5 @@ You can tell from other projects that apparently worked before:
 - [sploitGET](https://github.com/0xricksanchez/sploitGET).
 
 There were attempts to communicate via [requests](https://requests.readthedocs.io/en/latest/), but instead of a response I got a blank page.
-It is likely that the **curl** approach may even stop working.
 
-You could try using [libcurl](https://curl.se/libcurl/) via [PycURL](https://pypi.org/project/pycurl/#files), but I'm too old for that.
+Until recently, a simple approach using classical [curl](https://curl.se/) worked. But that approach stopped working too.
diff --git a/sploitus_assistant.py b/sploitus_assistant.py
@@ -11,10 +11,10 @@
     # 'accept-language': 'en-US,en;q=0.9,ru-RU;q=0.8,ru;q=0.7',
     'content-type': 'application/json',
     # 'dnt': 1,
-    # 'origin': 'https://sploitus.com',
+    'origin': 'https://sploitus.com',
     # 'referer': 'https://sploitus.com/?query=Moodle',
     # 'sec-ch-ua': '"Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"',
-    # 'sec-ch-ua-mobile': '?0',
+    # 'sec-ch-ua-mobile': '?0',/
     # 'sec-ch-ua-platform': '"Windows"',
     # 'sec-fetch-dest': 'empty',
     # 'sec-fetch-mode': 'cors',
@@ -28,6 +28,7 @@ class SploitusAssistant:
     Class work with [sploitus.com](https://sploitus.com/)
     `targets`: targets for information
     `headers`: headers for `http` request
+    `curl_cmd`: version/wrapper used [curl-impersonate](https://github.com/lwthiker/curl-impersonate) (default: `curl_ff109`)
     `targets_type`: info type for targets (`exploits` or `tools`)
     `sort`: sort results (`default`, `date` or `score`)
     `title`: hide or show titles (default: `False`)
@@ -39,6 +40,7 @@ def __init__(
             self,
             targets: List[str],
             headers: Dict[str, Any],
+            curl_cmd: str = 'curl_ff109',
             targets_type: str = 'exploits',
             sort: str = 'default',
             title: bool = False,
@@ -48,6 +50,7 @@ def __init__(
     ) -> None:
         self.targets = targets
         self.headers_dict = headers
+        self.curl_cmd = curl_cmd
         self.type = targets_type
         self.sort = sort
         self.title = title
@@ -64,7 +67,8 @@ def __run_curl_sploitus(self, target: str, output: list) -> None:
         '''
         self.headers_dict.update({'referer': f'{self.sploitus_url}/?query={target}'})
         headers_for_curl = ' '.join([f"-H '{k}: {v}'" for k, v in self.headers_dict.items()])
-        cmd = "curl -s '{search_url}/search' {headers} --data-raw '{data}' --compressed".format(
+        cmd = "{curl_cmd} -s '{search_url}/search' {headers} --data-raw '{data}' --compressed".format(
+            curl_cmd=self.curl_cmd,
             search_url=self.sploitus_url,
             headers=headers_for_curl,
             data=dumps(
@@ -132,7 +136,7 @@ def __scan(self) -> Dict[str, dict]:
         ],
         headers=headers,
         targets_type='exploits',
-        sort='default',
+        sort='score',
         title=False,
         offset=0,
         semaphore=8