Rewrite Webscreenshot Module

[TheTechromancer]

With the growing need for a full-browser helper, it may be worth it to revisit Playwright and see if we can improve our web screenshot capabiliy so that it's not reliant on gowitness.

We could also raise each of the network requests (including the rendered DOM) as their own HTTP_RESPONSE.

[Sh4d0wHunt3rX]

Not sure if this is related to Playwright, but that would be great if possible to add "Customizable automatic form filling" to bbot crawler like Katana has.

For example, here there is a xss: example.com/?s=

However, it needs bbot actually search something to get this, it will cool if bbot can fill the inputs such as logins, search, fields in general.

[TheTechromancer]

@liquidsec is working on a feature like this that tries common xss/sqli payloads in forms. This will effectively turn BBOT into a recursive web scanner.

[TheTechromancer]

Considering a rework of BBOT's web screenshot functionality. Here are the options as I see them:

Gowitness - Golang

• Pros:
  • Easy to install, supports all platforms/OS
• Cons:
  • Slow, unreliable

Playwright - Python

• Pros:
  • Stable, featureful
  • Supports python
• Cons:
  • Only supports Ubuntu, Debian (will not run on alpine/arch/redhat etc)
  • Very picky about browser installation
    • Wants to forcefully remove your existing installations

haylxon - Rust

Selenium - Java

• Pros:
  • Supports python
• Cons:
  • Java
  • Absolute pain to get working (chrome version + driver version must match)

[TheTechromancer]

I really wanted to use Playwright but the platform restriction is kind of a deal breaker. I wish there was a good low-level Python library for interacting with Chrome DevTools, but I couldn't find a Python project on par with golang's chromedp or rust's chromiumoxide.

However, there's another option, which is interacting directly with the Chrome DevTools API. It's actually pretty easy. Here's an example of how to take a screenshot:

import asyncio
import json
import websockets


"""
Step 1: launch chrome with:

    chromium --remote-debugging-port=9222 --headless
"""


async def capture_screenshot(url):
    # Connect to Chrome DevTools
    uri = "ws://127.0.0.1:9222/devtools/browser/ac22c802-1b4b-4edf-9818-f91166fedb44"
    async with websockets.connect(uri) as websocket:

        async def send_request(method, session=None, **params):
            to_send = {
                "id": 1,
                "method": method,
                "params": params
            }
            if session is not None:
                to_send["sessionId"] = session
            await websocket.send(json.dumps(to_send))

        # Create a new page/tab
        await send_request("Target.createTarget", url="about:blank")
        response = await websocket.recv()
        print(f"response 1: {response}")
        target_id = json.loads(response)['result']['targetId']

        # Attach to the page
        await send_request("Target.attachToTarget", targetId=target_id, flatten=True)
        response = await websocket.recv()
        print(f"response 2: {response}")
        session_id = json.loads(response)['params']['sessionId']

        # Enable the Page domain to receive events
        await send_request("Page.enable", session=session_id)

        # Navigate to the URL
        await send_request("Page.navigate", session=session_id, url=url)
        response = await websocket.recv()  # Wait for navigation to complete
        print(f"response 3: {response}")

        # Wait for the page to load completely
        while 1:
            message = json.loads(await websocket.recv())
            print(f"message: {message}")
            if message.get("method") == "Page.loadEventFired":
                break

        # Enable the Page domain to receive events
        await send_request("Page.disable", session=session_id)

        # Capture the screenshot
        await send_request("Page.captureScreenshot", session=session_id, format="png", quality=100)
        while 1:
            try:
                response = await websocket.recv()
                print(f"response 4: {response}")
                screenshot_data = json.loads(response)['result']['data']
                break
            except:
                pass

        # Close the page
        await send_request("Target.closeTarget", targetId=target_id)
        response = await websocket.recv()
        print(f"response 5: {response}")

        return screenshot_data

asyncio.get_event_loop().run_until_complete(capture_screenshot('http://example.com'))

[TheTechromancer]

Neat: https://github.com/lightpanda-io/browser