Legacy gas limits prohibit Livepeer Payments in BYOC

[its-DeFine]

might be Relevant to the issue: #3744

Impact: High. Cannot process BYOC jobs into Livepeer.

Problem: When trying to sent Micropayments to a Bring your own container(BYOC) type of job using the Livepeer go implementation for a Gateway and Orchestrator the job request is not able to be processed. The Orchestrator and gateway containers show the following errors:

• Gateway Error:
  job_id= capability=<capability name> processing job request job_id= capability=<capability name> Failed to get token from Orchestrator [<orchestrator url>⁠](<orchestrator url>) err=<nil>

• Orchestrator error:
  could not get ticket params err=insufficient sender reserve but is think it s on your side now

Gateway Config

`services:
gateway:
image: livepeer/go-livepeer:latest
container_name: vtuber_network-gateway-test
volumes:
- ./data/gateway:/data
ports:
- 9999:9999
command: [
"-gateway",
"-orchAddr=,
"-orchSecret=${LIVEPEER_ORCH_SECRET:-orch-secret}",
"-httpAddr=gateway:9999",
"-httpIngest=true",
"-maxTicketEV=17000000000000",
"-maxTotalEV=17000000000000",
"-maxFaceValue=1000000000000000",
"-maxPricePerUnit=17000000000000",
"-network=arbitrum-one-mainnet",
"-ethUrl=https://arb1.arbitrum.io/rpc",
"-ethPassword=",
"-dataDir=/data",
"-v=6",
]
networks:
- default

networks:
default:
name: vtuber_network
external: true`

Orchestrator Config
`#############################################

Livepeer VTuber environment

#############################################

------------ Livepeer orchestrator (go-livepeer) ------------

LIVEPEER_NETWORK=arbitrum-one-mainnet
ORCHESTRATOR_PORT=9995
LIVEPEER_ORCH_SECRET=
ETH_RPC_URL=
ETH_PASSWORD_FILE=/root/.lpData/.ethpass # Mount config/ethpass with the keystore password
ETH_ADDRESS=
ETH_ORCH_ADDRESS=
PRICE_PER_UNIT=16000000000000
TICKET_EV=16000000000000
AUTO_ADJUST_PRICE=false
MAX_GAS_PRICE=1000000000

------------ Worker registration ------------

LIVEPEER_ORCH_URL=https://vtuber.ultimaratio.link:9995
LIVEPEER_ORCH_SKIP_VERIFY=true
ORCHESTRATOR_HOST=# public IP or hostname for serviceAddr, e.g. 65.188.126.223
CAPABILITY_NAME=Frank111 # IMPORTANT: Fill the capability name that you gave us in the form here!

CAPABILITY_PRICE_PER_UNIT=16000000000000

CAPABILITY_PRICE_PER_UNIT=0

------------ Pixel Streaming / Unreal ------------

VTUBER_SESSION_DIR=/home/ubuntu/vtuber_sessions
VTUBER_TCP_HOST=host.docker.internal
VTUBER_TCP_PORT=7777`

Livepeer request code:

` def create_livepeer_headers(self, capability_config: Dict[str, str]) -> Dict[str, str]:
"""Create proper Livepeer headers for gateway requests, including extra timeouts"""
job_header = base64.b64encode(json.dumps({
"request": json.dumps({"run": capability_config["run_command"]}),
"parameters": json.dumps({}),
"capability": capability_config["capability_name"],
"timeout_seconds": 1
}).encode()).decode()

    headers = {
        'Content-Type': 'application/json',
        'Livepeer': job_header,
        'Livepeer-Orch-Search-Timeout': '2s',
        'Livepeer-Orch-Search-Resp-Timeout': '1s'
    }
    return headers
    `

How our logic works:
Our script sends one payment request x seconds, you can think of this as Total Ticket EV = Payment for x seconds of hosting our app.

What is happening:
Legacy network perceives this as a "transcoding" request offered for a huge payment. From a logical perspective such edge cases where either prohibited from Go-Livepeer logic or if not programmatically prohibited should had received less rigorous testing. But could also be due to incorrect gas calculation from Legacy Arbitrum state

Suggestion for resolution:
As a temporary solution we can set a tested PRICE_PER_UNIT variable and multiply with y = timeout_seconds variableto reach TICKET_EV=16000000000000 then still route that job request as we would normally do. But this is not guarantee to work properly at all times if the issue is due to legacy gas calculation rather than hardcoded limits

[ad-astra-video]

There are a couple things that can help here: 1) allow Orchestrators to accept work if desired when insufficient sender reserve happens if the sender reserve can at least cover that ticket faceValue (see implementation where setting to 1 would achieve this), 2) get some better documentation of the sender reserve needing to be 0.36 ETH. This requirement can be lowered by reducing avgGasPrice.

See new test that illustrates how this works and that 0.36 ETH should enable better up time (PR #3748). Note there is also sender caching by the Orchestrator for up to 2 days. This was increased a while ago to help keep network operating when there are short periods of eth rpc outage (PR #2849). So there are two things that can be in play here that are competing: gateways want an updated max float when they increase reserve and Orchestrators want to cache the max float so that OrchestratorInfo responses can be fast.

All that said, @rickstaa is low reserve gateways a priority right now? 0.36 ETH is about $1,510 USD which can be recovered in about 2 days after removing from reserve. I think documentation on this will help.

[rickstaa]

All that said, @rickstaa is low reserve gateways a priority right now? 0.36 ETH is about $1,510 USD which can be recovered in about 2 days after removing from reserve. I think documentation on this will help.

@ad-astra-video thanks for the quick investigation! I agree that with the current ETH price the 0.36 ETH reserve requirement to prevent issues with gas spikes feels reasonable to me. I did update the documentation and in the meantime will look for an owner to adjust the average gas parameter (see #3744 (comment)).