How to monitor external task/job progress via API?

[ChrisLynchHPE]

I'm using Pode to develop my own REST API to be a front-end to another job scheduler service, which brokers assignment of resources when a new job is scheduled. I use the Pode State feature to keep track of these assignments. Can anyone recommend a process to essentially perform a status poll of this external job scheduler? I've tried to use Pode Tasks, but that appears to be buggy and you have to define multiple tasks up front.

Here is my route code:

Add-PodeRoute -Method Post -Path '/api/jobs/checkOut' -ScriptBlock {

    $RequestBody = $WebEvent.Data

    if ($RequestBody.Count -eq 0 -or $null -eq $RequestBody) {

        Write-PodeJsonResponse -StatusCode 400 -Value @{ ErrorID = 'Invalid_Request_Body'; ErrorDescription = "The missing JobID and JobURL in request body." }

    }

    elseif (-not $RequestBody['JobID']) {

        Write-PodeJsonResponse -StatusCode 400 -Value @{ ErrorID = 'Invalid_Request_Body'; ErrorDescription = "The missing JobID in request body." }

    }

    elseif (-not $RequestBody['JobURL']) {

        Write-PodeJsonResponse -StatusCode 400 -Value @{ ErrorID = 'Invalid_Request_Body'; ErrorDescription = "The missing JobURL in request body." }

    }

    else {

        # again, ensure we're thread safe
        Lock-PodeObject -ScriptBlock {

            $FoundDuplicate = $state:CurrentJobs.GetEnumerator() | ? { $_['JobID'] -eq $RequestBody['JobID'] }

            if ($FoundDuplicate) {

                Write-PodeJsonResponse -StatusCode 409 -Value @{ ErrorID = 'Duplicate_JobID'; ErrorDescription = "JobID already exists." }

            }

            else {

                $state:CurrentJobs.GetEnumerator() | ? { $_.State -eq 'Stopped' } | Select -First 1 | Out-Host

                $Result = $state:CurrentJobs.GetEnumerator() | ? { $_.State -eq 'Stopped' } | Select -First 1

                if ($null -eq $Result) {

                    Write-PodeJsonResponse -StatusCode 406 -Value @{ ErrorID = 'No_Available_Slots'; ErrorDescription = "No available slots." }

                }

                else {

                    $Result.JobID = $RequestBody['JobID']
                    $Result.JobUrl = $RequestBody['JobURL']
                    $Result.State = "Running"

                    # Need to connect back to AppVeyors API to get the job start time
                    $JobStatus = Invoke-RestMethod -Uri $Result.JobUrl -Method Get -Headers ([AuthHeaders]::GetAuthHeaders())
                    $Result.Start = $JobStatus.build.started

                    # Will need to add logic here to validate the job is still running by using Invoke-RestMethod to $Result.JobUrl
                    Write-PodeJsonResponse -Value $Result

                    # Need to have Pode to continuously check the job status as a scheduled task
                    Invoke-PodeTask -Name 'MonitorJob' -ArgumentList @{ GroupID = $Result.Name }

                    Save-PodeState -Path $cache:PodeStateConfig

                }

            }

        }

    }

}

Here is the Pode Task code:

Add-PodeTask -Name "MonitorJob" -Timeout (60 * 120) -TimeoutFrom Start -ScriptBlock {

    param ([String] $GroupID)

    # Lookup the group assigned to the job from the caller to ensure it's being monitored
    $FoundJob = $state:CurrentJobs | ? Name -eq $GroupID

    if ($GroupID -eq 'Internal') {

        return $true

    }

    else {

        # Returns True if still running, False if not
        $JobStatus = Invoke-RestMethod -Uri $FoundJob.JobUrl -Method Get -Headers $state:Headers -ErrorAction SilentlyContinue

    }


    if ($JobStatus -and -not [String]::IsNullOrEmpty($FoundJob.JobUrl)) {

        $FoundJob.State = "Running"

    }

    # Likely lost track of the job
    elseif ($JobStatus -and [String]::IsNullOrEmpty($state:CurrentJobs[$j].JobUrl)) {

        $FoundJob.State = "Stopped"
        $FoundJob.JobID = $null

    }

    else {

        $FoundJob.State = "Stopped"
        $FoundJob.Appliances.Clear()
        $FoundJob.JobID = $null
        $FoundJob.JobUrl = $null

    }


    Save-PodeState -Path $cache:PodeStateConfig

}

And for completeness, here is the Pode State config:

{
  "CurrentJobs": {
    "Value": [
      {
        "Name": "Group1",
        "Appliances": [
          null,
          null,
          null,
          null
        ],
        "JobID": "",
        "JobUrl": "",
        "State": "Stopped",
        "Start": "0001-01-01T00:00:00",
        "End": "0001-01-01T00:00:00"
      },
      {
        "Name": "Group2",
        "Appliances": [
          null,
          null,
          null,
          null
        ],
        "JobID": "",
        "JobUrl": "",
        "State": "Stopped",
        "Start": "0001-01-01T00:00:00",
        "End": "0001-01-01T00:00:00"
      },
      {
        "Name": "Group3",
        "Appliances": [
          null,
          null,
          null,
          null
        ],
        "JobID": "",
        "JobUrl": "",
        "State": "Stopped",
        "Start": "0001-01-01T00:00:00",
        "End": "0001-01-01T00:00:00"
      },
      {
        "Name": "Group4",
        "Appliances": [
          null,
          null,
          null,
          null
        ],
        "JobID": "",
        "JobUrl": "",
        "State": "Stopped",
        "Start": "0001-01-01T00:00:00",
        "End": "0001-01-01T00:00:00"
      }
    ],
    "Scope": []
  }
}

Here is how the caller makes the POST call to this API:

Invoke-RestMethod -uri http://api.internal:8080/api/jobs/checkOut -Method POST -body @{JobID = "1.10.10"; JobURL= "https://external.api.to.monitor.com/api/projects/proj1/build/1.10.10"}

[ChrisLynchHPE]

I should also ask, does this scale well? How are these Pode Tasks treated internally? When you define a task in the Pode Server config, is it a single task, or is a unique task created each time the task is called?