[WIP] feat: sdk

[ycjcl868]

Summary

UI-TARS Agent SDK for different device's operations.

Examples

Desktop

import { GUIAgent } from '@ui-tars/core';
import { Device } from '@ui-tars/operator-nut-js';

const abortController = new AbortController();
const guiAgent = new GUIAgent({
  device: new Device(),
  // openai params
  model: {
    provider: 'openai_compatible',
    baseURL: 'https://<your_endpoints>.huggingface.cloud/v1/',
    apiKey: 'sk-proj-1234567890',
    model: 'UI-TARS-7B-SFT',
    headers: {},
  },
  signal?: abortController.signal,
  onData?: ({ data }) => {
    console.log(data);
  },
  onError?: ({ data, error }) => {
    console.error(error);
  },
  systemPrompt?: 'override system prompt',
});

await guiAgent.run('send "hello world" to x.com'); // instruction

Planning

const guiAgent = new GUIAgent({});

const planningList = await reasoningModel.invoke({
  conversations: [
    {
      role: 'user',
      content: 'buy a ticket from beijing to shanghai',
    }
  ]
})
/**
 * [
 *  'open chrome',
 *  'open trip.com',
 *  'click "search" button',
 *  'select "beijing" in "from" input',
 *  'select "shanghai" in "to" input',
 *  'click "search" button',
 * ]
 */

for (const planning of planningList) {
  await guiAgent.run(planning);
}

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 17.98%. Comparing base (e378581) to head (2cc4051).

Files with missing lines	Patch %	Lines
packages/operators/nut-js/src/index.ts	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   17.99%   17.98%   -0.02%     
==========================================
  Files          61       62       +1     
  Lines        1378     1379       +1     
  Branches      210      210              
==========================================
  Hits          248      248              
- Misses       1103     1104       +1     
  Partials       27       27              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.