Skip to content

enhancement: implement getContext method for device emulation #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

tusharmctrl
Copy link

LocalBrowserProvider Configuration

This PR explains and have added the use-case on how to use the clientType setting when configuring a HyperAgent instance with the LocalBrowserProvider.

Usage

Use the clientType parameter to control whether the browser context should emulate a desktop or a mobile device.

const agent = new HyperAgent({
  llm: llm,
  browserProvider: "Local",
  clientType: "desktop", // Options: 'desktop' or 'mobile'
  debug: true,
});

Notes

  • Default value: "desktop"
  • This solution will work on both LocalProvider and HyperBrowserProvide.

@@ -45,6 +45,7 @@ export class HyperAgent<T extends BrowserProviders = "Local"> {
: LocalBrowserProvider;
private browserProviderType: T;
private actions: Array<AgentActionDefinition> = [...DEFAULT_ACTIONS];
private clientType: "desktop" | "mobile";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you take the types directly from the devices offered from the devices list ?
Something like private clientType?: keyof typeof devices;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the constructor too can you assign the clientType on the object instance ?
Like this.clientType = params.clientType

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - I tried this, but it wouldn't give actual suggestions to user - reason being, playwright doesnt do export on Devices type. So, considering user perspective - asking user to select mobile / desktop - would make more sense since we're not able to provide suggestions.


if (device === "mobile") {
const iPhone = devices["iPhone 12"];
return await this.session.newContext({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass in the entire device object, don't specify individual params here

 this.session.newContext({...devices[device]})

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that - that wouldn't work because, there's this property where sharp is contradicting.

When we pass the entire object, it will throw an error like:

Error: Image to composite must have same dimensions or smaller
    at Sharp.toBuffer (/home/tushar-rupani/missionctrl/HyperAgent/node_modules/sharp/lib/output.js:163:17)
    at compositeScreenshot (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/tools/agent.ts:38:6)
    at runAgentTask (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/tools/agent.ts:156:31)
    at HyperAgent.executeTask (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/index.ts:335:14)
    at <anonymous> (/home/tushar-rupani/missionctrl/HyperAgent/src/agent/test.ts:13:18)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason behind that is deviceScaleFactor

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And isMobile too.

public getSession() {
if (!this.session) {
return null;
}
return this.session;
}

public async getContext(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add this method on the abstract base class that they inherit from ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants