WebXR UX improvements

[elalish]

A nice discussion was started in #2381 about what the most important differences are between Scene Viewer and WebXR modes of AR on Android. WebXR has major advantages over Scene Viewer especially in terms of DOM customization, so we want people to use it. However, we also need to know where it comes up short so that we can fix any gaps. Here are a few comments from the other thread.

[elalish]

From @ukito-pl:

WebXR mode is very cool, gives a lot of extra features and possibilities. Generally it would be my "go to" technology regarding AR, but it is currently supported only on chrome and samsung browsers and doesn't work on IOS devices at all. Having extra features that are only available on specific devices/browsers is not a good idea and limiting a solution only to Android and Chrome is also not a good idea. Until Safari and other browsers implement WebXR I don't think I will use it in my cross-platform projects.

Regarding WebXR implementation:

• there is no user guidance, model just shows up in front of user and places on the floor when it finds it. But there is no prompt or anything saying that user should point the camera on the ground and move a phone from side to side to find a floor.
• no scale information when scaling model

[elalish]

We don't include user guidance by default because we want to keep the size of the library small and not be in charge of string localization. However, we give you the ar-status and ar-tracking attributes to help you make UI for this. We have a basic example here, but perhaps we should flesh it out some more.

What scale info would you like and for what use case? We have a snap at 100%, but honestly if scale matters it seems like ar-scale="fixed" is the more common desire.

[elalish]

As for platform inconsistency, I totally agree. Our only real hope is that Apple sees the light and implements WebXR. Still, even with Scene Viewer, it is inconsistent with Quick Look and even more so with the 3D experience on the webpage, at least if you use any of model-viewer's more advanced features. We look at WebXR as consistency between AR and 3D, with a Quick Look fallback on iOS.

[elalish]

From @ideathingdev:

WebXR is great, the only missing feature stopping me from using it is the possibility for the user to capture photos and videos as this is highly requested (and expected, unfortunately) by clients.

[elalish]

Yes, we've heard this a lot. We need a new web API for this, but it's being worked on. We'll implement it as soon as it's available.

[elalish]

From @maciekglowka:

I use WebXR as a default, BUT if a user opens link in an app (which is a lot of cases) than it uses Android WebView instead of Chrome - and then SceneViewer is the only option and a nice fallback. Otherwise my page would be broken (which it kind of is at the moment;).

[elalish]

Great! Yes, unfortunately WebXR is not coming to WebView (security concerns and such). However, we're working on getting it into WebLayer, which is the cool new version.

[milesgreen]

WebXR is definitely the way forward, as it opens so many more options and interactions. Apple will hopefully catch up before too long, and we know that WebXR work is progressing in WebKit, so hopefully will appear in Safari, one day.

As per the discussion over here #2279, the subject of the calibration process is a significant UX challenge. It's a process that needs to be intuitive for the uninitiated to easily understand. One of the considerations is what to show the user during the calibration process, which is a necessary part of the AR experience.

By comparison, here's what SceneViewer and AR QuickLook do during the calibration process:

• SceneViewer only shows interaction prompts during calibration process and no model. The model itself is only revealed once the object is placed / anchored to a point in the scene. From there it can be viewed, moved, scaled, etc.

• iOS AR QuickLook shows a small translucent preview of the model, locked front-on to the screen (like it's a 2D layer) along with interaction prompts to guide the user. Only once it's placed / anchored to a point does it then transition into 3D space.

My preference is to not display the 3D model until anchored. The reason being that in my observations, often, once a user sees a 3D object on screen (before it's anchored), they stop attempting to scan the environment, they focus on the object, which can lead to unexpected results - for example, the model is not being presented at accurate real-world scale as calibration has not been completed.

Still, I don't think the calibration process should be left empty, as that would be very unhelpful to the user. A great deal can be done with the DOM layers, especially combined with the ar-status and ar-tracking events. A combination of messaging and animated iconography can really help guide the user. The original Poster image could be used as a transparent overlay.

That said, the above approach of not displaying the model and showing more detailed DOM guidance can all be done today. You can hide the model by manually reaching into the ThreeJS scene and adjusting visibility. Which could be made even easier with the plan to support Show/Hide model parts. So for now, my use case is catered for.

Additional UX considerations that could help the calibration and placement process:

• Point Cloud Feedback. Consider displaying a point cloud of dots around the environment as features are detected. This gives user the idea that something has happened, and encourages them to scan further / wider to build up more points and fill in the blanks.

• Tap to Place Target. Consider rather than automatically dropping the 3D object as soon as the scene is tracked, instead display a target point, like a round circle, that sits flat on the plane, wherever the device is pointed. The user is then instructed to Tap to Place the object. This helps them place the object exactly where they want, when they are ready.

These are just my thoughts. Interested to know how others feel about it.

[elalish]

Point Cloud feedback is nice, but again, this not available in WebXR yet. Also, we've found that often by the time there are points in the point cloud, there is also a plane and the object has been placed on it. If it can't find a plane, it's because it also can't find many points.

The advantage of placing the object initially world-locked instead of our old screen-locked way is that for the user to see around the object, they have to move their phone (instead of just turning it). This is exactly the motion ARCore needs for scanning.

[milesgreen]

Thanks for the insights Emmett. Super interesting and helpful to understand all the thinking that has gone into this.

[maciekglowka]

The point cloud display have recently appeared in the Scene Viewer and I have to agree that it helps to communicate the callibration. Even if it is fake-ish it shows some kind of room analysis in progress.

I have been working with the AR for the last couple of months and what is also visible to me is the ever improving calibration time. I am usually testing on a cheapish Samsung M21 and in the good conditions I hardly have to move the phone around anymore. If it is almost instant, then the calibration process is not so much of an issue anymore.

As of the target point placement - it seems like a more convenient method to position the model, as nothing will obscure the room etc. However it requires an additional action from the user. So the user would have to notice and read some kind of message beforehand. For some applications it might be better, but for some might end up not working - the user will abandon the process (as would happen with a too long calibration). I am working on an e-commerce project, so the users would be casual. And in this case I believe if there are less steps there is a bigger chance that the user will actually display the model.

[milesgreen]

However it requires an additional action from the user

It's a great point. In general, the fewer clicks the better. It's a fine balance.

[AndresSalomon1990]

Love this post! I would like to comment a few UI/UX things I have seen in other AR environments in my short experience with webAR:

1. Totally agree that the user tap the screen and place the 3D model where he/she wants. Here you can show a cursor or something like that (maybe something more modern) that moves with the camera and whenever is clicked, it places the model.
2. Another could be to show instructions after placing the model on how to scale, rotate and move the model with intuitive animations like the hand in Scene Viewer that shows how to detect a surface. Also this could be done by steps: first scale like this, detect the event/interaction and then enable the rotation and so on.
3. A recenter button that can be used if I move the camera to other angle, once it's clicked, it moves the model in front of me.
4. Possibility to add audio to the models and their animations or environment audio. I don't know if this already can be done. Could be useful for instructions also.
5. Maybe this is too much, but the possibility to add more than 1 model to the AR scene. Like creating a more complex scene, as you can do with A-Frame.

Will comment here if I found more ideas 😋

Anyway, I really enjoy working with Model Viewer and AR! Thank you for your hard work! 💪

[david-rhodes]

1. Are there any plans to implement support and/or slots for title, share_text, and link parameters for feature parity with scene-viewer, as documented here: https://developers.google.com/ar/develop/java/scene-viewer?
2. Any plans to implement a "record" or "screenshot" functionality, as is available with scene-viewer?

As one potential problem, scene-viewer intents can be activated automatically without user interaction, but WebXR cannot due to security protocols. There's probably no solution for this, but it could be a blocker for some users.

[elalish]

We actually have a much more flexible API for 1 with WebXR than Scene Viewer or Quick Look has. Simply add any HTML you want as child nodes of the model-viewer element and they will show up in both 3D and AR. If you want to restrict them to AR only, you can scope the CSS using our ar-status attribute. Here's an example.

Yes, we're working on 2, but we're waiting on a new API that's being implemented in Chrome.

We have found most users will not accept the AR camera permissions (for either Scene Viewer or WebXR) if they haven't first seen the model in 3D. Just some UX food for thought.

[david-rhodes]

We actually have a much more flexible API for 1 with WebXR than Scene Viewer or Quick Look has. Simply add any HTML you want as child nodes of the model-viewer element and they will show up in both 3D and AR. If you want to restrict them to AR only, you can scope the CSS using our ar-status attribute. Here's an example.

Thank you for the example. Yes, I know we can add custom HTML elements, but scene-viewer has "defaults" for these features and WebxR does not. Convenience does have value, especially for smaller development teams and for things considered "feature parity." Passing in an attribute or parameter is simpler than writing and styling my own share functionality, for example.

We have found most users will not accept the AR camera permissions (for either Scene Viewer or WebXR) if they haven't first seen the model in 3D. Just some UX food for thought.

Thank you for this information--I did not know that. Do you have analytics that demonstrate this, or is it from internal UX testing?

[elalish]

It's from internal UX testing. It sounds like we should build a new example with HTML that matches the Scene Viewer default. Would that help?

[david-rhodes]

That would definitely help!

[maciekglowka]

Hi all again,

we were shooting some AR promo videos today and I had the occasion to see for a longer bit how other / new to subject people use webXR with model-viewer. So I have two more insights :) (it is not entirely new, it overlaps bit with the above)

• while I like that the 3d model appears immediately (before finding surface) it is confusing for the users. If you don't know how it works you don't notice the moment floor is found (the rectangle around the model is not enough). So the users try to move the model before it is properly placed and it gets frustrating. So, maybe the model preview before finding the surface should be semi-transparent, wireframe, white whatever ? I have to dig more into ar-status and try first to show a message 'you can now move and rotate' or smth though.
• I'd really like to have more events fired to measure in production how users are really interacting with AR. I have a GA event at the moment to check if they enter at all, but would love to see eg. if the users got to the point of actually moving and rotating the object :)

[felipe-chamas]

So, maybe the model preview before finding the surface should be semi-transparent, wireframe, white whatever ?

I really liked this idea!

[elalish]

Okay, I made a change to allow immediate movement and I think it's a significant improvement: #2606 Hopefully you agree and thank you all for the discussion! It'll be part of our next release, but in the meanwhile you can test it on modelviewer.dev. I opted to skip the wireframe and such because that's a lot more complicated and slower, and also because this is even less distracting.

[AndresSalomon1990]

Just tried it, awesome improvement! You rock @elalish !!!

[maciekglowka]

Okay, I made a change to allow immediate movement and I think it's a significant improvement: #2606 Hopefully you agree and thank you all for the discussion! It'll be part of our next release, but in the meanwhile you can test it on modelviewer.dev. I opted to skip the wireframe and such because that's a lot more complicated and slower, and also because this is even less distracting.

Ok. Thanks! I guess we have to make now big enough prompts to make it clear that the model is not properly placed yet :)

[elalish]

I'd like to direct everyone's attention to a PR that changes the behavior of two-finger rotation and scaling. I'm on the fence about merging it, so input would be appreciated! #2619

[hybridherbst]

Adding some thoughts here based on Twitter discussion regarding the general WebXR vs. system UX and the new UX example for AR.

1. I'd prefer if there's some solid UX "by default" and it can be turned off if really wanted - the standard astronaut doesn't have a failure prompt or similar.
2. A single type of "failure" might not be enough; Is it too dark? Not enough surface detail? Phone not moving? I'm not sure what WebXR gives here, maybe SceneViewer had the "more data" advantage in that regard.

Some tests (and things we're fighting with in client projects) to check if UX is good enough for AR stuff for people outside the tech bubble - all of these are based on real-world observations of how people desperately try to use AR experiences on their phones:

1. Place phone on desk, click "View in AR". If you've never used AR, what prompts can make you pick up the phone and look at some (ideally free) surface?
2. Lay in bed at night, phone pointing up, and click "View in AR". What lets you know what you're doing wrong?
3. Visit relatives that have these glass desks. Hold phone in hand while pointing at that desk, click "View in AR". (the model will never place while looking at that highly reflective/refractive thing). What to display?
4. Hold phone, click "View in AR", assume model places immediately, and don't move at all (a lot of people do this). How do you learn that you can move around that "photo" on your phone?
5. Click "View in AR", then without looking at your phone talk to someone for 10s, holding your phone randomly/idly. Now, look back at your phone. The model will be randomly placed somewhere (but placed), all you see is the camera feed. How do you find your model / how do you learn what to do?

[elalish]

These are all good points. I'm now leaning away from the concept of AR "failures" (some of which are on the user, and others of which are on ARCore) and instead trying to meet the user where they are. Hence now in MV the object is immediately placed in space and interact-able regardless of floor tracking. If a floor is found, it'll fall onto it, but if not, the user is not bombarded with how they're doing it wrong.

This is motivated partly by the above discussions, partly by my desire for simple UX, and partly because we don't ship strings (localization is intended to be external to MV). But it is certainly a trade-off. Your point 5 is interesting because this is a big and tricky UX problem for AR in general, especially for more story-focused cases. Basically, how do you get your user to move their body and phone to draw their attention to what you want them to care about when you can't control their camera? I suppose one simple answer might be to enable tap-to-place all the time (kind of like tap to recenter in our pan example). Still, discoverability remains tricky.

[felipe-chamas]

I want to propose a new possible problem with WebXR
Somehow I always felt that WebXR was slower than Scene-Viewer, but whenever I tried it on modelviewer.dev it felt fast enough.
A theory emerged between my coworkers, and we tried to prove it, but I don't know if we have enough evidence.
We believe a lot of heavy duty tasks from a website keep running when the WebXR experience is launched, unlike Scene-Viewer which is launched in another app. We recorded the DevTools Performance tab testing the model-viewer button on modelviewer.dev and on one of our client's websites.
Using the same model, on the customer website we feel the experience latency is higher to inputs and although the FPS feels the same, the FPS of the ThreeJS scene feels slower.
Also, not entirely sure if this is evidence, but comparing both experiences on performance tab summary (only while the experience is open) we can see the following charts:

On modelviewer.dev

On our customer website

We think it'd be nice to somehow keep the site "on hold" while the user interacts with WebXR. Maybe this is a bigger problem that affects all of immersive-web, maybe it's not a problem at all. But we would like to get some inputs on what you guys might think.

[elalish]

Well, the whole point of WebXR is that the page is able to keep running its scripts, which is why the experience is so customizable. The vast majority of that should be MV rendering the scene, same as in 3D mode. If the website is running a bunch of other heavy tasks concurrently, my first question is why? That seems like a much more important problem to solve, regardless of WebXR.

I'm also not quite sure what you're describing; are you referring to page load performance, model load performance, or FPS during interaction? I don't know what your 6,000ms charts are referring to. I'm also not sure how the FPS can feel both the same and much slower. Can you describe a bit more precisely?

[felipe-chamas]

That seems like a much more important problem to solve, regardless of WebXR.

Completely agree, but unfortunately it's not my slow website to fix. We make everything we can to deliver a small sdk, but it really depends on the website in the end.

are you referring to page load performance, model load performance, or FPS during interaction

I'm using performance as a way to try to prove the website is running on the background of the experience, which you just confirmed. Thanks

I'm also not sure how the FPS can feel both the same and much slower

I see it really got a little bit confusing. The camera movement doesn't seem to have a delay from the mobile moving to the screen feedback of the captured image. The 3D movement on the other hand, sometimes feel like there's a small delay on interactions and on camera movement. Those delays are higher and noticeable on heavy websites.

Thanks for the reply, @elalish . I understand the whole point of WebXR is that the page is able to keep running its scripts, we also need it for our own experience. I'm just thinking about the consequence this brings when comparing to SceneViewer. It feels like an issue that will be solved over time with more computational capable hardware, but in the meantime we are running A/B tests between WebXR/SceneViewer and this may be a reason our users usually leave a better feedback to SceneViewer.

[elalish]

Interesting; you're drawing a distinction between camera movement and 3D movement, which is interesting since in both cases the delay is in rendering the 3D scene. Are you comparing WebXR mode to Scene Viewer, or WebXR to regular 3D on the page? If you can post any screen recordings of what you mean that could help a lot. Also, any user feedback you have regarding WebXR vs Scene Viewer would be much appreciated.

[hybridherbst]

I think the point is that the non-isolation of model-viewer's WebXR mode is a performance issue compared to the isolated nature of a SceneViewer experience - in the latter, there's a guarantee that the device does the best it can to present the experience, in the former it's up to whatever the website does and model-viewer can't guarantee anything.

In extension, that means that services providers (e.g. us, R2U, ...) cannot guarantee that performance is good, even when to a customer it appears as if the "viewer" is taking over the entire page and thus is a new page in spirit. Thus, choosing WebXR over SceneViewer right now is a risk because of this coupling.

The question I'd see in here would be, is there a way that model-viewer can remedy that, or will there always be a need to build around it (e.g. actually make a new page with just model-viewer for the WebXR mode, ask implementors to attach to some callbacks themselves, ...) that require "cooperation with the website around" instead of the current way it "just works" when using SceneViewer.

I'm not sure if there are web technologies that already help with that, e.g. what happens when I press the "fullscreen" button on a html video? Does that do something different?

[elalish]

In fact the fullscreen API is exactly what WebXR uses, so the comparison to video is apt. And yes, the way to guarantee site isolation is by navigation, which is no more jarring than the switch to a native app. No, there's not much else we can do from our side, as this is how the web works.

[julienbfabre]

@elalish any update on the ability to make photos and videos when using WebXR? I know you said you were waiting on some Chrome API, is that moving forward?

Wouldn't it be possible to achieve that already without some specific API, like 8th Wall does it?

[SGECZ]

ar-scale="fixed" is what you mean? Does that only affect WebXR - not the model viewer?

I am asking cause we use it for machines and the real world size is kind of important and it would be nice to have optical feedback that you are really back at 100% after scaling. like it is now in sceneviewer/quicklook

[elalish]

Yes, any attribute that starts with ar only affects the AR modes (WebXR, SceneViewer, & QuickLook if possible). I would highly encourage any product use cases to set ar-scale="fixed".

[SGECZ]

thank you!

[butacm]

Hi, i would also need the scaling percentage because the user needs to be able to scale it down and than up again to its real size, and that's a specific request from our client because the machines need to be visualised in the real size but also having the possibility to scale them.

[elalish]

@butacm You might consider using ar-scale="fixed" and making your own UX with child DOM and setting scale instead to change the scale manually.

[jukibom]

We're still unable to move over to WebXR as a default, there are few things about the UX which aren't quite up to the same standard as Scene Viewer

• Latency of the object to the camera feed seems worse, though I suppose this will get less of an issue over time - it's mainly on older devices
• Moving a wall-mounted object to a different wall seems more difficult, it doesn't want to snap for some reason
• Objects can be accidently moved outside of view if something recalibrates during a drag action and there's no way for the user to recover other than restarting XR

[elalish]

That's quite helpful, thanks. Any chance you could file issues on these with screen recordings of WebXR and SceneViewer? The third in particular I'm surprised is any different between them.

[vincentfretin]

Add support for ar-placement="any" and ar-placement="both".

Currently we can set ar-placement="floor" and ar-placement="wall".
On Android WebXR it works fine. On iOS, model-viewer doesn't currently support wall (see #3989) but QuickLook do supports wall (vertical) and "any". The alignments "vertical" and "any" actually both snap to floor or wall when I tested it, it doesn't enforce the model to stay on the floor though.
I would like to have ar-placement="any" for Android WebXR and QuickLook so that a model can be snapped on both floor or wall, like in QuickLook.
That would be the first step, but I'm actually more interested by having a mode ar-placement="both" (but it would works only on Android WebXR) to snap on floor and wall. For example placing a fridge, you want it the closest on the wall and always on the floor, right?

Am I the only one wanting something like this? Does anyone already worked on a change for that? If you have any pointer where and how to modify the current model-viewer code to support it, I'll appreciate it.

[elalish]

PRs welcome! This is probably the most relevant part of the code: https://github.com/google/model-viewer/blob/master/packages/model-viewer/src/three-components/ARRenderer.ts#L509

As for other interest, it may have some relation to this draft PR: #4030