Native CoreAudio renderer with spatial audio #1399
Conversation
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
3 times, most recently
from
5d249f2 to
9c1b8ca
Compare
|
I've posted a build of this for anyone who would like to try it out. Be sure to read the release notes. |
|
Thank you Andy - came across this by accident today, super excited to see this in action - I'm on an M3 so hope to be able to test out and let you know my feedback. |
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
from
9c1b8ca to
c2c2dc4
Compare
|
Please try with the newest build from https://github.com/andygrundman/moonlight-qt/releases which fixes a bug with the surround channels. |
Worked beautifully! There are audio drop outs occasionally but not sure if related to this specific build (need to isolate in more testing), still there with low bitrates ('m running wired PC to Wifi 6E MacBook Pro which seems to have ample bandwidth even at 4K 120fps). Is there a way to enable disable on the fly to help me test? This is really great work Andy, thank you - I see your latest changes too for the MacOS end of the UI. Do you have a new release I can grab or happy to compile myself if you can include the steps involved (sorry not experienced on that side of things). |
Thanks! I've uploaded a 3rd build, which adds a few options in settings:
Note about head-tracking: This is not a properly signed build, and my understanding of Apple's security is that the two spatial audio features that require entitlements will be prevented from working when you install an unofficial build. If head-tracking works after you enable it, or if you see the text "Render mode: personalized spatial" in the overlay, please let me know. |
Just tried it out and head-tracking is working perfectly using AirPods Pro (Gen 2) and MacBook Pro M3. The only issue I'm having is infrequent audio drop outs but doesn't appear to be as a result of your build so I'll need to dig into that further - perhaps just wifi related albeit on 6E. I tested with some games (Mafia, Fort Solis as a couple examples), 7.1 test videos etc. The Mac is finally a killer portable gaming machine! Thank you Just retried with personalised spatial head-tracking and all good there also. |
Well that's great news, I guess this is a beta entitlement and maybe they'll lock it down later. Hopefully they realized it's a bit silly to lock out head-tracking, knowing someone's "head pose" doesn't really feel like sensitive user data.
Does it work better on ethernet? Does it work better if you only stream stereo audio? How about compared to the release version (7.1 should get downmixed by SDL in the release build) but use the same bitrate, I think. I am wondering if we need to give the user more control over the audio bitrate and the 5ms packet size. The problem with this is that anything along these lines would need a Sunshine patch as well. I'm hoping it's just a bug in the way I'm handling the audio somewhere. I admit most of my testing has been with ethernet, as I have some annoying wifi issues on my M1 Pro MBP as well as an M4 iPad using 6E, and have a hard time getting a smooth video stream without ethernet. It's the kind of thing that sometimes clears up if you disable Bluetooth or reboot, or there's always the chance it's my Unifi setup. The iPad is my only 6E device and it's real fast in a speedtest, but Moonlight video and audio both stutter a lot at any bitrate. |
|
I have ordered an ethernet adaptor so will see if that improves things and report back. Attached a small video in case it helps with the audio cutting out and the metrics showing: Interestingly after some fiddling it appears to go improve a little if I do not enable the "mute host speakers whilst streaming" option - actually scratch that, think was just chance. I dug this thread up where Steam Streaming speakers were suspected: https://www.reddit.com/r/cloudygamer/comments/i9otf3/audio_stuttering_with_moonlight/ |
ah I should add that I'm on the dev 15.1 Beta (24B5035e) |
|
Bit more digging here - ethernet does resolve so the audio stuttering is WiFi + MacOS related. Seeing lots of Mac related audio stuttering threads out there (e.g. #159) pointing to potential Mac services, none of the suggestions work for me.. its not bitrate/throughput related but I do see it occur when the blue overlay shows packets dropped due to resync >0% Any thoughts/ideas as to how to resolve would be much appreciated as the beauty of Moonlight for me is to be portable |
|
Thanks for implementing this feature. I am on the M1 Pro and it works with no issues with my AirPods Max. Head tracking also works. I am on wifi and I don't have any issues like audio stuttering. |
|
I have iPadOS 18 Public Beta installed on my iPad Pro. Will Spatial Audio along with head tracking work on it? How can I get access to the iPadOS Beta version of Moonlight? |
So far this is only for macOS, but iOS/tvOS is definitely the next task. |
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
2 times, most recently
from
32f576d to
aa85077
Compare
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
from
aa85077 to
2660e79
Compare
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
from
2660e79 to
7eb840b
Compare
NOTE: this patch depends on a patch to moonlight-common-c, see [this PR](moonlight-stream/moonlight-common-c#95). * Adds an audio stats overlay that works with all current renderers, showing common info such as bitrate and packet loss. It is blue and in the upper-right, and will appear whenever the video overlay is enabled. * Audio renderers are able to add more lines to the overlay (the upcoming CoreAudio patch uses this). * Added bitrate/FEC display to both video and audio stats. * Consolidated the 3 FPS lines into one to save a bit of space. * All time-based stats are now microsecond-based, improving accuracy of very fast events.
* Inspired by an example app from Apple. [1] * Needs an M1 or newer Mac. * Operates in a standard passthrough mode for stereo or when you have enough real channels (HDMI). * When headphones or built-in Macbook speakers are used and a 5.1 or 7.1 stream is being sent by Sunshine, this will render the surround channels into high quality spatial audio. * Supports optional head-tracking (enable in settings). * Supports personalized HRTF if you've scanned your ears with your iPhone. This can greatly improve spatial audio for many people. Known issues: * System sound menu does not indicate spatial audio is active or that multichannel audio is playing. * If Moonlight is in Game Mode and you toggle back and forth by say, swiping out of full screen to another app, the audio may stutter a bit. [1] https://developer.apple.com/documentation/audiotoolbox/generating_spatial_audio_from_a_multichannel_audio_stream
andygrundman
force-pushed
the
andyg.coreaudio-spatial-mixer
branch
from
7eb840b to
43c4581
Compare
|
Heads up. Merging this together with #1180 results in segfault when enabling the stats overlay. When I was initially testing 1180 I did so by merging the two PRs together and it worked fine but I noticed the author said that the enhancements on macOS could add ~6ms but I was only seeing a 1ms difference so I went about remerging and building. I must have done something wrong the first time around because subsequent builds caused a segfault and no matter the order in which I merged the code together, it segfaulted. The nature of how 1180 was kept current with master, by merging master into the branch makes it hard to rebase. I've had to merge these together by rebasing #1410 onto master then this one on top of that, then merge 1180 into the rebased 1399. Sorry, I have not found the time to troubleshoot the segfault |
Hi! I'm just going to drop this here, but it should be considered alpha quality and nowhere near ready to be merged. But it does work quite well for me and I'd like to get it out there for some feedback and more eyes/ears on it. I am pretty impressed at how good it sounds and the fact that CoreAudio doesn't even break a sweat doing spatial processing on 8 channels of PCM. It would prefer to take 12 channels actually, but that's a problem for another day...
Planned features:
[1] https://developer.apple.com/documentation/audiotoolbox/generating_spatial_audio_from_a_multichannel_audio_stream