Repository navigation
[SB] SAM Support (dGPU-detection, clipboard-detach handling) #93
Description
Activity
For this we need to implement support for the embedded controller. Some progress has been made in jakeday/linux-surface#286 (comment), but I can't give you any time-frame when (or even a guarantee that) this will be implemented.
I'll rename this issue so we can keep it open as tracking-issue for the other one.
- changed the title
[-]SB1 DGPU support[/-][+][SB] SAM Support (dGPU-detection, clipboard-detach handling)[/+]on Feb 24, 2020 Hi qzed!
Thank you for all your support.
I've been using the custom kernel for quite a while.Is there any progress that has been made since your last post on February about the dGPU?
Best,
Felipe
Unfortunately not. I've fixed the link in my comment above as that has been somehow broken. @kitakar5525 managed to power on the dGPU but at the moment that requires some not so clean patching as there are some driver conflicts IIRC.
Hi everyone!
Any news about the dGPU?
Thanks!
Also on SB1, please let me know how I can help with this issue
The thing missing right now is manpower. I.e. for the dGPU, someone would need to update the patch in jakeday/linux-surface#286 (comment) (so that it has a chance of being accepted upstream) and put together a driver that does the appropriate ACPI calls. For the other stuff, there probably needs to be a special HID driver that attaches to the HID-over-I2C device.
Is SB1 inherently tricky in how it does dGPU? I guess I'm surprised SB1 is the only one without dGPU atm. Any possibility of leveraging what's already been done for SB2 or others?
Been a while since I've done linux drivers but I'll start looking into it. Might take a while to figure out how @kitakar5525 got where he was
The fundamental parts are different than on the SB2. Specifically, it looks like the dGPU power is (at least in part) controlled by the embedded controller (SAM, in constrast to SB2 this is not SAM-over-SSH but SAM-over-HID). Even if it's not actually controlled by SAM, you need to set up communication with it as that otherwise leads to errors (jakeday/linux-surface#286 (comment)). That's what that one patch does: https://github.com/qzed/linux-surface-sam-hid/blob/master/patches/5.1/0001-Add-quirk-for-Surface-SAM-I2C-address-space.patch.
After that, calling the ACPI functions to turn on the dGPU should succeed. Ideally, you'd want to wrap those into a driver. Since they're also being called by the _ON and _OFF methods of the power region, you might get away with actually just using the standard PCIe power methods. Similar to https://github.com/torvalds/linux/blob/69119673bd50b176ded34032fadd41530fb5af21/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c#L1262-L1265 and https://github.com/torvalds/linux/blob/69119673bd50b176ded34032fadd41530fb5af21/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c#L1294-L1299.
Reacted by chiekkuHey @qzed,
I'm trying to reproduce your work via the ACPI calls and so far I'm not having any luck. I applied the i2c quirk patch to NixOS's 20.03 default kernel successfully, however it looks like my dGPU is not powering on with either of those ACPI calls. The double call toacall_vgbi_dgpureturns{0x00, 0x00, 0x00, 0x00}both times, and the other callacall "\_SB.PCI0.RP05.HGON"returns0x0called.Any idea where I'm going wrong with this? Maybe I have the wrong constants?
@richardcq Is there anything interesting in dmesg? How did you check the dGPU status? Looks like the ACPI call succeeded, so it should have at least sent the data to the EC. I don't own an SB1, so I haven't been able to test his myself. Maybe @kitakar5525 can help you more.
Here's the full run I did. When I tried the first time I did both the
lspcimethod and I checkedlshw -class displayas well to see if a second display popped up. I also don't have any/sys/module/acpi/parameters/debug_*, not sure why.$ acall(){ echo "$1" | sudo tee /proc/acpi/call >/dev/null && sudo cat /proc/acpi/call;echo;} $ VGBI_UUID_DGPU="{0x69 0x5C 0xD0 0x6F 0xE3 0xCD 0xF4 0x49 0x95 0xED 0xAB 0x16 0x65 0x49 0x80 0x35}" $ acall_vgbi_dgpu(){ acall "\_SB_.PCI0.LPCB.EC0_.VGBI._DSM $VGBI_UUID_DGPU 1 1 0";} $ acall "\_SB.PCI0.RP05.HGON" 0x0called $ lspci -nn | grep -i nvidia $ dmesg | tail -n 10 --- snip, random USB device messages --- $ acall_vgbi_dgpu {0x00, 0x00, 0x00, 0x00} $ lspci -nn | grep -i nvidia $ acall_vgbi_dgpu {0x00, 0x00, 0x00, 0x00} $ lspci -nn | grep -i nvidia $ dmesg | tail -n 10 --- same USB device messages --- [33698.222533] ACPI Warning: \_SB.PCI0.LPCB.EC0.VGBI._DSM: Argument #4 type mismatch - Found [Integer], ACPI requires [Package] (20190816/nsarguments-59)160 remaining items
In fact when I say that it beaks the per screen HiDPI detection I mean the following: I have indeed two monitors, one embedded (HiDPI) and an external without HiDPI. Without your systemd service, for instance, if I move firefox from one screen to another, it scale automatically ( Wayland... ), but with your systemd file, I do not have this behavior anymore.
After several hours of trial and error it is still purely random if my dGPU works or not. I also get the error "... has fallen off the bus" if it does not work and it does not depend on what "power/control" shows. Maybe the solution by @fematarazzo is not for Surface Book 1 or Ubuntu 20 (my setup). Could you please give my a link to the thread in the NVIDIA forums?
I think that it actually shouldn't be necessary to interface with the sysfs
dgpu_powerattribute at all. Runtime PM (as inpower/control) should theoretically be enough (but you might have to toggle that once in case the state doesn't match).So theoretically the
PRP5PowerResource is connected to the dGPU. Meaning if you request the device to be powered on in any way (load a driver, setpower/controltoon), it should turn on automatically since that's what ACPI is for. That works via the_ON/_OFFmethods of that power resource, which callHGON/HGOF. So that's (more or less) exactly what writing todgpu_powerdoes, just the "proper" way via the standard ACPI power management methods.If I remember correctly, however, the dGPU PCI device doesn't even show up at boot? That would make runtime PM inaccessible. Also I don't really know why that is...
Another thing I can't quite make sense of is the corresponding
_DSMfunction. It could be that that's somehow required for the initial setup.So you could maybe try:
- Blacklist any dGPU drivers (nvidia, nouveau, just use the long list here) for testing.
- Cold boot (no restart).
- If the dGPU is not visible, run
echo 1 | sudo tee /sys/bus/platform/devices/MSHW0041:00/dgpu_dsmcall. This should (hopefully) make the dGPU visible as PCI device. - Run
cat /sys/bus/pci/devices/<dgpu>/power/control, this should returnon. - Run
cat /sys/bus/pci/devices/<dgpu>/power_state, this should not returnD3cold. Might showunknown, so probably good if you tell us what it shows exactly. - Run
echo auto | sudo tee /sys/bus/pci/devices/<dgpu>/power/control. - Run
cat /sys/bus/pci/devices/<dgpu>/power_state, this should now returnD3cold. - Run
echo on | sudo tee /sys/bus/pci/devices/<dgpu>/power/control. - Run
cat /sys/bus/pci/devices/<dgpu>/power_state, this should returnD0,D1,D2, orD3hot.
If things still work after that you can un-blacklist the nvidia modules and try to load the driver and test the dGPU (e.g. run
nvidia-smi).Unfortunately dGPU is not visible after cold (or warm) boot. Furthermore, step 3 does not make it visible. So far only writing 1 to dgpu_power works.
However, writing 1 to dgpu_dsmcall after it is visible changes something:
- The error
...fall off the bus...stops. Note that during this error it seems the driver tries to load, fails, and starts from beginning, can be seen by executingps -ef | grep nvidia lsmod | grep nvidiagives, showing the driver was now able to load:
nvidia 35278848 1 drm 557056 12 drm_kms_helper,nvidia,i915,ttm- New error is shown again and again:
[ 678.310625] nvidia 0000:01:00.0: can't change power state from D3cold to D0 (config space inaccessible) [ 678.310673] NVRM: This is a 64-bit BAR mapped above 4GB by the system NVRM: BIOS or the Linux kernel, but the PCI bridge NVRM: immediately upstream of this GPU does not define NVRM: a matching prefetchable memory window. [ 678.310679] NVRM: This may be due to a known Linux kernel bug. Please NVRM: see the README section on 64-bit BARs for additional NVRM: information. [ 678.312172] nvidia: probe of 0000:01:00.0 failed with error -1 [ 678.312240] NVRM: The NVIDIA probe routine failed for 1 device(s). [ 678.312243] NVRM: None of the NVIDIA devices were initialized. [ 678.312748] nvidia-nvlink: Unregistered the Nvlink Core, major device number 506 [ 678.669751] nvidia-nvlink: Nvlink Core is being initialized, major device number 506- The error
This kinda indicates that the actual power state of the dGPU doesn't match with what the ACPI think it is/should be:
[ 678.310625] nvidia 0000:01:00.0: can't change power state from D3cold to D0 (config space inaccessible)Specifically, the message is displayed when config space access fails. That failure makes sense given that the device is in D3cold according to the kernel. D3cold means hard off, no power, so no access. To transition out of D3cold, an external (relative to the PCI subsystem) method needs to be used. That is normally ACPI (and should be in this case). However, due to calling around in "random" ACPI code with the
dgpu_poweranddgpu_dsmcallattributes, it's very likely that what the kernel thinks the ACPI state is vs. what the state actually (in PCI) is differs.So what I think happens here is: Driver tries to load, runtime PM kicks in and wants to turn on the card. ACPI subsystem says card is already on so nothing needs to be done on that side. PCI subsystem tries to enable card but fails because card is actually in D3cold and config space access fails.
So, once you get to this step, try toggling runtime PM. I.e. run
echo on | sudo tee /sys/bus/pci/devices/<dgpu>/power/control. That should hopefully cause the kernel to trigger a transition to the off/D3cold state. Since ACPI (probably) thinks it's on this should run the transition to turn it off on the ACPI side and do nothing on the PCI side (hopefully).Also you should blacklist any driver that can load against the device. Otherwise it will try to load as soon as it becomes visible and might mess with runtime PM. Better to try and do that after you have a stable device.
The
fallen off the busmessage is from the nvidia driver IIRC? Again, try to only load the driver after you've toggled runtime PM successfully.I've had another look at the DSM and there's a
RP05.ATCHfield. It seems that that guards some parts of the power transitions and could indicate whether the dGPU is attached (one) or not (zero). The field is initialized to zero and the only place it is changed seems to be in the DSM (here and here).So I guess that's supposed to be run when the base connection changes?
That would mean a bring-up sequence for the GPU should probably look like this:
- Run DSM call (
dgpu_dsmcall). This enables RPM transitions via ACPI. - Toggle runtime PM (
power/control) to synchronize between ACPI and PCI subsystems.
Unfortunately (as you've mentioned) for some reason the DSM call doesn't make the dGPU visible. I guess that'd mean that it's not recognized as "attached".
- Run DSM call (
You could maybe try poking at the ACPI values with the
acpi_callmodule (especially the\_SB.PCI0.RP05.ATCHfield).So, after a lot of trial and error now ;-) I believe I got it stable.
Seems that the script that was already mentioned here is the right one to turn it on:#!/bin/sh echo 1 | tee /sys/bus/platform/devices/MSHW0041:00/dgpu_power echo 1 > /sys/bus/pci/rescan setpci -H1 -s 01:00.0 6a.b=81 setpci -H1 -s 01:00.0 4.w=0407 echo 1 > /sys/bus/pci/rescan setpci -s 01:00.0 4.w setpci -s 01:00.0 6a.bIt was mentioned in this nvidia forum thread. However, reading further in this thread it seems that some kernel options are required as well, mentioned here and another one here. So to sum it up, my kernel options currently are
nouveau.modeset=0 pci=realloc nvidia-drm.modeset=1 pcie_port_pm=off pcie_aspm=off. I don't know what effects these options have but at least it has no negative side effect on the power consumption. According to powertop it can go down to 4W.pcie_port_pm=offshould prevent the card from going into D3cold. You might want to check temperatures of the card, especially if it gets warm to the touch when not used or during suspend. Could be that that's been improved with recent nvidia drivers, but that was (is?) a problem on my SB2.Ideally, we'd find a way to get this into the kernel in a way to make its runtime PM work (as we have for the SB2) and/or as a driver against the MSHW0041 device (which seems related to detaching/hot-plugging the dGPU), but that might not be easy (or work at all) due to some edge-case differences between how Windows and Linux handle runtime PM for PCI devices. That's pretty much impossible to do sanely the way the procedure looks now.
Something that may help debugging this further is enabling debug logs for ACPI PowerResource transitions via
echo "file drivers/acpi/power.c +p" | sudo tee /sys/kernel/debug/dynamic_debug/control echo "file drivers/pci/pci-acpi.c +p" | sudo tee /sys/kernel/debug/dynamic_debug/control echo "file drivers/pci/pci.c +p" | sudo tee /sys/kernel/debug/dynamic_debug/controlbefore running that script. Or better yet, enable that at boot via the kernel parameters
dyndbg="file drivers/acpi/power.c +p" dyndbg="file drivers/pci/pci-acpi.c +p" dyndbg="file drivers/pci/pci.c +p"This should give some messages like this one, which should give us a bit more insight into what's going on with ACPI and PCI, at least on the kernel side.
So far it seems that only SB1 are affected and as this thread is not that busy anymore the workaround using this start script (and the kernel parameters) seems to be somewhat satisfactory. Additionally, it is already an older device so there will be not many new user. I also don't notice a higher energy consumption in standby etc. So I'd suggest to put the workaround on the SB1 page in Wiki (I could do this) till someone has the time to dig deeper into that. What do you think?
So far it seems that only SB1 are affected
The SB2 and SB3 do use a (slightly) different mechanism that doesn't involve the embedded controller. There were some initial hick-ups with that as well, but we've been able to fix those in kernel. So the runtime PM mechanism does work on those devices. It's just the SB1 that doesn't want to play along (likely due to the EC being involved somehow I think).
So I'd suggest to put the workaround on the SB1 page in Wiki (I could do this) till someone has the time to dig deeper into that. What do you think?
Sure, feel free to do that.
So what's the workaround?
EDIT: just blind finally saw it here #93 (comment)
I have no idea what I'm doing wrong but I can't get the dgpu working on my SB1.
I've followed the workaround described above. However:
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | egrep "(OpenGL vendor|OpenGL renderer|OpenGL version)" OpenGL vendor string: Intel OpenGL renderer string: Mesa Intel(R) HD Graphics 520 (SKL GT2) OpenGL version string: 4.6 (Compatibility Profile) Mesa 22.0.5That's what is installed and running:
uname -a Linux Skye 5.18.12-surface #1 SMP PREEMPT_DYNAMIC Thu Jul 21 01:16:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux dpkg-query -l '*nvidia*' Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-========================-==========================-============-======================================== un libgldispatch0-nvidia <none> <none> (no description available) un libnvidia-encode1 <none> <none> (no description available) un nvidia-384 <none> <none> (no description available) un nvidia-390 <none> <none> (no description available) un nvidia-common <none> <none> (no description available) ii nvidia-dkms-515 515.65.01-0ubuntu0.22.04.1 amd64 NVIDIA DKMS package un nvidia-dkms-kernel <none> <none> (no description available) un nvidia-kernel-common <none> <none> (no description available) ii nvidia-kernel-common-515 515.65.01-0ubuntu0.22.04.1 amd64 Shared files used with the kernel module un nvidia-kernel-source <none> <none> (no description available) ii nvidia-kernel-source-515 515.65.01-0ubuntu0.22.04.1 amd64 NVIDIA kernel source package un nvidia-libopencl1-dev <none> <none> (no description available) ii nvidia-prime 0.8.17.1 all Tools to enable NVIDIA's PrimeRight after boot I have:
sudo cat /sys/bus/pci/devices/0000:01:00.0/power/control on sudo cat /sys/bus/pci/devices/0000:01:00.0/power_state D0The dgpu is recognized on the PCI bus:
lspci -k 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 08) Subsystem: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers Kernel driver in use: skl_uncore 00:02.0 VGA compatible controller: Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07) Subsystem: Microsoft Corporation Skylake GT2 [HD Graphics 520] Kernel driver in use: i915 Kernel modules: i915 00:05.0 Multimedia controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Imaging Unit (rev 01) Subsystem: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Imaging Unit Kernel driver in use: ipu3-imgu Kernel modules: ipu3_imgu 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model Subsystem: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model ..... 01:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 940MX] (rev a2) Subsystem: Microsoft Corporation GM108M [GeForce 940MX] Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia .....Obviously the nvidia driver is used, which is confirmed by:
sudo lshw -numeric -C display *-display description: VGA compatible controller product: Skylake GT2 [HD Graphics 520] [8086:1916] vendor: Intel Corporation [8086] physical id: 2 bus info: pci@0000:00:02.0 logical name: /dev/fb0 version: 07 width: 64 bits clock: 33MHz capabilities: pciexpress msi pm vga_controller bus_master cap_list rom fb configuration: depth=32 driver=i915 latency=0 resolution=3000,2000 resources: irq:144 memory:a0000000-a0ffffff memory:90000000-9fffffff ioport:8000(size=64) memory:c0000-dffff *-display description: 3D controller product: GM108M [GeForce 940MX] [10DE:134B] vendor: NVIDIA Corporation [10DE] physical id: 0 bus info: pci@0000:01:00.0 version: a2 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress bus_master cap_list rom configuration: driver=nvidia latency=0 resources: irq:146 memory:ba000000-baffffff memory:c0000000-cfffffff memory:a2000000-a3ffffff ioport:4000(size=128) memory:b9700000-b977ffffThe nvidia driver is loaded with the following messages in dmesg:
..... [ 20.878092] surfacebook1_dgpu_switch:sb1_dgpu_sw_hgon: turned-on dGPU via HGON [ 20.878987] pci 0000:01:00.0: [10de:134b] type 00 class 0x030200 [ 20.879021] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00ffffff] [ 20.879043] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0fffffff 64bit pref] [ 20.879065] pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff 64bit pref] [ 20.879078] pci 0000:01:00.0: reg 0x24: [io 0x0000-0x007f] [ 20.879091] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 20.879157] pci 0000:01:00.0: Enabling HDA controller [ 20.879315] pci 0000:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:1c.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) [ 20.880071] pci 0000:01:00.0: BAR 1: assigned [mem 0xc0000000-0xcfffffff 64bit pref] [ 20.880093] pci 0000:01:00.0: BAR 3: assigned [mem 0xa2000000-0xa3ffffff 64bit pref] [ 20.880110] pci 0000:01:00.0: BAR 0: assigned [mem 0xba000000-0xbaffffff] [ 20.880119] pci 0000:01:00.0: BAR 6: assigned [mem 0xb9700000-0xb977ffff pref] [ 20.880124] pci 0000:01:00.0: BAR 5: assigned [io 0x4000-0x407f] [ 21.959126] nvidia: module license 'NVIDIA' taints kernel. [ 21.959129] Disabling lock debugging due to kernel taint [ 21.990878] nvidia-nvlink: Nvlink Core is being initialized, major device number 505 [ 21.990885] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 515.65.01 Wed Jul 20 14:00:58 UTC 2022 [ 22.146494] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 515.65.01 Wed Jul 20 13:43:59 UTC 2022 [ 22.163106] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver [ 22.186235] ACPI Warning: \_SB.PCI0.RP05.PXSX._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20211217/nsarguments-61) [ 22.589006] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1 [ 22.750068] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint. [ 22.763687] nvidia-uvm: Loaded the UVM driver, major device number 503. .....So I really can't see what the problem could be.
Does anybody have an idea how to further debug this?
Stupid me! The nvidia driver package
nvidia-driver-515was missing.Interesting that nothing complained about it and none of the installed nvidia packages did have the driver as a prerequisite.
It simply didn't work.Sorry for the lenghty post with all the detail. Everything is fine now.
Well not really everything. Some of the gnome stuff under Wayland dosn't like the nvidia setting, but this is another story.i'm unable to correctly install the nvidia drivers when using the surface kernel, but unable to use the touchscreen when using generic. what do i do? am i fucked? do i have to choose whether i'm gaming or using the touchscreen on boot forever?
Hello, I'm just curious is DGPU support on SB1 is on the road map?
Excellent work on this!