Repository navigation
Surface Pro 11 (Intel): the infrared camera, face login, and a software-ISP path #2260
eidos-case
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Updated 2026-10-08. The Howdy reader at the end is new. With the infrared
driver going upstream, libcamera hands it raw monochrome frames, and the old one
read them as colour: Howdy found a face in none of them. And the rows of an
older frame that show in the infrared picture are an IPU7 bug, with a fix on
linux-media (Kernel, below).
Updated 2026-10-04. Much has moved since 31 August, and this page now
says so:
imx681(@AndreGilerson's) is on linux-media atv8, and
vd55g0is a separate driver of ours, in review with ST before itgoes to the list;
ov13858patches are accepted, and Surface Pro 11 (Intel, Lunar Lake) — status, and how to live with it today #2268 keepsthe current list;
data, and these files now carry only what was measured.
This is a companion to @AndreGilerson's #1990, not a replacement for it. Read
that first: it carries the
imx681andvd55g0drivers, and a full IPU7hardware-ISP setup for the two colour cameras --
ipu7-camera-hal.patchwithimx681-uf.json,ov13858-uf.jsonand both gcss binaries, bridged intov4l2loopback so ordinary applications just work. If you want the best picture
out of the front and rear cameras, that is the route.
What is here is the part that route does not cover, plus an alternative for the
part it does:
surface-camera-bridge.shtakesimx681-uforov13858-ufand exits on anything else; there is no vd55g0 init. GPIO 1 drives the emitter on this machine. For Patch enabling speakers, mic, camera, suspend and hibernate, and touchpad configuration for surface pro 11 intel. #1990's driver the patch
is in Patch enabling speakers, mic, camera, suspend and hibernate, and touchpad configuration for surface pro 11 intel. #1990's comments; for the driver going upstream, see the illuminator
under Kernel below.
rather than a better one: drivers on their way to mainline, against a real
ISP with Intel's own tuning that has to be rebuilt after kernel updates. If
you would rather not carry an out-of-tree stack, this works.
Tested on Ubuntu 26.04 with kernel 7.0.0-30-generic and libcamera 0.7.0, in
August; the infrared driver since on 7.3-rc kernels.
The hardware
All three hang off the IPU7's input system. The IPU7's processing system --
the hardware ISP -- is not supported by mainline, so libcamera's software ISP
does the work through the
simplepipeline handler.Kernel
The
imx681driver is @AndreGilerson's, and it is on linux-media now, at v8,with his review folded in. The
vd55g0driver going upstream is a separateone, in review with ST before it goes to the list. #1990 carries both in
their out-of-tree form. What follows is what we added around them; the current
state of each patch, with links, is in #2268.
ov13858: power the sensor before talking to it. The driver assumes therails and clock are already on when probe() reads the chip ID. That holds where
ACPI power resources do it, and not where an INT3472 companion registers them as
regulators, a clock and a reset GPIO for the sensor driver to consume. Without
this the rear camera does not probe at all.
On linux-media, v3,
[email protected], not merged yet.ov13858: cap analogue gain at 15.5x.V4L2_CID_ANALOGUE_GAINisadvertised as 0..0x1fff and is honest to 0x7ff. At 0x800 the gain halves, and
above that it wanders: 0x1000 behaves as 0x400 did, 0x1fff as 0x7ff. An
autoexposure loop that raises its request and gets back less signal cannot
converge. The cap goes to
0x07c0rather than 0x7ff, because that is whatov13b10 and ov08x40 use, it works out as 15.5x -- the usual ceiling for these
parts -- and it measures as the actual peak: 14.23x against 0x7ff's 13.71x.
On linux-media, v2: https://patch.msgid.link/[email protected].
vd55g0: drive the infrared illuminator on ACPI platforms. Which of thesensor's four GPIOs drives the illuminator comes from the
st,ledsproperty,which ACPI does not provide. For #1990's driver, a DMI match in the driver
supplies it -- GPIO 1 on this machine -- in a patch in #1990's comments; note
that the assignment must go in the ACPI branch of probe, because
vd55g0_parse_dt_gpios_array()is guarded bydev->of_node.For the driver going upstream, ST and the media maintainers agreed on another
place:
ipu-bridgesuppliesst,ledsfrom a DMI match, and the driver needs nochange. That patch follows the driver; a test version has been run here.
ipu-bridge: sensor entries.ov13858and the rear sensor's 180-degreerotation quirk are accepted, in
media/next. Theimx681entry goes with thatdriver's series; the
vd55g0one will follow its driver. The rear camera alsoneeded flip controls in
ov13858before libcamera could turn its pictureupright; those are accepted too.
int3472: support the POWER1 GPIO type. Needed for the rear camera's rails.Jakob Berg Jespersen's patch for this landed instead, in
media/next.staging/ipu7: snoop the input system's writes. The staging IPU7 driver hasthe input system write capture buffers without snooping the CPU's cache, and on
x86 nothing invalidates the cache before a buffer is read. So a program that
reads frames with the CPU, Howdy among them, can get rows of an older frame
wherever the scene moved. On the infrared camera here that was 301 frames of
698 at 644x604, and none with snooping on. The rear camera at full size shows
none either way: its 26 MB buffers do not survive in the cache. The ipu6
driver's IPU7 support, which is to take over from this one, leaves snooping off
too; that is untested here.
On linux-media, sent 2026-10-07: https://patch.msgid.link/[email protected].
libcamera
A sensor helper for the imx681. Without one libcamera takes the raw V4L2
gain code for the gain itself: the AGC reads an analogue gain of 1020, decides
the sensor is enormously over-amplified, and holds the exposure down. Mean luma
23.8 against 110.7 with the helper. Twelve lines:
Sent to libcamera-devel as
[email protected], and held atour request until the kernel driver lands: a helper should not come ahead of
its driver. Until it is in a release, rebuild the packaged version rather than
installing a self-built library over it:
Ubuntu 26.04 ships libcamera 0.7.0-1ubuntu2, which has no
CameraSensorHelperfor the Sony IMX681. Without one the software AGC reads the raw V4L2 gain code
as a gain multiplier and holds the exposure down.
This is a twelve-line addition. Rather than install a self-built libcamera over
the packaged one, rebuild the same version with the patch applied, which is
what a distribution backport does and what an archive update will cleanly
replace.
Verifying the rebuild took
The sensor helpers compile into
libipa, which is linked into the IPA modules,not into
libcamera.so. So check:Note the
ipa/directory in that path. Checking/usr/lib/x86_64-linux-gnu/libcamera/ipa_soft_simple.sofinds nothing, becausenothing is there -- and an empty result from a non-existent file reads exactly
like a missing patch.
The list of sensor names in
libcamera.soitself is a different database, thestatic sensor properties, and will not contain the helper.
The visible confirmation is that this warning stops appearing:
vd55g0still warns with #1990's driver, and that is expected -- there is nohelper for it, and its tuning omits Agc so libcamera never asks.
What the rebuild changes, honestly
The warning goes and the AGC's gain model becomes correct. The picture was
already usable: on a night scene with lamps in frame it measures 21-24 % clipped,
which sounds alarming and is a white ceiling under three lamps rendered as
white. Judge this camera by the frame, not the percentage.
A mild green cast remains on neutral surfaces, spread about 11. That is the
absence of a colour matrix, which was removed deliberately: this sensor's
subject is usually backlit, and a matrix turns clipped regions magenta until the
software ISP learns to desaturate highlights.
Tuning files. These go in
/usr/share/libcamera/ipa/simple/.Neither colour file carries a colour matrix. The matrices that worked here are
Intel's own tuning for these exact modules, read out of the
.aiqbfiles besidethe Windows driver, and these files carry only what was measured on this
machine. Earlier versions of the rear camera's file carried them, corrected for
libcamera's white balance; it no longer does, and without them its colour is
flatter.
For the front camera there is a second reason. Intel's matrices for it work,
and cannot survive a clipped frame: on a portrait against a window, 29 % of the
frame clipped turns 22 % of it magenta. A front camera's subject is usually
backlit, so that is the normal case rather than the edge case.
The infrared tuning omits
Agcentirely, so libcamera leaves exposure and gainwhere they are set. Face authentication cannot use a camera whose exposure
depends on what the previous capture left behind. That applies with #1990's
driver. With the driver going upstream libcamera passes the infrared frames
through unprocessed, and the file does not apply.
Face login
Howdy is not packaged for Ubuntu; build from source. It needs
libinih-devandlibevdev-dev, which nothing pulls in, and dlib compiled from pip.Howdy reads a camera with
cv2.VideoCaptureand a/dev/videoNpath, andneither half works here:
whatever format is requested.
the node against each camera's
SystemDevices, and all three cameras hereclaim all 32 nodes. The first always wins, and it is not the infrared one.
The way through is Howdy's own plugin seam. The reader at the end of this post
is built on the libcamera Python bindings, selected with
recording_plugin = libcameraanddevice_path = CAM3-- a camera id substringrather than a path. It also drives the illuminator, on at start and off in
release(), so a crash cannot leave the emitter lit.It reads whichever format libcamera gives it: XRGB8888 with #1990's driver,
which reports the sensor under Bayer codes, or raw R8 with the driver going
upstream, since libcamera 0.7 has no software ISP for a monochrome sensor. It
reads R10 too, untested. It checks which format it got, because
validate()replaces one without saying so, and for a raw format it asks for the full
644x604: at the default 320x240, which is binned, the IPU7 reports a packet too
long on nearly every frame.
Set the sensor exposure in
/etc/howdy/config.ini, in lines; 700 worked here,and the reader uses 400 if it is unset. Keep it under about 70 % of the frame --
1200 lines with the default frame of the driver going upstream -- where the
illuminator stays lit. Enrol several
models at varied poses: one model passed at certainty 3.4 against a 3.5
threshold, which is not a login, and four take it to 2.7. A different person
measures 6.5 and is refused.
PAM goes in
gdm-password, which serves both the login screen and GNOME's lockscreen, as
sufficientso a failure falls through to the password:Note
/etc/pam.d/sudomay be pointless on your machine: if your user hasNOPASSWDin sudoers, sudo never consults PAM for authentication at all.An open question: can the two paths coexist?
I do not know, and it matters to anyone who wants both good colour pictures and
face login on the same machine.
The infrared path here goes through the mainline
intel_ipu7_isysandlibcamera's simple pipeline handler.
intel/ipu7-driversbuildsCONFIG_VIDEO_INTEL_IPU7=m, the same module, so installing it replaces thein-tree one.
vd55g0is not among the sensor drivers it carries, so whether theinfrared camera still enumerates afterwards depends on whether the out-of-tree
input system presents an equivalent media graph. Plausible; untested here.
Upstream, the ipu6 driver is taking over the IPU7, so on newer kernels the
in-tree module is no longer
intel_ipu7_isys.The
50-disable-ipu7-isys.confin #1990 is not the obstacle -- it is a PipeWirerule that hides raw ISYS nodes from the device list, and the kernel module stays
loaded. I assumed otherwise at first and was wrong.
If someone has run both, that answer is worth more than the rest of this page.
Related work on IPU6
@dmanresa-saes' #2253 covers the Surface Pro 7+ on IPU6 and reaches the hardware
ISP with Microsoft's own
.aiqbcalibration. Two things there are directlyrelevant here even though the platform differs.
It reports the same thing about the infrared camera, found independently: the
"all black" frame is an unlit illuminator hanging off the sensor's strobe pads,
with nothing in ACPI describing it. Our DMI table naming GPIO 1 is one machine's
instance of that.
It also carries a decoder for the CPFF/
.aiqbcontainer, and a detail we didnot have: mainline ISYS delivers RAW10 LSB-aligned while the vendor tuning
assumes Windows' 12-bit alignment. If that holds on IPU7 as well, it would
explain the pre-scale the rear camera's matrices needed here, and it is worth
checking before anyone uses Intel's matrices on this machine.
Known gaps
The illuminator has a duty limit. In August it lit about seven sessions in
ten here, cause unknown. Measured since: it holds while the exposure stays
under about 70 % of the frame, fades just above that, and goes out at 72-75 %;
the board's LED driver, by the look of it. The sensor's own auto exposure goes
past that in the dark, so face login needs a fixed exposure below the limit,
which the Howdy reader sets. The sensor can also cap its own exposure
(
MAX_EXPOSURE_LINES); where that limit should come from is a question for ST.No highlight rolloff in the software ISP. Once a channel pins at maximum the
ratios between channels are false, and a linear colour matrix faithfully
amplifies the falsehood. This is what keeps the front camera's matrix switched
off. Adding rolloff would fix it for every sensor on the simple pipeline.
The infrared camera shares the front camera's name in camera applications.
The name comes from the camera's
Locationproperty, built fromV4L2_CID_CAMERA_ORIENTATION. #1990's driver never creates that control, sothere the camera falls through to the sensor model. The driver going upstream
reads the orientation from the firmware, as a sensor driver should, so the
camera is listed as "Internal front camera" -- the same as the colour one,
since both genuinely face front. The naming logic lives in the applications,
not the library:
cameraName()insrc/apps/cam/main.cpp, duplicated in qcamand again in PipeWire's libcamera plugin, which is what reaches a desktop
Camera application. Telling the two apart was put to libcamera-devel as an RFC,
[email protected], and narrowed there to appending themodel when two cameras share a location.
The rear camera lands about 24 % darker than Windows on the same scene
minutes apart. Its AGC pins at both ceilings -- 33.3 ms and the sensor's real
14x -- while Windows exposes the same scene at 30 ms and ISO 200. Windows lifts
midtones through a tone curve; libcamera multiplies linearly under a fixed gamma.
Things that cost time here
/dev/v4l-subdevNis not an identity. The number is assigned in probe orderand moves when the set of devices or their properties changes. Look the sensor
up under
/sys/class/video4linux/v4l-subdev*/name.systemctl rebootover SSH is silently blocked by GNOME's session inhibitor.Use
reboot -i, and checkuptime -srather than whether SSH answers.libcamera/ipa/ipa_soft_simple.so. Checkinglibcamera/ipa_soft_simple.sofinds nothing, and nothing from a file that doesnot exist reads exactly like a missing patch.
The tuning files
ov13858.yamlimx681.yamlvd55g0.yamlThe Howdy reader
It is also in #2267. Save it as
recorders/libcamera_reader.pybeside Howdy'sother recorders, and
add the
libcamerabranch to_create_reader()invideo_capture.py.All reactions