Interest in GSoC 2026: Windows 8/10 GUI agent

[Νίκος Τσούλος]

Hello, my name is Nick, and I’m a 3rd year CS student in the Department of Informatics & Telecommunications at NKUA (Greece) with an interest in Windows kernel and internals. The 2026 GSoC Windows GUI agent project caught my eye. I've been browsing the relevant code and trying to create a high level overview of the project, while grasping a few technical points. 

1) The project description states that "some progress has been made to create a full WDDM driver... but the driver isn’t working correctly yet." Is this work-in-progress available in a specific branch or repository that I have missed?

2) From the work done so far, what have been the main caveats/blockers/non-trivial parts in practice?

3) Would a display-only WDDM driver completely remove the need for Desktop Duplication API you currently rely on for the capture path?

Thank you for your time.

[Rafał Wojdyła]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 3/6/26 13:51, Νίκος Τσούλος wrote:
> Hello, my name is Nick, and I’m a 3rd year CS student in the Department
> of Informatics & Telecommunications at NKUA (Greece) with an interest in
> Windows kernel and internals. The 2026 GSoC Windows GUI agent project
> caught my eye. I've been browsing the relevant code and trying to create
> a high level overview of the project, while grasping a few technical
> points.

Hello!

> 1) The project description states that "some progress has been made to
> create a full WDDM driver... but the driver isn’t working correctly
> yet." Is this work-in-progress available in a specific branch or
> repository that I have missed?
>
> 2) From the work done so far, what have been the main caveats/blockers/
> non-trivial parts in practice?

Currently we have no usable WDDM code. I was researching the subject
several years ago but wasn't able to proceed then due to poor documentation
at the time and the complexity of the WDDM API. The documentation state
definitely improved since then, and there is some other prior work
to look at.

> 3) Would a display-only WDDM driver completely remove the need for
> Desktop Duplication API you currently rely on for the capture path?

Display-only driver should be enough for the case of an emulated GPU
(which we use now). It would let us manage the render buffer and get
display surface damage areas directly without using the DirectX Desktop
Duplication APIs.

I'll mention other potential avenues for improving the GUI agent:

1. Improve the performance of the current display capture solution by using
a ring buffer for frame data so the capture thread doesn't play event
ping-pong with the main thread.
2. Improve/rewrite the window tracking logic. Display capture is only
one part of the equation, we're also tracking all changes to visible
windows (display driver knows nothing about that). Currently this is
done via polling, which is really inefficient. IIRC the reason for that
was because notifications via windows hooks were unreliable: they need
DLL injection under the hood which is rather problematic (32bit vs 64bit
processes, different security contexts for things like the login
screen/winlogon etc).
3. Add a visual capture debug "log" capable of saving window snapshots
along with their metadata. Something like that would be really useful
for debugging the window tracking logic.

- --
Best regards,
Rafał Wojdyła
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEVwOm02mHaIsSbeWChaL2sHYatbsFAmm6prUACgkQhaL2sHYa
tbsgmwf+Mn0ZL/TMzzp0/iFAfz/6xWli5ToyijoVK4y01GQx/5molAyCS3t1QgCH
ZBdWIlvikVsabJRAUhQc2iLWB1JlMoXYnmNdevWbknKhBFSv/HTJYD8MDqjtK6y3
0Ag1XzJSwCqdtglIdwAR8Y3xtZNt6i876F6gwDi3a+oAJJG7bRGZqAPkA6R6SOFg
Wt5jwjbfynmvgqf9BmQr6ClfapRBU8OGmdcOmmFt9YLkZdDYsAaogkjPgTVGWTzL
XqN4dGs3lRGqfqw+jJpGb1flGKKUlrHQteYn0ATvTpeOxSezfR8UPaJJHzp07R4p
TnSVmUeTdFNsgDPpu5jw2lkVYeMxaA==
=WiQr
-----END PGP SIGNATURE-----