The Wakefield committers may hold off-line discussions from time-to-time. Since these meetings are just for the committers and invited wayland experts, summary minutes will be recorded here as time permits
Online Zoom Meeting 9am PDT 17th Aug 2023
Attendees: Niels de Graef, Victor D'yakov, Phil Race, Olivier Fourdan, Maxim Kartashev, Kevin Rushforth, Jonas Adahl, Alexander Zvegintsev
As suggested, I implemented the approach when the session is not closed right away, but after a few seconds of inactivity. It gave a significant boost for certain scenarios, e.g. taking pixel by pixel in 50x50 area took 375s before the fix and only 28s after the fix.
Other than that working on various HiDPI scenarios for taking screen data in X11 compatibility mode.
We have made some progress since we last met, Vulkan rendering pipeling work is ongoing, we are actually able to render a colored rectangle on the screen, still the bulk of the work was setting up the environment to be able to start making progress, it is almost done.
We also implemented input support for testing through our Weston plugin, so the robot can generate keyboard events, mouse clicks, including absolute positioning, etc. We are getting ready to start running tests in our environment, but we are not there yet because the rest of the toolkit is not ready to start testing.
I finished working on tooltips, menus, now all the stuff works more or less, because the positioning left for the Wayland server entirely with only hint that we can give. It works pretty well for us, but things may change in the production, with all variety of desktop configurations.
I am finishing the clipboard support and plan to get to work on DnD soon.
I think, that primary focus for our work in medium term is going to be Vulkan and getting the IDE window up on the screen and interactable to show our users the progress being made.
The hints you refer to positioning are a bit vague, did it go where you wanted it to go, you don't know it went there, but it actually did go there?
It is probably the most complicated part of the protocol I have dealt with so far. I can post a link to the protocol interface, called the Positioner, but it will not make things clearer.
In short, you throw away all the calculations you did in the Java code, except for the coordinates relative to the top left corner of the parent window, and then you should suggest to Wayland what to do if that particular position is not suitable, for example, if it is between screens, on the edge of the screen, off the screen. I think this logic in Wayland is quite similar to the Java code we have.
The Wayland API does not allow you to use the results of calculations made by common Java code, because the result of this code is in an absolute position on the screen.
So this is a mismatch between what Wayland expects and the way our code has been written historically?
There's nothing fundamentally wrong with either side, it's just that they come from two different worlds and it's going to be hard to make them meet.
Yes, basically the work that we do in Java right now for positioning and popups, all that work is done on the Wayland server side, and you can only nudge it in the right direction that you think is preferable for you by giving hints.
There is something about you posted on a Wakefield mailing list about libei. Could you please elaborate what are doing there?
The idea for our testing so far has been to rely on a plugin for Weston to use the Weston instance and run tests there. The only ramining part that was not implemented was the input emulation.
One of the approaches to achieve that is to use more or less standard way to use the lebei. I asked if Weston supported it so we could just implement it through libei. But it will not be supported, so we decided to use another approach.
people might be interested in Peter Hutterer’s GUADEC talk about libei: https://www.youtube.com/live/dVVyokoQl2k?t=15312
He is the maintainer of libinput and the author of libei. It is a good talk about input capturing and what libei is.
Gnome 45 will be released with libei support in a less than a month.
X11 compatibility mode mostly seems to be pretty stable, the only issue we have is related to the Meta.disable_unredirect_for_display(global.display) workaround. It seems to work.
I did some pretty exaustive testing, we can run thousands of test with screencaptures, whatever else, the token does seems to remain valid, unless we run this test that does the fullscreen capture and everything goes south.
Is there a way to do this Meta.disable_unredirect_for_display(global.display) in some kind of startup file?
You can try to use an extension(basically a js) which loads with a system
In Gnome 45 and xdg-desktop-portal 1.18 you also get the same token behavior(reuse token and restore session) as in Screencast for Remote Desktop, so you can do a remote control.
For X11 compatibility mode, XWayland has direct support for libei, so XTest calls will call libei internally. This allows to control mouse and keyboard outside the X11 server, which is not possible today.
For example, clicking on the title of a window to move it to the front should work.
However, it does not use tokens, but a dialog for user confirmation.
Is this permission a one-time thing that lasts forever?
No, the XWayland will keep the connection for some time, if there is no activity from the client for a certain amount of time the connection will be closed and a user will be asked again.
It does not use the token. From the Wayland point of view you never know if some legit X11 application or a rogue client is trying to hijack your pointer.
What if I need to keep this permission between reboots?
In this case, you would need to use Remote Desktop Portal with a token and use libei directly, not XWayland.
For native Wayland, if we use libei directly - we are good, but for XWayland it gets better only when user is there and can click on a session.
It is not even a login session , it is XTest - libei behind the scenes session.
I forgot to mention one more thing, the splash screen support is also finalized.
I managed to get it to the screen center, the idea is to show a screen sized transparent window and position against it and hope that the Wayland will place the window exactly matching the screen size.
I am not sure how well it will work in a multi-monitor configuration.
There may be some glitches, such as animations showing windows appearing, it will not work well with tiling setups.
It does not look like a future-proof solution.
The proper solution for the splash screen is to work on the splash screen role window.
We could use a X11's splash screen implementation for a while, it does not depend on XToolkit, and does not have such issues with positioning.
It will be problematic in many compositors that use Wayland to do coordinate systems in a compositor. In Gnome terms, this is called fractional scaling mode, and it may cause the splash screen to be blurry.
Online Zoom Meeting 9am PDT 22nd June 2023
Attendees: Victor D'yakov, Alexey Ushakov, Phil Race, Kevin Rushforth, Maxim Kartashev, Olivier Fourdan, Alexander Zvegintsev
Update from Alexander :
Screencast support for AWT Robot was integrated into JDK 21
Mostly in good shape.
Some issues, failing tests with
- non-default UI scale
- resolution other than native
- tests which gain focus / raise window by mouse click
- some crashes - not related to screen cast (that one is fixed)
- still need to fix some issues
Crash seen now is in GTK 3 when using file dialog - not seen on X.org, but probably our bug
Still losing token sometimes, although worked through lots of confusion and found that CI
system inside Oracle periodically cleans out ~/.java where the token was stored
Temp. change to store it in ~/.awt until that is resolved.
Still see token lost sometimes - unclear why.
1) A tight loop invoking a small JDK app that uses Screencast-based Robot code
was able to run 64K times (ie 64K JVM invocations) - before some unrelated XIM leak caused a hang.
So not obviously the number of invocations
2) Number of reboots ? We reboot CI test systems after each task
3) Something done by one of the tests that invalidates it ?? - perhaps for the rest of the session ?
Dividing up full battery of tests and running the subsets over and over seems so far to have not
provoked invalidating the token. It was expected that this would show the problem and narrow down
the cause but none the wiser yet.
May have a workaround for the full screen catpure failure mentioned above
Alt-F2, then type "lg", to invoke looking glass.
then in the evaluator type
It immediately takes effect for the current session.
If that solves it might be an ubuntu specific issue
[UPDATE: this did indeed cure it : tracked as JDK bug https://bugs.openjdk.org/browse/JDK-8310333]
possibly the gnome extension might disable it for all sessions
Tried the JDK 21 robot screencast code with their hidpi fixes and found some issues with multi-screen setup and also found some issues with hidpi solution.
These seem to be problems in their code not in the JDK code. Working on it.
Also working on supporting hw accel in the pure wayland branch
Online Zoom Meeting 9am PDT 25th May 2023
Attendees: Victor D'yakov, Alexey Ushakov, Phil Race, Kevin Rushforth, Maxim Kartashev, Olivier Fourdan, Jonas Adahl, Alexander Zvegintsev
we have pushed several changes related to the X11 compatibility :
JDK-8307779 - Relax the java.awt.Robot specification
JDK-8280993 - [XWayland] Popup is not closed on click outside of area controlled by XWayland
CSR JDK-8307456 approved for the java.awt.Robot taking screenshots issue wth Screencast. This fix is also targeted to JDK 21.
Drag and Drop does not work in java -> wayland app direction: while this DND fix has been released on the system side(1, 2), changes are still required on the JDK side.
Our code it quite picky for the drop target, and requires it to be a toplevel with WM_STATE property, whereas the Wayland server uses dummy windows without this property set.
I already have a fix for this on the JDK side that I want to put in 21, but first I want to make sure I didn't break anything. In any case, it will be active when the Wayland session is active.
We still have test failures, some of them are tests that really was not so stable, and the timing thing on Wayland vs Xorg makes them fail.
Interesting categories of failures are number of crashed that seem to be in pipewire code.
It is a mixture on the thread that attached to the JDK where we actually calling something and occasionally some threads that pipewire kick of on its own.
We have seen at least one that says "stack smash", terminating.
And there is one that is not a crash, with one particular test on particular systems, which we still need to analyze, related to be not able to get the screen data.
Some of the crashes are in tests that do a lot of captures in a loop.
There is one test which repeatedly moves a mouse cursor and taking a pixel color at mouse position. It makes around ~11000 of getPixelColor requests, which takes no time on Xorg, but can time out with the new API. Alexander said that one request can be 3000x slower on his machine comparing to X11 implementation. So anything that captures pixels in a loop over and over again is a potential problem.
Sometimes when a test fails, test harness makes its own screencapture from another VM, so we wondering about the concurrency of all these libraries we are using now.
It could be a mixture of bugs in our code, bugs in pipewire that are fixed in a later version, or bugs are not yet fixed.
Some comments about threads, I would first check that you have handled the processes callback from pipewire, and also check that you have not enabled the real time pipewire thread(It is normally not enabled, as you should explicitly enable it).
Regarding performance, you are using the screencasting not for screencasting, not the way it is intended, so it is not strange that it takes so much longer time, due to security checks, sessions processing, streams and all that stuff.
The workaround for this particular test case is to keep this stream open, reducing the number of re-negotiation, permission checking, etc.
It reminds me one optimization, that we did for metal pipeline, we have display sync thread which started and stopped on each redraw request.
So the optimization was to have a keepalive counter that allows us to keep at alive enought to catch subsequent requests.
Probably this approach can be used here.
I do not want to overblow this, there are few tests/use cases which really suffers from this, most cases do screen capture a couple of times. So this enhancement could be done later, I don't think that we should rush on that and redone all of that in JDK 21.
All of the TCK tests pass, it is a good step that we pass all the conformance tests. It is the some of functional tests failing.
The plan is to ship this screencast code in 21, and if somebody happens to run on Wayland it should work modular this crashes. We are not gonna say it is supported, we hoping it should practically for most cases.
XTest will be wired to libei, which will actually move your mouse pointer using the Wayland compositor.
The good news is that libei 1.0.0rc1 was released a couple of days ago so we are working hard to try get everything in place, and it is moving nicely.
I fixed a few bugs in our implementation of Wayland toolkit regarding scaling multiscreen environment with different scaling.
What I did was that I tried to keep the buffer associated with a window in sync with the scaling of the window of the output where the window is on(use maximum). So if I move window from an output with 200% scale to an output with 100% scale, the buffer actually shrinks, but now I am not sure that I really have to do this, because the window to my does not change at all.
Wayland seems to scale the buffer correctly. Do I really need to go through the hassle of adjusting the buffer size and have the buffer scaled to the highest available scale of outputs and leave all work to Wayland?
I would suggest changing the buffer whenever it makes sense. If you don't do this, there are two issues:
- when the compositor scales down fonts will look slightly more blurry and will be less sharp
- If you're working on two monitors (say, 100% and 200%) on a battery-powered machine, and your main program is always on a lower-scaled monitor, all that drawing at 200% and scaling down work will increase power consumption and decrease battery life.
Right now it means that you always check at what outputs you are actually on, an take the maximum scale.
There is an extension for the fractional scaling that give you preferred scaling. (Gnome 44+)
We are in progress of implementing Vulkan rendering for pure Wayland prototype
I recall that there was a HiDPI fix for X11 compatibily mode on Wakefield mainling list.
Is there any chance that it could be targeted for 21?
It depends on a fix I merged yesterday, so there is no chance that it will go to 21.
Online Zoom Meeting 9am PDT 27th Apr 2023
he topic of the day was a discussion around Alexey's email to the mailing list ..libei
proposing what we might do for the "full" wayland native port and the various points it raised
Note: Alexey will add a fleshed out version on the project wiki.
The proposal suggests we'd code to the low level wayland APIs rather than using a high level toolkit (ie GTK4)
As previously observed we should explore both. Probably we could get bootstrapped a lot faster with GTK4
and it would give flexibility for backends so we'd probably not need to create an OGL and Vulkan backend ..
if Vulkan was an option on that distro then we'd expect the underlying Cairo (?) based renderer to be
using it or have it as an option.
Likely there are pros and cons to both options. GTK4 might make assumptions or so things in a way
that isn't compatible with some other functionality. A low-level port might mean a lot of re-inventing
the wheel and having to use functionality that is distor-specific but would have been hidden / abstracted
by the platform GTK 4.
So due diligence on both options, and not expecting completely smooth sailing for either.
Ease of implementation and maintenance and portability are strong arguments, but so are flexibility and
internal consistency in behaviour and behavioural compatibility with the existing X11 implementation.
So if doing it "right" is more work, we'll have to absorb that, and we want a stable base so we aren't
in a perpetual maintenance game.
Mario observed that we need to not get ahead of ourselves because we still have the list
of technical challenges that need to be resolved that apply to X compat mode and to a native port too.
We'll work on those but whilst they work through the eco-system over 2 years (?) we could be
doing something useful ..
One element of the proposal not directly related to wayland is we could merge EDT and Toolkit thread
since we no longer need these separate for applets. Possibly, but we kick off a new thread for modal
dialogs too .. could this be special cased ? Might depend on whether you can limit what happens
in such a case but I'm not sure you can.
We do have these documented on the wiki but it needs another pass to have a consitent format
- what the problem is
- what JDK can do
- what we need from the wayland ecosystem
- how much of a blocker it is
- can we "tweak" the spec - at the cost of making backports harder
We should look at this soon so we can get started on spec tweaks we think may be unavoidable but acceptable ..
Eg What's acceptable for screenshots ? We can't deprecate but can we make something optional ?
Alex Z: Is the SplashScreen API supportable ?
Apparently splash screens display in top lefe (the window positioning problem)
A splash screen (output only) surface role could probably be proposed upstream.
it'd just need a fallback implementation for compositors that doesn't support it, e.g. a dialog
Maybe similar to this PiP request : https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/132
For more info on roles: https://wayland-book.com/surfaces/roles.html
Add input to this existing issue : https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/67
Is there a library ? Like ATK ?
There are existing DBUS based A11Y APIs ..
For the accessibility protocol Jonas is talking about: https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/
For a quick overview of what Emmanuele did for accessibility in GTK4: https://blog.gtk.org/2020/10/21/accessibility-in-gtk-4/