The Wakefield committers may hold off-line discussions from time-to-time. Since these meetings are just for the committers and invited wayland experts, summary minutes will be recorded here as time permits
Online Zoom Meeting 9am PDT 25th May 2023
Attendees: Victor D'yakov, Alexey Ushakov, Phil Race, Kevin Rushforth, Maxim Kartashev, Olivier Fourdan, Jonas Adahl, Alexander Zvegintsev
we have pushed several changes related to the X11 compatibility :
JDK-8307779 - Relax the java.awt.Robot specification
JDK-8280993 - [XWayland] Popup is not closed on click outside of area controlled by XWayland
CSR JDK-8307456 approved for the java.awt.Robot taking screenshots issue wth Screencast. This fix is also targeted to JDK 21.
Drag and Drop does not work in java -> wayland app direction: while this DND fix has been released on the system side(1, 2), changes are still required on the JDK side.
Our code it quite picky for the drop target, and requires it to be a toplevel with WM_STATE property, whereas the Wayland server uses dummy windows without this property set.
I already have a fix for this on the JDK side that I want to put in 21, but first I want to make sure I didn't break anything. In any case, it will be active when the Wayland session is active.
We still have test failures, some of them are tests that really was not so stable, and the timing thing on Wayland vs Xorg makes them fail.
Interesting categories of failures are number of crashed that seem to be in pipewire code.
It is a mixture on the thread that attached to the JDK where we actually calling something and occasionally some threads that pipewire kick of on its own.
We have seen at least one that says "stack smash", terminating.
And there is one that is not a crash, with one particular test on particular systems, which we still need to analyze, related to be not able to get the screen data.
Some of the crashes are in tests that do a lot of captures in a loop.
There is one test which repeatedly moves a mouse cursor and taking a pixel color at mouse position. It makes around ~11000 of getPixelColor requests, which takes no time on Xorg, but can time out with the new API. Alexander said that one request can be 3000x slower on his machine comparing to X11 implementation. So anything that captures pixels in a loop over and over again is a potential problem.
Sometimes when a test fails, test harness makes its own screencapture from another VM, so we wondering about the concurrency of all these libraries we are using now.
It could be a mixture of bugs in our code, bugs in pipewire that are fixed in a later version, or bugs are not yet fixed.
Some comments about threads, I would first check that you have handled the processes callback from pipewire, and also check that you have not enabled the real time pipewire thread(It is normally not enabled, as you should explicitly enable it).
Regarding performance, you are using the screencasting not for screencasting, not the way it is intended, so it is not strange that it takes so much longer time, due to security checks, sessions processing, streams and all that stuff.
The workaround for this particular test case is to keep this stream open, reducing the number of re-negotiation, permission checking, etc.
It reminds me one optimization, that we did for metal pipeline, we have display sync thread which started and stopped on each redraw request.
So the optimization was to have a keepalive counter that allows us to keep at alive enought to catch subsequent requests.
Probably this approach can be used here.
I do not want to overblow this, there are few tests/use cases which really suffers from this, most cases do screen capture a couple of times. So this enhancement could be done later, I don't think that we should rush on that and redone all of that in JDK 21.
All of the TCK tests pass, it is a good step that we pass all the conformance tests. It is the some of functional tests failing.
The plan is to ship this screencast code in 21, and if somebody happens to run on Wayland it should work modular this crashes. We are not gonna say it is supported, we hoping it should practically for most cases.
XTest will be wired to libei, which will actually move your mouse pointer using the Wayland compositor.
The good news is that libei 1.0.0rc1 was released a couple of days ago so we are working hard to try get everything in place, and it is moving nicely.
I fixed a few bugs in our implementation of Wayland toolkit regarding scaling multiscreen environment with different scaling.
What I did was that I tried to keep the buffer associated with a window in sync with the scaling of the window of the output where the window is on(use maximum). So if I move window from an output with 200% scale to an output with 100% scale, the buffer actually shrinks, but now I am not sure that I really have to do this, because the window to my does not change at all.
Wayland seems to scale the buffer correctly. Do I really need to go through the hassle of adjusting the buffer size and have the buffer scaled to the highest available scale of outputs and leave all work to Wayland?
I would suggest changing the buffer whenever it makes sense. If you don't do this, there are two issues:
- when the compositor scales down fonts will look slightly more blurry and will be less sharp
- If you're working on two monitors (say, 100% and 200%) on a battery-powered machine, and your main program is always on a lower-scaled monitor, all that drawing at 200% and scaling down work will increase power consumption and decrease battery life.
Right now it means that you always check at what outputs you are actually on, an take the maximum scale.
There is an extension for the fractional scaling that give you preferred scaling. (Gnome 44+)
We are in progress of implementing Vulkan rendering for pure Wayland prototype
I recall that there was a HiDPI fix for X11 compatibily mode on Wakefield mainling list.
Is there any chance that it could be targeted for 21?
It depends on a fix I merged yesterday, so there is no chance that it will go to 21.
Online Zoom Meeting 9am PDT 27th Apr 2023
Attendees: Victor D'yakov, Alexander Zvegintsev, Olivier Fourdan, Niels de Graef, Phil Race, Maxim Kartashev, Kevin Rushforth
Alexander, I know you submitted a link to a proposed change to the wakefield sandbox repo, and our plan is to go to open jdk mainline code review as well, right?
Please make it much more verbose, as your last submission was really poor on explanation.
Yes, I posted it to the Wakefield sandbox a few weeks ago as a preliminary review of code changes(1, 2) several weeks ago. No work has been done in the last two weeks as we are now doing a test sprint.
I will post it to the openjdk repo next week to get it pushed out before the fork of JDK21. It will have more explanations.
Nikita Gubarkov is doing the HiDPI, it relies on me integrating cached bounds and insets for GraphicsDevice into the mainline, and that has not happened yet.
Does his code overlap with Alexander's or are they complementary?
I didn't have a chance to check the Alexander's code, but from the descriptions they probably are complementary.
We also need to understand the risks of anything that has to do with changes to the X11 compatibility mode, we've had some discussions about how we're going to test all this.
We need to be sure that whatever we put in there is not going to break anything else.
But the clock is ticking, because RDP1, feature freeze is June 8, so we have 6 weeks.
We'll need a CSR to carefully review, it will also require Joe Darcy to jump on it.
There will be a separate PR for changes: code changes and spec changes to make it easier to review(dependent on each other). I'll post a head-up message to wakefield mailing list.
About libEI, progress is still being made, and we are getting closer to something stable and ready to merge . One of the latest protcol changes , to not depend on protobuf anymore.
We are still planning to make it Gnome 45 and Fedora 39.
For libEI we are expecting last few API tweaks, obviously we want to do all that before we declare the API stable.
How is the pure Wayland stuff going on? I understand that it is not going into JDK21
There was no significant progress in the last few months.
The spec changes that are goint into JDK21 should be the same as needed as for the pure Wayland port. There could be some additional things needing to relax.
Regarding TCK, there are lots of discussions about how we actually adequately test this from the functional perspective on different distros.
and we still have in issue with Oracle Linux 9.1 that xdg-desktop-portal-gnome is too old, so it keeps asking the user permission for every screenshot.
9.2 going to be released in couple of weeks, normally you could try it on CentOS and it should work on RHEL as well.
Online Zoom Meeting 9am PDT 30th Mar 2023
Attendees: Maxim Kartashev, Alexey Ushakov, Alexander Zvegintsev, Victor D'yakov, Kevin Rushforth, Olivier Fourdan, Jonas Adahl
xdg-desktop-portal-gnome package was missing in 9.1, but was present in 9.0.
It now comes with 9.1 only with fresh install, system update of an old 9.1 installation didn't work for me, have to install it manually.
While the system now successfully reports version 4 of org.freedesktop.portal.ScreenCast. It is minimum required for restore_token functionality.
xdg-desktop-portal-gnome package comes outdated for both 9.0 and 9.1(latest available 41.2), to be able make screenshot without confirmation for every attempt we need at least 42+ version..
Added handling of negative scenarios when the user partially or completely denies access to screen data.
Now we throw a security exception only if the user denied screen capture at all.
If user is allowed to access only some displays, image data of others will be displayed in black, no exceptions or warnings will be thrown.
If user wants to change previously made decision, newly added Robot.resetRestoreToken() method can be used.
Changed the storage location of restore_token, now instead of a simple file, we use Preferences API.
It's still a file, but now it's a standardized XML file, still storing the token in plaintext.
I don't think this is a problem though.
I also checked probably the most popular product using ScreenCast API - OBS, Open Broadcast Software:
They also use store this token in plaintext in their configuration files.
I also checked how the prototype behaves with Plasma Desktop Session.
When I get the screen data, everything goes without errors, but the data itself is black.
Apparently, it's because I have an Nvidia graphics card.
Another interesting thing is that in response to a call to the Start method, I get a stream that has size, but no position.
This can be a problem, for example when we have two displays with the same resolution.
It is not clear how to distinguish the stream of one screen from the other and thus get the data from the requested display.
We have plans to make it, including introduction of header files, in JDK 21 LTS.
We will know more on our next meeting in 4 weeks from now.
We are usually not making any too risky changes 1-2 weeks before the fork(June 8th), we have only just two months for all this work.
It is not our ultimate goal, but a good opportunity.
We have just begun to continue our work on pure Wayland prototype.
Nikita Gubarkov has some important changes that could be useful for compatibility mode,
he was able to support HiDPI for different monitors using access for underlying wayland subsystem.
He said he will be able to port this changes, and probably, to include them into JDK 21 LTS release.
It is already integrated in our runtime shipped with our products, looks like all the issues resolved so it is time to move to OpenJDK
If this will requires CSR, please do not submit it right before the fork, give it an extra 1-2 weeks.
We could implement some limited support for screenshots for pure wayland.
General idea: as long as we can make at least some tests run just a single window, and don't rely on their absolute position,
but calculate their offsets just from the top left corner of just one window, we can implement robot color picking, and even screenshoting functionality for this window,
by grabbing pixels from associated Wayland buffer.
This will work for some tests, especially performance measuring will be more accurate(e.g. color picking at some points frequently).
Alexander, can we get recent changes for taking screenshots with ScreenCast API, before they reach JDK 21?
pipewire headers requires legal approval before placing them at openjdk github, but I can make a version to use headers files stored in system and share it.
Online Zoom Meeting 9am PDT 2nd Mar 2023
Attendees: Alexey Ushakov, Alexander Zvegintsev, Victor D'yakov, Kevin Rushforth, Phil Race, Olivier Fourdan, Jonas Adahl
(DBUS_SESSION_BUS_ADDRESS, XDG_SESSION_TYPE and WAYLAND_DISPLAY)
Phil: agreed but our CI system cleans some of these out too, before jtreg even has a chance.
So multiple things to address.
Jonas : how important is it to support restore tokens in persist mode on RHEL9.x/OL9.x
Phil: This is the support to allow permissions to do screen capture to persist across sessions.
It is very much needed for our automated CI testing systems which reboot after every test task.
Would be hard to support a release where it isn't possible.
RHEL bug about backport restore token portal support: https://bugzilla.redhat.com/show_bug.cgi?id=2174946
On Ubuntu 22.04 have pipewire-0.3 and spa-0.2 - are these API stable ?