xdg-desktop-portal/issues/850

[X11 compatibility]: Robot.mouseMove does not visually move mouse cursor

[X11 compatibility]: XTest emulated mouse click does not bring window to front..

[X11 compatibility]: Click on title to request focus test failures

The java.awt.Robot class provides methods to emulate input; keyPress(), keyRelease(), mousePress(), mouseRelease()

Current robot implementation doesn't work for Pure Wayland port at all, but it mostly works for X11 compatibility mode.

However, there are some cases when it fails for X11 - when trying to reach outside XWayland  server with using XTest API:

JDK-8280995 X11 compatibility: Robot.mouseMove does not visually move mouse cursor

JDK-8280990 X11 compatibility: XTest emulated mouse click does not bring window to front.

JDK-8280988 X11 compatibility: Click on title to request focus test failures

Implementing this via libEI will also fix the above issues

X11:

Can be used in X11 compatibility mode with the above-mentioned limitations, otherwise we'll have to use the same options as for Pure Wayland.

Covered by documentation changes in JDK-8307779 Relax the java.awt.Robot specification

Pure Wayland:

Solution #1. Not yet usable

This will probably be implemented by libEI. However, its shipping time(even estimated) is not yet known.

Solution #2. Not yet usable

 You can do most of this using the RemoteDesktop portal. Once permission is given, you can use it to completely control the keyboard and mouse.

However, you can't keep the permission as you can with restore_token  in the ScreenCast API, so it's not suitable for automated testing.

The native Wayland splash screen window usually appears somewhere in the upper left corner of the screen.

This is due to Wayland not allowing windows to position themselves.

This will need a Wayland protocol extension to add a role for an output-only, centered surface.

it'd just need a fallback implementation for compositors that doesn't support it, e.g. a dialog

Add input to this existing issue : wayland/wayland-protocols/-/issues/67

Implementation will probably be similar to this MR for Picture-In-Picture video: wayland/wayland-protocols/-/merge_requests/132

Quote from Maxim Kartashev's e-mail:

There's a quality-of-service problem with running via the compatibility layer as under certain circumstances native X windows look blurry. Users with small(ish) HiDPI displays tend to enable fractional scaling and with that enabled (regardless of the actual scale), XWayland pretends that the screen size is smaller and then pixel-stretches the resulting window according to the global scale. This works as a temporary solution, but people get quickly tired of looking at text that is blurry.

See https://gitlab.gnome.org/GNOME/mutter/-/issues/402 and https://github.com/swaywm/sway/issues/5917 for some more details.

wakefield-dev/2023-February/000081.html

Use the same solution as for the Pure Wayland toolkit

Preferred: emulate mouse click with libei

modify tests to use another API to bring window to front (e.g XMapRaised)

Covered by documentation changes in JDK-8307779 Relax the java.awt.Robot specification. Test modifications are still needed.

Click on window's body instead of decorations, request focus in other way

Covered by documentation changes in JDK-8307779 Relax the java.awt.Robot specification. Test modifications are still needed.

// gcc windowSpawnDoS.c -o windowSpawnDoS -lX11 && ./windowSpawnDoS
#include <X11/Xlib.h>
#include <unistd.h>

int main(void) {
    XInitThreads();
    Display* dpy = XOpenDisplay(NULL);

    Window root = DefaultRootWindow(dpy);
    for (int i = 0; i < 500; ++i) {
        Window ww = XCreateSimpleWindow(dpy, root, 50, 50, 50, 50, 0, 0, 0);
        XMapWindow(dpy, ww);
    }

    for(;;) {
        XEvent e;
        XNextEvent(dpy, &e);
    }
}

It may be faced when running some tests which are trying to map a lot of windows, e.g. one for every GraphicsConfiguration.

If you have 210 GraphicsConfigurations per display on Xwayland,  then running such test in 2 display configuration will lead to crash.

It is doable as of now, but not in a standard and convenient way. It can be done using some shell specific APIs, e.g. org.gnome.Shell.Screenshot DBUS API (which will not work for KDE, it has another API)

It has several drawbacks:

• There is no direct access to the screenshot data. Screenshot is saved to png file, after that you have to read it, decode it.
• Filesystem footprint. You need to save it to some file(its filesystem can be slow). It can be workarounded by using shared memory object though.
• Using DBUS API in this case is really slow. For instance, just a sync DBUS call to take a 1000x1000 screenshot and save it to the shared memory object file is ~14 time slower(it is just saving, without decoding and extracting data) comparing to full X11 implementation.

So, it would be great to get some standard API to get screenshot data in memory. I see that there is an open bug against this issue.

Specification change needed to handle user confirmation of screen capture (user can deny/partially allow screen capture).

Solution #1.

Specifically, it provides org.freedesktop.portal.Screenshot#Screenshot which can be used to create a screenshot.

However, it has some drawbacks(at least on Gnome):

• It does not allow to take a specified screen area, only whole screen.
• It asks a confirmation on each screenshot request, so it is unusable for automated testing. (note, this was fixed for GNOME 43, not yet delivered to most distros)
• It saves a screenshot to disk, which must be read and deleted(may be slow depending on drive).
• A user can deny the request. In this case as a possible solution we can throw a SecurityException for Robot#createScreenCapture (javadoc update required). Currently it is thrown only if readDisplayPixels permission is not granted(which is not suitable here).
• There is no way to disable the "screen flash" after screenshot

There is also org.freedesktop.portal.Screenshot#PickColor API which can't be used for Robot#getPixelColor needs. It does not allow to take a pixel color at specified location, but interactively asks a user to pick a color with mouse cursor.

Solution #2. Preferred

Finally, another possibility is to use the Screencast portal.

This might be a bit of overkill, but it avoids several of the problems mentioned in the screenshot and color picker portals.

• implementation is more complicated comparing to Screenshot API
• A user can deny the screenshot request too.
  
  • Permission to make screenshots can be stored with restore_token (string, introduced in xdg-desktop-portal 1.12), that needs to be stored safely somewhere.
    
  • new permission required if display set is changed or rearranged, might be an issue in case of remote automated testing.
• no intermediate file, screenshot data can be obtained from memory
• each display has its own stream, in case if requested screenshot area covers several displays resulting screenshot need to be combined from pieces.

Each solution is viable, but #2 seems to be more preferable.

However we may want to implement both of approaches:

If some new display got or disconnected, new permission request from user is required. In case of automated testing it may be a stopper. But we can fallback to a solution #1.

Prototype is available here.

• -Dawt.robot.screenshotMethod=dbusScreencast to switch on screenshot via Screencast(x11 for switch off). Enabled by default on Wayland
• -Dawt.robot.screenshotDebug=true to print debug info, disabled by default

CSR Approved JDK-8307456

Review in process: openjdk/jdk/pull/13803

JDK code it quite picky for the drop target, and requires it to be a toplevel with WM_STATE property, whereas the Wayland server uses dummy windows without this property set.

We need to disable this behaviour when the Wayland session is active.

Javadoc for java.awt.Robot#mouseMove says:

    /**
     * Moves mouse pointer to given screen coordinates.
     * @param x         X position
     * @param y         Y position
     */
    public synchronized void mouseMove(int x, int y) {

Despite the fact that this mouse movement is successfully emulated within the XWayland server(following mouse press emulation will be in a right place), the mouse cursor does not visually move.

With current specification of java.awt.Robot#mouseMove it may be a conformance issue.

Use the same solution as for the Pure Wayland toolkit

Preferred: see earlier point on working with RemoteDesktop portal(can't keep the user permission between sessions)

JDK21: Covered by JDK-8307779 Relax the java.awt.Robot specification

Use workaround.

This also requires a change in documentation, as there are "blind spots" which, if clicked on, will not close the popup window(e.g. window title or area in the side dock without application icons)

Fixed in JDK21

// gcc hugeFrame.c -o hugeFrame -lX11 && ./hugeFrame
#include <X11/Xlib.h>
#include <unistd.h>

static Display* display;

int main(void) {
    int width = 22000;
    int height = 25000;

    display = XOpenDisplay(NULL);

    Window win = XCreateSimpleWindow(display, DefaultRootWindow(display), 0, 0, width, height, 0, 0L, 0L);
    XMapWindow(display, win);
    XFlush(display);

    sleep(1);
}

/var/log/syslog shows:

gnome-shell[1248]: WL: error in client communication (pid 1248)
gnome-shell[1298]: (EE)
gnome-shell[1298]: Fatal server error:
gnome-shell[1298]: (EE) wl_shm@5: error 1: invalid size (-2094967296)
gnome-shell[1298]: (EE)

where -2094967296 looks like an integer overflow of 4 * 22000 * 25000

So it fails to create a pool, but doesn't handle it gracefully.

Excerpt from Olivier's mail:

Also worth noting that the XRandR emulation is per window/X11 client, whereas the root window is shared between all X11 clients, but maybe we could send a fake ConfigureNotify event to the given client, I would need to check if that's doable.

This notification would be helpful.

This MR adds notification, however it has several issues:

1. There is no event when you are trying to change to native resolution from another.
2. XRRScreenChangeNotifyEvent always has native resolution reported. (we are not using it though)
3. It might be a minor one, but on X11 session there is no events reported if you are trying to change to the same resolution.