Remember that the allocated VRAM will also be consumed by the each framebuffer (width * height * 2 byte per pixel * 2+ for swapping) as well as any other system needs.
In general, JavaFX will try to use no more than
-Dprism.targetvram=xx, freeing textures when this value is exceeded. Least used textures will be discarded, and recreated on need. The default for this setting is calculated as 75% of the maxvram setting, equal to 45M for the example of 90M. This setting may be overly aggressive for some applications, and experimentation with a larger value and
-Dprism.poolstats=true may result in more performance.
Platform Specific Notes
The video memory is usually set using a kernel boot parameter. To check the kernel boot arguments, use:
# cat /proc/cmdline
looking for something like vmalloc=xxxM. Changing this value (or setting it if it its missing) is done by changing the u-boot script.
A different way to check for the current value is:
# root@nitrogen6x:~# dmesg | grep vmalloc
Video Memory allocation is covered in the Raspberry Pi page.