Page 1 of 1

Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 14:54
by Beerholder
Not sure whether to place this here or in features, but since it is quite low level dev stuff I thought I'd post it here anyways... After about 3 weeks of messing around with CEF3, lots of dead ends, ending up using Awesomium and more sigsegvs than I bargained for, I finally managed to open a web browser embedded inside the Minetest game and have it render to an ITexture buffer inside a formspec.

Why? Because I can I guess ... The good (skip to 2:15 for the actual demo, first part is small overview of code changes). The sound actually does work, it's just that I did not record any sound in the video.

vlcsnap-2017-03-16-14h16m26s498.jpg (204.22 KiB) Viewed 13470 times

Bit of background: The proof of concept basically uses the Chrome code base (CEF3) underneath. As per the Chrome processing model, the main application (in this case, Minetest) is separate from the browser process so browser crashes will not crash the main application. Awesomium is a wrapper around the Chromium Embedded Framework (CEF3) and takes care of managing the browser sub-processes and the complexities of passing messages, handling browser events in the main program and providing access to screen buffers for off-screen rendering.

I managed to get the off-screen rendering to work with formspecs by providing a new browser formspec element that, when parsed, will open a so called Awesomium web view. The web view provides it's buffer for copying into other buffers. All that remained was to create a GUI image and copy the web view buffer into the image's texture. This will of course work for arbitrary textures, so the next logical step is to copy the web view buffers to node textures, allowing players to place YouTube video blocks, HTML documentation blocks, etc.

Awesome, right? Well, now for the bad .....

CEF3 is an absolute beast. When I started I found that the Chromium source consisted of 27000 files, it took me hours and hours to compile. There are binary packages for download of 200 - 300 MB. The core library is 60+ MB (uncompressed). It is under continuous development so there are no real stable branches. One version works, the next one crashes for no apparent reason. There are samples online showcasing off-screen rendering e.g. the Chrome browser in 3D applications but these don't take into consideration on how to embed CEF3 in an existing large code base such as Minetest. The CEF samples are no real help either. It looks like I would need to manually write code to share the browser buffers with the main Minetest program. Fun! -_-

TL;DR, CEF3 is a pain to use. I've been hacking around for about 2 weeks now, and would probably need another 2 weeks to get any off-screen rendering to work.

In comes Awesomium. It takes care of all the nasty things one does not want to take care of. I was actually able to put the above proof of concept together in 2 days. It's awesome! But: *Awesomium is closed source*... It is free to use in Open Source not for profit projects, but still, closed source. With 65MB, the libawesomium shared library is still large as it contains libcef of course.

Since the browser runs in a separate process, all sound handling (e.g. YouTube videos, audio played in the browser) is done by that process. That is great for formspecs, but if you'd have a node you would expect the volume to lower the further away you are from the node. To redirect the audio processing somehow, no clue yet how that could be done.

There are alternatives such as embedding Gecko. That will introduce a dependency on the Gecko code base and the Gecko engine would then have to run inside the Minetest process unless I would custom roll a sub-process model for Gecko. Also, the Gecko API doesn't look that simple so lot's of coding would probably be required to get that to work.

So that's the bad... But then of course, what do think about this kind of functionality? Besides this looking cool (and even cooler when I implement casting pages to nodes), is it worth pursuing. I can come up with a few use cases, and there is the possibility of abuse (e.g. a web browser node with configurable URL could link to inappropriate content) beyond what is already possible (e.g. insulting signs).

Also, what about the approach I took? Are these kinds of code base changes worth it? It seems to be minor so far as Awesomium provides a singleton, but still this adds a dependency on (yet) another (closed source) library. Would it be better to keep this outside of the main Minetest code base, package this as a mod (with Lua -> native bridge)? But in that case, is there currently a way to access to the texture buffers? As in, how to go from Awesomium/ CEF3 -> Lua -> formspec/ node ITexture buffers?

I'm really interested in what you think: Any feedback, whether it's good or bad, much appreciated!! :)

And then there are a bunch of TODO's I will still need to look into:
  • More robust code, e.g. it crashes when I exit a game and I need to put some safety mechanisms in place when the browser sub-process crashes
  • Passing events (e.g. mouse clicks) to the embedded browser
  • Implement copy to node texture buffer for web browser nodes
  • See if I can modify the Lua browser mod to allow for configurable URLs
  • The IGUIImage is not properly created, it is not registered in the UI element children
  • Make a fork of Minetest and upload my changes to the fork

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 15:24
by maikerumine
Just saw your video... WOW!!!
You changed minetest immensely!!!!

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 16:11
by Byakuren
I'm against the inclusion of any proprietary dependencies or libraries.

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 16:34
by Beerholder
Byakuren wrote:I'm against the inclusion of any proprietary dependencies or libraries.

Yeah I guess it is indeed still very worthwhile to put in the extra effort and try to get CEF3 to work instead of using the Awesomium wrapper. As far as I can tell, CEF3 and Chromium are both (revised/ new) BSD, so that should not be a problem with LGPL right?

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 18:09
by Byakuren
I don't think it would be an issue.

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 16, 2017 18:21
by BBmine
WOW! I could see problems with having this on servers tho. People could (maybe) download viruses onto the server's computer.

So that's where you've been all these weeks away from me XD

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Fri Mar 17, 2017 08:17
by Beerholder
Well the server is safe as the embedded Chrome is only started/ used on the client. My other thinking is that I should not provide a generic browser mod myself but leave that to the modders. They can hook into the new formspec element or node type and then if they want they could hard code in their mod that only certain pages can be shown. E.g. a form with buttons for tutorial 1, 2, etc. only linking to tutorial pages or hard coding the part and only allowing the video hash to be configured on a block. Worst that could happen then is getting rickrolled XD

On a more serious note, I am going to check out the CEF3 IPC mechanism. There is already some message passing mechanism to communicate UI events to the browser sub-processes (e.g. clicking the back button), there should be something then to pass raw bytes of the screen update rects back to the main process which I can then poll in the Minetest game loop.... I am already able to launch the sub-processes right now using CEF3 only (so without Awesomium), it's just the IPC part that I will have to implement in the sub-process and the polling in the game loop. Easier said than done tho ;)

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Fri Mar 17, 2017 17:23
by Beerholder
Success!!! I managed to get it to work with purely CEF3! So no more closed source Awesomium, and I actually did not have to do anything to get the IPC to work either! I am quite surprised, and happy I got it to work so quickly! And only about 5 segmentation faults today XD

I created a mt_cef wrapper following the same sort of principles as Awesomium to make it easier to manage browser lifecycle, providing a MinetestBrowser singleton for easy access across the code, allowing it to create "WebPages" and replaced the Awesomium calls with my own custom rolled one. I'll continue hacking around when I get home tonight :-D

I can't wait to implement this on nodes ;-)

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Sat Mar 18, 2017 00:28
by Beerholder
Forked 0.4.15, merged the code and pushed my changes my repo here:

Some very minimal documentation on how to build, the changes I made, simple API description and the example mod [MOD] Browser [browser] can be found here:

This is still very alpha quality, e.g. closing web pages in this CEF3 version is not implemented yet so every punch creates a new browser, I don't keep track of opened web pages in my singleton, at this moment there is nothing whatsoever to pass events to the browser and I am getting ABM timeout warnings for some reason -_- Also, resizing the Minetest window leads to spectacular failure (in the sense that for some reason it spawns multiple browsers ... are FormSpec elements parsed anew on resize?????), but then again maybe if you try it will just core dump (or start playing Rick Astley XD) when you resize. Anyways, handle with care for now and use at your own risk ;-)

I'll definitely want to make it way more robust before venturing into the node web browser textures, and might want to change the code style a bit as well while I am at it, as not in line with the Minetest code. But that's for tomorrow... However sharing is caring so if you're interested feel free to have a look and most def let me know if you have questions or concerns!

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Sun Mar 19, 2017 08:18
by nrz
It's interesting

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Mon Mar 20, 2017 11:02
by nrz
We should have an external library, not compile it with minetest itself as it's a very huge library maybe you can provide a PR to minetest with and external library

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Tue Mar 21, 2017 17:40
by Beerholder
Hi nrz thanks, I am most definitely looking into the build and linking against aspect. In the end libcef should obviously not be packaged with or be part of Minetest as it is indeed too big. I currently have it in the minetest/bin directory in my dev env but they do not need to be there (my current setup is still very much a dirty hack). I need to properly put the -DCEF3_ENABLED flag in place as well. Possibly I could use an existing install of Chrome to link against so I need a proper find for the library in the makefile, as well as load runtime. I will look into all of this and once there is a better way to build and link and when more stable, create PRs for devs to review as you suggested.

In the meantime, after a lot of digging through the Minetest code, nodes now also have very basic preliminary browser support:

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Tue Mar 21, 2017 22:07
by MineYoshi
This is awesome... +10

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Sat Mar 25, 2017 00:14
by Inocudom
What we have here is Minetest adopting a useful feature that Garry's Mod is well known for. Personally, I feel that Minetest deserves this feature way more than Garry's Mod ever did, so please keep up the good work.

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Wed Mar 29, 2017 22:57
by sofar
Why not adding some generic layer that can act as e.g. an X11 display or supports the VNC protocol? Then you could interface with almost anything, not just a single program.

Wayland comes to mind. Have minetest be a wayland compositor.

Re: Finally got Chrome to work inside Minetest ...

PostPosted: Thu Mar 30, 2017 17:07
by Beerholder
Actually dummy X server (formerly known as xvfb) did cross my mind at some point :-) but obviously X11 only. To make it generic would be a good idea and should be perfectly feasible tbh (all it needs to do is copy buffers to textures and pass events to the underlying application, anything supporting that like xvfb, its Windows equivalent or an embedded VNC client should have support for that)

I'll see if I can refactor it into a generic interface at some point, it would most certainly make it a lot more powerful. Can't wait to run my bash shell or running my Xfce desktop in Minetest ;-)