Open Cobalt, Issues and Opportunities 2019

[John Dougan]



This is a grab-bag of comments about the current state of the Open Cobalt project and its artifacts. It is mostly issues because those are more easily distinguished. Within a category, points are mostly in no particular order. I didn’t look at OpenQwaq at the time because its license and the license terms for Open Cobalt mandated by the grants were incompatible.

• Top level
  • What is it for? It was a research project, now is there any thing to build on it that people want? Coolness is insufficient.
  • This ties into the general VW/VR/Cyberspace problem of non-uptake. Several times there has been an iteration of the hype cycle and a fad period, but VWs haven’t caught on except in niches. Why? Is there anything we can do about that? Compare with uptake of embedded virtuality devices like smartphones, tablets, thermostats, etc.
  • Is there anything you can do in VR that is better than in person?
  • Small devices have taken over the world, and none of the developed STs really work on them well, as far as I can tell. For that matter, why would you want a VW on your phone?
  • In VW, like in 3D games and other places where a performance race exists, user perception of performance is important. The difference between 30 and 60 fps with lots of avatars and geometry is huge, and the users don’t usually care what the underlying language is. Modern STs appear to be a terrible place for this, unless you are making a research system where you can sacrifice some performance for development flexibility.
  • The original project was subject to the whims of grant requirements, so many obvious issues had to be ignored.
• Development
  • I should have all the necessary parts and keys available for someone to start up OC again. It may take a little while to find all of them. Some parts (the formerly Duke hosted resources) are basically backup archives of the file systems that will need some work with Mantis, SqueakSource, and a Web Server to get working again. Whoever wants them is going to need an estimated 6-10 GB of space I can download into.
  • I’m documenting a lot of issues here, but that doesn’t mean the project is unsalvageable. It does mean that it’s going to take significant time and effort by people who know what they are doing or are willing to learn.
  • In general, productizing is much harder than you think.
  • The media toolchain we developed against is partially obsolete. This is why I fixed the OBJ/MTL importers as that format seems to have some longevity, but OBJ/MTL doesn’t support complex content. Thinking about it , I think I made some late changes to the Collada/SketchUp importer at the same time. Somewhere I have a written rant at the abysmal state of the 3D media tools at the time.
  • In general, anyone picking this up for any other purpose than research is going to need to come up with a strategy for getting needed media content built, optimized, and into the scenes. This includes a media licensing issue. Most 3D art, for instance, is only licensed for re-distribution in a hard to reverse engineer format, which is pretty much contrary to the original OC goals. Places like OpenGameArt and CC licensing help a great deal, but don’t cover everything and often need adjustment. To do this adjustment you need a competent 3D artist. Similar issue for other media types.
• Architecture
  • Sharing takes place at the entire scenegraph level, one scene per island. There is a provision to possibly have more than one, but it never proved necessary and would have only added complexity.
  • The rendering process locks all the islands then render the in-island scenegraphs. We have to do the locking like this because the see-through portals potentially make all the island scenegraphs into one giant scene.
  • This is really coarse, and doesn’t scale well. Also has a hard time representing information hidden from a subset of users, as multi-player games require.
  • Older versions of Croquet used a 2 scenegraph format, one in and one outside of the island. This may be a better place than the current in-island architecture as you could make the in-island one less of a graphical system and more of a higher level semantic scene description not tied to 3D or limited by replicable FP. However you would have to maintain twice the code and solve the synchronization issues between the 2 trees. As I understand it, the synch problems were why they originally switched to the single in-island scene approach.
  • There were I/O blocking problems in a number of places. Biggest was when the initial scene snapshot was being loaded/saved into/from the island as even small scenes are big. This might have been partly mitigated by the planned (but never finished) move away from ImageSegments as the snapshot format.
  • Proper platform multi-threading might be a solution to some of the issues. However ST dogma is that you don’t want multi-threading on platform threads at the ST level and the one ST I know of that does that is proprietary and Windows only.
• Implementation Dependencies
  • Uses an old Squeak, needs porting to something with closures and the JIT. Do modern Squeaks even have the necessary Croquet VM support anymore? (replicable FP, etc.)
  • HUD/In-Helmet UI uses TweakUI. I like Tweak, but apparently I’m the only person besides Andreas who did. Parts of the TeaTime Islands are built on Tweak Islands.
  • It is using an old version of OpenGL that needs to be updated, probably changed to something WebGL/OpenGL ES compatible. A lot of the VM OGL support code is from the layers of prior attempts to do portable 3D in Squeak, so that’ll have to be rewritten if porting outside of Squeak.
  • External space chat uses Jabber/XMPP, which appears to have fallen out of favour and the replacements are mostly proprietary. Probably best to stick with it for now until a better open alternative arises.
  • Because of the ST object model and how it is usually implemented, most STs and derivatives have an issue with memory bandwidth when running on a C optimized processor. It also makes it difficult to use shared memory to get around the lack of platform threads. You could write a bunch of C/C++ and interface to that, but why bother with ST then?
  • Matthew had some ideas and code for moving to a more modern Squeak.
  • The Xrfb system used to support in-world shared desktop applications was a crude implementation with serious security and performance issues, but the basic idea is solid. It should also be usable for non-Cobalt applications. I spent some time working on this external to OC and think that with modern containerization the backend could be rebuilt and be very solid.
  • After doing a number of personal projects in this area, I have concluded that Pharo, though an interesting and worthwhile project, isn’t Smalltalk and its historical version compatibility issues make it a bad platform for a low headcount fringe project and may need to shift platforms without a big rewrite.
  • This isn’t saying I’m totally happy with the state of Squeak either. OTOH in the Squeak dev tools I can do a “senders of…” on a method name in the method list and it will do the right thing.
• Scenegraph and Rendering
  • The scenegraph design is a late 90’s approach that matches well with the state of OpenGL at that time; Dave Smith did a good job but it is complex (not unreasonably so) and not well documented. It can’t use a lot of modern approaches and optimizations because the change patterns are less predictable than in games (which can be designed to make pre-calculation profitable).
  • There is a LOT of functionality in TObject, the root of the scenegraph node classes. If I recall correctly this is because Alan likes that kind of top focused design because he believe it to be better for manipulation and extension by less technical users (Morph has the same structure). This decision may need to be revisited.
  • The rendering architecture is complex, hard to debug, and not well documented. I think this is mostly because of the see-through portals. On the other hand, I haven’t needed to fix a lot of issues either.
  • Bug in skybox timing, it appears to be using the wrong ttime for rotation calculations when viewed through a portal. I’m betting this is part of a more general issue with the portal rendering code.
  • There is a memory race in the shader code, as there is a chance shader objects will get moved by the GC at the wrong time. If the version of OGL is updated, this just gets worse as modern OGLs rely much more on shaders.
  • The raycasting system to determine where the mouse is pointing is slow and interwoven with the rendering to avoid propagating messages into the island. The performance might be solvable by moving to an octree for indexing the mesh faces. Bounding geometry is tight spheres, it may be worthwhile to look into changes.
• TeaTime
  • Cobalt still uses the “Interim TeaTime” because that worked well enough to ignore. The transactional approach of the full TeaTime has issues because the large amount of multi-user changes to the in-island scenegraphs leads to frequent commit conflicts, rollbacks, and retries, hurting real-time performance. Before he died, Andreas and I had some discussion about how to improve that situation for scaling, but that would be another research project.
  • The cryptography and protocol in the Island/Router communications was bad in 2012 and is hopelessly obsolete in 2019. Maybe try with SSL? Websockets? This is part of what Matthew looked into. OTOH, you may not care about security and this could be ignored.
  • Matthew had a lot of ideas and a bunch of code for improved TeaTime infrastructure as part of his thesis.
  • It would be nice to reexamine the NAT router hole punching code. It has been awhile, things have probably changed in this space, and UPnP is considered a security issue.
  • There is a garbage collection problem in how external references are managed in the island. When the project closed, I wasn’t sure about how to handle it.
  • I was pretty happy with the general WAN Island/Router discovery approach in OC and we had enough flexibility in the architecture to try experiments. There are a couple known issues in the SSDP implementation used for LAN discovery, but shouldn’t be too bad to fix. Ideally we’d interface to the platform SSDP, if such exists. It would also be nice to be able to use Bonjour/Zeroconf/DNS-SD service discovery as it is much less chatty.
  • The texture cache architecture (which is built on the island router) needs looking at and possibly extended or redone to handle other media types. I think Qwaq/Teleplace did this and in the original Croquet builds there was a second cache system by the UMN group that we decided not to use.

--

John Dougan
jdo...@acm.org

[Eugen Leitl]

On Thu, Sep 12, 2019 at 06:04:24PM -0700, John Dougan wrote:

Whoah, lots of really meaty bits here.

> This is a grab-bag of comments about the current state of the Open Cobalt
> project and its artifacts. It is mostly issues because those are more
> easily distinguished. Within a category, points are mostly in no particular

> order. I didn???t look at OpenQwaq at the time because its license and the

> license terms for Open Cobalt mandated by the grants were incompatible.

IANAL, but afaik we still can't just take code from a GPLv2 project to import
into the MIT license project without it becoming a GPLv2 project. This might
or might not cause license impedance mismatch issues with the rest of the
Smalltalk ecosystem, which tends to be MIT.

> - Top level
> - What is it for? It was a research project, now is there any thing

> to build on it that people want? Coolness is insufficient.

Open source VR is certainly fringe de la fringe (0.8% of Steam are Linux users,
and only about 1% of all Steam users own a VR headset, stats from
https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam?platform=combined ,
whoever is bothering with OpenHMD is negligible) but I believe in scratching our own itch.

> - This ties into the general VW/VR/Cyberspace problem of non-uptake.

> Several times there has been an iteration of the hype cycle and a fad

> period, but VWs haven???t caught on except in niches. Why? Is

> there anything
> we can do about that? Compare with uptake of embedded virtuality devices
> like smartphones, tablets, thermostats, etc.

We're one or two technology iterations away from a mobile AR platform. There are industrial
niche users ( https://www.youtube.com/watch?v=b4nNtN7XBi8 https://www.slideshare.net/esug/ui-prototyping-with-smalltalk-at-thales )
which can have significant impact despite small numbers.

> - Is there anything you can do in VR that is better than in person?

Collaboration in an energy-poor world is going to have to substitute for a lot of flying.
Entertainment, obviously. Outdoors untethered AR is pretty a whole world waiting to be filled.
Whatever is impossible in person is only limited to our imagination.

> - Small devices have taken over the world, and none of the developed

> STs really work on them well, as far as I can tell. For that matter, why
> would you want a VW on your phone?

Not on my phone. But on an AR wearable, certainly.

> - In VW, like in 3D games and other places where a *performance
> race* exists,

> user perception of performance is important. The difference
> between 30 and

> 60 fps with lots of avatars and geometry is huge, and the users don???t

The question is how much your Cog VM is bottlenecking issues which happen
largely on the GPU. Right now GPUs have issues with relatively low resolutions
like 2880 x 1600 (combined) on 300 EUR gear, so perhaps that's not yet quite a problem.

If we look at SecondLife then such distributed systems can be successful without
fancy graphics and 120+ fps. A lot of the power of SecondLife was due to the
toolkits available in-game, which engaged users in the same way Minecraft
does today.

> usually care what the underlying language is. Modern STs appear to be a
> terrible place for this, unless you are making a research system
> where you
> can sacrifice some performance for development flexibility.

Is there any borderline half-featured alternative for open source VR?
Particularly, one you could develop in-world? Like inspecting and modifying live objects.

I'm not aware of any. The drive towards JavaScript and browser environments is
dangerous, because it causes a lock-in into a fast-paced enterprise-dominated
ecosystem which long-term will trap users into telemetry-riddled TeleTubby land
by Facebook, with Intelligence API inside(tm).

> - The original project was subject to the whims of grant

> requirements, so many obvious issues had to be ignored.

> - Development
> - I should have all the necessary parts and keys available for

> someone to start up OC again. It may take a little while to find all of
> them. Some parts (the formerly Duke hosted resources) are
> basically backup
> archives of the file systems that will need some work with Mantis,
> SqueakSource, and a Web Server to get working again. Whoever
> wants them is
> going to need an estimated 6-10 GB of space I can download into.

Can you share the result via an online drive, until we can get something
up via GitHub or GitLab?

> - I???m documenting a lot of issues here, but that doesn???t mean the
> project is unsalvageable. It does mean that it???s going to take

> significant
> time and effort by people who know what they are doing or are willing to
> learn.

This is definitely the bottleneck here.

> - In general, productizing is much harder than you think.
> - The media toolchain we developed against is partially obsolete.

> This is why I fixed the OBJ/MTL importers as that format seems
> to have some

> longevity, but OBJ/MTL doesn???t support complex content. Thinking

> about it ,
> I think I made some late changes to the Collada/SketchUp importer at the
> same time. Somewhere I have a written rant at the abysmal state of the 3D
> media tools at the time.

Blender seems to have made giant strides since. I'm not aware of
anything else that is open source and powerful.

It would be very nice to have in-world building tools, which need not to be elaborate.
Second Life allowed some quite impressive builds, and even without direct
access of Smalltalk. See also Minecraft worlds.

> - In general, anyone picking this up for any other purpose than

> research is going to need to come up with a strategy for getting needed
> media content built, optimized, and into the scenes. This
> includes a media
> licensing issue. Most 3D art, for instance, is only licensed for
> re-distribution in a hard to reverse engineer format, which is
> pretty much
> contrary to the original OC goals. Places like OpenGameArt and
> CC licensing

> help a great deal, but don???t cover everything and often need

> adjustment. To
> do this adjustment you need a competent 3D artist. Similar issue
> for other
> media types.

I think we need in-world user-generated content that is generated with
a permissive license that takes an explicit opt-out. You need not a lot
to prime the pump to make the environments attractive, so I don't think
this is a serious issue right from the start.

> - Architecture
> - Sharing takes place at the entire scenegraph level, one scene per

> island. There is a provision to possibly have more than one, but it never
> proved necessary and would have only added complexity.

> - The rendering process locks all the islands then render the

> in-island scenegraphs. We have to do the locking like this because the
> see-through portals potentially make all the island scenegraphs into one
> giant scene.

> - This is really coarse, and doesn???t scale well. Also has a hard time

> representing information hidden from a subset of users, as multi-player
> games require.

> - Older versions of Croquet used a 2 scenegraph format, one in and

> one outside of the island. This may be a better place than the current
> in-island architecture as you could make the in-island one less of a
> graphical system and more of a higher level semantic scene
> description not
> tied to 3D or limited by replicable FP. However you would have
> to maintain
> twice the code and solve the synchronization issues between the
> 2 trees. As
> I understand it, the synch problems were why they originally switched to
> the single in-island scene approach.

> - There were I/O blocking problems in a number of places. Biggest was

> when the initial scene snapshot was being loaded/saved into/from
> the island
> as even small scenes are big. This might have been partly
> mitigated by the
> planned (but never finished) move away from ImageSegments as the snapshot
> format.

Has something been done in that direction in OpenQwaq yet?

> - Proper platform multi-threading might be a solution to some of the
> issues. However ST dogma is that you don???t want multi-threading

> on platform
> threads at the ST level and the one ST I know of that does that is
> proprietary and Windows only.

I noticed that the classic suggestion is to use microservices running on the
same machine to load all the cores available. This can be done e.g. with
Docker or similar containerization running headless Smalltalk VMs.

> - Implementation Dependencies
> - Uses an old Squeak, needs porting to something with closures and

> the JIT. Do modern Squeaks even have the necessary Croquet VM support
> anymore? (replicable FP, etc.)

OpenCroquet runs on Squeak 5.2 and 6.x https://blog.krestianstvo.org/en/open-croquet-for-squeak-6/
Both Pharo and Squeak are based on Cog VM https://github.com/OpenSmalltalk/opensmalltalk-vm
so presumably.

> - HUD/In-Helmet UI uses TweakUI. I like Tweak, but apparently I???m the

Tweak is still not ported, afaik.

> only person besides Andreas who did. Parts of the TeaTime
> Islands are built
> on Tweak Islands.

> - It is using an old version of OpenGL that needs to be updated,

> probably changed to something WebGL/OpenGL ES compatible. A lot of the VM
> OGL support code is from the layers of prior attempts to do
> portable 3D in

> Squeak, so that???ll have to be rewritten if porting outside of Squeak.

Long-term the only option is Vulkan, but that's a massive can of worms.

> - External space chat uses Jabber/XMPP, which appears to have fallen

> out of favour and the replacements are mostly proprietary.
> Probably best to
> stick with it for now until a better open alternative arises.

Yeah, federated open protocols for p2p systems, please.

> - Because of the ST object model and how it is usually implemented,

> most STs and derivatives have an issue with memory bandwidth when running
> on a C optimized processor. It also makes it difficult to use
> shared memory
> to get around the lack of platform threads. You could write a bunch of
> C/C++ and interface to that, but why bother with ST then?

> - Matthew had some ideas and code for moving to a more modern Squeak.
> - The Xrfb system used to support in-world shared desktop

> applications was a crude implementation with serious security and
> performance issues, but the basic idea is solid. It should also be usable
> for non-Cobalt applications. I spent some time working on this
> external to
> OC and think that with modern containerization the backend could
> be rebuilt
> and be very solid.

> - After doing a number of personal projects in this area, I have
> concluded that Pharo, though an interesting and worthwhile project, isn???t

> Smalltalk and its historical version compatibility issues make it a bad
> platform for a low headcount fringe project and may need to
> shift platforms
> without a big rewrite.

This is somewhat surprising to me, and not in a good way. I hoped that the Pharo
platform and community would have been of help.

> - This isn???t saying I???m totally happy with the state of Squeak
> either. OTOH in the Squeak dev tools I can do a ???senders of?????? on a method

> name in the method list and it will do the right thing.

> - Scenegraph and Rendering
> - The scenegraph design is a late 90???s approach that matches well

> with the state of OpenGL at that time; Dave Smith did a good job
> but it is

> complex (not unreasonably so) and not well documented. It can???t use a lot

> of modern approaches and optimizations because the change
> patterns are less
> predictable than in games (which can be designed to make pre-calculation
> profitable).

> - There is a LOT of functionality in TObject, the root of the

> scenegraph node classes. If I recall correctly this is because Alan likes
> that kind of top focused design because he believe it to be better for
> manipulation and extension by less technical users (Morph has the same
> structure). This decision may need to be revisited.

> - The rendering architecture is complex, hard to debug, and not well

> documented. I think this is mostly because of the see-through portals. On

> the other hand, I haven???t needed to fix a lot of issues either.
> - Bug in skybox timing, it appears to be using the wrong ttime for
> rotation calculations when viewed through a portal. I???m betting this is

> part of a more general issue with the portal rendering code.

> - There is a memory race in the shader code, as there is a chance

> shader objects will get moved by the GC at the wrong time. If the version
> of OGL is updated, this just gets worse as modern OGLs rely much more on
> shaders.

> - The raycasting system to determine where the mouse is pointing is

> slow and interwoven with the rendering to avoid propagating messages into
> the island. The performance might be solvable by moving to an octree for
> indexing the mesh faces. Bounding geometry is tight spheres, it may be
> worthwhile to look into changes.

> - TeaTime
> - Cobalt still uses the ???Interim TeaTime??? because that worked well

> enough to ignore. The transactional approach of the full TeaTime
> has issues
> because the large amount of multi-user changes to the in-island
> scenegraphs
> leads to frequent commit conflicts, rollbacks, and retries, hurting
> real-time performance. Before he died, Andreas and I had some discussion
> about how to improve that situation for scaling, but that would
> be another
> research project.

What do you think of this approach https://blog.krestianstvo.org/en/krestianstvo-luminary-for-open-croquet-architecutre-and-virtual-world-framework-in-peer-to-peer-web/
?

> - The cryptography and protocol in the Island/Router communications

> was bad in 2012 and is hopelessly obsolete in 2019. Maybe try with SSL?
> Websockets? This is part of what Matthew looked into. OTOH, you may not
> care about security and this could be ignored.

Or you could just leave this part to the environment. I would use a wrapper
like Wireguard to protect the transport. There are also encrypting network overlays
that produce globally routable IPv6 addresses across either IPv4 or IPv6 as
transport.

> - Matthew had a lot of ideas and a bunch of code for improved TeaTime

> infrastructure as part of his thesis.

> - It would be nice to reexamine the NAT router hole punching code. It

> has been awhile, things have probably changed in this space, and UPnP is
> considered a security issue.

What is the IPv6 situation with Croquet/Cobalt? It could solve the NAT issue
fundamentally. I'm not sure inasmuch e.g. the Gun DB approach to reflectors
wouldn't also act as mediators. Since WebRTC session initiation is also a
robustly solved problem, though outside of Smalltalk ecosystem.

> - There is a garbage collection problem in how external references
> are managed in the island. When the project closed, I wasn???t

> sure about how
> to handle it.

> - I was pretty happy with the general WAN Island/Router discovery

> approach in OC and we had enough flexibility in the architecture to try
> experiments. There are a couple known issues in the SSDP implementation

> used for LAN discovery, but shouldn???t be too bad to fix. Ideally we???d

> interface to the platform SSDP, if such exists. It would also be
> nice to be
> able to use Bonjour/Zeroconf/DNS-SD service discovery as it is much less
> chatty.

> - The texture cache architecture (which is built on the island

[Alan Grimes]

Eugen Leitl wrote:
> What is the IPv6 situation with Croquet/Cobalt? It could solve the NAT issue
> fundamentally. I'm not sure inasmuch e.g. the Gun DB approach to reflectors
> wouldn't also act as mediators. Since WebRTC session initiation is also a
> robustly solved problem, though outside of Smalltalk ecosystem.

I have a 3rd world ISP called Verizon that only supports IP4. =((

(it's listed as number 5 in this table)
https://en.wikipedia.org/wiki/List_of_multiple-system_operators#Top_multichannel_video_service_providers_in_the_United_States_by_number_of_subscribers


--
Clowns feed off of funny money;
Funny money comes from the FED
so NO FED -> NO CLOWNS!!!

Powers are not rights.