macOS keyboard input within embedded browser

[Callum Prentice (Callum)]

I am taking a look at a slew of issues related to keyboard input within an embedded browser on macOS and could use some platform specific help.

Regular keyboard input i(within an HTML form for example) works as expected and the input methods for non-English characters characters such as katakana appear to do so as well.  However, any site that uses the lower level char/key/modifier codes does not and that is what I am trying to fix.

This site (apologies for the really annoying 90s chip tune) seems to be a good test case - https://www.lexaloffle.com/bbs/widget.php?pid=52365 and doesn't work inside an embedded browser instance.

Keyboard events arrive in the CEF media plugin via the "key_event" and "text_event" messages - the latter for unicode input. There are a large number of strange conditionals within the "key_event" processing ("tab key up: for example) but if I remove all of those, just send bare codes into CEF and remove the "text_input" code, everything works nicely - except the foreign language input.  

I've spent some time tracing back into the macOS specific ObjC specific code but quickly got lost - has anyone else looked at the same issue and made any progress?

Thank you!

--

CALLUM PRENTICE | Principal Engineer

LINDEN LAB | Create Virtual Experiences

[Dayturn]

I know this sounds painful, but my recommendation is to rewrite all keyboard input for the macOS version using Apple frameworks.

This is what you will have to do anyway if you want the viewer to survive the transition to Apple Silicone.

Cheers,

Geir Nøklebye

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.
To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAOx%2Bzh3DuzD3TyLNY9ueD6LMw2Q0%2BOFoS_PEiihHnq%3DQT4_XBA%40mail.gmail.com.

[Callum Linden]

Interesting and not the first time I’ve heard that expressed as an option but sadly not a possibility right now. 

I know there was a perfect effort some years back to switch from Carbon to Cocoa and as far as I know, that’s what we use now. Will that not be supported on the new Apple Silicon machines? 

On Oct 23, 2020, at 1:10 AM, Dayturn <geir.n...@dayturn.com> wrote:

I know this sounds painful, but my recommendation is to rewrite all keyboard input for the macOS version using Apple frameworks.

[mayuka Yoshikawa]

Hi,

I haven't tried Mac OS lately, yes often see odd curious work? on input method on Mac/Linux. Now using SL on CentOS, btw :-)

And I haven't used Catalina because there are much more OS-related issues(bugs) by Apple, and I would love to hear you have evaluated Big-Sur on Apple Silicon (perhaps ARM-A8 based), it is going well? and we still have to use on Rosetta? Apple will release new ARM-based Macbook (Pro) next month, I am very much wondering how SL works.

-May

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/2C974DFB-5A4C-4C4B-994F-EDDCE79630EC%40lindenlab.com.

[Callum Prentice (Callum)]

Yes, I am personally looking forward to the new ARM/AS based but we haven't started to investigate what is involved in making the Viewer work on them.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/b6a6a965-0f59-1494-06bc-d9f356268d58%40gmail.com.

[mayuka Yoshikawa]

Thanks information, anyway I will purchase new Macbook because my Mac(book/book pro/mini)s are really old so I am looking forward to checking SL on Rosetta :-)

-May

[Geir Nøklebye]

When it comes to Carbon, the viewer is still dependent on it, so the switch to Cocoa is not complete:

llwindowmacosx-cpp

llwindowsmacosx.h

llappdelegate-objc.mm

llappviewermacosx.cpp

llfilepicker.h. 

All of these can be eliminated (as I have done) with the exception of llwindowmacosx-cpp. In addition llcommon is linked with Carbon. 

Apple announced multiple times before Catalina that Carbon would not be ported to 64-bit, but anyway for macOS 10.15 (Catalina) they silently went ahead and ported it to 64-bit without any announcement or documentation.  – Presumably because one of the big ISVs (and we all know who that is) did not have their code de-carbonated in time for the Catalina launch. Without this, the viewer as-is would not run on Catalina and the upcoming macOS 11 (Big Sur). 

It is unlikely that Carbon will be ported to run the ARM instruction set, but will run in Intel emulation under Rosetta 2 for an interim period.

---

Apple have clarified (per my request) that OpenGL will be supported initially on Apple Silicon in order not to break too many applications during the transition period which they have stated is 2 years. It is still unclear if the OpenGL Legacy profile the viewer depends on will be supported, as it might be difficult to implements on Apple’s tile based GPU that is integrated on the Apple Silicone chip. That would leave us with support for OpenGL 3.2 only.

They have further stated that there will be no support for GPUs from 3rd party developers or initial support for e-GPUs. It is also highly unlikely Apple will sign a system level driver for Vulkan.

 

We can expect OpenGL (deprecated) to be supported for as long as Apple ships Intel based systems which will be for the transition period of 2 years, but depending on how well the transition progresses, this period might be cut shorter like it was during the PowerPC to Intel transition where the period in reality was less than a year. 

They have stated developers must move their OpenGL applications to Metal, and Metal is only supported with Objective-C and Swift. 

It is not possible to write applications  in C++ (or any other language) that directly interface with the Metal frameworks and libraries. So for the viewer to use a Metal renderer Objective-C++ and modules must be enabled for the code base. Currently the large number of BOOL in the code prevents that from happening. Alternatively a complete rewrite in Swift is required.

How much of the low level access to macOS the viewer currently use, and which originates in Carbon, will be supported is at this stage unclear. Some developers with cross platform code bases have reported they can’t even compile their libraries on Apple Silicon. 

What is clear with macOS 11 (Big Sur) and Xcode 12 is that Apple is merging App-Kit and UI-Kit (iOS) into one new framework; Swift UI which makes it possible to write universal applications that both run on macOS and iOS from the same code base.  This is Apple’s stated direction, and with that follows an even stronger commitment to Swift and deprecation of Objective-C++ for system frameworks. 

---

We will most likely see the first Apple Silicone systems shipped at the end of the year after an announcement in November. If the viewer will run on those systems with Rosetta 2 emulation is anyones guess, but if it does, it most likely will be rather sluggish.  

At WWDC 2020 Apple showcased an Intel based game running on Rosetta 2, but it has later been revealed that this demo ran at both the lowest resolution and lowest quality setting, in addition to the game engine already using Metal. 

In my testing the current SecondLife viewer runs fine on macOS 11 beta (Big Sur) on Intel based systems – even gaining a slight increase in FPS over macOS 10.15 (Catalina) on the same hardware. Other developers and beta testers also report improved performance on Big Sur for Intel – which could be attributed to the complete removal of 32-bit system level code that still exist in Catalina. 

It can also be noted that there will be no support for virtual machines emulating Intel on Apple Silicon like SoftPC/VirtualPC did on the PowerPC based Macs. Meaning there is no way of running Intel based Windows on these systems in a VM. The Parallels demo of Linux running in a VM on Big Sur at WWDC 2020 was an ARM compiled version of Linux. 

Geir, 

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/2C974DFB-5A4C-4C4B-994F-EDDCE79630EC%40lindenlab.com.

[Callum Linden]

> When it comes to Carbon, the viewer is still dependent on it, so the switch to Cocoa is not complete:
> llwindowmacosx-cpp
> llwindowsmacosx.h
> llappdelegate-objc.mm
> llappviewermacosx.cpp
> llfilepicker.h.
> All of these can be eliminated (as I have done) with the exception of llwindowmacosx-cpp. In addition llcommon is linked with Carbon.
>
> Apple announced multiple times before Catalina that Carbon would not be ported to 64-bit, but anyway for macOS 10.15 (Catalina) they silently went ahead and ported it to 64-bit without any announcement or documentation. – Presumably because one of the big ISVs (and we all know who that is) did not have their code de-carbonated in time for the Catalina launch. Without this, the viewer as-is would not run on Catalina and the upcoming macOS 11 (Big Sur).

Interesting. I had a feeling the Carbon -> Cocoa transition was incomplete but since the viewer still seemed to work on recent versions of macOS, I assumed I was mistaken.

> Apple have clarified (per my request) that OpenGL will be supported initially on Apple Silicon in order not to break too many applications during the transition period which they have stated is 2 years. It is still unclear if the OpenGL Legacy profile the viewer depends on will be supported, as it might be difficult to implements on Apple’s tile based GPU that is integrated on the Apple Silicone chip. That would leave us with support for OpenGL 3.2 only.

Yep. We were aware of the impending OpenGL deprecation - sounds like the switch to ARM will make the work to deal with it larger in scope and even more of a priority.

> It is not possible to write applications in C++ (or any other language) that directly interface with the Metal frameworks and libraries. So for the viewer to use a Metal renderer Objective-C++ and modules must be enabled for the code base. Currently the large number of BOOL in the code prevents that from happening. Alternatively a complete rewrite in Swift is required.
>
> How much of the low level access to macOS the viewer currently use, and which originates in Carbon, will be supported is at this stage unclear. Some developers with cross platform code bases have reported they can’t even compile their libraries on Apple Silicon.
>
> What is clear with macOS 11 (Big Sur) and Xcode 12 is that Apple is merging App-Kit and UI-Kit (iOS) into one new framework; Swift UI which makes it possible to write universal applications that both run on macOS and iOS from the same code base. This is Apple’s stated direction, and with that follows an even stronger commitment to Swift and deprecation of Objective-C++ for system frameworks.
>
> ---
>
> We will most likely see the first Apple Silicone systems shipped at the end of the year after an announcement in November. If the viewer will run on those systems with Rosetta 2 emulation is anyones guess, but if it does, it most likely will be rather sluggish.
>
> At WWDC 2020 Apple showcased an Intel based game running on Rosetta 2, but it has later been revealed that this demo ran at both the lowest resolution and lowest quality setting, in addition to the game engine already using Metal.
>
> In my testing the current SecondLife viewer runs fine on macOS 11 beta (Big Sur) on Intel based systems – even gaining a slight increase in FPS over macOS 10.15 (Catalina) on the same hardware. Other developers and beta testers also report improved performance on Big Sur for Intel – which could be attributed to the complete removal of 32-bit system level code that still exist in Catalina.
>
> It can also be noted that there will be no support for virtual machines emulating Intel on Apple Silicon like SoftPC/VirtualPC did on the PowerPC based Macs. Meaning there is no way of running Intel based Windows on these systems in a VM. The Parallels demo of Linux running in a VM on Big Sur at WWDC 2020 was an ARM compiled version of Linux.

Fascinating - thank you for all that information. Sounds like there is a lot to tackle on the macOS side and I expect it will be more of a priority once the uplift project is closer to completion and resources become available.

[Geir Nøklebye]

The issue with removing Carbon from llwindowsmacosx.cpp has to do with a function used by the color picker there does not exist any equivalent for in Cocoa.
Most macOS applications updated to use the system color picker, but the last time I researched this, it does not work at all with the OpenGL legacy profile, so you’d have to update to the OpenGL 3.2 (standard) profile to even use it.

There are some applications that don’t use the system color picker interface, such as Affinity Photo, which also is a cross platform application, but it is not open source, and it also use the OpenGL 3.2 profile so most likely it use the system framework implementing their own interface on top of it.

So that is actually a bit of a pickle.

[Henri Beauchamp]

On Sun, 25 Oct 2020 21:54:45 +0100, Geir Nøklebye wrote:

> The issue with removing Carbon from llwindowsmacosx.cpp has to do with
> a function used by the color picker there does not exist any equivalent
> for in Cocoa.

The system color picker is not even used by the viewer (well, it could
be, but I'm still to hear about someone using it): we have got the XUI
color picker for that purpose... In fact, I removed the system color
picker code from my own viewer sources years ago.

I'm not a macOS specialist (I know about nothing Apple-related), but I
do not see any color picker related piece of code in llwindowsmacosx.cpp.

The only other mentions of Carbon I see in the sources are (with a
simple 'grep -ri carbon *' in linden/):

- indra/llcommon/CMakeLists.txt: find_library(CARBON_LIBRARY Carbon)
- indra/llcommon/CMakeLists.txt: target_link_libraries(llcommon ${CARBON_LIBRARY})

These could be moved (nothing "carbonized" any more in llcommon) into
indra/newview/CMakeLists.txt (until newview gets decarbonized too).

- indra/llplugin/slplugin/slplugin-objc.mm: // This is a bit of voodoo taken from the Apple sample code "CarbonCocoa_PictureCursor":
- indra/llplugin/slplugin/slplugin-objc.mm: // http://developer.apple.com/samplecode/CarbonCocoa_PictureCursor/index.html
- indra/llplugin/slplugin/slplugin-objc.mm: // Needed for Carbon based applications which call into Cocoa
- indra/llplugin/slplugin/CMakeLists.txt: # Mac version needs to link against Carbon

These may be the actual problem to remove Carbon entirely: are any
plugins/CODECs still using it (CEF, VLC, gstreamer and all the CODECs
they use) ?...

- indra/llwindow/llwindowmacosx.h: BOOL mHandsOffEvents; // When true, temporarially disable CarbonEvent processing.

This (and its initialisation to FALSE in llwindowmacosx.cpp) can be
removed entirely (a left out from old code: not used any more).

- indra/llwindow/llwindowmacosx.cpp:#include <Carbon/Carbon.h>
- indra/llwindow/llwindowmacosx.cpp: // Voodoo for calling cocoa from carbon (see llwindowmacosx-objc.mm).
- indra/llwindow/llwindowmacosx.cpp: // This largely mirrors the old implementation, only sans the carbon parameters.
- indra/llwindow/llwindowmacosx.cpp: // By registering a global CarbonEvent handler for mouse move events, we ensure that
- indra/llwindow/llwindowmacosx.cpp: // By registering a global CarbonEvent handler for mouse move events, we ensure that

llwindow is not even linked against Carbon any more (and does not
need it): the #include can safely be removed, as well as the
deprecated comments.

- indra/newview/llappviewermacosx.cpp:#define LL_CARBON_CRASH_HANDLER 1
- indra/newview/llappviewermacosx.cpp:#ifdef LL_CARBON_CRASH_HANDLER
- indra/newview/llappviewermacosx.cpp:#include <Carbon/Carbon.h>

These can be removed too: the define only causes the #include to be
performed, but the viewer compiles without it since no code in there
uses Carbon any more...

- indra/newview/macview.r:#include <Carbon/Carbon.r>

Not even sure what macview.r is all about... I don't have it in my
viewer (removed it years ago, without side effect)...

- indra/newview/llappdelegate-objc.mm:#include <Carbon/Carbon.h> // Used for Text Input Services ("Safe" API - it's supported)

This one is *still in use*, by the languageUpdated method (in the
first two lines of that method)... It would need a Carbon-less
equivalent.

- indra/newview/llfilepicker.h:#include <Carbon/Carbon.h>

Probably something needs to be done here (perhaps did you want to
mention the system file picker and not the color picker) ?

Personally, I solved this problem years ago: I wrote an XUI based file
picker that works for all OSes, is natively non-blocking (no need for
complex and hacky threading scheme such as with GTK2), and allowed me
to remove the dependency on Carbon (Mac) and GTK2 (Linux)...

- indra/newview/macview_Prefix.h:// MBW -- This doesn't work. There are some conflicts between things in Carbon.h and some of the linden source.
- indra/newview/macview_Prefix.h://#include <Carbon/Carbon.h>

Just comments to remove.

- indra/newview/Info-SecondLife.plist: <key>LSRequiresCarbon</key>

Totally useless, I think... I don't have that line in the plist
file for my viewer, without side effect.

On Sat, 24 Oct 2020 09:20:01 +0200, Geir Nøklebye wrote:

> We will most likely see the first Apple Silicone systems shipped at
> the end of the year after an announcement in November. If the viewer
> will run on those systems with Rosetta 2 emulation is anyones guess,
> but if it does, it most likely will be rather sluggish.

I suppose you are kidding, right ?... There is no way the viewer will
run at an acceptable speed if the x86 code needs to be emulated on an
ARM CPU (even with on-the-fly code translation and reuse)... Not to
mention OpenGL !

I see only two possibilities for ARM-Macs and SL viewers:

1.- The rewrite of the renderer for Vulkan (in the hope that a Vulkan
driver or "translator" will be available some day for those Macs)
together with the addition of Neon maths support (to replace the
SSE2 maths used in the viewer, when compiled for ARM).

2.- Using the Linux virtualization capability Apple announced, which
still involves the steps taken in 1, but with the added hope,
should Apple fail to provide Vulkan under macOS, that the hardware
virtualization for Linux will allow to run genuine Vulkan applications
under the latter... This means LL will at least have to resume Linux
support as well. :-P

Note that the work to achieve compatibility with future ARM-based Macs
is HUGE. Seeing how LL threw away Linux support on the pretext it took
them too much time (I'm still laughing at that stupid argument), I am
rather sceptical about their involvement in a Mac-ARM port...

Henri.

[Geir Nøklebye]

@Henry:

If you for once would step outside your nerd rage every time someone mentions Apple, you would discover that nobody has claimed the system color picker is used by any viewer, but it is the route every other modern macOS app have taken after removing Carbon from the viewer. For iOS it is in reality the only route. 

Just because you don’t see anything in your train wreck of a Mac viewer code for the color picker in llwindowsmacosx.cpp does not mean it does not exist in other viewers. 

Specifically you have two lines of code with Carbon dependency:

NColorPickerInfo  info;     http://mirror.informatimago.com/next/developer.apple.com/documentation/Carbon/Reference/Color_Picker_Manager/color_picker_mgr_ref/data_type_7.html

info.placeWhere = kCenterOnMainScreen;    The documentation for kCenterOnMainScreen has been scrubbed from the Apple developer site

The rest of your rant is more or less irrelevant, but the for the general audience:

The dependency  in llfilepicker.h  can easily be fixed like in this commit. 

The dependency in llappviewermacosx.cpp is fixed in this commit.

The dependency in llappdelegate-objc.mm is fixed in this commit. 

[Henri Beauchamp]

On Mon, 26 Oct 2020 08:13:04 +0100, Geir Nøklebye wrote:

> @Henry:
>
> If you for once would step outside your nerd rage every time someone
> mentions Apple,

There is not a single bit of "rage" or "rant" towards Apple in my
message !!! O.O

I'm only being factual, so please do calm down yourself from your nerd
fanaticism towards Apple ! :-D (see how easy it is to put badly inferred
emotions behind people neutral written words ?)

> you would discover that nobody has claimed the system
> color picker is used by any viewer, but it is the route every other
> modern macOS app have taken after removing Carbon from the viewer.
> For iOS it is in reality the only route.
>
> Just because you don’t see anything in your train wreck of a Mac
> viewer code for the color picker in llwindowsmacosx.cpp does not
> mean it does not exist in other viewers.
> Specifically you have two lines of code with Carbon dependency:
>
> NColorPickerInfo info;

I really do not understand your worry about that code and around
the color picker: I removed that code in my viewer, and nobody ever
complained they could not use the system color picker... Again, we
got an XUI one, that does a better job and is uniform over all OSes.

The LLWindowMacOSX::dialogColorPicker() method (and anything using
it) can simply be removed to avoid the dependency on Carbon.

> The rest of your rant is more or less irrelevant,

Again not a rant: just a factual list of the current traces of Carbon
in LL's viewer code (there are even less traces in mine).

> but the for the general audience:
>
> The dependency in llfilepicker.h can easily be fixed like in
> this commit

> The dependency in llappviewermacosx.cpp is fixed in this commit

> The dependency in llappdelegate-objc.mm is fixed in this commit

Excellent ! I'll reuse those myself, with due credits for their
author.

If I'm not mistaken, this leaves us with the SLPlugin dependency on
Carbon.

Henri.

[fuer...@gmx.net]

> step outside your nerd rage

There was no ranting, Henri was being very helpful. And I'm thankful for his much appreciated input, that he has been giving in great detail over and over again in this list for 10 years at least, for sure.
Just my two cents.

MartinRJ

-----Ursprüngliche Nachricht-----
Von: opensou...@lists.secondlife.com <opensou...@lists.secondlife.com> Im Auftrag von Henri Beauchamp
Gesendet: Montag, 26. Oktober 2020 09:35
An: opensource-dev <opensou...@lists.secondlife.com>
Betreff: Re: [opensource-dev] macOS keyboard input within embedded browser

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20201026093515.97333ef0392598cd48bbc0de%40free.fr.

[Henri Beauchamp]

Sadly,while your commit does remove the dependency on Carbon *headers*
for the compilation of llappdelegate-objc.mm, the viewer still needs
to be linked against the Carbon library, else I get at link time:

: "_TISCopyCurrentKeyboardInputSource", referenced from:
: -[LLAppDelegate languageUpdated] in llappdelegate-objc.o

: "_TISGetInputSourceProperty", referenced from:
: -[LLAppDelegate languageUpdated] in llappdelegate-objc.o

: "_kTISPropertyInputSourceLanguages", referenced from:
: -[LLAppDelegate languageUpdated] in llappdelegate-objc.o

: Symbol(s) not found for architecture x86_64
: Linker command failed with exit code 1 (use -v to see invocation)

The languageUpdated method code needs more work done on it to
remove the Carbon calls (if at all possible)...

Henri.

[Geir Nøklebye]

Yes, I already said that it needs to be linked with the Carbon library, which happens in llcommon, because as you discovered, there is a number of low level calls to the operating system being used throughout the code that originates from Carbon. Some might be replaced by POSIX calls to the embedded BSD system. Other will possibly need Foundation, but then you hit the BOOL issue face on.

I have not researched the full effect of not linking at all with Carbon.

I am a bit unsure how updated the BSD system actually is. For Catalina they switched to zsh as the default shell because changes in the bash licensing left them with a very updated version (or open source the system). The same happened with the majority of the open-source code they used for macOS server, so they just killed it like that.


Geir,

> --
> Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
> ---
> You received this message because you are subscribed to the Google Groups "opensource-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

> To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20201026100852.c229f064424d419475f9c42c%40free.fr.

[Geir Nøklebye]

>
> If I'm not mistaken, this leaves us with the SLPlugin dependency on Carbon.
>

There should not be any Carbon dependency at all in SLPLugin. That is a separate app bundle with its own linking which just happens to be copied into main viewer app bundle, and invoked from there.

The LL source and also mine has all the Carbon code removed or commented out for the SL Plugin. It should only use headers from Cocoa and AppKit.

Of course it should be completely rewritten as a proper macOS plugin, but that is a completely different discussion.

You can check if the SLPlugin use Carbon with the lsof command while it runs.
In comparison lsof on the viewer will find /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Extras2.rsrc is open and in use.

[Henri Beauchamp]

On Mon, 26 Oct 2020 13:11:06 +0100, Geir Nøklebye wrote:

> > If I'm not mistaken, this leaves us with the SLPlugin dependency on
> > Carbon.
>
> There should not be any Carbon dependency at all in SLPLugin.

Indeed... I removed the linking against Carbon in SLPlugin (and it must
be noted I do not link llcommon either against Carbon in my viewer: more
on that below), and SLPlugin did link just fine...

The only mention of Carbon is in indra/llplugin/slplugin/slplugin-objc.mm:

// Needed for Carbon based applications which call into Cocoa

NSApplicationLoad();

So I was afraid NSApplicationLoad() was doing low level calls to Carbon,
thus why SLPlugin would be linked against Carbon in its CMakeLists.txt.

But no: all mentions of Carbon (find_library(CARBON_LIBRARY Carbon) and
${CARBON_LIBRARY} occurrences) can indeed be removed entirely from
indra/llplugin/slplugin/CMakeLists.txt. :-)

> You can check if the SLPlugin use Carbon with the lsof command while
> it runs.

Nope, I do not have any bare metal Mac to test it on. I'm just using a
VirtualBox VM under Linux with Catalina + Xcode 11 (to verify my viewer
at least builds fine under macOS), but I cannot start the resulting
viewer binary on it, by lack of OpenGL drivers for macOS VBox VMs...

On Mon, 26 Oct 2020 12:41:44 +0100, Geir Nøklebye wrote:

> Yes, I already said that it needs to be linked with the Carbon library,
> which happens in llcommon,

The linkage against Carbon in llcommon is the wrong place to do it
(there is no call done to Carbon in llcommon itself)...
For my viewer, I link against Carbon only at the application level:
SLPlugin (now removed since it was not needed in the first place, as
you properly diagnosed), and the viewer binary itself.

I'd recommend (as I already wrote in my first message) to move the
linkage directives for Carbon from llcommon/CMakeLists.txt to
newview/CMakeLists.txt.

> because as you discovered, there is a number of low level calls to
> the operating system being used throughout the code that originates
> from Carbon. Some might be replaced by POSIX calls to the embedded
> BSD system.

That would be a nice thing to do to get rid of that last dependency
in llappdelegate-objc.c...

> Other will possibly need Foundation, but then you hit the BOOL issue
> face on.

Got rid of all (*) BOOLs years ago in my viewer.. . So if you want to
test a Foundation-based solution, I'd say that my viewer sources would
be a good base for it. There is just the "typedef S32 BOOL;" remnant
left in llcommon/stdtypes.h, but you can safely remove it on the
condition to change line 50 of llwindow/llpreeditor.h to read:
#if LL_DARWIN
// Darwin's clang does not know how to convert vector<int> to vector<bool> !
typedef std::vector<S32> standouts_t;
#else
typedef std::vector<bool> standouts_t;
#endif

Henri.

(*) Excepted where they do make sense, such as for some Windows and SDL
libraries calls.

[Dayturn]

On 26 Oct 2020, at 15:19, Henri Beauchamp <sl...@free.fr> wrote:

Yes, I already said that it needs to be linked with the Carbon library,
which happens in llcommon,

The linkage against Carbon in llcommon is the wrong place to do it
(there is no call done to Carbon in llcommon itself)...
For my viewer, I link against Carbon only at the application level:
SLPlugin (now removed since it was not needed in the first place, as
you properly diagnosed), and the viewer binary itself.

I'd recommend (as I already wrote in my first message) to move the
linkage directives for Carbon from llcommon/CMakeLists.txt to
newview/CMakeLists.txt.

Actually, for Xcode it does not matter where in the app bundle the library is referenced for linking, because if referenced once it will be available everywhere in the project (for the Objective-C code that is).

I guess it is done in llcommon as that is the first build target of the viewer project, so referencing it there ensures cmake shoves into the project and can be seen by all of the project build targets. (Note SLPlugin is a separate build project that is linked separately, as is the mac-crash-logger).

The C++ code, does however, have to have the linking set up properly per build target in the cmake files.

If you in your VM Finder right click on the CoolVLViewer1.28.X app you can open it with the Show Package Content from the pop up menu to see the structure. As you will see there is not separate libs like llcommon, llrender etc, but just one executable under the MacOS directory. 

So everything in the viewer build project is mashed up in that executable. 

Externally referenced libraries are in the Resources directory, but that is not really where they should be and creates issues for signing and notarization of the viewer. This is also where the SLPlugin and mac-crash-logger app bundles are placed, but that is also not ideal for signing reasons. In addition it necessitates some crazy sym-linking.

In reality the macOS version of the viewer should be completely redone as a proper Xcode project with the expected folder structure. Then you could more or less entirely remove the py build files (manifest etc) and not have to generate the project with cmake that results in all kinds of oddities all the time. It would also generate a proper installer and simplify signing and notarization a lot.

That also makes it much easier to generate a universal viewer with both Intel and ARM binaries in the app bundle. 

because as you discovered, there is a number of low level calls to
the operating system being used throughout the code that originates
from Carbon. Some might be replaced by POSIX calls to the embedded
BSD system.

That would be a nice thing to do to get rid of that last dependency
in llappdelegate-objc.c...

Other will possibly need Foundation, but then you hit the BOOL issue
face on.

Got rid of all (*) BOOLs years ago in my viewer.. . So if you want to
test a Foundation-based solution, I'd say that my viewer sources would
be a good base for it. There is just the "typedef S32 BOOL;" remnant
left in llcommon/stdtypes.h, but you can safely remove it on the
condition to change line 50 of llwindow/llpreeditor.h to read:
#if LL_DARWIN
// Darwin's clang does not know how to convert vector<int> to vector<bool> !
typedef std::vector<S32> standouts_t;
#else
typedef std::vector<bool> standouts_t;
#endif

I have added a file to llcommon: llbool.h and removed the BOOL, FALSE and TRUE definitions from stdtypes.h in order to be able to be very specific where that BOOL is used. At this time I have not gotten sufficiently far in converting BOOLs for it to be very useful but getting there…

I’ll have a closer look at the consequences of not linking with Carbon, but it might be a tougher call in the short term.

On top of that I discovered Apple has completely removed (or hidden) all Carbon documentation from developer.apple.com – probably the only way to force Adobe to not use Carbon :-). I guess one can still scour the Carbon header files…

I am going to yell at them in the developer forum as well over that. :-O

Henri.

(*) Excepted where they do make sense, such as for some Windows and SDL
libraries calls.

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20201026151927.06deeba17beb7cea0861145d%40free.fr.

[Henri Beauchamp]

On Mon, 26 Oct 2020 20:04:27 +0100, Dayturn wrote:

> Actually, for Xcode it does not matter where in the app bundle the
> library is referenced for linking, because if referenced once it
> will be available everywhere in the project (for the Objective-C
> code that is).
> I guess it is done in llcommon as that is the first build target
> of the viewer project, so referencing it there ensures cmake shoves
> into the project and can be seen by all of the project build targets.

Not really, no... llcommon used to be a shared/dynamic library (consumed
by the viewer and SLPlugin). That's why it had many dependencies linked
directly to it. Nowadays, it is a static library and does not need to
be linked individually against any library it consumes, unless they are
themselves static libraries that are not used anywhere else in the
whole project.

The linking to the dynamic libraries is best done at the executable
level (this way, the linker got less work to do). This is even more
true of static libraries used everywhere in the project (such as
boost or APR ones, for example): this way you ensure only one copy
of each static library function in use by the project libraries/
components will be included in the final binary.

Finally, llcommon does not make any use of Carbon. Linking against
it does not make any sense (and is in fact a no-op, as far as ld is
concerned).

The fact that you see the Carbon dependency "contaminating" all the
other libraries and executable linking to llcommon is simply due to
the fact that in the CMakeLists.txt for llcommon, the Carbon library
is (wrongly) added to the target_link_libraries for it.

The cmake files in LL's viewer would need quite a good cleanup (I
partly did it in my viewer, but still got work to do on it).

> In reality the macOS version of the viewer should be completely
> redone as a proper Xcode project with the expected folder structure.

I beg to differ...

This is cmake's business to do that: the viewer being a cross-
platform software, it should not (and actually cannot) follow any
pre-determined proprietary layout (such as BOTH for Visual Studio and
Xcode). Instead, cmake's generators ensure that a proper laid out
build tree is produced to build the viewer under the various platforms
and tools/compilers.

The problem you are (rightfully) noticing about the Xcode project
bad build tree layout, is that the cmake generator for Xcode did not
yet get converted to the newer Xcode versions (there have been some
communication problems between Apple and cmake's team, apparently)
but it is slowly getting worked out...

> Then you could more or less entirely remove the py build files
> (manifest etc) and not have to generate the project with cmake
> that results in all kinds of oddities all the time. It would also
> generate a proper installer and simplify signing and notarization
> a lot.

A lot more could be done with just cmake. It's just a choice from
the original project maker to use more or less Python (or any other
language) scripting.

Henri.

[Geir Nøklebye]

Carbon is not a single library that is being linked with, but it is a framework living in /System/Library/Frameworks/Carbon.framework, which again has 9 sub frameworks.

So when you think cmake links Carbon to the llcommon build target, what you in reality do is associate Carbon with the project, meaning you make the full top level Carbon framework available to the entire project, and then the build system will make the entry points into the portions of the framework referenced by your code, specifically by the #include <Framework_name/Header_filename.h> statements dispersed throughout.

https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Tasks/IncludingFrameworks.html

[Dahlia Trimble]

I've been hacking a bit on Apple platforms lately and thought I'd share some of what I've found, which aren't really related to input issues with the viewer and CEF but are related to tangential points I've seen in this thread:

Google has recently merged MetalAngle into Angle which provides OpenGL ES 2.0 and much of ES 3.0 on top of Metal. I've tried it with a OpenGL ES 2.0 project I have and it worked on both my mac and iPad. I'm not that familiar with the SL viewer code but I was under the impression that most of the OpenGL used is from a 2.1 profile (please correct me if I'm wrong here) and this should map to ES 3.0 for the most part with relatively few changes. It's probably not an optimal solution but might get a native viewer running on Apple Silicon and buy you some time until a better solution can be worked out. I'm not yet aware of any App Store apps using this yet but I don't know of any reasons why it wouldn't be accepted (famous last words).

https://github.com/google/angle

Vulkan is available on macos and iOS through MoltenVK which is supported by Khronos and LunarG and is part of the Vulkan SDK which they publish. I've used it and it seems to work fairly well on my 2014 Mac mini running Catalina, and my iPad mini running iPadOS 14.0.1. It's supposedly not 100% compatible with the full Vulkan spec but I have not yet run into any problems due to API completeness.

https://www.lunarg.com/vulkan-sdk/

There is a c++ wrapper for Metal which is supposedly feature commplete but I haven't tried it (yet).

https://github.com/naleksiev/mtlpp

-Dahlia

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/5D94FEC4-089A-4147-880A-5C9502F7F7F5%40dayturn.com.

[Henri Beauchamp]

On Tue, 27 Oct 2020 01:34:55 -0700, Dahlia Trimble wrote:

> I'm not that familiar with the SL viewer code but I was under the
> impression that most of the OpenGL used is from a 2.1 profile

OpenGL v2.1 is the strict minimum to run the viewer, but it is also
largely insufficient by today's standards, and results in many
rendering glitches and slow downs, with many shaders unavailable
or degraded.

I'd personally not run the viewer any more against such a low level
OpenGL driver...

You can get a 'feel' of it by installing a VirtualBox Linux VM and
running a viewer on it (a Windows VM won't even allow to run the
viewer). The result is... Eeeeek ! :-D

Henri.

[Geir Nøklebye]

Thanks Dahlia,

There are a lot of work arounds and “hacks” one can do to make things work short term, but if I were in LL’s shoes - being a commercial business and all, I would seek to use the Apple developer tools as an entry point not only into Macs, but perhaps much more importantly into iOS. Apple’s stated direction, with the first proof of the pudding in Xcode 12, is the ability to build universal applications that will run on both iOS and macOS from the same build and installer. 

This opens a vast new market opportunity for Linden Lab, and you might of course fear the iOS devices are underpowered, but they will in essence have the same Apple Silicon processors with the same GPUs as their macOS siblings. Current benchmarking shows the iPads have comparable performance to MacBook Pro configurations.

My biggest fear of course, in the above scenario, is that the backend does not scale at all if you let a horde of iOS users loose on it. 

Geir, 

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAAQTD4WqtkT%2BSKTeKE4hfuhZrYGLoaPpE%2BmK3bTTWce7J8Y1Eg%40mail.gmail.com.

[Dahlia Trimble]

I think if you run a linux VM in virtualbox you get the llvmpipe version of mesa which is a software emulator. You can tell by looking at the Help->About dialog in the viewer and if it says llvmpipe or vmware for the OpenGL driver. It's actually quite fast for a software emulator but only provides core profile 3.1 I believe.

The Metal version of Angle should provide such nice shader goodies as multiple render targets from ES 3+ so should work with the "Advanced Lighting" settings in the viewer, but yeah I kinda agree that a newer profile, at least 4+ would be a better choice and preferably 4.6 if the viewer would use it, but then again perhaps a Vulkan rewrite would be time better spent. ;)

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20201027100010.10eaf4c98af42a424b0c04cd%40free.fr.

[Dahlia Trimble]

Geir I certainly agree with you that using Apple's preferred APIs and tools directly would be a worthy choice for a viewer for Apple platforms. I'm not aware, though, of LL's current business practices and their available engineering talent and priorities. I spent most of my software development careen in a company 3-4 orders of magnitude ($ wise) larger than LL but we still had many constraints working against us and compromises were often chosen.

I

I also agree that modern mobile platforms, especially Apple's, are built on very capable hardware. E.g., I can run my emscripten viewer in Safari on my 5 year old iPad mini 4 and fly around a 5000 prim region and get FPS of 30 or more. Granted the rendering isn't quite as pretty as what a regular viewer can do but you can only get so far with WebGL 1.0 (Apple still doesn't support 2.0 in Safari) but it's still very impressive and I'm sure newer hardware is a lot better. My only complaint about iOS devices is Apple is rather stingy with RAM and SL/OpenSim regions tend to have a *lot* of big textures, but I'm sure some clever asset management could be conjured up which would help to mitigate any RAM problems.

One problem with LL's code and iOS would seem to be the LGPL license which I hear isn't all that compatible with the APP Store. LL could probably get around this since they own the source code and could publish without the LGPL restrictions. However, IANAL and there may be ways that TPV developers could use some of LL's code and still publish for iOS.

Anyway, apologies to the list for diverting so much from the original topic.

-Dahlia

[Henri Beauchamp]

On Tue, 27 Oct 2020 02:17:31 -0700, Dahlia Trimble wrote:

> I think if you run a linux VM in virtualbox you get the llvmpipe version of
> mesa which is a software emulator. You can tell by looking at the
> Help->About dialog in the viewer and if it says llvmpipe or vmware for the
> OpenGL driver. It's actually quite fast for a software emulator but only
> provides core profile 3.1 I believe.

It would depend on the Linux distro. On my newest build system
(Rosa 2016.1 (*)), I have:

Graphics card vendor: VMware, Inc.
Graphics card: Gallium 0.4 on llvmpipe (LLVM 3.9, 256 bits)
OpenGL version: 2.1 Mesa 13.0.4

But I also got an old notebook computer based on an Althon64 and ATI
Mobility Radeon 9700, that also runs OpenGL v2.1 under Mesa (20.2):
on that bare metal system, rendering glitches are even worst, especially
with rigged meshes skinning since there is no usable shader for them,
meaning they are badly/wrongly lit/shaded.

Henri.

(*) build system must be kept "old enough" so that the glibc/libstdc++
requirements are not too high and the built binaries will run on all
current Linux systems in use by your user base...

[Geir Nøklebye]

Just a short reply on RAM on iOS devices: current shipping iPhone and iPad models have from a minimum of 3 GB to max 6 GB for applications.

The latest shipping models have a GPU that share memory directly with the CPU so there is no need to copy between them. It is also tile based and it therefore diverts quite a bit from other GPUs in the market. What consequence that exactly has for a renderer for SL is way above my competence level, but I have seen a few comments that Metal is not necessarily great with dynamic content. 

In Apple’s to Metal migration documents they say the first step for an existing OpenGL application is to migrate to OpenGL 3.2 before starting to work on a Metal version. Otherwise it tends to be complex. Such a migration would benefit any viewer platform regardless.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAAQTD4VLzrOWP%2Bey5dkPZ2XSm0MxkVnx1qJVRb5arV3iNJ%3DAyQ%40mail.gmail.com.

[Tonya Souther]

(Oops. Sent this to Geir directly at first.)

As I understand it, the viewer uses OpenGL 2.1 ...but while Windows and Linux versions of the OpenGL drivers from the GPU makers also implement OpenGL 4 and that lets them gain speed improvements, the Apple version is strictly 2.1, with nothing later enabled if you ask for the 2.1 profile.

This is why performance on the Mac sucks rocks. I used to run two Mac Pros side by side, a dua-hex 3.3 GHz and a dual-quad 2.9 GHz, both with at least 48 GB of RAM and decent graphics cards (an R9-280X in the dual-hex, and a GTX 970 in the dual-quad). The dual-quad ran Linux, while the dual-hex ran macOS (all the way up to Catalina, though I had to hack it to get that running). Firestorm performance wasn't even close; on the same scene, at the same resolution, the macOS machine would run about 25 FPS, while the Linux machine would turn about 110.

If the viewer had been updated to a later OpenGL, it would have been a lot better, for multiple reasons. Now, LL's dug themselves a really deep hole, with Apple's help. I hope that Vulkan is doable, I really do; the alternative is for LL to have to write and maintain two renderers. Some of that work is going to have to be done anyway just to enforce a clean separation of graphics engine code from the rest of the viewer, but still, why reinvent wheels?

Yes, going to Metal gets you iOS. The problem there is that the UI needed is *different*; the viewer's UI as it now stands won't work well on a tablet, let alone a phone. Five minutes with Lumiya will prove that point. Reimplementing the UI to make it usable on such a device is even more massive an undertaking than rewriting the graphics pipeline.

I don't run Firestorm on the Mac any more. It's just too painfully slow. I do build and test it there, but that's it. If I could run Microsoft Office, a good photo editor (the GIMP is not; it drives me batty), and build Firestorm for the Mac on Linux, I'd drop Apple. I've been running macOS since 2002, so that's a remarkable statement.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/5C1A5659-C78F-4E07-BBC1-04DF7F1B6886%40dayturn.com.

[Geir Nøklebye]

Back to your original question:

There is a Carbon dependency in newview/llappdelegate-objc.mm languageUpdated that is tricky to resolve all the time it looks like Apple have scrubbed (or at least hidden) all Carbon documentation from the developer website. The header files just documents what the functions used do, but gives no hint for replacement.

Your best bet, IMO, is to generate a support request with Apple software engineering to help you resolve this issue and rewrite the code to use Cocoa and AppKit.

The same recommendation is for the color picker issue in llwindow, but in my estimation that is a bigger wash to resolve as the system color picker needs the OpenGL 3.2 profile. So that sounds like a project and not a support request.

Geir,

[Geir Nøklebye]

All,

An update after testing viewers on an Apple Silicon M1 Mac mini today, and the good thing is they all (SecondLife, Kokua, Cool VL, my own) runs, and seem to run reasonably well under Rosetta 2 emulation.

Performance is significantly better than on a Mac mini 2018 with the Intel 630 integrated graphics. Probably around a 3X performance increase.

For a viewer window of 1680x1080 running on a 27” non-Retina display, with graphics settings at high, objects at the fist High step (not fully turned up), ALM on and Shadows for both Sun/Moon/Projectors, no refelections, RendervolLODfactor of 4 and draw distance of 128; on my Linden home location it runs at about 10-20 FPS depending on how much is in view.

The SL viewer reports the following in the About floater:

You are at x, y, z in Washburn located at ec2-54-213-49-1.us-west-2.compute.amazonaws.com
SLURL: http://maps.secondlife.com/secondlife/Washburn/x/y/z
(global coordinates 265015.0, 250228.0, 28.3)
Second Life Server 2020-11-09.551942
Release Notes

CPU: VirtualApple @ 2.50GHz processor (2400 MHz)
Memory: 8192 MB
OS Version: Mac OS X 10.16.0 Darwin 20.1.0 Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:10 PDT 2020; root:xnu-7195.50.7~2/RELEASE_ARM64_T8101 x86_64
Graphics Card Vendor: Apple
Graphics Card: Apple M1

OpenGL Version: 2.1 Metal - 70.12.7

Window size: 1680x1044
Font Size Adjustment: 96pt
UI Scaling: 1
Draw distance: 128m
Bandwidth: 3000kbit/s
LOD factor: 4
Render quality: 3
Advanced Lighting Model: Enabled
Texture memory: 512MB
VFS (cache) creation time: November 18 2020 03:30:25
HiDPI display mode: 0



Note: The two notebooks with the M1 that is now on sale from Apple, are equipped with retina displays with would increase the render canvas to 4X the non-retina canvas.

Personally, I am pretty happy with that as it gives a window in which OpenGL still is around. – Which in my estimation is for as long as Apple sell Intel based Macs.

Now the big job is to get it to build as Apple Silicon native. First on to actually generating a viewer project with auto build on these machines, then the libs, then the viewer, then the renderer...

Cheers,
Geir

[Tonya Souther]

I'm impressed with these results. I'd frankly expected, at best, that the viewer would run a bit slower than native. I do wonder how much of that is the Metal OpenGL emulation as compared to the Intel GPU, but I don't know of a good way to find it out.

I do wonder how badly memory pressure will bite, since, as I understand it, the M1 Mac Mini will only be sold with integrated memory, and a maximum of 16 GB of it.

Moving the viewer natively to Apple Silicon will, in addition to the libraries, need a fair amount of rewriting around the SSE2 functions in llmath. I took a look at that and bounced; that's probably going to need the expertise of whoever it is at LL that knows the deep details of such things. IIRC, the ARM architecture defines its own vector math instructions, but how much work it takes to get there from here is very much a question.

I would expect, at least for a while, that any viewer would need to be built once again as a fat binary.

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/2D0078D1-982B-4B18-8D76-A4F395EB2D00%40dayturn.com.

[Geir Nøklebye]

The only machine I could get hold of right now was the 8GB configuration as the 16GB had shipping date of Jan 5, 2021.  But memory pressure is around 25 % with the viewer running and Rosetta 2 loaded, so unless you load up on many apps, it might not be too bad, and even then it can swap to very fast SSD. Xcode compiles never use tons of memory anyway, unless you try to do full app optimization. Besides Xcode on Intel will generate the ARM version too.

The performance increase is possibly a lot due to the unified memory architecture where nothing has to be copied between the CPU and GPU. We can look forward to more details on this in the coming weeks as the systems will be dissected and benchmarked through and through. 

I got autobuild and the required py version etc installed via macports.org, and it compiled all the tools on the machine during the install, after which I could configure the Xcode project. 

SSE2 is obviously one area, possibly the entire llmath, but then you have simd and a large number of libs optimized by Apple available.

The other hurdle is of course the renderer and the Apple GPU which is tile based, requiring a completely different approach to lighting on Metal than the deferred shaders currently in use. So many challenges ahead :-)

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c5Lg-dTCXPrzdwXQctsoVdrRrXke-47ANpoxQtVLm9Drw%40mail.gmail.com.

[Henri Beauchamp]

On Wed, 18 Nov 2020 14:43:06 +0100, Geir Nøklebye wrote:

> For a viewer window of 1680x1080 running on a 27” non-Retina display,
> with graphics settings at high, objects at the fist High step (not
> fully turned up), ALM on and Shadows for both Sun/Moon/Projectors, no
> refelections, RendervolLODfactor of 4 and draw distance of 128; on my
> Linden home location it runs at about 10-20 FPS depending on how much
> is in view.

And what do you get in your home location, with the same settings, when
running a Linux or even Windows viewer ?... I'd bet something in the
100+ fps range.

10-20 fps is really very low a frame rate (anything below 50fps is
pretty much sub-par; especially with a 128m draw distance).

Henri.

[Dahlia Trimble]

I wonder if it's using Apple's software implementation of OpenGL? Would be curious to see what a tool like GLView says.

I have a new M1 mini on order but Apple says I won't get it until sometime in December. :(

FWIW my laptop (Intel 8750H, nVidia 2080) gets 35ish FPS at my home location with maxed out settings except draw distance at 256, in a recent Firestorm, It's beachfront mainland with lots of water reflections. It seems to be  CPU bound as Task Manager says the GPU is running between 15-20% but it's saturating 2 cores of the CPU. My linux desktop does better with 50-55 FPS, same settings otherwise. It has a 4790K and a Radeon RX 5500. The 4790K usually has better single core performance than the 8750H in my laptop.

My current Mac Mini (Late 2014) gets 20ish FPS on Firestorm with default settings and window size at the same location.

-D

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20201118225727.dcbc6d44b794f12d4bbaab52%40free.fr.

[Geir Nøklebye]

@Henri,

There is nothing impressive about these numbers in absolute performance, but there is a health jump in performance on integrated graphics which brings the experience for SecondLife on machines like the Macbook Air and MacBook Pro 13” up to acceptable levels. These are machines that are typically bought in large quantities by students.

It is also worth noting the performance increase is seen on Intel code translated to ARM and running OpenGL on a GPU that is optimized for Metal. Other game engines and relevant applications reports 2-10X performance increase using Metal over OpenGL. Meaning there is scope for improving performance of the viewer on this architecture.
OTOH, there are also potential performance issues on the tile based GPU with dynamic content, as Metal performs best with content known at compile time, which we all know there is very little of in SecondLife.

If you install Windows on a Mac, you will get much higher FPS on the same hardware, but the primary reason for the difference is screen handling, where on macOS an application is never allowed to draw directly to the screen, but the output is composited to the desktop in a second pass at 60 Hz. – Which is why it is pretty pointless to generate more than 60 fps as that is what the compositor will send to the screen regardless.


Geir,

[Geir Nøklebye]

On the question: Is it using Apple’s implementation of OpenGL? The answer is yes, as it has been for many years.

The key to understanding this is that Apple for the majority of the existence of macOS X (meaning ten - we are now on macOS 11) only allowed the card manufacturers to write a barebones HAL, and implemented their own take on OpenGL on top of it.  This is also the core of the spat between NVIDIA and Apple, where they refused to supply sufficient information about their cards for Apple to implement their own OpenGL and later Metal on those cards.

This is also key to understanding why there never will be a Vulkan on the Mac unless there is both an official announcement and implementation on macOS by Apple. Anything else they simply will refuse to sign, so it won’t run. 

What GLView reports is:

OpenGL Version:  4.1 Metal - 70.12.7

OpenCL platform:  OpenCL 1.2 FULL PROFILE

Metal platform: 1v1, 1v2, 1_v3, 1_v4, 2_v1

Geir, 

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAAQTD4XLDL5PrWta5O43u9E2-3QY8%3DdmTM_abCYdZ-nsS38xxg%40mail.gmail.com.

[Dahlia Trimble]

Thanks Geir.

I'll assume "4.1 Metal" is different than "Apple Software Renderer 4.1 APPLE" which is what I was curious about.

I've heard Apple has no plans to provide a Vulkan implementation but there is MoltenVK which is part of the Vulkan SDK and is supported by Khronos. I'm aware that there are multiple apps on the App Store which use it. It's not a driver but rather a thin translation later between Vulkan and Metal so I don't think Apple would need to sign it as a kext as it only exists in user space. There was a brief period when it wasn't accepted for iOS but that was due to a patch that accessed a private API which was fixed shortly after it was discovered, and now it's currently blessed by Apple.

-D

[Tonya Souther]

Uhm, Geir..what's MoltenVK, then? Sure seems to me like Vulkan for macOS...

No, it's not going to be as fast as native Metal. But it's also not going to require double the investment in writing and maintaining two renderers.

Unfortunately, the only way to tell for sure is to have both Metal and Vulkan renderers, and that's a mountain of work...

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/435ED98A-AC29-47CD-BEF5-B1EAE1D1BFFC%40dayturn.com.

[Dayturn]

MoltenVK is an attempt to implement Vulcan on top on Metal, but if I had a business to run I would not, and I would certainly not, make a business decision on a major development project on a foundation Apple could just block with a simple revocation of the dev certificate. 

In addition you insert a dependency in your product where development of perhaps the most fundamental component potentially could significantly lag Apple system software releases thereby locking you out of addressing new HW releases for a period of time. 

Apple, for good and for bad, has a tendency to spring out new tech on an unsuspecting dev community that is disruptive if you have not followed their directions on what is smart to do.

Both Adobe and Microsoft now sit there essentially without products optimized for the new platform for a long time because they both learned virtually nothing from the PowerPC to Intel transition where they seriously lagged behind, and lost Mac customers all over while insisting to only use their own dev tools. 

Which comes back to chew their behind this time around too, because not only must they tailor the apps to a new HW arch, but they must first fix their tooling. Meanwhile small contenders such as Affinity zip past Adobe with optimized products at HW launch. 

Also, using bog standard Metal and the Apple frameworks, it will be possible from one source tree to generate a viewer that no only runs on macOS, but also iOS, iPadOS and possibly tvOS (when the new, more capable AppleTV is launched in 2021). (This is a bit of promise-ware right now as it works better from iPadOS to macOS than the other way around, but we can expect a bigger follow up at WWDC2021 and Xcode 13 or whatever they might call it). 

Despite the pain both in effort and cost of maintaining what essentially would be two different products, there is also a potentially significant upside in addressing Apple’s entire product range as it becomes more and more capable. (Essentially the M1 chip is a small iteration of the current high end iPad chip with modifications for a wider range of ports, form factors and better cooling). 

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c4VQWa6y_Xov6E6vWpOOZSJ5b0BereE1TFJ%3D8dyp%3D2q%2Bw%40mail.gmail.com.

[Tonya Souther]

MoltenVK is open source. Worse comes to worst, you include it in your source tree. Still beats maintaining two renderers.

For some folks, having the same code run on the desktop and tablet and phone is an unalloyed good thing. That's not going to work for the viewer, though. It depends on too many UI functions that just don't exist in iOS. Don't believe me? Spend some time with Lumiya. It'll show you how you would need to approach a viewer UI on iOS.

Apple wants people to make software specific to their platform. That may be fine for Microsoft, but LL would be pushing it - and while most of LL are Mac users, as well as a big chunk of the creator community they want to cater to, in the end they have to make a cold, hard business decision about where to spend developer resources.

[Geir Nøklebye]

Sure it is open source, but still a high business risk as you make yourself dependent on some random maintainer or get stuck maintaining it yourself.

MoltenVK is completely fine for hobby projects some of us are engaged in. Not so if your livelihood and business depend on it.

Cheers,
Geir

[Tonya Souther]

Eh, I'm not sure maintaining MoltenVK is worse than maintaining a second renderer. And even if it does get dropped by the originator, chances are there are enough people interested to help you keep it maintained.

Basically, I think you're badly underestimating the cost of two renderers here.

[Geir Nøklebye]

Dahlia,

The only thing Apple had to do was to write the HAL to their own GPU, as they already had their implementation of OpenGL in box. The OpenGL issue is not a technical one, but a strategic one for Apple. They don’t want dependencies on other suppliers for the total integration of their platform; both HW and SW. 

---

The issue with MotenVK is the business risk as long as you don’t have a commitment from Apple. Without such commitment, they can cut the cord on it at any time as they have done with other techs. 

For LL, which now has been in the market for 17 years with a viewer, the decision they will have to make it what technology and platforms can support them through the next 17 years. If the answer to that is that they want to provide their service also on Apple kit, going with anything but 100% supported technology from Apple for the most core of your product; the renderer, does not compute. 

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAAQTD4VEVWwtT8uP%2Bas0MfW0FPdshtC7D-dnOAC5axJMZ5VAVw%40mail.gmail.com.

[Tonya Souther]

That's just it, though: MoltenVK interfaces with Metal just like an application does, so Apple can't just cut the cord on it. It's a translation layer. Indeed, for a while it wasn't usable on iOS because it used an internal interface, but they rewrote it to only use public interfaces, and Apple approved MoltenVK-based apps on iOS. As far as the OS is concerned, it's just another Metal app.

Sure, they can revoke it on iOS, but there's no such mechanism in place on the Mac - and if they move in that direction, then there will be a lot more people affected than just Second Life users.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/606F5861-0FBF-4BD8-AAFF-DAE4B7C77B76%40dayturn.com.

[Tonya Souther]

You're right in that Apple is putting effort into libraries and HW integration. MoltenVK benefits from that just as Metal native apps do.

The bigger issue is that after implementing two renderers, maintenance costs are still doubled: as LL adds in advancements to the renderer, if they have two renderers, they have to make those changes twice, have to have staff expert in both, yada yada yada. I'm not talking about hardware/OS platform changes here, but changes to SL itself: things like materials support, as one example. Those costs are avoided with one renderer across multiple platforms.

In short, the initial investment is doubled (or worse, if they keep OpenGL for Windows and Linux), and ongoing maintenance is doubled, all for the sake of compatibility with a platform they can't exploit anyway in iOS et al. That's why native Metal doesn't make sense.

On Thu, Nov 19, 2020 at 9:55 AM Geir Nøklebye <geir.n...@dayturn.com> wrote:

I don’t underestimate it at all. :-)

I have said (for years) the initial investment is most likely 10 man-years worth of effort, $6 mill direct cost and 18 months development time. I have no problem tripping that (cost) estimate once reality kicks in. 

Once past the initial investment it is possibly cheaper to maintain than roll your own open source foundation, as you will get so much development effort from Apple put into the libs and HW integration. 

On Windows you can have a much softer progression without blowing up the entire codebase.

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c7x-nK57O-1-sxV8BnQGnDdJ_oa6XxbxRtP51d1znx4dw%40mail.gmail.com.

[Geir Nøklebye]

Of course there is such mechanism on macOS, and it is called Notarization and Signing. Or do you in earnest suggest LL distribute the viewer unsigned or not notarized (when the code has been changed to satisfy the notarization requirements)?

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c4UfJTLKxghBXpUa0JtWn4HL7ERMi-RA%3DHD%3Df4NqMBvvg%40mail.gmail.com.

[Tonya Souther]

They're distributing it unsigned now, and can't sign it until they replace VLC version 2 in the plugin. (Been there, done that, got the battle scars from Firestorm's QA people.)

But Apple has no reason whatsoever not to notarize the code. As far as they're concerned, a MoltenVK application *is* a Metal application.

[Dahlia Trimble]

Geir's dire FUDish warnings probably do have some merit but given how thin of an abstraction MoltenVK is, I'd think it far less likely that it would fall victim to the wrath of Apple than many of the other dependencies the viewer relies upon.

That said, it's common in the many rendering engines I'vs studied that they have a separate layer for each graphics API they target. Vulkan usually gets you Linux and Android. Direct3D gets you Windows, Metal gets you Apple, and OpenGL gets you older Linux and the web via Emscripten. There's usually a well defined interface description that each layer implements. Often one implementation might get more attention and new features than others but they usually sync up over time.

I don't believe MoltenVK is as risky as Geir suggests, at least not in the ways he describes. I rather consider the risk is that Metal might implement new features which Vulkan may not cleanly overlap and this might make it difficult to use such features. By using MoltenVK you're stuck with the feature set intersection between Vulkan and Metal, and while that intersection may currently be quite large, it may not be in the future as the APIs diverge.

However, these modern APIs don't really offer much benefit to older, single-threaded renderers and the LL code base would likely require very significant refactoring before any large performance benefits might be realized. Lacking such a refactoring effort, it may be a better choice to stick to OpenGL and use a OpenGL-><other API> translation should OpenGL support disappear on any given platform. Unfortunately, there's already a rather substantial technical debt that's accumulated in the viewer code base and such an approach likely make it worse.

Regardless, I've not often seen LL take user suggestions such as those in this thread and implement them; more often they work in secret and do their own thing. It's also not my place to tell them how to run their business.

-D

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c6Aq4MbbMhXdjYfmMiVXfX_f8owKqa3MpUGrE3fo8PvWg%40mail.gmail.com.

[Geir Nøklebye]

You need to dig further into that...

[Tonya Souther]

OK, I'll byte. Why would Apple not notarize code that uses MoltenVK to access Metal?

[Geir Nøklebye]

On 19 Nov 2020, at 17:13, Tonya Souther <tonya....@gmail.com> wrote:

all for the sake of compatibility with a platform they can't exploit anyway in iOS et al.

I am afraid you are completely wrong on that.

Also, investing in a “renderer” maintained by some random open source developer and that runs in userland is asking for performance issues, and all kinds of compatibility issues ahead, and would essentially be another Sansar. So good luck with that. ;-)

[Tonya Souther]

OK, how do you make the viewer UI - which is heavily dependent on interactions with more than a single click on an inworld object or UI element, and has things that need clicking on much smaller than the usual iOS button - usable on a table, never mind a phone?

And it's not a renderer, it's an interface layer, used by more than just this one project.

[Dahlia Trimble]

Calling Khronos "some random open source developer" is a bit like calling the IETF "some kid in a basement".

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/C36E66C0-FBD6-4663-9FD8-1F8B72EDBF2D%40dayturn.com.

[Geir Nøklebye]

The decision tree is actually quite short and simple. 

The question you have to ask is: Do I still want to offer my product on Apple platforms?

If the answer is no, the only thing you have to do it let the Mac viewer limp along on the current code base till OpenGL support is removed from macOS, and then it is over.  ETA: less than 2 years. 

If the answer is yes, the next question you have to ask is: How do I achieve technology stability and minimize my effort for the remaining period I want to offer my product on Apple platforms?

The obvious answer to that question is to use Apple technologies to the extent possible for my product because it both covers (mostly) smooth transitions and progression through Apple HW and system software releases, minimizing my efforts over time to stay on the platform, and leaves me to concentrate on developing my product.

The alternative answer is rolling my own, including using technologies and frameworks with open ended development and maintenance horizons. 

Over time it increases my burden as there is nothing to soften my blows when Apple makes unexpected moves, change platform, introduce products I currently don’t know that I need or want, deprecates and removes technologies they no longer want, need or like. 

It leaves me the freedom to totally control and develop my product from scratch, including my freedom to continously develop and maintain workarounds and tradeoffs to stay on Apple products.  

[Geir Nøklebye]

On 19 Nov 2020, at 23:23, Tonya Souther <tonya....@gmail.com> wrote:

OK, how do you make the viewer UI - which is heavily dependent on interactions with more than a single click on an inworld object or UI element, and has things that need clicking on much smaller than the usual iOS button - usable on a table, never mind a phone?

In case you have not noticed both iOS, iPadOS and tvOS has full support for game controllers which would help users browse, navigate and interact with the world.

https://developer.apple.com/documentation/gamecontroller?language=objc

In case you have not noticed, both iOS and iPadOS now have almost complete support for keyboards, trackpads and mice. In addition iPadOS has pen support which would work great for building and manipulating objects. 

https://developer.apple.com/videos/play/wwdc2020/10617/

https://developer.apple.com/design/human-interface-guidelines/ios/overview/mac-catalyst/

https://developer.apple.com/videos/play/wwdc2020/10109/

Making small and precise selections and click interaction has worked very well since early version of iOS, where the most frequently used is advancing web pages from the list of tiny numbers that often exist on web pages and forums. By making the view zoomable, you can hone in on an item which otherwise is difficult to select. 

By enabling system assistive technologies, you can usually completely navigate any menu structure and UI by the keyboard or even voice. 

[Geir Nøklebye]

An additional update to my previous message:

A large number of UI elements including the main menu and most floaters (probably all) can be moved out of the rendered 3D view and presented to the user with standard UI elements in a separate layer, or for macOS separate floaters that can even be moved to another screen.

On 19 Nov 2020, at 23:23, Tonya Souther <tonya....@gmail.com> wrote:

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c62c41iyFdBiGwEke-RKnfkqe8n7ExVM1h8EDMTBRwcCw%40mail.gmail.com.

[Dayturn]

As far as Apple, they actually are since they have announced it as deprecated technology that will be removed from the system.

Context is king! ;-)

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CAAQTD4U4Paw6rFOzpBqQU2FRP18frZ97ZQRRx-B3K3_s3yQnrA%40mail.gmail.com.