MQTT Feature ...

[mar...@feldtmann.online]

Well, Version 12 offers a new feature: MQTT protocol. If you want to use it, how could it be supposed to work? I wanted to make a quick shot to connect to RabbitMQ ... and look if I could do stuff with RabbitMQ and its MQTT support and VA.

The first point (do not forget!): search the documentation. Well, beside the information, that MQTT is a new feature in version V12 - nothing .... hmmm

Ok, then load the feature and look for example usage code, perhaps test code .... nothing ... hmmm

At this point I closed VA, started VisualStudio Code IDE and starting a new project with C#/.NET6 with the RabbitMQ connection library and the freeImage graphics library wrapper and started coding ... viewing their documentation how to do stuff ...

Marten

[Mariano Martinez Peck]

Hi Marten,

With the newer frameworks, instantiations prioritize documenting the code itself with good and detailed method comments, class comments, etc. You can see this with OsProcess, all async frameworks, Unicode support, MQTT, etc etc. So the very first thing I would do is check class comments and method comments. For example for SstMqttClient. Inclusion of MQTT in the online documentation would be coming in the future.

There are actually examples, and they are also shipped as a feature. See the map "Server Smalltalk - MQTT Examples", the application SstMqttExamples and the class SstMqttClientExample.

Finally, there are two presentations that you may want to see:

1) An MQTT Client for the Enterprise: https://youtu.be/AGCN8qsxYbw
2) Let’s deploy that Smalltalk project!  (the project deployed here is a MQTT demo) : https://youtu.be/r7RxtVJeGa8

Regards,

Mariano

--
You received this message because you are subscribed to the Google Groups "VAST Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/ec559b85-4170-4df6-8450-7721da278aa2n%40googlegroups.com.

--

Mariano Martinez Peck Senior Software Engineer  mp...@instantiations.com  @MartinezPeck  /mariano-martinez-peck  instantiations.com

[Richard Sargent]

On Tuesday, February 28, 2023 at 10:16:22 AM UTC-8 Mariano Martinez Peck wrote:

Hi Marten,

With the newer frameworks, instantiations prioritize documenting the code itself with good and detailed method comments, class comments, etc. You can see this with

Hi Mariano, I am all in favour of good comments all the way through, don't doubt that for a moment. However, comments at the class level are no substitute for "start here" types of documentation. At the least, an otherwise not-yet-documented feature should ship with a Read Me style technical note that points out what a person should look for and how to get started. [That was one of the things that drove me crazy with the Java world. Documentation was simply listing the classes and their methods. Nothing to provide a coherent understanding of what's important.]

It might be worthwhile using the Comments section of the configuration maps / features to provide minimal pointers. e.g. When the map/feature is loaded, display its comment(*) which might be as simple as "Refer to ... and ... to get started with feature XYZZY" on the Transcript at the end of the loading.

(*) I realize the feature map comments already include encoded information about the feature, so perhaps something more structured. It could be as simple as how abt.cnf was split into phases.

[mar...@feldtmann.online]

Hello Mariano,

thanks for the information about this. The answer itself - well, shows a decision I rather would not like to see. Documentation of the code is correct and should be done and more technical details could be written there (thats the way Gemstone does it ... have online documentation but more detailed information in class comments)

But I would rather not like to see, that the official online documentation can be split into two parts: the "old" code and the "new" code. The new code is not mentioned - but only the old code is mentioned.

And as Richard also mentioned - how to texts, the first step texts should be not inside the code.

I would argue, that if github projects would only be documented in the pure-code files, the success would not be there.

But I will look at the code again .....

[Mariano Martinez Peck]

Hi Richard, 

Please find my answer below.

Hi Mariano, I am all in favour of good comments all the way through, don't doubt that for a moment. However, comments at the class level are no substitute for "start here" types of documentation. At the least, an otherwise not-yet-documented feature should ship with a Read Me style technical note that points out what a person should look for and how to get started. [That was one of the things that drove me crazy with the Java world. Documentation was simply listing the classes and their methods. Nothing to provide a coherent understanding of what's important.]

It might be worthwhile using the Comments section of the configuration maps / features to provide minimal pointers. e.g. When the map/feature is loaded, display its comment(*) which might be as simple as "Refer to ... and ... to get started with feature XYZZY" on the Transcript at the end of the loading.

(*) I realize the feature map comments already include encoded information about the feature, so perhaps something more structured. It could be as simple as how abt.cnf was split into phases.

 

We are already doing that but rather than at the config map level we are doing it at the app level. Basically, this is our process now:

1. We add comments to the application class to give that higher level context. This is the appropriate place to talk about design, architecture, main entry points, etc. Later, these become the top-level overview documentation in the online documentation. For example:

UnicodeSupport (subapp) comment goes to https://www.instantiations.com/vast-support/documentation/1200/#page/pr/UnicodeSupport-1.629.2.html# 
 OsProcessApp app comment goes to https://www.instantiations.com/vast-support/documentation/1200/#page/pr/OsProcessesApp-1.html#
 EsAsynchronousSupport app comment goes to https://www.instantiations.com/vast-support/documentation/1200/#page/pr/EsAsynchronousSupport-1.660.1.html

and so on...

2. We are making this change so that we can keep documentation updated and in sync

3. We try to provide very detailed examples that help educate while showing how it works

All this has created a sustainable methodology for our team to maintain the pace of development while ensuring comments stay up to date.  And we have very strict documentation standards now for class/method comments compared to years ago. 

Unfortunately for MQTT we didn't have the time to do that because for this release we had severe time constraints.

Finally, for VAST 2024, we are also evaluating to improve our internal build process to automatically generate this doc on every internal build. 

Regards,

[mar...@feldtmann.online]

Tough I have only tested the sending with the MQTT client, it works pretty well with RabbitMQ. Have to make more tests with TLS communication and subcribing topics ... the idea here was to have VA (together with FreeImage oder ImageMagick) processes working on graphical jobs containing images (retrieving via RabbitMQ) and manipulate them and send them (via RabbitMQ) to other VA processes (on different machines) for storing them in directory (all processes are headless tasks and should be working on Windows AND Linux machines ...).

The jobs under Windows are working quite nice (the interaction with FreeImage) and I wrote an initial interface to work with ImageMagick (MagickWand API) ... under windows no problem - BUT under Linux I had lots of problems (especially all stuff handling png files ...)

[mar...@feldtmann.online]

Ok, I stopped the VAST project at this stage.

The VA-MQTT client seems to be ok and useable, it is a fast as the .NET RabbitMQ.Client (CSharp) code (of course this is a full AMQP client). The VA-MQTT client is pretty stable and sending 3169 graphic files (in a batch operation) to RabbitMQ is no problem for that client (and not for RabbitMQ). The documentation for RabbitMQ.Client is less good here -- there introduction code shows code, which is not stable. From 3169 files the transfer lost around 600 file each run (but it was much faster :-)) - if you add sleep() statements the code goes more stable. You have to read further the documentation to set the right options (confirm options for the transmission) to get the transfer stable and secure. This is a not a good way.

The project itself could be done in VA, due to the fact, that the "heavy" work (and HD images are only the start, later on 4K images would the normal ones)  is done in external c libraries and VA is only manager here ...

I stopped the VA project, simply due to the fact, that neither FreeImage nor ImageMagick could be wrapped in a stable way under Linux for VA (here Ubuntu 22.04). The code worked under Windows - but Linux ? No way. Because I needed the code for headless systems under Linux, I go now the C# way.

I have to admit, that the .NET6 (Core) system is pretty good. Create a new project (via a template), get some library code via "dotnet get application ..." (this took most of my time), do dome imports in the C# code, then a "dotnet build" and a "dotnet run" and all stuff is build (and ready for distribution) and can be tested.

[Joachim Tuchel]

Mariano,

the first two of your three links point to a 404 page...

I'd like to add a little comment to this discussion about documentation - even if it is unrelated to MQTT or any other specific feature.

I think your approach makes a lot of sense for people who have been working with a techmology in other contexts and want to use them in Smalltalk now. When you know your building blocks and mechanics and class or method names are similar to what you've been using before in Python, JS or Rust, you'll have an extremely fast start into a new library.

This is not, however, the way things are happening in VAST for many users. Consider your users being long-time Smalltalkers who've been using VAST for decades. They now finally get a feature that allows them to play with the JS, Python or C# guys around them. They now can finally use some tool or api they haven't been able to use on their project before. For many users I know, this new feature in VAST is their first encounter with a tool or API. They need some introduction into the basics - with examples in VA Smalltalk. The alternative would be to start learning this new thing in Python or C# and come back to VAST once they have taken the first few hurdles and transfer their new knoweldge to Smalltalk method sends. Let's be honest: if I need to learn to use something in Python first and can do it, I may as well stick with Python and use it. Why not continue on the "everything new in X and couple it wiht Smalltalk" approach taken in so many projects? These days, everybody is talking about micro services, which supports this model.

I remember myself starting Seaside in the naive belief I could do Web Programming without learning all that JS, HTML, CSS stuff. It took me some time to understand I have to learn all that AND how to combine all of this with Seaisde and Smalltalk. At the moment, I am starting my first real complex XML mapping project and again have the feeling I'd better learn the basics of all these more complex things in Java or Python and see if I can achieve the same in Smalltalk (because there is a body of code in Smalltalk already). As you can read in another thread on this forum, I feel quite lost already a few hours after I started my first experiments. I must admit the documentation of the XML mapping stuff in VAST probably dates back to the IBM days, but that is only an explanation for the situation, but doesn't really help me progress.

Maybe this comment helps you think about writing a few basic tutorials in addition to good and extensive class / method / config map comments. VAST is progressing so fast and there are lots of great improvements coming our way, it would be sad if we couldn't use them because of missing documentation.

Joachim

Mariano Martinez Peck schrieb am Mittwoch, 1. März 2023 um 13:17:28 UTC+1:

[Joachim Tuchel]

Marten,

could you be more specific about your problems? Are there problems with missing features of API calls (callbacks etc), memory problems  or does ImagMagick work worse on Linux than on Windows (which I doubt)?

Joachim

mar...@feldtmann.online schrieb am Samstag, 4. März 2023 um 20:53:32 UTC+1:

[mar...@feldtmann.online]

Hello Joachim,

imagemagick works very stable under Linux - we have used it for years now - but we've used so far only the command line version. No problem. We wanted to change our picture-data-storage to sockets and not to files and by that we might get additional 20% performance win, remove file polling and so on. So we need server processes, expecting commands from RabbitMQ, do the work and send different commands to RabbitMQ back for additional backend processes.

Actually VA has all stuff needed ... and the binding of the MagickWand-C-API does not look that difficult ... but if it does not work (and it works under Windows) ... then it is not worth the time ...

[Joachim Tuchel]

Marten,

same here: we use ImageMagick via OsProcessStarter / Command Line - works very well on Linux and Windows. It is even faster and stable on Linux, afaict.

But for the rest of your story: I still don't understand what did work and what didn't. Do you have problems with MQTT or with API calls to MagickWand? I'm a pita, but I think we have a use case for MagickWand here in the near future....

Joachim

[mar...@feldtmann.online]

Its the API call to MagickWand ... so, to get more into details (your fault). This is the first example from the public API page ...

copyAndConvertExample
"MSKMagickWandExamples copyAndConvertExample"

| wand api |

"handy code to access the api"

api := MSKMagickWandAPIWrapper Default.

"well to initialize the system"

api magickWandGenesis.

"get a new wand"
wand := api newMagickWand.

"get a build-in graphic for testing and put it into the wand"
(api magickReadImage: wand path: 'logo:') = MagickFalse
   ifTrue:[ self halt: 'error' ].

"reset the iterator!"

api magickResetIterator: wand.

[ (api magickNextImage: wand) ~= MagickFalse ] whileTrue: [

  "***** now: set the new filename and write the build-in graphic as png ***"

  api
    magickWriteImage: wand path: '/datadisk/projects/picture-postprocessing/test-pictures/pattern/test.png' .
].

"release the data"

api
   destroyMagickWand: wand ;
   magickWandTerminus.

All APi calls are done via synchronous API calls (to get fixed memory locations). Under Windows it works, under Linux the new filename is set, but it crashes during "magickWriteImage".It created the test.png file, but with file size "zero".

magickWriteImage is defined as 

MagickWriteImage := self createPlatformFunction: #('MagickWriteImage' nil (pointer pointer) int64)

and the internal definition is <c: int64 'libMagickWand-7.Q16HDRI.so':MagickWriteImage pointer pointer>.

The code works under Linux with "test.gif", "test.jpeg" - but NOT with "test.png" or "test.webp" ... so after all pretty useless. There must be still a general error in the whole structure ... the CreateThumbnail-API Call creates a file under Linux with the correct size, but with BLACK content. I am still not sure, how an enum should be returned - but I assume its an int64.

[Gabriel Cotelli]

Regarding de documentation issue, I've personally found Diátaxis ( https://diataxis.fr/  ) a very compelling idea for technical documentation authoring.

[Richard Sargent]

On Mon, Mar 6, 2023 at 12:08 PM Gabriel Cotelli <g.co...@gmail.com> wrote:

Regarding de documentation issue, I've personally found Diátaxis ( https://diataxis.fr/  ) a very compelling idea for technical documentation authoring.

_That_ does look interesting. There are multiple needs for documentation, and those four distinctions seem like they would do a great job.

Well designed examples could fulfill one of the quadrants needs. (The How To guides, perhaps.)

--
You received this message because you are subscribed to a topic in the Google Groups "VAST Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/va-smalltalk/3Ra7Yr0IDbY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/79c4c14a-c63e-49c6-bfb1-57766e41f7a9n%40googlegroups.com.

[Joachim Tuchel]

Marten,

that is too bad. I was hoping to use MagickWand for a few tricks in our App as well. Sounds like it's not worth trying at the moment. I wonder if this is a VAST or an ImageMagick problem. The way I read your code, it seems like MagickWand is the problem, although that's hard to believe... ImagMagick is used in so many tools that I guess problems are reported and fixed quite fast...

otoh, it is good to know that it's not an MQTT problem with VASt that you are struggling with. I look forward to finding the time to experiment with it soon (side project no. 983 or so)...

Joachim

[Joachim Tuchel]

Hi Mariano.

well documented code is a good thing, no discussion about it.

The fact however, that the documentation page finds zero results for the keyword 'MQTT' is suboptimal, imo. Not even the presentations you link to are mentioned. 

There's not even mention of the fact that you can load a feature that handles MQTT and if the user wishes to read documentation, they can look into class comments of the classes x, y or z. I'd also wish for a short first steps intro using at least some MQTT implementation. 

My impression is that you need to understand MQTT first and once you know exactly how to use MQTT, you can easily find out how to do it in VAST as well. 
This may be good for a majority of VAST users, but if you come from the other end where you might have heard that using MQTT for your project might be a good idea, VAST is not helpful to you. Imagine being a team member and ypur manager asks you to find out if using MQTT for a new feature in your VAST project might work and what amount of work it might be. How would you start if MQTT is completely new to you and you have zero idea of what it is?

Just my 2cts on this (3 years old) discussion.   

Joachim

Mariano Martinez Peck schrieb am Dienstag, 28. Februar 2023 um 19:16:22 UTC+1:

[Esteban A. Maringolo]

Hi Joachim,

I appreciate your feedback on this. Because of these issues, and as part of a general modernization effort on our side, we are planning to overhaul our entire documentation platform.

Our goal is to make it significantly more accessible for developers. Transitioning documentation that has been built up over decades is certainly not an easy task, but it is one we are committed to completing.

Best regards,

Esteban Maringolo VAST Consultant Senior Software Developer  emari...@instantiations.com  @emaringolo  /emaringolo  instantiations.com

To view this discussion visit https://groups.google.com/d/msgid/va-smalltalk/5e229256-ea47-457a-bfeb-24087e1aa442n%40googlegroups.com.