how does MWS (MUMPS WEB SERVER) works ??

[Adam Šída]

Hello mumps guru's

as disscussed over mail with mr. Habiel, I would like to ask you here about MWS's architecture.

I found MWS server as one of few option to fetch M-data over www, but very little info about its architecture and quite difficult to read  M code (especially abbreviated one) to understand how it works.

I found also other tools to publish M-data over http (QEWD, mgsi/mg-dbx,...) but none seem to fit my needs.

I am interested at the moment in how MWS handles incoming requests and what happend on OS level and M executable level when request is processed.

to explain what I look for,....old school web servers such as apache runs process per request, or ...or worse mgsi runs xinetd and then runs new process,...

compared to nginx or node.js which just spawn new threads for request and thus can run with minimal resource requirement...

I am quite aware about M executable architecture and its no-thread-use limitations.

can somebody please explain how MWS works ? is there some optimisation, so that once created www-handling process is reused again and again ?

or even to define how many of such processes can await in background to handle more requests in parallel.

or I would appreciate if somebody points me, where to start in MWS code to understand what I am interested in ?

I run M in docker container, I learn M lang over nights :) (as senior SQL consultant I love QUERY and ORDER commands). But still can't find practical application for M as without good www backend is hard to use with modern HTML UI frameworks.

thank you in advance

AS.

[chris....@gmail.com]

A  Mumps based web server is one option.  If you wish to use a more industry standard web server (Apache, Nginx or IIS) then you might want to look at mg_web:

 

https://github.com/chrisemunt/mg_web

 

Both options will allow you to process HTTP requests in Mumps code.  Of course, when generating an HTTP response it will be easy to include data held in the Mumps database.

 

Chris.

--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/fce9329e-838c-4c84-9e55-b31821e89ac6n%40googlegroups.com.

[Sam Habiel]

Chris,

Your reply was moderated; I made you a regular member.

--Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/05e701d8032b%24a60fccc0%24f22f6640%24%40gmail.com.

[Sam Habiel]

Adam

I will reply in more detail tomorrow. I need to focus on my day's work today.

Answers to your questions:

1. MWS uses the JOB command in M on a connection. It basically creates a new process.

2. The process stays around (I think for about 10 seconds) a bit for the same connection before it dies.

3. If another connection comes around, it creates a new process. There is no "thread pool" concept in MWS. However, since I wrote the GT.M code, GT.M has added the ability to pass sockets around to different processes, so it's now possible to implement that, but I haven't really bothered.

--Sam

--

[Christopher Edwards]

I know the MWS supports HTTP persistent connections as well.  Persistent connections allow the reuse of an M Job over the same HTTP connection and has a significant speed up in performance. I can’t remember if it was implemented on YottaDB (as Sam says he wrote that code). 

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CABHT961gZgVyfA5E-UhNCmprV7tm8U-dahWexMk0RC%2ByT%2BAv0w%40mail.gmail.com.

--

Christopher Edwards
cje...@gmail.com

[Raman Sailopal]

An alternative is the mgweb server developed by Rob Tweed.

This allows the easy development of APIs, calling MUMPS code to generate data that then automatically returns the data in JSON format.

This makes it possible to present mumps "back-end" data to a traditional "front-end" via a rest api.

I created a demo of what's possible using YottaDB, mgweb-server and Metabase as a front-end

https://github.com/RamSailopal/YottaDB-Demo

Ram

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CALEz0YOn3La62%2B7w36m8MFJBbtAt8R1Mwe35kqiPa-_w3%2Ba2HQ%40mail.gmail.com.

[Adam Šída]

thank you for all responses, may be I asked my question badly, so sorry for confusion...

I would like to avoid xinetd, apache, tomcat, etc and I'd like to have as direct connection (between M and browser) as possible.

from what I read until now I believe that M would have, lets say, quite higher transactional perf, comparing to eg. MSSQL+MSIIS, so I thought it would be great to not to loose it in extra processes, especially when M-lang can compile and do even sock comm :)

Today it is so easy to put lightweight reverse proxy (such as traefik is) in between browser and backend (taking care of TLS, auth, etc.), so it should be pretty straight to communicate with M over pure http.

+ this is something I can't accomplish with enterprise SQL DB at all.

is there something wrong about my idea from senior mumps point of view ?

I will wait for Sam's answer.

thank you all again

A.

čt 6. 1. 2022 v 20:50 odesílatel Raman Sailopal <rsai...@gmail.com> napsal:

You received this message because you are subscribed to a topic in the Google Groups "Hardhats" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hardhats/JQ5zg0B8Wpg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hardhats+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CAHUezNzKBc7QqkJ4%3DEp%3DtpexaXAdZx0pRc%3DU5h6rhOAsDQeq4A%40mail.gmail.com.

--

Adam Šída

online:

gsm: +420 720 694 593

e-mail: adam...@gmail.com

 

address:

Jugoslávská 2687

415  01, Teplice

Česká republika/Czech rep.

[Sam Habiel]

Adam,

Let me try to answer your question fully today.

Source code here: https://github.com/shabiel/M-Web-Server/blob/master/src/_webreq.m

Documentation for socket programming can be found here: https://docs.yottadb.com/ProgrammersGuide/ioproc.html#using-socket-devices

go -> job starts the web server. The webserver is started using this job command. $SYSTEM of 47 is assigned to GT.M, and YottaDB.

 I $P($SY,",")=47 J start^%webreq(PORT,,$G(TLSCONFIG),$G(NOGBL),,$G(USERPASS),$G(NOGZIP)):(IN="/dev/null":OUT="/dev/null":ERR="webreq.mje"):5  ; no in and out files please.

After this, go -> job exits. The webserver is now running in the background.

On line 44, we open the TCP socket like this:

 I %WOS="GT.M" O TCPIO:(LISTEN=TCPPORT_":TCP":delim=$C(13,10):attach="server"):15:"socket" E  U 0 W !,"error cannot open port "_TCPPORT Q

Line 49 is the use command to use the socket. GT.M has two modes, and we need to ensure we are running in byte mode (aka M mode). We don't want to interpret data as UTF-8.

 I %WOS="GT.M" U TCPIO:(CHSET="M")

Next, we listen for 5 concurrent connections max.

 I %WOS="GT.M" W /LISTEN(5) ; Listen 5 deep - sets $KEY to "LISTENING|socket_handle|portnumber"

The hard part of the code is the next block.

Upon connection, we reach this block. First thing, we do for loop to wait for 10 seconds each loop around waiting for a message.

Each loop, we check to see if $KEY has the information we need. Different versions of GT.M had different possible values. The old code is commented out and is no longer in use.

If $KEY has "CONNECT", then we take the socket identifier, detach it from the parent job, and start a process using the job command with that socket.

I %WOS="GT.M" D G LOOP

. ;

. ; Wait until we have a connection (inifinte wait).

. ; Stop if the listener asked us to stop.

. FOR W /WAIT(10) Q:$KEY]"" Q:$G(NOGBL) Q:($E(^%webhttp(0,"listener"),1,4)="stop")

. ;

. ; We have to stop! When we quit, we go to loop, and we exit at LOOP+1

. I '$G(NOGBL),$E(^%webhttp(0,"listener"),1,4)="stop" QUIT

. ;

. ; At connection, job off the new child socket to be served away.

. ; I $P($KEY,"|")="CONNECT" QUIT ; before 6.1

. I $P($KEY,"|")="CONNECT" D ; >=6.1

. . S CHILDSOCK=$P($KEY,"|",2)

. . U TCPIO:(detach=CHILDSOCK)

. . N Q S Q=""""

. . N ARG S ARG=Q_"SOCKET:"_CHILDSOCK_Q

. . N J S J="CHILD($G(TLSCONFIG),$G(NOGBL),$G(TRACE),$G(USERPASS),$G(NOGZIP)):(input="_ARG_":output="_ARG_")"

. . J @J

. ;

. ; GT.M before 6.1:

. ; Use the incoming socket; close the server, and restart it and goto CHILD

. ; USE TCPIO:(SOCKET=$P($KEY,"|",2))

. ; CLOSE TCPIO:(SOCKET="server")

. ; JOB START^%webreq(TCPPORT):(IN="/dev/null":OUT="/dev/null":ERR="/dev/null"):5

. ; SET GTMDONE=1 ; Will goto CHILD at the DO exist up above

. ; ---- END GT.M CODE ----

;

QUIT

To summarize all of this: A parent job listens around for up to 5 connections at a time, and when it gets a connection, it starts off a new job with that connection.

You had another question after this: How are the jobs reused for other connections or the current connection for future requests?

The parent listening job loops around, and stays around forever until stopped. It's the same block above, which does I %WOS="GT.M" D G LOOP

The child jobs handling each connection have a loop that is done using GOTO:

I %WOS="GT.M"&$G(HTTPLOG) ZGOTO 0:NEXT^%webreq ; unlink all routines; only for debug mode

G NEXT

The ZGOTO 0 is a trick to make GT.M relink all the routines. This is hard to understand, so ask me about it later when you have more experience.

So what the goto is really doing is saying, listen for the next request FROM THE SAME CONNECTION.

A few lines down from the NEXT tag, we find these lines:

R TCPX:10 I '$T G ETDC

I '$L(TCPX) G ETDC

We try reading from TCPX for 10 seconds. If we don't get any data (IF '$TEST), we go to ETDC

If we somehow get data but it's empty (shouldn't happen), also go to ETDC.

Now what does ETDC do?

ETDC ; error trap for client disconnect ; not a true M trap

D:HTTPLOG LOGDC

K:'$G(NOGBL) ^TMP($J),^TMP("HTTPERR",$J)

C $P

HALT ; Stop process

This just stops the process after doing some logging.

Summarizing this part:

1. The parent listener process runs forever until asked to stop.

2. The child job handling the connection will keep reading more from the same connection (pipelining) for 10 seconds, then dies.

Now, a few more items to talk about surrounding this code:

1. I didn't write the first draft of this code. Kevin Muldrum did. I adapted it to be a general purpose web server, added lots of Unit Tests, and made it work on GT.M/YottaDB.

2. GT.M kept adding features to sockets. One very nice recent feature is the ability to pass sockets between processes. If we implement this, our code will become much faster still than it is right now. Now, we have to start a new process for each new connection. It's now possible to start a children pool in advance and use these to service the processes. I am not interested in implementing this right now, but if you want to do it, please do, and I will take the pull request.

--Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/CADbfgDmu_oAXEVJgz48JE8aM18e-mDNR7r3qccSTMP%2BND-4_3Q%40mail.gmail.com.

[rtweed]

If you're happy to have a reverse proxy between a browser and an M system, surely you might as well be using NGINX or Apache to provide that functionality as well as the industrial-strength web server functionality they provide, then use mg_web as the very lightweight direct connection to the M system?

For the reasons you should be doing this, see:

 

- https://yottadb.com/building-rich-web-applications-with-yottadb/
- https://github.com/chrisemunt/mg_web/blob/master/why_mg_web.md

[rtweed]

I'd just add that mgweb_server makes use of the mg_web connector/interface that is the work of Chris Munt.  mgweb_server just adds an abstraction  layer and lots of ready-made APIs/interfaces to make the creation of M-based REST services really quick and easy.

[OldMster]

Agreed - I use mg_web with Apache on Linux, using Xinetd to manage the connections.  It really is very simple, and very robust.