jPOS 3.0 and beyond

[Alejandro Revilla]

Hi All,t

I wanted to provide an update on my ideas for the project in general and the next version—jPOS next—in particular.

TL;DR jump to the final bullet points.

Since I founded jPOS, I have always welcomed contributions of all kinds as long as they don’t break existing functionality. This approach has helped to build a strong and diverse community of contributors. However, over the course of 25 years, some contributions have become obsolete or unnecessary, while others have been integral to the development and evolution of jPOS, of course.

Additionally, as technology and programming languages have evolved, some code also becomes outdated. For example, when I started jPOS, Java 1.0 was the current version and we didn’t have basic features such as the java.util.Map class or the java.util.concurrent package. We had to use Hashtables and invent our own synchronization mechanisms. We also had to create our own thread pools due to the limitations of the “green threads” available at the time. The java.time package did not yet exist, and so on. Annotations? what’s that?

I created jPOS as a way to learn Java, which was a new programming language in 1997. As someone who was familiar with C and C++ and had read the book “The Gang of Four,” I implemented various design patterns in the jPOS codebase, such as Visitors, Factories, Builders, Flyweight, Composites, Composition, etc. These patterns can be seen throughout the jPOS codebase, but perhaps could be simplified using more modern class hierarchies, like say a Map! instead of an ISOComposite/ISOComponent.

To give a tiny example, when I wrote the jPOS Logger, which includes the LogSource and LogListeners, people were still using System.out for logging. At the time, the later popular log4j framework had not yet been released by IBM’s alphaWorks.

The Java language has undergone tremendous improvements and is now faster, more efficient, and more enjoyable to use. Features such as pattern matching, string templates, and triple-quote syntax have made code more readable. Additionally, Project Loom, which introduces virtual threads (formerly called fibers), is a game changer for jPOS and the TransactionManager. Project Panama is an exciting development that will enable us to connect with other languages and utilize standard machine learning tools that can be beneficial in the payments industry. For example, these tools can help us improve risk management systems.

There are many more benefits to this, but in order to take advantage of these opportunities, we need to follow the Java release cadence, which may have been initially resisted. It is important to keep up with the latest updates and advancements.

Getting jPOS to run with the latest releases of Java (19, 20 and beyond) require not only changes to jPOS, but also changes to some of its dependencies. For example, some don’t support Java Modules (introduced in Java 9, but now enforced from 17 and beyond). Not a big deal for jPOS itself that has very few dependencies and the only elusive one for now is SSH support, a feature not very often used and on track to get fixed by the sshd project, but when it comes to jPOS-EE, we have many many more dependencies, and the migration process becomes quite a challenge.

But the future with the latest Java versions is brilliant, and we need to do whatever it takes to get there.

To give you an idea of the exciting potential of what’s to come, I made a few changes to the TransactionManager to take advantage of VirtualThreads from project Loom. As a result, I was able to process around 1 million transactions in just 60 seconds, reaching a peak of 47,000 transactions per second using 100,000 sessions. And I honestly think this is just the beginning! We have so much more potential to explore. 100,000 in-flight transactions with an SLA of <5s is quite impressive for a single node.

As you begin to work with larger scales on single nodes, certain problems that were previously not noticeable become more pressing. Synchronizations here and there (TSpace, Logger to name a few) become sensitive and need to be addressed, and once you start playing with this, a few minutes later you find yourself realizing that a LockSupport.parkNanos gives you 1000 TPS more than a Thread.sleep, and learning about things like Thread.onSpinWait(). Very exciting!

But in order to achieve that goal, we must lighten our baggage of old code. We need to rock the boat a bit.

So, these are the main takeaways of this long (sorry) email.

• jPOS 3.0 will deprecate a lot of classes available in jPOS 2.x series. Those will start to be marked as Deprecated in the 2.x series. Some will be small deprecations, some will be big (i.e. no more PAUSE on the TransactionManager as there’s no need to now that we can have up to a million sessions and having no continuations simplifies a lot the very sensitive TM code and participant implementation, that will for example, safely use ThreadLocals).

• We have quite exciting improvements to the TransactionManager not limited to the use of VirtualThreads targeting for example context isolation (i.e. read-only entries), context validation (done by the TM), etc. We may have to tweak the existing API a bit in order to do that.

• The ISOMsg set of fields use integers as the field names. In order to seamlessly handle inner TLV, DataSets, and esoteric proprietary field hierarchies, we plan to use String instead of integers, in an as less invasive way as possible, so that you can address for example fields like 126.5.AA.

• Telemetry is a big debt. We have HdrHistogram in the TM, but with very little tooling beyond being able to upload your .hmgr files to http://jpos.org/histogram (you knew that, right?). That will change and people will be able to easily plug their preferred telemetry backend.

• The Logger subsystem will change in order to be more compact, support distributed tracing and telemetry as well as business oriented payloads (i.e. custom, compact, business-meaningful JSON payloads posted to ELK). A single LogEvent should be able to feed for example a Prometheus backend as well as a business-specific ElasticSearch, while providing compact storage in say a MongoDB (just to show an example). And it has to be super fast! in order to keep up with this new Loom based TM.

• The Spaces, specially TSpace will need a lot of attention, but hopefully, they’ll keep their API that has been serving us very well for over 25 years.

• Some stuff will vanish. compat_1_5_2 will stay around as a dependency, but will be disconnected from the project. Same goes for many jPOS-EE modules that were not heavily used. We’ll try to send jPOS-EE through a strict diet in order to focus on the components we heavily use. Some not core to jPOS-EE but good to have may graduate to their own repositories.

About contributions

Up until now, PRs have been chaotic. People would send a PR based on what could be useful for her/him at any given time, but totally disconnected with the project roadmap. The fun part is that people send PR, but rarely maintain the PR in the future, track its dependencies for possible upgrades or security issues. and rarely contribute documentation. We find ourselves, 10 years later, trying to figure out why some component does something. Beyond basic and obvious features, bug fixes or performance improvements that we are happy to accept, we’ll need some kind of discussion and coordination before a PR gets accepted. Without being too strict, we need something like the former JSRs, now JEPs (Java Enhancement Proposal). We can reuse the name and call it the jPOS Enhancement Proposal, and we are happy to study them.

jPOS Enhancement Proposals (JEPs)

I suggest starting with something like JEP-2 (the JEP template) and eventually improve it to make it more jPOS specific. If you have an idea worth contributing to jPOS, we suggest you:

• Do your PoC of course.

• Informally test the waters on Slack, or the jPOS-Users mailing list.

• Write a JEP, perhaps post the JEP in jPOS Discussions.

before sending a PR.

New releases

2.x series:

• 2.1.8 will be released soon.

• 2.1.9-SNAPSHOT will start flagging stuff as Deprecated. Very limited features will be added as we will concentrate in the 3.x series (both on jPOS as well as jPOS-EE side).

• At some point (perhaps 2.1.9, perhaps 2.1.10), jPOS 2 will become a LTS version.

3.x series:

• 3.0.0-SNAPSHOT is already available under branch next and available in jPOS maven repo.

• We like to say it’s ready when it’s ready, jPOS 3.0 will be released when it’s ready, but it’s reasonable to think the initial 3.0 release will be available during Q2/2023.

And that’s it. Happy 2023 and thank you for your support!

--

@apr

[QSwitch]

Hi @apr

I really excited to see what jPos 3 feels like.

Would you also consider the abilty to compile jPos into native code with tools such as GraalVM ? Sure it will improve the performance, especially the startup time.

Regards.

[Alejandro Revilla]

jPOS (3.0) runs on GraalVM. We just have some issues with one dependency. JLine, that we use for CLI commands, but I'm sure those will be resolved soon. The easiest way to run it is inside Quarkus. You can do that using simple code like this:

https://gist.github.com/ar/2cebcd03e251fa8b8350fbcf948f6c7d

Kind regards

--

@apr

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/256342ed-c110-475f-b1e3-3b607d2b3c52n%40googlegroups.com.

[galih.l...@gmail.com]

hi @Alejandro Revilla 

can you share the spec of the server that you use for the 1 million transactions in just 60 seconds.

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAAgSK%3DmPdsho7VfbmNsZi%3DCcPsE%2BPJhrSz1j1uhZVGwkatcEyw%40mail.gmail.com.

[Alejandro Revilla]

That would be my MBP.

[Agustín Izaguirre]

Hi @apr and folks

Great writeup! 

I agree that improving upon observability is one of the main things that would really benefit JPOS as a whole.

To be honest I'd like to hear your opinion on implementing something like Micrometer for JPOS as there are a lot of good ideas over that project.

In our case we plugged a crude handrolled ELK listener with MDC threadLocals with a lot of success (jpos pluggable log listeners made it a breeze). 

This really accelerated digging through logs by being able to partition through interesting dimensions like channel-host/port, client, brand, bin, etc and being able to visualize with little effort.

I'd also take a look on supporting OpenTelemetry out of the box, might prove worthwhile. These tools are really a major productivity boost.

Still, excited to see what's next. JPOS 3.0 looking to be a complete game changing release!

---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/256342ed-c110-475f-b1e3-3b607d2b3c52n%40googlegroups.com.

[Agustín Izaguirre]

Hey all,

Here's a blog post on micrometer support for Quarkus for example explaining the reasoning behind it. It's a good quick read https://quarkus.io/blog/micrometer-metrics/

Digging a bit seems other projects also support Micrometer out of the box. 

JPOS could maybe do a simple facade around Micrometer and then maybe something like SPI for different implementations with a sane default. I'd love to start a proper discussion over this.

[Alejandro Revilla]

Thank you for your comments. Micrometer is indeed in the radar and the way to go. We plan to replace our early and timid use of HdrHistogram with it.

OpenTelemetry is also on the roadmap for application specific data, and JFR for internal jPOS stuff that I’m sick of people destroying while aggregating logs, “because our [Dev?]Ops team want to flatten your XML logs”. 

I’ll follow up with additional comments, sorry for the short (mobile) reply, but I didn’t want this interesting thread to stay unanswered.

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/ca4827fd-d1a3-4d5d-a352-3aad4484c33fn%40googlegroups.com.

--

--

@apr

[Alejandro Revilla]

When we first started talking about jPOS observability, vsalaman, who is usually ahead of the pack in terms of new technology, suggested using Prometheus right away and it could have been a great solution for a long time, and perhaps it still is. I usually don’t like to be too much opinionated in terms of which technology to enforce, so I went with something very lightweight and internal to jPOS, HdrHistogram.

Thing is that while you can upload some HdrHistogram files to http://jpos.org/histogram to do some forensics, that doesn't qualify as observability.

Now in hindsight, Prometheus is still great, but as you say, going with Micrometer is a more flexible approach, and that’s what we plan to use in jPOS 3.0.

Regarding your specific ELK listener with MDC ThreadLocals, that’s going to become dangerous in jPOS 3.0 where you can have tens or hundreds of thousands virtual threads. ThreadLocals can become quite resource consuming and you may have to keep an eye on that. but there are many ways to achieve the same, especially if you’re using the TransactionManager. You just need an ELK reporting participant.

Regarding OpenTelemetry, I consistently monitor CNCF developments, and indeed, OpenTelemetry is on the radar. However, before addressing it, there are some internal logging needs that take precedence.

The existing Logger/LogSource/LogEvent trilogy, written 25 years ago, needs an overhaul.

I want LogEvents to be less tied to the listeners. Right now, a LogEvent has a LogSource, and said source has a list of log listeners. I think it’s totally OK to send the same LogEvent to different groups of listeners.

In addition, I have a recurring problem with people destroying our logs.

I cry everytime I see a nice log like:

<log realm="org.jpos.q2.qbean.SystemMonitor" at="2023-06-24T10:25:50.710">
  <info>
           OS: Mac OS X (13.4.1)
         Java: 1.8.0_332 (Amazon.com Inc.) AES-secure
  environment: default
 process name: 64...@apr-2.local
    user name: apr
         host: apr-2.local/127.0.0.1
          cwd: /opt/local/e-factura/srl
   free space: 508.5 GiB
      version: 2.1.4-SNAPSHOT (2a9b3f6)
     instance: f3d7ab8b-137b-47cc-b2e1-d5a7b16d4e64
       uptime: 00:00:04.927 (1.618652)
   processors: 16
       drift : 0
memory(t/u/f): 331/35/296
     encoding: UTF-8
     timezone: GMT-03:00 (GMT-03:00) -03:00
watch service: sun.nio.fs.PollingWatchService
        clock: 1687613150 2023-06-24T13:25:50.707Z
 thread count: 66
 peak threads: 66
 user threads: 57
 ...
  </info>
</log>

converted into

2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - <log realm="org.jpos.q2.qbean.SystemMonitor" at="2023-06-24T10:25:50.710">
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - <info>
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - OS: Mac OS X (13.4.1)
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - Java: 1.8.0_332 (Amazon.com Inc.) AES-secure
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - environment: default
...
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - </info>
2023-06-24 10:25:50 INFO  org.jpos.q2.qbean.SystemMonitor - </log>

and I’m being generous here, sometimes I get to see that info totally flattened, very difficult to reformat to make it readable. This has information redundancy, loss of structure, lack of readability, and our elements such as ‘warn’, ‘info’, ‘debug’, ‘opertor’, lose their structural meaning.

That’s information we need when debugging jPOS issues, but it’s not information the regular user or operator needs, so I’m considering moving all that to JFR. If that approach flies and we don’t get to see major roadblocks, and it doesn’t end up being a crazy idea, the whole jPOS logging system may change, freeing people to use whatever logging framework they want, and logging all the “important” things they want, such as:

2023-06-17 10:00:00 INFO  com.example.MyApplication - User starts the application.
2023-06-17 10:00:01 INFO  com.example.MyApplication - User is about to set a variable 'a'.
2023-06-17 10:00:02 INFO  com.example.MyApplication - User sets the variable 'a' to a value of '1'.
2023-06-17 10:00:03 INFO  com.example.MyApplication - Variable 'a' has been successfully set to '1'.
2023-06-17 10:00:04 INFO  com.example.MyApplication - User confirms the value of 'a'.
2023-06-17 10:00:05 INFO  com.example.MyApplication - The current value of variable 'a' is '1'.
2023-06-17 10:00:06 INFO  com.example.MyApplication - User verifies the value of 'a'.
2023-06-17 10:00:07 INFO  com.example.MyApplication - Confirmed that the value of variable 'a' is indeed '1'.
2023-06-17 10:00:08 INFO  com.example.MyApplication - User ends the operation on variable 'a'.
2023-06-17 10:00:09 INFO  com.example.MyApplication - User prepares to close the application.
2023-06-17 10:00:10 INFO  com.example.MyApplication - User closes the application.
2023-06-17 10:00:11 INFO  com.example.MyApplication - Application has been successfully closed.

That’s the current situation. We anticipate some breaking changes in jPOS 3.x, but I believe they will bring substantial benefits.

--

@apr

[Agustín Izaguirre]

Something to keep in mind is that Prometheus is a pull based approach instead of push based.

About the ThreadLocals, yes, it's something that we might have to rework when we finally start using virtual threads (ScopedValues maybe?). All depends on how that contextual information is propagated from thread to thread (or task).

On the other hand we've decided to be cautious and not greenlight virtual threads yet. We believe they still have some rough edges that need to be ironed out. Specifically we are worried about deadlocking issues in thirdparty code and given that pinning on synchronized is something being worked on we decided to wait. See reddit thread

Going back to observability, internally we've gone for the Micrometer approach. We instrument some JPOS components and we use the elastic APM agent to reliably deliver those metrics to Elastic/Kibana for visualization. This so far seemed the least intrusive approach but requires some cooperation from the components (Channels, Participants, TxnManager, SysMonitor, etc).

In short what we'd like to do is simply to move all the metrics related stuff (counters, gauges, rates) to the micrometer API and maybe add a JPOS implementation using the same api if needed.

> flattened, very difficult to reformat to make it readable

We had that too, though I might say that we have found the SystemMonitor and TxManager logs pretty useful to troubleshoot some difficult issues.

Here is a shortened snippet on how we bridge from JPOS to sl4j.

public LogEvent log(LogEvent ev) {

    ByteArrayOutputStream eventMessageOutputStream = new ByteArrayOutputStream();
    ev.dump(new PrintStream(eventMessageOutputStream), "");
    String eventMessage = eventMessageOutputStream.toString();
    logger.info(eventMessage); //actual implementation differs a bit

[Alejandro Revilla]

Fully agree with your post, including not greenlighting Loom. We tested it extensively for servers, channels, and network related jPOS code, but the TransactionManager is tricky because it runs user participants not under our control. We'll keep the old TM handy (perhaps renamed as LegacyTransactionManager) just in case.

Regarding Micrometer, you may want to transition from your custom instrumentation to the jPOS standard one for servers, muxes, channels and ISOMsgs going in and out. We tried to keep the number and cardinality of metrics low.

As for your code snippet, it looks quite similar to the FrozenLogEvent.

https://github.com/jpos/jPOS/blob/master/jpos/src/main/java/org/jpos/util/FrozenLogEvent.java#L24-L45

--

@apr

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/7c30cce9-93a1-4d10-85ff-9e872758ddf9n%40googlegroups.com.