Hi All,t
I wanted to provide an update on my ideas for the project in general and the next version—jPOS next—in particular.
TL;DR jump to the final bullet points.
Since I founded jPOS, I have always welcomed contributions of all kinds as long as they don’t break existing functionality. This approach has helped to build a strong and diverse community of contributors. However, over the course of 25 years, some contributions have become obsolete or unnecessary, while others have been integral to the development and evolution of jPOS, of course.
Additionally, as technology and programming languages have evolved, some code also becomes outdated. For example, when I started jPOS, Java 1.0 was the current version and we didn’t have basic features such as the java.util.Map
class or the java.util.concurrent
package. We had to use Hashtables and invent our own synchronization mechanisms. We also had to create our own thread pools due to the limitations of the “green threads” available at the time. The java.time
package did not yet exist, and so on. Annotations? what’s that?
I created jPOS as a way to learn Java, which was a new programming language in 1997. As someone who was familiar with C and C++ and had read the book “The Gang of Four,” I implemented various design patterns in the jPOS codebase, such as Visitors, Factories, Builders, Flyweight, Composites, Composition, etc. These patterns can be seen throughout the jPOS codebase, but perhaps could be simplified using more modern class hierarchies, like say a Map! instead of an ISOComposite/ISOComponent.
To give a tiny example, when I wrote the jPOS Logger, which includes the LogSource and LogListeners, people were still using System.out
for logging. At the time, the later popular log4j
framework had not yet been released by IBM’s alphaWorks.
The Java language has undergone tremendous improvements and is now faster, more efficient, and more enjoyable to use. Features such as pattern matching, string templates, and triple-quote syntax have made code more readable. Additionally, Project Loom, which introduces virtual threads (formerly called fibers), is a game changer for jPOS and the TransactionManager. Project Panama is an exciting development that will enable us to connect with other languages and utilize standard machine learning tools that can be beneficial in the payments industry. For example, these tools can help us improve risk management systems.
There are many more benefits to this, but in order to take advantage of these opportunities, we need to follow the Java release cadence, which may have been initially resisted. It is important to keep up with the latest updates and advancements.
Getting jPOS to run with the latest releases of Java (19, 20 and beyond) require not only changes to jPOS, but also changes to some of its dependencies. For example, some don’t support Java Modules (introduced in Java 9, but now enforced from 17 and beyond). Not a big deal for jPOS itself that has very few dependencies and the only elusive one for now is SSH support, a feature not very often used and on track to get fixed by the sshd project, but when it comes to jPOS-EE, we have many many more dependencies, and the migration process becomes quite a challenge.
But the future with the latest Java versions is brilliant, and we need to do whatever it takes to get there.
To give you an idea of the exciting potential of what’s to come, I made a few changes to the TransactionManager to take advantage of VirtualThreads from project Loom. As a result, I was able to process around 1 million transactions in just 60 seconds, reaching a peak of 47,000 transactions per second using 100,000 sessions. And I honestly think this is just the beginning! We have so much more potential to explore. 100,000 in-flight transactions with an SLA of <5s is quite impressive for a single node.
As you begin to work with larger scales on single nodes, certain problems that were previously not noticeable become more pressing. Synchronizations here and there (TSpace, Logger to name a few) become sensitive and need to be addressed, and once you start playing with this, a few minutes later you find yourself realizing that a LockSupport.parkNanos
gives you 1000 TPS more than a Thread.sleep
, and learning about things like Thread.onSpinWait()
. Very exciting!
But in order to achieve that goal, we must lighten our baggage of old code. We need to rock the boat a bit.
So, these are the main takeaways of this long (sorry) email.
jPOS 3.0 will deprecate a lot of classes available in jPOS 2.x series. Those will start to be marked as Deprecated in the 2.x series. Some will be small deprecations, some will be big (i.e. no more PAUSE
on the TransactionManager as there’s no need to now that we can have up to a million sessions and having no continuations simplifies a lot the very sensitive TM code and participant implementation, that will for example, safely use ThreadLocals).
We have quite exciting improvements to the TransactionManager not limited to the use of VirtualThreads targeting for example context isolation (i.e. read-only entries), context validation (done by the TM), etc. We may have to tweak the existing API a bit in order to do that.
The ISOMsg set of fields use integers as the field names. In order to seamlessly handle inner TLV, DataSets, and esoteric proprietary field hierarchies, we plan to use String instead of integers, in an as less invasive way as possible, so that you can address for example fields like 126.5.AA
.
Telemetry is a big debt. We have HdrHistogram in the TM, but with very little tooling beyond being able to upload your .hmgr
files to http://jpos.org/histogram (you knew that, right?). That will change and people will be able to easily plug their preferred telemetry backend.
The Logger subsystem will change in order to be more compact, support distributed tracing and telemetry as well as business oriented payloads (i.e. custom, compact, business-meaningful JSON payloads posted to ELK). A single LogEvent should be able to feed for example a Prometheus backend as well as a business-specific ElasticSearch, while providing compact storage in say a MongoDB (just to show an example). And it has to be super fast! in order to keep up with this new Loom based TM.
The Spaces, specially TSpace will need a lot of attention, but hopefully, they’ll keep their API that has been serving us very well for over 25 years.
Some stuff will vanish. compat_1_5_2
will stay around as a dependency, but will be disconnected from the project. Same goes for many jPOS-EE modules that were not heavily used. We’ll try to send jPOS-EE through a strict diet in order to focus on the components we heavily use. Some not core to jPOS-EE but good to have may graduate to their own repositories.
About contributions
Up until now, PRs have been chaotic. People would send a PR based on what could be useful for her/him at any given time, but totally disconnected with the project roadmap. The fun part is that people send PR, but rarely maintain the PR in the future, track its dependencies for possible upgrades or security issues. and rarely contribute documentation. We find ourselves, 10 years later, trying to figure out why some component does something. Beyond basic and obvious features, bug fixes or performance improvements that we are happy to accept, we’ll need some kind of discussion and coordination before a PR gets accepted. Without being too strict, we need something like the former JSRs, now JEPs (Java Enhancement Proposal). We can reuse the name and call it the jPOS Enhancement Proposal, and we are happy to study them.
jPOS Enhancement Proposals (JEPs)
I suggest starting with something like JEP-2 (the JEP template) and eventually improve it to make it more jPOS specific. If you have an idea worth contributing to jPOS, we suggest you:
Do your PoC of course.
Informally test the waters on Slack, or the jPOS-Users mailing list.
Write a JEP, perhaps post the JEP in jPOS Discussions.
before sending a PR.
New releases
2.x series:
2.1.8 will be released soon.
2.1.9-SNAPSHOT will start flagging stuff as Deprecated. Very limited features will be added as we will concentrate in the 3.x series (both on jPOS as well as jPOS-EE side).
At some point (perhaps 2.1.9, perhaps 2.1.10), jPOS 2 will become a LTS version.
3.x series:
3.0.0-SNAPSHOT is already available under branch next
and available in jPOS maven repo.
We like to say it’s ready when it’s ready, jPOS 3.0 will be released when it’s ready, but it’s reasonable to think the initial 3.0 release will be available during Q2/2023.
And that’s it. Happy 2023 and thank you for your support!
--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/256342ed-c110-475f-b1e3-3b607d2b3c52n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAAgSK%3DmPdsho7VfbmNsZi%3DCcPsE%2BPJhrSz1j1uhZVGwkatcEyw%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/ca4827fd-d1a3-4d5d-a352-3aad4484c33fn%40googlegroups.com.
When we first started talking about jPOS observability, vsalaman, who is usually ahead of the pack in terms of new technology, suggested using Prometheus right away and it could have been a great solution for a long time, and perhaps it still is. I usually don’t like to be too much opinionated in terms of which technology to enforce, so I went with something very lightweight and internal to jPOS, HdrHistogram.
Thing is that while you can upload some HdrHistogram files to http://jpos.org/histogram to do some forensics, that doesn't qualify as observability.
Now in hindsight, Prometheus is still great, but as you say, going with Micrometer is a more flexible approach, and that’s what we plan to use in jPOS 3.0.
Regarding your specific ELK listener with MDC ThreadLocals, that’s going to become dangerous in jPOS 3.0 where you can have tens or hundreds of thousands virtual threads. ThreadLocals can become quite resource consuming and you may have to keep an eye on that. but there are many ways to achieve the same, especially if you’re using the TransactionManager. You just need an ELK reporting participant.
Regarding OpenTelemetry, I consistently monitor CNCF developments, and indeed, OpenTelemetry is on the radar. However, before addressing it, there are some internal logging needs that take precedence.
The existing Logger/LogSource/LogEvent trilogy, written 25 years ago, needs an overhaul.
I want LogEvents to be less tied to the listeners. Right now, a LogEvent has a LogSource, and said source has a list of log listeners. I think it’s totally OK to send the same LogEvent to different groups of listeners.
In addition, I have a recurring problem with people destroying our logs.
I cry everytime I see a nice log like:
<log realm="org.jpos.q2.qbean.SystemMonitor" at="2023-06-24T10:25:50.710">
<info>
OS: Mac OS X (13.4.1)
Java: 1.8.0_332 (Amazon.com Inc.) AES-secure
environment: default
process name: 64...@apr-2.local
user name: apr
host: apr-2.local/127.0.0.1
cwd: /opt/local/e-factura/srl
free space: 508.5 GiB
version: 2.1.4-SNAPSHOT (2a9b3f6)
instance: f3d7ab8b-137b-47cc-b2e1-d5a7b16d4e64
uptime: 00:00:04.927 (1.618652)
processors: 16
drift : 0
memory(t/u/f): 331/35/296
encoding: UTF-8
timezone: GMT-03:00 (GMT-03:00) -03:00
watch service: sun.nio.fs.PollingWatchService
clock: 1687613150 2023-06-24T13:25:50.707Z
thread count: 66
peak threads: 66
user threads: 57
...
</info>
</log>
converted into
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - <log realm="org.jpos.q2.qbean.SystemMonitor" at="2023-06-24T10:25:50.710">
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - <info>
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - OS: Mac OS X (13.4.1)
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - Java: 1.8.0_332 (Amazon.com Inc.) AES-secure
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - environment: default
...
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - </info>
2023-06-24 10:25:50 INFO org.jpos.q2.qbean.SystemMonitor - </log>
and I’m being generous here, sometimes I get to see that info totally flattened, very difficult to reformat to make it readable. This has information redundancy, loss of structure, lack of readability, and our elements such as ‘warn’, ‘info’, ‘debug’, ‘opertor’, lose their structural meaning.
That’s information we need when debugging jPOS issues, but it’s not information the regular user or operator needs, so I’m considering moving all that to JFR. If that approach flies and we don’t get to see major roadblocks, and it doesn’t end up being a crazy idea, the whole jPOS logging system may change, freeing people to use whatever logging framework they want, and logging all the “important” things they want, such as:
2023-06-17 10:00:00 INFO com.example.MyApplication - User starts the application.
2023-06-17 10:00:01 INFO com.example.MyApplication - User is about to set a variable 'a'.
2023-06-17 10:00:02 INFO com.example.MyApplication - User sets the variable 'a' to a value of '1'.
2023-06-17 10:00:03 INFO com.example.MyApplication - Variable 'a' has been successfully set to '1'.
2023-06-17 10:00:04 INFO com.example.MyApplication - User confirms the value of 'a'.
2023-06-17 10:00:05 INFO com.example.MyApplication - The current value of variable 'a' is '1'.
2023-06-17 10:00:06 INFO com.example.MyApplication - User verifies the value of 'a'.
2023-06-17 10:00:07 INFO com.example.MyApplication - Confirmed that the value of variable 'a' is indeed '1'.
2023-06-17 10:00:08 INFO com.example.MyApplication - User ends the operation on variable 'a'.
2023-06-17 10:00:09 INFO com.example.MyApplication - User prepares to close the application.
2023-06-17 10:00:10 INFO com.example.MyApplication - User closes the application.
2023-06-17 10:00:11 INFO com.example.MyApplication - Application has been successfully closed.
That’s the current situation. We anticipate some breaking changes in jPOS 3.x, but I believe they will bring substantial benefits.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/7c30cce9-93a1-4d10-85ff-9e872758ddf9n%40googlegroups.com.