Migration from d3 to jBASE

[Tom Jeske]

Hello experienced jBASE users!   My company is migrating to jBASE from d3 in the near future.  I am one of several software engineers who maintain and enhance our current d3 system.  We have over 250 users (AccuTerm) and 1000+ web users - mostly flash connect and .NET.   We're currently running on a 12 year old AIX box but we also have 13 other VM's connected via OSFI and NFS to offload the main AIX box.  We also have many thousands of programs from 20+ years of development. 

I would really appreciate some stories from people who've been in this swamp full of alligators.  What was your migration strategy?  How did you maintain your development efforts AND a conversion effort?  How long did it take start to finish?  How much 'outside consulting'  did you need?  All advice and stories gratefully accepted. 

TIA

[stope19]

Hi Tom,

I'm sure many others will throw in ideas on the conversion, but it would be interesting to understand why you are moving from D3 to jBase. I have no affiliation with either platform (have been out of MV for many years) - but am always keen to know the motivation behind such decisions. Was is cost? Features? Support? Something else?

cheers,

dave

[Rick Weiser]

Hi TIA,

This might not be the best place for this thread.  There is a jBASE form on Google that might serve you better.

https://groups.google.com/forum/#!forum/jbase

RIck

[Wols Lists]

On 17/02/17 14:38, Rick Weiser wrote:
> Hi TIA,
>
> This might not be the best place for this thread. There is a jBASE form
> on Google that might serve you better.

MIGHT.

I would have thought this was a better place to find people who know
both D3 *and* jBase. And a better place to find people who have migrated
from one version of Pick to another.

We did a migration from PI to UV. The first thing to do would be to copy
all your source across, and try to compile on jBase. There's probably a
bunch of minor glitches that might be loads of grunt work to fix but no
real problem. FIX THEM ON THE D3 SIDE. You want as much of your code as
possible to work on both systems.

Work out how to get the data across. I can't remember how we did it. A
virtual tape save/restore will probably do.

You can then plan the live switch-over, having worked out what you need
to do.

It's more effort up front, but trying to make as much of your code
cross-compatible before you start is a good idea. Maybe set up a git
repository, develop on one system, and push and test on the other before
deploying. Then your development and porting work can proceed in
parallel, knowing that no new "one system only" code is being developed.

Cheers,
Wol

[Rick Weiser]

Yes, MIGHT.

People who have migrated to jBASE MIGHT not be on this forum.  They would however be on the jBASE forum.

The question is platform specific.  If the question was general, then yes this forum would be the place to post.  There MIGHT be things in jBASE that can help him with the transition, that he MIGHT not be able to get here.

[Ian Harper]

I'm not affiliated with the original poster but my company is looking to do the same migration some time in the future. From my perspective jBase looks a lot more modern than D3. It seems to me that there's been a lot of development efforts on Rocket's other MV DBMS (U2) and D3 has been given a lower priority.

[Tom Jeske]

In response to stope19: Management made the decision but shared these reasons: Less costly site licensing model, better connectivity to other databases, encryption (DREM), able to use C for development.

[Dawn Wolthuis]

We are working with folks who are migrating from D3 to SQL Server using MVON. So, the target is different, but the project is likely very similar.

 

In case it is useful, I see these implementations as a SCIENCE. Here is a broad sketch and some tips that might be helpful.

 

Sys Admin

GOAL: A target environment configured properly for running a MultiValue application for starters, eventually have it ready for production.

 

Have your high availability, DR, security, scalable live environment ready to go (like using AlwaysOn for SQL Server), or anything else required. [I don't like sys admin, so I'm pouring all of the platform setup into one big Sys Admin category. Use Zumasys for the cloud or at least use their expertise in this area.]

 

Compile

GOAL: Everything compiled so the application can be launched.

 

Getting to 90% is fast, perhaps even amazingly fast, because it is simply another implementation of the MV data model and languages. 

 

You might hit a few new compiler deltas (where the new compiler doesn't do exactly what the D3 compiler did with the same code and no prior D3 applications compiled in the new environment have used some construct in their source code). Those are almost always fast to resolve, whether with a quick work-around where you do a bulk change to the code, or, more typically, with a simple change to the compiler to take into consideration your unique use of the language. jBASE, like MVON, is backed by fast, smart, engineers who have seen everything and can likely help you get rid of compile errors in very short order.

 

The key in this phase is to keep and manage a list and the communication to get the changes in short order. I said these projects are SCIENCE, but they are all about relationships too, of course.

 

Initial data migration

GOAL: Export and import (save and restore account, for example) and have enough data to be able to run the application and be able to test out all of the features.

 

Where in the previous step, any needed changes are usually done by the vendor, this step often finds areas where some "data cleansing" is in order -- places where the data was not right in the first place but D3 was cool with it anyway. The application owner sometimes wants or needs to write a data cleansing routine so that when the data lands in the new environment, the issues that were found are resolved.

 

Errors, obvious ones

GOAL: Turn the application over to those who know how to use it to do the next round of testing. 

 

The easy part of testing is the first time you launch the application in the new environment. If something has not been configured, compiled, or migrated properly. when you get started you hit an error, fix it, hit another, fix it. Any such issues should be managed like compile and data issues, identifying whether the vendor or the application owner needs to do something and managing the issues list. 

 

Suggestion: I like using github issues with a huboard (kanban board for github issues) to manage issues across multiple organizations. Your vendor might have other approaches.

 

iNconsistencies, “deltas”

GOAL: Have the application and data migration in shape to be live

 

This is the hardest step to complete, I think. Now you have an application with data (might be a duplicate of your live data at a point in time), and it looks like it runs. Ah, but it might not be an exact match with some variation on the input or by using some feature deep in the application. There could be some hidden "deltas" (things that act differently in the new environment).

 

It seems to be a common issue with projects that no one wants to test a system that looks the same as the one currently running and is just a different platform. Asking (or even telling) users who know the current system to test the new one is the number 1 spot for delays with the project. Sites can sit on a very-close-but-has-not-been-fully-tested system for a long time without preparing a test plan, without getting anyone who likes to test, without getting the resources of people who know the current system well enough to test, etc. 

I also put more focus on testing and managing the deltas than on "documentation" as I like doc that is a by-product of the process. However, there will be a need for good media approaches (email, written doc, meetings, video, postings) throughout the project. Document any inconsistencies between the old and the new that are planned for the new environment. In our case, for example, a site might document how to use Excel or PowerBI directly against their SQL Server data.

Change-over strategy. 

GOAL: Have a high level strategy and then a detailed plan for how you are going to move from one to the other. 

 

Sometimes sites like to populate both their D3 database and, in our case, SQL Server, using a utility to do so and start writing reports against the new environment. In other words, they might do OLTP on the old system until they are ready to move, but start by moving read-only activities to the new environment. I don't know if that is common with jBASE, as that might be because one of the reasons to get to SQL Server is because users like having access to their data or sites like the security model they can put in place to then give users access within proper parameters. But it might work well in that environment too as you can prove the data is all in the right places when having parallel data. 

 

Some sites like to do parallel processing in the two systems, sometimes for a month, to make sure all of the i's are dotted in the new system before they stop maintaining the old. This is more common when a site cannot seem to get a priority on testing from users or if the application is so mission critical that they decide it is wise to verify a full month of transactions, reports, and sys admin (backups, etc).

 

The most common strategy is likely picking a "bite the bullet" date and switch over on that date, with the data migration being one key to making that work. If you do start "streaming the data" to the new environment in advance, then you can do an instant change-over, else there would be downtime to port the data.

 

Execute the plan and go LIVE

GOAL: Do it. Go live with your application on the new platform.

Afterwards, there are always those little things that you decide to do after the change-over. Do those now and react to anything else that crops up.

 

Even if you are sick of the project by this time, don’t forget the celebration. After all, projects like this are all about relationships, I mean SCIENCE.

Best wishes with your new (ad)venture.  --Dawn

Dawn M. Wolthuis

ONgroup Intl

M: 616-901-6293

P: 800-573-0300 x706

S: Dawn.Wolthuis

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms

[Don Robinson]

Hello Tom,

Dawn pretty much covered the big picture but the devil is in the details.

I worked on a Universe to jBase project for about a year. After a near disaster, we brought in Jim Idle, one of the creators of jBase.

He installed Perforce to track changes, both source code and Unix scripts, etc. A five user version with 20 client/workspaces is free which should be enough.

A major hurdle is that jBase, at least version 3.x, only allowed a single copy of a program. We ran into the same program name with different code in multiple accounts. Over 40 BP files in total. Universe handles this with the RUN command but jBase didn't at that time, I think it supports RUN now.

I worked on an Ultimate to Universe conversion and the biggest hurdle was the fixed (nailed) port numbers. I wrote a program to update the file Universe uses to set the port number.

I've worked on a couple of other conversion.

All of them took a lot longer and a lot more work than expected.

Besides the mods required to compile the programs, there are OS issues and worse of all, people!

As Dawn said, getting people with the application knowledge to do SERIOUS testing is a real struggle! Some drag their feet because they don't understand the reason for converting, don't like change, feel "left out" of the planning, etc.

My suggestion is to assign people full time to the project and DON'T expect them to do their current job in their spare time, there won't be any spare time.

My 2 cents worth.

Don Robinson

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+un...@googlegroups.com

[Glen Batchelor]

  Yes, yes and more yes. Even integration/migrations have the same problems. The biggest issue I've encountered is having the right people, with full knowledge of their own responsibilities, in the places they need to be during the critical steps. Testing? LOL yeah right. No one is going to admit that they don't really fully understand their job activity and why they have to do things a specific way. They just do it that way because no one complains that something is broken or that their work is causing other problems. SMH

GlenB

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

For more options, visit http://groups.google.com/group/mvdbms

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

[John Doe]

Below, not necessary in the proper order, are some of my experiences.

I did a D3 to jBASE 3 conversion in 2002-2003 on Windows server and it was a pain in the butt.

Unlike we were told. there were major differences between D3 and jBASE 3.

In 2007 we've explored a transition to jBASE 4 but based on required changes has been decided to forego any jBASE update and move to a different platform.

 

If you get D3 it's like getting an empty house to which you add furniture(programs) to your liking.

When you get jBASE all you get is a pile of bricks.

To understand the above metaphor you have to take into account that jBASE does not have any login procedure, no security whatsoever except for Windows login, nothing.

Beside the program conversion we had to build the account and user framework from scratch and that is no easy task.  Windows login does not tell you who the user is so we had to implement our custom login that has custom access rights for each user.

While it sounds simple, in reality it took about half a year just to build a frame for accessing jBASE.

 

The short synopsis:

We had about 3000 basic programs.

It took about 4500 hours/person to finish the conversion that is 1 programmer full time + 3 part time for 1 year.

Note that because of the volume we had to prioritize the basic conversion therefore step 1 was to put at the beginning of each of the 3000 PICK programs a call to a subroutine which recorded in a central file each call to each program.  After a week we had a list of about 500 programs to look into ASAP.  If I remember correctly the average time to convert 1 program was about 50 minutes which included testing.  We had 5 minutes deals as well as daylong conversions notably for programs that had dynamic calls such as

"CALL @name(…"

 

In my experience PICK systems are the slowest in the market.

The very unscientific test results below refer to run times for basic programs with 100 being the fastest.

Universe   100

jBASE 3     98

OpenQM    57

D3              32

Flash D3    90  (inferred from other people test results)

 

jBASE pretends that being compiled in an executable form the programs are somehow better or at least better performing.  Baloney.

There is absolutely no advantage.  The programs are not portable between machines of different architecture or different operating systems and you are dependent on a third party C compiler.  jBASE compiles the basic source code into C then invokes the resident C compiler to build the executable.

Once made an executable, the program has to run together with the "license" program.  Every so often the program communicates with the "license" and will abort if the license is not present so build here and run there is just an illusion.

 

Before buying, the jBASE PR department was all milk and honey with unrealistic promises such as "restore the PICK save tape and voila everything works".

After we paid the dough all we've got was "jBASE is not PICK" and "This is not a bug, it's a feature".  No bugs we've reported have ever been fixed.

As a bonus they had a program called PORTBAS that supposed to smooth any conversion issue but in practice proved to be of very limited use.

 

Before jumping in, I suggest you dip your toes into jBASE world by downloading their free developer version.  You should try to develop a simulation of your current system and see what problems you encounter.

I know that the life altering decisions are often made by people who have no clue beyond the sales pitch but in this case at least you have the opportunity to make a case based on your own experience.

 

I know nothing about current version jBASE 5 but here are some points to take into account about jBASE 3 and 4.

 

1) jBASE has a lot of reserved words therefore we had to change many variable names in about 3000 programs.  Some program names had to be changed because were incompatible with jBASE.

 

2) Some instructions are not supported for example TCLREAD, some have a slightly different syntax such as

    IF A

    THEN

has to be changed to

    IF A THEN

some have different functionality because jBASE is case sensitive and does not have any CASING ON/OFF instruction.

 

3) Different SYSTEM() values

    Different OCONV/ICONV conversion codes

    Different INCLUDE syntax

    Different Q pointers

 

4) Because jBASE is case sensitive we had problems when users entered commands with the wrong case therefore we had to build our own command interpreter similar to JSH and also made sure that all inputs were converted to upper case when necessary.

 

4) There is no "phantom" or job scheduler, we had to write our own.

5) Programs are searched in the system path therefore MD pointers to programs are no longer functional.  In our case we've moved all common programs in a "common" directory that is shown in the path after the current directory.

 

6) As of jBASE 4 all programs are loaded in dll's  therefore development is a chore because each time you compile, in order to run the latest version, you have to log off then log on again.

 

7) As of jBASE 4, the named common is no longer working as per the manual so we were unable to use it to transfer data between programs so we aborted any attempt to "upgrade" to jBASE 4.

 

It's too long ago so I don't recall every issue I had but I can tell you there were many.

Of course the decision makers were frustrated because the rose sales pitch and the reality were so far apart to the point of accusing us programmers of sabotage but in the end we found other organizations that were able to confirm the myriad of problems we had.
Good luck.

[Dawn Wolthuis]

Yes, I agree with Don that the details are very important and need to be managed with good communication, etc. Our projects are not as complex as what you described, however. I talked to the chief architect of MVON# who happens to know jBASE well. He confirmed that jBASE does some things quite differently than other MV platforms, including ours (SQL Server or Oracle with MVON or MVON#), so that might be the reason for some added pain points that Don mentioned, although I suspect some of those have been addressed by jBASE since that time.

I'm passing along here what I learned in the conversation (with the chief architect of MVON#) about jBASE that might make for added complications compared to what I am used to (I went from Prime Information to UniData to MV ON SQL Server). 

1. Unlike all other MV platforms, jBASE doesn't run in a p-machine (aka bytecode machine, run machine, virtual machine). D3 runs in the D3 run machine. MVON# runs in .NET, UniData runs in the UniData run machine, TAFJ (Temenos's replacement for jBASE) runs in the JVM etc. 

Only jBASE opts not to have the luxury of the memory management and other features of a run machine. Most applications that are written today run in a run machine. So, you will have some details regarding this "feature" of jBASE, no doubt.

For example, with jBASE, it is not a piece of cake to get to the O/S and you need to link everything new in.

 

2. Apparently the concepts of MD/VOC and of MV accounts, in general, was not implemented in the same way for jBASE. So, Don might be giving you a good heads up on that front regarding some of the pain points in doing a change-over to jBASE rather than to another MV environment such as OpenQM or MVON. Those were written to be a more precise emulator of UniVerse and other PICK/MV platforms. All other MV environments have a master dictionary (or vocabulary file) approach where the VOC entry IS the definition and it is easy to put multiple accounts on one server. So, be sure you know how to handle a dev, test, and live accounts properly.

From what I know, jBASE is definitely a workable platform for companies to choose even if the projects to get from here to there with jBASE are likely more complex than what we end up doing with our MV implementations. Our initial goals included emulating everything as close to existing platforms as feasible while expanding the interoperability of MV applications with everything else (SQL Server, .NET, Xamarin, ...).

[Note: I wanted to clarify this lest anyone think that a project to get to SQL Server would be as complex as what Don is describing. It is still a project for sure, but perhaps oddly, it is likely a closer match to the source platform.]

--Dawn

Dawn M. Wolthuis

Take and give some delight today

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

For more options, visit http://groups.google.com/group/mvdbms

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

[geneb]

On Thu, 16 Feb 2017, Tom Jeske wrote:

> Hello experienced jBASE users! My company is migrating to jBASE from d3
> in the near future. I am one of several software engineers who maintain

I'd be curious to know what the drive is to move from D3 to jBASE.

g.

--
Proud owner of F-15C 80-0007
http://www.f15sim.com - The only one of its kind.
http://www.diy-cockpits.org/coll - Go Collimated or Go Home.
Some people collect things for a hobby. Geeks collect hobbies.

ScarletDME - The red hot Data Management Environment
A Multi-Value database for the masses, not the classes.
http://scarlet.deltasoft.com - Get it _today_!

[Ed Clark]

I have to echo the question that others have asked: Why jBase? Is this a definite conversion, or are options still open?
As many have pointed out in the past on this forum, you can do pretty much anything you need to do on any of the currently available multivalve databases. Some things are easier on some platforms than others, and each has its own attractions, but a competent programmer can do anything that they need to on any platform, including D3, assuming that they are running on modern hardware.

If the move is definite, then you should post on the jBase google group, and also talk to their support people. They’ve done this before. 15 years ago I had to do a conversion to jBase, and they were very supportive at that time.

The biggest hurdle that you may run up against is case sensitivity. D3 is case-insensitive by default in both source code and data. “A”=“a”. jBase, last I knew was case sensitive in both. SO
SELECT CUSTOMER-FILE WITH NAME = “SOMEONE”
will not return records where NAME is “someone” or “Someone”, unless you fiddle with the dictionary. In basic code:
crt “SOMETHING”
EQU NULL TO “”
are both illegal syntax because “crt” needs to be “CRT”, and “NULL” is a reserved word.
IIRC, jBase provides a conversion program which will try and fix the casing in programs but it’s not 100%.

Other issues depend on what features of D3 you use, and what 3rd party tools you employ.
Do you use SB+ or another 4gl? Something to provide ODBC or a web server? You need to find out if these work with jBase and what alternatives you have.
Do you use RUNOFF much? I don’t think jBase supports it.

During the conversion I did to jBase, I found that most things using the spooler and job scheduler needed to be changed because command syntax was different.
I found that many of our ACCESS queries were extremely slow resource hogs on jBase, and many needed to be rewritten.

[Wols Lists]

On 17/02/17 16:59, Ian Harper wrote:
> I'm not affiliated with the original poster but my company is looking to
> do the same migration some time in the future. From my perspective jBase
> looks a lot more modern than D3. It seems to me that there's been a lot
> of development efforts on Rocket's other MV DBMS (U2) and D3 has been
> given a lower priority.

One reason for which, is that IBM actually put a load of effort into
updating U2, and that Rocket bought the "genuine Pick" assets a lot
later than U2.

I suspect Rocket is chucking the same effort into genuine Pick, as in to
U2, but the genuine Pick has had far less time in which to show any results.

Cheers,
Wol

[Peter McMurray]

I am afraid that this sounds like all the wrong people are on the selection committee. Almost certainly dyed in the wool OSS people living in the dark ages.

The first question that should be asked is 

"What do I want to do that I can't do in D3". The answer to which is arguably nothing. One can call anything these days and relevant to that C is best used for embedded languages requiring a small footprint. It is a minefield for the average programmer. Most of us who grew up with Assembler and understood low level programming are now vanishing. Modern shops tend to look for low price programmers who can churn out stuff and lack the deep skills for memory management, testing and the like necessary for C.

Next question
" How much investment do I have in D3"

Answer millions of dollars, so why am I walking away from it?

Is JBase a suitable alternative for a Pick environment when there are so many other better platforms? Oh I forgot. it's cheaper. If I may be forgiven for quoting a famous US comedian "The Road Less Travelled! Maybe there's a reason"

[Jay LaBonte]

Tom. I have to agree with Peter here. I do not know your position within your organization, but it sounds like you are being tasked with managing this conversion. 

Since it appears the decision to move to jBase has been made, and that fact that you are asking these questions now is a huge red flag to me. These are question that should have been researched BEFORE making the decision as to which database in which to migrate and you should have already performed a cost analysis, and built your project plan. 

The next issue I see here is that it appears part of the decision to migrate was due to performance issues, yet it seems a decision was made to change hardware and the database without really knowing where the issue actually reside. My bet is that D3 would be find if you simply upgraded hardware and release levels. 

I may be wrong about some of this, and I am speculating based on the type of statements you made and questions you asked.

If the decision has not yet been finalized, then I would suggest contracting someone with this type of experience and have them review the overall plan before committing to such a massive undertaking which could end up costing more than your original D3 investment.

Jay L.

Sent from my iPad

--

You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+un...@googlegroups.com

[John Doe]

If the issue is the licensing fees then better look at OpenQM which is not the fastest but is about twice as fast as D3 and as far as I know is working fine and has great support. 
There are free developer versions for jBASE, OpenQM, Universe, Unidata and UniVision.
http://www.jbase.com/products/demo/
http://www.openqm.com/cgi/lbscgi.exe?t0=downloads
http://www.rocketsoftware.com/demos
http://www.edp.co.uk/solutions/univision.htm

On Friday, February 17, 2017 at 6:46:14 AM UTC+2, Tom Jeske wrote:

[J Bramley]

Tom,

My name is John Bramley and I'm in charge of MultiValue development at Rocket Software. First, I'd like to highlight that Rocket has made a significant investment in D3, since the purchase of the technology from TigerLogic in November 2013. From reading this thread it sounds like your company is running on a very old version of D3. There has been a significant amount of work done on D3 in the last 12 years and your company might not be aware of the advancement in the technology. You mention: 'Management made the decision but shared these reasons: Less costly site licensing model, better connectivity to other databases, encryption (DREM), able to use C for development.' All of which can be accomplished working with D3 and Rocket.

I'd be more than happy to meet with yourself and your management team to present the latest D3 release and our plans moving forward with the product. At least you'd have clear understanding of what your getting with your current investment in D3 and possibly avoid a very costly and time consuming migration to jBase.

Please let me know how I can be of assistance.

John

[Rick Weiser]

Tom,

I think your company should take John up on his offer.  I have known John for many years and he is a straight shooter.

It can't hurt to talk to him.  You may find the discussion useful.

Just my 2 cents,

Rick

[David Knight]

I have not met John, but my 2c worth is in support of d3. Particularly d3/win. I've read with interest some of the comments in this thread, and while I have no practical experience with multiple mv versions; I can say that I have personally found d3 to be much faster than Universe. A product I nearly switched to many years back.

I have found d3 to be very stable, and the latest releases to be bug free. By way of full disclosure, it would be true to say I do not push the envelope that much, but with that said; I'm extremely happy.

In short, my application started life 20+ years ago on Advanced Pick, including being installed on AP/DOS, AP/Native and then various incarnations after that. I was using SB+ v2.3.2 as a dev system, and later designbais. Everything always migrated without recompile or any syntax issues in all that time.

Therefore, your company would be well served to at least try the latest version of d3. I suspect it would take about an hour and you would be up and running. Then migrate to different dev technologies. Far easier, less stress; and pretty much guaranteed success. Oh, and probably cheaper too.

My 2c worth.

[Peter McMurray]

Hi John Doe
I am surprised at your remark on speed. QM is undoubtedly an excellent product however I believe you must have been comparing unoptimised D3 code for speed.

In fact D3 optimised is blisteringly fast. A recent rather silly remark by an IT person led me to do some tests.

The person declared that NTFS could not handle more than 10,000 records in a folder, even with an index, in a reasonable time frame.

IN fact NTFS uses B+ trees and hashing which has been incorporated into the D3 FSI superbly.

We have credit file transaction files and invoice files that exceed half a million records with no degradation in performance. A few special records that are used all day every day exceed 250,000 bytes. I had planned to break them down before I realised that D3 did that for me with the MATREAD.

I created 250,000 genuine invoice records on an old Compaq AMD Athlon with 4Gb of Ram and timed random reads at over 10 per millisecond.

At month end a client with an Intel I7 and 8 GB of Ram creates PDFs for several thousand customer statements and credit card movement reports by vehicle in under a second. We then sort these PDFs alphabetically in D3 and shoot them out to multi tray HP printer all neatly combined for posting - preparation time again under one second.

I choke when I see people trying to get through large string reads because the MATREAD is and always has been so massively efficient. It actually breaks a record down into a table of contiguous variables in memory, one per attribute. D3 takes this a step further and shunts large variables off to a separate area leaving the base address in the table. A well organised Item with the latest value at the front of the attribute works brilliantly.
I am sure that even an antiquated AIX box could out perform these benchmarks significantly.

[John Doe]

The fact that you've been able to do so many records with D3 is irrelevant because maybe using other MV system you would have done more.
First I've mentioned that my tests were unscientific that is were very limited in scope.
Second I've mentioned that the tests refer to basic performance not file performance.
Third, I did not pretend to test the latest offerings but relatively old stuff such as jBASE 3.
Fourth, all MV systems mentioned were loaded on the same machine so while not scientific the results are indicative of basic performance which include array parsing, matrix processing, for loops, gosubs, subroutine calls and other generic calculations.
More often than not, when a vendor advertises better performance is more or less related to better hardware not better software.  Of course if you compare Universe on a Pentium 2 and D3 on a cluster of Xeon servers with solid-state drives you will see a stellar performance increase.
I've stayed away from file benchmarking because is more or less relative to file optimization techniques specific to each vendor and at the time I was not in the business of benchmarking each system but in the business of evaluating the talk versus the walk.
Well, if you believe that what I said is not true you can always download the free developer versions on the same machine, preferably the latest, and provide us with current performance comparisons.  Like I said before apparently Flash D3, which by the way I have never tested myself, seems to be up with the big boys but this is for somebody else to prove.

Below is an old thread from 2003
https://groups.google.com/forum/#!searchin/comp.databases.pick/benchmark|sort:relevance/comp.databases.pick/jYd-X4eupYY/GEk4WrFcMpIJ

[Peter McMurray]

Fascinating John Doe. Apparently you are using information that is 15 years old and consider it relevant. In fact I do discus this with others using the systems. 

In particular I discussed the specific QM v D3 with a Pick system manager who has been doing it for 37 years to my certain knowledge. He currently uses QM and his only regret is that it is significantly slower than D3.

Personally I have used and supplied Pick systems for 40 years so the software I am using for comparison is identical as I wrote it.

In fact the first time I saw a change that was so massive that you didn't need to time it was Reality 2.4 to 3.0. Since then I have been through many systems from DEC farms and Unidata, many versions of Unix and Universe,Advanced Pick and D3. Currently my office machines have several versions of D3 and QM on Windows 10 and clients are using D3 on Windows 7 PCs and Linux virtual machines.

I must admit that I am intrigued as to how you measure Basic performance and exclude File performance and how this could possibly be relevant to a commercial system. Perhaps you don't understand that even a simple actuarial loop calculation will involve time slice paging to and from disc.

[stope19]

On Wednesday, 22 February 2017 07:49:45 UTC+11, Peter McMurray wrote:

<snip> Perhaps you don't understand that even a simple actuarial loop calculation will involve time slice paging to and from disc.

Maybe you mean 'page faulting'  rather than 'time slice paging' ?  

[geneb]

On Tue, 21 Feb 2017, Peter McMurray wrote:

> commercial system. Perhaps you don't understand that even a simple
> actuarial loop calculation will involve time slice paging to and from disc.
>

Wait, what? If the system is touching the disks at /all/ during non-I/O
ops, you've got a RAM problem on your hands. :)

[Glen Batchelor]

Unless you are running Windows.... 

[geneb]

On Tue, 21 Feb 2017, Glen Batchelor wrote:

> Unless you are running Windows....
>

That just means the host OS screwed you, not the DB environment. :)

[John Doe]

All I see you do is to talk about D3 versus your expectations and about how many satisfied clients you have.  Good for you.
If you have numbers that compare in a fare way various MV systems and not just pub talk please be free and publish them.
For me fare way means testing with some real life programs that do something useful not just add A to B for a million times.
Of course things may have changed in the last 12 years and I don't pretend that I am know-all like you and you are right that I am kind of stupid and I didn't know "that even a simple actuarial loop calculation will involve time slice paging to and from disc."
Well, I think that Tom Jeske got enough food for thought so as far as I am concerned I consider this thread closed unless there are some hard numbers to look at.
Have a nice day.

[David Knight]

Hi John,

Things seem to be escalating, and not in a good way!!!

I can only give anecdotal evidence, but it is realistic because I used the same iron. I'll try to be brief.

I am NOT a Universe expert, so please take that into account. A number of years back, I was looking for a dev environment to replace my ageing character-mode SB+ system. The application was a non-flash system because I was not brave enough to Flash SB+ for fear it would break! As one cannot easily run mixed non-Flash and Flash; I left it non-Flashed. Along came designbais as a dev tool, but at the time ONLY worked on Universe.

As a small developer, I only have ONE Windows PC which does everything, to develop on. designbais staff helped me to get Universe going, helped me port data and got me going with designbais under UV. I used that for a few years trying to redevelop with little success; BUT observationally I found this setup slow. Really slow. Fast forward a couple of years, and designbais released support for d3/win; so I immediately ported what was in designbais/UV over to designbais/d3win ON THE SAME IRON. The difference was staggering. Now, to be fair, designbais on D3/Win HAS TO use flashed code; so I think the first thing everyone needs to accept here is that Flashed is definitely the way to go with d3; any comparison with non-Flashed will fail. Also I realise "staggering" is not a statistical value; but I will testify that the difference was so clear as to be startling.

If I've understood part of your post correctly, you have multiple mv systems available on the same iron? I also note you said you do not want to test yourself, and that's fair enough too. I just wonder if I can pique your interest enough to try? Flash-compiling is easy to do and then you can re-run your own 'real world' comparative test; and then maybe you will find that D3 is not the slowest of the bunch?

Or not, I guess that's up to you.

FWIW, I agree with you that some of the best benchmarks are real-world apps doing real-world stuff on identical iron. It just struck me that perhaps you have an the perfect opportunity to provide those number?

Thoughts?

David

[Martin Phillips]

HI all,

 

It is unfortunate that, even after all these years, there is no acceptable standard benchmark test for multivalue that reflects a real application mix of computation and file i/o. It is fairly simple to construct a test that shows product A and being faster than product B or the other way around if you do this based on knowledge of the internal architecture of the products. The majority of comparison tests that I have seen are based on totally unrealistic repeated operations such as building a 10Mb string by appending one character at a time or copying a large file in a manner that might be significantly affected by differences in hash order.

 

We recently undertook some detailed performance analysis with the aim of making QM faster than one of the other major products for a comparison based on a set of tests that were largely dictated by the client. Aside from pointing us at some areas where we could make improvements (most of which have now been done), these tests revealed that performance measurement is not as easy as it sounds. One of the slightly surprising discoveries was that the performance of the same test (no i/o involved) could differ by as much as 30% from one run to the next, a difference that in many cases exceeded the gains we were trying to make. Further analysis showed that these variations are in some way attributable to the operating system manages processes and totally out of our control. We are still trying to understand the finer detail of this discovery.

 

Another “feature” that we had overlooked in previous tests was that today’s processors change their clock speed automatically to balance between performance and power usage. Simply changing the computer’s power optimisation settings could change the test outcome considerably though this is likely to affect all products in the same way when doing comparisons.

 

For most applications, I suspect that actual performance is influenced more by program design than by the underlying database architecture. If performance is an issue, simply fixing poor design may resolve it to the extent that relative performance of different products is not an issue and choice of product can be based on price, functionality or quality of support services rather than performance comparison.

 

 

Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton NN4 6DB, England
+44 (0)1604-709200

 

 

[John Doe]

David,

HAD multiple MV systems.
I am no longer in the MV business world.Maybe one of the more qualified players will devise some benchmark that reflects real life requirements including of course file management.  Anecdotal testimony regarding installation, conversion, ease of use, connectivity or support is also very useful.  In the end people have to become aware that total cost versus total benefits has to balance their budgets the same way they do when they buy a car or a house and to be aware about all costs not just the license fee which may be insignificant in rapport to total cost.

John.

[Peter McMurray]

Hi Stope

I did not mean page faulting. The gigabytes of memory around today skip that. It was definitely an issue with Reality on 16Kb of core so in those days we mapped everything. I meant Time-slice as we are talking about multi-user systems. Even back in my university days on an IBM 360 student programs were kicked in and out on a time share basis. I chose actuarial because I worked in insurance many years ago and some issues could take a lot of calculation.

As for real world calculations when I see a 1000+ programs flying through in a few seconds on a compile versus the several minutes on other setups, when I see several thousand PDFs produced in under a couple of seconds I feel competent to say that X is faster than Y; especially when my feelings are confirmed by another with many years experience.

Multi-value is all about disk access so I chose reads and in particular mentioned the benefit of MATREAD, that based on my experience I am aware many people do not understand.

Also I can confirm Martin's experience that the underlying OS can cause timing variations.

Which gets us back to my initial observation that the wrong questions are being asked by the people involved.

What do they think cannot be done in D3?

Why do they think a massive rebuild to a totally different type of system is going to be better or cheaper?

I can tell then now that the ludicrous situation of case-sensitivity is going to be a massive issue. This was an issue that first arose with Unix back in the '80s when it appears that designers did not realise that ASCII is designed to be easily case-insensitive. A quick look at http://www.asciitable.com/ will demonstrate this if anyone is in doubt. Those of us coming from a binary coded hexadecimal background (NCR) could see this instantly. I can remember being told in a quite serious manner, by a senior supplier developer, that the accepted wisdom was all item-ids should be numeric. Of course this does not solve the issue of operators using SMITH or Smith nor does it solve the issue of program varName or VARNAME. Scanning data and programs for this alone is a monster.

I avoided optimising my D3 code for quite a while as all my programs are sub-routines but having taken the leap have never regretted it.

[JJCSR]

I am a former D3 client (ver 7.4.3, way long ago), converting to REALITY in late 2008 / early 2009.   One question I do not see posed in this thread, which was very high on my list of needs but may be of no significance to the migrating company, herein, deals with the potential use of either/both DR/Fail-Over ("fail-over" in the form of "redundant" servers).   Please keep in mind - DR is not necessarily FAIL-OVER   At the time of my conversion from D3, I was running on Windows, and, so I was advised, D3/Win did not have the capability (at that time - and on that version) to do the redundant processing that I required.   And, I was not wanting to convert to Unix/Linux/AIX.

Although we have yet to implement DR (Disaster Recovery), we have been running Fail-Over with redundant-processing / transaction-logging servers for about 2 years.   DR is something that we are considering for future implementation, with the secondary server installed in a remote location, some 10-15 miles away.  

Again, maybe these issues (DR/Fail-Over) are not pertinent to the company, as yet, but if either/or seems like something that may well be required, eventually, it may well be worth making the effort to get answers, now, as to the ability of the eventual MV system that will be installed.   When I say, "...may well be required", I do not mean by company-owners, alone.   Outside elements, such as audit firms, could be the deciding factor.

Good luck with your project.

Jim Cronin

Kittery Trading Post

Kittery Maine

On Thursday, February 16, 2017 at 11:46:14 PM UTC-5, Tom Jeske wrote:

[Gerd Förthmann]

Hi,

I have been working with several MV flavours over the years (mainly ADDS Mentor, early versions of D3/Win and UD) and since the last 3 years now with D3/Linux. The biggest gripe I have with D3 is indexing, which is just rubbish!
It sort of works with Basic (KEY) but sorting on indexed attributes is just as slow as on non-indexed ones. And why do you still have to index on A-correlatives and not on Dict items like in U2?

So if you can get that to work  that would be a big step forward.

Another thing that doesn't work properly is UTF-8 support. For English speaking countries that obviously doesn't matter but for us in Germany for instance it is a BIG problem since every UTF-8 character counts as 2. And since I work for an International company with sites all over the world support of local languages is paramount if we want to introduce our software at every site.

BTW the company I'm working for also plans to migrate from D3 to jBase starting in China and we are all running on the latest version of D3 already.

--

[Peter McMurray]

Hi Mecki

At last the elephant in the room is raised. UTF-8 is the universal storage medium of the internet.

Pick data is UTF-8,JSON-UNICODE compliant over all 17 planes.

I wrote UTF-8 compliant edit routines in D3 Basic over 6 years ago and sent the test data to D3 support asking for the very minor change required to render the report generator fully compliant.

I was promised that for 10.2 and was very disappointed to find a silly statement in the manual that UTF-8 is being converted to ANSI. WHAAAT!

Is this because some system programmers are obsessed with C in 16 or 32 bit and confuse program code with data?

"Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as end of string.Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as end of string." Quote from Wikipedia.

Bob Rasmussen made a minor change to Anzio for me to use and Pete Schellenbach was concerned about displaying double width characters. I did data tests for Pick storage using  INUIT and Chinese characters in Notepad as well. I cannot remember off the top of my head but I do not believe that there are any double width characters in Chinese as standardised by Mao, they are found in the several earlier sets of characters.

Pick Attribute, Value and sub-value marks fall in the special character non Unicode area so there is no problem. It takes one line of Basic Code to recognise what you are dealing with. Surely inserting that into the master system code is not that difficult for OCONV and the like.

At the time I had prospects in Singapore and Moscow. Well life is what happens when you are planning something else - in my case heart attacks. As I fade into the distance nothing would delight me more in this arena than to see D3 designers wake up.

[Wols Lists]

On 25/02/17 20:52, Peter McMurray wrote:
> I was promised that for 10.2 and was very disappointed to find a silly
> statement in the manual that UTF-8 is being converted to ANSI. WHAAAT!

How on earth is this conversion anything else but lossy?

aiui, ansi has 127 characters. UTF-8 has thousands. How are you supposed
to encode thousands of possible values, when you only have 127 unique
values to store them in?

I really don't understand what you (or Rocket) are saying, but that
statement you've quoted just does NOT make sense to me :-(

Cheers,
Wol

[Glen Batchelor]

UTF-8 encoding could be retained as a blob inside a D3 file but data field char translation still has to be done to ASCII due to char byte space as you said. Also indexing and other byte specific tech in the engine was tweaked for ASCII charset.

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

[Gerd Förthmann]

Wol,

Welcome to 8-bit bytes without which MV couldn't work anyway.

Unlike the 7-bit ASCII character set UTF-8 uses 1 to 4 8-bit bytes
(octets) for each character so you can store more than a Million unique
values. That's why UTF-8 is the most popular encoding of the internet.

All they need to do to make D3 truly UTF-8 compliant is to count
characters instead of bytes.

Here is a simple example:

ÄÖÜäöüß - How many characters do you see here?

Well, I see 7.

And how long is that string?

I would also say 7.

But according to D3 it is 14 since this string of 7 UTF-8 characters is
14 bytes long.

Now take this string - aäoöuüs

How long is that one?

7, 10 or 14?

And what would you expect to see if you use this code?

VAR = "aäoöuüs"
CRT VAR[1,5]
CRT VAR 'L#8'

[Anthony Youngman]

On 26/02/17 09:06, Gerd Förthmann wrote:
> Wol,
>
> Welcome to 8-bit bytes without which MV couldn't work anyway.

No. I understand that completely. the point is, aiui, ANSI is a
seven-bit character set - or have they updated the standard?

>
> Unlike the 7-bit ASCII character set UTF-8 uses 1 to 4 8-bit bytes
> (octets) for each character so you can store more than a Million unique
> values. That's why UTF-8 is the most popular encoding of the internet.
>
> All they need to do to make D3 truly UTF-8 compliant is to count
> characters instead of bytes.
>
> Here is a simple example:
>
> ÄÖÜäöüß - How many characters do you see here?

Except, aiui, that is NOT ansi. So if D3 is capable of displaying it
correctly, then Rocket's statement that they're converting everything to
ansi is just - demonstrably - wrong.

>
> Well, I see 7.
>
> And how long is that string?
>
> I would also say 7.
>
> But according to D3 it is 14 since this string of 7 UTF-8 characters is
> 14 bytes long.
>
> Now take this string - aäoöuüs
>
> How long is that one?
>
> 7, 10 or 14?
>
> And what would you expect to see if you use this code?
>
> VAR = "aäoöuüs"
> CRT VAR[1,5]
> CRT VAR 'L#8'
>

I get all that - treating a utf-8 string as if it were an
extended-ascii/ansi byte string is going to lead to stupid results, but
equally, saying you're going to squeeze a metric tonne (utf-8) into a
pint pot (ansi) is going to lead to equally stupid results :-)

(the only place that would work is on a dead star :-)

Cheers,
Wol

[Peter McMurray]

Hi Wol and others

PICK Data is 8 bit and US-ASCII is 7 bit.

Pick uses some characters as delimiters from the extended Ascii set so it is not fully compatible with extended Ascii but because this is allowed for in Unicode it works fine.

All Pick code is designed around using these delimiters so it works fine with UTF-8 which can never have these characters as data.

As I said I have fully tested all the Unicode range in UTF-8 Pick.

One simply does character input instead of field input. input getByte,0 instead of input getFld. If the character is < 128 then it is US-ASCII if it is greater then it is the start of a Unicode string that tells you how many bytes make up the character.

This allows one to happily use back arrow, forward arrow and insert keys within a field input instead of having to replace the entire field to alter it.

Better still one does this in a tiny sub-routine which is by far the best way of using D3.

Having captured a field and ensured that it is of valid character type e.g. numeric etc. one can then call a second subroutine if necessary to do field validation either date, within limits or read a record etc.

It is lightning fast and it fits the Pick data model perfectly. LIST works perfectly. It is only when there is a length count on the display field that it gets tangled because LIST counts bytes instead of characters.

If one is in any doubt then this is where I use VME (The only place I use VME for data) the GROUP and DUMP GX verbs show the data exactly as stored on the disk. Better still this data is compatible with the outside world in any language.

[Peter McMurray]

Addendum to above message. I demonstrated with a Pick input however it was my intention to use XAML and F# as the major input routines and simply pass the screen capture back to the Pick database. However it appears that someone has decided to make a complete mess of that by fooling around with ANSI which is not compatible with PICK. y acute is character 253 and y umlaut is character 255. It may well be that because these are not valid individual characters in Unicode it wont matter. But why do it in the first place?

[Kevin King]

At the risk of coming off like a heretic, this is why I think Multivalue solutions could benefit from using JSON as the native storage format. Unlimited depth, native UTF-8 support, and no more delimiters.

Would love to hear other people's input on the idea. QM has native JSON support, but that's the only one that I know of that has given anything more than a passing glance to JSON in the storage engine.

On Feb 26, 2017 3:19 PM, "Peter McMurray" <pgmcm...@gmail.com> wrote:

Addendum to above message. I demonstrated with a Pick input however it was my intention to use XAML and F# as the major input routines and simply pass the screen capture back to the Pick database. However it appears that someone has decided to make a complete mess of that by fooling around with ANSI which is not compatible with PICK. y acute is character 253 and y umlaut is character 255. It may well be that because these are not valid individual characters in Unicode it wont matter. But why do it in the first place?

--

[Peter McMurray]

Hi Kevin

I am a great believer in "If it works don't fix it"

All Pick data is built around delimiters and they work fine.

I can see considerable massive issues introducing backslash escapes into the terabytes of data that are stored using Pick delimiters.

Pick can store JSON now as every Pick item is a Name/Value pair. and in a well-designed Pick Database every attribute is a Name/Value pair with the Name being the A attribute and S being a synonym.

I have dived further into the D3 manual and can see where the manual writer has become massively confused. They state

"NOTE—FlashBASIC does not currently support UNICODE, therefore the item is converted to ANSI and back to UNICODE after the trigger completes."

ANSI is an 8 bit code that has nothing to do with Unicode they should have just said that all data is passed as 8 bit character strings.

[Bob Rasmussen]

While recognizing the risk of contributing to a thread which I have only
skimmed, I must point out that this use of the term "ANSI" is imprecise at
best. There are many ANSI standards. But when it comes to character sets,
I think you (Peter) are referring to ISO character sets. There are many of
these too, most notably ISO-8859-x. For instance, ISO-8859-1 "Latin-1",
which is mostly the same as Windows codepage 1252, "Western". (Windows
codepages usually have characters assigned to hex-80 through hex-FF, which
ISO 8859 leaves unassigned.)

But I concur on your statements about UTF-8, including that it does not
collide with values used as Pick delimiters.

On Sun, 26 Feb 2017, Peter McMurray wrote:

> Hi KevinI am a great believer in "If it works don't fix it"

> --
> You received this message because you are subscribed to
> the "Pick and MultiValue Databases" group.
> To post, email to: mvd...@googlegroups.com

> To unsubscribe, email to: mvdbms+un...@googlegroups.com

> For more options, visit http://groups.google.com/group/mvdbms
>
>

Regards,
....Bob Rasmussen, President, Rasmussen Software, Inc.

personal e-mail: r...@anzio.com
company e-mail: r...@anzio.com
voice: (US) 503-624-0360 (9:00-6:00 Pacific Time)
fax: (US) 503-624-0760
web: http://www.anzio.com
street address: Rasmussen Software, Inc.
10240 SW Nimbus, Suite L9
Portland, OR 97223 USA

[Kevin King]

I do not claim any mastery of UTF-8, but Bob, if that were true wouldn't you expect an ASCII 254 to be considered valid in a JSON string that is implicitly being interpreted as UTF-8?

We have done many experiments with this and have found that anything over ASCII 127 needs to be encoded otherwise it is considered an error in the JSON string.

So how can an attribute mark be a valid UTF-8 character and yet cause a JSON parsing operation to fail?

On Feb 26, 2017 5:04 PM, "Bob Rasmussen" <r...@anzio.com> wrote:

While recognizing the risk of contributing to a thread which I have only skimmed, I must point out that this use of the term "ANSI" is imprecise at best. There are many ANSI standards. But when it comes to character sets, I think you (Peter) are referring to ISO character sets. There are many of these too, most notably ISO-8859-x. For instance, ISO-8859-1 "Latin-1", which is mostly the same as Windows codepage 1252, "Western". (Windows codepages usually have characters assigned to hex-80 through hex-FF, which ISO 8859 leaves unassigned.)

But I concur on your statements about UTF-8, including that it does not collide with values used as Pick delimiters.

On Sun, 26 Feb 2017, Peter McMurray wrote:

Hi KevinI am a great believer in "If it works don't fix it"
All Pick data is built around delimiters and they work fine.
I can see considerable massive issues introducing backslash escapes into the
terabytes of data that are stored using Pick delimiters.
Pick can store JSON now as every Pick item is a Name/Value pair. and in a
well-designed Pick Database every attribute is a Name/Value pair with the
Name being the A attribute and S being a synonym.

I have dived further into the D3 manual and can see where the manual writer
has become massively confused. They state
"NOTE—FlashBASIC does not currently support UNICODE, therefore the item is
converted to ANSI and back to UNICODE after the trigger completes."
ANSI is an 8 bit code that has nothing to do with Unicode they should have
just said that all data is passed as 8 bit character strings.

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com

To unsubscribe, email to: mvdbms+unsubscribe@googlegroups.com

For more options, visit http://groups.google.com/group/mvdbms

[Kevin King]

On Feb 26, 2017 3:59 PM, "Peter McMurray" <pgmcm...@gmail.com> wrote:

Hi Kevin

I am a great believer in "If it works don't fix it"

Oh I agree but could we be missing opportunities because we are (loosely) limited to 3 dimensions in a record and have spotty I18N support?

All Pick data is built around delimiters and they work fine.

And there are people who will say that SQL doesn't support Multivalue data and that works "fine". My toaster doesn't give me internet access and that's fine. The efficacy of "fine" therefore must be defined by a larger context. In term of the Multivalue industry, I think we impose artificial limitations because we are "fine" with technology that is still largely based on a 50+ year old context. Could we do more in the global marketplace by looking at more current contexts like deeply nested objects and industry standard I18N?

I can see considerable massive issues introducing backslash escapes into the terabytes of data that are stored using Pick delimiters.

Oh I agree. Never suggested it was an easy transition... Just floating an idea.

Pick can store JSON now as every Pick item is a Name/Value pair. and in a well-designed Pick Database every attribute is a Name/Value pair with the Name being the A attribute and S being a synonym.

I think then we may not agree on what JSON is or what it brings to the party. Yes, we can store JSON in Multivalue data stores today. But we lose much of the value of JSON of we convert it to delimited MV format. I was merely wondering if making JSON the foundation of the underlying storage if we might bring new, exciting, and needed abilities into the platform?

[Glen Batchelor]

Supporting JSON does not automatically make it unicode compatible. JSON is just another markup format. Getting D3 to support JSON does nothing the change the reliance on ASCII charset for specialized data functionality such as indexing and soundex. No, JSON is not going to make D3 unicode compatible even if it's adopted as an optional storage methodology. The database itself needs to be rewritten to have byte-space encoding support and the indexing functionality needs to be optimized to work based on bit patterns and not ASCII strings.

--

[Kevin King]

Glen, agreed.  I'm not suggesting that converting to JSON implicitly makes anything internationalized.  I am, however, floating the idea that there may be ways that MV storage could be improved and JSON has a couple of interesting tick marks in the benefits column.

--

-K

[Glen Batchelor]

 If internationalization is an issue and it hasn't already been addressed then frankly the ship sailed 10-15 years ago. The global market is here to stay and even on Infor SX.e I have issues with ISO8859-1 conversion where UTF-8 is required and our Progress data tables are still in ISO8859-1. Even with encoding support you can only handle 1 charset per storage segment so do you really think that the MV will be MV if you have to support multiple encodings? I think, at that point, you loose the efficiency of what makes MV such a lean database.

[Wols Lists]

On 26/02/17 22:34, Kevin King wrote:
> At the risk of coming off like a heretic, this is why I think Multivalue
> solutions could benefit from using JSON as the native storage format.
> Unlimited depth, native UTF-8 support, and no more delimiters.

Which is why, in my "forever toying with the idea of writing an Open
Source Pick", I'm looking at using utf-8 delimiters. Last I looked,
however, I think all the user-defined characters are three bytes long
and maybe had other quirks I considered undesirable too. I was trying to
find something where checking a single byte said "this is a delimiter",
and the code could then dig deeper if needed.

Cheers,
Wol

[Wols Lists]

On 27/02/17 00:13, Kevin King wrote:
> I do not claim any mastery of UTF-8, but Bob, if that were true wouldn't
> you expect an ASCII 254 to be considered valid in a JSON string that is
> implicitly being interpreted as UTF-8?
>
> We have done many experiments with this and have found that anything
> over ASCII 127 needs to be encoded otherwise it is considered an error
> in the JSON string.
>
> So how can an attribute mark be a valid UTF-8 character and yet cause a
> JSON parsing operation to fail?

Because you're confusing characters and bytes? To be valid utf-8, an
attribute mark needs to be a two-byte character. So sticking a Pick
attribute mark in the middle of a utf-8 JSON blob *will* cause something
to blow up.

I can't remember the rules, but the first byte in a UTF-8 character
always starts with a 0 bit, and all subsequent bytes start with a 1 bit.
So 0xFE can NOT be a valid UTF-8 character, because it doesn't start
with a 0 bit.

Cheers,
Wol

[Peter McMurray]

Hi Wol

Attribute marks work well with Unicode in UTF-8 because they are eight bits but invalid UTF-8 Unicode.

All Pick systems are designed to pull out PICK delimiters first so the fact that they are invalid Unicode is great they work as single characters and the lead bit is 1 not zero.

I think you may be confusing US-ASCII which is 7 bit encoding one byte per character and therefore a leading zero bit with UTF-8 Unicode that uses all 8 bits in a variable number of bytes per character.

To ensure compatibility with US-ASCII if the lead bit is zero in UTF-8 Unicode then it is a single US-ASCII character all other characters take two or more bytes..

[Gerd Förthmann]

Peter,

I appreciate that you tested to input and display every possible UTF-8
character in D3.

But Input and display are not the problem; you don't even have to change
any code for that.

But with this statement you nearly hit the nail on the head:
"List works perfectly. It is only when there is a length count on the

display field that it gets tangled because LIST counts bytes instead of
characters."

With other words LIST only works "perfectly" as long as your data only
contains ASCII characters since every display field has a length count.
And the same applies to Basic reports and string handling including
OCONV with MC conversions.

As I said before - if you want D3 to be truly UTF-8 compliant it ALWAYS
has to count characters not bytes!

And as long as it doesn't it is a more than valid reason to swap to a
database that does.

> --
> You received this message because you are subscribed to
> the "Pick and MultiValue Databases" group.
> To post, email to: mvd...@googlegroups.com

> To unsubscribe, email to: mvdbms+un...@googlegroups.com

[Peter McMurray]

HI Mecki

I am afraid that you are wrong. List works perfectly with UTF-8 characters from any plane so long as you do not limit the length of the display thus proving the storage is fine.

However I am grateful for you raising the topic that I have been chasing D3 for over many years.

I posed the question "what can Jbase do that D3 can't"

Following your lead I decided to look at Jbase again and was stunned they document what I have been saying about the ease of change to UTF-8 in Pick.

I am sold Rocket better get a move on before they lose all international customers.

http://www.jbase.com/pdf/docs/jBASE-Internationalization.pdf 

[Kevin Powick]

On Tuesday, 28 February 2017 00:18:02 UTC-5, Peter McMurray wrote:

HI Mecki

I am afraid that you are wrong. List works perfectly with UTF-8 characters from any plane so long as you do not limit the length of the display thus proving the storage is fine.

Storing your data is one thing, manipulating it is another.  As already pointed out by Mecki, at least a few, common Basic language features will not work correctly with UTF-8 data that requires more than one octet of storage.

Examples:

LEN()

SEQ()

COL1()

COL2()

FOLD()

MY.CHARS = MY.REC[1,4]

There are too many possible "gotchas" with storing UTF-8 data in current versions of D3 to make it a good idea.

--

Kevin Powick

 

[Peter McMurray]

Hi Kevin

I find it disappointing that you and Mecki make somewhat ridiculous statements without bothering to read the detail of my statements and the JBASE article I referred to.

Firstly D3 counts bytes not characters. 

That is why the List command works provided you do not limit the number of bytes in an attribute.

 List fileName A1 for example will happily list whatever bytes it finds in the first attribute thus doing a basic test that the data stored in the first attribute is successfully delimited by the attribute mark without confusing it with UTF-8 Unicode.

The commands that you mention are obviously the ones that I use to control the data

input getByte,0 gets the first byte

seq(getByte) tells me the value of the byte received. if it is less than 128 it is US-ASCII which is only valid if it is the first byte in a sequence otherwise it is part of a UTF-8 sequence and edited accordingly.

col(1) etc. allow me to break the field at the appropriate point as the manual says "col1() allows nonsystem delimited strings to be handled like dynamic arrays"

Mecki makes the similarly silly statements that all characters are two bytes and that input does not need altering.

In fact most German characters are single byte US-ASCII in UTF-8 and it is only the special characters with diaresis such as umlaut or the unique characters such as the lower case sharp s " the grapheme ß, called Eszett"  that are more than one byte long.

I am intrigued with the idea that he seems to think the D3 single byte backspace will somehow clear that without altering the input.

The major reason I dropped further work on QM was because i wanted to use Unicode and in my opinion it was a ridiculous mistake to go to 16 bit, This may have seemed an easy fix but it only covers the basic levels of unicode and introduces the major problems of endiness that UTF-8 was designed to bypass regardless of machine architecture.

[Kevin Powick]

Hi Peter,

I find it silly that you conflate terms like Unicode, UTF-8, etc., yet ignore most of the examples given to you that demonstrate why D3 BASIC does not correctly handle data with a character encoding that results in more than one byte per character. (e.g. UTF-8 above ASCII 127).

It really is trivial to write a few simple test programs in D3 Windows that demonstrate my assertion.  The one below fails spectacularly on the UTF-8 data found in REC<2>

I agree that D3 can store the data but, again, it does not correctly process that data via several, commonly used D3 BASIC functions.

REC = ''
REC<1> = 'I can eat grass';* English
REC<2> = 'ὕαλον ϕαγεῖν δύναμαι· τοῦτο οὔ με βλάπτει';* Classic Greek

CRT REC<1>
CRT REC<2>
*
CRT LEN(REC<1>)
CRT LEN(REC<2>);* Fail
*
CRT REC<1>[1,5]
CRT REC<2>[1,5];* Fail

CRT FIELD(REC<2>,'λ',1);* Fail
CRT SEQ('λ');* Fail

:RUN BP TEST

I can eat grass

ὕαλον ϕαγεῖν δύναμαι· τοῦτο οὔ με βλάπτει

15

82

I can

ὕα

ὕ

206

--

Kevin Powick

[Peter McMurray]

Well Kevin the only failure in your example is the stupidity of deliberately printing a string of bytes that you are aware is incorrect. I can see that it is pointless continuing.

I suggest that you read the JBASE document and gain a better understanding of the issue.

[Kevin Powick]

Peter,

Incorrect string of bytes?  LoL.  It's 100% valid UTF-8 data.  The very type of data you erroneously claim D3 handles without issue.

The only stupidity is your denial of the fact that various, common D3 BASIC functions do not work correctly with UTF-8 data.

The only relevance of that jBASE document is that as of version 4.1, the product has undergone substantial enhancements to allow for "internationalization" via compatibility with UTF-8.  So substantial are the changes, that it takes 55 pages to explain them and the considerations users must make for going down that path. 

All that being said, I do agree with you on one item.  It is pointless to continue this conversation.

--

Kevin Powick 

[Gerd Förthmann]

If somebody is silly here then it's neither Kevin nor me.
All we tried to point out is that the D3 UTF-8 implementation is seriously flawed.

BTW I am German and work in Germany - and yes, I work with D3 and we use those unique German characters here in Germany every day.
And it is not unusual to have 5 or more 2 byte characters in a 30 character string.
When I refer to 'German' characters I mean characters that are unique to the German alphabet (ÄÖÜäöüß) (even though those are also used in other languages like Finnish and Turkish).
And those are all 2 bytes in UTF-8.
All other letters in the German alphabet are also in the English alphabet and of course single byte ASCII.

Before we upgraded to UTF-8 we were using the German character set in wIntegrate, where the letter ä for instance is stored as }.
So when we were told that D3 now supports UTF-8 we decided to upgrade and convert all those substitute ASCII characters in our database to UTF-8 instead migrating to another database.
Which wasn't without problems because one of those substitute characters is /.
Only then to find out that we suddenly had all sort of problems with our software not working properly any more.

Thank you for pointing out, that Backspace doesn't work in D3 with UTF-8 characters too - I hadn't even noticed up to now.
So when a user reports missing characters I now know what could have caused it.
One more reason to ditch D3.

And how silly is this statement? "the List command works provided you do not limit the number of bytes in an attribute."
When I use LIST or SORT I usually want to display data in columns.
And to do that I use dictionary items; and dictionary items require a justification in attribute 9 and a length in attribute 10.
Our software uses LIST in procs to display data on the screen or to print reports - and that doesn't work properly regardless if you think it is silly or not.

Cheers - end of thread for me

[Ian McGowan]

It seems like a great idea to have the ability to write a record to json in Unidata DIR-type or Universe Type-19 files, and vice versa.  But using it as the storage engine sounds like a really large undertaking that would have all kinds of impacts to how data is queried, indexed, locked etc.  Martin could probably provide a more informed opinion, but it sounds like the kind of thing that would require some severe surgery to the internals of the MV databases.

MongoDB uses json data structures, and it's interesting how they decouple the database engine from the storage engine to allow multiple options.  So there's at least one other system that lets you choose a different engine.  Perhaps it would be easier to write a Basic -> Javascript compiler and run on Mongo instead? ;_)

https://www.percona.com/blog/2016/01/06/mongodb-revs-you-up-what-storage-engine-is-right-part-1/

[Ian McGowan]

After googling a little more, and reading this article http://mvdeveloper.blogspot.com/2015/09/qm-and-vfs.html, it seems like QM has that pluggable storage engine.  A very cool concept!  I wonder what query performance is like for the non-native engines though?

[Peter McMurray]

Hi Mecki

I believe that we would all appreciate your organisation reporting the salesman that stated D3 supports UTF-8 as that sort of stupidity hurts us all.

Unfortunately my posts have not been read with care. I have persistently made the point that ALL PICK DATABASES CAN STORE UTF-8 without alteration.

D3 needs only minor alteration to enable UTF-8 characters to be handled correctly in Basic and report generation. In fact sensible use of the current features enable UTF-8 to be used in Basic in most circumstances. Silly examples of pulling out a byte where a string is required do not assist the discussion.

In particular LIST could be fixed with a very minor alteration to count characters instead of bytes.

SORT has only ever worked for US-ASCII since all other users have Locale requirements easily met in Unicode and particularly UTF-8 in Pick.

Apparently some people find it to difficult to read a manual so I will quote from the excellent Jbase write up that explains everything and points out the very minor alterations required for Pick. Warning to Rocket: Jbase are actively targeting D3, see the latest alterations to CASING for example.

" The Future and UTF-8

The industry is converging on UTF-8 and Unicode for all internationalization. Microsoft NT is built on a base of Unicode; AIX, Sun, HP/UX all offer Unicode support. All the new web standards, such as HTML, XML, etc., support Unicode. The latest versions of Netscape Navigator and Internet Explorer both support Unicode. UNIX support for Unicode for directory names is provided via UTF-8.

Because of these difficulties, the major Unix distributors and developers foresee Unicode eventually replacing older legacy encodings, primarily in the UTF-8 form.

UTF-8 will more than likely be used exclusively in:

 Text files (source code, HTML files, email messages, etc.)

 File names

 Standard input and standard output, pipes

 Environment variables

 Cut and paste selection buffers

 Telnet, modem, and serial port connections to terminal emulators" 

In regard to terminal emulation both Anzio and Accuterm support UTF-8

[Kevin Powick]

On Friday, 3 March 2017 18:09:01 UTC-5, Peter McMurray wrote:

 

" The Future and UTF-8

The industry is converging on UTF-8 and Unicode for all internationalization. Microsoft NT is built on a base of Unicode

People need to stop conflating Unicode and UTF-8.  Unicode is a standard for the consistent encoding, representation, and handling of text.  UTF-8 is one of several "character encodings" used to implement the Unicode standard.  BTW, Windows is UTF-16.

--

Kevin Powick

[Peter McMurray]

Still at it Kevin.

Do not quote a JBASE document as being written by me.

UTF-8 is the Unicode form designed to avoid the issues created by using 16 or 32 bit encoding. In particular it is the ideal format for Pick which is an 8 bit environment and it is 100% compatible with US-ASCII i.e. practically all Pick Data to date.

It is accepted as the default for XML and required for standard email programs. It is used by 88.4% of all Web pages as of January 2017. It is even used for Unix file names.

QM had a business reason for using 16 bit that limits the use whilst addressing a large chunk of their European market. That is perfectly reasonable and it does not change the fact that D3 would be best served by UTF-8 with a minimal change from byte emphasis to character emphasis as JBASE has done. the extra code would not impact the speed  and use of most current D3 installations as UTF-8 is 100% US-ASCII compatible. I can assure you that Microsoft does not limit itself to UTF-16 as I did all my document and data capture testing using Microsoft and the appropriate character sets. I double checked the results by copying my FSI results to the VME and using the DUMP command. By the way I can handle your Greek example perfectly in D3 including word counts, extractions and character counts. All I want is that the OS do the simpler counting so that Access works properly and a cople of commands that do the associated ICONV/OCONV more easily.

By the way there are normalisation techniques to solve SORT issues that could be readily incorporated into the current D3 that only handles US-ASCII correctly anyway at the moment.