Help in migrating to Cache from Pick

[ChrisB]

Hi, We are currently migrating our Pick based system to Cache and one of the main reasons is to speed up the processing time. The system is basically an accommodation enquiry system and when overloaded by users the processing time is considerably downgraded. Any tips on amending programs to suit the Cache enviroment would be much appreciated. Thanks

[Dave]

Hi Chris, are you using D3 (or another non-case-sensitive flavor)? If so, the biggest headache you will have will be the fact that Caché is case sensitive.  This includes dictionary names, item IDs, logic code, etc.  The second biggest headache is the limit of string length, item size, and routine size (unless this has changed in more recent versions - we're still working with 2009.x for the moment).  If you have really large programs (or programs with lots of large includes) they will not compile and will need to be broken up into subroutines.

Once you get beyond those issues, the rest will seem easy.  Step one is to get your programs to compile in Caché, step two is to make sure they work.  Then I would start to work on performance (i.e. indexes).  Once you're fully migrated into Caché you can start taking advantage of the Caché-specific tools like creating classes and working with objects, using the built in classes for networking/sockets/file (OS file) manipulation, etc. and you should be able to gain some additional performance gain there too.

Dave

On Tue, Sep 11, 2012 at 10:38 AM, ChrisB <cba...@blueyonder.co.uk> wrote:

Hi, We are currently migrating our Pick based system to Cache and one of the main reasons is to speed up the processing time. The system is basically an accommodation enquiry system and when overloaded by users the processing time is considerably downgraded. Any tips on amending programs to suit the Cache enviroment would be much appreciated. Thanks

--
You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

[Rich Taylor]

Chris,

 

As Dave indicated some the responses will depend on where you are coming from.  If you are coming from D3 then case sensitivity becomes a issue due to D3 being case insensitive .  If you are coming from Universe or Unidata then this will not be a problem as both those environments are case sensitive also.

 

One other thing that Dave had mentioned has been dealt with.  For practical purpose we no longer have a limit on routine (program) size.  While on the topic of program one thing you will notice is that our compiler is tighter than many others in the MV world.  Some examples:

·         Spaces in line label definitions;  HANDLE.LINE :  is on legal  due to the space before the:

·         Reserve word conflicts

·         Missing concatenation operators;  LINE = SomeData “ additional text added to the end”.  The quoted text get interpreted as a formatting option due to the : missing between SomeData and the quote

 

For further feedback I need to understand if you are referring to issues that may come up in migration or if you are looking for ways to get better performance out of Caché.  Let me take a brief shot at both

 

Issues beyond the syntax items show above:

·         Reading large blocks of text into memory.  I have run into cases where, rather than using READSEQ to read in a large text file, the record is put into directory type file and input using READ.  The issue here is that Caché has a 3.6 million character limit on strings.   You can handle this in two ways.  If you want to stick with purely MV Basic commands you can do READSEQ to handle the file line-by-line.  A better way would be to use our File Stream handling API.   The program would still be written in MV Basic, you would just use our API in place of MV Basic statements.

·         Socket handling uses our standard Sockets API

·         HTTP handling uses our standard API

·         Web Services would need to be implemented using our methodology, which is very easy by the way.  You could still just call an existing MV Basic subroutine from the web service.

 

Performance

·         Caché adds some new in-memory structures that can be used in place of Dynamic arrays.   I have used both these techniques in MV Basic programs to achieve vast improvements in performance.

o   Cache Arrays – these are arrays that can be indexed by any string value.  For example it is a common technique to build a dynamic array with two VM separated attributes.  The first contains a code and the next a value related to that code.  When you need that value you do a locate on the first attribute to find the position of the value in the second attribute.  With a Caché array you can make the index of the array the code.  So instead of doing and locate on CODE you would do:  VALUE = LOOKUPARRAY(CODE).

o   There are also listing structures, called $LISTs, that are binary in nature and extremely fast

·         Use our Class API’s.  There are many cases where we have an API to handle a particular function that performs better.  This can be due to simplifying code or just due to that function be highly optimized.  A great example is email.  In Cache we handling this completely within an API rather than having to call out to OS level functions.

 

We can help out more as specific issues come up.  One of the greatest advantages InterSystems and Caché brings to the MV World is options and support.   You have already found this Community forum and I would continue to use it.  Post here any time you have something that you need to  deal with.  Of course your Sales Engineer is a great resource for these issues.  Finally,  our support group is world class. 

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

--

[James Westley Farrell]

Actually using streams for handling any large hunks of data is a very nifty way to handle them.  In a previous Cache tip I replied that you can set the end of line character to @fm instead of the default crlf and use a stream to build huge dynamic arrays really quickly. 

Using a file stream is probably the fastest and least costly way to handle large files on disk as well.  I've used a combination of file and global streams in MVBasic programs just lately to build a nifty parser for HL7 files, such beasts being pretty huge when you can see something like 30,000 accessions in a day.  (You just can't beat MVBasic for string slicing and dicing, no matter what my COS cohorts here in the shop might say :) )  Leveraging a combination of traditional dynamic arrays and Cache multidimensional arrays along with a combination of global and file streams, you can take a legacy MV app and give it some mindblowing speed.

Rich, it's well worth devoting a short thread or two to streams.  They're an underappreciated workhorse that can really speed up an application.

PS:  I'm still here, surfacing for a breath of fresh MV air every once in a while.  The MUMPS world hasn't completely assimilated me yet... resistance is so far not futile :)

Bill

--
"He is your friend, your partner, your defender, your dog. You are his life, his love, his leader. He will be yours, faithful and true, to the last beat of his heart. You owe it to him to be worthy of such devotion" -- Unknown

[Lee Burstein]

Watch for an upcoming Tip on Stream Objects, especially with EXECUTE, in November.

<image001.jpg>

 

From: intersy...@googlegroups.com [mailto:intersy...@googlegroups.com] On Behalf Of ChrisB
Sent: Tuesday, September 11, 2012 10:38 AM
To: InterSy...@googlegroups.com
Subject: [InterSystems-MV] Help in migrating to Cache from Pick

 

Hi, We are currently migrating our Pick based system to Cache and one of the main reasons is to speed up the processing time. The system is basically an accommodation enquiry system and when overloaded by users the processing time is considerably downgraded. Any tips on amending programs to suit the Cache enviroment would be much appreciated. Thanks

--
You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

--
You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

--
"He is your friend, your partner, your defender, your dog. You are his life, his love, his leader. He will be yours, faithful and true, to the last beat of his heart. You owe it to him to be worthy of such devotion" -- Unknown

--
You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

Lee H. Burstein

Technical Trainer

InterSystems

Office: 617-225-3145
Home Office: 302-477-0180
Cell: 302-345-0810

[James Westley Farrell]

Awesome, Lee.  In the meantime, here's a piece of code I adapted and included in one of my library classes.  You hand it a pathname.  It reads the directory and gives you back a dynamic array with information about each file in it.  It goes like this:


/// Return a list of files in a directory.  Result is returned in a dynamic array (@fm-delimited)
/// with each attribute containing three values separated by value marks (@vm):<br />
/// full_pathname @vm date_modified @vm file_size
ClassMethod ListDirectory(Pathname As %String = "") As %String [ Final ]
{
 
 If Pathname="" Then Return ""

 Result = "%Library.GlobalCharacterStream"->%New()
 Result->LineTerminator = @fm ; * set the line terminator to @fm to make the stream builder give you a dynamic array

 FilesRS = "%Library.ResultSet"->%New("%Library.File:FileSet") ; * now ask the system to hand you a directory listing as a result set
 FilesRS->Execute(Pathname,"*.*")

 Loop
 While FilesRS->Next()
 
     FileFullPath      = FilesRS->Get("Name")
     FileDateModified  = FilesRS->Get("DateModified")
     FileSize          = FilesRS->Get("Size")

     Result->WriteLine( FileFullPath : @vm : FileDateModified : @vm : FileSize ) ; * so each attribute has three values: the path, the modified-date, and the file size
    
 Repeat

 Result->Rewind() ; * point the stream back to the beginning
 Return Trim( Result->Read(3641144), @fm ) ; * return one hig-honkin' dynarray
}


To use this, copy this classmethod into your library class and recompile the class.  Then, if for instance your class is called MyLibrary and lives in your package called MyPackage and you wanted to list C:\Some\Path, you'd write:

TheListing = "MyPackage.MyLibrary"->ListDirectory("C:\Some\Path")

and one return, the variable TheListing would have a dynamic array in it where each attribute has three values:  the complete path name of a file, the date it was last modified, and the size in bytes.  (I'm sure there may be more fields you "could" have, but these are the ones I know about.  Lee? Rich?)

Long and short, this is mind-bogglingly faster than using OPEN, SELECT, LOOP WHILE READNEXT, READ, accumulating a dynarray with <-1> (which is probably the slowest possible way to build one).... REPEAT.  Anywhere you need to build a large dynamic array, using a global stream (and setting the line terminator character to @fm like in the example above) is much faster.

Note:  the Write() method will let you append to an attribute (without writing the @fm "EOL" character) and the WriteLine() method writes the EOL character (in our case, a field/attribute mark.

Looking forward to what other goodies Lee will turn us on to.  Perhaps you might want to expand on this in your upcoming tip.

These are just the couple I know anything about.  A trip through the Class Browser says there are more treasures to be had.

Bill

[chrisb]

Hi Dave,

 

Many  thanks for your response.

 

Although we use both D3 and MVbase all our development has been carried out in MVbase so we do not have the problem with Cache being case sensitive. So far we have reached steps 1 & 2 with little difficulty and any new developments will be carried out using the Cache specific tools though we do not wish to amend out current suite of programs too much. I am really after any tips on how to improve performance on this current suite such as using globals rather than the traditional open/read/write Pick commands and also ways of dealing with dynamic arrays etc.

 

Chris

For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en checked for viruses and spam by CanIt.
http://www.canit.3d.net.uk/

[Jason Warner]

Chris,

I would avoid writing straight to globals in your MV code. Doing so
avoids any triggers you might have on your writes. We do use globals
every once in a while to read data quickly (avoiding an open on the
file). When doing this, make sure you look at the MD for the global
name. While Cache does name globals after your files most of the time
sometimes the global associated with your MV file might be named
something like ^D3 or some such thing.

One trick we use is to put information to pass between programs in
process private globals. They come in the form of ^||GlobalName. The
benefit of using them is that they are only available to the process
they are written in and are cleaned up once that process dies. We find
them particularly useful in our green screen programs. Our system uses
a PROC based menu to launch MV programs. In try/catch blocks, we put
information into a specific process private global. When the program
hits the PROC again, we check that global and if it has information, we
pass it to a crash handler that asks the user for information about the
crash. We then email the stack traces, code snippets and user
information from the error. This has allowed us to clean up our code
because we now have an automated process for handling errors (rather
than relying on users to call us).

That is another thing that helped us a ton. Use Cache's exception
handling. It is really good and allows for some powerful error handling
on your end. You can even derive from Cache's exception base class to
create your own exceptions.

We have been moving our code more towards objects instead of MV files
because that is where the true power of Cache in MV lies. Getting MV
projections along-side SQL and .Net access is huge in terms of
productivity for us. Also, some things in Cache ONLY work in objects
and sometimes only with COS code (&html<>, &sql<> and $$$ macros are
some that come off the top of my head).

We were some of the early adopters for MV in Cache, so I can't remember
what issues we ran into. However, if you have some specific questions,
the members on this board are very friendly and will be able to help.
One thing I do remember is that Cache was extremely helpful during our
conversion. Their support was actually one of the most important
factors in us going with Cache.

Jason

On Wednesday, September 12, 2012 1:30:17 PM, chrisb wrote:
> Hi Dave,
>
> Many thanks for your response.
>
> Although we use both D3 and MVbase all our development has been
> carried out in MVbase so we do not have the problem with Cache being
> case sensitive. So far we have reached steps 1 & 2 with little
> difficulty and any new developments will be carried out using the
> Cache specific tools though we do not wish to amend out current suite
> of programs too much. I am really after any tips on how to improve
> performance on this current suite such as using globals rather than
> the traditional open/read/write Pick commands and also ways of dealing
> with dynamic arrays etc.
>
> Chris
>

> *From:*intersy...@googlegroups.com
> [mailto:intersy...@googlegroups.com] *On Behalf Of *Dave
> *Sent:* 11 September 2012 16:04
> *To:* intersy...@googlegroups.com
> *Subject:* Re: [InterSystems-MV] Help in migrating to Cache from Pick

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en checked for viruses and
> spam by CanIt.
> http://www.canit.3d.net.uk/
>

[Tony Gravagno]

I was an early cheerleader for ISC and still am when given the chance,
but unfortunately for lack of any recent work with Caché the limited
skills I had have waned. (Use it or lose it.) I still spend most of my
time in D3 and other platforms these days. So this comment is as much
a suggestion to Chris as a question to everyone else.

I think the first objective is to get your D3 system ported to Caché
as-is. Make sure everything works 100% the same in both platforms so
that you feel confident enough to go live. Then when everything is
stable in the live environment start taking advantage of the value-add
which Caché offers.

I was working on a Universe migration to Caché a few years ago and we
were looking at some extensive XML code which made use of UV-specific
calls. That code HAD to be migrated to Caché %XML functions. Java code
also required serious changes to port away from UOJ, like in D3 you
would need to port to Zen from FlashCONNECT. But pretty much
everything else stayed the same - we weren't re-writing code during
the migration effort unless it was absolutely necessary, even for
optimizations.

With regard to using globals in BASIC, as above, I suggest just doing
the port, and _then_ introduce globals where they make sense. Globals
and other features in BASIC should be used judiciously, not just
because they're there. Once you're in Caché, there is a wealth of
other features that you will want to use - but probably not in the
heat of a migration effort.

HTH
Tony Gravagno
Nebula Research and Development
TG@ remove.pleaseNebula-RnD.com
remove.pleaseNebula-RnD.com/blog (see Caché blogs)
Visit http://PickWiki.com! Contribute!
http://Twitter.com/TonyGravagno
http://groups.google.com/group/mvdbms/about



> From: Jason Warner
> Chris,
>
> I would avoid writing straight to globals in your MV code....

[James Westley Farrell]

What Tony said, bigtime.  The elephant can be consumed entirely, but one bite at a time.  Get everything working as-is (which *should* be fairly easy) and test that much thorough. 

One thing that we found (almost) irritating at first but in the long run a great tool for improving the quality of our code was the fact that unassigned variables cause an exception in Cache.  We had no idea how many had been lurking in our suite that UV had simply glossed over for years.  Even though you *can* turn that feature off, I'd suggest that you don't and leverage it as a tool to improve your code now.

One more suggestion since Jason wisely included it:  Try...Catch...End Try is a marvellous construct.  There are examples of writing your own exception handling class in the archives of this group.  You can really run with that to improve the robustness of your code.  Code that can think for itself and attempt to work around situations rather than wait for operator/programmer intervention is a pretty groovy thing.

Process Private Globals are awesome.  You will eventually forsake COMMON and named COMMON almost entirely.  You can keep and pass nearly anything in them, including instances of classes, arrays, stuff that was odd or clumsy to pass before now seems to be a whole lot easier.  At least to me it does.  YMMV, but for me Cache seems to be everything I ever wanted out of Pick or UV but never seemed to get.  Now I have it.  (Yeah, I'm a total Cache fanboy convert.)

Until next time!
Bill

--
You received this message because you are subscribed to the Google Groups "InterSystems:  MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

[chrisb]

Rich

 

First of all thanks for all your info . We have nearly converted all our software to be compatible with Cache and are now looking at ways of increasing performance. The concept of cache arrays looks interesting as we use dynamic arrays extensively in conjunction with commands such as LOCATE, INSERT etc but not sure how to use these cache arrays as you have suggested.

 

Thx

 

Chris

checked for viruses and spam by CanIt.
http://www.canit.3d.net.uk/

--

[James Westley Farrell]

There is the rare case where you might want to read/write straight from/to a global, but it isn't often encountered in the MV world.  I have one such case in my HL7 parser (written in MVBasic) where I break down huge HL7 input files into an archive global whose nodes look like

^HL7( OriginalFileName, AccessionId, MessageLineNumber )

Later, I have to go back and re-create "mini" files for selected accessions.  These files can only contain messages that pertain to one accession.  These "mini" files are destined to go through our QA system to test changes in the Order Entry and Lab suites, so they have to be narrowed down from the great-big "normal" HL7 files that may have messages for, say, 2000 accessions in them.  This way we can control testing down to a gnat's behind.

The real accession data lives in an MV file along with a multivalued attribute containing the original file where I got the data.   It's awesome-fast to leverage CMQL to do the SELECTS to find the few accessions I want to build "mini" files for, then leverage $Order() to zip through one narrow scope of messages in order to recreate custom output files.  For this narrow purpose, I can use MV objects for the MV files and a handy Cache global for the huge stuff that would be horribly unwieldy in MV alone.

Off the top of my head, I could think of no "normal" MV case where you'd read/write on a global.  Once you get used to using Cache' objects you become very spoiled very fast :)  That ability alone to leverage the power of both classic MV and classic Cache' at once is incredibly powerful.

Bill

On Wed, Sep 12, 2012 at 5:35 PM, Jason Warner <jas...@brashers.com> wrote:

Chris,

I would avoid writing straight to globals in your MV code. Doing so avoids any triggers you might have on your writes. We do use globals every once in a while to read data quickly (avoiding an open on the file). When doing this, make sure you look at the MD for the global name. While Cache does name globals after your files most of the time sometimes the global associated with your MV file might be named something like ^D3 or some such thing.

One trick we use is to put information to pass between programs in process private globals. They come in the form of ^||GlobalName. The benefit of using them is that they are only available to the process they are written in and are cleaned up once that process dies. We find them particularly useful in our green screen programs. Our system uses a PROC based menu to launch MV programs. In try/catch blocks, we put information into a specific process private global. When the program hits the PROC again, we check that global and if it has information, we pass it to a crash handler that asks the user for information about the crash. We then email the stack traces, code snippets and user information from the error. This has allowed us to clean up our code because we now have an automated process for handling errors (rather than relying on users to call us).

That is another thing that helped us a ton. Use Cache's exception handling. It is really good and allows for some powerful error handling on your end. You can even derive from Cache's exception base class to create your own exceptions.

We have been moving our code more towards objects instead of MV files because that is where the true power of Cache in MV lies. Getting MV projections along-side SQL and .Net access is huge in terms of productivity for us. Also, some things in Cache ONLY work in objects and sometimes only with COS code (&html<>, &sql<> and $$$ macros are some that come off the top of my head).

We were some of the early adopters for MV in Cache, so I can't remember what issues we ran into. However, if you have some specific questions, the members on this board are very friendly and will be able to help. One thing I do remember is that Cache was extremely helpful during our conversion. Their support was actually one of the most important factors in us going with Cache.

Jason

On Wednesday, September 12, 2012 1:30:17 PM, chrisb wrote:

Hi Dave,

Many  thanks for your response.

Although we use both D3 and MVbase all our development has been
carried out in MVbase so we do not have the problem with Cache being
case sensitive. So far we have reached steps 1 & 2 with little
difficulty and any new developments will be carried out using the
Cache specific tools though we do not wish to amend out current suite
of programs too much. I am really after any tips on how to improve
performance on this current suite such as using globals rather than
the traditional open/read/write Pick commands and also ways of dealing
with dynamic arrays etc.

Chris

*From:*intersystems-mv@googlegroups.com
[mailto:intersystems-mv@googlegroups.com] *On Behalf Of *Dave

*Sent:* 11 September 2012 16:04

*To:* intersystems-mv@googlegroups.com

<mailto:CacheMV@googlegroups.com>

To unsubscribe from this group, send email to

CacheMV-unsubscribe@googlegroups.com
<mailto:CacheMV-unsubscribe@googlegroups.com>

For more options, visit this group at
http://groups.google.com/group/CacheMV?hl=en

--
You received this message because you are subscribed to the Google
Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com

<mailto:CacheMV@googlegroups.com>

To unsubscribe from this group, send email to

CacheMV-unsubscribe@googlegroups.com
<mailto:CacheMV-unsubscribe@googlegroups.com>

For more options, visit this group at
http://groups.google.com/group/CacheMV?hl=en checked for viruses and
spam by CanIt.
http://www.canit.3d.net.uk/

--
You received this message because you are subscribed to the Google
Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to

CacheMV-unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/CacheMV?hl=en

--
You received this message because you are subscribed to the Google Groups "InterSystems:  MV Community" group.
To post to this group, send email to Cac...@googlegroups.com

To unsubscribe from this group, send email to CacheMV-unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

--

[Tony Gravagno]

MV is limited to 3 dimensions with AM/VM/SVM. When dealing with XML, JSON, exploded bill of materials, complex multi-tier business relationships, nested objects, and other n-tier tree structures, there are plenty of opportunities to go way beyond that. In the MV world people tend to reformat data and convince themselves that 2 or 3 dimensions are plenty good for any task, but the reality is that we are still spending time to reorganize data to accommodate limitations in the model. Even if re-writing code into Caché classes, or using the fine set of %functions that are available, ultimately n-dimensional constructs get serialized to MV limitations just to get the data to fit into the traditional file structures.

 

The Caché global gives us an opportunity to think outside of the traditional MV box once we've transitioned away from it. I think this feature is under-marketed to a market where everyone has already bought into the concept of multi-dimensional data. It's been a while since I've spent serious time in Caché but I don't think the full power of Caché globals has been made available "elegantly" to MV. So it seems there is more there than most people know, and less there than what could be. That seems to me like a recipe for great things to happen in terms of marketing, sales, and engineering.

 

Tony Gravagno

Nebula Research and Development

TG@ remove.pleaseNebula-RnD.com

remove.pleaseNebula-RnD.com/blog

Visit http://PickWiki.com! Contribute!

http://Twitter.com/TonyGravagno

http://groups.google.com/group/mvdbms/about

[James Westley Farrell]

And there's an almighty point to be made, Tony.  I still tend to think MV three-dimensional architecture.  My sentiment is still if you have more than three dimensions, y'oughtta think about dividing your layout a little more finely.  That's just me and thirty years of Pickalike talking -- YMMV.

What I haven't figured yet is how to exploit many-dimensional Cache globals with objects.  I learned how to write nice objects for MV and am completely addicted to them.  I haven't had a huge need yet for writing objects for big-A Cache globals, but I can smell the rain a-comin', the deeper I get into the mysteries of HL7.  For now, $Order()-ing through the small chunks I need is plenty fast as (1) I know exactly where I'm going to start and (2) the scope of where I'm going to operate is going to be limited.  In any case, it beats the tar out of opening up a directory full of sequential files, READSEQ-ing down to the bits I need (of which there may be several chunks in one file, if I have opened a file they *may* be in at all) and moving on to the next.  Having those monsters pre-parsed and trimmed down, living in a global saves disk room and guesswork when I need to produce testbed files for QA.

You can't beat MVBasic for string slicing and dicing.  The stock MUMPS parser may take a half hour on the big alpha server to process down a day's worth of HL7 monstrosities.  On my PC with a personal edition of Cache, I can process a week's worth to have at the ready with MVBasic in about 10 minutes.  It's all about having the right commands to do the heavy lifting in a line or two of code instead of ten or twenty lines of COS to get the same effect.  I can produce a testbed of mini-files for 50 accessions in about five seconds.  There's no analog on the "big shop" side of the house.

Tell me MV don't rock. :)

I believe it's the ability to leverage the best of MV and the handy bits of classic Cache together that makes the difference. 

B

--

You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com

To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com

For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

[Ed Clark]

I half-agree Tony. The Cache multidimensional array structure allows unlimited dimensions, but also gives you unlimited rope to hang yourself on. Using globals directly is a maintenance nightmare. The power of Cache is in classes. You can design objects at a high level with very complex structures and interrelationships, and not have to know a thing about the globals that are used to implement the class. Everything that you need to use objects is available from mvbasic. The only time you have a limitation where "ultimately n-dimensional constructs get serialized to MV limitations just to get the data to fit into the traditional file structures" is if you need to use the CMQL query language, which can only work on 3-dimensional files. If you need CMQL then you can't do anything complex with globals anyway, If you don't need cmql, then you can avoid the limitations of the %MV.Adaptor parent class and have very rich objects. And since objects have a built-in capability to "serialize", once you have a class you could still extract data to use with cmql, or use other tools like DeepSee or Zen Reports.

I think it's somewhat on purpose that "the full power of Caché globals hasn't been made available "elegantly" to MV" because the goal at intersystems, both from a development and marketing standpoint, is to focus on objects.

I'm not saying don't touch globals. I can think of several scenarios where you might want to touch globals directly. Jim Farrell mentioned a few places where he is using globals for an optimization, and most of the ones I can think of are also optimizations.

> --
> You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com

[Tony Gravagno]

HL7 is basic EDI which people have been doing with MV for years. This is another case of "just port it first and optimise later". But I'd think that there would be a Caché HL7 Working Group somewhere, with COS developers focused on best practices for parsing and building huge HL7 docs. Given the benefits of classes, I'd think that there should be a "standard" class out there which parses docs into nested objects. Parsing HL7 should be as easy as calling a lower-level %XML function - except that this would be a higher-level wrapper class. The next step would be to pass the parsed data to MVBASIC for application-specific logic. But in this scenario there's that "impedance mismatch" between objects and MVBASIC. It might be much more convenient to save a visit or lab result object (accessions, etc) as-is rather than breaking that up into pieces that MV can digest. But like Ed said, in order for queries to work we can't do that. Sure, MV logic would have to be re-written to handle this anyway. What I'm suggesting is that for new code written by someone coming to Caché from MV, or to MV from Caché, there really shouldn't be this broad gap between the environments. Again as Ed suggests, classes and objects might be the answer to all of this, but (and remember I'm rusty here so I'll qualify with "I believe") within the class code we still need to break down data elements for MV structures and then re-hydrate them on the way out. That tier of effort for the developer and system resources should not exist.

 

Fun discussion, thanks.

T

[James Westley Farrell]

Heh, oh no doubt it's easier with objects.  Due to the nature of the project I'm working on I haven't had a load of time to design it the way I'd love to.  Y'know how it goes out in the real world vs the comfort and safety I'm used to in the software house:  "Apply a fix hot-n-now and we'll worry about doing it right 'some time down the road'(tm)."

Having dropped back to Cache 5 hasn't helped my case, but I still have my 2012 personal edition sandbox on my PC where I can MV as much as I want to.  I kinda feel like the dirty old man, slinking off to look at MV-porn but having to present my COS "church-face" in the weekly meetings.

Y'know how it can get when y'do whatcha do to survive.

B

--

You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

[Ed Clark]

I think that Intersystem's Ensemble has extensive support for HL7. If you're dealing with HL7 then there's a good chance that you should be using Ensemble (which of course is based on Cache and contains the same mv capabilities)

re: "I haven't had a huge need yet for writing objects for big-A Cache globals". Hopefully you won't be doing this much. Class definitions are very flexible and can map just about any global that you want to migrate from a legacy mumps system, but for new development you should be creating a class and not thinking about the global underneath.

I'm not sure how valid it is to say that "within the class code we still need to break down data elements for MV structures and then re-hydrate them on the way out." MV can handle multidimensional arrays and $LIST constructs. (though it's true that mvbasic doesn't have complete support for more esoteric things like "naked references"). I agree that MV has the best string slice-and-dice functions, but Object Script does pretty well too and has some unique functions. A little Object Script in a class method can go a long way and add loads of power for your mv code to call.

> --
> You received this message because you are subscribed to the Google Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com
> To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

> within the class code we still need to break down data elements for MV structures and then re-hydrate them on the way out.within the class code we still need to break down data elements for MV structures and then re-hydrate them on the way out.

[James Westley Farrell]

Wistful sigh, Ed.  Ensemble isn't even a notion around here.  We just did a BIG UPGRADE to Cache 5.0.2.

SeewuddImean?

What I mostly do these days is without or by hand.

[Rich Taylor]

Ensemble is the tool to use if you are heavily into messaging (HL7, X12, EDIFACT,..) or integration.  Ensemble is Caché plus so you have all the power of Caché.

 

As to MV structures and classes, if I understand you correctly Tony, you do not need to do any handling of the MV structure from the class.   You simply extend from %MV.Adaptor the adaptor will take care of mapping properties to attributes and such.

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

--

[Rich Taylor]

Chris,

 

Let me start with some basics on Globals.  All subscripts in a Globals are automatically in Ascending order.  For example if I create a global with three entries “B”, “C”, and “A” they are actually held in the global in alpha order not the order they were entered.  So in this example the global, let’s say it was called RST, would look like this if dumped:

 

RST("A")="letter a"

RST("B")="letter b"

RST("C")="letter c"

 

You can see that the subscript of the array is not limited to being numeric and it is in order.  This can often eliminate the need to locate and insert that would be necessary in a dynamic array.  For code tables this becomes even more beneficial.  For example take the sample MV code below.

 

CodeTable<1,-1> = “AB” ; * code

CodeTable<2,-1> = “123” ;* factor

CodeTable<1,-1> = “CD”

CodeTable<2,-1> = “234”

CodeTable<1,-1> = “DE”

CodeTable<2,-1> = “54”

 

ProcessCode = “CD”

Locate(ProcessCode,CodeTable,1;idx) then

                ProcessFactor = CodeTable<2,idx>

End else

                ProcessFactor = 0

End

 

The code table could have been loaded from a table or other source. 

 

Using a Cache Array would look like this

 

DIM CodeTable()             ; * note the empty dimensions.  This tells the compiler to use a Cache Array

 

CodeTable(“AB”)  = “123”

CodeTable(“CD”)  = “234”

CodeTable(“DE”)  = “54”

 

ProcessCode = “CD”

ProcessFactor = $Get(CodeTable(ProcessCode))              ; * the $get() function protects against undefined array positions replacing the else clause of the locate.

 

This syntax allows for cleaner code and much faster execution particularly if this lookup is done often on large arrays.

 

 

Another example is building list of invoices that are to be in order then later updating a Dynamic array.  Doing a locate and insert at each invoice would not perform well for large arrays  Instead do this:

 

 

Dim InvoiceList()

 

loop while readnext InvoiceNo do

                * Process the invoice

                * at the end add the invoice to the list

                InvoiceList(InvoiceNo) = “”          ; * the value of the array is meaningless in this example

Repeat

 

* now update a MV record with the list of invoices

* $order iterates through a Caché array returning each new subscript until it runs out.  At that point the variable is set to the empty string.

InvoiceNo  = “”

InvoiceArray = “”

Loop

                InvoiceNo = $order(InvoiceList,InvoiceNo)

Until InvoiceNo = “” do

                InvoiceArray<1,-1>

repeat

 

MyRecord<some attribute> = InvoiceArray

 

 

I have use both these techniques to get performance improvements from 33% to insane levels.   The later technique took a 5 hour process down to 2 minutes.  I expect that this was a isolated “Perfect Storm” of benefits, but it is fun to talk about J

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

[James Westley Farrell]

I dunno if I'd call that an isolated perfect storm at all, Rich.  There are lots of cases where in traditional MV we'd often be forced to use something like:

Aggregate = ""
Loop
* do some stuff
Aggregate< -1 > = StuffToAggregate
Repeat

for string building, sometimes for output, sometimes for small file building.  Often streams can help.  Other times you can leverage globals to do sorting work like you outlined in your previous message since they'll do the sorting work for you automagically.  A lot of times for creating CSV templates where I might use the same data to output columnar data in two or three different formats, I'll gather the data into a global then use lists.  Cache lists are a little like dynamic arrays, but you can sling them into a global node pretty conveniently.  From what I can tell, if you have a pretty wide row, $LISTGET works a lot faster than DCounting a dynamic array then using dyn< index > to pull columns back out to put into whatever format you're outputting the report in this go-around.

Dim ReportRows()
RowCounter = 0
*read data and get the columns you'll need for this row
RowCounter += 1
Row = $ListBuild( TheFirstCol, TheSecondCol, TheThirdCol,...)
ReportRows( RowCounter ) = Row
... until you've got the whole report gathered.  Then for each format you might do this

RowCounter = ""
Loop
   RowCounter = $Order( ReportRows( RowCounter ) )
Until RowCounter = "" Do
   ThisRow = ReportRows( RowCounter )
   NumberOfColumns = $ListLength( ThisRow )
   Line = ""
   For Idx = 1 To NumberOfColumns
      If Line <> "" Then Line := ","
      Line := DQuote( $ListGet( ThisRow, Idx, "" ) )
   Next
* here you could write the line to a flat file, send it down a tcp connection, or wherever this report of many using the same data would go

The combination of using a local Cache array plus lists to reuse columnar data speeds up the process of outputting the same report data tremendously -- and no "scratch files" to go back and clean up.  (I like *that* part a LOT!)  Granted, it's not something you'd use all the time, but when ya need it, it's handy to reach out and grab.  I've always hated writing scratch files partly because of the mess that almost never gets cleaned up, partly because of the cost in disk writes, partly from having to touch something that's almost permanent more than once.  This same technique is blinding fast for CRT paging, for paging reports in small CSP pages (if you don't feel like using a whole Zen thing -- or in my case, you can't use Zen and have to make do)... you can think of a lot of ways Cache will give you tools to work around almost any situation.

B

[Rich Taylor]

I didn’t mean the MV syntax was isolated, just the ability to gain that vast of an improvement.  I think that gain had a lot to do with the process and environment rather than the syntaxes involved.  I can still get some big improvements with these techniques.

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

[Jason Warner]

Another cool thing about Cache arrays is that they are basically
red-black trees. This means you can insert really fast and retrieve
really fast. This also means that they auto-sort like Rich said. This
allows you to process long lists of unstructured data and sort it as
you process it. For example, you could put together a list of sellers
in an auction ordered by date and then by business name.

Dim SaleResults()

while reading line from file
parse data out of line

SaleResults(date, busName, "sold") += soldAmt
SaleResults(date, busName, "numSold") = $get(SaleResults(date,
busName, "numSold"), 0) + 1
SaleResults(date, busName, "something") ...
repeat

Later to output the data by date:

date = ""

loop
date = $order(SaleResults(date))
while while date # ""
crt "Info for sale " : date

busName = ""

loop
busName = $order(SaleResults(date, busName))
while busName # ""
crt " " : busName : " sold " : SaleResults(date, busName, "numSold")
: " unit(s) for " : SaleResults(date, busName, "sold") "md2$"
repeat
repeat

These are off the top of my head so there may be some issues in the
code, but the principles are sound. These two code snippets will output
your data in Date order followed by Business Name order. The same code
in MV required lots of locates and value marks and the structure was
hard for my relatively new MV head to wrap itself around.

> image003
>
> *From:*intersy...@googlegroups.com
> [mailto:intersy...@googlegroups.com] *On Behalf Of *chrisb
> *Sent:* Thursday, September 13, 2012 6:25 AM
> *To:* intersy...@googlegroups.com
> *Subject:* RE: [InterSystems-MV] Help in migrating to Cache from Pick

>
> Rich
>
> First of all thanks for all your info . We have nearly converted all
> our software to be compatible with Cache and are now looking at ways
> of increasing performance. The concept of cache arrays looks
> interesting as we use dynamic arrays extensively in conjunction with
> commands such as LOCATE, INSERT etc but not sure how to use these
> cache arrays as you have suggested.
>
> Thx
>
> Chris
>

> *From:*intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>
> [mailto:intersy...@googlegroups.com] *On Behalf Of *Rich Taylor
> *Sent:* 12 September 2012 15:32
> *To:* intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>
> *Subject:* RE: [InterSystems-MV] Help in migrating to Cache from Pick

>
> Chris,
>
> As Dave indicated some the responses will depend on where you are
> coming from. If you are coming from D3 then case sensitivity becomes

> a issue due to D3 being case *insensitive* . If you are coming from

> Universe or Unidata then this will not be a problem as both those
> environments are case sensitive also.
>
> One other thing that Dave had mentioned has been dealt with. For
> practical purpose we no longer have a limit on routine (program)
> size. While on the topic of program one thing you will notice is that
> our compiler is tighter than many others in the MV world. Some examples:
>
> ·Spaces in line label definitions; HANDLE.LINE : is on legal due to
> the space before the:
>
> ·Reserve word conflicts
>
> ·Missing concatenation operators; LINE = SomeData “ additional text
> added to the end”. The quoted text get interpreted as a formatting
> option due to the : missing between SomeData and the quote
>
> For further feedback I need to understand if you are referring to
> issues that may come up in migration or if you are looking for ways to
> get better performance out of Caché. Let me take a brief shot at both
>

> *Issues*beyond the syntax items show above:

>
> ·Reading large blocks of text into memory. I have run into cases
> where, rather than using READSEQ to read in a large text file, the
> record is put into directory type file and input using READ. The
> issue here is that Caché has a 3.6 million character limit on
> strings. You can handle this in two ways. If you want to stick with
> purely MV Basic commands you can do READSEQ to handle the file
> line-by-line. A better way would be to use our File Stream handling
> API. The program would still be written in MV Basic, you would just
> use our API in place of MV Basic statements.
>
> ·Socket handling uses our standard Sockets API
>
> ·HTTP handling uses our standard API
>
> ·Web Services would need to be implemented using our methodology,
> which is very easy by the way. You could still just call an existing
> MV Basic subroutine from the web service.
>

> *Performance*
>
> ·Caché adds some new in-memory structures that can be used in place of

> Dynamic arrays. I have used both these techniques in MV Basic
> programs to achieve vast improvements in performance.
>

> oCache Arrays – these are arrays that can be indexed by any string

> value. For example it is a common technique to build a dynamic array
> with two VM separated attributes. The first contains a code and the
> next a value related to that code. When you need that value you do a
> locate on the first attribute to find the position of the value in the
> second attribute. With a Caché array you can make the index of the
> array the code. So instead of doing and locate on CODE you would do:
> VALUE = LOOKUPARRAY(CODE).
>

> oThere are also listing structures, called $LISTs, that are binary in

> nature and extremely fast
>
> ·Use our Class API’s. There are many cases where we have an API to
> handle a particular function that performs better. This can be due to
> simplifying code or just due to that function be highly optimized. A
> great example is email. In Cache we handling this completely within
> an API rather than having to call out to OS level functions.
>
> We can help out more as specific issues come up. One of the greatest
> advantages InterSystems and Caché brings to the MV World is options
> and support. You have already found this Community forum and I would
> continue to use it. Post here any time you have something that you
> need to deal with. Of course your Sales Engineer is a great resource
> for these issues. Finally, our support group is world class.
>
> Richard S Taylor
> Sales Engineer
> InterSystems Corporation
> Office: 443-340-8614
> FAX: 440-815-5805
>

> image003
>
> *From:*intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>
> [mailto:intersy...@googlegroups.com] *On Behalf Of *ChrisB
> *Sent:* Tuesday, September 11, 2012 10:38 AM
> *To:* InterSy...@googlegroups.com
> <mailto:InterSy...@googlegroups.com>
> *Subject:* [InterSystems-MV] Help in migrating to Cache from Pick

>
> Hi, We are currently migrating our Pick based system to Cache and one
> of the main reasons is to speed up the processing time. The system is
> basically an accommodation enquiry system and when overloaded by users
> the processing time is considerably downgraded. Any tips on amending
> programs to suit the Cache enviroment would be much appreciated. Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en
>
> checked for viruses and spam by CanIt.
> http://www.canit.3d.net.uk/
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

[Rich Taylor]

On $LISTs, Yes these are another technique for speeding things up.  I have used that also.  You also have some need functions like LISTFROMSTRING and LISTTOSTRING that can convert delimited string into a list.

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

From: intersy...@googlegroups.com [mailto:intersy...@googlegroups.com] On Behalf Of James Westley Farrell

Sent: Friday, September 14, 2012 4:43 PM
To: intersy...@googlegroups.com

[ChrisB]

Hi Rich,

 

First of all thanks for all your help and your info on private globals has been really useful. Also thanks to all the other posters, your help and ideas have been much appreciated.

 

In our migration I have decided for the time being to make use of these private globals though have one problem. In our current software I build up a large dimensioned array where each array contains information for a hotel together with it’s availability and price etc.Once this dimensioned array has been built up  I then have to sort this  array by price (either ascending or descending as required by user) and this involves use of LOCATE and re-building the data into another array according to the sort order. Now you mentioned (with your invoice example) that this could be done on the spot so to speak, but what happens if say one hotel has a price of $300.00 and another hotel has the same price, how can these be assigned to a private global in a specific sort order.

 

many thanks

 

ChrisB

[Jason Warner]

Chris,

I would add a second dimension to the global that is the hotel name.
That way you can hold more than one hotel per amount.

^||Hotels("300.00", "Mariott", ....) = ...
^||Hotels("300.00", "Hilton", ....) = ...

When sorting arrays in Cache, I make the first node my first sort
parameter and then my next node my next sort parameter and so on. Any
other nodes I want to include come after all of my sorting nodes.

Look at my example with dealers sorted by sale date and business name
for a better example. You can also $order() backwards through the
global by adding a -1 as the last parameter to your $order() statement.

Jason

On Wednesday, September 19, 2012 9:28:08 AM, ChrisB wrote:
> Hi Rich,
> First of all thanks for all your help and your info on private globals
> has been really useful. Also thanks to all the other posters, your
> help and ideas have been much appreciated.
> In our migration I have decided for the time being to make use of
> these private globals though have one problem. In our current software
> I build up a large dimensioned array where each array contains
> information for a hotel together with it’s availability and price
> etc.Once this dimensioned array has been built up I then have to sort
> this array by price (either ascending or descending as required by
> user) and this involves use of LOCATE and re-building the data into
> another array according to the sort order. Now you mentioned (with
> your invoice example) that this could be done on the spot so to speak,
> but what happens if say one hotel has a price of $300.00 and another
> hotel has the same price, how can these be assigned to a private
> global in a specific sort order.
> many thanks
> ChrisB

> *From:* Rich Taylor <mailto:Rich....@intersystems.com>
> *Sent:* Friday, September 14, 2012 9:17 PM

> *To:* intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>

> *Subject:* [SPAM] RE: [InterSystems-MV] Help in migrating to Cache

> image003
>
> *From:*intersy...@googlegroups.com
> [mailto:intersy...@googlegroups.com] *On Behalf Of *chrisb
> *Sent:* Thursday, September 13, 2012 6:25 AM
> *To:* intersy...@googlegroups.com

> *Subject:* RE: [InterSystems-MV] Help in migrating to Cache from Pick

>
> Rich
>
> First of all thanks for all your info . We have nearly converted all
> our software to be compatible with Cache and are now looking at ways
> of increasing performance. The concept of cache arrays looks
> interesting as we use dynamic arrays extensively in conjunction with
> commands such as LOCATE, INSERT etc but not sure how to use these
> cache arrays as you have suggested.
>
> Thx
>
> Chris
>

> *From:*intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>
> [mailto:intersy...@googlegroups.com] *On Behalf Of *Rich Taylor
> *Sent:* 12 September 2012 15:32
> *To:* intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>

> *Subject:* RE: [InterSystems-MV] Help in migrating to Cache from Pick

>
> Chris,
>
> As Dave indicated some the responses will depend on where you are
> coming from. If you are coming from D3 then case sensitivity becomes

> a issue due to D3 being case *insensitive* . If you are coming from

> Universe or Unidata then this will not be a problem as both those
> environments are case sensitive also.
>
> One other thing that Dave had mentioned has been dealt with. For
> practical purpose we no longer have a limit on routine (program)
> size. While on the topic of program one thing you will notice is that
> our compiler is tighter than many others in the MV world. Some examples:
>
> ·Spaces in line label definitions; HANDLE.LINE : is on legal due to
> the space before the:
>
> ·Reserve word conflicts
>
> ·Missing concatenation operators; LINE = SomeData “ additional text
> added to the end”. The quoted text get interpreted as a formatting
> option due to the : missing between SomeData and the quote
>
> For further feedback I need to understand if you are referring to
> issues that may come up in migration or if you are looking for ways to
> get better performance out of Caché. Let me take a brief shot at both
>

> *Issues*beyond the syntax items show above:

>
> ·Reading large blocks of text into memory. I have run into cases
> where, rather than using READSEQ to read in a large text file, the
> record is put into directory type file and input using READ. The
> issue here is that Caché has a 3.6 million character limit on
> strings. You can handle this in two ways. If you want to stick with
> purely MV Basic commands you can do READSEQ to handle the file
> line-by-line. A better way would be to use our File Stream handling
> API. The program would still be written in MV Basic, you would just
> use our API in place of MV Basic statements.
>
> ·Socket handling uses our standard Sockets API
>
> ·HTTP handling uses our standard API
>
> ·Web Services would need to be implemented using our methodology,
> which is very easy by the way. You could still just call an existing
> MV Basic subroutine from the web service.
>

> *Performance*
>
> ·Caché adds some new in-memory structures that can be used in place of

> Dynamic arrays. I have used both these techniques in MV Basic
> programs to achieve vast improvements in performance.
>

> oCache Arrays – these are arrays that can be indexed by any string

> value. For example it is a common technique to build a dynamic array
> with two VM separated attributes. The first contains a code and the
> next a value related to that code. When you need that value you do a
> locate on the first attribute to find the position of the value in the
> second attribute. With a Caché array you can make the index of the
> array the code. So instead of doing and locate on CODE you would do:
> VALUE = LOOKUPARRAY(CODE).
>

> oThere are also listing structures, called $LISTs, that are binary in

> nature and extremely fast
>
> ·Use our Class API’s. There are many cases where we have an API to
> handle a particular function that performs better. This can be due to
> simplifying code or just due to that function be highly optimized. A
> great example is email. In Cache we handling this completely within
> an API rather than having to call out to OS level functions.
>
> We can help out more as specific issues come up. One of the greatest
> advantages InterSystems and Caché brings to the MV World is options
> and support. You have already found this Community forum and I would
> continue to use it. Post here any time you have something that you
> need to deal with. Of course your Sales Engineer is a great resource
> for these issues. Finally, our support group is world class.
>
> Richard S Taylor
> Sales Engineer
> InterSystems Corporation
> Office: 443-340-8614
> FAX: 440-815-5805
>

> image003
>
> *From:*intersy...@googlegroups.com
> <mailto:intersy...@googlegroups.com>
> [mailto:intersy...@googlegroups.com] *On Behalf Of *ChrisB
> *Sent:* Tuesday, September 11, 2012 10:38 AM
> *To:* InterSy...@googlegroups.com
> <mailto:InterSy...@googlegroups.com>

> *Subject:* [InterSystems-MV] Help in migrating to Cache from Pick

>
> Hi, We are currently migrating our Pick based system to Cache and one
> of the main reasons is to speed up the processing time. The system is
> basically an accommodation enquiry system and when overloaded by users
> the processing time is considerably downgraded. Any tips on amending
> programs to suit the Cache enviroment would be much appreciated. Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en
>
> checked for viruses and spam by CanIt.
> http://www.canit.3d.net.uk/
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en
>
> --
> You received this message because you are subscribed to the Google
> Groups "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com

> <mailto:Cac...@googlegroups.com>

> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com

> <mailto:CacheMV-u...@googlegroups.com>

[Ed Clark]

you could use multiple subscripts:

rates(300.00,"Hilton")="can be empty"
rates(300.00,"Mariott")="could be a list of previous stays"
or
rates(300.00,4,"Hilton")="can be empty"
rates(300.00,5,"Mariott")=""
where the second subscript is another ordering criterion, maybe a rating or availability.

Instead of using process private variables, though, you might want an in-memory array or %variable if you need "global" scope

> <image001.jpg>

> <image001.jpg>

[Rich Taylor]

A Quick suggestion would be to build your array as:

 

Hotels(price,HotelID) = $list  of hotel information. 

 

This would put the information first in Ascending price order then within price by Hotel.  Using the $order function you can traverse this array in either forwards or backwards directions so you can get the effect of changing the sort from ascending to descending without having to rebuild the array.  Also note that $ order works on on subscript of the array.  So the fully traverse the array by price then show all the hotels will require nested loops using $ order on the different subscript levels.

[James Westley Farrell]

One possible solution is to remember hotels by price.  Say, if you identify hotels by item id you could possibly keep a private global named Price and its primary node could be the room price you're bidding

Dim Price()

Loop While ReadNext Do somestuff to gather pricing info

If $Data( Price( DollarAmount ) = 0 Then ; * there's no room price at this dollar amount yet
    Price( DollarAmount ) = HotelId
End Else
    Locate HotelIds In Price( DollarAmount ) By "AL" Setting AMC Else Ins HotelId Before Price( DollarAmount ) ; * by price, by item id
* or you could just stick id's on the back with < -1> if the string is short OR best yet, use another dimension and order through that
End

Repeat

At the end of that, you can do several things.  You can start at the low price and go to the highest by

DollarAmount = ""
Loop
    DollarAmount = $Order( Price( DollarAmount ) )
While DollarAmount <> "" Do
...
Repeat

To go from highest to lowest price, change the $Order function to
    DollarAmount = $Order( Price( DollarAmount ), -1 )

or you can pick an arbitrary price in between to start searching.  The deal is, when "DollarAmount" comes up nil, you're done.

That's SO much faster than trying to sort by Locate's.  Saves me a huge amount of compute (and user complaint) time :)

Bill

[Gandalf]

Actually, that won't order things correctly...

You need to be careful, when you want numeric ordering, to use canonic numbers... "300.00" is NOT a canonic number, because of the trailing 0's.

You need to use the Unary Plus (+) operator to make sure it's a canonic number, before using it as a subscript (when you are trying to sort numerically).

Also, it would be better to have a simple master global (or local), and keep the index separate.

Then you could have multiple indices created, to be able to sort quickly on different fields...

You could store the data in a global or even a local array, in a simple form:

Set id=$i(^data),^data(id)=hotelinfo ; something maybe like: $lb("Hilton",300,"Smoking","King","WiFi")

And then make an index:

Set ^priceindex(price,id)=""

You can then use $order to find the hotels, in ascending or descending order, or even find a range, very quickly.

This approach also eliminates the problem of having multiple hotels with rooms for the same price.

That's the beauty of Caché local and global arrays.

Scott Jones

[Jason Warner]

You are right. That would be text sorting and bad. It was typed quickly
as going through my email today. Also interesting point on the index.
Hadn't thought of doing that. However, for a quick throwaway sort on
data, building an array or process private global the way I mentioned
would not be bad.

> <javascript:>>

> > *Sent:* Friday, September 14, 2012 9:17 PM

> > *To:* intersy...@googlegroups.com <javascript:>
> > <mailto:intersy...@googlegroups.com <javascript:>>

> > *From:*intersy...@googlegroups.com <javascript:>
> > [mailto:intersy...@googlegroups.com <javascript:>] *On Behalf Of

> *chrisb
> > *Sent:* Thursday, September 13, 2012 6:25 AM

> > *To:* intersy...@googlegroups.com <javascript:>

> > *Subject:* RE: [InterSystems-MV] Help in migrating to Cache from
> Pick
> >
> > Rich
> >
> > First of all thanks for all your info . We have nearly converted
> all
> > our software to be compatible with Cache and are now looking at
> ways
> > of increasing performance. The concept of cache arrays looks
> > interesting as we use dynamic arrays extensively in conjunction
> with
> > commands such as LOCATE, INSERT etc but not sure how to use these
> > cache arrays as you have suggested.
> >
> > Thx
> >
> > Chris
> >

> > *From:*intersy...@googlegroups.com <javascript:>
> > <mailto:intersy...@googlegroups.com <javascript:>>
> > [mailto:intersy...@googlegroups.com <javascript:>] *On Behalf Of

> *Rich Taylor
> > *Sent:* 12 September 2012 15:32

> > *To:* intersy...@googlegroups.com <javascript:>
> > <mailto:intersy...@googlegroups.com <javascript:>>

> > *From:*intersy...@googlegroups.com <javascript:>
> > <mailto:intersy...@googlegroups.com <javascript:>>
> > [mailto:intersy...@googlegroups.com <javascript:>] *On Behalf Of

> *ChrisB
> > *Sent:* Tuesday, September 11, 2012 10:38 AM

> > *To:* InterSy...@googlegroups.com <javascript:>
> > <mailto:InterSy...@googlegroups.com <javascript:>>

> > *Subject:* [InterSystems-MV] Help in migrating to Cache from Pick
> >
> > Hi, We are currently migrating our Pick based system to Cache
> and one
> > of the main reasons is to speed up the processing time. The
> system is
> > basically an accommodation enquiry system and when overloaded by
> users
> > the processing time is considerably downgraded. Any tips on
> amending
> > programs to suit the Cache enviroment would be much appreciated.
> Thanks
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "InterSystems: MV Community" group.
> > To post to this group, send email to Cac...@googlegroups.com

> <javascript:>
> > <mailto:Cac...@googlegroups.com <javascript:>>

> > To unsubscribe from this group, send email to

> > CacheMV-u...@googlegroups.com <javascript:>
> > <mailto:CacheMV-u...@googlegroups.com <javascript:>>

> > For more options, visit this group at
> > http://groups.google.com/group/CacheMV?hl=en
> <http://groups.google.com/group/CacheMV?hl=en>
> >
> > checked for viruses and spam by CanIt.
> > http://www.canit.3d.net.uk/
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "InterSystems: MV Community" group.
> > To post to this group, send email to Cac...@googlegroups.com

> <javascript:>
> > <mailto:Cac...@googlegroups.com <javascript:>>

> > To unsubscribe from this group, send email to

> > CacheMV-u...@googlegroups.com <javascript:>
> > <mailto:CacheMV-u...@googlegroups.com <javascript:>>

> > For more options, visit this group at
> > http://groups.google.com/group/CacheMV?hl=en

> <http://groups.google.com/group/CacheMV?hl=en>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "InterSystems: MV Community" group.
> > To post to this group, send email to Cac...@googlegroups.com

> <javascript:>
> > <mailto:Cac...@googlegroups.com <javascript:>>

> > To unsubscribe from this group, send email to

> > CacheMV-u...@googlegroups.com <javascript:>
> > <mailto:CacheMV-u...@googlegroups.com <javascript:>>

> > For more options, visit this group at
> > http://groups.google.com/group/CacheMV?hl=en

> <http://groups.google.com/group/CacheMV?hl=en>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "InterSystems: MV Community" group.
> > To post to this group, send email to Cac...@googlegroups.com

> <javascript:>

> > To unsubscribe from this group, send email to

> > CacheMV-u...@googlegroups.com <javascript:>

> > For more options, visit this group at
> > http://groups.google.com/group/CacheMV?hl=en

> <http://groups.google.com/group/CacheMV?hl=en>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "InterSystems: MV Community" group.
> > To post to this group, send email to Cac...@googlegroups.com

> <javascript:>

> > To unsubscribe from this group, send email to

> > CacheMV-u...@googlegroups.com <javascript:>

> > For more options, visit this group at
> > http://groups.google.com/group/CacheMV?hl=en

[James Westley Farrell]

The best yet.  Good one, Scott.

[ChrisB]

Hi, Further to our migration from Pick to Cache, can anyone tell me how to code the following in Pick code using cache globals, ie using ^PRICES and ^PRCTAB etc without the OPEN/READ commands etc,

 

    DIM PRCTAB(45)

    MAT PRCTAB=''

    OPEN 'PRICES' TO PRICES ELSE STOP

    PRCKEY='000001'

    MATREAD PRCTAB FROM PRICES,PRCKEY THEN

          GOSUB XXXXXX

          END ELSE

          ERRMSG='XXXXXXXXXX'

    END

 

Any help would be much appreciated, thanks Chris

[Ed Clark]

Well, first off, do you have a good reason to change it? Looks like boilerplate, reliable mv code which will continue to work as you expect in Cache.
If you want to change it, you've got 2 choices: just rewrite it using global references, or rewrite it using class methods. I'll show both, but there's no sense in just rewriting it without introducing objects.

Global references:
PRCKEY="000001" ;* if you are going to be using object script directly at some point, get used to using double quotes instead of single quotes for strings. mvbasic of course accepts single and double quotes and backslashes for quoting, but object script only uses double quotes.
if $data(^PRICES,PRCKEY) THEN GOSUB XXXXXX ELSE ERRMSG="xxxxxxxx" ;* use $data to determine is the "item" exists

That's all :) In gosub xxxxx, where you would have referenced PRCTAB(1), use ^PRICES(PRCKEY,1). Any changes you make to ^PRICES are made on disk, so there's no WRITE of any kind necessary.
If you want to make changes that don't go immediately to disk, use a local array:

DIM PRCTAB()
$KILL PRCTAB ;* If you are using this in a loop, as is likely, you want to clear any previous item before merging.
$MERGE PRCTAB=^PRICES(PRCKEY)

and then in gosub xxxxx refer to PRCTAB(1) instead of ^PRICES(PRCKEY,1). When you want to "write", use $MERGE to copy PRCTAB into ^PRICES(PRCKEY,1)

There are a lot of downsides to coding like this. First off, you can't be sure that the PRICES file uses the ^PRICES global, so you need to either make sure tat the PRICES file is created using ^PRICES, or figure out the path. For example:
OPEN "PRICES" TO FP ELSE ABORT
GLOBALNAME=FILEINFO(FP,2)
which retrieves the global the file uses. But you need to open the file to do this.

This will not work at all on directory-type files, because items in dir files are stores as os files instead of global nodes.
You would need to take extra steps to handle locking. You can use MATREADU in basic and get standard results, but if you are using globals you need to manually take and clear locks.
This would not fire any triggers or update any indexes.
You have just changed one type of code to another with no real advantage and some big disadvantages. You really want to use objects:

Using objects:
First, use PROTOCLASS to create a class for the PRICES file. If you haven't tried that already, it's fairly simple and we can help you with it.
Once you have a class:

item="MVFILE.PRICES"->%OpenId(PRCKEY)
if IsObject(item) then ;* if the item isn't on file, %OpenId returns an empty string "" instead of an object reference variable.
gosub "xxxxxx"
end else
errmsg="xxxxxxx"
end.

In the gosub, use item->property name to access the properties. For example:
prc=item->Price
item->Price=newprice

and to save the results:

st=item->%Save()
if st#1 then cry $SYSTEM.Status->GetErrorText(st)

The class handles reading and writing. You have named properties instead of numbered attributes in an array. The class can have methods added to do validation. Even better, the class can have methods which do processing on the item that you would normally have to keep in a subroutine.

[Ed Clark]

I was so intent on dismissing the idea of using globals directly that without thinking I made it a little more complicated than the simplest case.
Cache supports 2 different types of files stored in globals. The default is the inode-type, where every item is a single global node. The second type is the anode-file where every attribute is a global node. My description for using global references was for anode-type files. So if you had an mv item:

ID
0001 A
0002 B
0003 C

in a file using global ^GLOB, in a default inode-type file it would be stored as:

^GLOB("ID")="A^B^C"

where ^ in the data is an attribute mark--not to be confused with the caret used to identify a global reference. In an anode-type file, it would be stored as:

^GLOB("ID",1)="A"
^GLOB("ID",2)="B"
^GLOB("ID",3)="C"

My description shows how to use the data in an anode-type file. The &SAVEDLISTS& and &PH& files that are created by Cache in a new account are aide-type files.

For anode type files, you would have to use dynamic array references on the globals:

CRT ^PRICES(PRCKEY)<1>
instead of
CRT ^PRICES(PRCKEY,1)
The data in an anode-type is stores as a dunmaic array in the global, so there isn't an equivalent to using MATREAD. The only equivalence is to a simple READ.
If you wanted to reference the attributes as an array, you could use the functions MATBUILD/MTPARSE to convert the data to and from dimensioned arrays.

[Rich Taylor]

Chris,

 

I will both agree and disagree with Ed on some points.  In my experience doing direct global reference is great when you want to get the absolute best performance  on read only access of data.  I would almost NEVER use this to update production records.  This is due to many of the reasons Ed stated; locking, triggers, indexes and such.

 

I know that your goals are to increase processing speed.  What I have found is that reading directly from globals can provide a boost, but what really kicks things up is to look at the in memory structures that you may build in your processes.  I would look for places where you do locate/insert and locate/access or similar syntaxes.  Replacing this with Caché arrays can provide some significant performance improvements.  This is because you can key the Caché array by the index value that you were previously would have had to locate on.  The reference now becomes much more direct.

 

As Ed stated, if it isn’t broke I would not ‘fix’ it.

 

To that end you probably want to look at some of your bigger processes and do a code profile to see where the time is being taken before making wholesale changes.  This tool is %SYS.MONLBL in Caché.  You can search the documentation for this.  Just be aware that the ‘routine’ that being profiled is not necessarily called the same thing.  You need to look at line two of the cataloged VOC entry to see the internal routine name that is used by Caché.  This is the code you will actually profile.  If you need any help running this let us know.

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

From: intersy...@googlegroups.com [mailto:intersy...@googlegroups.com] On Behalf Of ChrisB

Sent: Thursday, October 04, 2012 9:51 AM
To: InterSy...@googlegroups.com

--

[James Westley Farrell]

Rich,

You know what I would LOVE to see?  If MVBasic had the SPLIT() and JOIN() functions from CacheBasic.  We already have MATPARSE and MATBUILD to turn dynamic arrays into Pick dimensioned arrays and back.  It would a help to be able to do the same thing with Cache arrays using a native MVBasic split()/join() function instead of having to make a library classmethod to cross languages to get at a functionality.  I use Cache arrays a LOT (being a sometimes PHP programmer) because of their increased speed and convenience (like being able to use names as keys).

Just thinkin' out loud.
Bill

[Rich Taylor]

Well, not sure about Caché arrays, but you can do this with lists.  You can use the $LISTFROMSTRING() and $LISTTOSTRING() functions.

 

Richard S Taylor
Sales Engineer
InterSystems Corporation
Office: 443-340-8614
FAX: 440-815-5805

 

From: intersy...@googlegroups.com [mailto:intersy...@googlegroups.com] On Behalf Of James Westley Farrell

Sent: Friday, October 05, 2012 9:16 AM
To: intersy...@googlegroups.com

[Chris B]

Rich & Ed,

 

Thanks for your prompt response to my query and for the help and advice. I have used the CPG’s now with the $ORDER function which has been a great help in speeding up the sorting of data (primarily by price either ascending or descending). The only remaining area which slows the processing speed is when the system tries to ascertain a price for a hotel. The price data is held as such;

 

     1> 16355]16359]16362]16366]16372]16375]16389]16391]      etc etc     these are the date fields

     2>   4500]4200]4100]4500]4200]4100]4400]4600]4300]                  etc  - these are the prices for the dates

 

If an enquiry is for eg 10nights, the system has to go through each date band and ascertain the price accordingly, so for example it could be 3 nights at 45.00, 2 nights at 42.00, 5 nights at 41.00 . The dat es can stretch over a whole year and can contain quite a large array. To obtain the price I first locate the arrival date in attr<1> and then do a for /next loop until the end date (start date + 10 nights) has been passed and then total up the prices. It is this subroutine that calculates the price that is the most time consuming. Apart from that working with Globals has been really useful though not always sure how to make the most of Cache’s utilities as they do not always seem to be documented or am I missing something?

 

Many thanks,  ChrisB

checked for viruses and spam by CanIt.
http://www.canit.3d.net.uk/ --

[James Westley Farrell]

I use that a lot when I'm stuck in the COS world and denied the joy of MV dynarrays :)  Lists are like little dynamic arrays (but a bit faster if you leverage them well!)

[Gandalf]

On Friday, October 5, 2012 11:34:04 AM UTC-4, Chris B wrote:

Rich & Ed,

 

Thanks for your prompt response to my query and for the help and advice. I have used the CPG’s now with the $ORDER function which has been a great help in speeding up the sorting of data (primarily by price either ascending or descending). The only remaining area which slows the processing speed is when the system tries to ascertain a price for a hotel. The price data is held as such;

 

     1> 16355]16359]16362]16366]16372]16375]16389]16391]      etc etc     these are the date fields

     2>   4500]4200]4100]4500]4200]4100]4400]4600]4300]                  etc  - these are the prices for the dates

 

If an enquiry is for eg 10nights, the system has to go through each date band and ascertain the price accordingly, so for example it could be 3 nights at 45.00, 2 nights at 42.00, 5 nights at 41.00 . The dat es can stretch over a whole year and can contain quite a large array. To obtain the price I first locate the arrival date in attr<1> and then do a for /next loop until the end date (start date + 10 nights) has been passed and then total up the prices. It is this subroutine that calculates the price that is the most time consuming. Apart from that working with Globals has been really useful though not always sure how to make the most of Cache’s utilities as they do not always seem to be documented or am I missing something?

Did you really mean: 4 nights at 45.00 (16355,16356,16357, 16358), 3 nights at 42.00 (16359,16360,16361), and 6 nights at 41.00?

If I were doing this in COS (or wanted to use globals in MVBasic), I'd set this up as:

^DatePrice(16355)=45

^DatePrice(16359)=42

^DatePrice(16362)=41

...

To find the price for the first night, and find how many nights that is valid for, you would use the following expression:

Set initdate=$o(^DatePrice(beginDate+1),-1,price),nextdate=$o(^(initdate),1,nextprice)

You can then calculate how many nights fall into the first range, and if you need to go to further ranges...

You should check if nextdate is "", because that indicates you've reached the end of your range of valid dates...

This should be fast, because it uses the B-tree to quickly find the first date, instead of a linear scan... (and yes, $LIST can be faster than character delimited lists,

but it is still a linear scan (i.e. O(n) instead of O(log(n))) to find that first date)

Scott Jones

[Michael Cohen]

Hi Chris,

 

I meant to send this last week, but it became a lost draft…

 

Reading a global returns the same result as a READ: a dynamic array delimited by marks chars.

 

If your sub XXXXXX does little with the array and you don’t mind changing code from:

PRCTAB(1)

to:

PRCTAB<1>

 

then the following would work:

PRCKEY='000001'

IF $D(^PRICES(PRCKEY)) THEN PRCTAB=^PRICES(PRCKEY); GOSUB XXXXXX ELSE ERRMSG='XXXXXXXXXX'

 

Notes:

1. $D checks for the existence  of the item

2. I don’t know whether it is a bug or expected, but if I change your line 2 to:

   MAT PRCTAB=”M”

the MATREAD resets unused values to empty, removing my “M”

3. could use MATPARSE to convert

 

 

 

 

From: intersy...@googlegroups.com [mailto:intersy...@googlegroups.com] On Behalf Of ChrisB
Sent: Thursday, October 04, 2012 9:50 AM
To: InterSy...@googlegroups.com
Subject: [InterSystems-MV] Re: Help in migrating to Cache from Pick

 

Hi, Further to our migration from Pick to Cache, can anyone tell me how to code the following in Pick code using cache globals, ie using ^PRICES and ^PRCTAB etc without the OPEN/READ commands etc,

--