large array library

[mattschinkel]

I have created a lib for creating 1 large array.

I tested and added these files to SVN
\include\jal\large_array.jal
\sample\18f452_large_array.jal

Tested on 18F, It took lots of code to do this, but works great.

Let me know what you think. As you know, you can not currently have an
array that takes up more then 256 bytes memory.



-- Notes:
-- Supports byte array with up to 2048 entries
-- Supports word array with up to 1024 entries
-- Supports dword array with up to 512 entries
-- Creates an array named large_array_1
-- Use "alias new_array_name is large_array_1" to rename the array
--
-- Example:
-- const dword ARRAY_SIZE = 600 -- choose number of array
variables
-- const dword ARRAY_VARIABLE_SIZE = 2 -- choose size of variables
(byte*2)
-- include large_array -- include the array library
-- alias test is large_array_1 -- rename/alias the array to
test
--


Matt.

[Sebastien Lelong]

Hi Matt,

2009/11/7 mattschinkel <mattsc...@hotmail.com>

I have created a lib for creating 1 large array.

I tested and added these files to SVN
\include\jal\large_array.jal
\sample\18f452_large_array.jal

Tested on 18F, It took lots of code to do this, but works great.

Nice ! Even if I've never had to use such large array, I'm sure lot of people will be happy to find this lib
 

-- Example:
-- const dword ARRAY_SIZE = 600          -- choose number of array
variables
-- const dword ARRAY_VARIABLE_SIZE = 2    -- choose size of variables
(byte*2)

Maybe these constants should be prefixed with "LARGE_" to emphasize this is very specific to large array, not "common" array.

What do you think guys ?


Cheers,
Seb

[mattschinkel]

> Maybe these constants should be prefixed with "LARGE_" to emphasize this is
> very specific to large array, not "common" array.

You are right,I forgot to rename those

Matt

[William]

Hi Matt,

You've got a nice library there, and I suspect it may be pretty fast
too.

But we'd like more features. :-)

Here are some half-baked ideas.

1. A compile-time constant for total max memory usage for large
arrays is fine.
2. We can get by with just your dword-width arrays, and do a little
index arithmetic and shifting when dealing with byte and word widths
2. We'd like to be able to allocate more than one array, and
preferably a mix of different widths (byte, word, or dword)
3. We'd like to do the allocation at run-time, not at compile time:
var word handle1 = large_byte_array(300)
var word handle2 = large_byte_array(300)
var word handle3 = large_word_array(999)
var word handle4 = large_dword_araay(400)
notes: we have a mix of byte arrays (handle1 and handle2),
word array: handle3, and a dword array(handle4)
yes, the numbers above may be more than your typical 18F ram
max. They are just for example.
4. We need a cool way to actually use the 'dynamic' large arrays. No
ideas yet on that... So until then:
x = get_large_byte_array(handle1, some_index)
put_large_byte_array(handle2, some_index, x)
not nearly so cool as your approach, but we can do more than
one array, and we can do a mix of arrays.
5. It would be a real 'bonus' if we could 'free' the array after use,
but that can wait until later. :-)

William

[mattschinkel]

> You've got a nice library there, and I suspect it may be pretty fast
> too.
>
> But we'd like more features. :-)
>
> Here are some half-baked ideas.

Idea's are always welcome :)

> 1.  A compile-time constant for total max memory usage for large
> arrays is fine.

like I have now?

-- const dword ARRAY_SIZE = 600 -- choose number of array
variables
-- const dword ARRAY_VARIABLE_SIZE = 2 -- choose size of variables
(byte*2)

would take up 600x2 bytes

> 2.  We can get by with just your dword-width arrays, and do a little
> index arithmetic and shifting when dealing with byte and word widths

I wanted to make functions with byte*CONSTANT like:

const byte SOME_CONSTANT = 2
function large_array_1'get(byte*SOME_CONSTANT in address) return
byte*SOME_CONSTANT is

but the compiler would not allow me, therefore I had to make 3
seperate functions for byte, word, dword

> 2.  We'd like to be able to allocate more than one array, and
> preferably a mix of different widths (byte, word, or dword)

I agree, I wanted to wait for your feedback before I do this. more
arrays would just require renaming variables & copying code, I can add
some but how many? maybe 4? and yes, you would then be able to have a
mix of widths.

> 3.  We'd like to do the allocation at run-time, not at compile time:
>         var word handle1 = large_byte_array(300)
>         var word handle2 = large_byte_array(300)
>         var word handle3 = large_word_array(999)
>         var word handle4 = large_dword_araay(400)
>         notes: we have a mix of byte arrays (handle1 and handle2),
> word array: handle3, and a dword array(handle4)
>         yes, the numbers above may be more than your typical 18F ram
> max. They are just for example.

I'm not sure what you are trying to do here. I like having the array
allocated at compile time so it does not take up time/cycles. Using an
array allocated during compile must be faster then using an array
allocated during run time.

I don't know how to allocate an array during run time. I suppose a
different set of arrays could be made for allocating during run time.
Is there a specific reason to allocated during run time?

I suppose you want to create an array while running then delete it and
use the memory space for something else??

> 4. We need a cool way to actually use the 'dynamic' large arrays.  No
> ideas yet on that...  So until then:
>         x = get_large_byte_array(handle1, some_index)
>         put_large_byte_array(handle2, some_index, x)
>         not nearly so cool as your approach, but we can do more than
> one array, and we can do a mix of arrays.

dynamic meaning an array created during run time? I guess you won't be
able to make it work like a real array: x = get_large_byte_array
[some_index] so that's fine.

> 5.  It would be a real 'bonus' if we could 'free' the array after use,
> but that can wait until later. :-)

true



Thanks William, If you would like array's to be created during run
time you may add this feature to the lib. If you do create this
feature, feel free to add your name to the top of the lib as one of
the authors.

During the building of this lib I was only concerned to have an array
(s) that would work like any other array and be as fast as possible
while the circuit is running. I wanted to use only constants and have
no variables for allocating the array. I still wish Kyle would add
large arrays to jalv2, but this works just as well.

Matt.

[William]

Matt,

I think I'll just give up on the dynamic array idea, as too ambitious.

How about this instead: you could come up with some sort of external
template and 'generator' tool, perhaps in Python, that would generate
specific instances of your 'array library' on demand (of an end-
user)? Something like:

make_large_array myarray1 dword 800 > myarray1.jal
make_large_array myarray2 byte 2048 > myarray2.jal

This would give us the ultimate in efficiency, and also the ability
for multiple arrays of various sizes.

But if you prefer to stick with your original approach, may I suggest
that you cut and paste it into 3 different libraries, one for each:
byte, word, and dword? This would allow you to streamline the code
and make it easier to understand and maintain.

William

[mattschinkel]

> I think I'll just give up on the dynamic array idea, as too ambitious.

It sounds great, if you have a need for it. There may be a need for it
later in the future, so we can dream untill then :)

I'm sure I'll eventually think of an excuse for needing a dynamic
array.

> How about this instead: you could come up with some sort of external
> template and 'generator' tool, perhaps in Python, that would generate
> specific instances of your 'array library' on demand (of an end-
> user)? Something like:
>
> make_large_array myarray1 dword 800 > myarray1.jal
> make_large_array myarray2 byte 2048 > myarray2.jal
>
> This would give us the ultimate in efficiency, and also the ability
> for multiple arrays of various sizes.
>
> But if you prefer to stick with your original approach, may I suggest
> that you cut and paste it into 3 different libraries, one for each:
> byte, word, and dword? This would allow you to streamline the code
> and make it easier to understand and maintain.

I put it all into one lib instead of 3 different libs so that the user
can choose size of variables without changing lib includes, it makes
it more like a regular array and easier to use. If you are writing a
jal program and have an array, you may often change your mind about
what variable size you want, or you may want your program
customizable. It is quite important for my fat32 lib to be memory
space customizable. Here's a simple example of why you would want byte
word & dword all in one file:

const word MY_USER_DATA = 100
if MY_USER_DATA < 256 then
const byte BYTE_SIZE = 1
else
const byte BYTE_SIZE = 2
end if

const dword LARGE_ARRAY_VARIABLE_SIZE = BYTE_SIZE
include large_array
ALIAS entry_location is large_array_1

you cannot put an include within an if statement.


I don't know if a python script is needed, why not just have 4 lib
files? or one lib file with code copied. How many large arrays can fit
on a PIC anyways? If you have 4, byte*1 arrays with 512 entries, you
are filling up your PIC's with 2048 bytes. How many large arrays would
you plan on using?

I don't think this lib is hard to maintain unless we do add dynamic
array's, but maybe dynamic array's can have a seperate library since
they must be in the form large_array(address, byte).


Matt

[William]

Hi Matt,

If you don't want to streamline the code, could you add some
documentation on how it works and describe your strategy, and variable
naming convention?

For example, I am guessing, based on looking at your 'get' routine,
that the lowest range of indexes for the large array, will be found in
the _1a section, then _1b, etc. But then there is some further
switching back and forth that I don't understand yet.

Another question - what happens to arrays that seem to be declared but
never used? Are they wasted, or does the JALV2 compiler re-use that
space?

What is the worst-case 'waste' of memory? I think it is only 255*4
bytes but I am not sure.

Thank you,

William


On Nov 7, 12:41 am, mattschinkel <mattschin...@hotmail.com> wrote:

[mattschinkel]

> Another question - what happens to arrays that seem to be declared but
> never used? Are they wasted, or does the JALV2 compiler re-use that
> space?

There are many regular array's that are not being used, the compiler
ignores them & only uses the ones that the constant asks for.

If you make a variable and don't use it, it does not get allocated in
memory, it's a feature of jalv2. The compiler also ignores functions &
procedures that do not get used.

> What is the worst-case 'waste' of memory? I think it is only 255*4
> bytes but I am not sure.

yes, the worst case is 256*4, so that's 1026 bytes of your pic memory,
I don't think anyone would wish to use up that much memory but I may
be wrong.

I can add some documentation to the lib, I will give you a short
example of how I wrote the code.

Matt.

> > Matt.- Hide quoted text -
>
> - Show quoted text -

[mattschinkel]

-- This is how it works (I have not tested this code, but it is a
short version of the same lib)

-- example usage
const dword LARGE_ARRAY_SIZE = 300 -- choose number of array
variables
const dword LARGE_ARRAY_VARIABLE_SIZE = 1 -- choose size of
variables (byte*1)

-- include large_array -- include the array library
-- alias test is large_array_1 -- rename/alias the array to
test

if LARGE_ARRAY_VARIABLE_SIZE == 1 then -- if array has byte variables

-- create vars for if large_array_size is between 256 and 512
var byte large_array_byte_1b[256]
if !(LARGE_ARRAY_SIZE == 256) then
var byte large_array_byte_2b[LARGE_ARRAY_SIZE - (256)]
end if

-- create vars for if large_array_size is between 0 and 256
if !(LARGE_ARRAY_SIZE == 0) then
var byte large_array_byte_1a[LARGE_ARRAY_SIZE]
end if

-- get a value from a large byte array
function large_array_1'get(word in address) return byte is
var byte data

-- if "LARGE_ARRAY_SIZE > 256" we will use "b" array's. "a" arrays
will get ignored by compiler.
if LARGE_ARRAY_SIZE > 256 then -- notice constants here,
if address >= 256 then
data = large_array_byte_2b[address - 256]
else
data = large_array_byte_1b[address - 0]
end if
elsif LARGE_ARRAY_SIZE > 0 then -- and constants here.
data = large_array_byte_1a[address - 0]
end if

return data
end function

-- store a value into a large byte array
procedure large_array_1'put (word in address, byte in data) is
-- if "LARGE_ARRAY_SIZE > 256" we will use "b" array's. "a" arrays
will get ignored by compiler.
if LARGE_ARRAY_SIZE > 256 then -- notice constants here,
if address >= 256 then
large_array_byte_2b[address - 256] = data
else
large_array_byte_1b[address - 0] = data
end if
elsif LARGE_ARRAY_SIZE > 0 then -- and constants here.
large_array_byte_1a[address - 0] = data
end if
end procedure
if LARGE_ARRAY_VARIABLE_SIZE == 2 then -- if array has word variables
-- code here
end if

[William]

On Nov 10, 11:51 am, mattschinkel <mattschin...@hotmail.com> wrote:
> If you make a variable and don't use it, it does not get allocated in
> memory, it's a feature of jalv2. The compiler also ignores functions &
> procedures that do not get used.
>

Hi Matt,

At the moment, the library is essentially 'one large array'.jal.
Which is a fine thing, but perhaps not as generally useful as it might
be, with some refactoring.

If you are going to take advantage of this 'automatic removal of
unused variables' feature of the compiler, then it seems to me you can
get rid of a huge amount of the code -- For example, for the large
'byte' arrays, you can just use these same 8 arrays for all cases of
large 'byte' arrays:

var byte large_array_byte_1h[256]
var byte large_array_byte_2h[256]
var byte large_array_byte_3h[256]
var byte large_array_byte_4h[256]
var byte large_array_byte_5h[256]
var byte large_array_byte_6h[256]
var byte large_array_byte_7h[256]
var byte large_array_byte_8h[256]

And then, for example, in the 'get' and 'put' routines, if your
LARGE_ARRAY_SIZE is smaller than 1792, then large_array_byte_8h will
never be referenced, and so the compiler will not allocate any space
for it. Likewise, if LARGE_ARRAY_SIZE is smaller than 1536, then
large_array_byte_7h won't be referenced.

Now, turning to some other points we've been discussing: You mention
that a person could make multiple copies of the library and thereby
have multiple large arrays-- not really, since almost every line of
the nearly 800 line library would need to be edited to prevent compile
errors - duplicate variable names, etc. That is why I was
brainstorming for ways to use some sort of template and generator
tool.

Regarding breaking up the library into 3 smaller libraries, for byte,
word, and dword: How about this: make it a two-level library-- the
name 'large_array.jal' could simply be a wrapper that includes the 3
lower-level libraries.

William

[mattschinkel]

> At the moment, the library is essentially 'one large array'.jal.
> Which is a fine thing, but perhaps not as generally useful as it might
> be, with some refactoring.
>
> If you are going to take advantage of this 'automatic removal of
> unused variables' feature of the compiler, then it seems to me you can
> get rid of a huge amount of the code --  For example, for the large
> 'byte' arrays, you can just use these same 8 arrays for all cases of
> large 'byte' arrays:
>
>  var byte large_array_byte_1h[256]
>   var byte large_array_byte_2h[256]
>   var byte large_array_byte_3h[256]
>   var byte large_array_byte_4h[256]
>   var byte large_array_byte_5h[256]
>   var byte large_array_byte_6h[256]
>   var byte large_array_byte_7h[256]
>   var byte large_array_byte_8h[256]
>
> And then, for example, in the 'get' and 'put' routines, if your
> LARGE_ARRAY_SIZE is smaller than 1792, then large_array_byte_8h will
> never be referenced, and so the compiler will not allocate any space
> for it.  Likewise, if  LARGE_ARRAY_SIZE is smaller than 1536, then
> large_array_byte_7h won't be referenced.

I think you may be right, at the time I had a reson for creating all
these var's, but I try to get rid of most of them if the compiler
allows.

> Now, turning to some other points we've been discussing: You mention
> that a person could make multiple copies of the library and thereby
> have multiple large arrays--  not really, since almost every line of
> the nearly 800 line library would need to be edited to prevent compile
> errors - duplicate variable names, etc.  That is why I was
> brainstorming for ways to use some sort of template and generator
> tool.

It's simple... just do a string replace of large_array_ with
large_array_2_ to get rid of vars with the same name, then copy the
lib to another .jal file.

In this case, I will change all large_array_ to large_array_1_ for the
main lib. I can easily make an exe to do this, I don't have so much
experience with scripting. Even the user can do it via notepad's "find/
replace all" feature.

> Regarding breaking up the library into 3 smaller libraries, for byte,
> word, and dword:  How about this:  make it a two-level library--  the
> name 'large_array.jal' could simply be a wrapper that includes the 3
> lower-level libraries.

yes, this could be done, but...

then the main lib will have only 3 lines:

include large_array_1_byte
include large_array_1_word
include large_array_1_dword
--maybe more if large_array_2_____

As I said before... byte word and dword must all be named
large_array_1___ so the user can choose it's var length (byte word
dword). Therefore the vars in each smaller lib will be the same
starting with large_array_1_. it will not produce an option to use
more combinations of var lengths.

maybe it's better if I just put a long
"------------------------------------------" between byte word dword?

We can either have more small files, or one large file. I think the
others would prefer that we don't clutter svn & releases with too many
files. If I split the lib up into 3 for byte word & dword, and if I
make 4 large array's, that is a total of 12 lib files + 1 main.

why hav 13 files when you can have just 4, or just one with a lib
generator. If you split the lib, it is harder to do a variable replace
via notepad.

Matt.

[mattschinkel]

Ok, now I remember the reason for all these vars, they cannot be
removed. If I make these vars:

var byte large_array_byte_1b[256]
var byte large_array_byte_2b[256]

and if I want an array size of 300, the compiler will allocate the
full 2 array's of 512 bytes.

so, I copied them over and each has a line like this:

var byte large_array_byte_1b[256]
if !(LARGE_ARRAY_SIZE == 256) then -- compiler gives an error on
zero sized arrays
var byte large_array_byte_2b[LARGE_ARRAY_SIZE - (256)] -- this
one will only allocate what is calculated
end if

and the same had to be made for each step.

There may be a way to remove more vars via more calculations, but it
will actually add more code and more complexity.

I do wish there was a way to reduce code, but this is the only way I
see.

Matt.

[Rob Hamerling]

Hi Matthew,

mattschinkel wrote:
> Ok, now I remember the reason for all these vars, they cannot be
> removed. If I make these vars:
>
> var byte large_array_byte_1b[256]
> var byte large_array_byte_2b[256]

A minor JSG issue for your next revision. When a variable is not
supposed to be accessed directly by a user program - I think this is the
case with your array - then its name should start with an underscore.

Regards, Rob.

--
Rob Hamerling, Vianen, NL (http://www.robh.nl/)

[mattschinkel]

May I add 3 more large array libs (as discussed) now that everything
is working good?

2 large arrays are required by my fat32 lib (depending on user
settings), and the other two will be spare

eg:

large_array_1.jal
large_array_2.jal
large_array_3.jal
large_array_4.jal