New features feedback / advice request

[cicciocb]

Hi All,

following the latest feedback I received from some of you, I understood that I must implement a feature enabling the management of binary data.

This could be done using strings but actually the strings are managed using the classic 'C' approach that uses the character ASCII 0 (NUL) as end of string.

The advantage is that they are very easy to manage and have a low memory footprint.

The drawback is that they inhibit the use of the character (NUL) as it will be considered as end of string.

For example, the string 

A B C D (NUL) E F G H I (NUL)

Will be cut at the first (NUL) and then considered as

A B C D

If the (NUL) appears at the first position of the string, all the string will be also considered null

For example

(NUL) A B C D (NUL)

will be considered as empty string.

I would like to receive your advice and suggestions on what I try to explain below before starting to code that into Annex and Annex32. 

The idea is the creation of a new kind of "object", the IObuffer.

The principle of this object is the capability to hold and manage binary data.

Ideally, it will be interfaced with any function requiring binary data handling.

Basically the IObuffer is a piece of RAM memory that can be written and read byte per byte or as a block.

It has a defined length and can be freely dimensioned and cleared.

There will be several IObuffers available, let's say up-to 5 (numbered from 0 to 4).

This is specified in the following examples with 'buffer_num' 

The IObuffer will have the following functions / commands :

IObuff.DIM(buffer_num, length)   ' dimension the buffer; it can be used again to resize the buffer

IObuff.CLEAR(buffer_num)   ' clear the buffer and returns the ram back

IObuff.WRITE(buffer_num, position, value)   ' write 'value' at the position 'position'

                                            ' value is a byte and position can be from 0

  ' to the length of the buffer-1

a = IObuff.READ(buffer_num, position) ' read a byte at the position 'position'

Another possibility could be to use it a kind of "special" array and use like this :

DIM IObuff(buffer_num, length) ' dimension

a = IObuff(buffer_num, position) ' read

IObuff(buffer_num, position) = a  'write

DIM IObuff(buffer_num, 0) ' clear

Then the IOBUFFERS could be interfaced like that :

With UDP :

UDP.READ_IOBUFF buffer_num  ' receive the message into the IObuffer 

UDP.WRITE_IOBUFF buffer_num, IP, port  ' write the buffer from the IObuffer

UDP.REPLY_IOBUFF buffer_num  ' reply from the IObuffer

With FILES:

' save the buffer

FILE.SAVE_IOBUFF buffer_num, "/buff.bin"

' append the buffer

FILE.APPEND_IOBUFF buffer_num, "/buff2.bin"

' read the buffer starting from the file position 0 for 500 bytes

FILE.READ_IOBUFF buffer_num, "/buff2.bin", 0, 500

' read the buffer starting from the file position 20 for 1500 bytes

FILE.READ_IOBUFF buffer_num, "/buff2.bin", 20, 1500

A little example

FILE.READ_IOBUFF 0, "/myfile.bin", 500, 1000  ' read 1000 bytes starting from the position 500

IObuff.DIM(0, 2)   ' redimension the buffer at 2 bytes

IObuff.WRITE(0, 0, 8)     ' write 8 at the position 0

IObuff.WRITE(0, 1, 0 )    ' write 0 at the position 1

FILE.SAVE_IOBUFF 0, "/myfile2.bin" ' save the buffer

With Serial port:

print IObuff(buffer_num)  'print all the content of the buffer

print IObuff(buffer_num, pos)  'print only the byte at the position 'pos' of the buffer

Serial.READ_IOBUFF(buffer_num) ' read all the characters present in the input buffer of the serial port in the IObuffer 

The same principle shall be applied with a similar syntax for I2C, SPI, etc .

NOTE:

The IObuffer, when used to RECEIVE data (for example in FILE.READ_IOBUFF) will be automatically dimensioned.

When used again to read data, it will be automatically redimensioned to fit with the received data.

To know how many data have been received, the function IObuff.LEN(buffer_num) shall be used.

The IObuffer dimensioning will be required only for writing purposes and, if used for receiving data, 

will be automatically redimensioned to fit with the received data.

The IObuffers will be automatically cleared (removed from the memory) when the program is run.

The size of the IOBuffer will be limited only by the free RAM available (that can be up-to 4MBytes for the ESP32 with PSRAM)

Please do not hesitate to give your feedback / advices / suggestions / points of view.

This should be a useful improvement for the benefit of all the users.

Thanks

cicciocb

[Electroguard]

Looking at the situation from a gadgets point of view, how would it expect a receiving gizmo to read and handle the numeric data sent from the gadget ... which would obviously include nulls (0).

If the gizmo can read and handle incoming numeric data without problem, could Annex just be given an additional 'switch' which could allow it to handle incoming data in the same manner as a receiving gizmo ?

[cicciocb]

Robin,

what do you mean for 'switch' ?

Actually it is not possible to receive or send data if that includes nulls (0).

I did several "patches" in several functions in the past such as SERIAL.BYTE, SPI.HEX$, I2C.readarray or I2C.writearray and maybe others that I forgot.

All these to enable the use of the character nul.

With this new approach all these functions should share the same logic and syntax.

As an example for UDP, this code will enable the use of this remote

udp.begin 10000

onUdp received

wait

received:

udp.read_iobuff 0

print "received "

for z = 0 to 9

  print iobuff.read(0, z),

next z

print 

return

Using the function a$ = udp.read$ will simply return an empty string as the content contains several times the character nul

[Bugs]

I find it difficult to think of how to do things until I actually /have/ to do it so excuse any silly remarks.

On Wednesday, 29 April 2020 15:42:29 UTC+1, cicciocb wrote:

Another possibility could be to use it a kind of "special" array and use like this :

DIM IObuff(buffer_num, length) ' dimension

a = IObuff(buffer_num, position) ' read

IObuff(buffer_num, position) = a  'write

 

Is "a" a special type of variable consisting of a single byte or is it the normal floating point (8 bytes?)

DIM a as byte?

 

A little example

FILE.READ_IOBUFF 0, "/myfile.bin", 500, 1000  ' read 1000 bytes starting from the position 500

For symmetry should you have a similar save option?

FILE.SAVE_IOBUFF 0,100,10, "/myfile2.bin" ' save 10 bytes from the buffer starting at position 100 

IObuff.DIM(0, 2)   ' redimension the buffer at 2 bytes

IObuff.WRITE(0, 0, 8)     ' write 8 at the position 0

IObuff.WRITE(0, 1, 0 )    ' write 0 at the position 1

 

could the "8" or "0" be a byte type variable? 

[cicciocb]

Bugs,

I admit that I was probably not very clear in my description so I'll try to clarify more following your questions.

The IOBUFFER is (will be) a special variable containing bytes.

There are (will be) 5 buffers available numbered from 0 to 4

The IOBUFFER.DIM(buffer_number, length) command will reserve memory space for it

For example IOBUFFER.DIM(0, 1000) define the IOBUFFER 0 with 1000 bytes

Then it will be possible to read and write from it using bytes (so all the values from 0 to 255)

To read from the function will be :

value = IOBUFFER.READ(buffer_num, position)

to write to it the function will be

IOBUFFER.WRITE(buffer_num, position, value)

Where :

buffer_num can be from 0 to 4

position can be from 0 to the buffer length -1

value can be from 0 to 255

When not required anymore, it can be cleared using the command

IOBUFFER.CLEAR(buffer_num)

This simple code create a buffer and fit it with the byte value 123

IOBUFFER.DIM(0, 100) ' buffer with 100 bytes

for z = 0 to 99

  IOBUFFER.WRITE(0, z, 123)

next z

Then the buffer will be connected on all the modules that requires access to binary data

For example, with the FILE, it will be possible to read binary data from the file using the command

' read the buffer #2 starting from the file position 0 for 500 bytes

FILE.READ_IOBUFF 2, "/buff1.bin", 0, 500

and to write this same buffer to another file

FILE.SAVE_IOBUFF 2, "/buff2.bin" 

or

FILE.WRITE_IOBUFF 2, "/buff2.bin" 

and, to print this same buffer

SERIAL.WRITE_IOBUFFER 2

or send it to the UDP

UDP.WRITE_IOBUFFER 2

or send it to the I2C

I2C.WRITE_BUFFER 2, i2c_address, register

and so on ...

The IObuffer will act like a channel to exchange binary data between functions.

Another advantage is that there will be a common syntax between all the functions

The IOBUFFER could also implement function to convert from/to other format simplifying the code

Hoping have been enough clear ... 

[Bugs]

Ok - I am still thinking...

I can do this as I can use a variable:-

IOBUFFER.DIM(0, 100) ' buffer with 100 bytes

data 0,1,255,99,10

restore

for z = 0 to 4

   read v

   IOBUFFER.WRITE(0, z, v)

next z

But can I do this?

dim v(4)=0,1,255,99,10

for z = 0 to 4  

   IOBUFFER.WRITE(0, z, v(z))

next z

or do I have to introduce another line in the loop:-

for z = 0 to 4  

   v1=v(z)

   IOBUFFER.WRITE(0, z, v1)

next z

I ask because the docs say "DIM A(100)              define a floating point array "

I am never sure what type a variable is (byte, integer, fp) so will Annex keep me right provided value is <256?

[cicciocb]

Yes Annex will do the conversion automatically provided that  the value is below 255.

However it will be possible to initialise the IOBUFFER like the DIM

IOBUFFER.DIM(0, 10) = 0,1,2,3,4,5,6,7,8,9

[Bugs]

Thanks - that has helped a lot.

I do not have a practical application but could there not be a requirement to output specific parts of the buffer?

If an output device needs say 4 bytes to set up a condition there could be a buffer containing all the sensible conditions (read from a config file at program start)

The program could select which condition to send - e.g to send the 4 bytes for condition 95 it might be

SERIAL.WRITE IOBUFFER 0, (95*4), 4

 

[cicciocb]

Yes, this can be implemented

[Bugs]

Ok - I will leave you in peace for a while.

Surprised that there has not been any comments from people who have requested a binary feature - hope it is not wasted effort.

[cicciocb]

Bugs,

I'm also surprised as I was expecting more feedback; probably people have other priorities.

My question was about the "complexity" as my first objective is to do something easy to understand.

So far I didn't find any "easier" alternative syntax, that's the reason that pushed me into asking for suggestions and proposal.

I cannot consider it as a waste of time as I'm sure that will be useful for many people.

I'll publish a beta version soon including these changes and we will see what happens.

[mcguinn]

cicciocb,

What you propose looks ok to me.
Haven't been able to give it deep thoughts yet, but I sense possibilities to
create rudimentary drivers for spi or i2c devices.

Regards
mcguinn

[cicciocb]

Thanks mcguinn for your feedback,

I would just understand if the result will enough clear for everybody.

Effectively this will open the gate to low level devices managements like drivers.

[Dwight]

Hi Cicciocb, others,

I wish I could add some value to this discussion.

I can't say I have ever needed binary communications that I was not able to solve some other way.

As Bug's stated "Surprised that there has not been any comments from people who have requested a binary feature - hope it is not wasted effort."

at any rate, thank you for all your efforts to improve and enhance Annex.

dwight

[jimbo]

This is a valuable feature and with your remote example, I can see the obvious benefit this will add. Thank you for your continuing efforts!

[cicciocb]

Thanks for your support but, basically, I was asking for suggestions.

I have a major issue on the syntax to be given to the commands as it could become very complicate when using with other commands.

Instead of put on hold waiting for more feedback, I would implement as I found it very useful even for my own needs.

I would just to have your opinion about the syntax :

I plan to include parenthesis around the IOBUFFERS commands all the time, in order to have the possibility to specify the number, the start position and the length.

For example, the syntax will be :

UDP.IOBUFF_WRITE(buf_num [, start[, length]]), IP$, port

UDP_IOBUFF_REPLY(buf_num [, start[, length]]), [, port]

FILE.IOBUFF_WRITE(buf_num [, start[, length]]), file$

SERIAL.IOBUFF_WRITE(buf_num [, start[, length]])

UDP.IOBUFF_READ(buf_num)

FILE.IOBUFF_READ(buf_num), file$, file_block_start, file_block_length

SERIAL.IOBUFF_READ(buf_num)

What do you think ?

cicciocb

[Bugs]

That looks good to me for file and serial - can't comment on UDP as have not used it much apart from sending remote IR strings.

I have a slight worry about the result of the  IOBUFF_READs (especially SERIAL) when the memory runs out due to longer than expected data stream.

With the 8266 I get crash and reboot if a response to WGET$ is too long.

[cicciocb]

The Serial will work exactly as Serial.input$ using the serial input buffer.

This is limited at 256 characters (or bytes).

I'm not sure but I think that the UDP is limited at 8192 but, in general, only little packets are exchanged using the UDP (at least for the scope of what Annex is supposed to do).

For the WGET is very difficult to control the RAM as it is impossible to know if the size of the answer will fit in the RAM.

I could put a kind of limiter but the unique alternative will be to return an empty result that could be even more complicate to understand.

Maybe a message saying "message too big"?

[Bugs]

I have not had a problem with the serial buffer as I usually use serial.chr$ and build the input string until the line terminator.

(Similar to the Visual Basic a$=LINE INPUT #1 function.)

This is a bit off the topic but anyway -

I currently only use WGET$ for sending email or Thingspeak or IFTT messages.

The problem only occurs with the 1k plus return message from the email providers. 

It would certainly be helpful if there was buffer size limit and an ERROR message or number.

[cicciocb]

Do you send emails using WGET ?

 

[Bugs]

Ah no - SORRY!

My program sends an IFTT trigger and an email at every re-boot and I have lost the place.

Forget what I said and I will go and watch snooker on TV.

[Andy Gadget]

I may be wide of the mark here but are you saying that the binary buffer WILL be able to handle the NUL character?

I recently looked at loading .bmp bitmap files from the web via an API but quickly realised this was impossible using the standard commands.  Would a binary buffer make this a possibility?  

[cicciocb]

It could be implemented also the support for binary through the web but an image is too big to be loaded from the web

[Andy Gadget]

Ah, I was thinking a 320 x 240 pixel bitmap would be around 76K which I thought may be possible, but I've realised I was forgetting the colour depth - The real size would be 226K which is a totally different box of frogs.

  

[Gerard Sontag]

Hello cicciocb & All,

Sorry for the delay but those times are not the best ones.

My feedback:

good to be able to have x'00'

about the name iobuff.clear it's more a destroy.

about sending the whole buffer to serial or file: there are cases where you don't know the final size in advance and you don't want to send the whole buffer. So either it will be

fine to be able to resize the buffer or/and have the ability to use the buffer as a stack ( with push and pop) and the index in this case will be manage by Annex (I vote for this option)

As someone else ask it will also be great to be able to access this buffer by halfword and word freeing to plit those half/words into bytes before putting them into the buffer.

My 2 cents

Gérard

[cicciocb]

Hi Gerard,

thanks for your feedback.

I'll take into account your remarks.

cicciocb

[Stuart Gillies]

This is a bit complex for me to appreciate the effect of it on what I am doing at the moment. But I did have a related request:

that instead of displaying a small square in place of non-printing characters, perhaps you could display e.g. [FF] i.e. the hex value in brackets.  Useful for diagnosis.

I haven't yet experienced the problem that your bigger idea would address.

My other suggestion is to make better use of the right hand side of the screen in the editor.  Lots of unused space on a landscape screen. Maybe put the log window at the side of the code window instead of below it? (If you have control over it)

[Carla Huisman]

Hello Cicciocb,

Because I just read your answer to Vincent Himpe  about  the 'array of strings' question, I followed your pointer to this place,

and studied your clear description of your IOBUFFER plan.

Sorry that I have no advice to add, but just the remark that I like the idea very much, and expect to make use of it often, when 

you present it.   Thanks for this (coming) addition !

@t.              

[ACarvalho]

About the limitation on WGET answer sizes, a parameter with the maximum size (or portion of the answer to get, like start and length) for the answer would solve the memory allocation issue on large answers.