Serial Monitor updates

[Jo mo]

And new thread for discussing about serial monitor improvements!

- correction of the "CR" text to "LF" (corresponding to the actual function)

- added the option CR only
- put the "CR+LF" as default option" !

remark: the results with "CR+LF" are the best looking on the  the serial monitor window.
with "CR" only it looks a bit buggy, because the text we are entering appears above the previously written text!
and for "LF" only it is better than "CR" but not wonderful !

For me, a better solution will be to show a "CR+LF" on the serial monitor windows for all the 3 option (CR+LF, CR,LF).
But for "CR"only, we will have to filter the unwanted LF on what is sent to the serial port interface,
and for "LF" only, we will have to filter the unwanted CR on what is sent to the serial port interface

Also, in that serial monitor window, it will be nice, to have two different colors for what is being sent to the serial port (the "local echo") and what is being received!
Maybe yellow for the sent datas (to FPGA) and green for the received datas (coming from the FPGA).
Those are the colors of the Alhambra serial LEDs.
What do you think guys?

Advice: Play a bit with the serial monitor actual options for the "enter" key before updating your icestudio to the new Wip and try those serial monitor changes!

[charli va]

Hello Jom!! These characters should be interpreted in a standard way for generic "terminal emulators".

For example, CR's mission is to return to the beginning of the line by overwriting the content (https://developer.mozilla.org/en-US/docs/Glossary/CRLF). That is why you have observed this strange behavior that may seem like a bug.

LF is line feed, its mission is to go to the next line without going to the beginning (in the next column as this escape character is sent).

So if you send CR+LF you get the result you need in that case, which is "go to start of next line".

But this behavior depends on your application and fpga side in our case.

And the reality is that many applications do what they want XD. That's why other classic serial terminals (for example, stty on linux) allow you to configure many combinations of CR and LF, this is a help output for stty:

man stty
# (output abridged..)
Input settings:
       [-]icrnl    translate carriage return to newline
       [-]inlcr    translate newline to carriage return
Output settings:
       [-]ocrnl    translate carriage return to newline
       [-]onlcr    translate newline to carriage return-newline
       [-]onlret   newline performs a carriage return

This idea could be very useful, allowing to configure different combinations for sending data and for receiving data, right now we only interpret the sending data.

I think that the important thing for the serial terminal is to be flexible, because there is no standard way to interpret the escape characters and although in the applications that you make in the fpga you can manipulate it if for example you use some soc or some design that uses another combination of codes, we have to be able to adapt to it.

For example, this summer I was collaborating with a soc that plays Forth and that was when I added this selector because I just needed a combination that we didn't have (at that time we only played one without being able to select).

Perhaps it would be nice to be able to configure free combinations of CR and LF for input and output and have some default combinations.

I have many ideas for the serial terminal, if you want we can brainstorm together to build the serial terminal of our dreams XD and even if you dare to develop some of the features I can supervise or help you. Right now I'm super focused on several very important icestudio things for the near future, but discussing the future of this module is very important.

The idea of the colors should be very interesting, but as a config.

One important thing about that is that this config could be saved as a "serial config file" or embeded  in the design, in this way when you open your design you have your serial config, or you pass your config to other people.

--
Has recibido este mensaje porque estás suscrito al grupo "FPGAwars: explorando el lado libre" de Grupos de Google.
Para cancelar la suscripción a este grupo y dejar de recibir sus mensajes, envía un correo electrónico a fpga-wars-explorando-el...@googlegroups.com.
Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/f845ae1f-718f-47f4-b048-230b16bd38cdn%40googlegroups.com.

[Jo mo]

Ola Carlos,

Yes, i understand that we may need sometimes all the possible combinations of LF, CR, LF+CR !

Your idea of LF, CR, LF+CR converter in reception ( a bit as we have in transmission after the enter key) is interesting !

i will think about it!

Maybe to decrease the possibilities of having thinks a bit messy , we can make tree different text parts in the serial terminal window:

1- for entering with the keyboard/ or pasting data to be sent (after an "enter" key press)

2- for sent data (with some timestamp info)

3- for received data  (with some timestamp info)

I will try to play a bit with this and see if i get something interesting ! 

And i will ask you if i need some technical help!

have a nice evening

[charli va]

Sorry Jom!! i understand for your mail that you dont have clear the escape characters! sorry me!

I'm preparing a list of features that i think should be interesting and we could implement together to make the best serial terminal emulator! :)

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/f3b490a5-d312-4c64-873b-7d4a028a2b9fn%40googlegroups.com.

[Jo mo]

Hello guys,

This last week i plaid a bit with the code of our serial monitor.

I am trying to get a more "bash like" workflow.

For now i managed to get:
- different colors for sent and received datas
- send data to the serial port only after an Enter-key
- and add some CR+line-feeds on the displayed datas (eg after an Enter key)

I also found a library which could expand positively the functionalities of our serial monitor.like:

-   Arrow navigation: Use left and right arrows to navigate in your input
-   Local History: Just like bash, access the commands you previously typed using the up and down arrows.

-   Tab-Completion: Provides support for registering your own tab-completion callbacks.

-   ...

But, as i am a dick in JS coding :-), even after trying hard following the instruction supplied in the github page, i had no succès importing it in our project!
I am getting an error "Uncaught SyntaxError: Cannot use import statement outside a module"

Interesting links about that library and is integration in other codes:

https://www.linkedin.com/pulse/xtermjs-local-echo-ioannis-charalampidis/
https://github.com/wavesoft/local-echo
exemple of the libraryhttps://codesandbox.io/s/phuuj1?file=/index.html

@ Charli: If you can find some time to have a look at this "import", it will be great ! Your eye will be more efficient than mine for sure ;-)

 Thanks

[Jo mo]

Hello Guys

i had a hard javascript time to get something working with the serial monitor. But here it is!

i added functions (see list at the end of that post), to improve the editing of our commands to be sent to the serial port.
For that i took a lot of code from localechocontroller.js and trying to improve it for some of the functions

The code is not yet cleaned at all (20 % of garbage in my main.js ), it has few minor bugs but it is working quite well ! You can test it for exemple with one of Democrito's math designs (sin, log, ...) or simply with  the TX pin connected to the RX pin (has i did).

How to install :
To try it, you can replace the 3 following files in your icestudio folder by the 3 files in that zip

index.html    located in folder     \resources\plugins\serial-term\
Main.js        located in folder     \resources\plugins\serial-term\js\
style.css      located in  folder    \resources\plugins\serial-term\css\

You can find a little "user manual" in the bottom part of the "serial monitor" starting / setting window.

Test it, and If you like it, i will clean that code and create a pull request

Of course, your questions and bug reports are welcome in that thread.

I hope now to have a bit more time to start playing with the lasts (great i am sure) Democrito's Oled graphics designs ;-)

Have a nice weekend

Big hug !

list of some of the new Features

- Automatic focus on terminal after any GUI button-clicks

-The local echo controller tries to replicate most of the bash-like user experience primitives, such as:

    Arrow navigation:  Use left and right arrows to navigate in your input

        Use up and down arrows  to navigate trough rows
    Word-boundary navigation: Use alt+left and alt+right to jump between words
    Word-boundary deletion: Use alt+backspace to delete a word
    Multi-line continuation: Break command to multiple lines if they contain incomplete quotation marks, boolean operators (&& or ||), pipe operator (|), or new-line escape sequence (\).
    Full-navigation on multi-line command: You are not limited only on the line you are editing, you can navigate and edit all of your lines.
    Local History: Just like bash, access the commands you previously typed using the Alt+Up  arrow and Alt +down arrow.
    remark: - when cursor is at the beginning or at the end of the entered text, using up and down arrows also navigates trough history.
                - History records all the commands sent (with the enter key) and also the last edited command  which was not yet been sent (with enter key) .
- A short serial monitor manual text inserted in the serial monitor starting windows
-...


Todo:
- auto fit on resize of the terminal windows
- use insert key for toggling insertion mode (insert or replace)
- (Maybe an optional) bell sound for received data '\u0007'
see https://www.npmjs.com/package/ansi-escapes?activeTab=explore

known bugs:

- Due to a limit on the terminal height, the cursor can not navigate upward/above the top limit (even when we scroll up).
so the scroll up is only useful for the history of the serial communication in RX an TX.
so we can have troubles on editing the beginning of large (pasted or fully manually entered) block of data.

- few problems on editing and echoing a pasted block including enter /carriage return (almost solved).

[Democrito]

Hello Joaquim,

I have done a quick test and it works great, I like it!

I have done the tests with the module that calculates the logarithm in base 10.

The fact that what is sent and what is received are of different colors helps a lot to see what is what.

The "up arrow" option to retrieve previously entered text was much needed because (at least for me) it's something I frequently need and makes testing easier.

Another thing I've tried is "Flush on enter". Thanks to this new option I will be able to fix two text editors for LCD and OLED, since if I sent letter by letter it created editing problems for me (I didn't have time to scroll in those projects and I had to resort to timers). Since it will now have this option, I will be able to focus it as if we were in an Arduino terminal.

Thank you very much for adding these functions.

A hug!

[charli va]

Hello Joaquim!!! great job!! With these new functions, the serial term begins to be a very good tool, thank you very much!

These days I will be working with him and telling them about the experience.

Thanks again!!!

--

Has recibido este mensaje porque estás suscrito al grupo "FPGAwars: explorando el lado libre" de Grupos de Google.
Para cancelar la suscripción a este grupo y dejar de recibir sus mensajes, envía un correo electrónico a fpga-wars-explorando-el...@googlegroups.com.

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/7602fb97-e064-4a68-8005-d1ef478c54bfn%40googlegroups.com.

[Jo mo]

Ola guys,

Thanks a lot Democrito and Charli for you tests and  future test.

I am glad that you like it ! 

Don't hesitate to share your suggestion to make the tool better. if i can , it will be with pleasure that i will try to address them !

Have a great Sunday.

[Jo mo]

Ola Boys and girls ;-).

I made a bit more progress on our  serial terminal :
- width an height of the terminal area is now automatically resizable ( by dragging the edges of his containing window)
- little cosmetic changes butons size and  positioning(to make it "more compatible" to the windows resizing possibilities),...

3 little bug solved:  
-   down arrow navigation(in multi-line edition) was shifting left
- the first history element available when pressing up arrow was -2 command and not -1 command.
- terminal echoing of cursor movements avoided outside the "flush on enter" mode


Remarks: in the history, we record all the commands followed by an enter key(sent to serial port) and also the unsent commands which were edited before a history navigation up or down arrows.
I limited the amount of recorded commands to 50.

- I started cleaning my code ( ~250 lines of code/comments  burned for now ;-) ).

You can download here the 4 files to copy in your icestudio sub-folders

index.html and manifest.json    located in folder     \resources\plugins\serial-term\

Main.js              located in folder         \resources\plugins\serial-term\js\
style.css           located in  folder        \resources\plugins\serial-term\css\

I will work a bit more on it and hopefully next iteration will be ready for a pull-request !

Have a good night !

[charli va]

Thanks Joaquim!!! testing in few minutes!! Awesome work!!! 

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/337af0b3-a358-4a35-92cd-cc3af317cd60n%40googlegroups.com.

[Democrito]

Good morning Joaquim,

I have tested it and will comment on what I see and possible improvements.

For normal use it works fine with no problems. But if I demand something from the terminal that is usually not normal, that's when I start to see problems.

* Those of us who use laptops have a reduced keyboard, so "shift+insert" is a bit more complicated for us. The ideal would be a "control + V" or "copy paste" from the right mouse button.

* It does not allow copy and paste when there are several "enter". This is important if we want to launch many lines at the same time.

* I think I have noticed that the "copy and paste" does not work well when the "Flush on enter" option is enabled.

The tests have been carried out with the design I made, the "Debugger" with the famous phrase "Per aspera ad astra". In that example, I first run the configuration OLED from the terminal and it works fine. And then I copy and paste everything else, and that's when (I guess) having a lot of "enters", it doesn't send the text as the user would expect.

Let me recommend the example of "Debugger I2C OLED" to see if it works well, since in that example a lot is demanded of the terminal.

I also noticed that disabling "Flush on enter" when I do "copy paste" concatenates the old copied with the new I want to copy.

I am attaching here the "Debugger" design with the phrase, for you to try it out.

If it is necessary to reload the configuration in the OLED a second time (or more times), it is recommended to turn off the terminal and the OLED and start again from the beginning, because if the OLED already has the initial configuration loaded it could cheat on that part and believe that you are accepting the new one.

I am aware that what you want to carry out has a lot of work!

A hug!

[Jo mo]

Ola Democrito;

Thank a lot for your test and comments,

- For the copy and paste,  we already have it from the right click menu !  But as you wrote not yet as ctrl + V.

I will try to see if i can make it (i know how to catch the crtl+V command in js, but not yet how to catch the system clipboard contents. I will have a look, it should be hidden in xterm.js file)

- for your second point i have not observe it yet , but i will try with you debugger ! As an exemple, i could launch 15 log calculus of log10 in one go (see my next /coming post) !

- for the " disabling "Flush on enter" when I do "copy paste" concatenates the old copied with the new I want to copy." i have seen that bug, it seams there is no "clear of the clipboard" but an add to the buffer specially (specially when we switch of "flush on enter") ( i really need to see how xterm.js handle this clipboard buffer)!

I will work on this bugs this weekend using your debugger !

A Big hug Democrito.

[Jo mo]

I noticed a nice feature for using the serial terminal in combination with a worksheettool (like excell) for testing some of our designs:

When we copy an entire column in an excel sheet a return is inserted after every cell.
So when we paste it in the serial monitor and send it(hit enter once), every cell contents will be considered as a separate command (with an enter after every one)

So if we take your exemple democrito (with your log10 block)
we can make eg 15 FPGA calculus in one go!
And then we copy the returned results(green) and past-it in the excel sheet (in a new column)
then we can plot the curve ( i am joining the excel file)!


remarks:

1 -  as we have no serial buffer at the output of Democrito's log designs, we need to be sure that we sent to the fpga a bigger number of digits than the number of digits that the fpga will generate in his output serial stream!
that is why i inserted somme digit after the radix dot in the datas that we send to the fpga.

2 - Excel should have "dot" instead of commas as decimal separator see link

3 - Interestingly, on the graph, we can see a little offset of 0.04 (1% error) on Fpga log (x) values ( for x >1000 ). this may be due to some rounding on your the log10 design.  

it is not very pretty, but we can compensate that by subtracting 0.04 to the log of all the x values above 1000.

[Jo mo]

I just added the ctrl + C and ctrl +V shortcuts in main.js

Thanks to a quick web search, for copy-pasting this copy-paste code :-)

[Democrito]

Hello Joaquim,

5 months ago I obsessively improved the logarithm modules and also tried to optimize as much as possible the real -----> Qn.m -----> real converters. The example you used must be one of the first ones I posted, now it doesn't deviate as much, what's more, the higher the number, the less deviation it has. I always recommend going to my GitHub for updates. One way to tell if it's up to date is by comparing the dates (post/GitHub).

I leave the specific link: https://github.com/Democrito/repositorios/tree/master/Maths/logarithms

In any case, I am attaching it here to go faster. You will be able to verify that in low numbers the error is similar (it is not excessive) and the greater the input values, the better precision it has.

A big hug!

[Jo mo]

Ah, yes, you are right, i used a version that i downloaded long time ago! Sorry!

But as an exemple for testing the serial monitor, it was ok! ;-)

I think, i fixed the concatenated (because not sent data) when pasting data in the direct send mode (flush on enter unchecked). it was a mistake on my side

see new main.js

AAARGH!!!  From this afternoon i am having problems for programing my colorligth boards with my two programmers (ft2232 and usbBlaster) !!  i need to try with another computer!

Sometimes computers sucks :-))

Have a nice evening.

[Democrito]

Joaquim is better now!

When I disable "Flush on enter" and copy/paste it no longer does the concatenation. That was especially the case when "Flush on enter" was disabled. One less mistake!

Get some sleep and try to rest well. When I'm stuck with a project, a good night's sleep helps me collect my thoughts when I come back on the attack.

I send you a lot of strength!

[Jo mo]

Hola Democrito,

After an electronic nightmare weekend? I finally managed to understand my  boards communicating problems! Yes there were few, this is why it was difficult to troubleshoot!

- a small ground wire was mechanically broken on my motherboard. causing a problem with both of my programmers(ft2232; usb blaster)

- my colorligth i5 board does not not communicate any more by jtag! it is somehow fried! ? I ordered another one today!

- my usb blaster may also have a problem now...

But in the end now I could program, with your i2c debugger ice file, my colorlight i9 with the ft2232 programmer.

So now i am trying to reproduce the remaining fault you have seen using the serial monitor.

but for now it seems ok! i copied al the 8 pages in one go, sent them :

and got on my SHxx screen:

once i got a shift of all pages down ?!?!

Then i played a bit just modifying the last page no.7 (replacing hex char by zeros) and it works as expected!

Your debugger is really nice to play with this screens!

So if you still have problems with the serial monitor, i will be great-full to let me know and give maybe other details to help me reproduce !

Thanks and have a nice evening

[Democrito]

Hello Joaquim,

I will only remember one thing:

You have to separate the data with an enter. Surely in this case for some reason you have done it right, but you always have to "give away" an enter, you can put as many as you want, but if there isn't one the data will be concatenated and who knows how the screen will interpret it.

I will propose several tests:

Step 0:

In each test I disconnect the FPGA so that it really starts from scratch (and thus also stop powering the screen, that is, everything is reset), I also disconnect the serial terminal (from the program, clicking with the mouse to "disconnect"). I reconnect the FPGA and then connect the serial terminal.

Step 1: 

Have the "Flush on enter" checkbox enabled

I copy and paste the configuration of the SH: 78 00 AE D5 80 A8 3F D3 00 40 8D 14 20 00 A1 C8 DA 12 81 FF D9 F1 DB 40 A4 A6 AF

Give away one or more "enters" and now copy everything else and paste it: copy and paste everything at once (all as one block).

7800B01002
7840000000000080808080800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080800000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000
7800B11002
78400000000000007F7F19191F0F00003C7E6A6E2C00007E7E04060080C0C08074764A7E7C00002C4E5A7A340000FEFE667E3C00003C7E6A6E2C00007E7E04060074764A7E7C000000000000F4F6CA7E7C00003C7E667F7F00000000000074764A7E7C00002C4E5A7A3400067F7F6600007E7E04060074764A7E7C0000666600000000000000
7800B21002
784000000000000000000000000000000000C0E0E0E0E0E0E0E0F8FFFFFFFFFFFFF8E0E0E0E0E0E0E0C00000000001010000000000000000000000000000000080C0C0C0C0C0C0C0C0E0FCFFFFFFFFFFFCE0C0C0C0C0C0C0C0C0800000000000000000000080C0C0C0C0C0C0C0C0C0F0FCFEFFFFFEFCF0C0C0C0C0C0C0C0C0C0808000000000
7800B31002
784000000000000000000000000000000000010307078FFFFFFFFFFFFBF1F1FBFFFFFFFFFF8F070703010000000000000000000000000000000000000000000003070F0F1F9FFFFFFFFFFFC7E3E1E3C7FFFFFFFFFF9F1F0F0F070300000000000000000003070F0F1F1FDFFFFFFFFFFFF7E7E1E1E3E7FFFFFFFFFFFF1F1F0F0F070300000000
7800B41002
784000000000000080C0606060C080000080800080000F1F1F1F0F87C7E3830707EFFF1F9F0F00000080808000000000000000E0E00080000000008080800000008080008C3F3FBF9FFFEF0F0707070F0F9F9F3FBF3F1C80808000000000808080000000008080E0E00F1F1F1F0F8FC7E78303030387878F0F1F1F1F0F000000000000000000
7800B51002
7840000000000000070F1818180F0700001F1F011F1F0000000000011F1F1900001F1F011F1F00000F1F1A1B0B0000000000001F1F011F1F00001D1D121F1F00001F1F0101000F1F191F1F0000000000001F1F0101000F1F191F0F00001D1D92DFFFE0C08F1F191F1F0000000000011F1F1900000F1F191F0F00000000000000000000000000
7800B61002
78400000000000C0E0F0C00000F0F080C080000080C040C08000000000000080C040408000C0E0F0C0000080C040C0800000C0C080C00080C04040800000F0F0000000000000000000000000000000000000000000003078FCFCFCFCFCFCFEFFFF3F3FFFFFFEFCFCFCFCFCFC7830000000000000000000000000000000000000000000000000
7800B71002
78400000000000000F0F0C00000F0F000F0F0000070F0D0D0500000000000005090B0F0600000F0F0C00000E0E090F0F00000F0F00000005090B0F0600000D0D000000000000000000000000000000000000000000000000000079FFFFFF7F7F3F3F3F3F7F7FFFFFFF7900000000000000000000000000000000000000000000000000000000

Give away one or more "enters"

I only get "snow". There is no image.

Repeat step 0 (Disconnect everything and reconnect, same with serial terminal).

Step 2:

Now disable "Flush on enter".

I copy and paste the configuration of the SH: 78 00 AE D5 80 A8 3F D3 00 40 8D 14 20 00 A1 C8 DA 12 81 FF D9 F1 DB 40 A4 A6 AF

Give away one or more "enters" and now copy everything else and paste it: copy and paste everything at once (all as one block).

7800B01002
7840000000000080808080800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080800000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000
7800B11002
78400000000000007F7F19191F0F00003C7E6A6E2C00007E7E04060080C0C08074764A7E7C00002C4E5A7A340000FEFE667E3C00003C7E6A6E2C00007E7E04060074764A7E7C000000000000F4F6CA7E7C00003C7E667F7F00000000000074764A7E7C00002C4E5A7A3400067F7F6600007E7E04060074764A7E7C0000666600000000000000
7800B21002
784000000000000000000000000000000000C0E0E0E0E0E0E0E0F8FFFFFFFFFFFFF8E0E0E0E0E0E0E0C00000000001010000000000000000000000000000000080C0C0C0C0C0C0C0C0E0FCFFFFFFFFFFFCE0C0C0C0C0C0C0C0C0800000000000000000000080C0C0C0C0C0C0C0C0C0F0FCFEFFFFFEFCF0C0C0C0C0C0C0C0C0C0808000000000
7800B31002
784000000000000000000000000000000000010307078FFFFFFFFFFFFBF1F1FBFFFFFFFFFF8F070703010000000000000000000000000000000000000000000003070F0F1F9FFFFFFFFFFFC7E3E1E3C7FFFFFFFFFF9F1F0F0F070300000000000000000003070F0F1F1FDFFFFFFFFFFFF7E7E1E1E3E7FFFFFFFFFFFF1F1F0F0F070300000000
7800B41002
784000000000000080C0606060C080000080800080000F1F1F1F0F87C7E3830707EFFF1F9F0F00000080808000000000000000E0E00080000000008080800000008080008C3F3FBF9FFFEF0F0707070F0F9F9F3FBF3F1C80808000000000808080000000008080E0E00F1F1F1F0F8FC7E78303030387878F0F1F1F1F0F000000000000000000
7800B51002
7840000000000000070F1818180F0700001F1F011F1F0000000000011F1F1900001F1F011F1F00000F1F1A1B0B0000000000001F1F011F1F00001D1D121F1F00001F1F0101000F1F191F1F0000000000001F1F0101000F1F191F0F00001D1D92DFFFE0C08F1F191F1F0000000000011F1F1900000F1F191F0F00000000000000000000000000
7800B61002
78400000000000C0E0F0C00000F0F080C080000080C040C08000000000000080C040408000C0E0F0C0000080C040C0800000C0C080C00080C04040800000F0F0000000000000000000000000000000000000000000003078FCFCFCFCFCFCFEFFFF3F3FFFFFFEFCFCFCFCFCFC7830000000000000000000000000000000000000000000000000
7800B71002
78400000000000000F0F0C00000F0F000F0F0000070F0D0D0500000000000005090B0F0600000F0F0C00000E0E090F0F00000F0F00000005090B0F0600000D0D000000000000000000000000000000000000000000000000000079FFFFFF7F7F3F3F3F3F7F7FFFFFFF7900000000000000000000000000000000000000000000000000000000

We have repeated the same as before, but with the "Flush on enter" deactivated. I did it on purpose so you have everything at hand.

Now everything is fine, the image looks perfectly.

My conclusion is that when "Flush on enter" is on, it might do something it shouldn't.

When you do the tests I recommend that you do Step 0 between each test, this way you make sure that there are no data type confusions, etc.

If you need me to do more tests, tell me and I'll tell you what I see.

May you have sweet and restful dreams.

[Democrito]

I've tried an echo circuit to see what it's actually sending and receiving, and something is going on there because it's not getting the data I expected.

For example, with ScriptCommunicator if I send the letter 'a' with enter, it comes out like this:

In red is what I send, in green is what is received. You can check that 97 (ascii decimal letter 'a' lowercase) carriage return (13) and newline (10) is sent.

However, 97 is received (this is correct) but then 0 and then 225...

There's something weird here and maybe that's what's driving you crazy. I don't understand why that happens.

Nothing I'm telling you has to do with your program, it's about seeing how data is sent and received.

If you don't know scriptCommunicator, tell me and I'll tell you how to configure it and all that.

I am attach an echo terminal, though I think you have to use a modified circuit.

[Jo mo]

Ola Democrito,

Thanks a lot for you detailed description! i could reproduce everything !

I started with your second post the echoing circuit (modified with a 25 to 12Mhz pll block in my case)!

i am not using script communicator for now, i use the serial monitor + my logic analyser and it gives me that.

- first comment, thanks to your script monitor test, i saw that in my serial monitor code i am wrongly sending the linefeed(10) before the CR (13) when "send on enter" is "CRLF".

So i switched the order in the code and got exactly what you got with script comunicator:

looking a this diagram, we can see that the 13 is completely absent and the 10 is present at the output but not recognized due to the fact that the level before it is 0 and not 1 (see yellow arrows)

Next, i selected "none" at "send on enter"(for removing the CR and LF)  and then put " abcdef " and hit enter in the terminal

we see first char is ok,

than "c" and "e" are almost ok,

and "b" , "d", "f" are loosed!

So :

1- somehow when sending to our serial  port the char very fast (with no waiting time between every char, as we do with "flush on enter" ) our rx,  tx blocks combo are getting mad !

Maybe we need to insert a little buffer some were!

2- sometimes,(like in thaht above case ) the Fpga tx line stay at zero after the end of the transmission. this means that the following/future data received by the serial terminal will also be misinterpreted!

So the one who finds how to improve  our rx and tx blocks wins a banana :-))

I will continue tonight with the bug of "loosen enter char" when we send datas to the oled screen !

Have a nice day!

[Democrito]

I have tested your (& @Carlos) serial terminal with an "Arduino Echo" and it works perfectly fine, both with the "Flush on enter " enabled and not enabled.

I have observed a very curious thing, look at this image:

There is the FPGA port, in my case the FT2232H that the Alhambra II has, and the Arduino serial port.

Note that although they are both two serial ports, they do not have the same name (USB Serial Port vs USB2.0-Serial).
Note that in the "path" it looks like the FT2232H assigns it as a virtual port, however the Arduino assigns it as a physical port.

I don't know if the virtual and physical port is correct, but it seems so to me.

Maybe the problem comes from there and it's not a buffer problem.

Along these lines, if I was correct, I have no idea how to fix this kind of problem because it is way beyond my technical knowledge on the subject.

Let's see if someone else can help us with this problem and knows how to distinguish this type of detail.

Attached the "Arduino Echo" that I have used, it is a very simple program.

[Jo mo]

Ola Democrito,

For the curious think (virtual vs physical port), i do not know, i suppose thaht it is a difference in the different drivers code !?

I can just say that my ft2232 is recognised (with the same parameters test has yours)

For the problem with the echo case with flush on enter, i solved it.

In the TX block the the "State" bit stays locked at zero ("parado" mode) after the first char transmission. The reason for this is, the "bits"counter counts from 0 to 10 instead of 0 to 9 (for the 10 transmitted bits)  see lines 120,12.

As in "flush on enter" or "paste mode mode" 10 bits wagoons are sent for every char , and each wagon is very close from each other (9us for 115200bps), counting 11bits instead of 10 in TX  block

creates an overflow. Because Tx working speed is 10% slower than the received signal !

@ Juan: Could you please crosscheck to be sure that i am not introducing a bug in you block! if everything is ok, maybe we can update your collection with a version 0.3 of the block!

@ Carlos: One days it will be nice to have a warning to the user somewhere when we load a design which contains old blocks versions (compared to the last version available in the ice collections)

Little remark: In the "Rx" block the rcv signal and the "data" bus gets out of the block at the same fpga clock cycle! 

And with my logic analyser, i saw that rcv is even generated slightly before the the "data" bus.

To be sure that the datas are really ready when the rcv signal is received by the tx block in this case, i delayed the rcv signal by one clock cycle! But anyway, here, it seams that the datas are read at the right time (clock cycle). So maybe not necessary to update the RX block!

And here is the final transmission:

I am joining the echo.ice file from Democrito but modified !

Have a nice day guys

[Jo mo]

Doing a quick search in the forum, i found the Fernando echo file (with his homemade ;-) different rx, tx blocks) compatible with multiple board frequencies.

So i made the same test and it better, but it also also has a little problem with fast successive transmissions (like flush on enter)

I am joining a copy of Fernando's  file (with some port removed), you have to modify the frequency ( corresponding to you board ) !

[Democrito]

Hello Joaquim,

I have tested your "Echo-Mod" program with ScriptCommunicator and the serial terminal and now everything is fine, no weird stuff.

As for the "I2C Debugger" the same thing still happens when "Flush on enter" is enabled. I went to the Arduino terminal because that is its natural way of working and the exact same thing happens. I have also observed that it continues to concatenate other copy/paste files.

I have been doing other tests apart, for example, I have used the logic analyzer (echoing), and... there is something that does not work well. We have to make sure that the TX and RX modules work well from the point of view of the logic analyzer.

I am very confused, because for some things it works fine and for others it stops working.

[Democrito]

I went here: https://nandland.com/uart-serial-port-module/

I took the verilog code to create an RX module.
I put it to the test and it worked without problems.
Then I put it in the "I2C debugger", and did the test of sending letter by letter, and it works fine.
After I enabled "flush on enter", it doesn't work.
Then I tried from the Arduino terminal, it doesn't work.

So the problem doesn't seem to be the RX module, at least not with the one you've modified, nor with the new one from nandland.

Maybe you're right and you need a buffer, or maybe for some reason I2C doesn't give it time to send everything if the "flush on enter" option is enabled.

I am attaching the nandland module, just out of curiosity. It's a draft because I was in a hurry. It has no baud rate, it only works at 115200.  The 104 comes from: 12_000_000 MHz/115200 baud, that's easy to modify.

[Jo mo]

Ola Democrito,

Yes, i still have to check my serial terminal code ! To make it work also with your debugger (in all  the cases)

In fact i realized just now that you debugger includes a RX block :-)  for info, on the ice file i joined the main mod was in the TX module.

The little mod i made in the RX module, i commented it at line (200). so if you want to try it,  you have to un-comment line 200 and comment 199.

But not sure that it will help, because the rcv signal is already slightly delayed (to the following rising edge of the system clock) outside the RX block. With a D flip flop block.

for the buffer, before the i2c module, i had a look at your schematic and as the i2c part seams to work at a frequency of 400Khz.

it is 4 times faster than the serial RX input ( 115.2Khz).  it should be ok! 

I am, now, am switching again to the debugger.ice test in combination with the serial monitor to  try sorting it out!

Thanks a lot for your help debugging all this,  together we will manage to get it right ;-)

Have a good sleep!

[beni...@gmail.com]

Hello Joaquin,

I think that in your example of Serial-echo for the Colorlight you have an error because the calculations are based in a clock of 12 Mhz and the colorlight has a clock of 25 Mhz frequency. 
I have created a modification of the project so you can change the frequency and the bauds in the TX and RX modules, son now they are compatible with all boards.

It is convenient for the next projects we consider that there are more people using boards in IceStudio with 25 Mhz clock or other frequencies different to 12 Mhz  

Thanks and regards

Fernando

[Jo mo]

Holà Fernando,

Yes, the echo_mod that i posted is an ice40 version (as Democrito and most users here are  on ice 40, i removed the pll 25 to 12 mhz in the file i posted here). But on my own test i used the 25 to 12Mhz pll module!

I tested your "universal serial echo" and it works perfectly!

I like the fact that the internal logic is indépendant of the system frequency value!  And with no additional ressources cost !

But, i think, we can not convince all the future "block" maker to use the same strategy ! (of not "using" at all the frequency value of their system clock);

When we use the PLL primitives for exemple it is always relying on our own system frequency value !

I think we all agree, that this 25-12 mhz question when we share our ice files is really cumbersome!

At some point we should improve the "convert board function" to somewhat "automatically add" a pll converter block in the icefiles that we load with this "convert board function" !

Because (taking apart the blocks you shared in your last post)   plenty of *.ice block  can be affected by the real system clock frequency of the user !

For know, maybe it is the right time to start making the ice-interface collection .

We can start by inserting there the

-  Two  RX , TX modules from juan (with my mod)

-  Fernando's universal blocks

But we have to rename all this blocks to avoid errors on the user side!

eg:

-  Serial_RX_SysClk12mhz

-  Serial_TX_SysClk12mhz

-  Serial_TX_Universal

-  Serial_RX_Universal

If we want to avoid big names, we can also use subfolder in the collection

  eg:   serial / RX / SysClk12mhz/  serial RX.ice

Personally, i prefer a combination of the two big names + subfolder (eg:   serial / RX /   Serial_RX_SysClk12mhz  )

What do you think guys

Have a nice day

[Jo mo]

Hola Democrito,

For the problem of Carriage Returns remove from the pasted text, it was my mistake!

It should be better with the joined main.js

At least, here the debugger controls quite well my SH Oled display

Remains maybe the "concatenation" Bug !  I will try to reproduce it

Have a nice evening guys

[Democrito]

Hello Joaquim,

I just did more tests. It still doesn't work (with "Flush on enter" enabled and if we send text with more than one enter), however I think everything is fine (it's just my opinion). What I think is happening is that for some reason it is not allowed to send text at the same time if there is more than one enter. And I think this because if I have "flush on enter" enabled and I copy and paste the code line by line and hit enter then it works fine.

I have tested it with ScriptCommunicator and from that program it works to send everything at once.

Another hypothesis that I have is that when we have activated " Flush on enter  " it interprets CR-LF as a code and not as a carriage return, so it adds something that does not exist and only interprets the CR once, which is at the end. If this were true, what it is doing is concatenating everything, because it only sees 1 enter (the one at the end) when we manually hit enter.

Another possibility would be to do a little experiment. Every time you detect an enter(13,10), create a pause in the serial terminal program (eg 500ms) before continuing to send the text and see what happens. If it worked, you would just have to trim the time until you find the limit where the error occurs. In this case it is to see if it is a PC buffer problem for example (to find some excuse).

[Democrito]

I remember once, at the beginning of this forum, they gave me a program made in python, which consisted of sending characters from the PC to the FPGA. The guy who sent me that code was using Linux and I was using Windows and on Windows it didn't work. With the carriage returns there was a problem, but I realized that it used the function "print hex(13,10)" [I don't remember this], but it's to give you an idea. It occurred to me to change it to how it's done in Basic and I changed it to "print chr(13,10)" and it worked. I'm telling you this in case you see something in the code that works that way.

Take everything I tell you as "a brainstorm"

[Jo mo]

Ola Democrito,

These enter disappearing it was my fault! to deal with terminal printing, i needed to replace them with other characters (due to term.write function from xterm.js ).

But for writing to the hardware serial port i should have put the "return" back ( which i did not do correctly before)

This one should be better in terms of multiple enters bug (i hope ;-)),

I also improved a bit the vertical cursor navigation!

Once again Democrito thanks a lot for all you ideas and all the time you spent for testing this.

Have a good day

[Obijuan]

Hi!

I am not following this thread in deep, but I would like to comment something about the serial port

In this thread [1] (in jun 2020), Alberto found some strange behaviour with the serial port, so I made some tests and fixed it

The serial tx works ok, but in general, if you want to design a circuit that read something from serial port, perform some calculations and sent back a response, it is MANDATORY to use a buffer for syncing the receiver and the transmitter

I am attaching here the echo circuit for testing. If you have a look at the transmitter, it includes a 1-byte buffer. So, if you use this transmitter, it should fixed all the issues related to the serial port

When no buffer is used, the syncronization between the receiver and the transmitter is very unstable: in some systems will work, but not in another. It is a kind of random.

@Joaquim:  The change you made to the transmitter, makes it work for you, but it would not work in general, as the busy signal is turned off before the last bit is sent. So, depending on how the circuit uses this block, it may work or not (if the circuit reads the busy signal, it will not work. If the circuit ignores it, it may or not work (in your case it works). But the general solucion to this problem is just adding at least a 1-byte buffer to the transmitter

The last version of the transmitter with buffer can be found in the Collection-stdio, under the devide folder: Device/Serial-tx-Bufferx1. This version works ok for any board. Just change the sclk parameter to your system frequency (12_000_000 for 12MHz boards. And 25_000_000 for 25Mhz board) (12Mhz is the default value)

Currently, the only well-organized, tested and documented collections are the stable ones:

    IceK: Constants
    IceWires: Wires and Buses
    IceIO: FPGA IO-pins
    IceGates: Logic gates
    IceMux: Muxes and demuxes
    IceCoders: Binary Encoders and Decoders
    IceFF: Flip-Flops

All the rest collections are temporary or may contain bugs. I am creating these stable collections from the bottom up. Currently I am working on the iceRegs collection (with registers) and then I will continue with the iceSRegs (with Shift registers). This collection will allow us to build the final stable collection for serial communications. In the mean time you can use the blocks from the stdio collection (or your own custom blocks)

I know that users may find it confusing: lots of collections, versions, bugs, etc... That is the reason why I am refactoring it from the bottom up. I only need some time (and patience form users). I am also waiting for Chali to integrate these stable collections into icestudio so that users can install it easily

Best regards, Obijuan

[1] https://groups.google.com/g/fpga-wars-explorando-el-lado-libre/c/MaXrq9llzcI/m/N4KX-vs_AwAJ

[Democrito]

@Joaquim: I confirm that with the new "main.js" everything has been solved. And also, it seems that the copy/paste concatenation has been resolved as well. I think you can already do a PR on GitHub.

@Obijuan: I've saved the TX module with one byte buffer, I'll use it in future projects!

Have a good night.

[Jo mo]

Hola Juan,

Thanks for your reply an for the link to the thread of  2020. It is exactly the problem we got here!
i tried your echo file and it works great !
Looking at all those RX-TX design i understood!

In the cases discussed here, the computer let no time at all between the datas. Means after a stop bit of 1 bits time, we got it send the start bit(0) for the next data. it means a char rate of 115200 / 10 = 11.5 [kChar/ sec]
This means that ideally all the logic inside the FPGA should run at this exact 10 "bits time frequency".

- RX  block needs only 9.5 bits time  (before trigger the RCV_tic),
- TX  block needs 10 bits time (before releasing the busy signal)
So RX it is a bit too fast for the simple TX block which loaded with the new char value without having finished to treat the previous one.

The 8 bit buffer (added) memorizes the char and release it 0.5 bits later (when the busy signal disappears), an this buffer is never overflowed because the char rate arriving from the computer is 10bits per cycle.
If the char rate was generate in the fpga and higher than the 115200 / 10 [char per sec]. the buffer will be overflowed  and some char will be loosen. but at least with the buffer some of the char will survive! without it nothing will be recognized by the computer!

- my TX  block mod needs only 9 bits time (before releasing the busy signal) and being ready to deal with the next char coming, it is working.
But as you wrote Juan, it is "dangerous" because we were close to loose the stop bits (which is set at the very last fpga clock cyclewith that mod).

To summarize, as you wrote, fast transfer ==> use of tx+buffer block !

@juan: Maybe, in the future interface collection, we can add a short "info" text inside the TX block mentioning this problem! And suggesting the use of the TX+buffer block when the char rate is at 100% of the transmission capacity !

Thanks Juan also for pointing us to the stdio collection.  i have it installed but never opened it :-) it hard to have a full picture of all this existing blocks made by people during all this years !

Just take your time for preparing and documenting all this, it is a precious, useful work!

One day, when we will search in the collection manager, it will be great to have a link to the part of your documentation explaining the use of that particular block we are read to pick-up!

@ Democrito: Thanks for your test, i will clean a bit more the code and do the PR at the beginning of this week!

Do not hesitate, to comment, if you see some more bugs or annoying workflow problems. "al hierro candente batir de repente" ;-)!

Have a nice week guys.

[Jo mo]

Hello guys,

So i finally made the pull request for the serial monitor v0.2!

In it, few little changes (that i can revert if you prefer):

- added a  prompt like sign "->" before  the command which were sent to the serial port (green text)

- When resizing the terminal window, the previous datas(sent or received) is sometimes displayed weirdly! So i decided (at least for now) to reset the dispay when we resize the window.
Nevertheless, if we were editing a command( without sending yet) it remains under edition after a resize! and the command history is also not reseted.

- Slightly better history navigation, i think

- Decreased the code size  from around 1800 to 1300 lines of code.
Still some room for code cleaning but it will not reduce a lot more the code size because anyway i will need to add more commented lines to comment the code and make it more "readable".

Again do not hesitate to comment here (as Democrito did previously, thanks to him) or ( on github) if you find somme bugs or possible improvement !

Have a nice evening

[Democrito]

Now we have a super serial terminal! Anything that can be added and improved will always be welcome.
I use it a lot so anything I see I'll report back!

[Juan Gonzalez Gomez]

I've tested it under linux and it works ok!!!!  Great contribution Joaquim! Thanks a lot 🙂️

I've just found a little bug. If all the options are disabled:

* local echo: unchecked

* Flush on enter: unchecked

* Hex view: Unchecked

* On enter sends: none

IF the circuit on the FPGA is just the echo (it re-transmits to the PC everything it receives), the expected behaviour should be that it is shown everything you type, in the same line. But it appears vertically. For example, if you type "hello", the expected behaviour is that every character is sent individually to the FPGA, it is echoed back, and the terminal should print the character on the same line. So you should see "hello" (in green color, and horizontally) in the terminal. but you get it vertically:

I've been testing the other modes and they work ok

Thanks a lot again for this contribution

Obijuan

PD.- I am attaching the echo circuit used for this test

El jue, 16 feb 2023 a las 22:02, Democrito (<spo...@gmail.com>) escribió:

Now we have a super serial terminal! Anything that can be added and improved will always be welcome.
I use it a lot so anything I see I'll report back!

--
Has recibido este mensaje porque estás suscrito al grupo "FPGAwars: explorando el lado libre" de Grupos de Google.
Para cancelar la suscripción a este grupo y dejar de recibir sus mensajes, envía un correo electrónico a fpga-wars-explorando-el...@googlegroups.com.
Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/f7f770be-4bb9-40d2-9b77-9c9c808ea6d1n%40googlegroups.com.

[Jo mo]

Ola Juan,

Thank a lot for your Linux test.

So it seams  we have no compatibility problems so far! Great news !

For the bug, yes,

In fact, i inserted a carriage return just before the first key press on the keyboard ( just following a  bunch of received data (callback to renderec function).

in this case the bunch is juts a single letter !

So you are right, it seams better to have them on the same line in thaht case.

Little question: when we have the exact same checkbox config but with Local echo active what do you prefer ?

1-    hh    (as it is now)

       ee

       ll

       ll

       oo

2-    hheelllloo    ( as it will be if i just simply remove the carriage return, to solve your discovered bug )

3-    hh ee ll ll oo  ( i can replace my  actual CR by a space (just when local echo is on) )

For the hex-view checked i will also rework with that same "in-line" logic (vs vertical distribution / CR):

for "hello" we will have:

- "Local echo" un-checked :   68 65 6c 6c 6f

- " Local echo" checked :        h68 e65 l6c l6c o6f

Of course, all the mods above will only be effective when "flush on enter" is off !

Improving all that should be quite easy, let me know your preferences, and then i will implement it quickly !

Have a nice weekend Guys !

[Jo mo]

Shit, i wrote all my hello letter in green and yellow color with this google group text editor. But they appear in black/white!

Sorry guys you will have to guess the colors ;-p

[Jo mo]

Has Year2023 is starting to be far from the black and white  screens era , i decided to try hacking thaht google group "Monochrom" fetischism. :-)

[Juan Gonzalez Gomez]

Option 2 is ok. That is the standard behaviour of terminals, like script communicator or similars

For the hex-view checked is ok what you has proposed

Thanks again. We are having a very powerful terminal! 

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/185f1938-5c8c-4fbd-b691-afb3b88cb638n%40googlegroups.com.

[Jo mo]

Ok,

i just pull requested somme little change for making it more intuitive, following Juan comments !

it tried to test it extensively with Juan echo file and also with the Democrito debbuger file (for driving an oled display). And found no problems so far !

Again, if you see something, ....

Have a nice weekend

[Obijuan]

Tested! I can confirm that the bug is gone

Thanks a lot Joaquim! 🙂️