Quest for an Universal way of organising icestudio block

[Jo mo]

Hello everybody,

First, sorry if the subject was already discussed.

So I am now using icestudio since about two weeks.

And one think which is still a bit difficult is to find a wanted component in the collection.

The collection manager is nice, it improves the navigation across the collections.

But, as the numbers of modules and collections increases, I’m asking if it will be interesting and feasible to create an “universal” structure. That will be followed by all the blocks/collection creators.

For example as early starting draft:

Legend: 

• the structure (in capital letters)
• the blocks (lowercase)
• the mothers collection of the block in parenthesis

 

-BASIC         

-INPUT_CIRCUITS 

            -BUTTONS

                        -DEBOUNCED

                                 -

            -ROTARY ENCODERS

                        -DEBOUNCED

                                  -

-PULL-UPS

              -

-BUS

            -SPLIT

                        - 8 BIT

                                    - separador1-7 (collection Jedi 1.1)

                                   - separador2-6 (collection Jedi 1.1)

                                   - …

                        - 9 BIT

            -MERGE

 

-LOGIC

            -GATES

            -DECODERS

            -COUNTERS

            -…

 

- INTERNAL MEMORIES

            -registers

                   -Flip Flops

                       

- OUTPUT_CIRCUITS

            - EXTERNAL MEMORIES

                        - EM638325BK- 8Mbit sdram

                        - GD25Q16CSIG - 2MB spi ram

 

            -I2C   

-Mcp23017 - drived trough i2C

            -…

 

-I2S

           

-SPI

            -Mcp23017 - drived trough spi

            -…

-GRAPHICS

            -LEDs

                        -seven segments

                        -led bargraph driver

-LCD

-VARIOUS

-vga

-hdmi

 

-FPGA CHIP SPECIFIC BLOCKS/NOT WORKING ON ALL FPGAS OR DEVBOARDS

            - BRAM-ECP5

- SERDES-ECP5

-…

 

-HOMEMADE_BLOCKS

 

-UNCLASSED BLOCKS

 

And every time someone creates a block it has to file in a new field called ”struct position” in the “edit/project information” window

For example: for the “separador1-7” bus splitter, we can enter in the ”struct position” field  “Bus/split/8bits”.

And if someone left a “struct position” field empty, it will appear in the UNCLASSED block section.

Another advantage of this structure system is that it can be easily translatable to other languages.

We can create a big file with structure translations of various languages and when the user select is own language, the structure will appear updated with it. Of course this means that when a block creator fills in the “struct position” fields he does it in is own language. And as icestudio knows user language, it will use the translation file to saves de field with the default language (eg Spanish ;-)). So the shared between users file will always be the same (Spanish) but this will be transparent for the user.

Of course, the block names, which appear at the tree branches extremities (eg. in my above tree: seven segments, mpc23017) will remain untranslated. (too difficult to be implemented )

So if you want, we can use this post for brainstorming the subject. Who knows, we may be able to agree on a “universal structure” adapted for icestudio workflow.

Sorry for that long post, I desperately hope not being too messy ;-)

Regards and have a good week-end

Joaquim

[Tim Rudy]

Hi Joaquim!

That is an interesting subject. (Do you mind if we mix English and Spanish - why not make it more messy?)

I wanted to create a collection manager (and still would like to, but last year and this year I just don't have open source time). So I can discuss what I see (and what charli va has mentioned). This is for high level design, and maybe the community (who knows more than I do) can give feedback about what's best for most users. What are "typical" use cases, what are the most advanced use cases that would be reasonable?

Desired Situation

A web page that is a central location, it would inform community of all Icestudio collections that exist. Collections can possibly be "vetted" or rated. It depends on how large the list gets, probably.

(Actually, I see the starting point exists: https://github.com/FPGAwars/icestudio-collections)

This repository or index would permit the user to:

1. download collection and install it

2. download more collections - we don't want users to be limited to only one collection that they are familiar with, the one they first installed when they got Icestudio - user can be aware of more, and expand their toolkit

3. select blocks to use from any collection, if user prefers to go by individual blocks

Note the question of **organizing** a collection as you discussed (and language translation, demanding standardization, or personalizing) is not addressed - it's for another section, below...

Possible Features

• search and select blocks and package blocks into a new collection (a custom collection)
  
• upload collections or block by users (for users without repositories)
  
• upload or link GitHub repositories with blocks or collections
  
• provide an API to integrate into Icestudio (Install these blocks or collections using Collection Manager right in Icestudio)
• standardization is a good thing - This exists already in the Icestudio definition of a collection, e.g. in user guide "A collection is composed by blocks and examples sorted by categories (directories). The package.json file is required and contains information about the collection."  There is a README.md. There is also a required way to declare the hierarchy (as it will appear in the Icestudio menu or Collection Manager), and translations, and there is even a tool to create a collection, iirc.
• the freedom for each collection creator to do their own thing is also good (but I guess they can still be required to meet standards - such as consistency in organizing, naming convention, capital or small letters - these things are not enforced currently)
• another issue you may have noticed is collections can have such long lists that the menu is difficult to navigate - size limits could be enforced

A Mock-up: For User organizing their own Collection(s)

This is the idea I had. But this has two things that are possibly a significant problem:

1. Only a crazy or advanced user like a company with many employees that use Icestudio would want to use this? Who would take the time to organize convenient collections that make use of categories, naming, separation? Like a library/catalogue system...

2. It becomes quite important to have an unambiguous reference back to the original source of any component/block (as you also show in your example, Joaquim). That would be technical, for possible updating of the block, or it's for attribution so the knowledge is not lost. Would it be a difficult data and metadata issue, a difficult issue in the world of the Internet and changing repos (with a solution like Semver used in software)?

The basic thing to notice is the user can select only what they want - they no longer need those huge collections. Then, they can either download, which by default uses the collection name that the component came from, or they can conveniently categorize what came from the left side, on the right side: They add and name the groups in their hierarchy, say. (The user below has hit "Add Group..." twice, and then selected or dragged components). The left side comes from one collection: It's a collection from the Icestudio index page/repo page mentioned above. Or, it's from any URL on the Internet, if the source has the correct metadata.

Just ideas. Hope we can generate a discussion and work out a collection ecosystem! - Tim

[Jo mo]

Hello Tim,

Thanks a lot for your reply.

For the languages, almost no problem, i understand well French, English, Portuguese and Spanish. Unfortunately i cannot write in Spanish and Portuguese. And it’s a shame because I was born in Portugal.

I agree with everything you wrote.

But about the efforts on the web page, i personally prefer putting them directly in icestudio. This to avoid painfull windows navigations between icestudio and the web browser.

For me the ideal situation is:

• All the collections are loaded when icestudio starts but with their blocks spreaded/visible in a standard structure scheme”, and not classed “by collection”. 

I found painful to have to search trough all the collections and collection subfolders to find eg ”a Nand gate with 5 inputs”. If it exist in some collection, it should be visible grouped with all other Nand gates (from all collection) at the same place.

eg: 

        logic/nand/nand2(xxxx collection)

        logic/nand/nand5(xxxx collection)
        logic/nand/nand5(yyyy collection)

Of course, the difficult part is define the “standard structure” which suit to everyone’s mind :-)

•  Also, when icestudio starts, an automatic collections update process could be done. With a dialog box informing the users about the collections being updated.

 

For me the point of icestudio, is making think easier by:

• Showing the circuit diagram visually.

• Allowing the user to write less Verilog code.

But if finding the right icestudio block  takes to much time, the user will end-up by re-writing himsef code in verilog  (that may already be just there, hidden in a collection).

In a way it is good, we will become more fluent in verilog. ;-)

Concerning the organisation of the exiting blocks (in the current collections),

As I understand, Icesudio is actually just using the folder/subfolder structure of the collection.zip file to show the blocks organised in the dropdown menus.

 So, I am volunteer to help reorganising the existing blocks in the right folder and this for every collection.

But First we need to define all together the “best standard folder structure”.  And then there is some work on icestudio code to show the dropdown menus with thaht fixed standard structure.

Tim I have seen just now in your github page the 74 family chips collection “ice-chips-verilog.zip3», I tried to add it in icestudio but it fails (invalid collection message).

I looked at the zip file contens and supposed that this collection is only compatible with a previous version of icestudio; Am i rigth?

I had the same message with the “FPGA-peripherals-master.zip collection ”

 Regards

 Joaquim

[charli va]

Hello !! First of all, thank you for your comments. They are very important to us.
Now I am working on the collection manager, I think this is a very important feature. I am trying to put all the current work and your ideas together in a public document to work together.
I think Tim's ideas are very helpful and the final solution is this way, I think the best way to organize blocks in collections is through "virtual collections" regardless of disk storage.

About the default disk organizations, Obijuan is refactoring the collections into small and better organized, check the FPGAwars repository if you want to see it, I am sure your ideas are very welcome.

Thanks!

--
Has recibido este mensaje porque estás suscrito al grupo "FPGAwars: explorando el lado libre" de Grupos de Google.
Para cancelar la suscripción a este grupo y dejar de recibir sus mensajes, envía un correo electrónico a fpga-wars-explorando-el...@googlegroups.com.
Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/10db4b11-5ecc-4e01-abfd-53f105c787d5n%40googlegroups.com.

[Jo mo]

Hello Charliva,

It's great to see that we are all thinking in about the same directions :-) This "virtual collection" concept looks promising to me.

Thanks and have a nice day.

[Tim Rudy]

Hi Joaquim, since I don't have your e-mail I want to ask: You mentioned that the collection zip file you downloaded above (um, .zip3?!) couldn't be recognized by Icestudio. Do you have a Mac? Linux? When you try to download and add collection, does it work or fail at this time? I should have asked this 2 years ago - Thanks, Tim

[Jo mo]

Hello Tim,

i am on Windows 10,

it is interesting that you ask that question, because 2 days ago, playing with the "coming" FpgaRiders Collections i again got problems of recognition by the Collection manager of the collection. and in fact what i saw, is :

If we insert a "readme.md" it almost freezes any following refresh trial of the collection manager.

with F12debug window, i got the error:

In fact, in the end, we have to change readme.md file to capital letters ( READMe.md) for making it work! 

Remark 1: in my icestudio, i am using the Preference/external collection way of loading collection. it links to a single folder were i play with the unziped collections.

Remark 2: if we have a lot of collection loaded, we also need to wait maybe one minute before the refresh of the CM completes!

also when refresh doesnt come i hit the "F12" key (debug window) to see if i have the above picture error.

But this "collection manager" recognition of a big amount of collection is buggy. ( eg at the time of this writing i have that refresh problem of the CM and again i see the same F12(debug )error described above".

 So i will by elimination, remove collection from the folder  to try locating my error , ...

[charli va]

Hi Joaquim! Ill try to prepare an explanación of how collection manager work. This afternoon check for no md files read.

All of the bugs that you find please tell us in this thread!!

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/e47e5909-75ea-4050-839a-6656063c82f2n%40googlegroups.com.

[Jo mo]

Hola Charli and Tim,

i just understood my mistake:

With icestudio, opened, saved and build files which are located in sub-folders of my external collection path!

so Icestudio (as usual) creates a new build folder with plenty of files that the collection manager will not like.

@Charli: of course any info about those files that we have to avoid in our collection folder and sub-folder are welcome!

Have a nice day guys!

[charli va]

About the ice-build dir, I'm thinking of changing it.

My first thought was that this build directory would be in the same folder as the project file.

But with examples or collections they generate a lot of confusion, because many times you are going to test something and you do it on the example itself.

I think maybe we should detect that it is in a "system folder" and send it in this case to the temp folder or advise the user to save the file somewhere else.

What is your opinion?

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/bc69a484-532b-4add-9bc9-c11721c3c230n%40googlegroups.com.

[Tim Rudy]

Yes, it would work to avoid any "system folder" which is in a list of things known to Icestudio. Can ask the user to browse and select the working location. Do you think of it as a "working location"? Some users will never see it or know about it, but some want to examine the files in there.

[Jo mo]

Hello guys,

1)

- when we share file in the forum, we always transmit the .ice file only. (the buid-dir will then be generated in the "leecher's" :-) computer. When i tryes to play with the design on his icestudio )

- also the size of the build dir is ~ 8 times bigger than the one of ice files (so not interesting for sharing)

So maybe it is not really necessary to have the build dir close to the ice file.

Maybe we could imagine storing all the build folders in c:/documents/icestudio/builds.(maybe even in a compressed format) and add a button "open buid folder in "explorer" or "at drive location" somewhere on icestudio GUI for those who want to see the .v, .bit files,....

2) if we dont do the above suggestion and want to keep the "buid dir" in the same folder as the ice file,

I think, in this case it will be interesting for us to make a difference between a block design and "board" design,

- board design = .ice files = design containing real FPGA ports/pins as in inputs and outputs

- block design = .ic files = design containing only virtual ports as inputs and outputs  ( and that the only ones allowed to be store in the "system collection folder" )

And those .ic block can be File/opened by icestudio but not allowed to be build trough icestudio command.

They can be modified and saved as .ic file in the "system dir".

And if a physical Fpga port is inserted during the modification, they can only be saved as .ice files and stored in a no system/collection folder.

[charli va]

Hi Joaquim, Tim!! i like a lot some of your ideas.

I think we could redefine this behaviour as this:

1) At startup ask at the user for a working directory as Tim suggested, in the user selected directory Icestudio create an "Icestudio" directory.   For example if you select $HOME/Documents, Icestudio will create $HOME/Documents/Icestudio

2) In this directory Icestudio create a "builds" directory, in this directory Icestudio create a folder for each design that could be build, verified,etc

3) In Icestudio we put a new icon in the launch bar with the "Open Build folder" and when you click , open the system folder navigator with the content

4) I think could be very interesting if in this directory we create python env and apio directories, in fact we could use this directory to merge all of the directories needed by Icestudio.

What do you think?

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/a67e289f-95db-4da9-a2d2-54258a73c852n%40googlegroups.com.

[Jo mo]

Hola charli,

for me 1), 2), 3),and 4) are 100% ok!

about 4), venv and apio are a bit less "user work related",  but to avoid spreading files every-were in the drive, i think it is a good idea to store the builds, venv, apio,... in a same folder.

This one can be:

- "fixed"as it is now (eg: c:\ users\xxx\.icestudio\)

- or chosen by the user at icesudio installation as you wrote.

i have a little preference for the "fixed" option.

If we have all the same folders (Os variations apart) it can be easier when communicating with the users ( in tutorials, or when trying to debug  their problems)

Have a good evening!

[Tim Rudy]

Point (3) is awesome and a help to the user.

About point (1) and (2) creating the "builds" directory in one location: This presents an issue that may come up later:

Users who have

 ..path/ExperimentA/proj.ice

 ..path/ExperimentB/proj.ice

will lose the working files history if the "builds" directory is called "ice-builds/Proj" (name of the project stays the same - or the user happens to reuse that name a lot).

Can be avoided by using a hash of the path + Proj. But that approach obscures things for the power user again. They would need to use the Open/Navigate button of point (3), otherwise they'll be searching for stuff at, say:

 c:\users\xxx\.icestudio\ice-builds\X6YTLW3K\main.v

Thanks, Regards

[charli va]

Hi Tim! i'm agree with you about the ice-build dir.  Your idea of the hash i think would be nice. We could do a hash with the absolute path.  In the next days i'll have some to test it.

Thanks for your feedback!

Hi Tim! I agree with you about the ice-build directory. Your hash idea, I think would be nice. We could hash the absolute path. In the next few days I will have some to test it.

Thank you for your comments!

Para ver esta conversación en el sitio web, visita https://groups.google.com/d/msgid/fpga-wars-explorando-el-lado-libre/8084da93-2fc0-4afd-aee2-55dac3cf9f53n%40googlegroups.com.