Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

A visual collaborative Unix Shell interface, codenamed "ShellHive"

41 views
Skip to first unread message

Omar Castro

unread,
Jun 5, 2014, 8:54:39 AM6/5/14
to
Hello everyone, I don't know if this is the right group to post this.

I am doing a dissertation, which objective is to simplify the creation and management of Unix Shell (SH) scripts, transforming the text script in a visual graph, which its focus is the processing of large amounts of data (in short, to do Big Data related tasks)

It also tries to simplify other things such as sharing and collaborating the graphs with other people, so that is easier to understand for people with high and low experience with SH scripts (such as trainees), it can also be used to teach other people how to write those scripts.

There is a public demo page here

paginas.fe.up.pt/~ei08158/thesis/public_demo.php

I would like you to play with the demo, then to send me feedback by clicking on the "send feedback" link on the top bar, it will be a relatively short form.

You can send feedback directly to this thread, but the feedback sent to the "send feedback" link allows me to organize the gathered information.

Thank you for for attention.

PS: If the feedback is positive I might consider and continue the project to a full featured web application :)

PSS: If this is not the right group to post this, where would you recommed me to post?

Kaz Kylheku

unread,
Jun 5, 2014, 9:57:23 AM6/5/14
to
On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
> There is a public demo page here
>
> paginas.fe.up.pt/~ei08158/thesis/public_demo.php

This redirects to "188.226.185.166" which wants my browser to execute
Javascript.

Which is blocked by NoScript.

Which is staying that way; thanks for playing!

Omar Castro

unread,
Jun 5, 2014, 10:08:48 AM6/5/14
to
Well, it is a collaborative application which uses websockets :)
and the name is not definitive, that, why i didnt buy an domain yet

thanks for trying!

Omar Castro

unread,
Jun 5, 2014, 11:46:49 AM6/5/14
to
Well, I decided to buy a domain for a short time

the demo link is now

shellhive.com/project/play/1

And one thing, the project is open source, here is the project link

https://github.com/OmarCastro/ShellHive

Kaz Kylheku

unread,
Jun 5, 2014, 12:47:16 PM6/5/14
to
On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
> Well, it is a collaborative application which uses websockets :)
> and the name is not definitive, that, why i didnt buy an domain yet

Okay, you seem real enough; I will try it.

Omar Castro

unread,
Jun 5, 2014, 1:04:51 PM6/5/14
to
Thanks!

However, the web app is not completed, so bugs are to be expected, I started developing the app Jan of this year, as a one-man team, so I was the developer, tester, designer, etc...

Kaz Kylheku

unread,
Jun 5, 2014, 2:33:58 PM6/5/14
to
On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
> Quinta-feira, 5 de Junho de 2014 17:47:16 UTC+1, Kaz Kylheku escreveu:
>> On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
>>
>> > Well, it is a collaborative application which uses websockets :)
>>
>> > and the name is not definitive, that, why i didnt buy an domain yet
>>
>>
>>
>> Okay, you seem real enough; I will try it.
>
> Thanks!
>
> However, the web app is not completed, so bugs are to be expected, I started
> developing the app Jan of this year, as a one-man team, so I was the
> developer, tester, designer, etc...

That's okay; completion/completeness is the kiss of death to any software
project: programming language, OS, application, ...

Kaz Kylheku

unread,
Jun 5, 2014, 2:48:47 PM6/5/14
to
On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
> Quinta-feira, 5 de Junho de 2014 17:47:16 UTC+1, Kaz Kylheku escreveu:
>> On 2014-06-05, Omar Castro <omar.ca...@gmail.com> wrote:
>>
>> > Well, it is a collaborative application which uses websockets :)
>>
>> > and the name is not definitive, that, why i didnt buy an domain yet
>>
>>
>>
>> Okay, you seem real enough; I will try it.
>
> Thanks!
>
> However, the web app is not completed, so bugs are to be expected, I started
> developing the app Jan of this year, as a one-man team, so I was the
> developer, tester, designer, etc...

It looks neat.

"Garbage-collect" your compiler-generated fifo's once the show terminates:
/tmp/fifo-*-* :)

You may want to work on adding some optimizing pass to get rid of useless
uses of fifos and cats, so that realistic command lines are generated
wherever possible.

Also, you may want to optionally take advantage of bash process substitution
in the code generator (non-POSIX feature of bash). For instance, command A can
send its standard output to B, and standard error to command C like this:

A 2> >(C) | B

bash sets up the file descriptors, eliminating the "fifology" of redirection
descriptor 2 to a named pipe.

I think that arbitrarily complex graphs can be compiled to a single command
this way. That has its advantages because it becomes a single job under
POSIX job control.

Omar Castro

unread,
Jun 5, 2014, 3:44:29 PM6/5/14
to
>
> It looks neat.
>
>
>
> "Garbage-collect" your compiler-generated fifo's once the show terminates:
>
> /tmp/fifo-*-* :)
>

No need, it uses Docker behind the scenes, so anything outside the filesystem directory gets deleted.

I used /tmp because since the objective of /tmp is to have files to be temporary


>
>
> You may want to work on adding some optimizing pass to get rid of useless
>
> uses of fifos and cats, so that realistic command lines are generated
>
> wherever possible.
>

My first script generator actually does this, but then I had to think the
use-case where the user splits and joins the application whenever the user
wants, that where it started to become difficult, and there is a feature called macros which my thesis advisor though it is a great idea to add, a macro in this case is a composition of interconnected commands, a macro can accept multiple inputs and multiple outputs, I implemeted it last week, but it is
buggy, so the best way to do this is my making named pipes.

After I finish my thesis (in next month), I would need to create an algorithm to find paths that would not need to create named pipes, and genereate optimized code

>
> Also, you may want to optionally take advantage of bash process substitution
>
> in the code generator (non-POSIX feature of bash). For instance, command A can
>
> send its standard output to B, and standard error to command C like this:
>
> A 2> >(C) | B
>
> bash sets up the file descriptors, eliminating the "fifology" of redirection
>
> descriptor 2 to a named pipe.
>
>
>
> I think that arbitrarily complex graphs can be compiled to a single command
>
> this way. That has its advantages because it becomes a single job under
>
> POSIX job control.


I'm aware of that actually, when you create a macro (buggy and incomplete as for now) you can create with a complete command line, like

A 2> >(C) | tee >(D) >(E) | B

and it generates a complete graph try creating a macro like this (shortcut is Shift + M)

with a command like this: grep "test" | tee >(gzip > file4.txt.gz) >(bzip2 > file4.txt.bz2) | sort > sorted.txt

the name of the macro must have one word and must not exist
its description can be anything, or empty
the command can be anything

let me give you an example:

name: compress-sort
description: an example macro that compresses and sorts it
command: grep "test" | tee >(gzip > file4.txt.gz) >(bzip2 > file4.txt.bz2) | sort > sorted.txt

An macro will be created with a generated graph based on the command

Omar Castro

unread,
Jun 6, 2014, 5:09:19 AM6/6/14
to
Late announcement, I added an introduction page to help users to use the application.

And good news, while sleeping, I found a solution to reduce the "fifology" of the generated commands, I'm gonna implement and test the solution, stay tuned...

Omar Castro

unread,
Jun 6, 2014, 7:37:26 AM6/6/14
to
I finished optimizing the generated commands, and I also added the "echo" command (was not in my plans though)
0 new messages