[9fans] data analysis on plan9
hugo rivera
since I discovered plan 9, about two years ago, I've been constantly
amazed by its simple yet quite powerful design.
From one year now, I am looking forward to move to plan 9 as my main
OS, but I am not able to do so because it lacks the data analysis
tools available in some other systems, like linux.
Because my work involves dealing with data coming from experiments in
astro-particle physics, I am more or less tied to data analysis
software like the R programming language, Python's Numpy, Cern's ROOT
and even gnuplot. While using them, I realized that most of the time I
deal with text files that go here and there as input or output of
small specific programs that perform a given task (I don't know if
this is the result of my Unix/Plan 9 background or just a
coincidence). Say I have a command 'clean' that removes undesired
points from a body of data, and another command 'four' that performs
the FFT; so they are used together as
clean data.txt | four > results.txt
so it occurred to me that one can create single commands to interact
among them to perform some analysis on data, just like in the original
Unix style. Awk can be used as glue among them, with some other small
glue utilities. Plotting data is another thing that I would like to
integrate into this, since plots are quite frequent while analysing
data, but I am not sure how.
Also, something similar to GSL (http://www.gnu.org/software/gsl/)
would be invaluable or maybe even indispensable.
Maybe some day I'll start to write some commands for plan 9 to begin
working on it, but I want to convince myself that this is worth the
time spent.
What do you think of this? my main concern is that perhaps the "do one
thing well" design falls short for data analysis. I've never seen
people work like this on data analysis before (but I do not think I am
the first to do it) because in general, they tend to use large data
analysis frameworks. I'd really appreciate some feedback on this from
people working on data analysis and also from the plan 9 community
(otherwise I wouldn't be writing here :-)
Saludos
--
Hugo
Federico G. Benavento
gsl-1.6.tbz GNU Scientific Library, native port.
/n/sources/contrib/pac/sys/src/lib/gsl-1.6.tbz
hth
--
Federico G. Benavento
Jason Catena
yet ported to Plan 9. Ron Minnich et al. seem to prefer gnuplot, and
reported that they generated data for it and used it in a paper, but
weren't specific whether the gnuplot ran on the same plan9 box or
another *nix.
From http://doc.cat-v.org/plan_9/IWP9/2008/trace.pdf pp. 19-21:
"4.1 Visualizing trace device output
Once we had the data, we needed a way to analyse the information. After
working with the data for a while, we realized that the output as
shown in Figure
1 would be very useful. No graphiing [sic] tool available to us in
Plan 9 or Linux
was able to create that output. In the end, we determined that gnuplot was the
most appropriate tool, but even then the data required significant processing to
get it into the proper form.
We wrote a suite of scripts usng rc, the plan 9 shell; acid, the Plan
9 debugger;
awk, and sed to generate data appropriate for plotting with gnuplot.
The createplot
script has the ability to filter out functions which ran for less than
a specified number of clock cycles, which is useful for reducing the amount of
noise in a plot. To generate a plot from the data collected earlier, discarding
functions which completed in less than 4000 cycles, we just ran:
plots/createplot /amd64/9k8pf 4000 ./trace > plotme
and fed the input into gnuplot."
Jason Catena
ron minnich
> I'd also be interested in knowing whether gnuplot or an equivalent is
> yet ported to Plan 9. Ron Minnich et al. seem to prefer gnuplot, and
> reported that they generated data for it and used it in a paper, but
> weren't specific whether the gnuplot ran on the same plan9 box or
> another *nix.
gnuplot on linux. even octave uses gnuplot. Not that it's great, but
there is not much else.
ron
J. R. Mauro
On Jul 9, 2009, at 14:40, hugo rivera <uai...@gmail.com> wrote:
> Hi,
> since I discovered plan 9, about two years ago, I've been constantly
> amazed by its simple yet quite powerful design.
> From one year now, I am looking forward to move to plan 9 as my main
> OS, but I am not able to do so because it lacks the data analysis
> tools available in some other systems, like linux.
> Because my work involves dealing with data coming from experiments in
> astro-particle physics, I am more or less tied to data analysis
>
> software like the R programming language, Python's Numpy, Cern's ROOT
There a plan 9 R port, isn't there? If not, there might be an R done
in python or something.
>
> and even gnuplot. While using them, I realized that most of the time I
> deal with text files that go here and there as input or output of
> small specific programs that perform a given task (I don't know if
> this is the result of my Unix/Plan 9 background or just a
> coincidence). Say I have a command 'clean' that removes undesired
> points from a body of data, and another command 'four' that performs
> the FFT; so they are used together as
> clean data.txt | four > results.txt
> so it occurred to me that one can create single commands to interact
> among them to perform some analysis on data, just like in the original
> Unix style. Awk can be used as glue among them, with some other small
> glue utilities. Plotting data is another thing that I would like to
> integrate into this, since plots are quite frequent while analysing
> data, but I am not sure how.
Plan 9 has plot program that fit well in a pipeline. Even gnuplot can
go in a pipeline.
>
> Also, something similar to GSL (http://www.gnu.org/software/gsl/)
> would be invaluable or maybe even indispensable.
> Maybe some day I'll start to write some commands for plan 9 to begin
> working on it, but I want to convince myself that this is worth the
> time spent.
I think it's worth it. Parts of this idea are already there (sum,
sort, join, plot)
>
> What do you think of this? my main concern is that perhaps the "do one
> thing well" design falls short for data analysis. I've never seen
> people work like this on data analysis before (but I do not think I am
> the first to do it) because in general, they tend to use large data
> analysis frameworks. I'd really appreciate some feedback on this from
> people working on data analysis and also from the plan 9 community
> (otherwise I wouldn't be writing here :-)
I know someone who does astrophysics analysis and visualization
(including movies) on a special "OS" he wrote that works entirely like
Unix pipes and filters. I think developing an anlysis framework as a
pipes-and-filters toolbox is great.
>
> Saludos
>
> --
> Hugo
>
J. R. Mauro
It wouldn't be too bad to translate gnuplot to plan9 plot/graph, would
it?
Roman V Shaposhnik
> > Also, something similar to GSL (http://www.gnu.org/software/gsl/)
>
> gsl-1.6.tbz GNU Scientific Library, native port.
>
> /n/sources/contrib/pac/sys/src/lib/gsl-1.6.tbz
I've been meaning to ask this for a long time -- do we have a
catalog of things in contrib?
Thanks,
Roman.
John Floren
http://plan9.bell-labs.com/wiki/plan9/Contrib_index/
--
"I've tried programming Ruby on Rails, following TechCrunch in my RSS
reader, and drinking absinthe. It doesn't work. I'm going back to C,
Hunter S. Thompson, and cheap whiskey." -- Ted Dziuba
erik quanstrom
the parsing seems a bit odd, at least for my contrib stuff.
contrib/list from contrib(1) does a better job of listing contrib
packages.
- erik
Steve Simon
-v is verbose
user and package restrict the output to that
user on that users package only.
see contrib(1)
-Steve