[Haskell-cafe] wanted: haskell one-liners (in the perl sense of one-liners)

[Thomas Hartman]

Okay, I am aware of

http://haskell.org/haskellwiki/Simple_unix_tools

which gives some implementation of simple unix utilities in haskell.

But I couldn't figure out how to use them directly from the shell, and
of course that's what most readers will probably wnat.

Or let me put it another way.

Is there a way to do

find -maxdepth 1 -type f | xargs du | perl -ane 'print "\$F[0]\n"' |
perl -e '$sum += $_ while <>; print "$sum\n"'

as a shell command that idiomatically uses haskell?

For non-perlers, that sums up the disk usage of all files in the
current directory, skipping subdirs.

print "\$F[0]\n

looks at the first (space delimited) collumn of output.

perl -e '$sum += $_ while <>; print "$sum\n"'

, which is I guess the meat of the program, sums up all the numbers
spewed out of the first column, so in the end you get a total.

So, anyone out there want to establish a haskell one liner tradition?

:)

thomas.
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Bas van Dijk]

On Friday 02 March 2007 14:48, Thomas Hartman wrote:
> ...

> But I couldn't figure out how to use them directly from the shell, and
> of course that's what most readers will probably wnat.

> ...

From Dons wiki article http://haskell.org/haskellwiki/Blog_articles I noticed
this blog with a nice tip on how to run Haskell expressions from the shell:

http://www.joachim-breitner.de/blog/archives/156-Haskell-on-the-Command-Line.html

regards,

Bas van Dijk

[David Roundy]

On Fri, Mar 02, 2007 at 03:17:10PM +0100, Bas van Dijk wrote:
> On Friday 02 March 2007 14:48, Thomas Hartman wrote:
> > ...
> > But I couldn't figure out how to use them directly from the shell, and
> > of course that's what most readers will probably wnat.
> > ...
>
> From Dons wiki article http://haskell.org/haskellwiki/Blog_articles I noticed
> this blog with a nice tip on how to run Haskell expressions from the shell:
>
> http://www.joachim-breitner.de/blog/archives/156-Haskell-on-the-Command-Line.html

I liked this trick (which involves writing bash functions that call ghc
-e), but am a little disappointed that I can't see how to import modules
using ghc -e, which rather severely limits what I can do with one liners.
Ideally you'd at least like to be able to do something like:

in my .bashrc:

if which ghc >/dev/null
then
function hrun { ghc -e "interact($*)"; }
function hmap { hrun "map($*)"; }
function hmapl { hrun "unlines.map($*).lines" ; }
function hmapw { hmapl "(unwords.map($*).words)" ; }
fi

then I'd like to run

echo hello world | hmap toUpper

but there's no way that I can see to import Data.Char either on the command
line, or as a flag passed by hrun to ghc. Ideally one would want the bash
function to always deal with importing Data.Char, one of the regexp
libraries, etc.

Any suggestions?
--
David Roundy
Department of Physics
Oregon State University

[Joachim Breitner]

Hi,

Am Freitag, den 02.03.2007, 09:25 -0800 schrieb David Roundy:
> echo hello world | hmap toUpper

$ echo hello world | hmap 'map Char.toUpper'
HELLO WORLD

(Whereas hmap is as defined in the original blog entry)

Greetings,
Joachim
--
Joachim Breitner
e-Mail: ma...@joachim-breitner.de
Homepage: http://www.joachim-breitner.de
ICQ#: 74513189

[Thomas Hartman]

Great tip! One question though.

What condition is

if which ghc >/dev/null

checking?

What bad thing will happen if you don't do this check?

Sorry this is more a bash question than a haskell question.

2007/3/2, Bas van Dijk <basva...@home.nl>:

[David Roundy]

On Fri, Mar 02, 2007 at 08:12:19PM +0100, Thomas Hartman wrote:
> Great tip! One question though.
>
> What condition is
>
> if which ghc >/dev/null
>
> checking?
>
> What bad thing will happen if you don't do this check?
>
> Sorry this is more a bash question than a haskell question.

It just checks whether ghc is in the path.

--
David Roundy
Department of Physics
Oregon State University

[Thomas Hartman]

To answer my original question, here's a few ways to accomplish what I
wanted with haskell

Perl is still a lot faster than ghc -e, but I guess if you wanted
speed you could compile first.

********************************************************************

thartman@linodewhyou:~/learning/haskell/UnixTools$ ls -l
total 16
-rw-r--r-- 1 thartman thartman 2726 Dec 20 07:56 UnixTools.hs
-rw-r--r-- 1 thartman thartman 82 Jan 7 07:18 echo.hs
-rwxr--r-- 1 thartman thartman 790 Mar 4 05:02 oneliners.sh
-rwxr--r-- 1 thartman thartman 646 Mar 4 04:18 oneliners.sh~

thartman@linodewhyou:~/learning/haskell/UnixTools$ ./oneliners.sh
haskell, ghc -e pipe
16

real 0m1.652s
user 0m0.600s
sys 0m0.030s
**********
haskell, hmap pipe
16

real 0m1.549s
user 0m0.410s
sys 0m0.200s
**********
haskell, two pipes
16

real 0m2.153s
user 0m0.900s
sys 0m0.370s
**********
perl, two pipes
16

real 0m0.185s
user 0m0.010s
sys 0m0.100s

thartman@linodewhyou:~/learning/haskell/UnixTools$

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat oneliners.sh
hmap (){ ghc -e "interact ($*)"; }
hmapl (){ hmap "unlines.($*).lines" ; }
hmapw (){ hmapl "map (unwords.($*).words)" ; }

function filesizes () {

find -maxdepth 1 -type f | xargs du
}

echo haskell, ghc -e pipe
time filesizes | ghc -e 'interact $ (++"\n") . show . sum . map ( (
read :: String -> Integer ) . head . words ) . lines '
echo "**********"

echo haskell, hmap pipe
time filesizes | hmap '(++"\n") . show . sum . map ( ( read :: String
-> Integer ) . head . words ) . lines'
echo "**********"

echo haskell, two pipes
time filesizes | hmapl "map ( head . words )" | hmap '(++"\n") . show
sum . map ( read :: String -> Integer ) . lines'
echo "**********"

echo perl, two pipes
time filesizes | perl -ane 'print "$F[0]\n"' | perl -e '$sum += $_
while <>; print "$sum\n"'

2007/3/2, Thomas Hartman <tphy...@gmail.com>:

[Donald Bruce Stewart]

There's some nice one liners bundled with h4sh:

http://www.cse.unsw.edu.au/~dons/h4sh.html

For example:

http://www.cse.unsw.edu.au/~dons/h4sh.txt

If you recall, h4sh is a set of unix wrappers to the list library.
I still use them everyday, though probably should put out a new release
soon.

-- Don

tphyahoo:

> . sum . map ( read :: String -> Integer ) . lines'

[Thomas Hartman]

That seems like a really great thing to have. But I had troubles installing it.

h4sh depends on hs-plugins.

And...
****************
thartman@linodewhyou:~/haskellInstalls/hs-plugins$ ./Setup.lhs configure
Setup.lhs: Warning: The field "hs-source-dir" is deprecated, please
use hs-source-dirs.
Configuring plugins-1.0...
configure: /usr/local/bin/ghc-pkg
configure: Dependency base-any: using base-2.0
configure: Dependency Cabal-any: using Cabal-1.1.6
Setup.lhs: cannot satisfy dependency haskell-src-any
thartman@linodewhyou:~/haskellInstalls/hs-plugins$
****************
Advice?

2007/3/4, Donald Bruce Stewart <do...@cse.unsw.edu.au>:

[Donald Bruce Stewart]

Yes, it definitely is a little lagged. It should be ported to use lazy
bytestrings too. I wsa more suggesting the one liners as examples of
haskell use in the shell.

tphyahoo:

[Thomas Hartman]

I think I could have most of the oneliner goodness of h4sh, without
having to do the module install, if I could figure out a way to
include modules with ghc -e.

or, alternatively some way to specify modules as a ghc flag, analogous to

perl -MPath::To::Module -e 'commands'

Can this be made to work?

(From http://haskell.org/haskellwiki/Simple_unix_tools, which seems to
be repeated somehow in h4sh, although I'm not completely certain on
that.)

***************************************

thartman@linodewhyou:~/learning/haskell/UnixTools$ echo "1234" | ghc
-e 'interact id'
1234
thartman@linodewhyou:~/learning/haskell/UnixTools$ echo "1234" | ghc
-e 'UnixTools.cat'

<interactive>:1:0:
Bad interface file: UnixTools.hi
UnixTools.hi: openBinaryFile: does not exist (No such file or directory)

thartman@linodewhyou:~/learning/haskell/UnixTools$ head -n23 UnixTools.hs
module UnixTools where
--
-- Some unix-like tools written in simple, clean Haskell
--
--

import Data.List
import Data.Char
import System.IO
import Text.Printf
....
-- The 'cat' program
--
cat = interact id

[Brandon S. Allbery KF8NH]

On Mar 4, 2007, at 6:31 , Thomas Hartman wrote:

> Setup.lhs: cannot satisfy dependency haskell-src-any

Used to be bundled, now unbundled. On debian/ubuntu check your
libghc6-*-dev packages. (libghc6-haskell-src-dev?)

--
brandon s. allbery [linux,solaris,freebsd,perl] all...@kf8nh.com
system administrator [openafs,heimdal,too many hats] all...@ece.cmu.edu
electrical and computer engineering, carnegie mellon university KF8NH

[Yitzchak Gale]

Thomas Hartman wrote:
> Is there a way to do

<perl one-liner gibberish>

> as a shell command that idiomatically uses haskell?

I used to do those in Perl, too, years ago.

I switched to the readable step-by-step style of the
Python shell when I moved from Perl to Python.
It is a whole different mentality about how to work,
but in the end it is equally powerful and beautiful.

Now that I am a Haskell person, I find that the Python
style is a perfect fit for Haskell shells such as GHCi.
The :def command in GHCi makes this really
powerful (though I find that I rarely need that much
power).

You can kind of fake the one-liner style in Python,
but it is awkward. I assume that the same would
be true in Haskell (though I admit that I never
tried it very seriously). Unless you define all kinds
of single-character abbreviations, in which case
why not just use Perl?

Regards,
Yitzchak

[Claus Reinke]

>I think I could have most of the oneliner goodness of h4sh, without having to do
>the module install, if I could figure out a way to include modules with ghc -e.

i confess myself to be among those who underappreciated ghc -e, until this thread:)

as Joachim said (thanks for starting this, btw;-), we can use qualified names

$ echo hello world | hmap 'map Char.toUpper'
HELLO WORLD

and to get at your other methods, the question is not how to include modules
with ghc -e; instead, recall that ghc -e supplies a command to be evaluated
within the context of its parameter module (single input for a ghci session):

$ cat Imports.hs
import Debug.Trace
helper x = trace "hi there" (x+1)

$ ghc -e 'helper 41' Imports.hs
hi there
42

as to the original question in this thread, my .bashrc now also has a few less
symmetric entries:

function hrunl { ghc -e "interact(show.($*).lines)"; }
function hrunw { ghc -e "interact(show.($*).words)"; }
function hrunwl { ghc -e "interact(show.($*).map words.lines)"; }

using which the one-liner becomes something like

find -maxdepth 1 -type f | xargs du | hrunwl "sum . map (read . head)"

(the find/du is better left in shell tool land, it seems, and default Num is Integer)

hth,
claus

[Thomas Hartman]

Can't seem to find these packages. Do I need to add another repo? Or
build from source?

thartman@linodewhyou:~/haskellInstalls/hs-plugins$ apt-cache search
libghc6 | grep ^libghc6-haskell
libghc6-haskelldb-dev - Haskell library for expressing database queries
thartman@linodewhyou:~/haskellInstalls/hs-plugins$ cat /etc/apt/sources.list

deb http://archive.ubuntu.com/ubuntu/ dapper main restricted
deb-src http://archive.ubuntu.com/ubuntu/ dapper main restricted

## Uncomment the following two lines to add software from the 'universe'
## repository.
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://archive.ubuntu.com/ubuntu/ dapper universe
deb-src http://archive.ubuntu.com/ubuntu/ dapper universe

deb http://security.ubuntu.com/ubuntu dapper-security main restricted
deb-src http://security.ubuntu.com/ubuntu dapper-security main restricted

2007/3/4, Brandon S. Allbery KF8NH <all...@ece.cmu.edu>:

[Thomas Hartman]

quoth brandon allberry in a private message (answering my unintended
private message)

On Mar 7, 2007, at 1:51 , Thomas Hartman wrote:

> Can't seem to find these packages. Do I need to add another repo? Or
> build from source?

You may have to backport; I found it in feisty (universe) in a quick
search, but it's not in dapper or edgy.

2007/3/7, Thomas Hartman <tphy...@gmail.com>:

[Thomas Hartman]

In the spirit of making easy things easy, here is a "haskell from
shell" one-line grepper, that uses regexen.

Now, if only I could get pcre-regex installed I would be quite
content. (Still stuck using posix RE for now.)

**************

thartman@linodewhyou:~/learning/haskell/UnixTools$ time ./q-words.sh
q
qua
quack
quacked
quackery
quackery's
quacking
quacks
quad
quad's

real 0m3.186s
user 0m1.900s
sys 0m0.810s

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat q-words.sh
cat /usr/share/dict/american-english | ghc -e 'interact $ unlines.
take 10 . filter ( \x -> x =~ "^q" :: Bool ) . lines' Imports.hs

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat Imports.hs
import Text.Regex.Posix

****************************

2007/3/2, Thomas Hartman <tphy...@gmail.com>:

[Thomas Hartman]

with PCRE. now (and yep, it's a lot faster than posix):

***************************************************************************

thartman@linodewhyou:~/learning/haskell/UnixTools$ ./q-words.sh

q
qua
quack
quacked
quackery
quackery's
quacking
quacks
quad
quad's

real 0m6.691s
user 0m2.460s
sys 0m3.000s

q
qua
quack
quacked
quackery
quackery's
quacking
quacks
quad
quad's

real 0m1.032s
user 0m0.750s
sys 0m0.100s
thartman@linodewhyou:~/learning/haskell/UnixTools$ cat q-words.sh
time cat /usr/share/dict/american-english | ghc -e 'interact $

unlines. take 10 . filter ( \x -> x =~ "^q" :: Bool ) . lines'

ImportsRegexPosix.hs

time cat /usr/share/dict/american-english | ghc -e 'interact $

unlines. take 10 . filter ( \x -> x =~ "^q" :: Bool ) . lines'

ImportsRegexPCRE.hs

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat ImportsRegexPosix.hs
import Text.Regex.Posix

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat ImportsRegexPCRE.hs
import Text.Regex.PCRE

thartman@linodewhyou:~/learning/haskell/UnixTools$

2007/3/7, Thomas Hartman <tphy...@gmail.com>:

[Thomas Hartman]

Okay, so much for PCRE match.

Can someone show me give me pointers on PCRE replace?

Ideal would be something with all the =~ s/// semantics from perl.
(Not sure if this is included in Text.Regex.PCRE. is it?)

In other words, how to do this with (preferrably) Text.Regex.PCRE ?

Of course in this case a POSIX regex is fine, but since PCRE is faster
and more expressive, and what I'm used to anyway, PCRE is probably is
what I will tend to use by default, unless portability is paramount.

****************************
thartman@linodewhyou:~/learning/haskell/UnixTools$ cat lines.txt
a b c
a b c
a b c

thartman@linodewhyou:~/learning/haskell/UnixTools$ cat lines.txt |
perl -ne '$_ =~ s/(\w) (\w) (\w)/$1 z $3/; print $_'
a z c
a z c
a z c
thartman@linodewhyou:~/learning/haskell/UnixTools$

[Thomas Hartman]

Just noticed a comment in

http://www.serpentine.com/blog/2007/02/27/a-haskell-regular-expression-tutorial/

which says there's no perl-like regex replace in the library, and links to

http://hpaste.org/697

which is an attempt at providing one.

Not sure if this is useful or not.

2007/3/7, Thomas Hartman <tphy...@gmail.com>:

[Chris Kuklewicz]

Thomas Hartman wrote:
> Just noticed a comment in
>
> http://www.serpentine.com/blog/2007/02/27/a-haskell-regular-expression-tutorial/
>
>
> which says there's no perl-like regex replace in the library, and links to
>
> http://hpaste.org/697
>
> which is an attempt at providing one.
>
> Not sure if this is useful or not.
>

Any given replacement routine is less than 10 lines of code and will do exactly
what you need.

A general replacement library has to contend with several things:
1a) What syntax/semantics?
1b) How do you supply a specification? Must it be the same type as the
regular expression or the data?
1c) How do you report errors?
2) Which regex-* backends to support?
3) What types to work on? [Char], Seq Char, ByteString, Lazy ByteString.
4a) If the backend/type supports lazy matching then does the replacing?
4b) What if the backend/type does not support lazy match or strictness is desired?
5) If there is laziness then can it handle infinite streams of input?
6) Is anyone smart enough to design this API without actual users?

Note that some approaches allow for much more efficiency than others. Taking a
normal ByteString and performing replacement to create a Lazy ByteString makes
sense, but is a bit of wrinkle.

But as you pointed to on http://hpaste.org/697 any given example of a
replacement routine will be very small, and easy to build on top of the regex-* API.

[Thomas Hartman]

Just thought I'd add another potentially helpful bit to this oneliner
/ shell scripting thread. Though to be fair, this perhaps strains the
definition of "one" in "one liner".

This takes a long file containing mostly numerical data, filters out
the numerical data, and sorts it numerically. (Not the same thing as
sorting alphabetically, which is what you get by default, or using the
unix sort utility). Maybe there's some flag to make unix sort util act
like this? Enh, who cares, now we have haskell. :)

Thanks to Thunder, Quicksilver, and whoever else it was on #haskell
who helped me out with this.

******************************************************

thartman@linodewhyou:~/learning/haskell/UnixTools>cat sortNumeric.sh | head -n10
cat out_select_char_length_csv.out | ghc -e '
interact $
unlines
map show

-- more efficient than -- reverse . take 10 . reverse
( \s -> drop (length s - 10 ) s )

Data.List.sort -- maybe not necessary?

map ( read :: String -> Integer )

Data.Set.toAscList . Data.Set.fromList -- more efficient than prelude nub
filter ( all Data.Char.isDigit ) . lines'

2007/3/7, Chris Kuklewicz <has...@list.mightyreason.com>:

[Thomas Hartman]

To answer my own post, the Data.List.sort *is* necessary.

Otherwise, you get alphabetic sort.

2007/3/20, Thomas Hartman <tphy...@gmail.com>:

> Just thought I'd add another potentially helpful bit to this oneliner
> / shell scripting thread. Though to be fair, this perhaps strains the
> definition of "one" in "one liner".
>
> This takes a long file containing mostly numerical data, filters out
> the numerical data, and sorts it numerically. (Not the same thing as
> sorting alphabetically, which is what you get by default, or using the
> unix sort utility). Maybe there's some flag to make unix sort util act
> like this? Enh, who cares, now we have haskell. :)
>
> Thanks to Thunder, Quicksilver, and whoever else it was on #haskell
> who helped me out with this.
>
> ******************************************************
>
> thartman@linodewhyou:~/learning/haskell/UnixTools>cat sortNumeric.sh | head -n10
> cat out_select_char_length_csv.out | ghc -e '
> interact $
> unlines

> . map show

>
> -- more efficient than -- reverse . take 10 . reverse

> . ( \s -> drop (length s - 10 ) s )
>
> . Data.List.sort -- maybe not necessary?
> . map ( read :: String -> Integer )
> . Data.Set.toAscList . Data.Set.fromList -- more efficient than prelude nub
> . filter ( all Data.Char.isDigit ) . lines'

[Thomas Hartman]

Same thing, but with better comments and whitespace.

thartman@linodewhyou:~/learning/haskell/UnixTools>head -n30 sortNumeric.sh

cat out_select_char_length_csv.out | ghc -e '
interact $
unlines

-- take 10 from the end

-- more efficient than
-- reverse . take 10 . reverse
. ( \s -> drop (length s - 10 ) s )

. map show -- convert Integer to String

-- sort numerically
. Data.List.sort

. map ( read :: String -> Integer ) -- convert String to Integer

-- Uniqify

-- more efficient than prelude nub

-- sorts too, but alphanumerically, whereas we want numerically
. Data.Set.toAscList . Data.Set.fromList

. filter ( all Data.Char.isDigit )

. lines
'

2007/3/20, Thomas Hartman <tphy...@gmail.com>:

[Dougal Stanton]

Quoth Thomas Hartman, nevermore,

> This takes a long file containing mostly numerical data, filters out
> the numerical data, and sorts it numerically. (Not the same thing as
> sorting alphabetically, which is what you get by default, or using the
> unix sort utility). Maybe there's some flag to make unix sort util act
> like this? Enh, who cares, now we have haskell. :)

Alas, foiled again! ;-)

sort flag:

-n, --numeric-sort

D.
--
Dougal Stanton

[Thomas Hartman]

You contributed

#12:14 < Thunder> @pl \s -> drop (length s - n) s
#12:14 < lambdabot> drop =<< subtract n . length

But, on second thought, 1) I can't use this as a drop-in replacement
for the non points free (right term?) version, and 2) I don't really
understand it.

Still, I would be curious to see if there is a way to make this useful somehow.

2007/3/20, Lutz Donnerhacke <lu...@iks-jena.de>:

> In iks.lists.haskell, you wrote:
> > This takes a long file containing mostly numerical data, filters out
> > the numerical data, and sorts it numerically. (Not the same thing as
> > sorting alphabetically, which is what you get by default, or using the
> > unix sort utility). Maybe there's some flag to make unix sort util act
> > like this? Enh, who cares, now we have haskell. :)
>

> sort -g
>
> > Thanks to Thunder
>
> I was not that helpful.

[Lutz Donnerhacke]

* Thomas Hartman wrote:
> You contributed
>
> #12:14 < Thunder> @pl s -> drop (length s - n) s
> #12:14 < lambdabot> drop =<< subtract n . length
>
> But, on second thought, 1) I can't use this as a drop-in replacement
> for the non points free (right term?) version, and 2) I don't really
> understand it.

I did not contribute, but polluted the channel with my own test.

In order to understand this construct you have to switch to the (-> a) Monad:

drop =<< subtract n . length

== do x <- subtract n . length
drop x

This construct consists of partially applied functions, i.e. functions
waiting for an argument.

If you apply an argument to the whole construct, it is applied to each line
seperatly.

So "drop =<< subtract n . length $ s" becomes:

let x = subtract n . length $ s
in drop x s

==

let x = length s - n
in drop x s

==

drop (length s - n) s

I did not expect this monadic approach from lambdabot and was somewhat
surprised. I assumed an application of "liftM2 drop" instead.

[Jules Bean]

Thomas Hartman wrote:
> To answer my own post, the Data.List.sort *is* necessary.
>
> Otherwise, you get alphabetic sort.

Which is why you do the Set trick at a different stage in the process,
like this:

>>
>> interact $
>> unlines
>> . map show
>>
>> -- more efficient than -- reverse . take 10 . reverse
>> . ( \s -> drop (length s - 10 ) s )

>> . Data.Set.toAscList . Data.Set.fromList -- move it *here* after
>> integer conversion and it sorts and nubs for you

>> . map ( read :: String -> Integer )

>> . filter ( all Data.Char.isDigit ) . lines'

_______________________________________________