Running a list of cmd's

[ryan....@gmail.com]

I'm trying to download a list of webpages to a directory, where the directory is my target
to be built:

"//*.matchpages" %> \out -> do
let year = takeBaseName out
let html = year <.> "matches.html"
need [html]
() <- cmd "mkdir" out
Stdout stdout <- cmd Shell "grep" "Match stats" html "|" "grep -oE" "'[[:digit:]]+.html'"

Now I need to run this function for each line of stdout, something like:

map (\x -> cmd ("curl http://afltables.com/afl/stats/games/"++year++"/"++x) "-o" (out </> xl)) (lines stdout)

I get type mismatches though (it needs an Action ()) which I can't figure out because I'm still new to Haskell.
Can you suggest a solution?

Thanks for any help,
Ryan

[Ryan Gonzalez]

On May 11, 2016 10:09 AM, <ryan....@gmail.com> wrote:
>
> I'm trying to download a list of webpages to a directory, where the directory is my target
> to be built:
>
>  "//*.matchpages" %> \out -> do
>         let year = takeBaseName out
>         let html = year <.> "matches.html"
>         need [html]
>         () <- cmd "mkdir" out
>         Stdout stdout <- cmd Shell "grep" "Match stats" html "|" "grep -oE" "'[[:digit:]]+.html'"
>
> Now I need to run this function for each line of stdout, something like:
>
>      map   (\x -> cmd ("curl http://afltables.com/afl/stats/games/"++year++"/"++x) "-o" (out </> xl)) (lines stdout)
>

What if you change the `map` to `mapM_`?

> I get type mismatches though (it needs an Action ()) which I can't figure out because I'm still new to Haskell.
> Can you suggest a solution?
>
> Thanks for any help,
> Ryan

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong.
http://kirbyfan64.github.io/

[Neil Mitchell]

Hi Ryan Eckbo,

First thing to note, the fact that directory creating rules kind-of
work was a bug in Shake that got fixed in Shake-0.15.6. I suggest you
upgrade, at which point things will fail, but that's probably good
since the directory creating rule would have half-worked and been very
confusing.

Fortunately, it should be as simple as switching that thing from being
a folder to a file, and using writeFile' to produce it.

As Ryan Gonzalez says, you should be able to fix the immediate issue
using mapM_. I suggest you get it working first, but once you have
something, a few pointers:

* I'd define `wget :: String -> FilePath -> IO ()` as a helper, so you
aren't always calling cmd with wget. Makes it easier to switch to
curl, or some other library later on. Plus simpler to use.

* Using grep is usually an anti-pattern, much better to parse with
Haskell if you can, even loosely. However, if you are learning
Haskell, probably one to leave towards the end.

* If instead of doing `wget` on each file in the `.matchpages` rule
you issued a `need` which fetched them all you would get automatic
parallelism, isolation, resumption, exception-handling, optional
staunch mode, better progress prediction etc. Typically a rule should
do 1 or 2 command line calls, and defer to other rules for the
remaining ones. (In the meantime, if you just want parallelism, look
at the forP function in Shake.)

Thanks, Neil

[ryan....@gmail.com]

Thanks Neil and Ryan.

Below is the result of using mapM_. I'm curious what the right way would be to force haskell
to recognize the correct instance type?

My current version of shake in stack is 0.15.5, so I'll try upgrading by making a yaml config file.

I tried looking for simple grep functionality in Haskell but the solutions I saw were overly verbose, when
I'm better at the language I'll try figure it out.

Lastly, if targets can only be files, but I want to be able to specify a directory to build, is the correct way
to make it a phony command?

Thanks!
Ryan

---
Build.hs:39:29:
No instance for (CmdResult b0) arising from a use of ‘cmd’
The type variable ‘b0’ is ambiguous
Note: there are several potential instances:
instance CmdResult GHC.IO.Exception.ExitCode
-- Defined in ‘Development.Shake.Command’
instance CmdResult System.Process.Internals.ProcessHandle
-- Defined in ‘Development.Shake.Command’
instance CmdResult CmdLine
-- Defined in ‘Development.Shake.Command’
...plus 11 others

[Neil Mitchell]

Hi Ryan,

> Below is the result of using mapM_. I'm curious what the right way would be to force haskell
> to recognize the correct instance type?

The return type of cmd is polymorphic, and mapM_ doesn't specify it,
so you should do:

mapM_ (\x -> unit $ cmd ("curl

http://afltables.com/afl/stats/games/"++year++"/"++x) "-o" (out </>
xl)) (lines stdout)

The unit function does nothing other than forcing the type result. See
the comment about "not using the result" in
https://hackage.haskell.org/package/shake-0.15.6/docs/Development-Shake.html#v:cmd.

> My current version of shake in stack is 0.15.5, so I'll try upgrading by making a yaml config file.

I wouldn't worry overly if it will be annoying. Hopefully it should
arrive in Stack LTS shortly.

> I tried looking for simple grep functionality in Haskell but the solutions I saw were overly verbose, when
> I'm better at the language I'll try figure it out.

I'd aim for something not grep based but more like:

[x | "Match":"stats":x:_ <- lines xs, takeExtension x == ".html"]

> Lastly, if targets can only be files, but I want to be able to specify a directory to build, is the correct way
> to make it a phony command?

See http://shakebuild.com/faq#how-can-i-depend-on-directories

Think of directories as containers for files. They exist or don't
pretty randomly, but if they have files, they must exist. In
particular, you can't depend on a directory with need or write a rule
to create a directory. Directories are created as needed - the rule
for bar/baz.exe will create the bar directory if necessary.

Thanks, Neil