Top-down code in namspaces

[Glen Mailer]

Hi everyone, I'm looking to get some opinions on code style.

Specifically, I like to write my code in a top-down.

What I mean by that is that within a file the highest-level functions sit at the top, and are implemented in terms of lower-level functions further down.

The idea is that through sensible naming, a reader should be able to stop reading at any point and still know what's going on.

I was recently watching the prismatic schema presentation from the 2013 conj, and noticed they too promoted having a "public" section at the top of the namespace.


The problem now is because of the single-pass nature of clojure's evaluation - simply writing code like this doesn't actually work.

There's a few approaches i've seen from reading other's code:

1. Have an "impl" namespace which contains all of the helper functions for the public "interface"
2. Make use of (declare) forms where necessary to improve forward readability
3. Don't worry about source order, just have the api functions further down the file and live with it

Some other options I considered include making really heavy use of (declare), or even defining some sort of (eval-reversed) macro which runs the code backwards.


I'd like to know if people are experiencing this issue, and how you all are resolving it?


Cheers
Glen

[Luc Prefontaine]

a) move out helpers or core code
out of the API name space
b) tag stuff left not part of the API
as private in the API name space
c) keep the API at the bottom
d) do the above iteratively as code
evolves

So far it's been workable ( > 20000
locs so far). Not too much name
space switches while improving
or fixing code.

Luc P

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Luc Prefontaine<lprefo...@softaddicts.ca> sent by ibisMail!

[u1204]

>Specifically, I like to write my code in a top-down.
>
>What I mean by that is that within a file the highest-level functions sit
>at the top, and are implemented in terms of lower-level functions further
>down.
>
>The idea is that through sensible naming, a reader should be able to stop
>reading at any point and still know what's going on.

In a functional programming language each function ought to be
independent and insensitive to order. Unfortunately, Clojure is
sensitive to order, as you can see.

Your top-down style of programming is really useful. You might
consider using a tangle function (It is only a few lines of code)
to extract the code in the order the compiler likes. This would
allow you to continue to write in the order humans like.

Instead of calling load to read the file, call your tangle function.

Tim Daly

[Sean Corfield]

On Jun 1, 2014, at 11:53 PM, u1204 <da...@axiom-developer.org> wrote:
> Instead of calling load to read the file, call your tangle function.

Whilst that might work from the REPL, it's not going to work with normal Clojure tooling and it would mean you couldn't just :require files written that way in the file's ns either.

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Phillip Lord]

Sean Corfield <se...@corfield.org> writes:
> On Jun 1, 2014, at 11:53 PM, u1204 <da...@axiom-developer.org> wrote:
>> Instead of calling load to read the file, call your tangle function.
>
> Whilst that might work from the REPL, it's not going to work with normal
> Clojure tooling and it would mean you couldn't just :require files written
> that way in the file's ns either.

Yeah, it would work for require, you just untangle as you go. Whether it
is worth the effort or not is a different question; it would break a lot
of other tools. My suspicision is that Tim is not a heavy tool user, but
I may be wrong about that.

Basically, I think the OP is stuck until/unless Clojure fixes it's
current (and in my belief broken) behaviour. If you want top-down
Clojure code, just start reading from the bottom.

Phil

[Gregg Reynolds]

On Sun, Jun 1, 2014 at 9:36 AM, Glen Mailer <glen...@gmail.com> wrote:

Hi everyone, I'm looking to get some opinions on code style.

Specifically, I like to write my code in a top-down.

What I mean by that is that within a file the highest-level functions sit at the top, and are implemented in terms of lower-level functions further down.

... 

The problem now is because of the single-pass nature of clojure's evaluation - simply writing code like this doesn't actually work.

There's a few approaches i've seen from reading other's code:

1. Have an "impl" namespace which contains all of the helper functions for the public "interface"
2. Make use of (declare) forms where necessary to improve forward readability
3. Don't worry about source order, just have the api functions further down the file and live with it

4.  Put your helper funcs ("defn-" stuff) in helpers.clj, without a call to ns at the top, then (load "helpers") at the top of the file that uses them.  You still get the effect you're looking for, with a one line "preface" that tells the reader where to look for more info.  Seems to work in a little test app (lein new app topdown):

;; in topdown/core.clj:

(ns topdown.core

  (:gen-class))

(load "helpers")

(defn -main

  "I don't do a whole lot, but I do call an internal function that lives in another source file."

  [& args]

  (hello))

;; in topdown/helpers.clj:

;; internal topdown helper fns

(defn- hello [] (println "Hello"))

HTH

Gregg

[Timothy Baldridge]

Another way of looking at Clojure code is not a "top down abstraction first" view, but as the building of a system from smaller parts. The best example of this is clojure/core.clj . Sure it's bootstrap code so it can be a bit verbose at times, but you start with nothing and end up with a complete system. Now picture this in a "top down" style...it'd be insanely complex. Even a "whole program compilation unit" compiler would have problems compiling this code, as some macros like let and loop are redefined in in the same file, and some macros are needed before those more complex forms can even be processed.

Most developers write Clojure code from the REPL in this case, it's pretty simple to compile files exactly the same way. Just as a programmer at the REPL would enter forms one at a time, So the compiler compiles in exactly the same way. The last thing I would want is two forms of compilation, one at the REPL, one during :require. 

So just a few thoughts. IMO, the simplicity of Clojure's compilation model as it stands now is a major strength, not a "broken" hack. 

Timothy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Phillip Lord]

Gregg Reynolds <d...@mobileink.com> writes:

> 4. Put your helper funcs ("defn-" stuff) in helpers.clj, without a call to
> ns at the top, then (load "helpers") at the top of the file that uses them.
> You still get the effect you're looking for, with a one line "preface"
> that tells the reader where to look for more info. Seems to work in a
> little test app (lein new app topdown):

This is not side-effect free (sorry for pun). I did this for my code,
and now slamhound doesn't work on it. Other tools too? I don't know.


https://github.com/technomancy/slamhound/issues/61

[Gary Trakhman]

It takes a while (a couple months) to get used to reading things upside-down, but I wouldn't want to go back.  Knowing with certainty that some called method is defined above in the compilation strategy simplifies code-reading and comprehension by minimizing where you have to look, and it also makes it easier to move stuff around with a text editor than what I imagine the alternative would provide.  

I haven't used a late-binding lisp in a large code-base, so I'm contrasting against my experience with Java.

[Gregg Reynolds]

(load "foo") is legal Clojure; if a tool can't handle it, that's either a bug or a deliberate limitation in the tool.  In the case of slamhound, I guess it's a deliberate limitation.  On the other hand, when I followed the instructions on the slamhound page and ran it against my directory it overwrote helpers.clj from

(defn- hello [] (println "Hello "))

to:

(ns hello)

which looks like very rude behavior to me - I won't be using slamhound anytime soon.  So I wouldn't say that using "load" has side-effects that break tools, but that some tools may make assumptions and have side-effects that do nasty things to legal code (or "encourage" a particular style).  I would not be surprised if other tools make similar assumptions about the structure of clojure source files.  Then it's a question of whether one wants to allow the limitations of tools to constrain one's use of the language.

-Gregg

[Gregg Reynolds]

On Tue, Jun 3, 2014 at 8:32 AM, Gregg Reynolds <d...@mobileink.com> wrote:

On Sun, Jun 1, 2014 at 9:36 AM, Glen Mailer <glen...@gmail.com> wrote:

Hi everyone, I'm looking to get some opinions on code style.

... 

4.  Put your helper funcs ("defn-" stuff) in helpers.clj, without a call to ns at the top, then (load "helpers") at the top of the file that uses them.  You still get the effect you're looking for, with a one line "preface" that tells the reader where to look for more info.  Seems to work in a little test app (lein new app topdown):

;; in topdown/core.clj:

(ns topdown.core

  (:gen-class))

(load "helpers")

(defn -main

  "I don't do a whole lot, but I do call an internal function that lives in another source file."

  [& args]

  (hello))

;; in topdown/helpers.clj:

;; internal topdown helper fns

(defn- hello [] (println "Hello"))

PS.  You can also do:

;; in topdown/core.clj

(ns topdown.core

  (:load "helpers"))

... etc...

;; in helpers.clj

(ns topdown.core)

[Luc Prefontaine]

Yeah, it's certainly hard, just tried it,
blood pressure increases in the head
and my eyes were bulging out of their
sockets.

A bit easier using these chairs
that allow you to flip upside
down, less strain on the neck and
no need to keep up your
balance every second or so.


My apology to the readers of the
above lines, I could not resist :)))


Luc P.

[Gary Trakhman]

I just turn the monitor upside-down ;-).

[Softaddicts]

Does not work on my iPad, I forget to lock the display every time.

Damn it...

Luc P.

Softaddicts<lprefo...@softaddicts.ca> sent by ibisMail from my ipad!

[Colin Fleming]

(load "foo") is legal Clojure; if a tool can't handle it, that's either a bug or a deliberate limitation in the tool.

This is not true. Cursive, for example, indexes Clojure projects in order to perform its magic. In IntelliJ, index data for a file is only allowed to depend on the contents of that file, not any other. This is a very common restriction in indexing systems, since otherwise you need to maintain dependency information with your index items and invalidate them when one of potentially many files is touched. Splitting a namespace across multiple files using load-file is impossible to handle correctly with this restriction. Cursive contains a metric truckload of heuristics to try to do intelligent things in this case (and generally does ok at it) since clojure.core does this, but I would definitely discourage anyone interested in tool support from doing this.

--

[Phillip Lord]

Gary Trakhman <gary.t...@gmail.com> writes:
> Knowing with certainty that some called method is defined above in the
> compilation strategy simplifies code-reading and comprehension by
> minimizing where you have to look, and it also makes it easier to move
> stuff around with a text editor than what I imagine the alternative
> would provide.

I never really care about backward or forward when jumping about code.
The editors take care of it for me.

> I haven't used a late-binding lisp in a large code-base, so I'm contrasting
> against my experience with Java.

It's a total pain in the ass. You write some lisp, start to debug it
carefully, and half way through hit a mispelt symbol.

(defun thing[fred x]
(funcall frd x))


In Clojure, the closest equivalent (hitting a symbol you have declared,
but not defined) is rare:

(declare frd)
(defun thing[fred x]
(frd x))


Although, pre-declaration doesn't prevent a related bug which is using a
symbol that you *think* is local but is actually global as in:

(defn blah [mp]
(do-stuff-to map))


Having said this this, having to declare things first and being forced
to write bottom up is also a pain. Win some, lose some.

Phil

[Phillip Lord]

Gregg Reynolds <d...@mobileink.com> writes:

>> This is not side-effect free (sorry for pun). I did this for my code,
>> and now slamhound doesn't work on it. Other tools too? I don't know.
>>
>> https://github.com/technomancy/slamhound/issues/61
>
>
> (load "foo") is legal Clojure; if a tool can't handle it, that's either a
> bug or a deliberate limitation in the tool.

If you read the bug report, you will see that I agree. The slamhound
developers don't and argue that (load "foo") is bad style. Who is right
is not so relevant; that there is such an argument is.

> So I wouldn't say that using "load" has side-effects that break tools,
> but that some tools may make assumptions and have side-effects that do
> nasty things to legal code (or "encourage" a particular style). I
> would not be surprised if other tools make similar assumptions about
> the structure of clojure source files. Then it's a question of whether
> one wants to allow the limitations of tools to constrain one's use of
> the language.

Or whether one's use of the language limits the ability to use tools. I
just make the statement that it good to know about these choices rather
than to make them by chance.

Phil

[Mars0i]

On Sunday, June 1, 2014 9:36:55 AM UTC-5, Glen Mailer wrote:

Hi everyone, I'm looking to get some opinions on code style.

Specifically, I like to write my code in a top-down.

What I mean by that is that within a file the highest-level functions sit at the top, and are implemented in terms of lower-level functions further down.

...

There's a few approaches i've seen from reading other's code:

...
Some other options I considered include making really heavy use of (declare), ...

I'm puzzled by the complexity of the solutions proposed.  (Although I like Luc's, since my computer doesn't have an auto-rotate function.)  I don't see this as a big deal at all. I just put a big declare statement at the top of the file.

More specifically, when initially coding related functions, I'll put the dependent ones below the ones they call.  After that chunk of code is done, I might move the higher-level functions up in the file.  Then I add the new functions to the declare statement by hand, or I periodically do something like:

grep defn mysourcefile.clj >> mysourcefile.clj
(Be careful to use two ">"s!)

and then I edit the junk at the end of the file into a declare statement at the top of the file.  And maybe if f I were ... lazier, I'd code a script that would update the declare in one pass.

That works for defn and defn-, and you might want to shorten the search term to 'def'.  It's more difficult if you use Clojure protocols.

[Mars0i]

On Wednesday, June 4, 2014 9:42:41 AM UTC-5, Mars0i wrote:

... Then I add the new functions to the declare statement by hand, or I periodically do something like:

grep defn mysourcefile.clj >> mysourcefile.clj
(Be careful to use two ">"s!)

and then I edit the junk at the end of the file into a declare statement at the top of the file.  And maybe if f I were ... lazier, I'd code a script that would update the declare in one pass.

OK, I couldn't resist my own implicit challenge.

#!/bin/sh
sourcefile="$1"
newsourcefile="new.$sourcefile"

newdeclare=$(echo '(declare' \
    `sed -n '/defn/s/(defn-* //p' "$sourcefile" | tr '\n' ' '` ')' \
    | sed 's/ )/)/')

sed "s/(declare .*/$newdeclare/" "$sourcefile" > "$newsourcefile"

This writes a new version of the file named new.<oldfilename>. Or if you either trust your script or trust your backups, and are on a system that includes the mighty ed editor, you can replace the last line with:

echo "1,\$s/(declare .*/$newdeclare/\nw\n" | ed "$sourcefile"

which edits the file in place, assuming that the previous version of the declaration was on one line.  You may want to use a different scriptable editor.

The messy part is the sed and tr line:

    `sed -n '/defn/s/(defn-* //p' "$sourcefile" | tr '\n' ' '`

The sed part finds all of the lines with "defn" in them, then substitutes the empty string for "(defn" or "(defn-".   'tr' then removes the newlines between the function names, replacing the newlines with spaces.  You'll need something a little more complicated if you put the parameter vector or anything else on the same line as the function name.  The 'echo' on the previous line, along with the final ')' adds "(declare" and its closing parenthesis.  Those two lines can be used by themselves to generate a declare statement from the command line. The 'sed' command after these lines isn't necessary; it just removes an unnecessary space before the closing parenthesis.

Obviously, there will be source files on which this won't work.  It's not worth making it foolproof.

It's a certainty that others would code this more elegantly or more succinctly.  It could be written in Clojure, obviously, but still wouldn't be foolproof unless someone hacks it from the Clojure parser.

[Reid McKenzie]

Clearly the solution is to use tools.analyzer and write a big def emitter

/s
Reid

[Colin Fleming]

I actually have an open issue for Cursive to do this automatically: #200. I'm starting to think a namespace sorter that automatically manages the declares might not be such a crazy idea.

[Phillip Lord]

Then integrate the whole lot into the Clojure compiler pipeline so that
it just works in the first place.

Reid McKenzie <rmcke...@gmail.com> writes:

> Clearly the solution is to use tools.analyzer and write a big def emitter
>
> /s
> Reid
> On 06/04/2014 10:27 AM, Mars0i wrote:
>>
>>
>> On Wednesday, June 4, 2014 9:42:41 AM UTC-5, Mars0i wrote:
>>

>> ... Then I add the new functions to the declarestatement by hand,

>> or I periodically do something like:
>>
>> grep defn mysourcefile.clj >> mysourcefile.clj
>> (Be careful to use two ">"s!)
>>
>> and then I edit the junk at the end of the file into a

>> declarestatement at the top of the file. And maybe if f I were
>> ... lazier, I'd code a script that would update the declarein one

>> pass.
>>
>>
>> OK, I couldn't resist my own implicit challenge.
>>
>> #!/bin/sh
>> sourcefile="$1"
>> newsourcefile="new.$sourcefile"
>>
>> newdeclare=$(echo '(declare' \
>> `sed -n '/defn/s/(defn-* //p' "$sourcefile" | tr '\n' ' '` ')' \
>> | sed 's/ )/)/')
>>
>> sed "s/(declare .*/$newdeclare/" "$sourcefile" > "$newsourcefile"
>>
>> This writes a new version of the file named new.<oldfilename>. Or if
>> you either trust your script or trust your backups, and are on a
>> system that includes the mighty ed

>> <http://www.gnu.org/fun/jokes/ed.msg.html> editor, you can replace the

>> <mailto:clojure+u...@googlegroups.com>.

>> For more options, visit https://groups.google.com/d/optout.

--
Phillip Lord, Phone: +44 (0) 191 222 7827
Lecturer in Bioinformatics, Email: philli...@newcastle.ac.uk
School of Computing Science, http://homepages.cs.ncl.ac.uk/phillip.lord
Room 914 Claremont Tower, skype: russet_apples
Newcastle University, twitter: phillord
NE1 7RU

[Luc Prefontaine]

I maintain that the average human
being looking at sed commands
would rather end up standing on his
head for a significant amount of
time to avoid it :)))

BTWY, I have been scripting under u*x
for a few decades by now.
I resort to it when nothing and I
mean nothing (think about Daffy Duck's
voice here) else can do it :))

Luc P.


Luc P.

>
>
> On Wednesday, June 4, 2014 9:42:41 AM UTC-5, Mars0i wrote:
> >
> > ... Then I add the new functions to the declare statement by hand, or I
> > periodically do something like:
> >
> > grep defn mysourcefile.clj >> mysourcefile.clj
> > (Be careful to use two ">"s!)
> >
> > and then I edit the junk at the end of the file into a declare statement
> > at the top of the file. And maybe if f I were ... lazier, I'd code a
> > script that would update the declare in one pass.
> >
>
> OK, I couldn't resist my own implicit challenge.
>
> #!/bin/sh
> sourcefile="$1"
> newsourcefile="new.$sourcefile"
>
> newdeclare=$(echo '(declare' \
> `sed -n '/defn/s/(defn-* //p' "$sourcefile" | tr '\n' ' '` ')' \
> | sed 's/ )/)/')
>
> sed "s/(declare .*/$newdeclare/" "$sourcefile" > "$newsourcefile"
>
> This writes a new version of the file named new.<oldfilename>. Or if you
> either trust your script or trust your backups, and are on a system that

> includes the mighty ed <http://www.gnu.org/fun/jokes/ed.msg.html> editor,

> you can replace the last line with:
>
> echo "1,\$s/(declare .*/$newdeclare/\nw\n" | ed "$sourcefile"
>
> which edits the file in place, assuming that the previous version of the
> declaration was on one line. You may want to use a different scriptable
> editor.
>
> The messy part is the sed and tr line:
>
> `sed -n '/defn/s/(defn-* //p' "$sourcefile" | tr '\n' ' '`
>
> The sed part finds all of the lines with "defn" in them, then substitutes
> the empty string for "(defn" or "(defn-". 'tr' then removes the newlines
> between the function names, replacing the newlines with spaces. You'll
> need something a little more complicated if you put the parameter vector or
> anything else on the same line as the function name. The 'echo' on the
> previous line, along with the final ')' adds "(declare" and its closing
> parenthesis. Those two lines can be used by themselves to generate a
> declare statement from the command line. The 'sed' command after these
> lines isn't necessary; it just removes an unnecessary space before the
> closing parenthesis.
>
> Obviously, there will be source files on which this won't work. It's not
> worth making it foolproof.
>
> It's a certainty that others would code this more elegantly or more
> succinctly. It could be written in Clojure, obviously, but still wouldn't
> be foolproof unless someone hacks it from the Clojure parser.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
--

[u1204]

> Hi everyone, I'm looking to get some opinions on code style.
>
> Specifically, I like to write my code in a top-down.
>
> What I mean by that is that within a file the highest-level functions sit
> at the top, and are implemented in terms of lower-level functions further
> down.

You could write a "(defn-defer ..." macro that asserts the signature
and caches the code body. The final function in your file would be
(do-defer ...) which does the actual (defn ... forms.

Tim Daly