I've recently been working on a game development project in Clojure
which is now starting to bear fruit. I thought people here might be
interested, and that it would be worthwhile to share some experiences
and perspectives.
The project is a steampunk-themed strategy game, and a playable in-
development version is available here:
Overall, I think Clojure is a fantastic language for game development.
Some thought on what makes it really compelling:
1. Immutable data structures work surprisingly well for games. I
represent the entire game state with a big defrecord, and the main
engine loop is basically:
a) Get commands from player / AI (e.g. "Move unit from A to B")
b) Convert each command into set of atomic updates ("Remove unit
from A", "Add unit to B")
c) Apply updates sequentially to the game state
d) Trigger any feedback to player e.g. animations, sounds
2. Concurrency support has been very helpful. With immutable game
state, it has been trivial to separate the game engine from the
renderer from the AI calculations - each effectively gets to operate
on its own "snapshot" of the entire game state. This can be a big
headache in many game engines that have to mediate access to a single,
mutable game state. I can also see even greater benefits when I
finally start adding some multi-player features.
3. High level, functional programming works great for rapid, iterative
and dynamic development. Of all the languages I've used, Clojure has
probably been quickest in terms of time required to add a given new
piece of functionality. For example, the map generator code probably
only took about 30 minutes to write, pretty good for a fractal
landscape generation system!
4. The Java interoperability support is fantastic - definitely counts
as one of the "killer features" of Clojure. I've been able to reuse a
lot of Java code libraries, both my own and standard libraries such as
Swing. As an example, I have a well optimised immutable persistent 2D
grid data structure in Java (basically a very efficient, spatially
partitioned map from (int,int) to Object) that I have been able to use
pretty much seamlessly in Clojure thanks to extend-protocol and
similar.
Some interesting observations / discoveries / surprises in the
process:
1. The game worked first time on Mac and Linux, despite having been
tested exclusively on Windows. Great kudos to to both the Java
platform and the Clojure libraries!
2. Reflection is *really slow*. After getting 20-100x speedups from
eliminating reflection warnings in the AI code, I now treat any
reflection warning in code other than one-time setup code as a bug to
be fixed immediately.
Finally, here are some things I think could be great improvements to
Clojure in the future from a game developer's perspective:
1. Better support for primitive parameters / return values / unchecked
arithmetic - in performance sensitive code, these are pretty much
essential. In a few cases in the graphics and AI code, I've had to
drop back to Java to get the performance I need.
2. It would be great to reduce the amount of memory allocations. Yes,
I know memory is plentiful and GC is very cheap, but it's still not as
cheap as stack allocation and any noticeable GC pauses are not good
for the player experience in interactive games. For this reason, I
find myself using reduce and indexed loops a lot more than I guess
would normally be idiomatic, and conversely tend to avoid some of the
lazy constructs and functions that generate sequences. While Clojure
is great for a strategy game, I'd probably hesitate to use it for a
real-time 3D game.
3. I think it would be great to have better support for circular
references - perhaps a two-pass compile? The reason this is
particularly acute in game development is that different subsystems
have quite a lot of inter-dependencies. AI evaluation system needs to
understand game state/engine so it can evaluate a position. Game state/
engine needs to understand units so it can manipulate them. Units need
to understand AI evaluation system so they can decide which actions to
take...... obviously it's possible to work around all this, but it's a
major pain, adds complexity and means that you need to structure code
to manage dependencies rather than in logical modules (which would be
easier to manage and maintain!)
Would love to hear thoughts, and particularly any other experiences
people have had in using Clojure for game development!
On Aug 13, 9:51 am, Mike Anderson <mike.r.anderson...@gmail.com>
wrote:
> 2. It would be great to reduce the amount of memory allocations. Yes,
> I know memory is plentiful and GC is very cheap, but it's still not as
> cheap as stack allocation and any noticeable GC pauses are not good
> for the player experience in interactive games. For this reason, I
> find myself using reduce and indexed loops a lot more than I guess
> would normally be idiomatic, and conversely tend to avoid some of the
> lazy constructs and functions that generate sequences. While Clojure
> is great for a strategy game, I'd probably hesitate to use it for a
> real-time 3D game.
There really is no way around this. We preallocate space for all our
OpenGL vertex, as well as all the FSM for the AI engine. This is
something we do even in our C++ code. We avoid creating objects
whenever possible, and preallocate at init time. This is especially
true in devices with smaller footprint like console/iphone/android,
etc.
Having said that, for prototyping OpenGL realtime 3d stuff on our
desktop,
clojure has been ok.
> 3. I think it would be great to have better support for circular
> references - perhaps a two-pass compile? The reason this is
> particularly acute in game development is that different subsystems
> have quite a lot of inter-dependencies. AI evaluation system needs to
> understand game state/engine so it can evaluate a position. Game state/
> engine needs to understand units so it can manipulate them. Units need
> to understand AI evaluation system so they can decide which actions to
> take...... obviously it's possible to work around all this, but it's a
> major pain, adds complexity and means that you need to structure code
> to manage dependencies rather than in logical modules (which would be
> easier to manage and maintain!)
I'm not sure what you mean by this, can you expand on this?
On Aug 13, 6:51 am, Mike Anderson <mike.r.anderson...@gmail.com>
wrote:
> Hello all,
> I've recently been working on a game development project in Clojure
> which is now starting to bear fruit. I thought people here might be
> interested, and that it would be worthwhile to share some experiences
> and perspectives.
Looks great. Thanks for sharing your experiences.
Do you plan to share the source code? Any reason you went with Swing
instead of OpenGL?
I assumed he didn't use OpenGL because it's a 2d tile game? Using OpenGL for 2d or 2.5d (isometric) is really only a good idea if you can assume the target has hardware OpenGL acceleration.
Even then you may not want to do that, due to battery concerns.
On Fri, Aug 13, 2010 at 2:06 PM, Brian Carper <briancar...@gmail.com> wrote: > Do you plan to share the source code? Any reason you went with Swing > instead of OpenGL?
<mike.r.anderson...@gmail.com> wrote: > 2. It would be great to reduce the amount of memory allocations. Yes, > I know memory is plentiful and GC is very cheap, but it's still not as > cheap as stack allocation and any noticeable GC pauses are not good > for the player experience in interactive games. For this reason, I > find myself using reduce and indexed loops a lot more than I guess > would normally be idiomatic, and conversely tend to avoid some of the > lazy constructs and functions that generate sequences. While Clojure > is great for a strategy game, I'd probably hesitate to use it for a > real-time 3D game.
This can be made a bit better by turning Escape Analysis on? Have you tried that?
The G1 collector is supposed to have lower latency. Have you tried it?
(GC choice/tuning is a loooong story, but some select suggestions follow)
First off, make sure you're at least using CMS (-XX:+UseConcMarkSweepGC) or G1 at all to begin with - the default collector is the throughput collector which will always do full GC:s of the old generation so that you are guaranteed to get pause times that scale linearly with heap size (unless you're generating *zero* non-young garbage).
With CMS, you can force shorter young generation pauses by limiting the size of the young generation (usually at the cost of more GC work in total; i.e., less throughput, because the older generation is more expensive to collect). I'm not sure at all how much control over VM options you have with Java WebStart though, but in general that would be my suggestion if the young generation pauses are too long. -XX:NewSize=SIZE and -XX:MaxNewSize=SIZE should be your friend here (and -XX:+UseParNewGC though I think that is always the default on modern VM:s anyway on multi-core machines). I'm making the assumption here that the allocation pattern is not such that the concurrent mark/sweep:s are the problem.
With G1, you can directly specify target pause times that G1 tends to be pretty good at maintaining in my experience. A recurring issue for me and at least some others seems to be that your application may have a workload that disallows certain memory regions to be collected within the pause time goal; this can lead to a build-up of uncollectable regions that may eventually cause a fallback to a full stop-the-world GC. The likelyhood of this happening is higher the lower the pause time goal you specify. An starter command line to play with (preferably with jdk 1.7) might be:
Funny you should mention this - I was about to post a question about
my own game when I saw your article. My issue is, I assume someone has
written minimax and/or alpha-beta pruning in Clojure (or a java
library that's easy to interop with). My case is slightly different in
that the turn order is not fixes - sometimes player X goes twice in a
row - but it ought to be pretty simple to plug into a standard AI
library. Does anyone know where I can find such a thing?
On Aug 13, 6:51 am, Mike Anderson <mike.r.anderson...@gmail.com>
wrote:
> I've recently been working on a game development project in Clojure
> which is now starting to bear fruit. I thought people here might be
> interested, and that it would be worthwhile to share some experiences
> and perspectives.
> The project is a steampunk-themed strategy game, and a playable in-
> development version is available here:
> Overall, I think Clojure is a fantastic language for game development.
> Some thought on what makes it really compelling:
> 1. Immutable data structures work surprisingly well for games. I
> represent the entire game state with a big defrecord, and the main
> engine loop is basically:
> a) Get commands from player / AI (e.g. "Move unit from A to B")
> b) Convert each command into set of atomic updates ("Remove unit
> from A", "Add unit to B")
> c) Apply updates sequentially to the game state
> d) Trigger any feedback to player e.g. animations, sounds
> 2. Concurrency support has been very helpful. With immutable game
> state, it has been trivial to separate the game engine from the
> renderer from the AI calculations - each effectively gets to operate
> on its own "snapshot" of the entire game state. This can be a big
> headache in many game engines that have to mediate access to a single,
> mutable game state. I can also see even greater benefits when I
> finally start adding some multi-player features.
> 3. High level, functional programming works great for rapid, iterative
> and dynamic development. Of all the languages I've used, Clojure has
> probably been quickest in terms of time required to add a given new
> piece of functionality. For example, the map generator code probably
> only took about 30 minutes to write, pretty good for a fractal
> landscape generation system!
> 4. The Java interoperability support is fantastic - definitely counts
> as one of the "killer features" of Clojure. I've been able to reuse a
> lot of Java code libraries, both my own and standard libraries such as
> Swing. As an example, I have a well optimised immutable persistent 2D
> grid data structure in Java (basically a very efficient, spatially
> partitioned map from (int,int) to Object) that I have been able to use
> pretty much seamlessly in Clojure thanks to extend-protocol and
> similar.
> Some interesting observations / discoveries / surprises in the
> process:
> 1. The game worked first time on Mac and Linux, despite having been
> tested exclusively on Windows. Great kudos to to both the Java
> platform and the Clojure libraries!
> 2. Reflection is *really slow*. After getting 20-100x speedups from
> eliminating reflection warnings in the AI code, I now treat any
> reflection warning in code other than one-time setup code as a bug to
> be fixed immediately.
> Finally, here are some things I think could be great improvements to
> Clojure in the future from a game developer's perspective:
> 1. Better support for primitive parameters / return values / unchecked
> arithmetic - in performance sensitive code, these are pretty much
> essential. In a few cases in the graphics and AI code, I've had to
> drop back to Java to get the performance I need.
> 2. It would be great to reduce the amount of memory allocations. Yes,
> I know memory is plentiful and GC is very cheap, but it's still not as
> cheap as stack allocation and any noticeable GC pauses are not good
> for the player experience in interactive games. For this reason, I
> find myself using reduce and indexed loops a lot more than I guess
> would normally be idiomatic, and conversely tend to avoid some of the
> lazy constructs and functions that generate sequences. While Clojure
> is great for a strategy game, I'd probably hesitate to use it for a
> real-time 3D game.
> 3. I think it would be great to have better support for circular
> references - perhaps a two-pass compile? The reason this is
> particularly acute in game development is that different subsystems
> have quite a lot of inter-dependencies. AI evaluation system needs to
> understand game state/engine so it can evaluate a position. Game state/
> engine needs to understand units so it can manipulate them. Units need
> to understand AI evaluation system so they can decide which actions to
> take...... obviously it's possible to work around all this, but it's a
> major pain, adds complexity and means that you need to structure code
> to manage dependencies rather than in logical modules (which would be
> easier to manage and maintain!)
> Would love to hear thoughts, and particularly any other experiences
> people have had in using Clojure for game development!
Paradigms of artificial intelligence programming: case studies in Common LISP By Peter Norvig has a full chapter on this (ch18), complete with code in Common LISP.
his "Artificial Intelligence: A Modern Approach" is also a good book if you are interested in game AI. 3rd edition just came out recently
On Fri, Aug 13, 2010 at 12:33 PM, Alan <a...@malloys.org> wrote: > Funny you should mention this - I was about to post a question about > my own game when I saw your article. My issue is, I assume someone has > written minimax and/or alpha-beta pruning in Clojure (or a java > library that's easy to interop with). My case is slightly different in > that the turn order is not fixes - sometimes player X goes twice in a > row - but it ought to be pretty simple to plug into a standard AI > library. Does anyone know where I can find such a thing?
> On Aug 13, 6:51 am, Mike Anderson <mike.r.anderson...@gmail.com> > wrote: >> Hello all,
>> I've recently been working on a game development project in Clojure >> which is now starting to bear fruit. I thought people here might be >> interested, and that it would be worthwhile to share some experiences >> and perspectives.
>> The project is a steampunk-themed strategy game, and a playable in- >> development version is available here:
>> Overall, I think Clojure is a fantastic language for game development. >> Some thought on what makes it really compelling:
>> 1. Immutable data structures work surprisingly well for games. I >> represent the entire game state with a big defrecord, and the main >> engine loop is basically: >> a) Get commands from player / AI (e.g. "Move unit from A to B") >> b) Convert each command into set of atomic updates ("Remove unit >> from A", "Add unit to B") >> c) Apply updates sequentially to the game state >> d) Trigger any feedback to player e.g. animations, sounds
>> 2. Concurrency support has been very helpful. With immutable game >> state, it has been trivial to separate the game engine from the >> renderer from the AI calculations - each effectively gets to operate >> on its own "snapshot" of the entire game state. This can be a big >> headache in many game engines that have to mediate access to a single, >> mutable game state. I can also see even greater benefits when I >> finally start adding some multi-player features.
>> 3. High level, functional programming works great for rapid, iterative >> and dynamic development. Of all the languages I've used, Clojure has >> probably been quickest in terms of time required to add a given new >> piece of functionality. For example, the map generator code probably >> only took about 30 minutes to write, pretty good for a fractal >> landscape generation system!
>> 4. The Java interoperability support is fantastic - definitely counts >> as one of the "killer features" of Clojure. I've been able to reuse a >> lot of Java code libraries, both my own and standard libraries such as >> Swing. As an example, I have a well optimised immutable persistent 2D >> grid data structure in Java (basically a very efficient, spatially >> partitioned map from (int,int) to Object) that I have been able to use >> pretty much seamlessly in Clojure thanks to extend-protocol and >> similar.
>> Some interesting observations / discoveries / surprises in the >> process:
>> 1. The game worked first time on Mac and Linux, despite having been >> tested exclusively on Windows. Great kudos to to both the Java >> platform and the Clojure libraries!
>> 2. Reflection is *really slow*. After getting 20-100x speedups from >> eliminating reflection warnings in the AI code, I now treat any >> reflection warning in code other than one-time setup code as a bug to >> be fixed immediately.
>> Finally, here are some things I think could be great improvements to >> Clojure in the future from a game developer's perspective:
>> 1. Better support for primitive parameters / return values / unchecked >> arithmetic - in performance sensitive code, these are pretty much >> essential. In a few cases in the graphics and AI code, I've had to >> drop back to Java to get the performance I need.
>> 2. It would be great to reduce the amount of memory allocations. Yes, >> I know memory is plentiful and GC is very cheap, but it's still not as >> cheap as stack allocation and any noticeable GC pauses are not good >> for the player experience in interactive games. For this reason, I >> find myself using reduce and indexed loops a lot more than I guess >> would normally be idiomatic, and conversely tend to avoid some of the >> lazy constructs and functions that generate sequences. While Clojure >> is great for a strategy game, I'd probably hesitate to use it for a >> real-time 3D game.
>> 3. I think it would be great to have better support for circular >> references - perhaps a two-pass compile? The reason this is >> particularly acute in game development is that different subsystems >> have quite a lot of inter-dependencies. AI evaluation system needs to >> understand game state/engine so it can evaluate a position. Game state/ >> engine needs to understand units so it can manipulate them. Units need >> to understand AI evaluation system so they can decide which actions to >> take...... obviously it's possible to work around all this, but it's a >> major pain, adds complexity and means that you need to structure code >> to manage dependencies rather than in logical modules (which would be >> easier to manage and maintain!)
>> Would love to hear thoughts, and particularly any other experiences >> people have had in using Clojure for game development!
>> Mike.
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
On Aug 13, 11:09 am, Wilson MacGyver <wmacgy...@gmail.com> wrote:
> I assumed he didn't use OpenGL because it's a 2d tile game?
> Using OpenGL for 2d or 2.5d (isometric) is really only a good idea
> if you can assume the target has hardware OpenGL acceleration.
> Even then you may not want to do that, due to battery concerns.
But there are some good OpenGL 2D game engines for Java, e.g.
Slick[1]. There's also Penumbra[2] which nicely wraps LWJGL for
Clojure. I ask this mostly because I'm making my own 2D game and
somewhat torn between Java2D and OpenGL. Penumbra is pretty solid in
my experience.
On Fri, Aug 13, 2010 at 4:51 PM, Brian Carper <briancar...@gmail.com> wrote: > But there are some good OpenGL 2D game engines for Java, e.g. > Slick[1]. There's also Penumbra[2] which nicely wraps LWJGL for > Clojure. I ask this mostly because I'm making my own 2D game and > somewhat torn between Java2D and OpenGL. Penumbra is pretty solid in > my experience.
We don't use Penumbra. On Java, we use JOGL, and on other targets we use pure OpenGL or OpenGL ES API for both compatibility and performance reasons. We have to ensure whatever OpenGL stuff we do, will work everywhere.
> What do you mean by "battery concerns"?
Laptops, Mobile devices and such. If you use OpenGL on a laptop, it will use the GPU assuming it's hardware accelerated. GPU is a big battery drain. On mobile devices such as iPhone and Android, it's even more so. Using OpenGL unnecessarily causes battery drain on your hardware.
Plus, if you use OpenGL, and your target is using those Intel Extreme Graphics, where the only thing it's Extreme at, is Extremely slow. :) You end up gain nothing but lose lots.
> I've recently been working on a game development project in Clojure
> which is now starting to bear fruit. I thought people here might be
> interested, and that it would be worthwhile to share some experiences
> and perspectives.
> The project is a steampunk-themed strategy game, and a playable in-
> development version is available here:
> Overall, I think Clojure is a fantastic language for game development.
> Some thought on what makes it really compelling:
> 1. Immutable data structures work surprisingly well for games. I
> represent the entire game state with a big defrecord, and the main
> engine loop is basically:
> a) Get commands from player / AI (e.g. "Move unit from A to B")
> b) Convert each command into set of atomic updates ("Remove unit
> from A", "Add unit to B")
> c) Apply updates sequentially to the game state
> d) Trigger any feedback to player e.g. animations, sounds
> 2. Concurrency support has been very helpful. With immutable game
> state, it has been trivial to separate the game engine from the
> renderer from the AI calculations - each effectively gets to operate
> on its own "snapshot" of the entire game state. This can be a big
> headache in many game engines that have to mediate access to a single,
> mutable game state. I can also see even greater benefits when I
> finally start adding some multi-player features.
> 3. High level, functional programming works great for rapid, iterative
> and dynamic development. Of all the languages I've used, Clojure has
> probably been quickest in terms of time required to add a given new
> piece of functionality. For example, the map generator code probably
> only took about 30 minutes to write, pretty good for a fractal
> landscape generation system!
> 4. The Java interoperability support is fantastic - definitely counts
> as one of the "killer features" of Clojure. I've been able to reuse a
> lot of Java code libraries, both my own and standard libraries such as
> Swing. As an example, I have a well optimised immutable persistent 2D
> grid data structure in Java (basically a very efficient, spatially
> partitioned map from (int,int) to Object) that I have been able to use
> pretty much seamlessly in Clojure thanks to extend-protocol and
> similar.
> Some interesting observations / discoveries / surprises in the
> process:
> 1. The game worked first time on Mac and Linux, despite having been
> tested exclusively on Windows. Great kudos to to both the Java
> platform and the Clojure libraries!
> 2. Reflection is *really slow*. After getting 20-100x speedups from
> eliminating reflection warnings in the AI code, I now treat any
> reflection warning in code other than one-time setup code as a bug to
> be fixed immediately.
> Finally, here are some things I think could be great improvements to
> Clojure in the future from a game developer's perspective:
> 1. Better support for primitive parameters / return values / unchecked
> arithmetic - in performance sensitive code, these are pretty much
> essential. In a few cases in the graphics and AI code, I've had to
> drop back to Java to get the performance I need.
> 2. It would be great to reduce the amount of memory allocations. Yes,
> I know memory is plentiful and GC is very cheap, but it's still not as
> cheap as stack allocation and any noticeable GC pauses are not good
> for the player experience in interactive games. For this reason, I
> find myself using reduce and indexed loops a lot more than I guess
> would normally be idiomatic, and conversely tend to avoid some of the
> lazy constructs and functions that generate sequences. While Clojure
> is great for a strategy game, I'd probably hesitate to use it for a
> real-time 3D game.
> 3. I think it would be great to have better support for circular
> references - perhaps a two-pass compile? The reason this is
> particularly acute in game development is that different subsystems
> have quite a lot of inter-dependencies. AI evaluation system needs to
> understand game state/engine so it can evaluate a position. Game state/
> engine needs to understand units so it can manipulate them. Units need
> to understand AI evaluation system so they can decide which actions to
> take...... obviously it's possible to work around all this, but it's a
> major pain, adds complexity and means that you need to structure code
> to manage dependencies rather than in logical modules (which would be
> easier to manage and maintain!)
> Would love to hear thoughts, and particularly any other experiences
> people have had in using Clojure for game development!
>> 3. I think it would be great to have better support for circular >> references - perhaps a two-pass compile? The reason this is >> particularly acute in game development is that different subsystems >> have quite a lot of inter-dependencies. AI evaluation system needs to >> understand game state/engine so it can evaluate a position. Game state/ >> engine needs to understand units so it can manipulate them. Units need >> to understand AI evaluation system so they can decide which actions to >> take...... obviously it's possible to work around all this, but it's a >> major pain, adds complexity and means that you need to structure code >> to manage dependencies rather than in logical modules (which would be >> easier to manage and maintain!)
> I'm not sure what you mean by this, can you expand on this?
Though I didn't write that paragraph, I have faced the same issue.
Suppose I have two functions in the same file, and one depends on the other:
This example is just a minor irritation, that I need to make sure all the functions in a file are in the right order.
A bigger problem is when the two functions are in different files/packages. Suppose I have files a.clj and b.clj and place some of my functions in each of those files based on some arbitrary categorization that makes intuitive sense to me. Then I realize that some of the functions in a.clj depend on some functions in b.clj, while some functions in b.clj depend on some functions in a.clj. Can I still use declare to resolve this circular dependency? Can I even "require" a.clj from b.clj when I have already "require"d b.clj from a.clj?
My solution so far has been to make sure that dependencies between packages are never two-way. I deliberately choose categorizations that won't result in circular dependencies between packages. I would much prefer to just refer to whatever function I wish, from whichever function I wish, and not need to think about which packages I am allowed to depend on or in what order functions must appear in a file.
On Fri, Aug 13, 2010 at 11:49 PM, Eric Lavigne <lavigne.e...@gmail.com> wrote: > Suppose I have two functions in the same file, and one depends on the other:
The (def g) in your example has the same effect as the (declare foo) in my example.
I discussed two problems. The first problem, which you addressed, was mostly just a warm-up for discussing a related problem that is more severe. Where can I put (def g) so that two files can "require" each other?
This is not a rare problem for me. Like Mike Anderson, I work around it by putting extra thought into which package-level dependencies I will allow, which sometimes necessitates creating more or fewer packages than I otherwise would have created.
> On Fri, Aug 13, 2010 at 11:49 PM, Eric Lavigne <lavigne.e...@gmail.com> wrote: >> Suppose I have two functions in the same file, and one depends on the other:
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
Suppose you make a file containing a (def foo) form for every defn in every file and then load that first? Does that solve the circular reference problem?
Eric Lavigne wrote: > The (def g) in your example has the same effect as the (declare foo) > in my example.
> I discussed two problems. The first problem, which you addressed, was > mostly just a warm-up for discussing a related problem that is more > severe. Where can I put (def g) so that two files can "require" each > other?
> This is not a rare problem for me. Like Mike Anderson, I work around > it by putting extra thought into which package-level dependencies I > will allow, which sometimes necessitates creating more or fewer > packages than I otherwise would have created.
> On Sat, Aug 14, 2010 at 12:13 AM, Wilson MacGyver <wmacgy...@gmail.com> wrote:
>> I rarely run into this. The few times I have, I just do
>> -- >> Omnem crede diem tibi diluxisse supremum.
>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscribe@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en
> I discussed two problems. The first problem, which you addressed, was > mostly just a warm-up for discussing a related problem that is more > severe. Where can I put (def g) so that two files can "require" each > other?
> This is not a rare problem for me. Like Mike Anderson, I work around > it by putting extra thought into which package-level dependencies I > will allow, which sometimes necessitates creating more or fewer > packages than I otherwise would have created.
On Fri, Aug 13, 2010 at 9:39 PM, Eric Lavigne <lavigne.e...@gmail.com> wrote: > This is not a rare problem for me. Like Mike Anderson, I work around > it by putting extra thought into which package-level dependencies I > will allow, which sometimes necessitates creating more or fewer > packages than I otherwise would have created.
Me three. I find it to be a *lot* of work to avoid circular dependencies.
<mark.engelb...@gmail.com> wrote: > On Fri, Aug 13, 2010 at 9:39 PM, Eric Lavigne <lavigne.e...@gmail.com> wrote: >> This is not a rare problem for me. Like Mike Anderson, I work around >> it by putting extra thought into which package-level dependencies I >> will allow, which sometimes necessitates creating more or fewer >> packages than I otherwise would have created.
> Me three. I find it to be a *lot* of work to avoid circular dependencies.
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
1. The "declare" macro may be handy for declaring multiple names at
once.
2. Maybe one could use the functions in clojure.repl or clojure-
contrib.ns-utils to write something that automatically forward
declares everything needed?
On Aug 13, 10:49 pm, Tim Daly <d...@axiom-developer.org> wrote:
> Suppose you make a file containing a
> (def foo)
> form for every defn in every file and then load that first?
> Does that solve the circular reference problem?
> Tim Daly
> Eric Lavigne wrote:
> > The (def g) in your example has the same effect as the (declare foo)
> > in my example.
> > I discussed two problems. The first problem, which you addressed, was
> > mostly just a warm-up for discussing a related problem that is more
> > severe. Where can I put (def g) so that two files can "require" each
> > other?
> > This is not a rare problem for me. Like Mike Anderson, I work around
> > it by putting extra thought into which package-level dependencies I
> > will allow, which sometimes necessitates creating more or fewer
> > packages than I otherwise would have created.
> > On Sat, Aug 14, 2010 at 12:13 AM, Wilson MacGyver <wmacgy...@gmail.com> wrote:
> >> I rarely run into this. The few times I have, I just do
> >> --
> >> Omnem crede diem tibi diluxisse supremum.
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Clojure" group.
> >> To post to this group, send email to clojure@googlegroups.com
> >> Note that posts from new members are moderated - please be patient with your first post.
> >> To unsubscribe from this group, send email to
> >> clojure+unsubscribe@googlegroups.com
> >> For more options, visit this group at
> >>http://groups.google.com/group/clojure?hl=en
One example is a contract programming job I did recently, writing software for identifying radioactive isotopes based on measurements of their emission spectra. In the end I had 7 Clojure source files (not including tests). I will show them in such an order that each file depends only on previous files.
I originally wanted to include the contents of the units file in the physics file, but then there would be a circular dependency between physics and io. The io code needs to interpret half-life values in files that provide information about isotopes, and those half-life values are meaningless if you don't know the relationship between seconds, hours, and years. Physics code needs access to io code quite frequently because, for example, you can't determine what portion of photons will penetrate a 1/4" thick iron plate without reading iron's transmission coefficients as a function of photon energy from a file. It's not a big deal to have a units package, but I don't typically create a new category for just two definitions that already fit well in one of the existing categories.
Also, there were two functions in the physics package that I moved to the math package so that functions in the io package could depend on them. These functions were responsible for sorting emission lines by energy and filtering out emission lines that would not be included in the analysis because their energies are often associated with false positives. These operations need to be performed on every isotope that is read from the application's isotope library, and the easiest way to arrange that is to call them from the function that reads the isotope library. Again, not a big deal, but I would not ordinarily include code in the math package that can only operate on emission data - that belongs in physics.
On Sat, Aug 14, 2010 at 1:37 AM, Wilson MacGyver <wmacgy...@gmail.com> wrote: > I realize that. I was pondering why I don't run into the the 2nd problem.
> In your code, how many files/name spaces are you creating? > And how many lines of code are in each file? I'm curious how you > organize your code.
> On Aug 14, 2010, at 12:39 AM, Eric Lavigne <lavigne.e...@gmail.com> wrote: > .
>> I discussed two problems. The first problem, which you addressed, was >> mostly just a warm-up for discussing a related problem that is more >> severe. Where can I put (def g) so that two files can "require" each >> other?
>> This is not a rare problem for me. Like Mike Anderson, I work around >> it by putting extra thought into which package-level dependencies I >> will allow, which sometimes necessitates creating more or fewer >> packages than I otherwise would have created.
> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your first post. > To unsubscribe from this group, send email to > clojure+unsubscribe@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en
Btsai wrote: > Continuing this train of thought...
> 1. The "declare" macro may be handy for declaring multiple names at > once.
> 2. Maybe one could use the functions in clojure.repl or clojure- > contrib.ns-utils to write something that automatically forward > declares everything needed?
> On Aug 13, 10:49 pm, Tim Daly <d...@axiom-developer.org> wrote:
>> Suppose you make a file containing a >> (def foo) >> form for every defn in every file and then load that first? >> Does that solve the circular reference problem?
>> Tim Daly
>> Eric Lavigne wrote:
>>> The (def g) in your example has the same effect as the (declare foo) >>> in my example.
>>> I discussed two problems. The first problem, which you addressed, was >>> mostly just a warm-up for discussing a related problem that is more >>> severe. Where can I put (def g) so that two files can "require" each >>> other?
>>> This is not a rare problem for me. Like Mike Anderson, I work around >>> it by putting extra thought into which package-level dependencies I >>> will allow, which sometimes necessitates creating more or fewer >>> packages than I otherwise would have created.
>>> On Sat, Aug 14, 2010 at 12:13 AM, Wilson MacGyver <wmacgy...@gmail.com> wrote:
>>>> I rarely run into this. The few times I have, I just do
>>>> -- >>>> Omnem crede diem tibi diluxisse supremum.
>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clojure@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+unsubscribe@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en
On Sat, Aug 14, 2010 at 10:33 AM, Eric Lavigne <lavigne.e...@gmail.com>wrote:
> I originally wanted to include the contents of the units file in the > physics file, but then there would be a circular dependency between > physics and io.
So what was the problem with splitting units out into it's own name space?
Circular dependencies between modules is a major code smell. Code where everything depends upon everything, or even close to that, is unmaintainable.
On Sun, Aug 15, 2010 at 10:43 AM, Brian Hurt <bhur...@gmail.com> wrote:
> On Sat, Aug 14, 2010 at 10:33 AM, Eric Lavigne <lavigne.e...@gmail.com>wrote:
>> I originally wanted to include the contents of the units file in the >> physics file, but then there would be a circular dependency between >> physics and io.
> So what was the problem with splitting units out into it's own name space?
> Circular dependencies between modules is a major code smell. Code where > everything depends upon everything, or even close to that, is > unmaintainable.
Circular dependencies are then the canary in the coal mine, warning you that your code is trending (hard) towards unmaintainability, You ignore the warning at your peril.
On Sun, Aug 15, 2010 at 7:45 AM, Brian Hurt <bhur...@gmail.com> wrote: > Circular dependencies are then the canary in the coal mine, warning you that > your code is trending (hard) towards unmaintainability, You ignore the > warning at your peril.
Avoiding circular dependencies can result in either: a) Everything ends up in one file. b) Everything ends up in zillions of small files with very complex dependency structures (and every time you move a function from one namespace to another to avoid a circular dependency, you have to locate every file that uses the relocated function and change the references to that function, all without the help of IDEs that do this kind of refactoring for you -- it can be a lot of work).
Both of these scenarios result in their own significant maintainability problems. I think it's much more ideal to be able to divide modules of functions into the appropriate granularity for the task at hand. I have found that for some tasks, circular dependencies arise very naturally at the desired level of granularity, and I don't like having to rework everything into something that avoids circular dependencies. This problem seems to be especially acute in rapidly evolving systems, because you're forced to constantly think about conforming to an acyclic structure at a time when you're focusing on something other than overall structure. I'd rather get things to work, and *then* when the code base becomes stable, figure out a more optimal organization.
On Sun, Aug 15, 2010 at 8:13 PM, Brian Hurt <bhur...@gmail.com> wrote:
> Circular dependencies between modules is a major code smell. Code where > everything depends upon everything, or even close to that, is > unmaintainable.
Sometimes there's simply no way around it. For instance, I recently had some python code that (stripped to its simplest form) had two classes, Document and Attachment, where Attachment was a specialised subclass of Document, but Document had a list of Attachments as one of its members. There was no way around packing the Document and Attachment classes into one big file, simply because I couldn't do circular module dependencies (python files are modules), but any refactoring of the code to remove the circular dependency left it *more* complicated than before.
