[Newbie] Observations and questions regarding long-running, adaptable, transparent software

[falcon]

I work on financial software--the kind that reads streaming data and
static data from various sources, transforms these streams through
various calculations and possibly decides to send out trades.

The logic inside this engine, how various decisions are made, what
calculations are used, what data is touched, is usually designed by
someone other than the programmer. Who ever works out the logic, it
is basically exploritary in nature. The domain expert may have a
general idea of what he wants and the 'design' is _always_ being
changed.

If there is a bug in the software or in the design, the bug often
can't be allowed to linger for weeks or days. Ideally, the running
server needs to be changed in matter of minutes. Again ideally, the
server shouldn't be brought off-line since it would loose the state it
has built up throughout the day (the stuff it has to 'remember') and
the server may be running many other strategies need to respoond to
the market in milliseconds.

As I've already mentioned, systems such as these basically need to
execute ideas in brains of non-programmers. These ideas change often,
very often. Many ideas are tried, many of them are discarded. With
such frequent changes, there is no way someone can build custom GUIs
to interact with and monitor the running strategy and add the kind of
logging which will allow "why the hell isn't this working" type of
questions to be answered.

Erlang is very good at concurrency and can probably handle working
with streams of data. Haskell, F#, etc. are very good at building
DSLs which let programmers quickly put together strategies. Java and C
++ (and C#) are very good as 'standard' languages which large
libraries provided by vendors, open source and built-in packages.
Excel, yes Excel, is great at combining user interface, simple logic
and the ability to change programs on the fly.

It looks like Clojure might be able to solve some of these problems as
well. I'm trying to figure out how well it fits in this environment.

I see that Clojure is designed to handle concurrent programming
(useful for multi-cores). I also notice that Clojure has the ability
to change programs while they are running; however, I haven't been
able to find examples of this. From what little I know of lisp/
scheme, DSLs are quite natural to them (even if they retain the lisp
syntax). Clojure obviously also has access to Java's libraries.

Is there a good solution to the problem of observing the program as it
executes, or easily peeking into the state of the server? The kinds
of trading strategies I described are often called 'black-boxes,' but
they are usually not true black boxes. There has to be an easy way to
keep track of what the trading brain is seeing and thinking. I once
saw a demo for small talk in which the programmer drew a little car,
attached the car to some code on one end and attached a steering wheel
on the other end and started 'controlling' the virtual car through the
virtual steering wheel...and in the time-span of a demo!

Is there any effort to bring FRP (FrTime, Yampa, etc.) library to
Clojure?

Any other ideas how Clojure can improve the kind of softare I
described above?

I'm not naturally inclined to like the lisp syntax, but there seem to
be enough benefits to real-world problems and I'm trying to get a
better feel for this paradigm. I'm also hoping tat Clojure is a
'jujutsu' language, where a single developer or a small number of
developers can take advantage of the language to 'fight off' large
problems or larger teams with bigger budgets and more unweildy weapons.

[Stuart Sierra]

Brief response to:

On Sep 16, 11:40 pm, falcon <shahb...@gmail.com> wrote:
> I also notice that Clojure has the ability
> to change programs while they are running; however, I haven't been
> able to find examples of this.

Most Lisps give you this for free, since the compiler is integrated
into the runtime. You can connect to a REPL on a running server, re-
def a function, and it takes effect immediately. There are several
REPL-over-a-socket experiments in the mailing list archives, but no
canonical implementation yet.

-Stuart

[Alexander Kjeldaas]

2008/9/17 Stuart Sierra <the.stua...@gmail.com>

I don't see how this should work.  The re-defined variable has thread-local scope and shouldn't be viewable in other threads.  If it was not so, then evaluating a form (foo ...) would mean that foo would have to be looked up through a ref.  Which in turn means that you need to be in a transaction.  This again means you wouldn't see the mutation that is done in another thread anyway.

Alexander

[Allen Rohner]

> I don't see how this should work.  The re-defined variable has thread-local
> scope and shouldn't be viewable in other threads.

Vars have thread-local scope. Defs are global.

Allen

[Rich Hickey]

Hmm...

def establishes the root binding of a var, said var being associated
with a name in a namespace. If the var already exists, def replaces
the root binding (but not the var). defn expands into def, so all this
applies to defn as well.

Root bindings are shared between threads, so a re-def will be seen by
already-running code in all threads.

binding establishes a per-thread binding for a var, hiding the root
binding and any prior bindings in this thread. A var that has a per-
thread binding can be set!, and those changes will be seen only in the
one thread. When the stack unwinds past the binding it is
disestablished, and the prior binding becomes visible again.

Idiomatic Clojure code that wants to treat a var as a true (thread-
local) variable should use binding and set!

While it is possible to make a globally visible change to a var with
def, the only appropriate scenarios for doing so are to fix a bug, or
for scratch vars at the repl.

Rich

[Rich Hickey]

On Sep 16, 11:40 pm, falcon <shahb...@gmail.com> wrote:

> I work on financial software--the kind that reads streaming data and
> static data from various sources, transforms these streams through
> various calculations and possibly decides to send out trades.
>

...

> Is there a good solution to the problem of observing the program as it
> executes, or easily peeking into the state of the server? The kinds
> of trading strategies I described are often called 'black-boxes,' but
> they are usually not true black boxes. There has to be an easy way to
> keep track of what the trading brain is seeing and thinking. I once
> saw a demo for small talk in which the programmer drew a little car,
> attached the car to some code on one end and attached a steering wheel
> on the other end and started 'controlling' the virtual car through the
> virtual steering wheel...and in the time-span of a demo!
>
> Is there any effort to bring FRP (FrTime, Yampa, etc.) library to
> Clojure?
>

I don't know of any yet. There is a CL package, Cells, that might make
sense for Clojure, esp re: spreadsheet-like behavior.

> Any other ideas how Clojure can improve the kind of softare I
> described above?
>

I think to the extent you use Clojure's data structure abstractions,
visualizations might take the form of general purpose UIs rather than
custom application-specific ones. Access to Swing and other Java libs
will make that easier.

You can attach Java debuggers to running Clojure programs, and I know
the enclojure folks are looking into Clojure-specific data structure
viewers.

You can fairly efficiently hang on to old versions of Clojure's
persistent data structures, since updates are incremental, and use
that to track decision making.

Clojure shares with Lisps good DSL capabilities, enhanced by the
broader reader support for data structures other than lists.

One thing missing from all of the languages you mention, including
Clojure, is sufficiently declarative logic (e.g. rules). That is
something I am actively researching for inclusion in Clojure.

> I'm not naturally inclined to like the lisp syntax, but there seem to
> be enough benefits to real-world problems and I'm trying to get a
> better feel for this paradigm. I'm also hoping tat Clojure is a
> 'jujutsu' language, where a single developer or a small number of
> developers can take advantage of the language to 'fight off' large
> problems or larger teams with bigger budgets and more unweildy weapons.

Keep up the good fight!

Rich

[Stuart Sierra]

On Sep 17, 12:45 pm, Rich Hickey <richhic...@gmail.com> wrote:
> While it is possible to make a globally visible change to a var with
> def, the only appropriate  scenarios for doing so are to fix a bug, or
> for scratch vars at the repl.

This brings up an idea I had a while back -- what if you wanted to
update a running system in a transaction? Say you have a bug that
requires multiple defs to fix. You want all those defs to take effect
at the same time. As I understand it, that's not possible with
Clojure right now, but didn't you mention that Vars and Refs used to
be the same thing?

Of course, there might be a significant performance penalty to do a
Ref lookup on every function call.

This is pure speculation, not a feature that I have any need for right
now.

-Stuart

[Rich Hickey]

On Sep 17, 3:40 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> On Sep 17, 12:45 pm, Rich Hickey <richhic...@gmail.com> wrote:
>
> > While it is possible to make a globally visible change to a var with
> > def, the only appropriate scenarios for doing so are to fix a bug, or
> > for scratch vars at the repl.
>
> This brings up an idea I had a while back -- what if you wanted to
> update a running system in a transaction? Say you have a bug that
> requires multiple defs to fix. You want all those defs to take effect
> at the same time. As I understand it, that's not possible with
> Clojure right now, but didn't you mention that Vars and Refs used to
> be the same thing?
>
> Of course, there might be a significant performance penalty to do a
> Ref lookup on every function call.
>

Exactly.

> This is pure speculation, not a feature that I have any need for right
> now.
>

It's an interesting idea, and something I tried, but the overhead was
too great.

Rich

[Raoul Duke]

> It's an interesting idea, and something I tried, but the overhead was
> too great.

i know less than nothing here, of course, but... any way to make the
default case of "nothing has changed" be somehow fast (enough), and
only the "oh poop things got edited" have to go into slow-mo for a
second to adjust?

[Rich Hickey]

Unfortunately not. If you want to see a consistent view of a set of
refs you must be in a transaction, and it's not workable to require
all fn calls be in transactions.

Rich

[Stuart Sierra]

On Sep 17, 7:13 pm, Rich Hickey <richhic...@gmail.com> wrote:
> Unfortunately not. If you want to see a consistent view of a set of
> refs you must be in a transaction, and it's not workable to require
> all fn calls be in transactions.

More speculation, again just out of curiosity: Say you have a system
where certain function definitions act like rules, and you want to
change those rules in a transactional manner. Would it be reasonable
to wrap just those functions with Refs?

On a related note, would it be possible to implement atomic, system-
wide transactions by suspending the current computation, replacing
some root bindings, and continuing? How do long-running systems like
Erlang handle this problem?

-Stuart

[Raoul Duke]

> How do long-running systems like
> Erlang handle this problem?

iiuc, erlang doesn't have transactions so it is a different situation.
when you redefine a function, the next time the (standard approach to
writing erlang actors) tail call goes back into the function it will
be actually going into the new definition. so of course that limits
exactly what can be changed. and it only allows for one version to be
running at a time so anything depending on it would have to be
upgraded somehow simultaneously? (well, erlang systems tend to be
written to handle errors well i guess so maybe you can step-wise
upgrade bits and pieces until it is all working again.)

[Shawn Hoover]

On Thu, Sep 18, 2008 at 7:11 AM, Stuart Sierra <the.stua...@gmail.com> wrote:

On a related note, would it be possible to implement atomic, system-
wide transactions by suspending the current computation, replacing
some root bindings, and continuing?  How do long-running systems like
Erlang handle this problem?

-Stuart

Erlang handles code replacement at the module level. Being process-based (referring to Erlang processes, not OS processes), Erlang can essentially give each process a persistent reference to each module it includes. The VM keeps up to 2 versions of hte code alive. When a module is replaced, new processes see the new module. Existing processes keep using the old module until they make a fully qualified function call, at which point they get the new module. If you replace the code again, processes running the old-old code die. (I'm rehashing the contents of http://erlang.org/doc/reference_manual/code_loading.html#12.2)

--Shawn

[Stuart Sierra]

On Sep 17, 2:00 pm, Rich Hickey <richhic...@gmail.com> wrote:
> > Is there any effort to bring FRP (FrTime, Yampa, etc.) library to
> > Clojure?
>
> I don't know of any yet. There is a CL package, Cells, that might make
> sense for Clojure, esp re: spreadsheet-like behavior.

Cells is an awesome package with abysmal documentation. Come to think
of it, Cells would work probably work better in Clojure than it does
in Common Lisp. Last I heard, the CL implementation relies on a
global "state counter" to track updates. Cloure's STM system could
take care of that more elegantly.

-Stuart

[Shawn Hoover]

This cell namespace was pasted the other day in IRC: http://paste.lisp.org/display/66688 (also some Java code: http://paste.lisp.org/display/66689). I'm not sure if it's the same as the CL Cells, but it does have to do with dependencies and updates and it uses Clojure transactions.

[falcon]

Google often brings me to the cells website, but I haven't been able
to figure out what it exactly is, you are right about the
documentation.

Although, speaking of documentation, I wish clojure also had some
printable documents, tutorial which I could pack in my bag and read on
the train. The videos are quite good though (I wrote the original
email before I finished watching the "Clojure for Java programmers"
video.

[falcon]

Looks like the FrTime dissertation was published this year:

Integrating Dataflow Evaluation into a Practical Higher-Order Call-by-
Value Language
By Gregory Cooper

http://dl.lib.brown.edu/pdfs/etd67.20080429180432.pdf

A quote from it:
"A technique similar to that employed by FrTime has been used to
implement a form
of dataflow for slot-based object systems like CLOS [33]. The basic
idea is to extend slot
accessor and mutator methods with code to implement dataflow updates.
In particular,
when an accessor is invoked from a signal-defining context, it records
a dependency as
well as returning a value. Likewise, when a mutator is invoked, it
iterates through its list of
dependents and re-evaluates them. This strategy was used to build the
one-way constraint
systems in Garnet [68] and Amulet [69] and has more recently been used
in the Cells [91]
library. None of these systems appears to support higher-order
reactivity or to address
glitches. Rather, they employ a depth-first update algorithm and avoid
infinite loops in
cycles by recomputing any given value at most once in a given update."

On Sep 18, 5:01 pm, "Shawn Hoover" <shawn.hoo...@gmail.com> wrote:
> On Thu, Sep 18, 2008 at 1:12 PM, Stuart Sierra

> <the.stuart.sie...@gmail.com>wrote:

>
>
>
> > On Sep 17, 2:00 pm, Rich Hickey <richhic...@gmail.com> wrote:
> > > > Is there any effort to bring FRP (FrTime, Yampa, etc.) library to
> > > > Clojure?
>
> > > I don't know of any yet. There is a CL package, Cells, that might make
> > > sense for Clojure, esp re: spreadsheet-like behavior.
>
> > Cells is an awesome package with abysmal documentation.  Come to think
> > of it, Cells would work probably work better in Clojure than it does
> > in Common Lisp.  Last I heard, the CL implementation relies on a
> > global "state counter" to track updates.  Cloure's STM system could
> > take care of that more elegantly.
>

> This cell namespace was pasted the other day in IRC:http://paste.lisp.org/display/66688(also some Java code:http://paste.lisp.org/display/66689). I'm not sure if it's the same as the

[Stuart Sierra]

On Sep 18, 5:17 pm, falcon <shahb...@gmail.com> wrote:
> Google often brings me to the cells website, but I haven't been able
> to figure out what it exactly is, you are right about the
> documentation.

I've met Ken Tilton (Cells' author) at LispNYC, so here's what I've
gathered -- you define a Model (a CLOS object), which contains Cells
(slots of that object). Each Cell is defined by an expression, which
may include references to other cells. Whenever the value of one Cell
changes, all the other Cells that depend on it are automatically re-
evaluated. You can also add triggers that call a function whenever a
Cell's value changes.

There's a global state counter. All Cells must be completely
evaluated at state N before they can advance to state N+1. So
circular dependencies work correctly.

Basically, it's event-driven programming, so it's cool for GUIs or
modeling physical systems, but Kenny has also applied it general-
purpose algorithms, like tree search.

Were it to be implemented in Clojure, you'd probably leave out the
Models (which aren't necessary) and then somehow attach a value
expression to a Ref (or Agent?). Transactions and immutability should
make it easier than in CL. Agents may even get you half-way there,
but I don't entirely understand Agents yet.

-Stuart

[MikeM]

Someone's been working on cells for clojure: http://paste.lisp.org/display/66688

[Randall R Schulz]

On Thursday 18 September 2008 15:28, falcon wrote:
> Looks like the FrTime dissertation was published this year:
>
> Integrating Dataflow Evaluation into a Practical Higher-Order
> Call-by- Value Language
> By Gregory Cooper
>
> http://dl.lib.brown.edu/pdfs/etd67.20080429180432.pdf

This document appears to have been removed from its server. Google
Scholar still indexes it (with only a single hit on the full, quoted
title) and (for now) has a HTML conversion of it, which I snatched,
just in case no other copy becomes available and that entry ages out of
Google's cache.

A little more poking around (using regular Google instead of Google
Scholar) turned up this:

<http://www.cs.brown.edu/research/pubs/theses/phd/2008/cooper.pdf>

Randall Schulz