Static pointers extension, an update

[Boespflug, Mathieu]

Hi all,

Simon PJ and a few of us at Tweag I/O have been hard at work on a
language extension for GHC to better support distributed programming
in general, with a particular focusing on improving the experience
using Cloud Haskell in particular. As has been recapped several times
now, CH suffers from the fact that sending closures around the cluster
to spawn computations is tedious and error prone. Some boilerplate is
required to make it work, and while there are Template Haskell
functions to help with the boilerplate, maintaining remote tables by
hand is still anti-modular, forces unnatural ordering of definitions
in source files, and offers poor static guarantees. The proposed
language extension, -XStaticPointers, aims squarely at resolving this.

Facundo submitted an early version of our work in August:

https://phabricator.haskell.org/D119

The current design (which has changed quite a bit) is documented in a
series of wiki pages, rooted here:

https://ghc.haskell.org/trac/ghc/wiki/DistributedHaskell

Type safety is a challenge in a distributed setting, especially in the
presence of (ad hoc) polymorphism and existentially quantified types.
Our aim is to to guarantee type safety without having to trust large
swaths of user library code (e.g. rank1dynamic). It turns out that to
limit the amount of trusted code to nearly nothing requires a change
to Data.Typeable (see the proposal linked above).

Our aim is to make a final push in the coming 10 days to get a very
simple version of the language extension merged into GHC just in time
for the 7.10 release early next year. We don't want to have to
revolutionize Data.Typeable in the process. Now would simply be the
wrong time for that! So we've outlined a "half way house" on the road
to the full language extension, which you should expect to see in GHC
7.12. Details here:

https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan

We're hard at work implementing the basic compiler support for this,
but we'd like your feedback concerning the "halfway house" we're
proposing above, in particular to understand whether the design will
at least allow for an implementation that meets the performance
constraints of some of you. Make sure to read the full StaticPointers
proposal first - it won't make much sense otherwise.

Note that there are a few discrepencies between the full proposal and
the halfway house, in particular in the treatment of encoders and
decoders. The implementation plan includes a rationale for *excluding*
decoders and encoders for StaticPtr from the base library. This
approach still needs to be made consistent across all wiki pages that
document the language extension proposal.

Best,

--
Mathieu Boespflug
Founder at http://tweag.io.

[Patrick Maier]

Hi Mathieu,

it's good to see progress on the StaticPointers extension.

> https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan

I have some comments and questions on the "half way house"
implementation plan from the point of view of HdpH.


(1) I welcome that serialisation of StaticNames is left to user
space. Looking at the current API, it seems that serialising a
FingerPrint is sufficient. However, from the point of view of HdpH,
with its tendency to nest many small Closures, a FingerPrint is still
quite big. I'd prefer serialising StaticNames as small integers, say
an index into an array, or a pre-computed perfect hash. I think this
could be done in user space if we had access to the set of all
StaticNames. So I'd like to add

allStaticNames :: [StaticName]

to the API.


(2) The implementation plan proposes

> In GHC 7.10, possibly only allow identifiers in the body of a static
> form, depending on the time required to implement floating of
> expressions to top-level adequately. This isn't a loss of generality,
> but is a loss of convenience for the user: one would have to float all
> expressions as top-level definitions manually.

I agree that it isn't a loss of generality but it would make HdpH
application code even clunkier than it is now. As an example, say one
wanted to create a task computing 'fib' of 'm + n', where 'fib' is a
top-level function and 'm' and 'n' are local variables. One would like to
write

$(mkClosure [| toClosure $ force $ fib (m + n) |])

where 'toClosure' and 'force' are HdpH library functions.

Instead, currently one has to manually abstract the local variables
that will form the Closure's environment, define the environment
abstraction

fib_abs (m,n) = toClosure $ force $ fib (m + n)

at the top-level, and then write

$(mkClosure [| fib_abs (m,n) |])

However, internally the 'mkClosure' macro calls 'static' on a dictionary
constructed from 'fib_abs' rather than on 'fib_abs' itself. So if
floating of 'static' expressions to top-level isn't supported, HdpH
users would additionally have to define

fib_abs_dict = mkCDict fib_abs

at the top-level. (This can probably be simplified by yet more
Template Haskell hackery.)


(3) The notes section in the implementation plan detail how the
compiler would process 'static e' where 'e' has some type 'a'. Two
steps, the creation of a DynStaticPtr and the type checking appear to
assume that 'a' is an instance of Typeable. Doesn't this limit
'static' to monomorphic types?


(4) I propose to rename 'deRefStaticPtr' to 'unsafeDerefStaticPtr',
and 'static' to 'unsafeStatic'. In 7.12, hopefully, there will be safe
versions of these primitives, recording and checking typereps and
avoiding 'unsafeCoerce'. However, users wanting to opt out of this
added level of safety for performance reasons could continue to use the
unsafe primitives.


Cheers,
Patrick

[Simon Peyton Jones]

| (1) I welcome that serialisation of StaticNames is left to user space.
| Looking at the current API, it seems that serialising a FingerPrint is
| sufficient. However, from the point of view of HdpH, with its tendency
| to nest many small Closures, a FingerPrint is still quite big. I'd
| prefer serialising StaticNames as small integers, say an index into an
| array, or a pre-computed perfect hash. I think this could be done in
| user space if we had access to the set of all StaticNames. So I'd like
| to add
|
| allStaticNames :: [StaticName]
|
| to the API.

Interesting idea. If you use an index into a table you immediately become dependent on having pretty much the exact same binary on each machine. I was thinking of serialising something tantamount to 'base:Data.Functor.runIdentity', which will work even if I add code, or new statics, and recompile.

(I'd want to be much less sloppy about TypeReps: type matching should be pretty rigorous, not just by name.)

But I accept that in particular cases you might know that the index is enough. 'allStaticNames' would let a user build a translation index <-> StaticName.

It seems a bit indirect. On de-serialiastion you'd have to
1. parse the index
2. translate to the StaticName
3. look up the StaticName
but step 3 of course is a hash-table look up that discovers... the index you had in step 1! A better alternative would be to split 'lookupStaticPtr' into two:

staticIndex :: StaticName -> Maybe Int -- Nothing if not in SPT
getStaticPtr :: Int -> Maybe DynStaticPtr -- Nothing if out of range

where the Int is the row of the SPT.

Simon
/d/optout.

[Simon Peyton Jones]

| (4) I propose to rename 'deRefStaticPtr' to 'unsafeDerefStaticPtr',
| and 'static' to 'unsafeStatic'. In 7.12, hopefully, there will be safe
| versions of these primitives, recording and checking typereps and
| avoiding 'unsafeCoerce'. However, users wanting to opt out of this
| added level of safety for performance reasons could continue to use
| the unsafe primitives.

I disagree. The intention is that there is nothing unsafe about defRefStaticPtr. Moreover 'static' is a language construct not a function.

The unsafeness comes, I believe, when you want to convert a DynStaticPtr to a StaticPtr a. That might involve unsafeCoerce (for now) but will be type safe in future

Simon

[Patrick Maier]

On 27/11/14 12:53, Simon Peyton Jones wrote:
> | (1) I welcome that serialisation of StaticNames is left to user space.
> | Looking at the current API, it seems that serialising a FingerPrint is
> | sufficient. However, from the point of view of HdpH, with its tendency
> | to nest many small Closures, a FingerPrint is still quite big. I'd
> | prefer serialising StaticNames as small integers, say an index into an
> | array, or a pre-computed perfect hash. I think this could be done in
> | user space if we had access to the set of all StaticNames. So I'd like
> | to add
> |
> | allStaticNames :: [StaticName]
> |
> | to the API.
>
> Interesting idea. If you use an index into a table you immediately become
> dependent on having pretty much the exact same binary on each machine.
> I was thinking of serialising something tantamount to
> 'base:Data.Functor.runIdentity', which will work even if I add code,
> or new statics, and recompile.

I don't think you'd have to run the same binary everywhere but you do
have to run the same application, with the same set of statics.

> But I accept that in particular cases you might know that the index is enough. 'allStaticNames' would let a user build a translation index <-> StaticName.
>
> It seems a bit indirect. On de-serialiastion you'd have to
> 1. parse the index
> 2. translate to the StaticName
> 3. look up the StaticName
> but step 3 of course is a hash-table look up that discovers... the index you had in step 1! A better alternative would be to split 'lookupStaticPtr' into two:
>
> staticIndex :: StaticName -> Maybe Int -- Nothing if not in SPT
> getStaticPtr :: Int -> Maybe DynStaticPtr -- Nothing if out of range
>
> where the Int is the row of the SPT.

Are you suggesting to serialise the index into the compiler-generated SPT?

HdpH could live with that. But it does introduce a dependency on the
compiler that wasn't there if the translation from index to StaticName
is done in user space. For any given set of statics, the compiler would
have to guarantee a consistent order of SPT entries, no matter where the
compiler is run. At the very least, this would mean that all binaries
have to be produced by the exact same compiler version. Which is doable
but does make multi-site distributed computing more arduous. (Remote
tool chain updates are a terrible chore.)


Cheers,
Patrick

[Patrick Maier]

I see; I had misread the plan in that respect.

With regard to future opt-outs from type safety, what is the plan (in
7.12) if the compiler comes across a 'static e' where the type of 'e' is
not Typeable? Will there be a compiler option -fno-typerep-static
to ignore these Typeable constraints and build an SPT without typereps?

Cheers,
Patrick

[Simon Peyton Jones]

Hmm.

For a start, I think you are assuming that we have (Eq StaticName) and (Ord StaticName). Let's document that!

Next, I was assuming more-or-less "the same binary" to get consistent behaviour.

You are, I think suggesting that you could get consistent behaviour with "the same set of statics" provided the SPT (and hence indices into the SPT) was kept in some canonical order. That would make you less sensitive to changes in code. But what "canonical order" did you have in mind? If you are going to rely on it, you'd better state its properties.


My original thought about serialising StableNames themselves was that you would get consistent behaviour even if the number of statics changes. That is much more robust. But I need to be more precise there too. What is the reasonably-robust StableName of
static (f (g a))
?
Perhaps something like
pkg:Module.<hash of "f (g a)">



I would love someone to write these issues up carefully. This big question is: if Haskell program A sends a message to Haskell program B, with a StableName in it, how does B know which function A intended? Might one of you attempt that?


For the short term: I think I still propose the split-up of lookupStablePtr as I propose below, but with some claim about a reasonably robust canonical ordering of stable names.

Simon

[Simon Peyton Jones]

| With regard to future opt-outs from type safety, what is the plan (in
| 7.12) if the compiler comes across a 'static e' where the type of 'e'
| is not Typeable? Will there be a compiler option -fno-typerep-static
| to ignore these Typeable constraints and build an SPT without
| typereps?

I had no such plan. Why would you want a non-Typeable e?

[Patrick Maier]

The main reason is that I want to be able to apply 'static' to
(parametrically) polymorphic terms.

I know that the StaticPointers proposal
[https://ghc.haskell.org/trac/ghc/wiki/StaticPointers] has a plan for
polymorphic statics in a type-safe way but I am not 100% happy with that
plan, for two reasons:

* Serialising StaticPtrs requires serialising typereps.

* I think the proposed plan would only work for rank1 types.

(HdpdH doesn't currently have a need for Closures of rank2 type but I
wouldn't want to rule it out.)

Cheers,
Patrick

[Simon Peyton Jones]

Well you can always unsafeCoerce it to (), and send that!
(And the reverse at the other end.)
That's effectively sending it without a TypeRep.

Simon

| -----Original Message-----
| From: distribut...@googlegroups.com [mailto:distributed-
| has...@googlegroups.com] On Behalf Of Patrick Maier
| Sent: 27 November 2014 15:33
| To: distribut...@googlegroups.com
| Subject: Re: Static pointers extension, an update
|

| --
| You received this message because you are subscribed to the Google Groups
| "Distributed Haskell" group.
| To unsubscribe from this group and stop receiving emails from it, send an
| email to distributed-has...@googlegroups.com.
| To post to this group, send an email to distributed-
| has...@googlegroups.com.
| To view this discussion on the web, visit
| https://groups.google.com/d/msgid/distributed-
| haskell/54774424.9090203%40glasgow.ac.uk.
| For more options, visit https://groups.google.com/d/optout.

[Patrick Maier]

I've spent some time thinking about more robust ways of serialising
StaticNames (attached, in markdown format).

Cheers,
Patrick

On 27/11/14 15:07, Simon Peyton Jones wrote:

[Simon Peyton Jones]

Patrick

Very good! Could we somehow get your material into the GHC wiki?
https://ghc.haskell.org/trac/ghc/wiki/StaticPointers

My main comment. In your proposal we seem to have
StaticPtr a
StaticName
StaticKey
Fingerprint
SPTIndex (Int)

That's too many! I suggest
StaticPtr a The full enchilada
StaticName The thing you serialise, a ByteArray#
SPTIndex A less robust thing you serialise, an Int

Even distinguishing the latter two is uncomfortable, but you argued strongly that a fingerprint is too long to transmit, and you wanted to transmit a SPTIndex instead.
StaticName and SPTIndex are both concrete, not abstract, so that you can serialise them.

So you can't get strings etc out of a StaticName. To do that, look it up in the SPT to get a StaticPtr.
staticIndex :: StaticName -> Maybe SPTIndex

StaticPtr should be abstract, but you should be able to extract stuff for debugging like
staticPtrModule :: UMI
staticPtrLoc :: (Int,Int)
staticPtrId :: Fingerprint -- Perhaps; I'm not that keen


Smaller things

* If toAscListSPT returns a list in ascending order, maybe you don’t
need to include the indices?

* Need to be explicit that SPTIndex is 0-indexed

[Patrick Maier]

On 02/12/14 14:21, Simon Peyton Jones wrote:
> Could we somehow get your material into the GHC wiki?
> https://ghc.haskell.org/trac/ghc/wiki/StaticPointers

Right, I'll try to upload it tonight.

> My main comment. In your proposal we seem to have
> StaticPtr a
> StaticName
> StaticKey
> Fingerprint
> SPTIndex (Int)
>
> That's too many! I suggest
> StaticPtr a The full enchilada
> StaticName The thing you serialise, a ByteArray#
> SPTIndex A less robust thing you serialise, an Int

I didn't bother thinking about what should should be abstract, what
concrete, and what internal.

Agree that StaticNameKey should be internal. I am treating Fingerprint
as part of the API; maybe it doesn't need to be. (However, Fingerprints
and their alleged properties as cryptographic hashes do play a crucial
role in the construction of StaticNames; there's no getting away from
that.)

What about DynStaticPtr though? Exposing DynStaticPtr would be
attractive for chancers (eg. HdpH developers) who just want to
unsafeCoerce what they find in the SPT to whatever type they think it
should have.

In general, I think there is a case for separating the Typeable/Dynamic
issues as much as possible from the implementation of StaticPtrs.

> Even distinguishing the latter two is uncomfortable, but you argued strongly that a fingerprint is too long to transmit, and you wanted to transmit a SPTIndex instead.
> StaticName and SPTIndex are both concrete, not abstract, so that you can serialise them.
>
> So you can't get strings etc out of a StaticName. To do that, look it up in the SPT to get a StaticPtr.
> staticIndex :: StaticName -> Maybe SPTIndex
>
> StaticPtr should be abstract, but you should be able to extract stuff for debugging like
> staticPtrModule :: UMI
> staticPtrLoc :: (Int,Int)
> staticPtrId :: Fingerprint -- Perhaps; I'm not that keen

You should also be able to get the same information out of a
DynStaticPtr. Consider this scenario: B receives a serialised static
pointer from A. B manages to look it up in its SPT and finds a
dsp :: DynStaticPtr. However, unwrapping dsp into sp :: StaticPtr t
fails because the ambient type StaticPtr t does not match the stored
TypeRep. To produce a sensible error message, one needs access to UMI
and source location.

> Smaller things
>
> * If toAscListSPT returns a list in ascending order, maybe you don’t
> need to include the indices?

It depends. Having explicit indices gives us the freedom to implement
the SPT as a collision-free hash table, at the expense of a somewhat
bigger table with a number of unused buckets.

If we insist that the indices are implicit [0 .. length SPT - 1] then
the SPT will be more compact but we have to have a collision resolution
mechanism, e.g. linear chaining.

I don't know whether hash collisions are a big drawback. But I would
argue in favour of the slightly more general type for toAscListSPT
because it constrains the implementation choices less.


Cheers,
Patrick

[Simon Peyton Jones]

| What about DynStaticPtr though? Exposing DynStaticPtr would be
| attractive for chancers (eg. HdpH developers) who just want to
| unsafeCoerce what they find in the SPT to whatever type they think it
| should have.

Yes, DynStaticPtr is a concrete type

data DynStaticPtr where
DSP :: Typeable a => StaticPtr a -> DynStaticPtr

and as such can readily be unpacked by clients to get at the underlying StaticPtr, whence they can get the bits and pieces.


It'd be good to talk to Mathieu and Facundo about merging your stuff and theirs, so there is only one version of the truth!

Simon

| -----Original Message-----
| From: Patrick Maier [mailto:Patric...@glasgow.ac.uk]
| Sent: 02 December 2014 17:31
| To: Simon Peyton Jones; Boespflug, Mathieu
| Cc: distribut...@googlegroups.com
| Subject: Re: Static pointers extension, an update
|

[Facundo Domínguez]

Hello,

I have updated the API in the Implementation Plan wiki [1]. This
proposal and this email represents only my personal opinion as the guy
trying to get the code out of the door.

I propose dropping unsafe uses of Dynamic, because I don't think it
achieves any compatibility in the face of future changes. IIUC every
user of Dynamic or DynStaticPtr would be forced to use unsafeCoerce at
some point and this in turn would materialize code that needs to be
changed when the interface is made truly safe.
Code like: unsafeCoerce $ (fromJust $ fromDyn d :: ()) :: StaticPtr YourType

I think that Patrick contributions deserve attention. Is there a need
to add them in the interim version? The FFI could be used to access
the SPT in the RTS to experiment with variants of the API.

We certainly could consider augmenting the StaticPtrInfo datatype with
more information deemed essential.

Best,
Facundo

[1] https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan#TheAPIofGHC.StaticPtrinterimversionfor7.10https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan#TheAPIofGHC.StaticPtrinterimversionfor7.10

> --
> You received this message because you are subscribed to the Google Groups "Distributed Haskell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to distributed-has...@googlegroups.com.

> To post to this group, send an email to distribut...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/distributed-haskell/618BE556AADD624C9C918AA5D5911BEF3F3F0DC8%40DB3PRD3001MB020.064d.mgd.msft.net.

[Boespflug, Mathieu]

It sounds pretty easy to augment the API with some variant of
sptToAscList without much impact on the rest, which if we do provide I
think is all Patrick needs to support his use cases, right?

--
Mathieu Boespflug
Founder at http://tweag.io.

On 4 December 2014 at 00:47, Facundo Domínguez

[Patrick Maier]

Hi Facundo,

> I propose dropping unsafe uses of Dynamic, because I don't think it
> achieves any compatibility in the face of future changes. IIUC every
> user of Dynamic or DynStaticPtr would be forced to use unsafeCoerce at
> some point and this in turn would materialize code that needs to be
> changed when the interface is made truly safe.
> Code like: unsafeCoerce $ (fromJust $ fromDyn d :: ()) :: StaticPtr
YourType

I am not fussed either way, as long as unsafeCoerce or
unsafeLookupStaticPtr are hidden in the Closure library.

Related to this: In a recent email exchange about your commit Simon
wrote

> So the big question is: what if there is a (static e) in a splice?
> Thus $(f (static x))? For now I suggest you simply make this illegal.
> Because you have to run that splice, so you don’t want to generate C
> constructor functions, stub files etc. Indeed initDsTc should really
> check that it doesn't need any of these fancy things.

I find this is somewhat alarming, for the following reason:
Suppose I want `static e :: StaticPtr t` where `t` is polymorphic (and
hence not Typeable). Because the type checker adds a `Typeable t`
constraint, I have to actually write

unsafeCoerce (static (unsafeCoerce e :: ())) :: StaticPtr t

I can't have users write unsafeCoerce, let alone double unsafeCoerce.
Fortunately, this code can be hidden in the HdpH mkClosure macro,
but only if I can use static inside TH splices.

> I think that Patrick contributions deserve attention. Is there a need
> to add them in the interim version? The FFI could be used to access
> the SPT in the RTS to experiment with variants of the API.

HdpH would like to serialise StaticPtrs as indices into the SPT rather
than as Fingerprints. I don't know whether the API of the RTS hash table
that you are using admits these accesses. But in the interim there is
another solution: add an array with all the StaticKeys, and expose

indexToStaticKey :: Int -> StaticKey
sptKeysToAscList :: [(Int, StaticKey)]

`sptKeysToAscList` is exactly what one needs for fingerprinting the SPT.
Plus, it is needed to implement `staticKeyToIndex :: StaticKey -> Int`,
which is the inverse of `indexToStaticKey`, by search through the list.
The search looks inefficient but it'll happen only once per StaticPtr
if you store `staticKeyToIndex fp` in an extra field of the `StaticPtr`
constructor. So I suggest to define `StaticPtr` as

data StaticPtr a = StaticPtr
{ staticPtrKey :: !StaticKey
, staticPtrIndex :: Int -- deliberately non-strict
, staticPtrInfo :: StaticPtrInfo
, deRefStaticPtr :: a }

mkStaticPtr :: StaticKey -> StaticPtrInfo -> a -> StaticPtr a
mkStaticPtr fp nfo e = StaticPtr fp (staticKeyToIndex fp) nfo e
-- Does `mkStaticPtr` need to execute before `sptKeysToAscList`
-- is accessible, eg. in TH splices? That should be fine, thanks to
-- `staticPtrIndex` being non-strict.

> We certainly could consider augmenting the StaticPtrInfo datatype with
> more information deemed essential.

For usability, the source location of the static expression would be
helpful.


How do you compute the StaticKey? I propose to compute the key from
a unique module identifier (pkgName-pkgVersion-fully.qualified.moduleName)
and a hash of the (canonicalised) source code of the static expression.
Using a fresh name instead of the source hash is ok in the interim,
but using a unique module identifier is crucial. Otherwise you might
get into trouble if an application depends on multiple versions of the
same package.


Great work, by the way.

Cheers,
Patrick

[Facundo Domínguez]

> I find this is somewhat alarming, for the following reason:

So far static can be used in splices and in quotations. deRefStaticPtr
only works when executed in the final program. If used in splice it
returns Nothing. There are a couple of TH tests in the current patch
that show this capabilities (TH_StaticPointers.hs and
TH_StaticPointers02.hs).

> How do you compute the StaticKey?

Currently:

GHC.Fingerprint.fingerprintString "<packageId>:<module-name>.sptEntry:<N>"

where

"<packageId>:<module-name>.sptEntry:<N>" is the name of the SPT entry.
Every occurrence of a static form produces a distinct SPT entry.

> indexToStaticKey :: Int -> StaticKey
> sptKeysToAscList :: [(Int, StaticKey)]

Would it suffice to provide

sptKeys :: [StaticKey]

? (similar to what you first proposed as allStaticNames)

Then you could define:

sptKeysToAscList = zip [0..] sptKeys

indexToStaticKey = memo (sptKeys !!)
staticKeyToAscList = memo (flip elemIndex sptKeys)

where memo is some memoization scheme.

> For usability, the source location of the static expression would be
> helpful.

Will look into that.

Facundo

> --
> You received this message because you are subscribed to the Google Groups
> "Distributed Haskell" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to distributed-has...@googlegroups.com.
> To post to this group, send an email to
> distribut...@googlegroups.com.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/distributed-haskell/547FC440.2090206%40glasgow.ac.uk.

[Simon Peyton Jones]

| I have updated the API in the Implementation Plan wiki

| https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan

Great. My comments

* Generally the API looks fine to me now.

* We should advertise explicitly that the next iteration will offer
a safe version of lookupStaticPtr, and probably deprecate unsafeLookupStaticPtr.

* The smallest thing that will get Patrick what he wants is
type SPTIndex = Int -- 0-indexed
sptKeys :: [(SPTIndex, StaticKey)]
Now he can use sptKeys to map SPTIndex <-> StaticKey.

Would that do, Patrick?

Documentation should say what programmers can expect; that is, to what
extent (if any) is an SPTIndex movable from one computer to another?

I'm inclined to include the SPTIndex in the result, to give the
implementation more wiggle room, e.g. to have a non-dense table.

* Patrick's remarks about how to implement StaticPtr (see snipped below)
are interesting but strictly NOT part of the API, and entirely up to
the implementor of StaticPtr.

| `StaticPtr` constructor. So I suggest to define `StaticPtr` as
|
| data StaticPtr a = StaticPtr
| { staticPtrKey :: !StaticKey
| , staticPtrIndex :: Int -- deliberately non-strict
| , staticPtrInfo :: StaticPtrInfo
| , deRefStaticPtr :: a }

* About splicing, this should be illegal
$(f (static e))
but this should be fine
$(f [| static e |])
That is, 'static' is illegal if you need to run it inside the compiler,
but it's fine to generate Haskell code that contains 'static'.

It's the latter that Patrick wants.

However to actually achieve this you have to add 'static' to
Language.Haskell.TH.Syntax. Time is short. Maybe Patrick can help?


Simon

[Simon Peyton Jones]

| I have updated the API in the Implementation Plan wiki [1]. This

| https://ghc.haskell.org/trac/ghc/wiki/StaticPointers/ImplementationPlan

PS: one last point.

I recall that you dealing with question of keeping StaticPtrs alive by registering each StaticPtr as a StablePointer too.

That seems possible, but it would be more direct simply to include the Static Pointer Table in the data structures that are traversed by the gc roots code. That can't be hard. Simon Marlow can advise. (Probably not the most urgent task, but I'd like to avoid complexity wherever possible.)

Simon

| -----Original Message-----
| From: distribut...@googlegroups.com [mailto:distributed-

| has...@googlegroups.com] On Behalf Of Facundo Domínguez
| Sent: 03 December 2014 23:47
| To: Simon Peyton Jones
| Cc: Patrick Maier; Boespflug, Mathieu; distributed-
| has...@googlegroups.com
| Subject: Re: Static pointers extension, an update
|

| > To post to this group, send an email to distributed-
| has...@googlegroups.com.

| > To view this discussion on the web, visit
| https://groups.google.com/d/msgid/distributed-
| haskell/618BE556AADD624C9C918AA5D5911BEF3F3F0DC8%40DB3PRD3001MB020.064
| d.mgd.msft.net.
| > For more options, visit https://groups.google.com/d/optout.
|
| --
| You received this message because you are subscribed to the Google
| Groups "Distributed Haskell" group.
| To unsubscribe from this group and stop receiving emails from it, send
| an email to distributed-has...@googlegroups.com.

| To post to this group, send an email to distributed-
| has...@googlegroups.com.

| To view this discussion on the web, visit
| https://groups.google.com/d/msgid/distributed-

| haskell/CA%2BQ%3D%3DSAQaDpcirwQKn-
| 6dsHxSftJy05qF7yTTMQghX%3D7OPF4pw%40mail.gmail.com.

[Facundo Domínguez]

> However to actually achieve this you have to add 'static' to
> Language.Haskell.TH.Syntax. Time is short. Maybe Patrick can help?

Beware that this is implemented in the current patch already.

> About splicing, this should be illegal
> $(f (static e))

In the current implementation this is allowed but I believe it causes
no troubles. The only limitation is that lookups will return Nothing
because the runtime SPT does not exist in the compiler. See
testsuite/tests/th/TH_StaticPointers02.hs for an example of how it
works.

We can disallow it if the implementation looks still too complex, but
with any luck we don't need to settle the matter immediately.

Facundo

> --
> You received this message because you are subscribed to the Google Groups "Distributed Haskell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to distributed-has...@googlegroups.com.
> To post to this group, send an email to distribut...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/distributed-haskell/618BE556AADD624C9C918AA5D5911BEF3F3F3081%40DB3PRD3001MB020.064d.mgd.msft.net.

[Boespflug, Mathieu]

On 4 December 2014 at 12:25, Facundo Domínguez

<facundo....@tweag.io> wrote:
>> However to actually achieve this you have to add 'static' to
>> Language.Haskell.TH.Syntax. Time is short. Maybe Patrick can help?
>
> Beware that this is implemented in the current patch already.
>
>> About splicing, this should be illegal
>> $(f (static e))
>
> In the current implementation this is allowed but I believe it causes
> no troubles. The only limitation is that lookups will return Nothing
> because the runtime SPT does not exist in the compiler. See
> testsuite/tests/th/TH_StaticPointers02.hs for an example of how it
> works.

Hm, I think that giving a compile time error to the user, ahead of the
splice executing, is much more user friendly than silently returning
an empty SPT, no matter whether that behaviour is documented
somewhere.