Draft smart contract runtime CAP & PR

128 views
Skip to first unread message

Graydon Hoare

unread,
Apr 22, 2022, 9:27:25 PMApr 22
to Stellar Developers
Hi,

The following draft CAP and core PR cover an initial proposed set of
components for the smart contract runtime environment:

https://github.com/stellar/stellar-protocol/pull/1176
https://github.com/stellar/stellar-core/pull/3413

Feedback is very welcome. I know there's a lot to digest, I've put as
much rationale as I can in the CAP, hopefully it all makes sense but
please ask questions if anything's unclear.

Thanks,

-Graydon

Nicolas Barry

unread,
Apr 26, 2022, 10:05:16 PMApr 26
to Graydon Hoare, Stellar Developers
Hey Graydon,
great to see we're making progress on this.

I'll start with a few questions!

Entry points
I do not see anything on that front. How are those exposed?

Host objects

Limits
I know this is a "TBD" section for the most part.
Are there limits on the size of the linear memory available? Are parts of it reserved? What about stack limits?

Limits on host objects.
When it comes to limits, like "total memory allocated by host objects", I imagine that this limit would be enforced at the transaction level, regardless of which VM allocates the object. So basically the limit would be enforced regardless of who allocates it?
Do you think there would also be limits at the VM level? For example, you could imagine having a global limit of 4MB, but each VM would only be able to allocate 1MB (basically, it's like having 4 levels of recursion). This may help with making it easier for contracts to predict when they will hit those limits.
If it was just one global limit, the "memory spike" caused by a contract invocation may cause the contract to fail depending on the calling contract, which is not great.

related to limits. As objects are immutable, I imagine that the policy for "freeing up" unused objects (at least from a limit enforcement point of view), would respect the rule of "not used, not counted"? There are different strategies for resource management (like only freeing up data in "batches" like in GC runtimes). Maybe it doesn't matter, but this should probably be specified.


Host value and host object types:
I find it weird that the doc uses raw integrals instead of enum names from the XDR (that are defined later in the doc, maybe that's the bug: start with the xdr).

Some of the types would benefit for some explanation, and in general what is the goal of the representation exposed to the contracts.

Why are there so many integer types?
I am counting U63, U32, I32 and then U64, I64, BigInt.
I imagine that the more types we have the more conversions need to be supported.

Why is there a 64 bit bitset type?
I would expect instead an arbitrary size bitset object (and maybe not even for v1).


`BigInt` for example: is the goal to expose the actual underlying data so that people can write arbitrary code to manipulate it?
I was expecting this to be completely opaque, and people would just have to rely on host functions to manipulate or access those properties.
I am just not sure if there is some weird tradeoff being made here: if people rarely use the "sign" within contracts, is this creating some weird overhead for those rare occurrences?

"string" vs "opaque"
Are strings utf8 strings? xdr strings are not great from an interop point of view with other systems and languages (in the existing protocol, we have additional constraints on strings), but if we want those strings to play nice with existing assets, we'll have to use the same rules.

"Price" seems redundant with "BigRat"
I think the only place where "Price" exists in the current protocol is when we deal with the DEX. I would leave it out for now, and just have a way (if we need to) for people to round a fraction that corresponds to a Price if needed via a host function.

How do we think of "BigRat" vs "double/float"?
Maybe "BigRat" should not be part of the first version of this and instead be added later depending on what people use in the standard library?

"Operation", "OperationResult" and  "Transaction" seem quite arbitrary at this stage as well.

"AccountID"
What is the purpose of this one? Is this supposed to be a "StrKey" type of thing or something used to identify the execution context? This may not be the right type in either case.

Nicolas




--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/CANhY3x5rTVj1nB8ykFfr7%3DJcNv-bT5%2B2X_m-n7tk2HmLx35cdw%40mail.gmail.com.

Leigh McCulloch

unread,
Apr 27, 2022, 1:16:30 AMApr 27
to Nicolas Barry, Graydon Hoare, Stellar Developers
Hi Nicolas,

I'll respond and chime in on some of your questions:

Why are there so many integer types?
I am counting U63, U32, I32 and then U64, I64, BigInt.
I imagine that the more types we have the more conversions need to be supported.

We get 32bit and 64bit types with wasm opcodes, and we also use these types throughout the existing ledger entries, so there's some value in supporting them. Could we upsize all 32bit types to 64bit types? Maybe, but negative I32 uses less resources than negative I64 when transmitted between the host and guest, so there's a cost consideration. Even if we eliminated some of these types from the contract developer, we're likely to internally use these types to optimize communication between host and guest. Are you thinking the contract developer interface should be simpler, or the host to guest interface?

To dig in on U63: it is an optimization. It exists because we can't fit a full 64bit type inside Val's because we need some space to use to discriminate between other types. Many ledger entries contain positive-only I64 values, and so to make those values cheap to transmit without additional host calls, U63 is used. U63 should be transparent to contract authors as the SDK will hopefully transparently convert a U63 or I64 to a local I64, and vice versa. You can see them in the SDK examples at the moment, as they work with i64 types and don't have to deal with U63 directly.

"string" vs "opaque"
Are strings utf8 strings? xdr strings are not great from an interop point of view with other systems and languages (in the existing protocol, we have additional constraints on strings), but if we want those strings to play nice with existing assets, we'll have to use the same rules.
Contracts themselves probably shouldn't distinguish between strings and opaque. I doubt we need to go down the path of utf8 safe strings, which come with additional costs if they're checked at conversion. However, for API and RPC services built on-top of contract data being able to distinguish between opaques intended to contain data and intended to contain strings
 
"Price" seems redundant with "BigRat"
I think the only place where "Price" exists in the current protocol is when we deal with the DEX. I would leave it out for now, and just have a way (if we need to) for people to round a fraction that corresponds to a Price if needed via a host function.
+1 I noticed same.
 
"Operation", "OperationResult" and  "Transaction" seem quite arbitrary at this stage as well.
+1 I noticed same, and it is unclear to me in any interop story why a contract would introspect the current transaction. A transaction will hopefully just be a container.

"AccountID"
What is the purpose of this one? Is this supposed to be a "StrKey" type of thing or something used to identify the execution context? This may not be the right type in either case.
I asked for AccountId. It's the identifier for referencing accounts in the existing XDR, and so if there is interop we need a first-class object that holds an AccountId. There's some examples of how it might be an input to contract invocations and used within a contract in the liqpool example: https://github.com/stellar/rs-stellar-contract-sdk/tree/main/examples/liqpool/src.

Cheers,
Leigh

On Fri, Apr 22, 2022 at 6:27 PM 'Graydon Hoare' via Stellar Developers <stell...@googlegroups.com> wrote:
Hi,

The following draft CAP and core PR cover an initial proposed set of
components for the smart contract runtime environment:

https://github.com/stellar/stellar-protocol/pull/1176
https://github.com/stellar/stellar-core/pull/3413

Feedback is very welcome. I know there's a lot to digest, I've put as
much rationale as I can in the CAP, hopefully it all makes sense but
please ask questions if anything's unclear.

Thanks,

-Graydon

--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/CANhY3x5rTVj1nB8ykFfr7%3DJcNv-bT5%2B2X_m-n7tk2HmLx35cdw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.

Leigh McCulloch

unread,
Apr 27, 2022, 3:05:41 AMApr 27
to Nicolas Barry, Graydon Hoare, Stellar Developers
Two types that I'm unsure about:

"Box"
It looks like this wraps a Val in an Object. What would happen if an object was wrapped in a val, wrapped in an object? What are the concrete uses for this? Would it be practical to avoid this until we need it? We could support Val's natively everywhere that they should be supported instead of requiring they be wrapped in an object.

"LedgerKey"
I'd like to see LedgerKey removed. It seems like an internal detail that is leaking into the Tx<->Host and Host<->Guest interfaces. I agree that there will be some subset of Objs/Vals that will be provided in some cases, like to identify ledgers, but ideally those identifiers are first-class objects and do not need wrapping in some other object type. I shared some thoughts about that here: https://github.com/stellar/stellar-core/pull/3413#discussion_r856264350.
 
Cheers,
Leigh

Graydon Hoare

unread,
Apr 27, 2022, 4:11:04 PMApr 27
to Nicolas Barry, Stellar Developers
On Tue, Apr 26, 2022 at 7:05 PM Nicolas Barry <nic...@stellar.org> wrote:

> Entry points
> I do not see anything on that front. How are those exposed?

In the section on the interface between host and guest environments,
the CAP mentions that WASM modules carry a list of exported guest
functions, which the host can call. The CAP does not specify which if
any are considered "entrypoints" as that's a higher-level concept
(i.e. the interface or mediation between any transactions that invoke
smart contracts and the host's decision to call a particular guest
function).

> Host objects
>
> Limits
> I know this is a "TBD" section for the most part.
> Are there limits on the size of the linear memory available? Are parts of it reserved? What about stack limits?

It says TBD :)

Speculating-ahead though: we must certainly impose a limit on the
linear memory, as with any memory resources allocated by a contract
implicitly through its use of host functions. The linear memory is in
many ways the easiest memory resource to bound: there's a single "grow
memory" WASM operation and it gives an exact size. A lot of the other
memory-metering will be more subtle due to allocations implicit in
host function calls.

> Limits on host objects.
> When it comes to limits, like "total memory allocated by host objects", I imagine that this limit would be enforced at the transaction level, regardless of which VM allocates the object. So basically the limit would be enforced regardless of who allocates it?

Yes, that is my assumption.

> Do you think there would also be limits at the VM level? For example, you could imagine having a global limit of 4MB, but each VM would only be able to allocate 1MB (basically, it's like having 4 levels of recursion). This may help with making it easier for contracts to predict when they will hit those limits.

Possibly. At present the design doesn't assign a VM owner to an
allocated object, and tracking the movement of objects shared between
them might be challenging (eg. if contract A calls contract B passing
object X, and A is finished using X, does B now own it?) but it's not
beyond the possibility of trying.

> If it was just one global limit, the "memory spike" caused by a contract invocation may cause the contract to fail depending on the calling contract, which is not great.

This is true of other aspects of the transaction environment. For
example if a caller is not authorized to perform some action via the
signatures it carries, or has not declared its requirement to read or
write a given ledger entry from a concurrency control perspective,
then a contract may fail or not-fail depending on calling context.
Even with execution step-limits, a contract called "right near the
end" of its caller's step-limit exhaustion should fail where it would
not when called from a more generous budget context. So I think the
memory accounting context issue is not different.

> related to limits. As objects are immutable, I imagine that the policy for "freeing up" unused objects (at least from a limit enforcement point of view), would respect the rule of "not used, not counted"? There are different strategies for resource management (like only freeing up data in "batches" like in GC runtimes). Maybe it doesn't matter, but this should probably be specified.

There is no mechanism for freeing objects. I could clarify in the CAP
that the intent is to only free all objects allocated by a transaction
at the end of the transaction, but at present the CAP says nothing
about transactions. I should also note that there is no way to trace
the memory graph of a WASM contract to know which objects it's "not
using". And besides, the execution budget for a smart contract is so
brief that it would fit within a typical GC period -- at which point
one might as well just wait until completion and free everything.

> Host value and host object types:
> I find it weird that the doc uses raw integrals instead of enum names from the XDR (that are defined later in the doc, maybe that's the bug: start with the xdr).

If you're referring to the bit-packing tags used on the 64 bit host
value representation, I should emphasize that they are not equal to
the SCValType discriminant values used on the multi-word SCVal XDR
datatype. They are similar (there are 8 tags in the host value
representation, vs. 9 cases in the XDR discriminant) and they are not
the same numbers, due to the u63 / not-u63 case being modeled as a
single low bit in the host value type. So for example "symbol" is tag
4 but in XDR it is denoted by SCV_SYMBOL which has value 5.

The SCObjectType discriminant values are the same as the host object
type codes, and so I could recycle them there (I do in the most recent
iteration of the implementation).

> Some of the types would benefit for some explanation, and in general what is the goal of the representation exposed to the contracts.

This is a fair point and where much of the casual feedback I've got so
far rests. I only have comparatively weak justification for inclusion
or exclusion of many of the XDR types -- as I noted in the CAP the
criteria are fairly subjective, based on "foreseen use in a lot of
contracts". I'm happy to adjust the list of object types, and if you
think the best approach is to err on the side of fewer at first /
leaving things out, that's fine by me.

> Why are there so many integer types?
> I am counting U63, U32, I32 and then U64, I64, BigInt.
> I imagine that the more types we have the more conversions need to be supported.

Well, all of u32, u64, i32 and i64 are present in every source
programming language we're dealing with and also occur freely in a
variety of XDR datatypes in the existing protocol. So I don't know
that it makes sense to exclude them as values in the communication
repertoire. And BigInt is completely different, it's not a standard
part of XDR or almost any PL, and it's always provided as a software
library with memory allocation and arbitrary storage requirements and
so forth. But it's useful for high resolution calculations and
cryptography.

The u63 is a size optimization: fitting a positive int64 inside a Val
without allocating an Object. The same issue/question occurs with
Symbol (a small string with a limited 64-character repertoire) vs.
String, or as you mention below BitSet (a small 60-bit set) vs.
overflowing to some putative arbitrary-size-bitset Object. In all
cases the small packed-in-a-Val type is intended to allow small /
common values to traverse the host/guest interface and have guest code
perform basic operations (insert/extract/compare) on the Val form
without allocating an Object and interacting with it at-a-distance
through host function calls.

u63 is visible in the _low level interface_ between guest and host (in
this CAP), but we've been experimenting with hiding it in the
_user-facing SDK_ bindings, transparently switching to an Object
holding int64 in the negative case. It's not completely clear to me
when such transparent-type-changing is good or bad to do: it's
convenient for users in some cases (fewer types and distinctions to
think about) but also obscures a cost difference (some values suddenly
cost more) where users might well be optimizing for cost. It's also
plausible to (say) have the SDK transparently overflow from intNN
types to BigInt, but .. again I think this is something one can
credibly argue that users would prefer _not_ to have done behind their
backs.

Whether to hide or reveal such representation distinctions to users is
more of an SDK issue, though, less a question for this CAP. The
question with respect to this CAP is whether it's _useless_ to provide
this set of options to the SDK -- if there's no point including
something in the interface because no SDK will ever use it at all
(either as a visible case or a hidden optimization).

If you think it'd help with the rationale, I can add a section
discussing the motivation to provide partially-redundant small-value
types as common special cases.

> Why is there a 64 bit bitset type?
> I would expect instead an arbitrary size bitset object (and maybe not even for v1).

Small-value optimization as mentioned above.

> `BigInt` for example: is the goal to expose the actual underlying data so that people can write arbitrary code to manipulate it?

If you mean "underlying data" in the sense of access to bytes making
up host-side BigInt Objects, no, I don't expect guests will want to be
(or be allowed to be) poking at the bytes inside them. I suspect
though you're asking about the SCBigInt and SCBigRat XDR types, which
are not types the user _operates_ on, they're transport encoding for
import/export of data from the host. They are necessary because XDR
has no native big integer type -- if we want users to be able to
import/export big integers at all, we have to define some projection
into XDR. When held in a host representation, though, the bytes will
be inaccessible (they're typically rearranged into arrays of
host-native-word-sized "limbs").

> I was expecting this to be completely opaque, and people would just have to rely on host functions to manipulate or access those properties.

For performing arithmetic or doing any non-IO operations on BigInt /
BigRat host Objects, this is correct. They're just for IO in the XDR
representation.

> "string" vs "opaque"
> Are strings utf8 strings? xdr strings are not great from an interop point of view with other systems and languages (in the existing protocol, we have additional constraints on strings), but if we want those strings to play nice with existing assets, we'll have to use the same rules.

Yeah, good point. I intend them to be UTF-8 only.

> "Price" seems redundant with "BigRat"
> I think the only place where "Price" exists in the current protocol is when we deal with the DEX. I would leave it out for now, and just have a way (if we need to) for people to round a fraction that corresponds to a Price if needed via a host function.

Ok. I don't feel strongly, I just put it in (along with other XDR
types) because it seemed likely ubiquitous in contracts.

> How do we think of "BigRat" vs "double/float"?
> Maybe "BigRat" should not be part of the first version of this and instead be added later depending on what people use in the standard library?

I left it out at first, but then Leigh suggested putting it in. There
is also an open question about whether to allow FP instructions at all
in user code; many people find FP worrying from a predictability
perspective, and there is some work to do in any given WASM
interpreter to ensure determinism. I don't feel strongly, can see
arguments either way.

> "Operation", "OperationResult" and "Transaction" seem quite arbitrary at this stage as well.

These were based on the assumption that contracts would likely be
constructing operations / transactions to submit back to core. If this
is an incorrect assumption, we can leave them out, I don't feel
strongly.

> "AccountID"
> What is the purpose of this one? Is this supposed to be a "StrKey" type of thing or something used to identify the execution context? This may not be the right type in either case.

This was included based on the assumption that contracts would very
likely receive (as invocation information) and sub-key their storage
by account IDs. If not, I am happy to change this also.

-Graydon

Graydon Hoare

unread,
Apr 27, 2022, 4:23:01 PMApr 27
to Nicolas Barry, Stellar Developers
On Wed, Apr 27, 2022 at 1:10 PM Graydon Hoare <gra...@stellar.org> wrote:
> > How do we think of "BigRat" vs "double/float"?
> > Maybe "BigRat" should not be part of the first version of this and instead be added later depending on what people use in the standard library?
>
> I left it out at first, but then Leigh suggested putting it in. There
> is also an open question about whether to allow FP instructions at all
> in user code; many people find FP worrying from a predictability
> perspective, and there is some work to do in any given WASM
> interpreter to ensure determinism. I don't feel strongly, can see
> arguments either way.

Leigh has corrected me here -- this arose from a conversation in which
I misunderstood his use of a floating point type in a prototype
contract, and a further comment on the ambiguity of the previous name
for the type (BigNum) as not being clearly integral. I really don't
care which set of large-number types we include at this stage; I
expect we'll have some more-used by some contracts than others, and
probably several more specialized ones in the future (eg. elliptic
curve fields); I was just trying to anticipate near-term use-cases. As
you mentioned elsewhere concerning prices, a big rational might be
useful for price calculation? I don't mind leaving it out though.

-Graydon

Leigh McCulloch

unread,
Apr 28, 2022, 3:25:29 PMApr 28
to Graydon Hoare, Nicolas Barry, Stellar Developers
In the XDR in CAP-46 there are several unbounded variable length types, SCBigInt contains magnitude<>, SCVec, SCMap, and SCObject's SCO_STRING and SC_BINARY are all unbounded arrays, which means they are valid up to Uint32 max size.

The XDR values used as inputs to transactions are not normally unbounded, otherwise transactions become expensive and impractical to transmit, parse, gossip. Other applications parsing transactions like wallets, Horizon, need the upper limits too.

We should probably specify some upper bound which I think means saying that SCBigInt isn't arbitrary sized, but has some epic size instead.

Or, maybe we should have shadow XDR definitions for transaction inputs such that transaction inputs have a set of bounds that are different to the limitations a contract can serialize to and from internally?

Leigh

--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.

Nicolas Barry

unread,
May 2, 2022, 8:59:11 PMMay 2
to Leigh McCulloch, Graydon Hoare, Stellar Developers
I don't think there was an update on this thread or the CAP.

It seems general consensus is to start with less types, and only add the ones we need as we identify use cases/justification for them, that way we don't block on trying to justify certain choices before we have the full picture.

For example, `LedgerKey` should be removed for now, and be re-added (or whatever is needed) when we specify how read/write sets are specified.

`string` is a funny one - and for the same reason, I would keep it out of the CAP for now. I don't know if we need them and if they need to be utf8 or "Stellar strings" (~ASCII).

On Leigh's question on putting bounds in the XDR: I don't think this is practical or future proof because of recursive types. What we should do is continue with what we've done so far: place a maximum bound on both the binary blob's size and on the recursion limit when allowing nesting.

One topic I wanted to follow up on as it's relevant to the CAP even at this stage is this:
> There is no mechanism for freeing objects. I could clarify in the CAP
> that the intent is to only free all objects allocated by a transaction
> at the end of the transaction, but at present the CAP says nothing
> about transactions. I should also note that there is no way to trace
> the memory graph of a WASM contract to know which objects it's "not
> using". And besides, the execution budget for a smart contract is so
> brief that it would fit within a typical GC period -- at which point
> one might as well just wait until completion and free everything.

Depending on how this is done, that last part may have pretty broad implications:
if what we keep in the list of host objects are all values ever created during a transaction execution, even a moderately complicated function could use up a lot of host objects, which sounds like a possible scalability issue (slow computers can still churn over lots of statements in a short amount of time).

I'm thinking that the number of objects allocated can realistically exceed the number of statements executed by a program.
If I am not mistaken the following code would add 4 new entries per loop iteration if `BigNum` was used:
    while i < m {
        i += 1; // 2 new objects: "1" and "i+1"
        r = (r-1)*i; // 2 new objects "r-1" and "(r-1)*i"
    }


As a consequence, this creates a weird coupling between the number of statements a program can perform and memory consumption limits.

I am especially worried that this could kill off any "crypto implemented as a smart contract" effort, which I think should be allowed (as it allows us to put together proofs of concepts).
At a minimum in the design rationale we should explain why this approach is viable even for moderately complicated computations given some limit (I think we could have relatively high limits for host objects like 10MB per transaction or more as we only have so many transactions executing in parallel). Like: how much memory would we need to perform the kind of math needed in zk/bullet proofs.


Graydon Hoare

unread,
May 19, 2022, 12:09:21 AMMay 19
to Nicolas Barry, Leigh McCulloch, Stellar Developers
I've opened a PR https://github.com/stellar/stellar-protocol/pull/1206

This contains several related changes to CAP-46 that hopefully address
a lot of comments, especially concerns about a lack of clear
requirements that were voiced by David in the protocol meeting. I also
removed most of the host objects that people voiced concerns about --
they're really more appropriate material for a later CAP. I left in
only the most uncontroversial objects: especially those that help
explain and motivate the distinction between host and XDR forms, and
the possibility of using solely immutable objects.

The issue of resource metering / limits remains mostly unaddressed due
to limitations of time in the implementation -- we're getting there
but we're not quite at that topic yet.

One significant part to note is that the host environment
implementation has changed from C++ to Rust in the past couple weeks.
We have been experimenting with Rust and concluded that it's adequate
for this purpose, can interoperate with stellar-core's existing C++
codebase well enough, and allows us significant opportunity for code
reuse between stellar-core and contract-testing contexts in the SDK.

-Graydon

Leigh McCulloch

unread,
May 24, 2022, 10:14:42 PMMay 24
to Graydon Hoare, Nicolas Barry, Stellar Developers
Hi Graydon,

union SCVal switch (SCValType type)
{
...
case SCV_OBJECT:
    SCObject* obj;
... 
};

I think we should change the value for SCV_OBJECT from SCObject* to SCObject. The XDR optional-data (*) is shorthand for a union and unnecessary because when the SCVal union has discriminant SCV_OBJECT, obj will always be set. The optional introduces an extra 4 bytes for every object that gets serialized, which is at least 50% extra size for some object types. The optional also unnecessarily introduces an Option<> in the Rust XDR lib which introduces unnecessary branches and failure paths that shouldn't exist.

I realize we have historically used XDR optionals on cyclic references, however XDR does not require this. Our Rust XDR lib already handles the cyclic reference without needing the optional. Our Go XDR lib also doesn't require the optional either as it introduces indirection on every union arm anyway. If the C++ XDR lib or other XDR libs do not support non-optional cyclic references we should be able to add support to them.

Cheers,
Leigh

Jonathan Jove

unread,
May 25, 2022, 10:38:28 AMMay 25
to Leigh McCulloch, Graydon Hoare, Nicolas Barry, Stellar Developers
The C++ library xdrpp doesn't support non-optional cyclic references. This came up in CAP-0023 (see the definition of ClaimPredicate specifically CLAIM_PREDICATE_NOT). It would be nice to fix this issue.

Leigh McCulloch

unread,
May 25, 2022, 12:05:19 PMMay 25
to Jonathan Jove, Graydon Hoare, Nicolas Barry, Stellar Developers
Alternatively, could we remove the cyclic type, by removing SCO_BOX? This object type seems not ideal. Storing val's in object boxes requires more host fns, and more host fn calls. Anywhere we have a val we should use a val rather than store it in an object box.

We can add SCO_BOX later on if we really really need it and deal with handling the cyclic ref in the C++ at that time.

David Mazieres

unread,
May 25, 2022, 1:30:37 PMMay 25
to Jonathan Jove, Leigh McCulloch, Graydon Hoare, Nicolas Barry, Stellar Developers
"'Jonathan Jove' via Stellar Developers" <stell...@googlegroups.com>
writes:

> The C++ library xdrpp doesn't support non-optional cyclic references. This
> came up in CAP-0023
> <https://github.com/stellar/stellar-protocol/blob/master/core/cap-0023.md> (see
> the definition of ClaimPredicate specifically CLAIM_PREDICATE_NOT). It
> would be nice to fix this issue.

To be clear, there are various ways to implement XDR unions:

1. Embed union members inside the union, like a C or C++ union.
However, since a structure cannot contain itself, this prevents
recursion or mutual recursion. This violates RFC4506.

2. Implement all union members as pointers. This means even if you
have an int in a union, you are going to need a separate allocation
to initialize that data structure, as well as malloc bookkeeeping
overhead.

3. Do some kind of adaptive thing.

Currently, xdrpp chooses option #1. However, I recognize that this is
an issue. Therefore, I have implemented #2 with the "-uptr" option to
xdrc in the cxx20 branch (which I will make work with stellar-core
soon). #2 is what most XDR compilers do.

I'm open to #3, but it might require sacrificing separate compilation
(or adding some kind of gross pragmas), as there's no way to detect
typedefs or mutual recursion across XDR files. In particular, two
design points that make sense would be A) embedding small values of less
than 8 bytes directly in the union and using pointers for larger ones,
or B) using pointers only in the cases of recursion. I'm not inherently
opposed to ditching separate compilation--this is how goxdr works,
because for efficiency goxdr benefits from knowing when union
discriminant is a type alias for bool. I'm also happy to entertain
other ideas on this front if it seems like something people want.

There's an ersatz version of #3 possible in the cxx20 branch today,
where you compile some .x files with the -uptr option and some without.
The two interoperate fine, since this only changes the implementation of
union types, not their interface.

David
Reply all
Reply to author
Forward
0 new messages