Re: Nostrscript/Nostr Extensions spec and ABI

[William Casarin]

On Mon, Jul 10, 2023 at 01:04:14PM +0200, Melvin Carvalho wrote:
>po 10. 7. 2023 v 9:28 odesílatel Semisol <h...@semisol.dev> napsal:
>
>> Starting this thread so that the ABI and other details for
>> Nostrscript/Nostr Extensions can be discussed. Here’s my thoughts:
>>
>> # Naming
>> “Nostr Extensions” is probably better than “Nostrscript” since it
>> represents what this actually does (extends a client), and the name
>> Nostrscript could be confusing since it could be interpreted as a
>> programming language.
>>
>> # Goals
>> - Allow for easily extending Nostr clients.
>> - Compatibility: Extensions should work on any platform: the web, mobile,
>> desktop, etc.
>> - Modularity: Not every feature has to be implemented by a client or
>> extension.
>> - Extensibility: It should be easy to add new capabilities for extensions,
>> such as custom feeds
>>
>> I’ll post my draft ABI later once I fully polish it.
>>
>
>Hi Semisol, thanks for sharing
>
>For those less familiar with nostrscript (or nostr!), there's some search
>results here:
>
>https://primal.net/search/nostrscript
>https://nostr.band/?q=%23nostrscript
>
>Aside: Primal also searches the fediverse, and is in general a pretty
>decent search experience (for news etc)
>
>From one of Will's original posts:
>
>web → javascript → 🌎🚀🌑
>nostr → nostrscript → 🤔💭❓

I like the name nostrscript for the branding, even if it is not
technically a script, right now in damus it's synonymous with
AssemblyScript + a custom nostr lib that communicates with damus.

I expose a few functions that scripts can call into the host:

```
enum Command {
POOL_SEND = 1,
ADD_RELAY = 2,
EVENT_AWAIT = 3,
EVENT_GET_TYPE = 4,
EVENT_GET_NOTE = 5,
NOTE_GET_KIND = 6,
NOTE_GET_CONTENT = 7,
NOTE_GET_CONTENT_LENGTH = 8,
}

declare function nostr_log(log: string): void;
declare function nostr_cmd(cmd: i32, val: i32, len: i32): i32;
declare function nostr_pool_send_to(req: string, req_len: i32, to: string, to_len: i32): void;
declare function nostr_set_bool(key: string, key_len: i32, val: i32): i32;
```

I've been thinking about changing this entire interface to just:

```
declare function nostr_cmd(json: i32, json_len: i32): i32;
```

where commands are serialized using json. This wouldn't be the best for
performance but would be nice for flexibility and avoiding the pain of
ABI between wasm hosts and guests.

Another alternative is something like what I have now:

```
declare function nostr_cmd(cmd: i32, val: i32, len: i32): i32;
```

Which provides a simple interface for sending a command (enum i32), a
string and the string length. This works well for many types of commands
but not all, which is why I have one-off functions for things that don't
fit that model.

I think reducing the surface area of the calls is a good thing for
implementation ease and future extensibility.

I haven't spent too much time thinking about this, it's just what I have
at the moment. Curious to see what others come up with.

Cheers,
Will

[William Casarin]

On Mon, Jul 10, 2023 at 08:29:44PM +0300, Semisol wrote:
>> On 10 Jul 2023, at 18:24, William Casarin <jb...@jb55.com> wrote:
>>
>> On Mon, Jul 10, 2023 at 06:19:34PM +0300, Semisol wrote:
>>>
>>> … snipped …
>>> My ABI exposes one import per command (I recommend you read it if you did not yet), which has some advantages, compared to a single interface:
>>>
>>> * You can know what a module can do beforehand, and modules that use
>>> unsupported features will never be able to load, which helps
>>> simplify development
>>> * You still get to retain the typing on arguments
>>
>> Can you link or ideally mail it as a patch(set) for review? Then I would
>> be able to respond inline here to comment on it.
>Here you go:
># Types
>- `u32`: Unsigned 32 bit integer.
>- `i32`: Signed 32 bit integer.
>- `u64`: Unsigned 64 bit integer.
>- `i64`: Signed 64 bit integer.
>- `usize`: Unsigned size, same as `u32`.

why not just use u32 directly? is this planned to change?

>- `byte[n]`: An amount of bytes of length n.
>- `discard`: Alias for `i32`, the application should discard the value.

This is strange to me, this isn't a wasm thing. wouldn't you just not
push a value on the stack if you don't want a return value?

>NOTE: Unsigned integers must be converted to signed while passing *to* wasm and
>must be converted back to unsigned when received *from* wasm.

why?

># Custom Sections
>The custom section `nostr` SHOULD be present on a module, which is a JSON object containing the following properties:
>- `name`: Module name
>- `version`: Module version as SemVer

I never bothered with this because I figured this meta information could
be included in a new nostr note type so that it can be covered by
publisher's signature. Then a hash of the wasm binary could be signed,
with the blob hosted elsewhere or in a note itself if its small enough.

># Structs
>
>### ByteBuf
>- `usize`: length
>- `byte[length]`: the actual data
>
>### String
>- `usize`: length
>- `byte[length * 2]`: string data, encoded in UTF-16 LE

I absolutely hate the utf-16 representation but it seems assemblyscript
forces it on us due to legacy javascript reasons =/

I would create my own script that doesn't make strings like this if I
had the time.

>Adds timer functionality.
>
>### Export: `nostr_timer__tick(): u32`
>If this function exists, this should be called every second.
>Returns 0 if it succeeded, 1 otherwise.

Do you have examples on what you would use this for?

>
>## nostr_persist
>
>Adds persistence functionality.
>
>### Import: `nostr_persist__persist_ephemeral(usize ptr): discard`
>The application should call this function when it wants to persist "ephemeral" data, such as state, through a reinstantiation, but not through an app reload/restart. This data should not be persisted after the app restarts or the module is restarted.
>
>### Export: `nostr_persist__load_ephemeral(usize ptr): u32`
>Loads the given `ByteBuf` that was previously persisted using persist_ephemeral upon module instantiation.
>Returns 0 if it succeeded, 1 otherwise.
>
>### Import: `nostr_persist__persist(usize ptr): discard`
>The application should call this function when it wants to persist data. The ptr should point to a ByteBuf.
>
>### Export: `nostr_persist__load(usize ptr): u32`
>Loads the given `ByteBuf` that was previously persisted using persist upon module instantiation.
>Returns 0 if it succeeded, 1 otherwise.

Do you have examples that use these? just trying to understand what this
is for

>## nostr_filter
>
>Adds filtering functionality.
>
>### Export: `nostr_filter__filter(Event event): u32`
>Should return 0 if this post should not be filtered, any other value otherwise. The value is used to convey why filtering was done.
>- `1`: Generic
>- `2`: Spam

nice! I guess the app can just detect this export and assume the module has filtering capabilities?

[Semisol]

> On 15 Jul 2023, at 21:10, William Casarin <jb...@jb55.com> wrote:

>
> On Mon, Jul 10, 2023 at 08:29:44PM +0300, Semisol wrote:
>>>> On 10 Jul 2023, at 18:24, William Casarin <jb...@jb55.com> wrote:
>>>
>>> On Mon, Jul 10, 2023 at 06:19:34PM +0300, Semisol wrote:
>>>>
>>>> … snipped …
>>>> My ABI exposes one import per command (I recommend you read it if you did not yet), which has some advantages, compared to a single interface:
>>>>
>>>> * You can know what a module can do beforehand, and modules that use
>>>> unsupported features will never be able to load, which helps
>>>> simplify development
>>>> * You still get to retain the typing on arguments
>>>
>>> Can you link or ideally mail it as a patch(set) for review? Then I would
>>> be able to respond inline here to comment on it.
>> Here you go:
>> # Types
>> - `u32`: Unsigned 32 bit integer.
>> - `i32`: Signed 32 bit integer.
>> - `u64`: Unsigned 64 bit integer.
>> - `i64`: Signed 64 bit integer.
>> - `usize`: Unsigned size, same as `u32`.
>
> why not just use u32 directly? is this planned to change?

The goal here was to in the future be able to adapt to Wasm64. It also helps with clarification of what is memory related

>> - `byte[n]`: An amount of bytes of length n.
>> - `discard`: Alias for `i32`, the application should discard the value.
>
> This is strange to me, this isn't a wasm thing. wouldn't you just not
> push a value on the stack if you don't want a return value?

Ah yeah I’ll remove that

>> NOTE: Unsigned integers must be converted to signed while passing *to* wasm and
>> must be converted back to unsigned when received *from* wasm.
>
> why?

Unsigned integers don’t exist in wasm when returning, so it has to be manually converted

>> # Custom Sections
>> The custom section `nostr` SHOULD be present on a module, which is a JSON object containing the following properties:
>> - `name`: Module name
>> - `version`: Module version as SemVer
>
> I never bothered with this because I figured this meta information could
> be included in a new nostr note type so that it can be covered by
> publisher's signature. Then a hash of the wasm binary could be signed,
> with the blob hosted elsewhere or in a note itself if its small enough.

I think having it be self contained is better than having to look up a note for this information. You can easily load from a file and don’t have to publish a note

>> # Structs
>>
>> ### ByteBuf
>> - `usize`: length
>> - `byte[length]`: the actual data
>>
>> ### String
>> - `usize`: length
>> - `byte[length * 2]`: string data, encoded in UTF-16 LE
>
> I absolutely hate the utf-16 representation but it seems assemblyscript
> forces it on us due to legacy javascript reasons =/
>
> I would create my own script that doesn't make strings like this if I
> had the time.

AS strings are just another class so you can implement your own string class

>> Adds timer functionality.
>>
>> ### Export: `nostr_timer__tick(): u32`
>> If this function exists, this should be called every second.
>> Returns 0 if it succeeded, 1 otherwise.
>
> Do you have examples on what you would use this for?

You can probably clean up some temporary state with this, but it isn’t that necessary so I will remove it.

>>
>> ## nostr_persist
>>
>> Adds persistence functionality.
>>
>> ### Import: `nostr_persist__persist_ephemeral(usize ptr): discard`
>> The application should call this function when it wants to persist "ephemeral" data, such as state, through a reinstantiation, but not through an app reload/restart. This data should not be persisted after the app restarts or the module is restarted.
>>
>> ### Export: `nostr_persist__load_ephemeral(usize ptr): u32`
>> Loads the given `ByteBuf` that was previously persisted using persist_ephemeral upon module instantiation.
>> Returns 0 if it succeeded, 1 otherwise.
>>
>> ### Import: `nostr_persist__persist(usize ptr): discard`
>> The application should call this function when it wants to persist data. The ptr should point to a ByteBuf.
>>
>> ### Export: `nostr_persist__load(usize ptr): u32`
>> Loads the given `ByteBuf` that was previously persisted using persist upon module instantiation.
>> Returns 0 if it succeeded, 1 otherwise.
>
> Do you have examples that use these? just trying to understand what this
> is for

The ephemeral part is for __reinstantiate, which helps reset the wasm memory since you cannot shrink it right now. This is so that you can transfer say relay connection handles through a reinstantiate.

The persistent part could be used to store state. Currently it’s useful only in a few cases, but in the future I think it would be a good idea to have some way to have module settings etc.

>> ## nostr_filter
>>
>> Adds filtering functionality.
>>
>> ### Export: `nostr_filter__filter(Event event): u32`
>> Should return 0 if this post should not be filtered, any other value otherwise. The value is used to convey why filtering was done.
>> - `1`: Generic
>> - `2`: Spam
>
> nice! I guess the app can just detect this export and assume the module has filtering capabilities?

yeah. the capabilities and permissions a module has is strictly defined by what it imports or exports

[William Casarin]

On Sat, Jul 15, 2023 at 10:01:28PM +0300, Semisol wrote:
>
>> On 15 Jul 2023, at 21:10, William Casarin <jb...@jb55.com> wrote:
>>
>> On Mon, Jul 10, 2023 at 08:29:44PM +0300, Semisol wrote:
>>>>> On 10 Jul 2023, at 18:24, William Casarin <jb...@jb55.com> wrote:
>>>>
>>>> On Mon, Jul 10, 2023 at 06:19:34PM +0300, Semisol wrote:
>>>>>
>>>>> … snipped …
>>>>> My ABI exposes one import per command (I recommend you read it if you did not yet), which has some advantages, compared to a single interface:
>>>>>
>>>>> * You can know what a module can do beforehand, and modules that use
>>>>> unsupported features will never be able to load, which helps
>>>>> simplify development
>>>>> * You still get to retain the typing on arguments
>>>>
>>>> Can you link or ideally mail it as a patch(set) for review? Then I would
>>>> be able to respond inline here to comment on it.
>

>>> NOTE: Unsigned integers must be converted to signed while passing *to* wasm and
>>> must be converted back to unsigned when received *from* wasm.
>>
>> why?
>
>Unsigned integers don’t exist in wasm when returning, so it has to be manually converted

wow you would think writing a wasm interpreter from scratch in C I would
remember this fact. probably because I support it transparently in one
of my stack value enums... thanks xD