SEP draft for Dynamic Asset Metadata

[Orbit Lens]

Draft for #96 (SEP for Issuing large numbers of Assets). Pull request: https://github.com/stellar/stellar-protocol/pull/183

========================

Preamble

SEP: <to be assigned>

Title: Dynamic Asset Metadata

Author: OrbitLens <orbit...@gmail.com>, Paul Tiplady <pa...@qwil.com>

Status: Draft

Created: 2018-09-30

Updated: 2018-09-30

Version 0.1.0

Simple Summary

This SEP extends SEP-0001 and adds the support of the dynamic asset metadata resolution. It describes a standard way to query asset metadata, thereby allowing the issuer to deal with an unlimited number of assets without defining each of them in stellar.toml file.

Motivation

Current SEP-0001 specification works excellent for issuers that manage only a few assets. However, stellar.toml has a size limit of 100 KB, effectively reducing the maximum possible number of assets to ~300 per file. At present time there is no standard way for the anchor to provide metadata for a large number of assets issued by the same account. There are plenty of use-cases that require thousands of separate assets: bonds, securities, futures, non-fungible tokens.

Specification

To allow dynamic asset properties resolution, the issuer implements described REST API endpoints and advertises the existence of a Dynamic Asset Metadata Service through the stellar.toml file. Top-level parameter ASSET_METADATA_SERVER should contain a fully-qualified URL of a metadata resolution service.

Example of stellar.toml:

ASSET_METADATA_SERVER="https://anchor.com/assets"

...

Asset Metadata Resolution Flow

• The client discovers the home_domain for the asset issuing account and downloads stellar.toml.
• Once the file is downloaded and parsed, the client searches for the asset by its code in [[CURRENCIES]] section.
• If the metadata for the asset was not found in [[CURRENCIES]] section, the client checks for the top-level ASSET_METADATA_SERVER parameter.
• If present, the client pulls a metadata from the remote server using the URL defined in ASSET_METADATA_SERVER parameter.

API Endpoints

Dynamic Asset Metadata Service exposes two REST API endpoints.

Resolve Asset Metadata Endpoint

GET <ASSET_METADATA_SERVER>/<ASSET_CODE>




This API endpoint returns the metadata for a particular asset. The result of the invocation should contain asset metadata following the parameter naming convention described in SEP-0001 for the [[CURRENCIES]] section.

Request

Request Parameters:

• asset_code - The code of an asset to look for.

Example: GET https://anchor.com/assets/AED




Response

On success, the endpoint should return 200 OK HTTP status code and TOML/JSON-encoded object containing asset metadata. Depending on the value of the "Accept" request header the endpoint should return the response in JSON format for "application/json" value or TOML format for "application/toml" and "*/*" wildcard (default).

Example:

code="AED"

name="United Arab Emirates Dirham"

issuer="GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWC6L7U"

display_decimals=2

List All Issued Assets Endpoint




GET <ASSET_METADATA_SERVER>

This API endpoint returns the list with metadata for all assets issued by the account. It follows Horizon REST API format convention.

Request

Request Parameters:

• cursor - Asset code from which to continue the search (referred also as paging_token in a result set).
• order - Results ordering - "asc"(default) or "desc".
• limit - Data page size. Default 20, maximum 200.

Example: GET https://anchor.com/assets/?cursor=BOB&order=asc&limit=2

Response

On success, the endpoint should return 200 OK HTTP status code and JSON-encoded object containing the list of the issued assets' metadata. A response result should contain records and navigation links following Horizon response convention.

Example:

{

  "_links": {

    "self": {

      "href": "https://anchor.com/assets/?cursor=BOB&order=asc&limit=2"

    },

    "prev": {

      "href": "https://anchor.com/assets/?cursor=BRL&order=desc&limit=2"

    },

    "next": {

      "href": "https://anchor.com/assets/?cursor=BSD&order=asc&limit=2"

    }

  },

  "_embedded": {

    "records": [

      {

        "code": "BRL",

        "issuer": "GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWC6L7U"

        "name": "Brazil Real",

        "paging_token": "BRL"

      },

      {

        "code": "BSD",

        "issuer": "GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWC6L7U"

        "name": "Bahamas Dollar",

        "paging_token": "BSD"

      }

    ]

  }

}

CORS headers

In order to comply with browser cross-origin access policies, the service should provide wildcard CORS response HTTP header. The following HTTP header must be set for both API endpoints:

Access-Control-Allow-Origin: *

Rationale

The "asset metadata" endpoint should support both TOML and JSON format to simplify its usage with different client types. Response in TOML format by default is intended to allow using the same parsing logic for both stellar.toml and API response.

The "list all" endpoint provides a convenient interface for exchanges, ledger explorers, infrastructure apps, and external services.

Discussion

• Should we enable multiple issuer accounts? It somewhat complicates the API (GET https://anchor.com/assets/AED-GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWC6L7U) but allows usage of the same server for all assets regardless of the issuer.
• Supporting both TOML and JSON format looks overcomplicated. Nevertheless using only TOML makes integration with external services more complicated (JSON is a clear winner here).

[Jed McCaleb]

This makes to me. Do people see a better way to solve this problem?

Jeremy's comment on the PR:

"I think that doing a signed merkle tree (depth tags and perhaps chunked leaf nodes) might be better because then you can do queries against untrusted edge servers to get proofs of the asset.

You can also do a merkle trie so you get higher assurance that you can't include conflicting entries"

Seems like that maybe over complicates it? It also seems like we could add that later.

--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.
To post to this group, send email to stell...@googlegroups.com.
Visit this group at https://groups.google.com/group/stellar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/6f7a1029-406d-4ce4-ab51-45a93d1a150b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Rubin]

I don't think that the proposed solution covers all bases.

1st off, the data itself should include a signature from the issuer's account asserting the validity of the metadata.

2nd off, we should have some way of checking consistency of the data provided across servers

3rd off, we should have a metadata TTL of some sort so that entries are cached appropriately & consistently

My proposed use of a signed hash tree/trie covers the first two fairly easily, the third is covered by specifying a ttl in the toml where the root is.

Having these authenticity checks is going to save headache down the line, when misconfigured servers accidentally serve stale data, data for a different anchor, etc. I don't think it adds much complexity to the API that gets exposed to the user at the API level, because we can hide the checking of these proofs in the library.

To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/CALGT9GMGEJ2yLwgjRmtv6_7Zs0vAF5n%2BsZjhQRwUWpVXLhBe%2BQ%40mail.gmail.com.

[Orbit Lens]

While I certainly agree with the extended validation approach proposed by Jeremy, I think that the implementation may require a significant amount of time. There are a lot of high priority protocol features, so spending time to update all SDKs and preparing reference code might just not worth it. All existing anchors have to be upgraded as well to support new verification format.

As Jed mentioned, we can always extend this SEP in the future, adding advanced validation features. 

Btw, if the standard of the anchor metadata discovery is going to be updated in the future to support extended validation, it would be great to add something like DKIF verification for stellar.toml, proposed by @Cajga (https://galactictalk.org/d/789-domainkeys-identified-federation-dkif).

[Tom Quisel]

The proposal makes sense to me! I like the idea of only supporting JSON for the responses to simplify. Can anyone think of a case where developers would prefer TOML?

Jeremy's idea of including a signature sounds like the right approach, although I agree that it'd complicate the implementation and probably shouldn't be in the first version. I also feel pretty good about the integrity already. The API should be served over HTTPS, and the API's URL comes from the asset's HOME_DOMAIN's stellar.toml file, also served over HTTPS. That gives a chain of trust back to the asset's issuing account. A breach where a SSL/TLS certificate is compromised seems to be at about the same level as the proposed signing server getting compromised, so I'm not sure adding the signature ups the security profile much.

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/CAJxYNoaTvq%2B73A-EPM4D6yuoFUfqxbQCSPk74QDTEry%2BHtr88A%40mail.gmail.com.

[Orbit Lens]

I updated my SEP proposal for Dynamic Assets Metadata Service and published a reference implementation for it (https://github.com/orbitlens/stellar-dynamic-asset-metadata-server).

Changes:

• Simplified API - single URL endpoint for all operations.
  
• Multiple issuer accounts support.
• Search by asset code and/or asset issuer.
• Only JSON output format.

Pull request for updated SEP draft: https://github.com/stellar/stellar-protocol/pull/224

Thanks to Tom and Jeremy for comments.

[George Kudrayvtsev]

Hi everyone, please welcome this thread back from the dead.

I'm bringing this thread back to life in light of today's asset-heavy world. I don't expect many of the old participants to join back in, but am hoping to field new discussion relevant to the Stellar network of today. With NFTs on Stellar gaining traction thanks to the low fees and native asset issuance mechanisms alongside SEP-1 being the ecosystem's trampoline into asset interoperability, the community would like to see SEP-14 ("Dynamic Asset Metadata") brought back into the fold [1].

A lot has changed since the initial SEP-14 was drafted, and I'd like to propose some changes that would simplify the standard / implementation.

To refresh everyone's memory, the SEP essentially proposed outsourcing the [[CURRENCIES]] field from SEP-1 into its own "asset metadata" server to overcome the stellar.toml file-size limitations of 100KB [2]. This would be a fully-qualified URL living in a top-level ASSET_METADATA_SERVER field on one's stellar.toml file, and it would serve JSON akin to Horizon's existing /assets endpoint.


The main complexity from SEP-14 I'd like to eliminate is the REST API. The current proposal [3] mandates that the ASSET_METADATA_SERVER server support filtering the currency metadata. While this is a nice feature, it should by no means be required. Asset issuers may not have "programmability control" over their hosting services (e.g. static websites, IPFS hosting, marketplace sub-domains, etc.), so this is artificially limiting.

In my view, we can simply have:

    ASSET_METADATA=https://example.com/.well-known/CURRENCIES.toml

This would correspond directly to a valid SEP-1 [[CURRENCIES]] entry [4]. No filtering, no API server, no JSON, just an "external link" to a TOML table. I don't think there's a need to make this a RESTful server unless there is direct ecosystem need. If there was, I think we could easily do it in a backwards-compatible way on top of this.

A second point, and one that was mentioning in the ancient thread, is metadata integrity. In my mind, there are two ways to approach this:

  1. Include an ASSET_METADATA_HASH field which is simply a SHA on the contents referenced by ASSET_METADATA.

  2. Define a separate SEP that lets issuers create proper Merkle-tree-like chains of integrity starting from a data entry on the issuing account and propagating to an arbitrary array of asset metadata.

In my view, (1) is brittle, as adding new assets would require updates to the top-level stellar.toml file, and there's no "audit trail" of why the metadata hash has changed. (2) is far more robust and moves the onus of integrity into the scope of another SEP, isolating the responsibility and scope of this one.

All comments on the above are appreciated! I will try to draft a PR describing the changes above shortly, though I hope my described changes above are clear.

Cheers,

-George

[John Wooten]

Sounds like a very intuitive first step to making dynamic asset metadata a reality - would love to see a PR. This is mission-critical for anchors that use the same issuer address across assets for consistency. Great call separating Merkle-like structures - def. could use just a basic hash reference to start.

Well done 

- John

--
You received this message because you are subscribed to the Google Groups "Stellar Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stellar-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/c7fa077a-9821-453f-a5eb-4f0e43e39406n%40googlegroups.com.

[Leigh McCulloch]

Great idea to try and simplify SEP-14 to see if we can solve the same problem without asset issuers needing to host a programmatic API. If we can do this with static files it definitely broadens the scope of deployment options and builds on the infrastructure asset issuers are already using.

I think a single CURRENCIES.toml file will face similar problems that are experienced today. Clients will need to download a really large file.

SEP-1 has this wonderful, but underused, feature where currencies can be stored in separate files. For example:

A .well-known/stellar.toml can contain:

VERSION="2.0.0"
NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015" 

 

[DOCUMENTATION]
ORG_NAME="Organization Name" 

 

[[CURRENCIES]]
toml="https://DOMAIN/.well-known/ABC.toml"

[[CURRENCIES]]
toml="https://DOMAIN/.well-known/DEF.toml"

And then the currency information can be stored in .well-known/ABC.toml and contain:

[[CURRENCIES]]
code="ABC"
issuer="GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWA6L7U"
display_decimals=7

This might reduce the main stellar.toml file somewhat, although if an asset issuer is needing to issue 20,000 assets, that will still result in a stellar.toml that is over 1MB in size.

If we added support for imports/includes to the stellar.toml, we could essentially paginate any stellar.toml data. For example:

A .well-known/stellar.toml can contain:

VERSION="2.0.0"
NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"  

INCLUDES=[ 

    "https://DOMAIN/.well-known/CURRENCIES_PAGE_1.toml", 

    "https://DOMAIN/.well-known/CURRENCIES_PAGE_2.toml",

]

 

[DOCUMENTATION]
ORG_NAME="Organization Name" 

And then the included toml at .well-known/CURRENCIES_PAGE_1.toml can contain:

[[CURRENCIES]]
toml="https://DOMAIN/.well-known/ABC.toml" 

 

[[CURRENCIES]]
toml="https://DOMAIN/.well-known/DEF.toml" 

And then the currency information can be stored in .well-known/ABC.toml

[[CURRENCIES]]
code="ABC"
issuer="GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWA6L7U"
display_decimals=7

There are downsides with this similar to a programmatic API would introduce. Clients must protect themselves from recursive includes. Not all clients might implement this, so stellar.toml's should try and keep critical information in the root as many do today.

Would this achieve a similar end?

To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/1827339852.1330676.1645196230109%40privateemail.com.

[George Kudrayvtsev]

Hi Leigh, thanks for your reply!

Great point about the existing "link to currency" feature. It's mentioned in the original issue I brought up [1], but is definitely worth resurfacing here. It's a mitigation people can use now while the churn around this proposal gets settled. A concern in said issue was that this may simply not scale enough? The stellar.toml limit is 100KiB, and if we use an "average template" of

[[CURRENCIES]]
toml="https://example.com/.well-known/ASSETCODE.toml"

then napkin math tells us that this scales to around ~14000 currency entries. Is that enough? I can easily see that being exceeded, especially if that someone is a marketplace or platform rather than an individual issuer.

In regards to generalizing SEP-14 to broadly support imports and includes, I think that pollutes the scope a bit much. However, I do like the idea of pagination to mitigate the "one large file" concern. We could expand the ASSET_METADATA field to be an array of URLs:

ASSET_METADATA = [

    "https://example.com/.well-known/CURRENCIES_1.toml",

    "https://example.com/.well-known/CURRENCIES_2.toml"

]

and the (optional?) ASSET_METADATA_HASH would be a corresponding array of hashes on the content. We could also make it a table in and of itself:

[[ASSET_METADATA]]

url = "https://example.com/.well-known/CURRENCIES_1.toml"

hash = "deadbeef"

[[ASSET_METADATA]]

url = "https://example.com/.well-known/CURRENCIES_2.toml"

hash = "cafebabe"

The latter seems more extensible, but is also far more verbose for something that we're trying to keep simple and straightforward. The former cleanly solves the pagination issue with a simple array. Thoughts?

-George

[Leigh McCulloch]

Scoping the contents of the referenced TOMLs seems like a good idea. Unbounded includes would be problematic for clients as they wouldn't know if they needed to parse linked TOMLs for their use case. Using your approach where includes contain specific information, clients only need to load the currencies TOMLs if they need to know about currencies.

I'd use the term CURRENCIES_TOMLS instead of ASSET_META_DATA, for consistency with the name of data sections that will be supported in it (CURRENCIES), and to follow suit with using the toml term for referencing another toml file like stellar.toml's already offer.

The simpler version is sufficient without hashes. These TOMLs are likely stored at the same host, and so any hash stored in the stellar.toml offers little security since both files would be hosted together. Even if not hosted together it's not clear what thread vector the hash will meaningfully protect against.

CURRENCIES_TOMLS = [

    "https://example.com/.well-known/CURRENCIES_1.toml",
    "https://example.com/.well-known/CURRENCIES_2.toml"
]

It'd be really interesting to hear from the original authors of SEP-14 to know if this would address the original problem or if it's missing anything.

To view this discussion on the web visit https://groups.google.com/d/msgid/stellar-dev/063e783f-6405-4009-8680-5d0ea7e64586n%40googlegroups.com.

[John Wooten]

Thank you, Leigh.

That clears up a lot. You hit it on the head with "wonderful, but underused." Will give it a shot

On a related note:

What if we replaced [[CURRENCIES]] with [[ASSETS]]?

I get Stellar started out anchored in cross-border stablecoins, but the network's expanded greatly to a wide range of diverse assets — perhaps we ought reference them so

[George Kudrayvtsev]

John, while I completely agree that [[CURRENCIES]] is outdated nomenclature at this point, it seems it'd be way too big of a break in terms of ecosystem compatibility if we replaced it with [[ASSETS]]. I think we're stuck with what we've got, though I'd love to be wrong.

Leigh, regarding CURRENCIES_TOMLS, I think that'd be fine, though it's a multi-plural so I'd prefer CURRENCY_TOMLS.

Regarding this comment:

> These TOMLs are likely stored at the same host, and so any hash stored in the stellar.toml offers little security since both files would be hosted together. Even if not hosted together it's not clear what thread vector the hash will meaningfully protect against.

The hash is intended to protect against a very specific threat vector that I think is important to accommodate: asset metadata changes after issuance. If we want to support NFTs in a meaningful way (and not just generic token-represented digital goods), we need to ensure that the toml file that describes a token cannot transparently change after the fact (or that such a change is at least traceable).

However, this integrity check probably belongs in a different place. For example, marketplaces could put the hash of the asset's [[CURRENCIES]] entry at the time of exchange into the transaction memo. Or this could be a property of the entry itself; I'm not sure. Those are just an off-the-cuff thoughts; the larger issue is around "asset meaning at purchase time." What good is using a toml file to describe a token if its description can change after you buy  it?

Ultimately, though, I think I agree that we should defer on that component. It offers little as presented here, and solving that problem needs a SEParate (heh) proposal.

I've created a PR with my proposed changes at stellar/stellar-protocol#1143.

Please have a look!

-George

[tomer]

Hi, this thread is too long for it's own good :) 

Given Orbit's well articulated reasoning for this approach and his willingness to support this long term in stellar.expert, we will be changing the status to Active. 

[John Wooten]

Perhaps a more straightforward solution is to amend the 100KB stellar.toml limit. What is the logic behind this?

In terms of issuing thousands of assets, I think code_template formatting covers a lot of the exceedingly numerous asset-issuing scenarios, at least in finance.

Furthermore, consider the single-reference toml solution for high-volume "marketplace platforms:"

A simple remedy is to have each entry in the "14000 currency entries" on the stellar.toml reference a toml with 99KB of asset data (can enforce 100KB or any cap at runtime), which should get you circa 100k relatively complex assets.

For instance, .well-known/stellar.toml can contain

|        [[CURRENCIES]]

|        toml="https://DOMAIN/assets/green-produce.toml"

And the currency information can be stored in assets/green-produce.toml and contain:

|        [[CURRENCIES]]

|        code="peas"
|        issuer="..."

|

|        [[CURRENCIES]]

|        code="green peppers"
|        issuer="..."

|

|        [[CURRENCIES]]

...

This works because Leigh's schema from Feb 22 uses [[CURRENCIES]] in "sub-tomls."

Notwithstanding, even single-depth toml referencing currently doesn't work with stellar.expert.

It seems like single-depth toml references for asset metadata would be an easy first step to more scalable issuer records. This could potentially just be a new version of SEP1 adding a toml field to the public-facing material.