[PATCH 7/7][RFC] apps: oob echo: add README

[Ben Levinsky]

Add README to discuss the following:
- motivation
- performance
- constrains on APU, RPU
- API

Signed-off-by: Ben Levinsky <ben.le...@xilinx.com>
---
apps/examples/oob_echo/README.md | 174 +++++++++++++++++++++++++++++++++++++++
1 file changed, 174 insertions(+)
create mode 100644 apps/examples/oob_echo/README.md

diff --git a/apps/examples/oob_echo/README.md b/apps/examples/oob_echo/README.md
new file mode 100644
index 0000000..c7a1a27
--- /dev/null
+++ b/apps/examples/oob_echo/README.md
@@ -0,0 +1,174 @@
+### Motivation
+Out of Band (OOB) transmission of large data buffers using RPMsg.
+
+### Performance
+- The relatively small size of RPMsg buffers does not work well with large amounts of data, such as video or radar streams. The current implementation also requires several copy operations for each message sent and received.
+- Some sort of "zero copy" approach is preferable.
+- It would also be good to be able to have DMA-capable devices able to move data directly into a message payload.
+
+### Standardization
+A number of different approaches have been suggested to get higher performance. There are zero copy implementations of the standard RPMsg protocol, and a number of OOB mechanisms proposed. It would be desirable to have a single, well-documented standard mechanism that can be easily reused.
+
+### Requirements
+These are some of the high-level requirements that the implementation must meet.
+
+- Zero-Copy of Payload
+-- The mechanism must allow the sending and receiving applications to read and write directly to and from the payload.
+- Arbitrarily-large Buffers
+-- The application should be able to set the size and number of buffers depending on the system configuration and system requirements, with no limits imposed by the implementation.
+- Adaptable to Constrained Memory Environments
+-- Conversely, the implementation should not set any minimums, so that it can be used in constrained environments,
+
+- Sender-Managed Resources
+--The sender of the message must maintain control of the allocation and use of the message payloads.
+
+- Compatible with existing implementation
+-- This implementation must be source-code compatible and interoperate with the existing implementation.
+
+- One RPMsg endpoint per pool
+-- There is only one RPMsg source and destination endpoint associated with each buffer pool.
+-- *Note: This is not an architectural constraint. It makes the API described below simpler and easier to implement. If the desired system requires support for multiple endpoints, the protocol described below can still be used.*
+
+### Architecture
+The approach suggested here is to maintain a shared memory buffer between the source and destination of the large messages. This buffer is divided into slots whose number and size are set at system startup or configuration time.
+When the application needs to send a large buffer, it gets the next available slot, writes the data it wants to send into the slot, flags the message as containing out of band data, and puts the index or address (TBD—based on further analysis and review), of the slot into the packet, along with the length of the data, and sends it via a normal RPMsg call.
+When the recipient gets a message with the flag indicating OOB data, it finds the slot in the buffer, and processes length bytes of data from that slot.
+Note: The buffer does not have to have the same number of slots as the RPMsg ring, it may have fewer, if not all the messages are going to use out-of-band data.
+
+### Protocol
+This adds a new protocol above the basic RPMsg send and receive calls. Currently, the payload of an RPMsg call is just a blob of data (void *). This approach uses the first 4 bytes of the payload as a header. The header contains a flag to indicate whether there is OOB data in the message and a message type. Each message type has it's own structure following the initial type header. For simplicity and convenience each field is 4 bytes long.
+
+#### OOB Data Message
+A data message containing out-of-band data has the following format:
+
+| OOB Flag/DATA | Offset/Index | Length |
+| ------------- | ------------ | ------ |
+
+1. OOB Flag/DATA – A flag indicating the message contains OOB data, and a type indicating that the message contains a pointer to the data.
+2. Offset/Index — Either the offset of the data from the beginning of the shared memory region, or the index of the slot (TBD—depending on further analysis).
+3. Length — The length of the data in the slot.
+
+- Acknowledgment
+An acknowledgment message may be sent when the recipient has finished processing the data. This provides a way for the sender to know when a slot is available for reuse. It has the following format:
+
+| OOB Flag/ACK | Offset/Index |
+| ------------------- | ------------ |
+1. OOB Flag/ACK—Flag indicating that the message contains OOB data and a type indicating that it is an acknowledgment.
+2. Offset/Index—Either the index or the offset of the OOB buffer.
+- Note: The notion of explicit acknowledgments assumes a very general implementation where out-of-order or concurrent processing of messages is possible.
+- Note: In a system with real-time deadlines, the timely receipt of an acknowledgement can serve as a notification that the receiver is meeting its timing requirements.
+
+#### Start-Up
+This message is sent whenever the OOB message passing system is initialized. The recipient can use this as an implicit indication that the sender is restarting the communication. It has the following format:
+
+| OOB Flag/INIT | Count | Address/ID | Size |
+| ------------- | ----- | ---------- | ---- |
+
+1. OOB Flag/INIT—Flag indicating that the message relates to OOB data, and a type indicating that it is an initialization message.
+2. Address/ID—The address or an identifier for the shared memory buffer to be used for OOB data. Note that which will be used is TBD.
+3. Count—The number of slots in the buffer.
+4. Size—The size of each slot in the buffer.
+- Question: Does this message need an acknowledgment?
+
+#### In-Band Data Message
+An in-band data message has a 0 in the header field, indicating that there is no OOB data, and no special processing to be done.
+
+| 0 | Data | ... |
+| - | ---- | --- |
+
+
+### Error Handling and Application Restart
+The protocol needs to be extended to deal with error handling and restart of the sending or receiving applications.
+
+### Protocol State Diagram(s)
+TBD.
+
+### System Configuration Considerations
+Reserve additional shared memory for buffers
+This approach requires additional shared memory for the OOB data storage, in addition to the memory used for the RPMsg data/vrings.
+
+##### Memory must be contiguous
+Since this memory is accessed from more than one CPU, some of which might not have an MMU, the shared memory for OOB data must be physically contiguous.
+
+##### One OOB buffer for each sender
+For each message sender, a separate OOB data buffer must be allocated. Note that this approach could allow for a single sender to use OOB messages for several different endpoints, as long as they can "see" the shared memory.
+
+##### RPU Constraints
+Buffer space must be withiInin the physical address space of the RPU
+##### APU Constraints
+The OS must provide a user-level mapping of buffer space to a process.
+
+
+# API
+This is a very high-level description of the proposed API for managing and sending buffers.
+
+##### Data Structures
+These are the immediately visible data structures associated with each message flow.
+
+buffers.
+
+ - Buffer Pool
+This is the contiguous shared memory area referred to above. It contains a fixed number of fixed-length buffers.
+
+- Free Buffer List
+This is some sort of storage that keeps track of which buffers in the pool are free or in use.
+
+##### Buffer Handling
+These are the minimal set of interfaces needed to set up and maintain the buffer pool.
+
+- Initialization
+Initialize the buffer pool.
+
+- Inputs
+Address, size, number of elements, source endpoint, destination endpoint.
+
+- Outputs
+Returns a buffer id or an error code
+
+- Get a Buffer
+Gets a buffer from the pool.
+
+- Inputs
+takes no arguments
+
+- Outputs
+returns a buffer index or error code (e.g. there are no free buffers)
+
+- Put a Buffer
+Returns a buffer to the pool.
+
+- Inputs
+The index of the buffer to be returned to the pool.
+
+- Outputs
+-- Returns 0 for success or an error on invalid index or index already free.
+- Notes
+-- This interface would typically be called by the sending application on receipt of an acknowledgment message from the remote end.
+
+
+
+##### Message Sending
+These are the interfaces required to send messages.
+
+- Send a Buffer
+This interface sends an OOBData message with the buffer index to the remote end.
+-- Inputs
+Takes a pool, and index, and a length.
+-- Outputs
+Returns 0 for success or an error code. (e.g. invalid index or failure of the underlying protocol)
+
+- Send an In-Band Data Message
+Sends a data type message with the specified buffer and length.
+-- Inputs
+Takes a pool, a message, and a length.
+-- Outputs
+Returns 0 for success or an error code.
+- Acknowledge a buffer
+Sends an acknowledgment message.
+-- Inputs
+Takes a pool, and a buffer index.
+-- Outputs
+Returns 0 for success or an error code.
+
+
+
--
2.7.4