ONF Document Type: Technical Recommendation
THIS SPECIFICATION IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE.
Any marks and brands contained herein are the property of their respective owners.
©2020 Open Networking Foundation. All rights reserved.
Open Networking Foundation, the ONF symbol, and OpenFlow are registered trademarks of the Open Networking Foundation, in the United States and/or in other countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or services of their respective owners.
This Technical Recommendations has been approved by the Project TST, but has not been approved by the ONF board. This Technical Recommendation is a new reference implementation document focus on v2.1.2 models, which has been approved under the ONF publishing guidelines for 'Informational' publications that allow Project technical steering teams (TSTs) to authorize publication of Informational documents. The designation of '-info' at the end of the document ID also reflects that the project team (not the ONF board) approved this TR.
Table of Contents
List of Figures
Description of Change
Initial version of the Reference Implementation document on streaming for TAPI v2.1.3
This ONF Technical Recommendation (TR) is a supplement to the Reference Implementation for the TRANSPORT-API (TAPI) [ONF TR-5XX.1].
The target architecture is provided in [ONF TR-5XX.1]. The figure below is a copy of the figure provided in the reference.
on the autonomous flow of information via TAPI from SDN-C to OSS/SDTN and from SDTN to OSS.
Streaming is the name for a type of mechanism that handles the providing of information from one system to another in some form of steady and continuous flow.
In the context of a Management-Control solution streaming is used primarily for the reporting (notification) of ongoing change of state of the controlled system from one Management-Control entity to another (usually superior) management-control entity. In this context as much of the information is derived from readings of instruments the flow is often called telemetry  .
The stream provides engineered flow such that peak load is averaged using some mechanism such as back-pressure and/or selective pruning of detail.
In the following discussion the term Controller will be used for any Management-Control entity (OSS, SDTN, SDN-C, EMS, NMS, Orchestrator etc.).
TAPI can be used in several different applications. The primary application is one where one Controller (provider) is providing an ongoing flow of state updates to a client (superior) Controller, as depicted in the figure above.
In this application the following assumptions apply:
Consequently, the TAPI provider aims to optimize the process of maintaining alignment for the client.
Note that TAPI is not intended to directly support:
This section examines the TAPI streaming capability in detail. Examples of UML and Yang are provided as appropriate.
The characteristics of streaming are described in general and are illustrated using example focusing on alarm reporting.
RESTCONF specifies a traditional form of notification where the assumption is that a relatively short queue of notifications will be available on the provider and alignment with current state (in the case of alarms, alignment with current active alarms) will be achieved by GET via RESTCONF.
An Event source/server streaming mechanism is made available as an alternative to traditional notifications and potentially, over time, as a replacement.
The streaming capability is distinct from TAPI Notification and is designed to better deal with scale and to provide an improve operational approach.
The method proposed allow the client to gain and maintain alignment with current state from the stream alone (with no need to get current state). The client can achieve eventual consistency by simply connecting to the relevant streams. The client will receive an ongoing view of change, assuming that the client is keeping up reasonably with the stream. The stream is designed to allow for some client delay with no loss of information fidelity.
When the client has a significant delay, there will be a loss of fidelity [A2] but no impact on eventual consistency. If the client has a very large delay, then a resync will be initiated by the provider. Resynchronization will be achieved simply by the client reconnecting to the stream from offset zero. This will again allow the client to achieve eventual consistency.
The streaming capability provides a reliable pipeline for reporting of change. This improves the information flow integrity and reduces the need for recovery and resynchronization.
The streaming approach is generally applicable to all information available over TAPI from the provider to client.
The streaming capability also offers an improved alarm structure (focusing on fundamental properties of the alarm and relegating legacy fields).
A management-control system, such as an Orchestrator, high level Controller, OSS etc., has the role achieving intended capability (intent, service etc.) by monitoring and processing information (e.g., alarms) from a managed-controlled system (network) such that the overall assembly of operations systems can determine actions to continue to support intent achieving revenue (via some assessment of services) and identify repair action prioritization (via analysis of problems). This system uses TAPI to acquire from the subordinate controller information from a fragment of the overall network, i.e., the devices monitored by the controller, where that information is positioned in terms of TAPI entities within a TAPI Context.
The client Controller maintains history and live views of the state of the things in the network so as to do the necessary analysis, hence that system uses a mechanism providing autonomous updates and does NOT query the provider Controller for states.
The overall solution is expected to have the following characteristics for the provider controller:
The primary focus for Streaming in TAPI 2.1.3 is simple and efficient ongoing alignment of a Controller (client) with a view presented by another Controller (provider).
The key characteristics of the TAPI Streaming solution:
The interface can offer many streams for a context. The client can determine, using calls on the provider, both the types of stream supported and the available streams that are active for connection and streaming.
This structure allows the provider to report the streams that it can support, regardless of whether they are active or not.
The provider can indicate the entity types supported by a stream, the storage and record strategies.
The segment-size and record-retention are free choices made by the provider depending upon system engineering.
Information may be divided into separate streams. There is no restriction on choice of division of the information into streams. A provider could choose to have a stream per class or to have streams that aggregate classes together that have similar lifecycles etc. It should be noted that for TAPI 2.1.3 ALL instances in the context of any object-class-identifier listed in record-content will be streamed, i.e., there is (intentionally) no filtering. [A5]
The supported-stream-type is also augmented with connection-protocol-details which provides a list of allowed-connection-protocols in freeform string (to be normalized through agreement). Candidate protocols include “websockets” [RFC6455] and “sse” [W3C SSE].
For the compacted log, the supported-stream-type information is augmented with compacted-log-details that includes tombstone [A6] -retention (delete retention) and compaction-delay settings. Both are free choices made by the provider depending upon system engineering.
This structure allows the provider to report the streams that are currently available.
The provider can indicate the
The streaming solution assumes that the provider is delivering information in sequence from a log.
In TAPI 2.1.3 a log approach oriented towards maintaining alignment is provided. The stream mechanism defined allows for different log strategies
This log approach has several key characteristics:
The other log strategies are partially supported. The TAPI 2.1.3 solution can be extended by conditional augmentation as appropriate.
There are two distinct aspects of alignment:
In the described application:
Once the client has identified the available streams to connect to, the client simply acquires the necessary authorization (see later [A12] ) and connects. The following provides a brief sketch of alignment. The process is discussed in far greater detail later in the document (see later ).
The next subsections consider the client connection to and receiving from a stream.
On initial connection, the client provides a null token. This causes the provider to stream from the oldest record. The client can continue to consume records from the stream ongoing.
The initial records received by the client will be for the entities that have not changed for a “long” time.
As the client continues to consume the stream it progresses past the Tombstone (delete) retention point (i.e., is receiving records that have a timestamp that is the less than the Tombstone retention (delay) from the current time), and recent tombstones will be received along with newer changes.
Compaction will remove multiple reports about the same entity, but as the stream progresses further it is possible that an update is received that overwrites previously received entity state or a tombstone is received that deletes an entity that was read earlier. This is where compaction had not yet removed the entity when the stream was started (potentially because the event causing the newer record had not yet occurred).
After some time, the client consumes past the compaction delay point (i.e., is receiving records that have a timestamp that is the less than the compaction delay from the current time). From this point onwards the client is receiving all recent changes and is aligned with network state as it was perceived by the provider at some recent point in time. Whilst beyond the compaction delay point the client will receive all event reports for the context.
If the controlled system stopped changing, then the client would eventually reach the newest record and would be aligned with the provider view of the state of the controlled system
Information fidelity is reduced if the client slips back by more than the compaction delay as compaction will remove some change detail.
The client will be forced to realign if it is delayed by more than the tombstone retention. The behavior of the provider at this point is equivalent to that when there is an initial connection.
The above detail can be summarized:
The stream-record allows for multiple log-records each of which includes a log-record-header and optionally a log-record-body.
The log-record-header provides information common to all records.
The Tombstone record may have only a header.
The log-record-header is as below.
Considering the fields in turn:
The log record body includes generalized content and is augmented with specific content depending upon the value of a control property, record-content.
The log-record-body includes:
It is augmented with an instance of a class. This this allows for any class from the model to be reported.
When determining the cause and effect of any behavior in the controlled system it is necessary to have visibility of relevant state and to know the time of each change of state. The time units must be sufficiently fine to allow all relevant event sequencing to be determined  .
The time of change at the source of change needs to be propagated as data in the stream and hence needs to be in the report of each instance of the thing (and in any stored form that exists between the source and the stream). This is recorded in event-time-stamp.
The time the record was logged is log-append-time-stamp.
The implementation solution may partition the log supporting a stream. This may cause stream content reordering. Clearly, event reordering across different sources is a fundamental behavior due to relativity and differential delay. It is expected that the order for events from each single state machine will be maintained.
Regardless, the timestamp granularity must be sufficient to ensure relevant chronological order can be recovered. This also assumes that robust time synchronization mechanism is present at each event source and
in the controller/controlled system as a whole.
TAPI is a machine-machine interface. As noted earlier, the client maintains a local repository representation of the information from the provider. The information is defined in terms of the context. As the context has been agreed (currently, up front during system design), the context is what the client system wants/needs to see.
Where the client needs less detail than is provided by a specific context the client can:
On this basis:
Any changes in the required information are handled by changes in the context. See later discussion on changing the context and spotlighting
The stream relates to the things in the context. Changes include:
See later on changing the context.
For the compacted log solution, whole entities are streamed on creation, deletion and change of a property. Hence if a single property in a CEP changes, the whole CEP is logged and streamed.
Note: Separation of properties that are slow rate change (config) from properties that are high rate change state and isolating independent state in separate entities is advisable. The alarm and state model provide for streaming allows a separation of small per-state entities from large slow changing config entities. The current TAPI OAM model also isolates properties related to monitoring results from properties related to configuration.
For the solution to operate reliably:
The solution can be tuned to balance pace to achieve consistency (eventual consistency) with the fidelity of information when under stress  .
Considering the supported-stream-type structure discussed earlier, there are two key time settings for the compacted log solution “tombstone-retention” and “compaction-delay”  .
TAPI 2.1.3 also offers a rudimentary capability for monitoring the streams. This allows an external client that has the appropriate capability and authorization to monitor which clients are connected and how their stream is performing.
Where this capability is supported each client, streaming connection is monitored for the id of the last record written to and read from the log. This allows an administrator to get a view of how delayed a client is.
In TAPI 2.1.3 this is for PoC analysis. It is expected that the feature will advance significantly in future releases as a result of experience gained from PoC activity.
Two options are explored, one provides full compaction support, the other uses a more traditional structure to feed a stream and provides a restricted emulation of compaction. Either can be used to support the current TAPI streaming solution sufficiently.
There may be other approaches that provide a suitable capability, i.e., reporting current state and change via whole entity reporting through the same single stream, so as to achieve eventual consistency with current state cost optimized potential for loss of fidelity.
The key consideration is the external behavior and not the specific mechanism used to support it.
The figure below shows a stylized view of a controller controlling a network and providing a stream to a client.
The key features highlighted in the diagram are:
Note that the assumption is that other pipelines are in place (not shown here) throughout the overall flow from device to client to ensure no relevant loss of information from the device and to ensure that the solution is always in a state of eventual consistency with current network state.
This approach uses the same pipeline mechanism but feeds the pipeline from a composite store. This does not offer the full compacted log capability. The behavior is as if the tombstone retention and the compaction delay are set to the same value (simplistically, the end of the log, but strictly any point in the log).
On client connection:
Current state may change during streaming of current state such that a change statement is appended to the truncated log.
Once the provider has sent current state, ensuring that all states that have not changed since connection of the client have been streamed to the client, it would then begin to stream from the truncated log. The truncated log may include some states that have already been streamed. As the client is necessarily designed to be able to deal with repeated statements this will not be an issue.
If the connection to the client is dropped, the client will reconnect providing the token of the last record it fully processed. The provider can code into the token any information that helps it determine what to stream next.
Clearly, the client must be able to retrieve current state in a shorter period than the log truncation time so that changes are not lost. If the log wraps during streaming of current state, the provider will have to restart the alignment (by dropping the connection and by ignoring the client token).
It would not be unreasonable for the provider to shorten the truncated log once alignment has been achieved.
The provider is expected to add a Tombstone record for every delete record.
The provider behavior is feeding the stream is very similar to a traditional behavior (send current in response to get then notify of change, other than the log is of whole entities).
In this realization the provider side does not carry out active compaction and hence does not provide the elegant degraded performance under pressure of the full compacted log approach.
A device raises an alarm once specific criteria have been met and clears the alarm once other specific criteria have been met. The criteria could involve:
The counts can be of the occurrence of other threshold crossing and hence the alarm definition at origin may be very complex.
The following sections work through the key features of an alarm solution as an example of usage of streaming.
The alarm streaming solution has a particular delete/tombstone behavior to provide the best performance. This is highlighted in Section ….
See system engineering below.
The following highlights key design decisions (in the context of justifying/explanatory information)
Note: It would be reasonable to also propagate other processed network property types through the same stream as alarms if the network property has similar characteristics to an alarm. For example, operational state is asymmetric in importance with a normal state and off-normal state where the normal state could be considered as equivalent to a clear.
The condition-detector is as below (descriptions omitted to save space).
Note that the Yang does not show the mandatory fields. The field enforcement will be applied to the Yang in the next TAPI release.
The following values population would be expected:
The condition-detector can be augmented with properties related to alarms as follows:
The alarm-detector-state indicates whether the alarm is ACTIVE, INTERMITTENT or CLEAR. This is the essential state of the alarm
The legacy-properties are provided to deal with the traditional alarm reporting properties. Alarm systems of the 20th century were based primarily on local lamps (initially filament bulbs) and bells. Lamps can only be on or off, and bells sounding or not sounding, so alarms were Boolean in nature. Where a detector was essentially multi-state it was converted into multiple Boolean statements.
The management of the equipments was essentially human only and local only (there were rarely remote systems). The device with the problem was the only possible indicator of importance and it had only three distinct bulbs to illuminate (filament bulbs tend to fail requiring costly replacement).
The devices were relatively simple in function and analysis of the detectors was crude. There was only the device to indicate severity. The device also could provide the best view as to whether a service was impacted, although clearly it had almost no knowledge.
In a modern solution with well-connected remote systems that increasingly analyse problems and where there is increasingly 'lights out' building operation, the device's guess at severity etc. is irrelevant. In addition, with sophisticated resilience mechanisms, the device cannot make any relevant statement on whether the customer service has been impacted.
Likewise, in a world where there were no remote systems and local management was the only practice, alarms had to be locally 'acknowledged'. Where there are remote systems, per alarm acknowledge is burdensome.
However, many solutions and operational practices continue to use the historic schemes. On that basis, the schemes are supported but relegated to optional. At this point in the evolution of control solutions legacy-properties are probably mandatory, however, it is anticipated that as control solutions advance the legacy-properties will become irrelevant.
The legacy-properties are:
This section further clarifies the rationale for the choices of mandatory and optional identifiers.
As noted earlier, the alarm clear will be followed immediately by a Tombstone record. As also noted, the deletion of an alarm detector, where the alarm was active prior to the deletion, will cause at least the logging of a Tombstone record. Allowing for compaction delay:
Whilst the timestamp for the alarm is not a Controller responsibility (other than for Controller generated alarms, e.g., disc full), there is an expectation of correct behavior.
A versatile approximate-data-and-time structure has been used for the event-time-stamp to allow representation of inaccuracies.
The time stamp for an alarm should be as follows:
The alarm detected condition shall be presented:
An alternative is to provide translation metadata to enable normalization from the native form at the client. This metadata can be provided separately from the alarm stream and related to each detector with a relevant mapping. There is not standard expression for translation metadata.
Under certain circumstances a detected condition may:
Initial TAPI deployments using TAPI 2.1.3 are applied to solutions where TAPI is feeding a controller, orchestrator or OSS etc. from a single context that is taking a view of the entire network for all layers.
It is assumed that at start up the Controller will form the necessary full context and will initialize various streams. It is also assumed that the Controller will recover from any internal problems to ensure that the streams allow the client to achieve eventual consistency.
There are a number of critical behaviors assumed from the underlying system (essentially use cases for other components):
This leads to a specific set of use cases. The first list of cases have not been expanded as they cover normal Controller behavior. The main focus here is the second list [A15] of use cases that correspond to TAPI interaction,
Note that the stream should have the characteristics discussed in various sections earlier in this document.
The following use cases are described briefly here and then illustrated in the sequence diagram. The use cases assume that Websockets over TCP is the chosen connection method
The following activities are carried out (many asynchronously)
The pipeline continues to operate with the client close to the most recent record
Major intermittent failure in the network, for example, several micro-bends with active percussive interference, overloaded client with reduce compute power available
Assume major intermittent failure as before in an environment will lower capacity.
The hybrid message sequence diagram below captures all relevant flows for the listed use cases. In the diagram the behavior of the Source (device) and TAPI Context are summarized. The diagram only shows presence and fundamental flow for these elements.
Two pipelines are shown in yellow. The TAPI context pipeline intentionally shows no detail as that is outside the scope of the interface definition.
The diagram shows coupled asynchronous parallel/concurrent repeating activities and independent asynchronous repeating activities each in a dashed box. Where the asynchronous activities are coupled there is a dashed line showing the relationship as a ratio of activity or as a 1 which indicates that "eventually" there will be the same number of activities in both asynchronous elements (as a result of a flow through both). Some activities are shown as nested. Where nested there is an indication where there are n repeats of the inner asynchronous activity for each of the outer activities. Buffers are shown to emphasize the asynchronous coupling. The compacted log is shown with a buffer symbol annotated with an "X" indicating compaction (the deletion of records in the log) and "0" indicating that the record is for all system time (where compaction will remove duplicates and hence contain the log size).
To the left of the figure is the client side (which initiates the connections etc.), shown as a stylized example. The client is shown with both a DB option and a compacted log option for storage. The critical features are the "Pong" and "Last commit". The majority of the client depiction is to explain last commit. Last commit is used by the client on reconnection to continue where it left off.
The external comms between client and provider is shown as a brown bar and is not considered in any detail. It is assumed that it is reliable (e.g., TCP) and is playing an active part in maintenance of the pipeline.
The client token is opaque so the client has no knowledge of sequence through the token, although there is also exposure of the sequence number, this is primarily intended for stream analysis (note that it may be beneficial as part of normal behavior to validate communication).
The middle of the diagram shows the provider and explains the basic flows related to initial connection, loss of connection and forced connection drop.
To the extreme right of the figure are vertical progression bars that show phases of interaction between the client and provider.
The following figure shows the use cases and relationship to the message hybrid message sequence chart in the figure above. The heading bars are the same as the righthand vertical bars on the previous figure. The flow across the figure assumes a chaining of the use cases in the order they are described in the earlier sections.
Beyond the current release there will be further improvements including the ability to dynamically adjust the stream behavior and the feeds to the stream as follows:
For each of the above it will be necessary to enhance the expression of capability such that the client can know what opportunities for adjustment are available. It is expected that this will be expressed using machine interpretable specifications.
The messaging is as defined below:
The following section considers Kafka as an example implementation of a compacted log and then discusses implications of compaction and some storage strategies.
Compaction is described in the Kafka documentation at https://kafka.apache.org/documentation/#compaction
With retention (for non-deletes) set to “forever”, the log becomes a single source of truth for absolute state (eventual consistency) and change of state (cost effective fidelity).
Client essentially reading next record in sequence:
Considering compaction delay, in general system load will cause the compaction to sometimes drift such that less compaction occurs that is ideal. In Kafka, compaction does not operate at a fixed cleaner point as the head segment is not compacted. When the head rolls to become a tail segment compaction can happen but may be delayed. The behavior is not fully deterministic as it depends upon segment fill and intermittency occurrence.
[WC3 SSE] https://www.w3.org/TR/eventsource/ Server-Sent Events
[RFC6455] https://tools.ietf.org/html/rfc6455 The websocket protocol
[ONF TR-5XX.1] TR-5XX.1 TAPI v2.1.3 Reference Implementation
This document uses terms defined elsewhere.
The primary purpose of this TR-5xx ….
Section where the cross reference is only relevant for abbreviation/acronym interpretation purposes):
Nigel Davis Ciena
Arturo Mayoral Telefónica
Kam Lam FiberHome
Pedro Amaral Infinera
Andrea Mazzini Nokia
Karthik Sethuraman NEC
Malcolm Betts ZTE
Jonathan Sadler Infinera
End of Document
This term loses relevance once the readings have been processe
and abstract but is often still used.
 This is distinct from the idea projected by Yang, i.e. that the interface expresses the contents of the repository. It is assumed here that the repository structure and content is a decision for the Controller designer. The only constraint is that the exposed information must be in TAPI form.
 It may be beneficial to add an indication of time granularity to assist in cause/effect evaluation. For this to be fully beneficial the accuracy of synchronization of time would also need to be determined).
 Compaction is used intelligently to reduce realignment time whilst minimizing probability of loss of detail.
 When assessing the system performance overall delay must be taken into account At this point the timings are given ignoring comms buffering. Depending upon the comms buffers there can be a significant additional delayed information in the stream.
[A1] Delete in
[A2] [AMLL]: Definition of the terms:
"loss of fidelity" and "eventual consistency" would improve the description.
[A3] [AMLL]: Definition or reference should be provided to improve readability and common understanding.
[A4] [AMLL] The "record-retention" format (String) may be confusing. Maybe, if the parameter may be expressed in Time or Capacity there should be two attributes for each different measurement unit.
[A5] [AMLL]: I need to see if I understood correctly. The Controller (Server) produces a set of supported streams, then each of these streams may include a list of "record-content" items such Connection, Node… according to the attribute enum options. Then, my understanding is that the provider somehow filter the relevant objects to be "streamed" in each stream in advance, and it is not the client which can flexibly decide how to configure the stream. Is it this correct?
[A6] [AMLL]: A novel reader may not understand the concept. I suggest some introduction to the concept somewhere.
[A7] [AMLL] This section should be structure in a way to be developed further with the rest of "log-record" and "log-storage" strategies description, such as:
3.8 Log strategy
3.8.1 Log-record strategies
3.8.2 Log-storage strategies
188.8.131.52 Whole entity on change
[A8] [AMLL]: Whole Entity On Change:…
[A9] [AMLL]: If there are several delete retention mechanisms, they should be described independently from the "log-record"/"log-storage" strategies. Otherwise if the delete retention is associated to the selected "log-record"/"log-storage" strategy, they should be explaining within.
Right now it is not very clear to me.
[A10] [AMLL]: If this is conditioned to the log-record/storage-strategy followed, I propose to include it within each specific strategy section for clarity.
[A11] [AMLL]: Same as previous comment.
[A12] [AMLL]: Include reference to section.
[A13] Is this the main difference between the two solutions?
[A14] [AMLL]: This seems to me more a UC than a general section. I would propose to include it in Section 4.
Maybe another interesting use case would be applying tapi-streaming to topology/connectivity-state changes monitoring, maybe a UC may compact sections 3.13, 3.14 and adds more concrete examples with a relevant TAPI model objects, such Topology or Connectivity.
[A15] Not mentioned both lists later.
[A16] [AMLL]: In general my idea of Use Cases description is a bit more applicated to specific network management purpose, e.g., Alarm management, Topology-Context synchronization, Connectivity-Service Lifecycle management… I think it is important not mix general streaming concepts provided by this model and the proposed framework in general and the specific use cases.
I think current proposed use cases could be somehow categorized, maybe something like Streaming Basics, Alarm Management, Context Alignment, Performance Management, Connectivity-Service lifecycle management…
Additionally I was mentioning in a previous comment, it would be helpful to assign a sort of coding to each UC for referencing/testing etc..
[A17] [AMLL]: Delete