TIPC

Transparent Inter-process Communication (TIPC) is a network communications protocol for Inter-process communication (IPC) that was designed for intra-cluster communication.

Features

Some features of TIPC:

• Location transparency of services in a network
• Auto-discovery mechanism
• Reliable transport
• Standard socket interface support
• Connectionless, connection-oriented and multicast messaging
• Subscription to network events
• Dual BSD/GPL licensed code

Implementations

TIPC protocol support was available in Linux kernel starting with version 2.6.16. It can also be built into earlier versions of the kernel as a supplementary module. The TIPC project currently provides an open source implementation of TIPC for other operating systems including Wind River's VxWorks and Sun Microsystems' Solaris. TIPC applications are typically written in C (or C++) and utilize sockets of the AF_TIPC address family. Support for D, Perl, Python, and Ruby is also available.

Bearer media

While designed to be able to use all kinds of bearer media, as of March 2012 implementations only supported Ethernet. The VxWorks implementation also supports shared memory which can be accessed by multiple instances of the operating system, running simultaneously on the same hardware.

Neighbouring node links

Link setup

A TIPC node periodically broadcasts Link Request messages on all configured media interfaces to detect neighbouring cluster nodes. If such a message is received by a node with no prior established link to the sending node, it replies with a unicast Link Response message. This establishes a Link between those two nodes.

Link continuity check

A background timer is maintained for each link. When a certain amount of time is passed without regular incoming traffic, a message is sent over the link to indicate to the counterpart that the link is still up. This message also contains an acknowledge for the last received Link Level Sequence Number, in order to allow the receiver to release sent packet buffers, and a Last Sent Sequence Number, allowing the receiver to detect gaps in the packet sequence. The continuity check mechanism allows rapid detection of communication media failure, or node crashes. The Link Tolerance is a configurable parameter for each link endpoint, determining how long the other link endpoint may remain unresponsive before the link is declared faulty, and reset. For Ethernet the default value of this parameter is 1.5 s. After a reset, the remaining link endpoint will continue to probe the link until it is re-established.

Logical network topology

The logical Network Topology is not necessarily equal to the physical one.

TIPC nodes with point-to-point links to each other (typically, but not always, fully meshed) are logically grouped and named a Cluster. Clusters can in turn be grouped in Zones. Again, the zones of a TIPC network have to be able to reach each other directly.

File:TIPC logical topology - zone-cluster-node.png

Logical topology of a TIPC network. A Zone consists of fully meshed Clusters which consist of fully meshed Nodes.

Addressing scheme

Unlike in most other network protocols like IP, the address is not attached to an interface but to the whole physical node. Also a node can only possess one single address which identifies it throughout the whole network.

The addressing scheme is mapped to the logical Network Topology. The human readable notation is <Z.C.N> where Z stands for Zone, C for Cluster and N for Node. Internally the address is represented as a 32-bit integer. Here, the 8 most significant bits identify the Zone, the next 12 ones the Cluster and the 12 least significant bits the individual Node. This allows for a maximum of 256 Zones, each containing 4096 Clusters, each containing 4096 Nodes.

Communication semantics

In order to be able to adapt to the user's needs, TIPC allows to choose from four different communication semantics:

• Unreliable connectionless messages (SOCK_DGRAM) - comparable to UDP
• Reliable connectionless messages (SOCK_RDM)
• Reliable connection-oriented messages (SOCK_SEQPACKET)
• Reliable connection-oriented byte streams (SOCK_STREAM) - comparable to TCP

Protocol operations

The protocol operations are optimized for networks matching the following assumptions:

• Most messages cross only one direct hop.
• Transfer time for most messages is short.
• Most messages are passed over intra-cluster connections.
• Packet loss rate is normally low; retransmission is infrequent.
• Available bandwidth and memory volume is normally high.
• For all relevant bearers packets are check-summed by hardware.
• The number of inter-communicating nodes is relatively static and limited at any moment in time.
• Security is a less crucial issue in closed clusters than on the Internet.

While being able to work within environments not complying to this list, TIPC will not be able to show its advantages over other protocols which were designed to work within a wider range of parameters.

Packet loss detection

Packet loss detection is done on the node-to-node link level. A packet is given a 16-bit Link Level Sequence Number which is incremented modulo 2¹⁶ − 1 every time a packet is created and sorted into the link's send queue. At the time a packet is taken from this queue and actually sent, its Link Level Sequence Acknowledge Number field is filled with the latest received in-sequence Link Level Sequence Number of the respective link. When a link endpoint receives the acknowledge it purges the messages with the equal and lower sequence numbers from its memory. After 16 incoming messages with no traffic in the other direction which would have been used to acknowledge them, a receiving node transmits an extra message in order to achieve this.

In case the sequence number of an arriving packet indicates that there were one or more previous packets lost, the receiving node calculates the gap and reports it to the sender who instantly retransmits the missing packets. For both connectionless and connection-oriented message flows, packet or byte-stream order is maintained from the receiving application's point of view.

Flow control

User-to-user flow control is only supported for established connections. Here, flow control is achieved by keeping a counter SENT_CNT on the sending side which is increased with every sent package. The receiving node counts the packages which were read by the receiving application. When this counter reaches the configurable number N, it informs the sender of this, which subtracts N from SENT_CNT. In case SENT_CNT reaches a certain limit, the sender stops sending packages until SENT_CNT falls under the threshold again.

The recommended value for N is at least 200, while the threshold for SENT_CNT should be at least 2N.

When the communication mode is connectionless, flow control can only be performed at the application level, implemented according to the particular characteristics of the application.

Apart from this, there are various mechanisms for congestion control in TIPC, both at bearer, link and node level. These are transparent for the user programs.

History

This protocol was originally developed by Jon Paul Maloy at Ericsson during 1996-2005 and was used by that company in cluster applications for many years, before subsequently being released to the open source community. The TIPC Project Team developed a freely available, portable implementation of the TIPC protocol.

Reference links

• TIPC project at SourceForge
• Technical documents at SourceForge

TIPC at the International Paralympic Committee

• TIPC at IPC.InfostradaSports.com (archived)