TIPC Programmer's Guide

A TIPC network consists of individual processing elements or nodes. Those nodes are arranged into clusters according to their assigned cluster identity. All nodes having the same cluster identity will establish links to each other, provided the network is set up to allow mutual neighbor discovery between them. It is only necessary to change the cluster identity from its default value if nodes in different clusters potentially may discover each other, e.g., if they are attached to the same subnet. Nodes in different clusters cannot communicate with each other using TIPC.

Each node has a 128 bit node identity which must be unique within the node's cluster. If the node will be part of a cluster, the user can either rely on the auto configuration capability of the node, where the identity is generated when the first bearer is attached, or he can set the identity explicitly, e.g., from the node's host name or a UUID. If a node will not be part of a cluster its identity can remain at the default value, zero.

Each node is also uniquely identified by a 32 bit hash number. This number is generated from the node identity, and is used internally by the protocol as node address. An extension to the neighbor discovery protocol guarantees that its value is unique within the cluster. This number is helpful both for protocol performance and for backwards compatibility.

The TIPC socket API provides three different address types:

• Service Address. This address type consists of a 32 bit service type identifier and a 32 bit service instance identifier. The type identifier is typically determined and hard coded by the user application programmer, but its value may have to be coordinated with other applications which might be present in the same cluster. The instance identifier is often calculated by the program, based on application specific criteria.


• Service Range. This address type represents a range of service addresses of the same type and with instances between a lower and an upper range limit. By binding a socket to this address type one can make it represent many instances, something which has proved useful in many cases. This address type is also used as multicast address, which will be explained in more detail later.


• Socket Address. This address is a reference to a specific socket in the cluster. It contains a 32 bit port number and a 32 bit node hash number. The port number is generated by the system when the socket is created, and the node hash is generated from the corresponding node identity as explained earlier. An address of this type can be used for connecting or for sending messages in the same way as service addresses can be used, but is only valid as long as long as the referenced socket exists.

When binding a socket to a service address or address range, the visibility scope of the binding must be indicated. There are two options, TIPC_NODE_SCOPE if the user only wants node local visibility, and TIPC_CLUSTER_SCOPE if he wants cluster global visibility. There are almost no limitations to how sockets can be bound to service addresses: one socket can be bound to many addresses or ranges, and many sockets can be bound to the same address or range. The service types 0 through 63 are however reserved for system internal use, and are not available for user space applications.

When sending a message by service address the sender may indicate a lookup scope, also called lookup domain. This is a node hash number, limiting the set of eligible destination sockets to the indicated node. If this value is zero, all matching sockets in the whole cluster, as visible from the source node, are eligible.

TIPC message transmission can be performed in different ways.

Datagram messages are discrete data units between 1 and 66,000 byte of length, transmitted between non-connected sockets of type SOCK_DGRAM or SOCK_RDM. Just like their UDP counterparts, TIPC datagrams are not guaranteed to reach their destination, but their chances of being delivered are still much better than for the former. Because of the link layer delivery guarantee, the only limiting factor for datagram delivery is the socket receive buffer size. The chances of success can also be increased by the sender, by giving his socket an appropriate delivery importance priority. There are four such priority levels. The default value is TIPC_LOW_IMPORTANCE, but this can be changed via the socket option TIPC_IMPORTANCE.

Furthermore, when there is receive buffer overflow, the sender can choose whether he just wants his message to be dropped, or if he should receive an indication about the failure to deliver. In the latter case, a potentially truncated (to the first 1024 bytes) copy of the original message is returned to the sender, along with the error code TIPC_ERR_OVERLOAD. This mechanism also works for other types of delivery failure, so a user may even encounter the error codes TIPC_ERR_NO_NODE and TIPC_ERR_NO_PORT. The sender reads the error code and and returned bytes as ancillary data from the socket. The default setting for this property is off, but it can be enabled via the socket option TIPC_DEST_DROPPABLE.

Datagram messages can be sent either by socket address, service address or multicast address.

• If a socket address is indicated the message is transmitted to that exact socket.


• When a service address is used, there might be several matching destinations, and the transmission method becomes what is often denoted anycast, i.e., that any of the matching destinations may be selected. When this is the case, the function translating from service address to socket address uses a round-robin algorithm to decrease the risk of load bias among the destinations. It should however be noted that this algorithm is node global, so unless a sender is alone on the node to use this address he has no guarantee that his particular messages will be evenly distributed among the destinations.


• The address type service range also doubles as multicast address. When an application specifies a service range as destination address, it does effectively instruct TIPC to send a copy of the message to all matching sockets in the cluster. Any socket bound to one or more instances inside the indicated multicast range will receive exactly one copy of the message, - never more. Multicast datagram messaging differ from the unicast/anycast ditto in one respect, - the lookup scope is always cluster global, and cannot be changed.

The risk of message rejection can be reduced by increasing the receive buffer size from the default value.

Datagram messages sent by service address may be subject to another mechanism intended to reduce the risk of delivery failure. Because of the transaction-free and non-atomic nature of binding table updates, a primary address lookup for a message may be successful on the source node, while it turns out that the destination socket has disappeared at arrival on the target node. In such cases, a secondary service lookup is attempted on the destination node, and only if that fails is the message dropped or returned to sender.

When a datagram is received in a socket, the receiver can read out the source socket address from the recvmsg() control block, as one would expect. In addition, it is possible to read out the service address the sender was using, if any. This feature might prove convenient in some cases.

Because of the lacking delivery guarantee for datagram messages, this transmission method should only be used when the programmer feels confident there is no risk of receive buffer overflow, or that he can handle the consequences. If he needs a more robust mechanism, with end-to-end flow control, he should instead consider using group messaging.

Connections can be established and used much like we are used to from TCP: the server creates a SOCK_STREAM socket and calls accept(); the client creates a blocking or non-blocking socket of the same type, does connect() on it and starts sending or receiving. The address types used can be any of service address or socket address (client side), or service address or service range (server side).

TIPC does however provide two varieties of this scenario, which may be useful in some cases.

• First, instead of being of type SOCK_STREAM, the sockets can be created as SOCK_SEQPACKET, implying that data exchange must happen in units of maximum 66,000 byte messages.


• Second, a client can initialize a connection by simply sending a data message to an accept()'ing socket. Likewise, the spawned server socket can respond with a data message back to the client to complete the connection. This way, TIPC provides an implied, also known as 0-RTT connection setup mechanism that is particularly time saving in many cases.

The most distinguishing property of TIPC connections is still their ability to react promptly to loss of contact with the peer socket, without resorting to active neighbor heart-beating.

• When a socket is ungracefully closed, either by the user or because of a process crash, the kernel socket code will by its own initiative issue a FIN message to the peer.


• When contact to a cluster node is lost, the local link layer will issue FIN messages to all sockets having connections towards that node. The peer node failure discovery time is configurable down to 50 ms, while the default value is 1,500 ms.


• To handle the very unlikely scenario of a half-finished, dangling connection, each socket endpoint maintains a 1-hour period timer to probe the peer if it has been silent during the past period. If there has been no response at the next timeout expiration the connection is aborted.

Group messaging can a little simplistically be described as datagram messaging, as described above, but with end-to-end flow control, and hence with delivery guarantee. There are however a few notable differences that must be described further.

• Messaging can only be done within a closed group of member sockets.


• A socket joins a group by calling the socket option TIPC_GROUP_JOIN with struct tipc_group_req* as argument. Part of this structure is a service address, where the type field indicates the group identity and the instance field indicates member identity. Hence, a member can only bind to one single service address, and nothing more.


• When sending an anycast message, the lookup algorithm applies the regular round-robin algorithm. However, it also considers the current load, i.e., the advertised send window, on potential receivers before making a selection.


• Just like with regular datagram multicasting, group multicast is performed by indicating a service range as destination. However, in group multicast only the lower value of the range is considered during lookup. This means that only those members which have joined the group with exactly that instance value will receive a copy of a sent multicast message.


• There is also a group broadcast mode which transmits a message to all group members, without considering their member identity. The sender indicates his intention to broadcast by using the send() primitive.


• There is a dedicated coordinated sliding window protocol in place to handle "traffic crunches", i.e., cases where many members may try to simultaneously send traffic to a the same destinations. This means that the delivery guarantee is valid even in such extreme cases.


• When receiving a message, the receiver uses recvmsg(), and can from the accompanying struct msghdr read out both source addresses of the sender, - its socket address and its bound service address.


• Apart from the message delivery guarantee, there is also a sequentiality guarantee. This guarantee is even valid between messaging modes, e.g., a member may send unicast, anycast, multicast or broadcast in any order, and the messages are guaranteed to be received in the same order in any particular recipient.


• When joining a group, a member may indicate if it wants to be eligible for loopback copies of his own sent anycast/multicast/broadcast messages. By default, this setting is off.


• When joining a group, a member may indicate if it wants to receive join or leave events for other members of the group. This feature leverages the service tracking feature, but contrary to other users, a group member will receive the events in the member socket proper. Because of this, it has become possible to issue a sequentiality guarantee; - a join event will always arrive before the first message from a new member, - a leave event is always delivered after the last message from the leaving member. An event is simply the reception of an empty out-of-band message, accompanied by the new member's two source addresses. I addition, a leave event message has the EOR bit set. The default value of the event subscription setting is off.

Message groups are both scalable and performant, but how much depends on their distribution across nodes and traffic pattern. On a single node with full traffic blast there is no problem establishing a group with e.g., 64 members, while there can be more if the group is distributed. If optimal performance for a sender is important, it is recommended to not let it switch between unicast/anycast and multicast/broadcast too frequently.