| The Linux RapidIO Subsystem
|The RapidIO standard is a packet-based fabric interconnect standard designed for
|use in embedded systems. Development of the RapidIO standard is directed by the
|RapidIO Trade Association (RTA). The current version of the RapidIO specification
|is publicly available for download from the RTA web-site .
|This document describes the basics of the Linux RapidIO subsystem and provides
|information on its major components.
|Because the RapidIO subsystem follows the Linux device model it is integrated
|into the kernel similarly to other buses by defining RapidIO-specific device and
|bus types and registering them within the device model.
|The Linux RapidIO subsystem is architecture independent and therefore defines
|architecture-specific interfaces that provide support for common RapidIO
|2. Core Components
|A typical RapidIO network is a combination of endpoints and switches.
|Each of these components is represented in the subsystem by an associated data
|structure. The core logical components of the RapidIO subsystem are defined
|in include/linux/rio.h file.
|2.1 Master Port
|A master port (or mport) is a RapidIO interface controller that is local to the
|processor executing the Linux code. A master port generates and receives RapidIO
|packets (transactions). In the RapidIO subsystem each master port is represented
|by a rio_mport data structure. This structure contains master port specific
|resources such as mailboxes and doorbells. The rio_mport also includes a unique
|host device ID that is valid when a master port is configured as an enumerating
|RapidIO master ports are serviced by subsystem specific mport device drivers
|that provide functionality defined for this subsystem. To provide a hardware
|independent interface for RapidIO subsystem operations, rio_mport structure
|includes rio_ops data structure which contains pointers to hardware specific
|implementations of RapidIO functions.
|A RapidIO device is any endpoint (other than mport) or switch in the network.
|All devices are presented in the RapidIO subsystem by corresponding rio_dev data
|structure. Devices form one global device list and per-network device lists
|(depending on number of available mports and networks).
|A RapidIO switch is a special class of device that routes packets between its
|ports towards their final destination. The packet destination port within a
|switch is defined by an internal routing table. A switch is presented in the
|RapidIO subsystem by rio_dev data structure expanded by additional rio_switch
|data structure, which contains switch specific information such as copy of the
|routing table and pointers to switch specific functions.
|The RapidIO subsystem defines the format and initialization method for subsystem
|specific switch drivers that are designed to provide hardware-specific
|implementation of common switch management routines.
|A RapidIO network is a combination of interconnected endpoint and switch devices.
|Each RapidIO network known to the system is represented by corresponding rio_net
|data structure. This structure includes lists of all devices and local master
|ports that form the same network. It also contains a pointer to the default
|master port that is used to communicate with devices within the network.
|3. Subsystem Initialization
|In order to initialize the RapidIO subsystem, a platform must initialize and
|register at least one master port within the RapidIO network. To register mport
|within the subsystem controller driver initialization code calls function
|rio_register_mport() for each available master port. After all active master
|ports are registered with a RapidIO subsystem, the rio_init_mports() routine
|is called to perform enumeration and discovery.
|In the current PowerPC-based implementation a subsys_initcall() is specified to
|perform controller initialization and mport registration. At the end it directly
|calls rio_init_mports() to execute RapidIO enumeration and discovery.
|4. Enumeration and Discovery
|When rio_init_mports() is called it scans a list of registered master ports and
|calls an enumeration or discovery routine depending on the configured role of a
|master port: host or agent.
|Enumeration is performed by a master port if it is configured as a host port by
|assigning a host device ID greater than or equal to zero. A host device ID is
|assigned to a master port through the kernel command line parameter "riohdid=",
|or can be configured in a platform-specific manner. If the host device ID for
|a specific master port is set to -1, the discovery process will be performed
|The enumeration and discovery routines use RapidIO maintenance transactions
|to access the configuration space of devices.
|The enumeration process is implemented according to the enumeration algorithm
|outlined in the RapidIO Interconnect Specification: Annex I .
|The enumeration process traverses the network using a recursive depth-first
|algorithm. When a new device is found, the enumerator takes ownership of that
|device by writing into the Host Device ID Lock CSR. It does this to ensure that
|the enumerator has exclusive right to enumerate the device. If device ownership
|is successfully acquired, the enumerator allocates a new rio_dev structure and
|initializes it according to device capabilities.
|If the device is an endpoint, a unique device ID is assigned to it and its value
|is written into the device's Base Device ID CSR.
|If the device is a switch, the enumerator allocates an additional rio_switch
|structure to store switch specific information. Then the switch's vendor ID and
|device ID are queried against a table of known RapidIO switches. Each switch
|table entry contains a pointer to a switch-specific initialization routine that
|initializes pointers to the rest of switch specific operations, and performs
|hardware initialization if necessary. A RapidIO switch does not have a unique
|device ID; it relies on hopcount and routing for device ID of an attached
|endpoint if access to its configuration registers is required. If a switch (or
|chain of switches) does not have any endpoint (except enumerator) attached to
|it, a fake device ID will be assigned to configure a route to that switch.
|In the case of a chain of switches without endpoint, one fake device ID is used
|to configure a route through the entire chain and switches are differentiated by
|their hopcount value.
|For both endpoints and switches the enumerator writes a unique component tag
|into device's Component Tag CSR. That unique value is used by the error
|management notification mechanism to identify a device that is reporting an
|error management event.
|Enumeration beyond a switch is completed by iterating over each active egress
|port of that switch. For each active link, a route to a default device ID
|(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written
|into the routing table. The algorithm recurs by calling itself with hopcount + 1
|and the default device ID in order to access the device on the active port.
|After the host has completed enumeration of the entire network it releases
|devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint
|in the system, it sets the Master Enable bit in the Port General Control CSR
|to indicate that enumeration is completed and agents are allowed to execute
|passive discovery of the network.
|The discovery process is performed by agents and is similar to the enumeration
|process that is described above. However, the discovery process is performed
|without changes to the existing routing because agents only gather information
|about RapidIO network structure and are building an internal map of discovered
|devices. This way each Linux-based component of the RapidIO subsystem has
|a complete view of the network. The discovery process can be performed
|simultaneously by several agents. After initializing its RapidIO master port
|each agent waits for enumeration completion by the host for the configured wait
|time period. If this wait time period expires before enumeration is completed,
|an agent skips RapidIO discovery and continues with remaining kernel
| RapidIO Trade Association. RapidIO Interconnect Specifications.
| Rapidio TA. Technology Comparisons.
| RapidIO support for Linux.
| Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005