Import Upstream version 1.8.5

[hcoop/debian/openafs.git] / doc / txt / rx-debug.txt

                               Rx Debug
                               --------

Introduction
============

Rx provides data collections for remote debugging and troubleshooting using UDP
packets.  This document provides details on the protocol, data formats, and
the data format versions.


Protocol
========

A simple request/response protocol is used to request this information
from an Rx instance.  Request and response packets contain an Rx header but
only a subset of the header fields are used, since the debugging packages are
not part of the Rx RPC protocol.

The protocol is simple. A client sends an Rx DEBUG (8) packet to an
address:port of an active Rx instance. This request contains an arbitrary
request number in the callNumber field of the Rx header (reused here since
DEBUG packets are never used in RPCs). The payload of the request is simply a
pair 32 bit integers in network byte order. The first integer indicates the
which data collection type is requested. The second integer indicates which
record number of the data type requested, for data types which have multiple
records, such as the rx connections and rx peers. The request packet must have
the CLIENT-INITIATED flag set in the Rx header.

Rx responds with a single Rx DEBUG (8) packet, the payload of which contains
the data record for the type and index requested. The callNumber in the Rx
header contains the same number as the value of the request, allowing the
client to match responses to requests.  The response DEBUG packet does not
contain the request type and index parameters.

The first 32-bits, in network byte order, of the response payload indicate
error conditions:

* 0xFFFFFFFF (-1) index is out of range
* 0xFFFFFFF8 (-8) unknown request type


Data Collection Types
=====================

OpenAFS defines 5 types of data collections which may be
requested:

  1  GETSTATS   Basic Rx statistics           (struct rx_debugStats)
  2  GETCONN    Active connections [indexed]  (struct rx_debugConn)
  3  GETALLCONN All connections [indexed]     (struct rx_debugConn)
  4  RXSTATS    Detailed Rx statistics        (struct rx_statistics)
  5  GETPEER    Rx peer info [indexed]        (struct rx_peerDebug)

The format of the response data for each type is given below. XDR is
not used. All integers are in network byte order.

In a typical exchange, a client will request the "basic Rx stats" data first.
This contains a data layout version number (detailed in the next section).

Types GETCONN (2), GETALLCONN (3), and GETPEER (5), are array-like data
collections.  The index field is used to retrieve each record, one per packet.
The first record is index 0. The client may request each record, starting with
zero, and incremented by one on each request packet, until the Rx service
returns -1 (out of range). No provisions are made for locking the data
collections between requests, as this is intended only to be a debugging
interface.


Data Collection Versions
========================

Every Rx service has a single byte wide debugging version id, which is set at
build time.  This version id allows clients to properly interpret the response
data formats for the various data types.  The version id is present in the
basic Rx statistics (type 1) response data.

The first usable version is 'L', which was present in early Transarc/IBM AFS.
The first version in OpenAFS was 'Q', and versions after 'Q' are OpenAFS
specific extensions. The current version for OpenAFS is 'S'.

Historically, the version id has been incremented when a new debug data type is
added or changed.  The version history is summarized in the following table:

    'L'  - Earliest usable version
         - GETSTATS (1) supported
         - GETCONNS (2) supported (with obsolete format rx_debugConn_vL)
         - Added connection object security stats (rx_securityObjectStats) to GETCONNS (2)
         - Transarc/IBM AFS

    'M'  - Added GETALLCONN (3) data type
         - Added RXSTATS (4) data type
         - Transarc/IBM AFS

    'N'  - Added calls waiting for a thread count (nWaiting) to GETSTATS (1)
         - Transarc/IBM AFS

    'O'  - Added number of idle threads count (idleThreads) to GETSTATS (1)
         - Transarc/IBM AFS

    'P'  - Added cbuf packet allocation failure counts (receiveCbufPktAllocFailures
           and sendCbufPktAllocFailures) to RXSTATS (4)
         - Transarc/IBM AFS

    'Q'  - Added GETPEER (5) data type
         - Transarc/IBM AFS
         - OpenAFS 1.0

    (?)  - Added number of busy aborts sent (nBusies) to RXSTATS (4)
         - rxdebug was not changed to display this new count
         - OpenAFS 1.4.0

    'R'  - Added total calls which waited for a thread (nWaited) to GETSTATS (1)
         - OpenAFS 1.5.0 (devel)
         - OpenAFS 1.6.0 (stable)

    'S'  - Added total packets allocated (nPackets) to GETSTATS (1)
         - OpenAFS 1.5.53 (devel)
         - OpenAFS 1.6.0 (stable)



Debug Request Parameters
========================

The payload of DEBUG request packets is two 32 bit integers
in network byte order.


    struct rx_debugIn {
        afs_int32 type;   /* requested type; range 1..5 */
        afs_int32 index;  /* record number: 0 .. n */
    };

The index field should be set to 0 when type is GETSTAT (1) and RXSTATS (4).



GETSTATS (1)
============

GETSTATS returns basic Rx performance statistics and the overall debug
version id.

    struct rx_debugStats {
        afs_int32 nFreePackets;
        afs_int32 packetReclaims;
        afs_int32 callsExecuted;
        char waitingForPackets;
        char usedFDs;
        char version;
        char spare1;
        afs_int32 nWaiting;       /* Version 'N': number of calls waiting for a thread */
        afs_int32 idleThreads;    /* Version 'O': number of server threads that are idle */
        afs_int32 nWaited;        /* Version 'R': total calls waited */
        afs_int32 nPackets;       /* Version 'S': total packets allocated */
        afs_int32 spare2[6];
    };


GETCONN (2) and GETALLCONN (3)
==============================

GETCONN (2) returns an active connection information record, for the
given index.

GETALLCONN (3) returns a connection information record, active or not,
for the given index.  The GETALLCONN (3) data type was added in
version 'M'.

The data format is the same for GETCONN (2) and GETALLCONN (3), and is
as follows:

    struct rx_debugConn {
        afs_uint32 host;
        afs_int32 cid;
        afs_int32 serial;
        afs_int32 callNumber[RX_MAXCALLS];
        afs_int32 error;
        short port;
        char flags;
        char type;
        char securityIndex;
        char sparec[3];        /* force correct alignment */
        char callState[RX_MAXCALLS];
        char callMode[RX_MAXCALLS];
        char callFlags[RX_MAXCALLS];
        char callOther[RX_MAXCALLS];
        /* old style getconn stops here */
        struct rx_securityObjectStats secStats;
        afs_int32 epoch;
        afs_int32 natMTU;
        afs_int32 sparel[9];
    };


An obsolete layout, which exhibited a problem with data alignment, was used in
Version 'L'. This is defined as:

    struct rx_debugConn_vL {
        afs_uint32 host;
        afs_int32 cid;
        afs_int32 serial;
        afs_int32 callNumber[RX_MAXCALLS];
        afs_int32 error;
        short port;
        char flags;
        char type;
        char securityIndex;
        char callState[RX_MAXCALLS];
        char callMode[RX_MAXCALLS];
        char callFlags[RX_MAXCALLS];
        char callOther[RX_MAXCALLS];
        /* old style getconn stops here */
        struct rx_securityObjectStats secStats;
        afs_int32 sparel[10];
    };


The layout of the secStats field is as follows:

    struct rx_securityObjectStats {
        char type;			/* 0:unk 1:null,2:vab 3:kad */
        char level;
        char sparec[10];		/* force correct alignment */
        afs_int32 flags;		/* 1=>unalloc, 2=>auth, 4=>expired */
        afs_uint32 expires;
        afs_uint32 packetsReceived;
        afs_uint32 packetsSent;
        afs_uint32 bytesReceived;
        afs_uint32 bytesSent;
        short spares[4];
        afs_int32 sparel[8];
    };



RXSTATS (4)
===========

RXSTATS (4) returns general rx statistics. Every member of the returned
structure is a 32 bit integer in network byte order.  The assumption is made
sizeof(int) is equal to sizeof(afs_int32).

The RXSTATS (4) data type was added in Version 'M'.


    struct rx_statistics {         /* General rx statistics */
        int packetRequests;        /* Number of packet allocation requests */
        int receivePktAllocFailures;
        int sendPktAllocFailures;
        int specialPktAllocFailures;
        int socketGreedy;          /* Whether SO_GREEDY succeeded */
        int bogusPacketOnRead;     /* Number of inappropriately short packets received */
        int bogusHost;             /* Host address from bogus packets */
        int noPacketOnRead;        /* Number of read packets attempted when there was actually no packet to read off the wire */
        int noPacketBuffersOnRead; /* Number of dropped data packets due to lack of packet buffers */
        int selects;               /* Number of selects waiting for packet or timeout */
        int sendSelects;           /* Number of selects forced when sending packet */
        int packetsRead[RX_N_PACKET_TYPES];    /* Total number of packets read, per type */
        int dataPacketsRead;       /* Number of unique data packets read off the wire */
        int ackPacketsRead;        /* Number of ack packets read */
        int dupPacketsRead;        /* Number of duplicate data packets read */
        int spuriousPacketsRead;   /* Number of inappropriate data packets */
        int packetsSent[RX_N_PACKET_TYPES];    /* Number of rxi_Sends: packets sent over the wire, per type */
        int ackPacketsSent;       /* Number of acks sent */
        int pingPacketsSent;      /* Total number of ping packets sent */
        int abortPacketsSent;     /* Total number of aborts */
        int busyPacketsSent;      /* Total number of busies sent received */
        int dataPacketsSent;      /* Number of unique data packets sent */
        int dataPacketsReSent;    /* Number of retransmissions */
        int dataPacketsPushed;    /* Number of retransmissions pushed early by a NACK */
        int ignoreAckedPacket;    /* Number of packets with acked flag, on rxi_Start */
        struct clock totalRtt;    /* Total round trip time measured (use to compute average) */
        struct clock minRtt;      /* Minimum round trip time measured */
        struct clock maxRtt;      /* Maximum round trip time measured */
        int nRttSamples;          /* Total number of round trip samples */
        int nServerConns;         /* Total number of server connections */
        int nClientConns;         /* Total number of client connections */
        int nPeerStructs;         /* Total number of peer structures */
        int nCallStructs;         /* Total number of call structures allocated */
        int nFreeCallStructs;     /* Total number of previously allocated free call structures */
        int netSendFailures;
        afs_int32 fatalErrors;
        int ignorePacketDally;    /* packets dropped because call is in dally state */
        int receiveCbufPktAllocFailures;  /* Version 'P': receive cbuf packet alloc failures */
        int sendCbufPktAllocFailures;     /* Version 'P': send cbuf packet alloc failures */
        int nBusies;                      /* Version 'R': number of busy aborts sent */
        int spares[4];
    };


GETPEER (5)
===========

GETPEER (5) returns a peer information record, for the given index.

    struct rx_debugPeer {
        afs_uint32 host;
        u_short port;
        u_short ifMTU;
        afs_uint32 idleWhen;
        short refCount;
        u_char burstSize;
        u_char burst;
        struct clock burstWait;
        afs_int32 rtt;
        afs_int32 rtt_dev;
        struct clock timeout;
        afs_int32 nSent;
        afs_int32 reSends;
        afs_int32 inPacketSkew;
        afs_int32 outPacketSkew;
        afs_int32 rateFlag;
        u_short natMTU;
        u_short maxMTU;
        u_short maxDgramPackets;
        u_short ifDgramPackets;
        u_short MTU;
        u_short cwind;
        u_short nDgramPackets;
        u_short congestSeq;
        afs_hyper_t bytesSent;
        afs_hyper_t bytesReceived;
        afs_int32 sparel[10];
    };