XDR - External Data Representation

XDR as a case study

• Sun RPC uses XDR.
• A good example of a “layer”.
• Interesting API.
• Powerful paradigm for creation of complex data structures.

XDR

• XDR provides a service associated with the OSI Presentation Layer.
• Common data representation
• Library (not part of the O.S.).
• Easy to port to new architectures.
• Independence from transport layer.

Data Conversion

• Asymmetric Data Conversion
• client always converts to the server’s data representation.
• Symmetric Data Conversion
• both client and server convert to some standard representation.

XDR Data Types

• boolean
• char
• short
• int
• long
• float
• double

• enumeration
• structure
• string
• fixed length array (1-D)
• variable length array (1-D)
• union
• opaque data

Implicit vs. Explicit Typing

• Explicit Typing means that each piece of data includes information about the type.
• Implicit typing means that the sender and receiver must agree on the order and type of all data.
• XDR uses Implicit Typing

XDR Programming

• Most XDR implementations are based on a buffer paradigm.
• A buffer is allocated that is large enough to hold an entire message.
• Individual data items are added to the buffer one at a time.
• XDR buffers can be attached to a file, pipe, socket or memory.

Conversion Terminology

• Converting from local representation to XDR representation is called Encoding.
• Converting from XDR representation to local representation is called Decoding.

XDR Buffer Creation

• There are a number of buffer creation functions in the XDR library.
• xdrmem_create
• destination for encoding -or- source for decoding is a chunk of memory.
• xdrstdio_create
• destination for encoding -or- source for decoding is a file descriptor.

XDR filters

• The XDR library includes an extensive set of filters that perform encoding/decoding operations.
• Each XDR stream includes an attribute that determines the specific operation that will be performed by a filter (encoding or decoding).

XDR Filters

• The filter to encode/decode an integer is called xdr_int:

bool_t xdr_int( XDR *xdrs, int *ip);

• the return value indicates success or failure.
• xdrs is a pointer to an XDR stream
• ip is a pointer to an integer.

xdr_int()

• If the XDR stream operation is XDR_ENCODE

int count;XDR *xstream;xdr_int(xstream, &count);

will convert (encode) the value of count to the integer representation used by XDR (big-endian) and put the result on the XDR stream.

xdr_int()

• If the XDR stream operation is XDR_DECODE

int count;

XDR *xstream;xdr_int(xstream, &count);

will get an XDR integer from the stream, convert (decode) the value to local integer representation and put the result in count.

Complex Data Filters

• The XDR library includes a set of filters designed to translate complex C data structures to and from XDR representation.
• Many of these filters make use of the simpler filters to convert individual components

xdr_array()

• The xdr_array() filter provides support for encoding/decoding a variable length array.

bool_t xdr_array( XDR *xdrs, char *arrp,

u_int *sizep, u_int maxsize, u_int elsize, xdrproc_t elproc);

sizep is a pointer to the size of the array.

elsize is the size of each element (in bytes).

elproc is a pointer to a function that can encode/decode individual array elements.

Inside xdr_array()

xdr_int(xdrs,&sizep);

for (i=0;i<*sizep;i++)

elproc(xdrs,arrp+elsize*i);

xdr_string()

• the string conversion filter is a little different since XDR strings have a maximum size.

bool_t xdr_string( XDR *xdrs,

char *string, u_int maxsize);

Problem!!

• We want to send an array of strings between processes.

• What is the problem (using xdr_array)?

• What is a possible solution?

Distributed Program Design

• Communication-Oriented Design
• Design protocol first.
• Build programs that adhere to the protocol.
• Application-Oriented Design
• Build application(s).
• Divide programs up and add communication protocols.

RPC Remote Procedure Call

• Call a procedure (subroutine) that is running on another machine.
• Issues:
• identifying and accessing the remote procedure
• parameters
• return value

Sun RPC

• There are a number of popular RPC specifications.
• Sun RPC is widely used.
• NFS (Network File System) is RPC based.
• Rich set of support tools.

Procedure Arguments

• To reduce the complexity of interface specification Sun RPC includes support for a single argument to a remote procedure.
• Typically the single argument is a structure that contains a number of values.

Procedure Identification

• Each procedure is identified by:
• Hostname (IP Address)
• Program identifier (32 bit integer)
• Procedure identifier (32 bit integer)

Procedure Identification

• Each procedure is identified by:
• Hostname (IP Address)
• Program identifier (32 bit integer)
• Procedure identifier (32 bit integer)
• Program Version identifier
• for testing and migration.

Program Identifiers

• Each remote program has a unique ID.
• Sun divided up the IDs:

0x00000000 - 0x1fffffff SUN

0x20000000 - 0x3fffffff SYS ADMIN

0x40000000 - 0x5fffffff TRANSIENT

0x60000000 - 0xffffffff RESERVED

Procedure Identifiers & Program Version Numbers

• Procedure Identifiers usually start at 1 and are numbered sequentially

• Version Numbers typically start at 1 and are numbered sequentially.

Iterative Server

• Sun RPC specifies that at most one remote procedure within a program can be invoked at any given time.
• If a 2nd procedure is called the caller blocks until the 1st procedure has completed.
• This is useful for applications that may share data among procedures.
• Example: database - to avoid insert/delete/modify collisions.

Communication Semantics

• To act like a local procedure (exactly one invocation per call) - a reliable transport (TCP) is necessary.
• Sun RPC does not support reliable call semantics.
• At Least Once Semantics (if the procedure returns)
• Zero or More Semantics (no reply)

Dynamic Port Mapping

• Servers typically do not use well known protocol ports.
• Clients known the Program ID (and host).
• A port lookup service runs on each host that contains RPC servers.
• RPC servers register themselves with the port mapper

The portmapper

• Each system which will support RPC servers runs a port mapper server that provides a central registry for RPC services.
• Servers tell the port mapper what services they offer.
• Clients ask a remote port mapper for the port number corresponsing to Remote Program ID.

More on the portmapper

• The portmapper is itself an RPC server!
• The portmapper is available on a well-known port (111).

Sun RPC Programming

• The RPC library is a collection of tools for automating the creation of RPC clients and servers.
• RPC clients are processes that call remote procedures.
• RPC servers are processes that include procedure(s) that can be called by clients.

RPC Programming

• RPC library
• XDR routines
• RPC run time library
• call rpc service
• register with portmapper
• dispatch incoming request to correct procedure
• Program Generator

RPC Run-time Library

• High- and Low-level functions that can be used by clients and servers.
• High-level functions provide simple access to RPC services.

High-level Client Library

int callrpc( char *host,

u_long prognum,

u_long versnum,

u_long procnum,

xdrproc_t inproc,

char *in,

xdrproc_t outproc,

char *out);

High-Level Server Library

int registerrpc(

u_long prognum,

u_long versnum,

u_long procnum,

char *(*procname)()

xdrproc_t inproc,

xdrproc_t outproc);

High-Level Server Library (cont.)

void svc_run();

• svc_run() is a dispatcher.
• A dispatcher waits for incoming connections and invokes the appropriate function to handle each incoming request.

High-Level Library Limitation

• The High-Level RPC library calls support UDP only (no TCP).
• You must use lower-level RPC library functions to use TCP.
• The High-Level library calls do not support any kind of authentication.

Low-level RPC Library

• Full control over all IPC options
• TCP & UDP
• Timeout values
• Asynchronous procedure calls
• Multi-tasking Servers
• Broadcasting

RPCGEN

• There is a tool for automating the creation of RPC clients and servers.
• The program rpcgen does most of the work for you.
• The input to rpcgen is a protocol definition in the form of a list of remote procedures and parameter types.

rpcgen Output Files

> rpcgen foo.x

• foo_clnt.c (client stubs)
• foo_svc.c (server main)
• foo_xdr.c (xdr filters)
• foo.h (shared header file)

Client Creation

> gcc -o fooclient foomain.c foo_clnt.c foo_xdr.c

• foomain.c is the client main() (and possibly other function) that call rpc services via the client stub functions in foo_clnt.c
• The client stubs use the xdr functions.

Server Creation

> gcc -o fooserver fooservices.c foo_svc.c foo_xdr.c

• fooservices.c contains the definitions of the actual remote procedures.

Example Protocol Definition

struct twonums {

int a;

int b;

};

program UIDPROG {

version UIDVERS {

int RGETUID(string<20>) = 1;

string RGETLOGIN( int ) = 2;

int RADD(twonums) = 3;

} = 1;

} = 0x20000001;