Host Networking

A message abstraction sends data in explicit chunks.

The key property is:

The receiver can tell where one message ends and the next message begins.

If the sender sends one message, the receiver observes one message boundary.

For example:

• sender sends message A;
• sender sends message B;
• receiver can distinguish A from B.

Important points:

• A message is a unit of communication.
• Message boundaries are part of the abstraction.
• Reliability, ordering, and loss behavior are separate questions.
  • A message interface may be reliable or unreliable.
  • It may preserve ordering or not.
  • These guarantees depend on the concrete protocol and implementation.

Common examples:

• UDP datagrams;
• HTTP requests/responses at the application level;
• many file-server and mail-server protocols;
• many request-response protocols.

A byte stream abstraction gives the application something like a pipe.

The application puts bytes in one end, and bytes come out at the other end.

The key property is:

There are no inherent message boundaries.

The sender may write data in some chunk size, but the receiver is not guaranteed to receive the data in the same chunk size.

Example:

Sender:
  send 1000 bytes
  send 1000 bytes

Receiver might see:
  recv 2000 bytes

or:
  recv 500 bytes
  recv 700 bytes
  recv 800 bytes

or many other possibilities.

The stream only preserves the sequence of bytes, not the sender’s application-level chunking.

TCP is the classic example:

• TCP provides a reliable byte stream.
• TCP does not preserve application message boundaries.
• If an application wants messages over TCP, it must define its own framing.

Common framing strategies:

1. Length prefix
   • Put a header before each message.
   • The header says how many bytes the message contains.
   • The receiver first reads the length, then reads exactly that many bytes.
2. Delimiter
   • Use a special byte sequence to mark the end of a message.
   • Example: line-based protocols.
3. Protocol-specific rules
   • HTTP has its own rules for determining where requests and responses end.

The important lesson:

TCP is a byte-stream interface. If your application wants messages, your protocol must reconstruct message boundaries.

A remote procedure call gives the programmer the abstraction of calling a function on another machine.

Conceptually:

result = remote_function(arguments)

From the caller’s perspective:

• the client calls a function;
• the request is sent to a remote machine;
• the remote machine runs some handler;
• the return value is sent back;
• the original call returns.

RPC is usually a higher-level abstraction built on top of messages or byte streams.

Typical properties:

• often synchronous;
• easy programming model;
• hides the network plumbing;
• resembles an ordinary function call.

But the abstraction hides a lot of complexity:

• marshaling / serialization of arguments;
• sending the request;
• dispatching to the correct remote handler;
• returning the result;
• handling errors, timeouts, retries, partial failures.

RPC can be implemented over:

• messages;
• byte streams;
• datagrams;
• custom transport protocols.

The lecturer emphasizes that the interface abstraction and the implementation strategy are separate.

A remote memory access abstraction lets one machine read or write memory on another machine.

Conceptually:

read remote memory range
write remote memory range

This is different from sending a message to the remote application and asking it to perform the read or write.

In a true remote-memory-access abstraction, the application on the remote side does not have to actively run a handler for each access.

There are several ways to think about this abstraction.

If an application wants to communicate with another endpoint, it must name that endpoint somehow.

Examples of addressing mechanisms:

• IP address;
• IP address + port;
• MAC address;
• DNS name;
• URL;
• Unix domain socket path;
• load balancer name;
• service discovery system;
• middleware / broker / registry such as ZooKeeper.

Addressing sounds simple, but becomes difficult in real systems.

Example:

A small service may begin with:

DATABASE_HOST = "db.example.com"

But later the system may need:

• multiple database replicas;
• failover;
• load balancing;
• geographic replication;
• service discovery;
• versioned deployments.

Then a single hard-coded address is no longer enough.

Addressing is therefore a real systems design problem, not just a syntactic detail.

A communication interface may be connection-oriented or connectionless.

Connection-oriented does not automatically mean reliable, and connectionless does not automatically mean unreliable.

They are separate dimensions.

However, common real-world combinations are:

The lecturer points out that reliability can be built in different ways:

• a protocol may provide acknowledgements and retransmissions;
• an application may resend requests until it gets a response;
• ordering may or may not be guaranteed.

Connections are useful because they provide a natural place to store reliability and ordering state:

• what has been sent;
• what has been acknowledged;
• what order messages belong to;
• what peer the state belongs to.

Without a connection, even defining ordering can become ambiguous.

For example:

• ordering from one thread?
• ordering from one process?
• ordering from one client address?
• ordering across multiple sockets?

A server often communicates with many clients at once.

The application needs to answer:

Which peer or connection should I handle next?

This is the demultiplexing problem.

Examples:

• Which client sent data?
• Which connection can I read from now?
• Which connection is ready for writing?
• Which pending request does this response belong to?
• Which connection had an error?

This becomes complicated on both server and client sides.

Server side:

• many clients may connect concurrently;
• some connections may be idle;
• some may be waiting for output buffers;
• some may have incoming requests.

Client side:

• one client may talk to multiple servers;
• responses may arrive in an unexpected order;
• the client must match responses to the correct outstanding request.

The main calls discussed are:

#include <sys/socket.h>

int socket(int domain, int type, int protocol);

int bind(int fd, const struct sockaddr *a, socklen_t al);
int listen(int sockfd, int backlog);
int accept(int fd, struct sockaddr *a, socklen_t *pal);
int connect(int fd, const struct sockaddr *a, socklen_t al);

ssize_t recv(int fd, void *buf, size_t len, int flags);
ssize_t send(int fd, const void *buf, size_t len, int flags);

ssize_t recvfrom(int fd, void *buf, size_t len, int flags,
                 struct sockaddr *a, socklen_t *pal);

ssize_t sendto(int fd, const void *buf, size_t len, int flags,
               const struct sockaddr *a, socklen_t al);

The interface looks small, but it covers many different communication modes.

socket(...) creates a socket and returns a file descriptor.

This fits into the Unix model:

• sockets are file descriptors;
• they can be inherited across fork;
• close works on them;
• for stream sockets, ordinary read and write can often be used;
• they can be used with multiplexing APIs such as select, poll, and epoll.

This integration with the file descriptor abstraction is powerful, but also leads to some historical complexity.

Conceptually, sockets cover a design space:

The two most common cases are:

1. Connected stream
   • TCP.
   • Reliable byte stream.
   • Requires connection setup.
2. Unconnected datagram
   • UDP.
   • Message/datagram-oriented.
   • No connection setup.

Unix domain sockets also exist:

• endpoints can be named by filesystem paths;
• they are used for local communication on the same machine;
• they can support stream and datagram modes;
• after connecting, the API looks much like network sockets.

The socket API supports multiple address formats:

• IPv4 addresses;
• IPv6 addresses;
• Unix socket paths;
• other address families.

Therefore, many calls use generic struct sockaddr * pointers plus explicit length values.

When the application passes an address into the kernel:

bind(fd, (struct sockaddr *)&addr, sizeof(addr));
sendto(fd, buf, len, 0, (struct sockaddr *)&addr, sizeof(addr));

the length is passed by value.

When the kernel returns an address to the application:

struct sockaddr_in client_addr;
socklen_t client_addr_len = sizeof(client_addr);

recvfrom(fd, buf, sizeof(buf), 0,
         (struct sockaddr *)&client_addr,
         &client_addr_len);

the length is passed by pointer.

Reason:

• before the call, the application tells the kernel how much space is available;
• after the call, the kernel updates the length to say how large the returned address actually was.

This looks awkward, but it is a consequence of using one interface for many address families.

A UDP server typically does:

int s_fd = socket(AF_INET, SOCK_DGRAM, 0);

struct sockaddr_in server_addr = ...;  // IP and port
bind(s_fd, (struct sockaddr *)&server_addr, sizeof(server_addr));

while (true) {
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);

    char buf[MAX_BUFFER_SIZE];

    rx_len = recvfrom(s_fd, buf, sizeof(buf), 0,
                      (struct sockaddr *)&client_addr,
                      &client_addr_len);

    tx_len = process_request(buf, rx_len);

    sendto(s_fd, response_buf, tx_len, 0,
           (struct sockaddr *)&client_addr,
           client_addr_len);
}

Important points:

• there is only one socket for communication;
• there is no listen or accept;
• every received datagram includes the sender address;
• every reply must specify the destination address.

A UDP client often does:

int fd = socket(AF_INET, SOCK_DGRAM, 0);

sendto(fd, request, request_len, 0,
       (struct sockaddr *)&server_addr,
       sizeof(server_addr));

recvfrom(fd, response, sizeof(response), 0,
         ...);

The client often does not explicitly bind.

The OS chooses an ephemeral local port.

Optionally, the client can call connect on a UDP socket to set a default peer address.

Then it can use send and recv instead of sendto and recvfrom.

A simple sequential TCP server does:

int s_fd = socket(AF_INET, SOCK_STREAM, 0);

struct sockaddr_in server_addr = ...;  // IP and port
bind(s_fd, (struct sockaddr *)&server_addr, sizeof(server_addr));

listen(s_fd, SOMAXCONN);

while (true) {
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);

    int c_fd = accept(s_fd,
                      (struct sockaddr *)&client_addr,
                      &client_addr_len);

    handle_connection(c_fd);
}

Important distinction:

• s_fd is the listening socket;
• c_fd is the connected socket for one client.

This simple server is sequential:

• it accepts one connection;
• handles it completely;
• then accepts the next connection.

That is easy to understand, but not good for concurrent servers.

A simplified connection handler:

void handle_connection(int c_fd) {
    while (true) {
        char buf[MAX_BUFFER_SIZE];

        rx_len = recv(c_fd, buf, sizeof(buf), 0);

        if (rx_len == 0) {
            // The peer performed an orderly shutdown.
            break;
        }

        if (rx_len < 0) {
            // Error handling required.
            break;
        }

        tx_len = process_request(buf, rx_len);

        send(c_fd, response_buf, tx_len, 0);
    }

    close(c_fd);
}

The lecture slides omit most error handling, but real networking code must handle errors after every system call.

Important cases:

• recv returns positive value:
  • received that many bytes.
• recv returns 0:
  • peer has closed or half-closed its sending side;
  • this is similar to EOF on a file.
• recv returns negative value:
  • error;
  • in non-blocking mode, may mean EAGAIN or EWOULDBLOCK.

The lecturer also mentions shutdown.

shutdown can half-close a connection:

• one side can say, “I will not send more data”;
• but it may still want to receive pending responses.

This is different from immediately closing the whole file descriptor.

When the application calls:

send(fd, buf, len, 0);

the OS does not necessarily put all len bytes on the wire immediately.

Typically, the OS copies some data into a socket buffer. Later the kernel and network stack send it out.

If the socket buffer does not have enough space, send may send only part of the data and return the number of bytes accepted.

Example:

ssize_t n = send(fd, buf, 1 << 30, 0);

If the application asks to send 1 GiB, the OS might only accept 64 MiB for now.

Then send returns the number of bytes accepted.

Correct code must check the return value and later send the rest.

Similarly:

recv(fd, buf, len, 0);

returns whatever data is currently available, up to len bytes.

If the application requests 128 bytes but only 64 bytes have arrived, recv may return 64.

This is especially important when implementing messages over TCP.

If the protocol says that a message is 1000 bytes, the application may need several recv calls to collect the full message.

A robust stream send usually looks like:

size_t off = 0;

while (off < len) {
    ssize_t n = send(fd, buf + off, len - off, 0);

    if (n > 0) {
        off += n;
        continue;
    }

    if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
        // Try again later if non-blocking.
        continue;
    }

    // Handle real error.
    break;
}

The exact code depends on blocking vs non-blocking mode, but the principle is:

Never assume one send sends the whole buffer. Never assume one recv receives the whole logical message.

For datagram/message sockets, the message boundary is part of the abstraction.

Therefore, the common model is:

• a datagram send is one message;
• the receiver receives one message at a time.

This avoids the TCP-style stream-framing problem.

However, datagram protocols have their own limits and error cases, such as maximum datagram size and possible loss.

By default, file descriptor operations are usually blocking.

Examples:

read
recv
write
send
connect
accept

Blocking means:

• recv blocks if there is no data;
• accept blocks if there is no incoming connection;
• connect blocks while the connection is being established;
• send may block if no buffer space is available.

The problem is that it is hard to predict how long an operation will block.

This is especially bad in servers:

• if one connection blocks;
• the whole thread may stop making progress;
• other ready connections may not be served.

A file descriptor can be switched to non-blocking mode.

Typical pattern:

int flags = fcntl(fd, F_GETFL, 0);
flags |= O_NONBLOCK;
fcntl(fd, F_SETFL, flags);

Some newer APIs, such as accept4, can return a non-blocking socket directly.

In non-blocking mode:

• if an operation can make progress, it does so;
• if it would block, it returns an error instead.

The common error cases are:

errno == EAGAIN
errno == EWOULDBLOCK

Portable code often checks both.

Example logic:

ssize_t n = recv(fd, buf, sizeof(buf), 0);

if (n > 0) {
    // received data
} else if (n == 0) {
    // orderly shutdown / EOF
} else {
    if (errno == EAGAIN || errno == EWOULDBLOCK) {
        // no data now; try again later
    } else {
        // real error
    }
}

Non-blocking mode avoids getting stuck, but it creates a new problem:

How does the application know when to try again?

Trying all sockets repeatedly would be inefficient busy waiting.

This leads to multiplexing APIs.

The traditional Unix server model:

1. Parent process creates a listening socket.
2. Parent runs an accept loop.
3. For each accepted connection, parent calls fork.
4. Child process inherits the connected socket FD.
5. Child handles the connection.
6. Parent goes back to accepting more connections.

Advantages:

• uses standard Unix abstractions;
• simple mental model;
• each connection handler is isolated in a separate process;
• useful for security and privilege separation.

Example: SSH.

A simplified SSH-style model:

• SSH server starts as root;
• it accepts a connection;
• it authenticates the user;
• child process drops privileges using calls such as setuid / setgid;
• then it runs the user’s shell with that user’s privileges.

This works well with Unix process protection.

Disadvantages:

• fork is expensive;
• process creation/destruction costs are high;
• does not scale well to high connection rates;
• not suitable for tens or hundreds of thousands of connections.

Historically, Unix also had inetd, a daemon that could listen on ports and start service programs for incoming connections.

This fits the accept-and-fork style.

A more modern variant is:

• parent accepts a connection;
• create one thread to handle that connection.

Advantages compared to fork:

• cheaper than process creation;
• shared address space;
• easier to share application state.

Disadvantages:

• still has nontrivial creation/destruction overhead;
• each thread needs a stack;
• thread metadata accumulates;
• synchronization is required for shared state;
• still scales poorly with huge numbers of connections.

This may be fine for smaller servers or long-lived low-rate connections, but not for large high-performance servers.

Both accept-and-fork and thread-per-connection use the same basic strategy:

Keep the socket interface synchronous, and use program concurrency to handle connection concurrency.

That means:

• one process/thread may block on one connection;
• other processes/threads can continue handling other connections.

The alternative is to change or augment the interface so that one thread can handle many connections asynchronously.

This is what select, poll, and epoll are for.

select asks the OS:

Among these file descriptors, which ones are ready for operations?

API:

int select(int nfds,
           fd_set *readfds,
           fd_set *writefds,
           fd_set *exceptfds,
           struct timeval *timeout);

It works with sets of file descriptors.

There are three main sets:

1. readfds
   • FDs where the application wants to know if reading is possible.
   • For a listening socket, “readable” means accept may succeed.
   • For a connection socket, “readable” means recv may return data.
2. writefds
   • FDs where the application wants to know if writing is possible.
   • Important after a non-blocking send failed with EAGAIN.
3. exceptfds
   • FDs where the application wants to know about exceptional/error conditions.

If timeout is NULL, select blocks until at least one FD is ready.

If a timeout is provided, select waits at most that long.

Return values:

• positive number:
  • number of ready descriptors;
• zero:
  • timeout expired;
• negative:
  • error.

select modifies the FD sets passed into it.

After select returns:

• FDs that are not ready are removed from the sets;
• ready FDs remain set.

Therefore, the application often has to rebuild the sets before every call.

select has major scalability issues.

Reasons:

1. FD sets are bitsets.
   • To check whether an FD is ready, the application uses something like FD_ISSET.
   • It may need to scan many possible FDs.
2. The operation is destructive.
   • The application must rebuild the sets repeatedly.
3. The kernel also has to inspect the sets.
   • For many connections, this becomes expensive.

Example:

If a server tracks 100,000 connections and only one has an event:

• application builds a huge read set;
• kernel scans it;
• kernel removes 99,999 entries;
• application scans to find the one ready FD;
• next loop, the set must be rebuilt.

This was acceptable in early Unix systems, where the number of connections was small. It is not ideal for modern large servers.

The read set is not always simply “all connections.”

If the server reads requests faster than it can process or reply, an attacker or aggressive client may make the server buffer too many pending requests.

So a server may temporarily remove a connection from the read set until it has finished sending earlier responses.

The write set also requires care.

Most sockets are writable most of the time.

If all sockets are always in the write set, select may constantly report them as writable.

Therefore, applications usually add a socket to the write set only when they have pending output that could not be sent immediately.

int epoll_create(int size);

int epoll_ctl(int epfd, int op, int fd,
              struct epoll_event *event);

int epoll_wait(int epfd,
               struct epoll_event *events,
               int maxevents,
               int timeout);

struct epoll_event {
    uint32_t events;
    epoll_data_t data;
};

Operations for epoll_ctl:

EPOLL_CTL_ADD
EPOLL_CTL_MOD
EPOLL_CTL_DEL

epoll_create creates an epoll object and returns a file descriptor for it.

This FD represents a stateful kernel object that remembers which file descriptors and events the application is interested in.

The old size parameter is mostly historical.

epoll_ctl changes the interest set.

Examples:

• add this FD and tell me about read events;
• modify this FD so I now care about write events;
• remove this FD from the epoll object.

The kernel keeps this state across calls.

The data field is opaque to the kernel.

The kernel stores it and gives it back when an event occurs.

Applications often use it to store:

• the FD itself;
• a pointer to a connection state structure;
• an application-level identifier.

This avoids an extra lookup when an event arrives.

Example idea:

event.events = EPOLLIN;
event.data.ptr = connection_state_pointer;
epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &event);

Later, epoll_wait returns the same pointer.

epoll_wait retrieves ready events.

Important distinction:

• the number of registered FDs may be huge;
• maxevents only says how many ready events to return in this call.

Example:

• 100,000 FDs registered;
• application passes an output array of size 64;
• epoll_wait returns up to 64 ready events.

This decouples:

• state management: which FDs are registered;
• event retrieval: which events are ready now.

This is the main scalability benefit.

For poll and epoll_wait, common timeout conventions are:

The lecturer briefly mentions another complexity.

Level-triggered behavior:

• the event is reported as long as the condition is true;
• example: FD is currently readable.

Edge-triggered behavior:

• the event is reported when the state changes;
• example: FD becomes readable or writable.

Edge-triggered mode can reduce repeated notifications, but requires more careful application logic.

The application must fully drain or process the available state, otherwise it may miss the next opportunity.

epoll works on file descriptors, not only network sockets.

It can also be used with:

• pipes;
• eventfds;
• files in some cases;
• timerfds;
• other kernel objects exposed as FDs.

A common pattern is to include a pipe or eventfd in an epoll set so another thread can wake up the event loop.

With epoll, a server can handle many connections with one thread.

The event loop is roughly:

while true:
    events = epoll_wait(...)
    for event in events:
        connection = event.data.ptr
        handle whatever is currently possible

This decouples:

• number of concurrent connections;
• number of OS threads.

A server may still use multiple threads to use multiple CPU cores, but it no longer needs one thread per connection.

This is the basis of many asynchronous runtimes and event loops, including ideas used by:

• Node.js;
• Python async frameworks;
• Rust async runtimes;
• many high-performance servers.

The cost:

The application must explicitly store all state that would otherwise live on a synchronous call stack.

Example:

If a non-blocking send sends only the first 3 KiB of a response, the application must remember:

• which response was being sent;
• how many bytes were already sent;
• which connection it belongs to;
• what to do when the socket becomes writable again.

In synchronous code, this state naturally lives on the stack. In asynchronous code, it must live in explicit connection/request state structures.

RDMA provides remote memory access.

The key primitive is a one-sided operation.

A one-sided operation means:

One side can initiate a memory read or write without the remote application actively executing a receive handler for that operation.

Examples:

• read memory from a remote machine;
• write data into memory on a remote machine.

Something still happens on the remote machine, of course. But it may be handled by:

• the OS;
• the NIC;
• RDMA hardware;
• firmware.

At the application interface level, the remote application does not have to run code for each access.

RDMA originated largely in:

• scientific computing;
• supercomputers;
• high-performance computing.

It is now also used heavily in:

• datacenters;
• machine learning training;
• distributed storage;
• high-performance key-value stores;
• systems that move large amounts of data.

Common RDMA-related technologies:

• InfiniBand;
• RoCE, meaning RDMA over Converged Ethernet;
• ibverbs;
• MPI implementations that use RDMA underneath.

There is not one universal socket-like RDMA API.

The most common low-level API is:

ibverbs

The name comes from InfiniBand, although verbs-style APIs are also used beyond pure InfiniBand contexts.

Another important interface is:

MPI - Message Passing Interface

MPI is common in HPC and supercomputing. It offers message passing and may also expose remote-memory-style operations. Many MPI implementations use RDMA internally when available.

Even though RDMA provides memory access, RDMA systems also support message send/receive.

Reason:

Remote memory access does not eliminate the need for coordination.

Shared memory is powerful, but shared state requires synchronization.

Problems:

• who owns a memory region?
• when is it safe to read?
• when is it safe to write?
• what if two nodes update concurrently?
• what if a pointer is followed while the remote side moves or frees the object?
• how are versions and consistency handled?

Sometimes sending a message is simpler and faster overall.

Example:

Instead of remotely traversing a complex data structure with many small RDMA reads, it may be better to send a request to the remote CPU and let it perform the operation locally.

The ibverbs interface is queue-based.

The application usually does not call a simple synchronous function like:

rdma_write(...);

and wait for it to finish.

Instead:

1. The application posts a work request into a queue.
2. The RDMA implementation performs the operation asynchronously.
3. A completion is later reported in a completion queue.

This makes the interface naturally asynchronous.

A queue pair is associated with a connection-like communication context.

It contains:

1. Send queue
   • used to send messages;
   • also used to initiate one-sided RDMA reads and writes.
2. Receive queue
   • used for receiving incoming messages;
   • the application posts buffers where incoming messages can be placed.

Important subtlety:

For message receives, the application must provide receive buffers in advance. The implementation needs somewhere to put incoming messages.

A completion queue reports completed operations.

A completion may mean:

• a send completed;
• an RDMA read completed;
• an RDMA write completed;
• a receive buffer was filled with an incoming message.

Typically, many queue pairs can share one completion queue.

This is useful because the application can check one completion queue to learn which operations completed across many connections.

This is similar in spirit to epoll:

• many outstanding operations;
• one place to ask what completed.

RDMA gives fine-grained control over completions.

When posting a work request, the application can ask for a completion signal.

Example flag:

IBV_SEND_SIGNALED

If every operation is signaled, the application gets a completion for every operation.

This is simple but may add overhead.

For performance, applications may signal only some operations.

Example:

• post five writes;
• ask for a completion only on the fifth;
• when the fifth completes, earlier ordered operations are known to have completed too.

This requires careful reasoning about ordering guarantees.

The lecture shows a simplified RDMA write example.

struct ibv_sge sge;

sge.addr   = (uint64_t)local_buf;
sge.length = DATA_SIZE;
sge.lkey   = local_mr->lkey;

struct ibv_send_wr wr = {0}, *bad_wr;

wr.sg_list = &sge;
wr.num_sge = 1;
wr.opcode  = IBV_WR_RDMA_WRITE;
wr.send_flags = IBV_SEND_SIGNALED;

wr.wr.rdma.remote_addr = remote_addr;
wr.wr.rdma.rkey        = rkey;

ibv_post_send(qp, &wr, &bad_wr);

struct ibv_wc wc;

while (ibv_poll_cq(cq, 1, &wc) < 1)
    ;

This example omits a lot of setup code.

Real RDMA setup includes:

• opening device context;
• creating protection domains;
• registering memory;
• creating queue pairs;
• connecting queue pairs;
• exchanging memory addresses and keys;
• handling errors.

The slide focuses only on the core transfer operation.

The ibv_sge structure is a scatter/gather entry.

It describes a local memory range.

Fields:

Scatter/gather exists because data may not be contiguous.

The interface can support operations like:

• take bytes from multiple local buffers and send/write them as one operation;
• read remote data and place it into local buffers.

In the simplified example, there is only one scatter/gather entry.

The ibv_send_wr structure describes the operation.

Important fields:

For an RDMA write:

• local scatter/gather list says where the local data is;
• remote address and rkey say where to put it remotely.

For an RDMA read:

• remote address and rkey say where to read from;
• local scatter/gather list says where to store the received data.

A crucial point:

ibv_post_send returning does not mean the RDMA operation has completed.

It only means the work request was accepted by the RDMA implementation.

The actual data transfer may still be pending.

The application must use completion information before assuming:

• the remote write is visible;
• the local buffer can be reused;
• the read data is available;
• the operation succeeded.

This is why the example polls the completion queue.

If an application has many outstanding operations, completions may arrive later and from many queue pairs.

The application therefore needs a way to match completions to its own state.

RDMA work requests usually include an application-provided identifier, often a work request ID.

The completion reports that identifier back.

The application can use it to find:

• which operation completed;
• which connection it belongs to;
• which buffer can now be reused;
• what state machine should run next.

Again, this is similar to asynchronous socket programming:

• high scalability;
• but explicit state management.

Before memory can be used for RDMA, it must be registered.

Registering memory creates a memory region.

The implementation returns keys:

The local key authorizes the local RDMA device to access the registered local memory.

It is used in scatter/gather entries.

Example:

sge.lkey = local_mr->lkey;

The remote key authorizes a peer to access the memory region remotely.

A peer needs the correct rkey to perform an RDMA read or write.

Example:

wr.wr.rdma.rkey = rkey;

The application must somehow send the peer:

• remote address;
• rkey;
• size / protocol information.

This exchange is part of the application or connection setup protocol.

The remote address used by RDMA is usually the address as seen by the remote application, not a raw physical address.

If RDMA is implemented in hardware, the NIC may need to translate that virtual address to a physical address.

Therefore, RDMA NICs may internally maintain translation information for registered memory regions.

This is analogous to page tables, but for the device’s access to registered memory.

At the interface level, the model is:

If the peer does not have the right rkey, it cannot access the memory region.

Implementations may also associate memory regions with particular queue pairs or protection domains.

This gives another level of authorization.

The lecture notes that security depends on the lower-level RDMA implementation and deployment context.

Common RDMA transports:

• InfiniBand;
• RoCE, RDMA over Converged Ethernet.

In a trusted supercomputer network, one may avoid expensive encryption because all nodes are controlled.

In less trusted networks, additional protection may be needed, such as Ethernet-level or IP-level security mechanisms.

RDMA is normally not something one exposes directly over the public internet.

RDMA can be very fast when the application is a good fit.

Advantages:

• avoids remote CPU involvement for one-sided operations;
• reduces software overhead;
• supports asynchronous high-throughput communication;
• works well for large structured data transfers;
• can be implemented efficiently in hardware;
• useful for HPC and ML workloads.

Good-fit examples:

• transferring matrix tiles;
• exchanging machine learning model weights;
• reading or writing known memory ranges;
• bulk data movement with little coordination;
• producer/consumer layouts where ownership is clear.

RDMA also has major drawbacks.

The application-level abstraction is not the same as the implementation.

Examples:

• messages can be implemented over TCP byte streams;
• RPC can be implemented over messages;
• remote memory access can be implemented in hardware, the OS, or a library;
• sockets expose file descriptors, but the protocol underneath may be TCP, UDP, Unix sockets, etc.

TCP gives a byte stream.

Therefore:

• no message boundaries;
• partial sends are possible;
• partial receives are possible;
• application protocols must define framing.

This is one of the most important practical lessons.

Non-blocking sockets prevent the application from getting stuck in one system call.

But then the application needs to know when to retry.

This leads to:

• select;
• poll;
• epoll;
• event loops;
• asynchronous runtimes.

select:

• bitsets;
• destructive update;
• scanning overhead;
• rebuilding sets.

poll:

• nicer array interface;
• not destructive in the same way;
• still O(n) per call.

epoll separates:

• registering interest;
• waiting for events.

This makes it better for large numbers of connections.

It also lets applications attach opaque state to events, which is useful for asynchronous event loops.

Accept-and-fork and thread-per-connection keep the programming model simple.

But they cost:

• process/thread creation;
• stacks;
• scheduling overhead;
• synchronization complexity;
• poor scalability with many connections.

Event-driven code can handle many connections with fewer threads.

But the application must explicitly track:

• pending requests;
• partial sends;
• buffers;
• timeouts;
• connection state;
• what to do when the next event arrives.

Sockets are mainly about messages and streams.

RDMA exposes remote memory operations.

This can be extremely fast for the right workloads, but it introduces:

• memory registration;
• keys;
• queue pairs;
• completion queues;
• explicit asynchronous state;
• synchronization problems.

RDMA helps most when:

• data locations are known;
• transfers are large or structured;
• coordination is simple;
• remote CPU overhead should be avoided.

Messages may be better when:

• operations are complex;
• shared state must be updated;
• pointer-heavy structures are involved;
• synchronization dominates the cost.