Asynchronous I/O

Applications usually need to simultaneously handle many event sources, such as keyboard, GUI interface, interprocess or most importantly network communication. Although many high-performance libraries and language features exist to solve that, it is useful to understand how those libraries work internally and which operating system interfaces they use.

A naive solution to this problem is to create a thread for each event source (e.g. TCP connection), however scalability of such an approach is limited. Creating and terminating a thread is not a cheap operation and each thread requires a significant amount of memory (at least for its stack). Hence, handling of millions of connections with such an approach is likely impossible.

The core of every high-performance networking application is usually an event loop processing events from multiple sources (TCP connections) in a single thread. This event loop usually runs in multiple threads, one for each available CPU core. Example of using this design are nginx web server, Node.js with its libuv library, all runtimes of languages implementing async/await style of programming etc.

Task assignment

Your goal is to implement a single-threaded event loop-based application, using the epoll system call that can handle timers, keyboard input, and TCP connections simultaneously. The base of the application (epoll loop, timer, and keyboard input) is already implemented in the provided template. Do not use multiple threads!

Steps:

  1. Download the epoll application template (C++ or plain C).

  2. Compile and run the application. See the instructions in the README of the corresponding repository.

  3. Extend the application by a TCP server that will listen on port 12345 (hints are provided below). Use epoll mechanism (a few related system calls) to handle events on TCP. Take into account multiple simultaneous connections. If you don't like our implementation, feel free to write the application yourself but keep existing functionality.

  4. For each TCP connection, your application should calculate the length of incoming lines and send the length back as a newline-terminated ASCII string immediately after receiving '\n', e.g.:

    receive: "Test string\n"
send: "11\n" // keep the connection established

  5. You can test basic functionality by using the netcat program:

    nc localhost 12345

  6. Upload your solution as .zip, .tar.gz, or .tgz archive to the BRUTE. Extracted archive must be compilable with the make, CMake, or Meson build systems. The upload system will test your solution, but the evaluation is manual.

    You can also check your solution using t.sh script provided in the application templates.

Using epoll

In UNIX, “everything” is represented as a file descriptor. Therefore, we are interested in monitoring those file descriptors for events such as “key was pressed”, “new client is connecting” or “a client sent us data”. All these events are represented by “file descriptor is readable/writable” events. Epoll is high performance mechanism, which enables waiting for multiple events on multiple file descriptors.

  • First, it is necessary to create epoll instance (context) using epoll_create1().

  • Then, file descriptors to be monitored can be added to or removed from the given epoll instance. The epoll_event structure and epoll_ctl() function are used for that. Each event can be edge or level triggered. When edge-triggered behavior is chosen (events |= EPOLLET), the application is notified about each event only once – e.g. you need to read all data from a given file – you will not be notified again, if the data are not read completely.

  • Finally, you can wait for events on all monitored file descriptors by calling epoll_wait() function in the main loop.

Creating TCP server in C

TCP provides a reliable, ordered, and error-checked transport of data streams. Streams do not preserve boundaries between messages. It means, that in one read you can receive only part of a sent message or multiple messages.

First of all, we have to include needed headers:

#include <sys/socket.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <fcntl.h>

Then, create a TCP socket (AF_INET → IPv4, SOCK_STREAM → TCP):

int sfd = socket(AF_INET, SOCK_STREAM, 0);

Create a socket address and bind the socket to this address:

short int port = 12345;
struct sockaddr_in saddr;

memset(&saddr, 0, sizeof(saddr));
saddr.sin_family      = AF_INET;              // IPv4
saddr.sin_addr.s_addr = htonl(INADDR_ANY);    // Bind to all available
interfaces
saddr.sin_port        = htons(port);          // Requested port

bind(sfd, (struct sockaddr *) &saddr, sizeof(saddr))

Our socket should be non-blocking. Why? By default, sockets are blocking, meaning that a call to read() blocks the calling thread until some data is received. While the thread is blocked (which can be for long time), events from other sources cannot be handled. Switching the socket (or any other file descriptor) can be performed with fnctl:

int flags = fcntl(sfd, F_GETFL, 0); # TODO: Error checking
flags |= O_NONBLOCK;
fcntl(sfd, F_SETFL, flags);         # TODO: Error checking

When you try to read from a non-blocking socket, which has no data available, you receive an error (EAGAIN or EWOULDBLOCK) instead of blocking.

We would like to create a server, and therefore we need to listen for connections:

listen(sfd, SOMAXCONN);

Now, we are able to be notified about incoming connections. In order to accept the incoming connection, call function accept:

cfd = accept(sfd, NULL, NULL);

Set the new file descriptor for non-blocking operations (same as above).

Then, we can communicate with the client using read() and write() system calls:

#include <err.h>
#define BUF_SIZE 42
char buffer[BUF_SIZE];
int count;
# Read data
count = read(cfd, buffer, BUF_SIZE);
if (count == -1)
    err(1, "read");
# Reply with read data
count = write(cfd, buffer, count);
if (count == -1)
    err(1, "write");

After the end of communication, close the connection:

close(cfd);

Hints

  • You probably want to create two new classes/structs.
  • You can use delete an object via its this pointer. See this C++ FAQ.
  • Consider fragmentation of incoming data (“Hello World!\n” can arrive as two different messages – “He” and “llo World\n”).
  • Don't close the connection after receiving first \n.
  • Read status of all system calls (especially read) – if an error occurs, read the errno variable.
  • Try understanding what do epoll edge and level triggers mean.
  • System calls like read and write can block unless the respective file descriptor is set to O_NONBLOCK mode.
  • You can use strace tool as a simple way of debugging various problems with your solution.