Transports and protocols (low-level API)

Gruvi is built around the transport/protocol abstraction layers that are documented in PEP 3156 and implemented by asyncio.

A transport is a standard interface to a communications channel. Different types of channels implement different interfaces. Since Gruvi uses libuv / pyuv, its transports are mostly wrappers around the various pyuv.Handle classes. Transport methods are always non-blocking.

A protocol is a callback based interface that is connected to a transport. The transport calls specific callbacks on the protocol when specific events occur. The callbacks are always non-blocking, but a protocol may expose certain protocol operations are part of their API that are switch points.

As a programmer you will use transports and protocols occasionally, but you will mostly use the higher-level interface provided by Gruvi. The exception is when adding support for a new protocol.

Transports

The following transport classes are available:

exception TransportError

A transport error.

class BaseTransport

Base class for pyuv based transports. There is no public constructor.

start(protocol)

Bind to protocol and start calling callbacks on it.

get_write_buffer_size()

Return the total number of bytes in the write buffer.

get_write_buffer_limits()

Return the write buffer limits as a (low, high) tuple.

set_write_buffer_limits(high=None, low=None)

Set the low and high watermark for the write buffer.

resume_reading()

Resume calling callbacks on the protocol.

As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes.

pause_reading()

Pause calling callbacks on the protocol.

This method may be called even if reading was already paused or a close is pending. See the note in resume_reading().

close()

Close the transport after all oustanding data has been written.

abort()

Close the transport immediately.

get_extra_info(name, default=None)

Get transport specific data.

The following information is available for all transports:

Name Description
'handle' The pyuv handle that is being wrapped.
class Transport(handle, server_hostname=None, mode='rw')

A connection oriented transport.

The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.Stream instance, so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY.

The server_hostname argument specifies the host name of the remote peer, if available and applicable for the handle.

The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw').

get_write_buffer_size()

Return the total number of bytes in the write buffer.

resume_reading()

Resume calling callbacks on the protocol.

As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes.

pause_reading()

Pause calling callbacks on the protocol.

This method may be called even if reading was already paused or a close is pending. See the note in resume_reading().

write(data, handle=None)

Write data to the transport.

writelines(seq)

Write all elements from seq to the transport.

write_eof()

Shut down the write direction of the transport.

can_write_eof()

Whether this transport can close the write direction.

get_extra_info(name, default=None)

Get transport specific data.

In addition to the fields from BaseTransport.get_extra_info(), the following information is also available:

Name Description
'sockname' The socket name i.e. the result of the getsockname() system call.
'peername' The peer name i.e. the result of the getpeername() system call.
'winsize' The terminal window size as a (cols, rows) tuple. Only available for pyuv.TTY handles.
'unix_creds' The Unix credentials of the peer as a (pid, uid, gid) tuple. Only available for pyuv.Pipe handles on Unix.
'server_hostname' The host name of the remote peer prior to address resolution, if applicable.
class DatagramTransport(handle, mode='rw')

A datagram transport.

The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.UDP instance.

The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw').

get_write_buffer_size()

Return the total number of bytes in the write buffer.

resume_reading()

Resume calling callbacks on the protocol.

As a relaxation from the requirements in Python asyncio, this method may be called even if reading was already paused, and also if a close is pending. This simplifies protocol design, especially in combination with SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol, to complete handshakes.

pause_reading()

Pause calling callbacks on the protocol.

This method may be called even if reading was already paused or a close is pending. See the note in resume_reading().

sendto(data, addr=None)

Send a datagram containing data to addr.

The addr argument may be omitted only if the handle was bound to a default remote address.

SSL/TLS support

SSL and TLS support is available by means of a special SslTransport transport:

class SslTransport(handle, context, server_side, server_hostname=None, do_handshake_on_connect=True, close_on_unwrap=True)

An SSL/TLS transport.

The handle argument is the pyuv handle on top of which to layer the SSL transport. It must be a pyuv.Stream instance, so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY.

SSL transports are always read-write, so the handle provided needs to support that.

The context argument specifies the ssl.SSLContext to use. You can use create_default_context() to create a new context which also works on Python 2.x where ssl.SSLContext does not exist.

The server_side argument specifies whether this is a server side or a client side transport.

The optional server_hostname argument can be used to specify the hostname you are connecting to. You may only specify this parameter if your Python version supports SNI.

The optional do_handshake_on_connect argument specifies whether to to start the SSL handshake immediately. If False, then the connection will be unencrypted until do_handshake() is called. The default is to start the handshake immediately.

The optional close_on_unwrap argument specifies whether you want the ability to continue using the connection after you call unwrap() of when an SSL “close_notify” is received from the remote peer. The default is to close the connection.

get_extra_info(name, default=None)

Return transport specific data.

The following fields are available, in addition to the information exposed by Transport.get_extra_info().

Name Description
'ssl' The internal ssl.SSLObject instance used by this transport.
'sslctx' The ssl.SSLContext instance used to create the SSL object.
pause_reading()

Stop reading data.

Flow control for SSL is a little bit more complicated than for a regular Transport because SSL handshakes can occur at any time during a connection. These handshakes require reading to be enabled, even if the application called pause_reading() before.

The approach taken by Gruvi is that when a handshake occurs, reading is always enabled even if pause_reading() was called before. I believe this is the best way to prevent complex read side buffering that could also result in read buffers of arbitrary size.

The consequence is that if you are implementing your own protocol and you want to support SSL, then your protocol should be able to handle a callback even if it called pause_reading() before.

resume_reading()

Resume reading data.

See the note in pause_reading() for special considerations on flow control with SSL.

do_handshake()

Start the SSL handshake.

This method only needs to be called if this transport was created with do_handshake_on_connect set to False (the default is True).

The handshake needs to be synchronized between the both endpoints, so that SSL record level data is not incidentially interpreted as plaintext. Usually this is done by starting the handshake directly after a connection is established, but you can also use an application level protocol.

unwrap()

Remove the security layer.

Use this method only if you want to send plaintext data on the connection after the security layer has been removed. In all other cases, use close().

If the unwrap is initiated by us, then any data sent after it will be buffered until the corresponding close_notify response is received from our peer.

If the unwrap is initiated by the remote peer, then this method will acknowledge it. You need an application level protocol to determine when to do this because the receipt of a close_notify is not communicated to the application.

close()

Cleanly shut down the SSL protocol and close the transport.

create_default_context(purpose=None, **kwargs)

Create a new SSL context in the most secure way available on the current Python version. See ssl.create_default_context().

Protocols

A Protocols is a collection of named callbacks. A protocol is are attached to a transport, and its callbacks are called by the transport when certain events happen.

The following protocol classes are available:

exception ProtocolError

A protocol error.

class BaseProtocol(timeout=None)

Base class for all protocols.

The timeout argument specifies a default timeout for various protocol operations.

connection_made(transport)

Called when a connection is made.

connection_lost(exc)

Called when a connection is lost.

pause_writing()

Called when the write buffer in the transport has exceeded the high water mark. The protocol should stop writing new data.

resume_writing()

Called when the write buffer in the transport has fallen below the low water mark. The protocol can start writing data again.

class Protocol(timeout=None)

Base class for connection oriented protocols.

The timeout argument specifies a default timeout for various protocol operations.

data_received(data)

Called when a new chunk of data is received.

eof_received()

Called when an EOF is received.

class DatagramProtocol(timeout=None)

Base classs for datagram oriented protocols.

The timeout argument specifies a default timeout for various protocol operations.

datagram_received(data, addr)

Called when a new datagram is received.

error_received(exc)

Called when an error has occurred.

The following class does not exist in PEP 3156 but is a useful base class for most protocols:

class MessageProtocol(message_handler=None, timeout=None)

Base class for message oriented protocols.

Creating transports and protocols

Transports and protocols operate in pairs; there is always exactly one protocol for each transport. A new transport/protocol pair can be created using the factory functions below:

switchpoint create_connection(protocol_factory, address, ssl=False, server_hostname=None, local_address=None, family=0, flags=0, ipc=False, timeout=None, mode='rw')

Create a new client connection.

This method creates a new pyuv.Handle, connects it to address, and then waits for the connection to be established. When the connection is established, the handle is wrapped in a transport, and a new protocol instance is created by calling protocol_factory. The protocol is then connected to the transport by calling the transport’s start() method which in turn calls connection_made() on the protocol. Finally the results are returned as a (transport, protocol) tuple.

The address may be either be a string, a (host, port) tuple, a pyuv.Stream handle or a file descriptor:

  • If the address is a string, this method connects to a named pipe using a pyuv.Pipe handle. The address specifies the pipe name.
  • If the address is a tuple, this method connects to a TCP/IP service using a pyuv.TCP handle. The first element of the tuple specifies the IP address or DNS name, and the second element specifies the port number or service name.
  • If the address is a pyuv.Stream instance, it must be an already connected stream.
  • If the address is a file descriptor, then it is attached to a pyuv.TTY stream if os.isatty() returns true, or to a pyuv.Pipe instance otherwise.

The ssl parameter indicates whether an SSL/TLS connection is desired. If so then an ssl.SSLContext instance is used to wrap the connection using the ssl module’s asynchronous SSL support. The context is created as follows. If ssl is an SSLContext instance, it is used directly. If it is a function, it is called with the connection handle as an argument and it must return a context . If it is True then a default context is created using gruvi.create_default_context(). To disable SSL (the default), pass False. If SSL is active, the return transport will be an SslTransport instance, otherwise it will be a Transport instance.

The server_hostname parameter is only relevant for SSL connections, and specifies the server hostname to use with SNI (Server Name Indication). If no server hostname is provided, the hostname specified in address is used, if available.

The local_address keyword argument is relevant only for TCP transports. If provided, it specifies the local address to bind to.

The family and flags keyword arguments are used to customize address resolution for TCP handles as described in socket.getaddrinfo().

The mode parameter specifies if the transport should be put in read-only ('r'), write-only ('w') or read-write ('rw') mode. For TTY transports, the mode must be either read-only or write-only. For all other transport the mode should usually be read-write.

switchpoint create_server(protocol_factory, address=None, ssl=False, family=0, flags=0, ipc=False, backlog=128)

Create a new network server.

This creates one or more pyuv.Handle instances bound to address, puts them in listen mode and starts accepting new connections. For each accepted connection, a new transport is created which is connected to a new protocol instance obtained by calling protocol_factory.

The address argument may be either be a string, a (host, port) tuple, or a pyuv.Stream handle:

  • If the address is a string, this method creates a new pyuv.Pipe instance and binds it to address.
  • If the address is a tuple, this method creates one or more pyuv.TCP handles. The first element of the tuple specifies the IP address or DNS name, and the second element specifies the port number or service name. A transport is created for each resolved address.
  • If the address is a pyuv.Stream handle, it must already be bound to an address.

The ssl parameter indicates whether SSL should be used for accepted connections. See create_connection() for a description.

The family and flags keyword arguments are used to customize address resolution for TCP handles as described in socket.getaddrinfo().

The ipc parameter indicates whether this server will accept new connections via file descriptor passing. This works for pyuv.Pipe handles only, and the user is required to call Server.accept_connection() whenever a new connection is pending.

The backlog parameter specifies the listen backlog i.e the maximum number of not yet accepted active opens to queue. To disable listening for new connections (useful when ipc was set), set the backlog to None.

The return value is a Server instance.

Endpoints

Endpoints wrap transports and protocols for client and server side connections, and provide a more object-oriented way of working with them.

class Endpoint(protocol_factory, timeout=None)

A communications endpoint.

The protocol_factory argument constructs a new protocol instance. Normally you would pass a Protocol subclass.

The timeout argument specifies a timeout for various network operations.

timeout

The network timeout.

close()

Close the endpoint.

class Client(protocol_factory, timeout=None)

A client endpoint.

transport

Return the transport, or None if not connected.

protocol

Return the protocol, or None if not connected.

switchpoint connect(address, **kwargs)

Connect to address and wait for the connection to be established.

See create_connection() for a description of address and the supported keyword arguments.

switchpoint close()

Close the connection.

class Server(protocol_factory, timeout=None)

A server endpoint.

addresses

The addresses this server is listening on.

connections

An iterator yielding the (transport, protocol) pairs for each connection.

accept_connection(handle, ssl=False)

Accept a new connection on handle. This method needs to be called when a connection was passed via file descriptor passing.

handle_connection(client, ssl)

Handle a new connection with handle client.

This method exists so that it can be overridden in subclass. It is not intended to be called directly.

connection_made(transport, protocol)

Called when a new connection is made.

connection_lost(transport, protocol, exc=None)

Called when a connection is lost.

switchpoint listen(address, ssl=False, family=0, flags=0, ipc=False, backlog=128)

Create a new transport, bind it to address, and start listening for new connections.

See create_server() for a description of address and the supported keyword arguments.

switchpoint close()

Close the listening sockets and all accepted connections.

switchpoint run()

Run the event loop and start serving requests.

This method stops serving when a CTRL-C/SIGINT is received. It is useful for top-level scripts that run only one server instance. In more complicated application you would call Hub.switch() directly, or wait on some kind of “request to shutdown” event.