Editor's note: In his upcoming session at O'Reilly's Emerging Technology Conference, Itamar will be showing attendees, step by step, how to use Twisted to develop networked applications. If you want to learn how Twisted can help you save large amounts of development time, don't miss his tutorial, slated for the opening day of Etech 2004.
Twisted is an open source networking framework, implemented in Python. It is designed to support both clients and servers and run on multiple operating systems and platforms. This article is a brief introduction to Twisted's capabilities and design goals.
Network programs are complex systems. They may need to run on multiple operating systems, support multiple protocols, operate in diverse environments, and integrate with existing systems. A mail client will typically need to support SMTP, POP3, IMAP, all running over TCP and TLS, and may need to support additional protocols as well (LDAP, for example). An integrated mail server must use SMTP (which in turn requires DNS client support), POP3, IMAP; but also HTTP for webmail support. Most existing networking frameworks are not suited to building these sorts of applications, as they tend to be either too high level and application-specific (e.g., web frameworks) or too low level and platform-specific (e.g., low-level C networking libraries). Custom development of new protocols is hard with these high-level frameworks, and takes too much effort with low-level frameworks — sending an email requires implementing a SMTP protocol client from scratch.
Twisted is an attempt to build a framework capable of supporting the needs of modern network applications, from simple custom protocol development to large-scale multi-protocol integrated systems. It provides:
|
Itamar Shtull-Trauring The Twisted event-driven networking framework (http://www.twistedmatrix.com) is a powerful, cross-platform toolkit for creating networked applications. In this tutorial you will learn, step by step, how to use Twisted to develop networked applications, and how this can save large amounts of development time. O'Reilly Emerging Technology Conference |
Twisted is divided into several packages, each providing different services. Typically, higher-level packages build on lower-level packages, allowing developers to depend only on those packages necessary to their application. The following sections describe some of Twisted's basic packages. There are many more, including a generic, remote-object protocol called Perspective Broker, relational database support, testing and debugging tools, and more.
twisted.internetAt Twisted's core, the twisted.internet package provides a
networking event loop. The event-loop networking model was chosen over threads
as it tends to be more scalable, integrates well with GUI applications, and is
supported by virtually all operating systems. In an event loop, a single thread
responds to network events (e.g. "socket readable"), deals with the
appropriate processing (reading from socket and passing data to the appropriate
handler), and then continues processing.
Much of the effort developing
twisted.internet involved abstracting these low-level,
platform-specific "events," wrapping them with generic APIs, and allowing the use
of the same code, regardless of the underlying platform. While
twisted.internet is not as fast as servers implemented on top of C
event loops, it is usually fast enough for all but the most demanding
applications.
High-level APIs for TCP, SSL, UDP, Unix domain sockets and other transports build on the event loop. Protocol implementation is separated from transport implementation, so protocols can run more or less transparently on top of transports of the same kind. A protocol developed for TCP will typically support SSL without any changes. For client connections, Twisted provides high-level functionality, such as reconnection on failure or disconnect, connection timeouts, and the ability to cancel connections, all in an implementation-generic way. The event loop also supports scheduling, thread integration, hooks for non-blocking DNS lookups, and other commonly required tasks.
The event loop is implemented atop operating-system-specific APIs. This
includes the standard select() system call, available on both Unix
and Windows, as well as other platform-specific event loops. Although the
networking code is generic, the event loop implementation is exposed to
developers who require platform-specific functionality, be it file descriptors
on Unix or events on Windows. In addition, the event loop can integrate with
GUI toolkits (GTK, Qt, Win32, Mac OS X's Cocoa, and so on), allowing generic
networking code to be used for GUI clients as well.
twisted.credBrowsing through freshmeat will show at
least 15 POP3 servers available for download. Different servers will support
different authentication sources and different storage backends, but large
parts of the implementations are duplicated. The twisted.cred
package is an attempt to solve this duplication of effort, by promoting server
development using generic components that can be reused in different contexts.
Authorization is separated from the application code if at all possible. Both
are separated from the protocol implementation. The twisted.cred
package provides classes and interfaces that allow separation of the different
development roles used to build a server in a consistent, flexible way.
These roles include:
Protocol developer
Creates the protocol implementation (e.g., POP3), which is designed to talk
to an abstract implementation of the backend, defined as an interface (e.g., an
IMailbox interface for a POP3 backend).
Application developer
Creates a backend for a specific protocol. There can be multiple backends,
for example, a Maildir IMailbox implementation as well as a MySQL backend
implementation created by a different developer.
Authorization developer
Creates generic authorization checkers for specific types of credentials,
which may be reusable across protocols. For example, a username/password
checker that uses /etc/passwd will probably work with both POP3
and HTTP.
Deployment
Tie together a protocol with an appropriate credential checker and backend
in order to build a running server. For example, tying a POP3 protocol to a
/etc/password credential checker and a MySQL mail storage.
Several protocols in Twisted use the twisted.cred package,
including HTTP, POP3, IMAP, and FTP, allowing them to share credential checkers
and to support pluggable backend implementations.
|
Related Reading 
Python in a Nutshell |
twisted.protocolsWhile some applications use Twisted exclusively to develop custom protocols, many applications use common, Internet-standard protocols. Twisted provides implementations of many of the more common ones, typically supporting both client and server implementations. These implementations do not enforce policies (e.g., the FTP code is not limited to serving files from the filesystem), allowing maximum developer flexibility. The supported protocols include HTTP, FTP, DNS, SMTP, POP3, IMAP, SSH, SIP, Jabber, TOC and Oscar (AIM), MSN, IRC, NNTP, and others.
In addition to these low-level protocol implementations, Twisted also
provides higher-level frameworks, implementing the functionality commonly
implemented on top of certain protocols. Two of these frameworks are the
twisted.web package, an object-publishing web server, and
twisted.names, a DNS server framework.
twisted.applicationComplex applications are more than just a single protocol implementation. To
aid applications development, tying together multiple subsystems, and to ease
deployment woes, Twisted provides the twisted.application package.
Applications are designed as trees of services, providing an abstraction for
related and dependent services.
An application developed in this way, using the interfaces and
infrastructure provided by this package, is easy to deploy. An application
deployment file is constructed either as a specially structured Python script
(which might load configuration information from external application-specific
files) or uses plug-ins for the mktap utility that comes with
Twisted. mktap generates "TAP" files, which are simple, pre-configured server
setups. These deployment files can then be run using twistd, a
command-line tool, which provides logging options, daemonization on UNIX,
profiling, and so on. For example, using the built-in "web" plug-in that
provides a web server configuration, makes starting a web server daemon in Twisted quite simple:
$ mktap web --path=/var/www
$ twistd -f web.tapBoth Python deployment files and TAP files can also be used to generated installable Debian or RPM packages. An experimental utility — which should be available in future versions of Twisted — allows easy generation of installable, self-contained Windows NT services.
Several applications, both open source and commercial, use Twisted. A brief overview of two of them demonstrates the power and flexibility of the framework.
Buildbot is a "system to automate the compile/test cycle required by most software projects to validate code changes." When a developer checks in code, the Buildbot build master is notified, and the build slaves (which can be run on different machines, and different operating systems such as Windows or Linux) are notified by the master (using the Perspective Broker protocol). The slaves check out the latest version of the code and run the applicable tests for the software. The status of the builds can be accessed via a web page, an IRC bot, or using a GUI tool, demonstrating Twisted's support for multiple protocols. The Twisted project itself uses Buildbot to run its tests.
Another project that uses Twisted is Quotient, an integrated messaging server. Quotient supports email — it acts as an SMTP server, POP3 server, and IMAP server, and also includes a sophisticated web-based mail-browsing and administration interface. Quotient also integrates VoIP management services using the SIP protocol. Using Twisted's multiple-protocol support and robust server architecture, Quotient can build a superior messaging server with capabilities that would be extremely hard to achieve with traditional single-purpose servers.
Of course, Twisted can be used to implement simpler systems, and has been used to build applications ranging from GUI VoIP clients to complex web applications. To learn more about Twisted, and how you can use it for your software, you can visit the Twisted web site. I will be giving a Twisted tutorial at the Emerging Technology Conference in February, and you can meet most of the Twisted development team at the PyCon conference this March.
Itamar Shtull-Trauring has contributed to Python and Zope and is one of the main developers of Twisted.
Return to Python DevCenter.
Copyright © 2009 O'Reilly Media, Inc.