How is twisted's Deferred implemented?

Question

Does it spawn a new thread underneath? If classical web server spawns a thread to serve a HTTP request and with Twisted web I have to spawn a Deferred() each time I want to query mysql - where's the gain? Looks like it doesn't make sens if it spawned a thread, so how's it implemented?

Answer 1

As others have said, a Deferred on its own is just a promise of a value, and a list of things to do when the value arrives (or when there is a failure getting the value).

How they work is like this: some function sees that the value it wants to return is not yet ready. So it prepares a Deferred, and then arranges somehow for that Deferred to be called back ("fired") with the value once it's ready. That second part is what may be causing your confusion; Deferreds on their own don't control when or how they are fired. It's the responsibility of whatever created the Deferred.

In the context of a whole Twisted app, nearly everything is event-based, and events are managed by the reactor. Say your code used twisted.web.client.getPage(), so it now has a Deferred that will be fired with the result of the http fetch. What that means is that getPage() started up a tcp conversation with the http server, and essentially installed handlers in the reactor saying "if you see any traffic on this tcp connection, call a method on this Protocol object". And once the Protocol object sees that it has received the whole page you asked for, it fires your Deferred, whereupon your own code is invoked via that Deferred's callback chain.

So everything is callbacks and hooks, all the way down. This is why you should never have blocking code in a Twisted app, unless on a separate thread- because it will stop everything else from being handled too.

Does that help?

Answer 2

A Deferred in Python doesn't do anything, it doesn't have anything to do with threads or processes or anything else.

It is an abstraction of a promise for future data to be delivered and how that data is mapped to the receiving callback functions.

This is documented very well.

How to Twisted achieves async behavior is documented very well on the Twisted documentation and newsgroup archives, simple Google searches will find what you need. It has nothing to do with the implementation of Deferred. If you still don't understand you need to ask that in another question for specific information, not in a comment to this answer.

That said, Twisted is kind of dead as far as concurrency goes, it will never scale to multi-core processors as well as something like Erlang, which is what I moved to from Twisted when Twisted quit scaling for me.