Credit: Brian Bush, Troy Melhase, David Beach, Martin Miller
You have a task that seems suited to multithreading, but you don't want to incur the overhead that real thread-switching would impose.
Generators were designed to simplify iteration, but they're also quite suitable as a basis for cooperative multitasking, also known as microthreading:
import signal
# credit: original idea was based on an article by David Mertz
# http://gnosis.cx/publish/programming/charming_python_b7.txt
# some example 'microthread' generators
def empty(name):
"" This is an empty task for demonstration purposes. ""
while True:
print "<empty process>", name
yield None
def terminating(name, maxn):
"" This is a counting task for demonstration purposes. ""
for i in xrange(maxn):
print "Here %s, %s out of %s" % (name, i, maxn)
yield None
print "Done with %s, bailing out after %s times" % (name, maxn)
def delay(duration=0.8):
"" Do nothing at all for 'duration' seconds. ""
import time
while True:
print "<sleep %d>" % duration
time.sleep(duration)
yield None
class GenericScheduler(object):
def _ _init_ _(self, threads, stop_asap=False):
signal.signal(signal.SIGINT, self.shutdownHandler)
self.shutdownRequest = False
self.threads = threads
self.stop_asap = stop_asap
def shutdownHandler(self, n, frame):
"" Initiate a request to shutdown cleanly on SIGINT.""
print "Request to shut down."
self.shutdownRequest = True
def schedule(self):
def noop( ):
while True: yield None
n = len(self.threads)
while True:
for i, thread in enumerate(self.threads):
try: thread.next( )
except StopIteration:
if self.stop_asap: return
n -= 1
if n==0: return
self.threads[i] = noop( )
if self.shutdownRequest:
return
if _ _name_ _== "_ _main_ _":
s = GenericScheduler([ empty('boo'), delay( ), empty('foo'),
terminating('fie', 5), delay(0.5),
], stop_asap=True)
s.schedule( )
s = GenericScheduler([ empty('boo'), delay( ), empty('foo'),
terminating('fie', 5), delay(0.5),
], stop_asap=False)
s.schedule( )Microthreading (or cooperative multitasking) is an important technique. If you want to pursue it in earnest for complex uses, you should definitely look up the possibilities of Christian Tismer's Stackless, a Python version specialized for microthreading, at http://www.stackless.com/. However, you can get a taste of cooperative multitasking without straying from Python's core, by making creative use of generators, as shown in this recipe.
A simple approach to cooperative multitasking, such as the one presented in this recipe, is not suitable when your tasks must perform long-running work, particularly I/O tasks that may involve blocking system calls. For such applications, look into real threading, or, as a strong alternative, look into the event-driven approach offered by module asyncore in the Python Standard Library (on a simple scale) and by package Twisted at http://twistedmatrix.com/products/twisted (on a grandiose scale). But if your application has modest I/O needs, and you can slice up any computation your tasks perform into short chunks, each of which you can end with a yield, this recipe may be just what you're looking for.
David Mertz's site, chock-full of idiosyncratic, fascinating ideas, is at http://gnosis.cx/; Christian Tismer's Stackless Python, the best way to do cooperative multitasking in Python (and much else besides), is at http://www.stackless.com/; Twisted Matrix, the best way to do event-driven (asynchronous) programming, is at http://twistedmatrix.com/.