Python Cookbook 2Nd Edition Jun 1002005 [Electronic resources] نسخه متنی

Recipe 19.19. Simplifying Queue-Consumer Threads

Credit: Jimmy Retzlaff, Paul Moore

Problem

You want to code a consumer thread which gets work requests off a
queue one at a time, processes each work request, and eventually
stops, and you want to code it in the simplest possible way.

Solution

This task is an excellent use case for the good old
Sentinel idiom. The producer thread, when
it's done putting actual work requests on the queue,
must finally put a sentinel value, that is, a
value that is different from any possible work request.
Schematically, the producer thread will do something like:

for input_item in stuff_needing_work:
work_request = make_work_request(input_item)
queue.put(work_request)
queue.put(sentinel)

where sentinel must be a
"well-known value", different from
any work_request object that might be put on the
queue in the first phase.

The consumer thread can then exploit the built-in function
iter:

for work_request in iter(queue.get, sentinel):
process_work_request(work_request)
cleanup_and_terminate( )

Discussion

Were it not for built-in function iter, the
consumer thread would have to use a slightly less simple and elegant
structure, such as:

while True:
work_request = queue.get( )
if work_request == sentinel:
break
process_work_request(work_request)
cleanup_and_terminate( )

However, the Sentinel idiom is so useful and
important that Python directly supports it with built-in function
iter. When you call iter with
just one argument, that argument must be an iterable object, and
iter returns an iterator for it. But when you call
iter with two arguments, the first one must be a
callable which can be called without arguments, and the second one is
an arbitrary value known as the sentinel. In the
two-argument case, iter repeatedly calls the first
argument. As long as each call returns a value
!=sentinel, that value becomes an item in the
iteration; as soon as a call returns a value
==sentinel, the iteration stops.

If you had to code this yourself as a generator, you could write:

def iter_sentinel(a_callable, the_sentinel):
while True:
item = a_callable( )
if item == the_sentinel: break
yield item

But the point of this recipe is that you don't have
to code even this simple generator: just use the power that Python
gives you as part of the functionality of the built-in function
iter!

Incidentally, Python offers many ways to make
sentinel valuesmeaning values that
compare equal only to themselves. The simplest and most direct way,
and therefore the one I suggest you always use for this specific
purpose, is:

sentinel = object( )

See Also

Documentation for iter in the Library
Reference and Python in a
Nutshell.