Linux Device Drivers (3rd Edition) [Electronic resources] نسخه متنی

5.6. Locking Traps

Many years of

experience
with locksexperience that predates Linuxhave shown that
locking can be very hard to get right. Managing concurrency is an
inherently tricky undertaking, and there are many ways of making
mistakes. In this section, we take a quick look at things that can go
wrong.

5.6.1. Ambiguous Rules

As has already been said above, a proper locking
scheme
requires clear and explicit rules. When you create a resource that
can be accessed concurrently, you should define which lock will
control that access. Locking should really be laid out at the
beginning; it can be a hard thing to retrofit in afterward. Time
taken at the outset usually is paid back generously at debugging
time.

As you write your code, you will doubtless encounter several
functions that all require access
to


structures protected by a specific
lock. At this point, you must be careful: if one function acquires a
lock and then calls another function that also attempts to acquire
the lock, your code deadlocks. Neither semaphores nor spinlocks allow
a lock holder to acquire the lock a second time; should you attempt
to do so, things simply hang.

To make your locking work properly, you have to write some functions
with the assumption that their caller has already acquired the
relevant lock(s). Usually, only your internal, static functions can
be written in this way; functions called from outside must handle
locking explicitly. When you write internal functions that make
assumptions about locking, do yourself (and anybody else who works
with your code) a favor and document those assumptions explicitly. It
can be very hard to come back months later and figure out whether you
need to hold a lock to call a particular function or not.

In the case of scull, the design decision taken
was to require all functions invoked directly from system calls to
acquire the semaphore applying to the device structure that is
accessed. All internal functions, which are only called from other
scull functions, can then assume that the
semaphore has been properly acquired.

5.6.2. Lock Ordering Rules

In systems with a large


number of locks (and the kernel is
becoming such a system), it is not unusual for code to need to hold
more than one lock at once. If some sort of computation must be
performed using two different resources, each of which has its own
lock, there is often no alternative to acquiring both locks.

Taking multiple locks can be dangerous, however. If you have two
locks, called Lock1 and
Lock2, and code needs to acquire both at the
same time, you have a potential deadlock. Just imagine one thread
locking Lock1 while another simultaneously takes
Lock2. Then each thread tries to get the one it
doesn't have. Both
threads will
deadlock.

The solution to this problem is usually simple: when multiple locks
must be acquired, they should always be acquired in the same order.
As long as this convention is followed, simple deadlocks like the one
described above can be avoided. However, following lock ordering
rules can be easier said than done. It is very rare that such rules
are actually written down anywhere. Often the best you can do is to
see what other code does.

A couple of rules of thumb can help. If you must obtain a lock that
is local to your code (a device lock, say) along with a lock
belonging to a more central part of the kernel, take your lock first.
If you have a combination of semaphores and spinlocks, you must, of
course, obtain the semaphore(s) first; calling
down (which can sleep) while holding a spinlock
is a serious error. But most of all, try to avoid situations where
you need more than one lock.

5.6.3. Fine- Versus Coarse-Grained Locking

The first Linux kernel that supported

multiprocessor systems was 2.0; it
contained exactly one spinlock. The big kernel
lock turned the entire kernel into one large critical
section; only one CPU could be executing kernel code at any given
time. This lock solved the concurrency problem well enough to allow
the kernel developers to address all of the other issues involved in
supporting SMP. But it did not scale very well. Even a two-processor
system could spend a significant amount of time simply waiting for
the big kernel lock. The performance of a four-processor system was
not even close to that of four independent machines.

So, subsequent kernel releases have included finer-grained locking.
In 2.2, one spinlock controlled access to the block I/O subsystem;
another worked for networking, and so on. A modern kernel can contain
thousands of locks, each protecting one small resource. This sort of
fine-grained locking can be good for scalability; it allows each
processor to work on its specific task without contending for locks
used by other processors. Very few people miss the big kernel
lock.^[3]

^[3] This lock still exists in 2.6, though it covers
very little of the kernel now. If you stumble across a
lock_kernel call, you have found the big kernel
lock. Do not even think about using it in any new code,
however.

Fine-grained locking comes at a cost, however. In a kernel with
thousands of locks, it can be very hard to know which locks you
needand in which order you should acquire themto
perform a specific operation. Remember that locking bugs can be very
difficult to find; more locks provide more opportunities for truly
nasty locking bugs to creep into the kernel. Fine-grained locking can
bring a level of complexity that, over the long term, can have a
large, adverse effect on the maintainability of the kernel.

Locking in a device driver is usually relatively straightforward; you
can have a single lock that covers everything you do, or you can
create one lock for every device you manage. As a general rule, you
should start with relatively coarse locking unless you have a real
reason to believe that contention could be a problem. Resist the urge
to optimize prematurely; the real performance constraints often show
up in unexpected places.

If you do suspect that lock contention is hurting performance, you
may find the
lockmeter

tool useful. This patch (available at http://oss.sgi.com/projects/lockmeter/)
instruments the kernel to measure time spent waiting in locks. By
looking at the report, you are able to determine quickly whether lock
contention is truly the problem or not.