There are two fundamental ways of kernel memory allocation, both of which are built on top of the kernel’s page allocator. (1) The slab allocator, allocates physically contiguous memory in the kernel’s address space, which is typically accessed with kmalloc(). Therefore, it’s faster, without needing to make address-space changes. In general, slab allocations are preferred as long as the size of memory needed is less than one physical page of 4 kilobytes. If allocation gets greater than 4K, the allocation of memory would get fragmented, groups of physically contiguous pages can get hard to find, and the speed would be sacrificed.

The other way of memory allocation is (2) vmalloc(), which returns memory that is virtually contiguous but may be physically scattered. This mechanism can only allocate entire pages, so it shouldn’t be used for small requests. In addition, excessive use of vmalloc() needs to be avoided since there’s the extra overhead of page-table changes. Then what needs to be done if I need a large allocation which must be physically contiguous? When it comes to a large memory allocation, the memory availability is more important than its allocation method. In this case, one can try the slab allocator, and then fall back to vmalloc().

3. Synchronization in Kernel

In order to synchronize different processes in the kernel, there are three main types of locks you can use. Note that these locks are used to protect the data structure not to protect code.

A. Spin Locks

When trying to acquire a spin lock while it is already held, the process/thread constantly checks if the lock is ready for availability while waiting (spin-waiting). As soon as the lock becomes available, the process can immediately acquire the lock and continue. The spinning prevents more than one process from entering the critical region. For that reason, spin locks should not be held for a long time. In the critical region of spin locks, you are not supposed to call (1) functions that touch user memory, such as copy_to_user(), copy_from_user(), (2) any semaphore functions or (3) any of the schedule functions, such as wait_event(), schedule().

The main thing is you cannot call potentially sleeping functions that can take awhile inside the critical region. For that reason, kmalloc() with the GFP_KERNEL flag, cannot be called inside the critical region, although kmalloc() with GFP_ATOMIC, which does not sleep, can be used. However, it’s strongly discouraged to use the latter because it can often fail.

Furthermore, spin locks can be used in interrupt handlers, whereas semaphores cannot be used. If a spin lock is used in an interrupt handler, you want to disable further interrupts to prevent any schedule functions or potentially sleeping functions from being called. Even if spin locks are used in places other than interrupt handlers, in general, you want to avoid interrupts. In that case, the following code can be used, since it conveniently disables interrupts and acquires the lock.

LKD

spin_lock_irqsave() saves the current state of interrupts, disables them, and then obtains the spin lock. spin_unlock_irqrestore() unlocks the spin lock and returns interrupts to their previous state.

If you enabled the interrupts, there’s no need to save the current state of interrupts before grabbing the lock. Then, the following code would do the job:

LKD

B. Sleeping Locks

Sleeping locks include semaphores and mutexes. These can cause a task to sleep. When trying to acquire a sleeping lock that is unavailable, the task is put onto a wait queue and goes to sleep. The processor is then free to execute other code. When the semaphore becomes available, one of the tasks on the wait queue wakes up so that it can acquire the semaphore.

C. Reader-Writer Locks

The Linux kernel also provides reader/writer variants of spinlocks and semaphores. Sometimes, it is safe for multiple threads to read data at the same time, as long as the data structure does not get modified. Therefore, the reader/writer locks allow multiple concurrent readers but only a single writer. If your data access follows this pattern, especially with a greater amount of reading than writing, the reader/writer locks are preferred.

LKD

The semaphore version of the reader-writer lock is

LKD

Resources

0. CS4118 http://www.cs.columbia.edu/~jae/