Integer underflow in AMD64 NonPaged pool calculation and suboptimal i686 pool scaling #28

New Issue

Open

opened 2026-06-12 16:41:51 +02:00 by harraiken · 0 comments

harraiken commented

2026-06-12 16:41:51 +02:00

Owner

1. Critical integer underflow on AMD64: In ComputeNonPagedPoolSize and ComputeMaximumNonPagedPoolSize, the size is calculated by unconditionally subtracting 4096 pages (16 MiB) from the total physical page count. However, the MM_MINIMUM_PHYSICAL_PAGES threshold for AMD64 is defined as 2048 (8 MiB). If the system attempts to boot with memory between 8 MiB and 16 MiB, the unsigned subtraction underflows to an extraordinarily large value. Consequently, the manager calculates a massive pool size, caps it at the maximum 128 GiB limit, and proceeds to blindly map the entire 128 GiB space. This immediately exhausts physical memory.
2. Suboptimal scaling discontinuity on i686: In ComputeNonPagedPoolSize, the code branches explicitly based on whether the system has less than or equal to 256 MiB of RAM. This introduces two problems:

A machine with exactly 256 MiB RAM is clamped to a 2 MiB initial non-paged pool. However, a machine with just one extra physical page (256 MiB + 4 KiB) falls into the else branch, allocating roughly ~8.1 MiB for the initial non-paged pool.
Capping the pool at 2 MiB for systems with 128 MiB or 256 MiB of RAM means utilizing less than 1% of the available physical memory, artificially starving the kernel of resources it could otherwise easily afford.

While reviewing the memory manager initialization functions (MM::Manager), I have identifies several issues in how the initial and maximum non-paged pool sizes are calculated across architectures. This includes a critical integer underflow bug on AMD64 that leads to an inevitable kernel panic on low-memory configurations, as well as a severe scaling discontinuity on i686. **1. Critical integer underflow on AMD64:** In ComputeNonPagedPoolSize and ComputeMaximumNonPagedPoolSize, the size is calculated by unconditionally subtracting 4096 pages (16 MiB) from the total physical page count. However, the MM_MINIMUM_PHYSICAL_PAGES threshold for AMD64 is defined as 2048 (8 MiB). If the system attempts to boot with memory between 8 MiB and 16 MiB, the unsigned subtraction underflows to an extraordinarily large value. Consequently, the manager calculates a massive pool size, caps it at the maximum 128 GiB limit, and proceeds to blindly map the entire 128 GiB space. This immediately exhausts physical memory. **2. Suboptimal scaling discontinuity on i686:** In ComputeNonPagedPoolSize, the code branches explicitly based on whether the system has less than or equal to 256 MiB of RAM. This introduces two problems: - A machine with exactly 256 MiB RAM is clamped to a 2 MiB initial non-paged pool. However, a machine with just one extra physical page (256 MiB + 4 KiB) falls into the else branch, allocating roughly ~8.1 MiB for the initial non-paged pool. - Capping the pool at 2 MiB for systems with 128 MiB or 256 MiB of RAM means utilizing less than 1% of the available physical memory, artificially starving the kernel of resources it could otherwise easily afford.

harraiken added the BUG ENHANCEMENT labels 2026-06-12 16:41:51 +02:00

harraiken added this to the ExectOS Development Board project 2026-06-12 16:41:51 +02:00

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: xt-sys/exectos#28