This merge request introduces the foundational infrastructure for the Memory Manager by implementing the initialization phase and critical bootloader fixes. While general-purpose heap allocation and freeing routines are not yet fully implemented, this changeset establishes the necessary groundwork for future memory management features.
Key features:
Scans and processes memory descriptors provided by the bootloader.
Dynamically initializes the memory layer based on the detected amount of available RAM.
Introduces stack allocation and stack release routines.
Lays structural groundwork for future physical and virtual memory management.
Implements basic non-paged pool allocator
Fixes multiple issues in the bootloader related to memory handling.
Corrects PAE-related memory handling, including #23.
Future Enhancements for Memory Manager
The current implementation of the Non-Paged Pool allocator is functional but lacks several key features:
Page Allocator:
A lockless fast-path for single-page allocations: Currently, every allocation requires acquiring a global spinlock. Implementing an atomic SList for 4KB pages would drastically reduce lock contention and improve performance in multi-core environments, as the majority of pool requests are for a single page.
Out-of-Memory (OOM) Mitigation: The allocator currently returns STATUS_INSUFFICIENT_RESOURCES immediately upon failure. Future versions should implement a "trimming" mechanism to reclaim memory from the file system cache (Segment Cache) or free up unused expansion pools before failing the request.
User-Mode OOM Killer: In scenarios where memory trimming is either too slow, too costly, or fails to reclaim enough pages, the system must have a "last resort" policy to maintain stability. An OOM (Out-of-Memory) Killer should be implemented to scan the process list, identify non-critical user-mode processes with high memory consumption, and terminate them to release physical pages back.
Threshold Events and Balance Set Monitoring: We need to implement system-wide counters for allocated/free pool pages and pair them with Kernel Events. This allows the kernel to notify the Balance Set Manager or other background workers to begin proactive memory management once specific consumption thresholds are met.
Commitment and Resident Available Tracking: To prevent system exhaustion where the system consumes so much RAM that critical kernel tasks cannot run. Implementing commit tracking ensures that every allocation is backed by a guarantee and that the kernel always maintains a safety margin of resident pages for crucial operations.
Paged Pool Support: Currently, the system only supports Non-Paged Pool, meaning every kernel allocation permanently occupies physical RAM. Implementing a Paged Pool is essential for scalability, as it allows non-critical kernel structures to be swapped out under memory pressure. This requires integration with the Fault Handler to support demand-paging.
Pool Allocator:
Per-Processor Lookaside Lists: Currently, all allocations are synchronized via a global spinlock, which will become a major bottleneck on multi-core systems. Implementing per-processor lookaside lists will allow the majority of small allocations (especially 4KB pages) to be satisfied locally without lock contention.
Deferred Free and Allocation Retry Logic: In high-stress scenarios, memory might be logically free but temporarily held in "deferred" lists (e.g., for performance batching). Implementing a retry loop that triggers FreeDeferredPool significantly increases the success rate of critical allocations.
Must Succeed: While it is primarily intended for critical paging and I/O paths within the kernel executive, legacy and specialized third-party drivers can also request this pool type. To maintain a robust NT compatibility layer, ExectOS should adhere to this contract: if a "Must Succeed" request cannot be satisfied, the system should immediately trigger a kernel panic.
Memory Manager:
Paged Pool Support and Demand-Paging: Currently, XTOS only supports resident (Non-Paged) memory, meaning every kernel allocation permanently consumes physical RAM. Implementing Paged Pool requires a deep integration with the Page Fault handler.
Hyper Space support: It allows the Memory Manager to perform "ephemeral" mappings without the massive performance overhead of global TLB flushes. It provides a per-CPU window that is essential for high-performance internal tasks.
Session Pool support: It provides isolation for GDI objects, fonts, and display driver structures. By isolating these resources within a Session Pool, we prevent memory leaks in one user session from affecting others.
Huge Page Support: Implement support for large page allocations (PDE-level) for performance-critical components.
This merge request introduces the foundational infrastructure for the Memory Manager by implementing the initialization phase and critical bootloader fixes. While general-purpose heap allocation and freeing routines are not yet fully implemented, this changeset establishes the necessary groundwork for future memory management features.
Key features:
* Scans and processes memory descriptors provided by the bootloader.
* Dynamically initializes the memory layer based on the detected amount of available RAM.
* Introduces stack allocation and stack release routines.
* Lays structural groundwork for future physical and virtual memory management.
* Implements basic non-paged pool allocator
* Fixes multiple issues in the bootloader related to memory handling.
* Corrects PAE-related memory handling, including #23.
## Future Enhancements for Memory Manager
The current implementation of the Non-Paged Pool allocator is functional but lacks several key features:
### Page Allocator:
- **A lockless fast-path for single-page allocations**: Currently, every allocation requires acquiring a global spinlock. Implementing an atomic SList for 4KB pages would drastically reduce lock contention and improve performance in multi-core environments, as the majority of pool requests are for a single page.
- **Out-of-Memory (OOM) Mitigation**: The allocator currently returns STATUS_INSUFFICIENT_RESOURCES immediately upon failure. Future versions should implement a "trimming" mechanism to reclaim memory from the file system cache (Segment Cache) or free up unused expansion pools before failing the request.
- **User-Mode OOM Killer**: In scenarios where memory trimming is either too slow, too costly, or fails to reclaim enough pages, the system must have a "last resort" policy to maintain stability. An OOM (Out-of-Memory) Killer should be implemented to scan the process list, identify non-critical user-mode processes with high memory consumption, and terminate them to release physical pages back.
- **Threshold Events and Balance Set Monitoring**: We need to implement system-wide counters for allocated/free pool pages and pair them with Kernel Events. This allows the kernel to notify the Balance Set Manager or other background workers to begin proactive memory management once specific consumption thresholds are met.
- **Commitment and Resident Available Tracking**: To prevent system exhaustion where the system consumes so much RAM that critical kernel tasks cannot run. Implementing commit tracking ensures that every allocation is backed by a guarantee and that the kernel always maintains a safety margin of resident pages for crucial operations.
- **Paged Pool Support:** Currently, the system only supports Non-Paged Pool, meaning every kernel allocation permanently occupies physical RAM. Implementing a Paged Pool is essential for scalability, as it allows non-critical kernel structures to be swapped out under memory pressure. This requires integration with the Fault Handler to support demand-paging.
### Pool Allocator:
- **Per-Processor Lookaside Lists**: Currently, all allocations are synchronized via a global spinlock, which will become a major bottleneck on multi-core systems. Implementing per-processor lookaside lists will allow the majority of small allocations (especially 4KB pages) to be satisfied locally without lock contention.
- **Deferred Free and Allocation Retry Logic**: In high-stress scenarios, memory might be logically free but temporarily held in "deferred" lists (e.g., for performance batching). Implementing a retry loop that triggers FreeDeferredPool significantly increases the success rate of critical allocations.
- **Must Succeed**: While it is primarily intended for critical paging and I/O paths within the kernel executive, legacy and specialized third-party drivers can also request this pool type. To maintain a robust NT compatibility layer, ExectOS should adhere to this contract: if a "Must Succeed" request cannot be satisfied, the system should immediately trigger a kernel panic.
### Memory Manager:
- **Paged Pool Support and Demand-Paging**: Currently, XTOS only supports resident (Non-Paged) memory, meaning every kernel allocation permanently consumes physical RAM. Implementing Paged Pool requires a deep integration with the Page Fault handler.
- **Hyper Space support**: It allows the Memory Manager to perform "ephemeral" mappings without the massive performance overhead of global TLB flushes. It provides a per-CPU window that is essential for high-performance internal tasks.
- **Session Pool support**: It provides isolation for GDI objects, fonts, and display driver structures. By isolating these resources within a Session Pool, we prevent memory leaks in one user session from affecting others.
- **Huge Page Support**: Implement support for large page allocations (PDE-level) for performance-critical components.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
This merge request introduces the foundational infrastructure for the Memory Manager by implementing the initialization phase and critical bootloader fixes. While general-purpose heap allocation and freeing routines are not yet fully implemented, this changeset establishes the necessary groundwork for future memory management features.
Key features:
Future Enhancements for Memory Manager
The current implementation of the Non-Paged Pool allocator is functional but lacks several key features:
Page Allocator:
Pool Allocator:
Memory Manager:
e2eff2b8361e01c52c0c40d54743e0to13399434731339943473to40d54743e0Ready for merging
I have not finished reviewing this PR yet, but for sure I miss some exports.
Pull request closed