Skip to content

[pciehp 1/4] Support 32 PCIe buses - #6226

Open
ilstam wants to merge 5 commits into
mainfrom
ilstam-hotplug-1
Open

ilstam wants to merge 5 commits into
mainfrom
ilstam-hotplug-1

Conversation

@ilstam

@ilstam ilstam commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Firecracker currently supports a single PCI bus. This PR is the groundwork for extending that to 32 buses.

@ilstam ilstam changed the title [pciehp 1/4] Support multi-bus PCI configuration and grant the OS PCIe hot-plug control [pciehp 1/4] Support 32 PCIe buses Sep 17, 2026
@ilstam
ilstam marked this pull request as ready for review September 17, 2026 17:15
@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.92308% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.07%. Comparing base (23b09b9) to head (13e76c0).
⚠️ Report is 32 commits behind head on main.

Files with missing lines Patch % Lines
src/vmm/src/devices/pci/pci_segment.rs 95.45% 1 Missing ⚠️
src/vmm/src/pci/bus.rs 96.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6226      +/-   ##
==========================================
+ Coverage   83.05%   83.07%   +0.01%     
==========================================
  Files         277      277              
  Lines       31481    31511      +30     
==========================================
+ Hits        26147    26178      +31     
+ Misses       5334     5333       -1     
Flag Coverage Δ
5.10-m5n.metal 83.30% <96.82%> (+0.01%) ⬆️
5.10-m6a.metal 82.68% <96.82%> (+0.01%) ⬆️
5.10-m6g.metal 80.15% <93.75%> (+0.01%) ⬆️
5.10-m6i.metal 83.31% <96.82%> (+0.02%) ⬆️
5.10-m7a.metal-48xl 82.67% <96.82%> (+0.02%) ⬆️
5.10-m7g.metal 80.15% <93.75%> (+0.01%) ⬆️
5.10-m7i.metal-24xl 83.28% <96.82%> (+0.02%) ⬆️
5.10-m7i.metal-48xl 83.29% <96.82%> (+0.02%) ⬆️
5.10-m8g.metal-24xl 80.15% <93.75%> (+0.01%) ⬆️
5.10-m8g.metal-48xl 80.15% <93.75%> (+0.01%) ⬆️
5.10-m8i.metal-48xl 83.28% <96.82%> (+0.01%) ⬆️
5.10-m8i.metal-96xl 83.28% <96.82%> (+0.01%) ⬆️
5.10-m9g.metal-48xl 80.15% <93.75%> (+<0.01%) ⬆️
6.1-m5n.metal 83.33% <96.82%> (+0.01%) ⬆️
6.1-m6a.metal 82.71% <96.82%> (+0.01%) ⬆️
6.1-m6g.metal 80.15% <93.75%> (+<0.01%) ⬆️
6.1-m6i.metal 83.32% <96.82%> (+0.01%) ⬆️
6.1-m7a.metal-48xl 82.70% <96.82%> (+0.02%) ⬆️
6.1-m7g.metal 80.15% <93.75%> (+0.01%) ⬆️
6.1-m7i.metal-24xl 83.34% <96.82%> (+0.02%) ⬆️
6.1-m7i.metal-48xl 83.34% <96.82%> (+0.01%) ⬆️
6.1-m8g.metal-24xl 80.15% <93.75%> (+0.01%) ⬆️
6.1-m8g.metal-48xl 80.15% <93.75%> (+<0.01%) ⬆️
6.1-m8i.metal-48xl 83.35% <96.82%> (+0.02%) ⬆️
6.1-m8i.metal-96xl 83.35% <96.82%> (+0.02%) ⬆️
6.1-m9g.metal-48xl 80.15% <93.75%> (+0.01%) ⬆️
6.18-m5n.metal 83.32% <96.82%> (+0.01%) ⬆️
6.18-m6a.metal 82.70% <96.82%> (+0.01%) ⬆️
6.18-m6g.metal 80.25% <93.75%> (+<0.01%) ⬆️
6.18-m6i.metal 83.32% <96.82%> (+0.02%) ⬆️
6.18-m7a.metal-48xl 82.70% <96.82%> (+0.02%) ⬆️
6.18-m7g.metal 80.25% <93.75%> (+0.01%) ⬆️
6.18-m7i.metal-24xl 83.34% <96.82%> (+0.01%) ⬆️
6.18-m7i.metal-48xl 83.35% <96.82%> (+0.02%) ⬆️
6.18-m8g.metal-24xl 80.25% <93.75%> (+<0.01%) ⬆️
6.18-m8g.metal-48xl 80.25% <93.75%> (+0.01%) ⬆️
6.18-m8i.metal-48xl 83.35% <96.82%> (+0.02%) ⬆️
6.18-m8i.metal-96xl 83.34% <96.82%> (+0.02%) ⬆️
6.18-m9g.metal-48xl 80.25% <93.75%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ilstam
ilstam added this pull request to stack #6230 September 17, 2026 17:16
@ilstam ilstam added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Sep 18, 2026

@Manciukic Manciukic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one nit.

Comment thread src/vmm/src/pci/bus.rs
impl PciBuses {
/// Create the root bus and one secondary bus per root port
pub fn new(secondary_buses: u8) -> Self {
let count = usize::from(secondary_buses) + 1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Nothing passes more than 0 here yet, but with u8 a caller could ask for up to 255 secondary buses. Anything above 31 has no ECAM coverage, and at 255 num_buses() would hit its expect. Worth catching it at construction time:

Suggested change
let count = usize::from(secondary_buses) + 1;
assert!(
secondary_buses < MAX_PCI_BUSES,
"PCI segment cannot have more than {} secondary buses",
MAX_PCI_BUSES - 1
);
let count = usize::from(secondary_buses) + 1;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Manciukic
Manciukic previously approved these changes Oct 2, 2026

@ShadowCurse ShadowCurse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm

Comment thread src/vmm/src/pci/bus.rs Outdated
ilstam added 5 commits October 6, 2026 12:02
Today Firecracker presents a single PCI bus to the guest, with no Type 1
devices other than the host bridge itself. Virtio devices appear as Root
Complex Integrated Endpoints attached directly to the host bridge, on
bus 0.

Native PCIe hot-plug needs root ports, and each root port starts a
secondary bus, so configuration cycles have to be able to reach buses
other than 0. Introduce PciBuses holding the buses a segment has; the
root bus plus one secondary bus per root port.

No visible change for the guest at this point, nothing creates root
ports and secondary buses yet.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Until now PCI_MMIO_CONFIG_SIZE_PER_SEGMENT was meant to cover every
device and function in a single bus because it was assumed that there's
a single PCI bus in the system. Since we are going to expand the
topology to support up to 32 buses this needs to be updated.

Rename PCI_MMIO_CONFIG_SIZE_PER_SEGMENT to PCI_MMIO_CONFIG_SIZE_PER_BUS
which is a more accurate name and move the constant to pci/bus.rs
instead of repeating its definition per arch. And add a new
PCI_MMIO_CONFIG_SIZE_PER_SEGMENT that represents the ECAM space for 32
buses.

The ECAM regions grows from 1 MiB to 32 MiB which is 's still well
inside the 256 MiB PCI_MMCONFIG_SIZE window which could be used to cover
every possible BDF (256 buses).

The test_mmio_bus() test used to access the entire 1MiB ECAM region, but
that's not possible after growing the ECAM region to 32 MiB since the
stack is not large enough to accommodate such a large local variable.
Adjust the test to instead check that the beginning and end of the ECAM
region can be accessed, whereas accessing just before or just after it
fails.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Two bus ranges are advertised to the guest via ACPI tables and both
describe bus 0 only. Subsequent patches will extend the bus topology so
secondary buses need to be advertised too.

Make the MCFG ECAM entry cover every bus a PCI segment can possibly
have, and add the full bus range of present buses in the host bridge's
_CRS.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
The FDT advertises a single PCI bus to the guest. Change this to
advertise the full range of buses present in the system.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Add an _OSC method to the host bridge. This can be used by the OS to
request it's granted control of certain capabilities (PCIe hotplug,
power management events, etc). Otherwise the platform keeps control of
them.

Grant whatever the guest asks for by returning the capabilities buffer
unchanged. Granting control of features Firecracker does not implement
is harmless. The guest only drives a feature if it also finds the
corresponding PCI capability.

Signed-off-by: Ilias Stamatis <ilstam@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: Awaiting review Indicates that a pull request is ready to be reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants