I am looking into adding functionality to the Linux kernel – specifically, the ability for users to open perf events for the resource control subsystem (resctrl), which doesn’t have perf support yet. Here’s what I’ve learned about the perf subsystem internals that might help if you’re a user or want to build similar functionality.
The perf_event_open System Call
The perf_event_open system call is defined in kernel/events/core.c. You can check the man page for detailed documentation.
The caller can supply a process, a CPU, and a cgroup ID for the entities they want to measure. The cgroup and process ID share the same input parameter, with the flag PERF_FLAG_PID_CGROUP controlling whether the parameter refers to a cgroup or a pid. The implementation handles locating these resources – for a process, it finds the task struct; for a CPU, it verifies the CPU is online; and for a cgroup, it passes the file descriptor to the cgroup.
Group Leaders in Perf
Since there’s significant code dealing with group leaders, it’s worth understanding how they work. The user passes the parameter group_fd to perf_event_open. Groups allow scheduling of events onto the PMU (Performance Monitoring Unit) hardware as a group – all or none. This matters because many PMUs have limits on the number of events they can track simultaneously, so the kernel time-multiplexes these groups onto PMUs.
A perf event can optionally have a group leader specified, but a group leader cannot have another leader. This creates a limited hierarchy – not a tree structure, just leaders with events under them. The perf_event_open handles most of this for PMU implementers. It resolves the group leader if specified and ensures all events in the group belong to a single hardware PMU. The perf subsystem doesn’t allow events from multiple PMUs in the same group because it’s hard to schedule events across multiple PMUs atomically.
Software events can be added to a hardware PMU group – this is allowed. When you add the first hardware event to a group that previously only had software events, there’s logic to move all the events to the hardware PMU.
Key Data Structures
Two important data structures link the perf_event struct to the PMU performing measurements and to the entity being monitored:
- perf_event_context – associated with the measured entity (task or CPU)
- perf_event_pmu_context – associated with a perf_event_context and the PMU
Multiple perf events can point to the same perf_event_context and perf_event_pmu_context. The perf_event holds a reference count on both structs it refers to. You can find documentation for struct perf_event_pmu_context in include/linux/perf_event.h.
Event Allocation and PMU Lookup
The perf_event_alloc function (called from perf_event_open) allocates and initializes the struct perf_event. It calls perf_init_event, which returns the PMU for the event. This is where the lookup happens from the event type to the PMU associated with that event type.
The perf_init_event function includes functionality that tries to initialize the event on different PMUs using the event type. Since PMUs can override the type field, perf_init_event follows these type redirections to find the actual PMU associated with the event. This is also where the kernel resolves dynamic PMU IDs – PMUs dynamically registered with the kernel are assigned an ID when registered, as mentioned in the man page.
Registering PMUs
Kernel code can register PMUs with perf_pmu_register. PMU developers specify their PMU’s behavior using fields on struct pmu, defined in include/linux/perf_event.h. The code in perf_event_open ensures the user has proper permissions and that the PMU supports all requested features.
Key fields that control supported behavior include:
task_ctx_nr– When set toperf_invalid_context, the PMU doesn’t support task contextcapabilities– Encodes PMU capabilities. For example, whenevent->pmu->capabilities & PERF_PMU_CAP_NO_INTERRUPTis set, the PMU doesn’t support sampling mode (only counter mode where users read values)
PMU Function Pointers
Struct pmu contains several function pointers for different PMU functionality:
event_init– initializes the perf_event structaddanddel– add and delete events to/from the PMUstartandstop– control event countingread– read event values
There are also optional functions. One function not marked as optional but actually is: sched_task. This allows the PMU to request a callback on every context switch on a specific CPU. To enable this, the PMU needs to call perf_sched_cb_inc, which enqueues the PMU’s CPU context onto a per-CPU callback list. If you don’t need this functionality, you can skip implementing the scheduling callback.
Example: Intel Uncore PMU
The Intel uncore PMU registration provides a small PMU configuration example. It’s not minimal because it implements the optional pmu_enable and pmu_disable functions, and passes attributes.
The attr_groups field is also optional – it allows PMU developers to create attribute files and directories in /sys/bus/event_source/devices/[pmu_name]/ for users to read PMU information. However, this can be left as NULL, as shown in the Alpha architecture example.
Summary
Understanding these perf subsystem internals is crucial when adding new PMU support to the kernel. We covered the major inputs to perf_event_open, the data structures associated with perf events, and how users specify their PMU functionality. Hope this serves as useful background when interacting with the code!