SONiC OS Profiling

About the Organization

<p>*APPLICATIONS ONLY OPEN TO US BASED COLLEGE STUDENTS*</p><p><br></p><p>1. Problem Statement</p><p><br></p><p>Today, SONiC KVM-based PR testing validates functional correctness (test pass/fail) but does not track resource consumption or performance characteristics over time. This creates several blind spots:</p><p><br></p><ul><li>Performance regressions (CPU spikes, memory leaks, excessive I/O) introduced by PRs go undetected until they reach production or nightly pipelines.</li><li>There is no historical baseline of resource utilization per feature branch, making it impossible to distinguish normal variance from genuine degradation.</li><li>Nightly test environments lack systematic profiling, so gradual performance erosion across releases is invisible until a tipping point is reached.</li><li>Root-cause analysis of production incidents is hampered by the absence of correlated profiling data from the CI/CD stage where the regression was introduced.</li></ul><p><br></p><p>Without a structured profiling database and automated anomaly detection, the team operates reactively—discovering performance issues in production rather than catching them at the PR gate.</p><p><br></p><p>2. Proposed Solution</p><p><br></p><p>Build an end-to-end performance profiling pipeline that runs alongside SONiC KVM-based test execution, stores results in a centralized database, and automatically flags regressions at the PR gate.</p><p><br></p><p>2.1 High-Level Architecture</p><p><br></p><p>The solution is composed of four core components that integrate into the existing CI/CD and nightly test workflows:</p><p><br></p><ul><li>Profiler Container – A lightweight Docker container packaging the community profiler (e.g., py-spy, perf, cProfile) that runs in sidecar mode alongside the SONiC KVM during test execution.</li><li>Profiling Database – A remote time-series/relational database (e.g., PostgreSQL + TimescaleDB or InfluxDB) that stores per-run profiling snapshots keyed by feature branch, commit SHA, test case, and timestamp.</li><li>Analysis Engine – A statistical analysis module that performs comparative, clustering, and outlier detection on profiling data to identify spikes against historical baselines.</li><li>PR Gate Integration – A CI pipeline step that queries the analysis engine after test completion and fails the PR if any metric exceeds the configured threshold.</li></ul><p><br></p><p>3. Mentorship Objectives</p><p><br></p><ul><li>Help the mentee learn about SONiC OS architecture, container ecosystem, and KVM-based testing infrastructure.</li><li>Help the mentee gain hands-on experience with performance profiling tools (py-spy, perf, pidstat) and statistical analysis techniques.</li><li>Help the mentee understand CI/CD pipeline integration and automated quality gates.</li><li>Work with virtual SONiC testbeds (vsonic) to develop and validate the profiling pipeline without physical hardware.</li><li><strong>This is a summer term program, Mentees are expected to work 40~ hours a week</strong></li></ul>