OpenVINO Toolkit

Mentors: Student: Mohamed Ashraf Ibrahim

Develop an autonomous multi-agent system using LangGraph to automatically analyze Hugging Face model architectures and generate the necessary code, tests, and documentation to add support for new models in the Optimum-Intel repository

Mentors: Student: Omkar Kabde

OTX supports 15+ detection models but has no built-in tracking.Every real-world video application (conveyor belt counting, asset monitoring, dwell-time measurement, traffic analysis, sports analytics, etc) needs stable object tracking across frames. This project integrates Multi-Object Tracking into OTX, adding five tracker implementations (ByteTrack, SORT, DeepSORT, OC-SORT, BoT-SORT), a SAM-2-inspired memory bank for re-identification, MOT17 benchmarking, and CLI support along with documentation. The design is decoupled: a new OTXTracker base class owns all temporal state while existing OTX detection models remain untouched.

Mentors: Student: Shehroz Kashif

A lot of everyday computer tasks—like navigating applications, managing files, and configuring settings—are repetitive and require multiple manual steps. Existing automation solutions rely on brittle scripts or cloud-based AI systems, which introduce latency, privacy concerns, and limited adaptability. This project aims to build a native desktop GUI agent that can understand natural language instructions and autonomously operate the computer interface. The system will combine Vision-Language Models (VLMs) for screen understanding and Large Language Models (LLMs) for reasoning and task planning, all running locally using OpenVINO for efficient, low-latency inference. The proposed solution uses a multi-agent architecture consisting of a Router, Planning, UI Grounding, Action Execution, and Reflection agents. The VLM (Phi-3.5 Vision) will interpret screenshots and map UI elements to coordinates, while the LLM (DeepSeek-R1 Distill Qwen 7B) will decompose tasks and generate action sequences. A PyQt-based desktop interface will allow users to interact with the agent, monitor execution, and configure settings.

Mentors: Student: Jayakaran

The existing GPU profiling workflow in OpenVino, through benchmark_app connects kernels to the model layers (primitives) and reports their execution times. However, this level of abstraction is too coarse to identify kernel instruction level insights. Intel GTPin, on the other hand, provides instruction level GPU kernel metrics like clock cycle counts, but is unable to directly map such metrics to specific inference or layers. This project aims to build a GTPin-centered profiling capability within OpenVINO to identify hotspot kernels and analyze their execution characteristics. The approach involves integrating controlled GTPin profiling into the inference flow, developing a custom GTPin tool for collecting kernel-level metrics, and progressively interpreting these metrics using signals from OpenVINO performance counters, and kernel metadata. The expected deliverable is a GTPin-based profiling workflow integrated into OpenVINO, capable of identifying bottleneck kernels, providing insight into their behavior, and offering indicative optimization directions, with validation using tools such as VTune.

Mentors: Student: asterich

This project aims to resolve the computational bottleneck in self-collision detection within the OCS2 framework for humanoid robotics. The current implementation in ocs2_self_collision operates on a pair-by-pair basis, iterating through collision pairs and calling PinocchioGeometryInterface::computeDistances() followed by Jacobian and penalty term calculations. This sequential approach is not SIMD-friendly by default and fails to utilize the parallel processing power of modern CPUs. To address this, I propose replacing the current computeDistances()-based distance computation with a high-performance implementation inspired by SIMD-amenable structures. A primary focus will be exploring / integrating SIMD-friendly data structures, such as Collision-Affording Point Trees (CAPT), to represent the storage and computation of collisions. This transition aims to transform the collision pipeline into a vectorized workflow that can be efficiently accelerated using AVX2 intrinsics. The primary deliverables include the clean integration of this accelerated engine into the ocs2_self_collision pipeline, accompanied by rigorous correctness checks against Pinocchio/FCL baselines. Furthermore, I will provide a comprehensive end-to-end latency evaluation using ocs2_mobile_manipulator to demonstrate the reduction in average and maximum compute cycle times. This work will significantly enhance the real-time performance of MPC-based humanoid control in complex environments.