Intel Video and Audio for Linux

Mentors: Student: Dec1s1on

In this project we want to generate disparity map and depth information from multiple 360-degree Images that can be helpful 3D scenes reconstruction.  I will focus on deep learning based depth map inference. I will model and train current state-of-the-art models with Tensorflow/Pytorch and compare their performance. I will convert the best performing model to ONNX representation which the DNN module (OpenVINO) can infer from. Thus integrating the model into libXcam for 360-degree depth map inference.

Mentors: Student: Jianyu Liu

Problem: FFmpeg DNN(Deep Neural Network) module has enabled OpenVINO backend inference on Intel GPU. While the FFmpeg decode and encode work on cpu currently memory copy between CPU and GPU, which lead to notable latency. It results in performance degradation when we do inference with some certain models. Therefore,I wan't to support the FFmpeg DNN inference GPU full pipeline, which means the FFmpeg DNN can do all the decode, encode and inference only on Intel GPU, without any memory copy. Method: Integrate ffmpeg dnn module and OpenVINO GPU inference with VaSurface GPU memory type Deliverables: A full gpu codec and inference pipeline on ffmpeg dnn module and a well writen document with user guide expamples

Mentors: Student: Shubhanshu Saxena

<p>This project focuses on implementing an asynchronous mechanism for model inference and batch execution in the TensorFlow backend of the FFmpeg Deep Neural Network module to boost model inference performance.</p> <p>The Tensorflow backend uses the TensorFlow C API, which currently does not provide functions for asynchronous execution. The support for async behavior can be provided using multithreading on the existing TensorFlow library functions. We will implement this behavior through detached threads that work independently of each other.</p> <p>Several inference frames will be combined to a single input tensor and executed together in a single batch to enable the batch mode. The DNN module authors saw a performance gain in the OpenVino backend with asynchronous batch inference against synchronous inference. A similar performance gain is expected from this project.</p>

Mentors: Student: Hongcheng Zhong

<p><strong>Versatile Video Coding (VVC)</strong> is the latest video compression standard after HEVC. <strong>GStreamer need to support VVC, and we should start from a VVC parser</strong>. Codec parsers are a core part of the Gstreamer hardware decoder. They are used in <code>gstvappi</code> and <code>gstcodecs</code> to split frames, get slice level information. VVC specification is finalized on 6 July 2021. It's a good time to start work on the VVC codec parser for GStreamer.</p>

Mentors: Student: Eugene Lyapustin

<p>360° video is getting popular during the past years and it is predicted to evolve [1]. Compared to virtual and augmented reality, 360° video is much more accessible for viewers. The cameras are getting cheaper, and many video sharing services have an option for 360° video support (YouTube, Vimeo, Facebook) as well as applications (e.g. VLC).</p> <p>360 video requires special transformations to be viewed: the most popular way to store the data at ERP and CMP projections.</p> <p>The goal of the project is to implement and optimize ERP and CMP conversion algorithms using GLES. The algorithm will perform 360° video transformations between ERP and CMP. [1] <a href="https://images.app.goo.gl/iVyk3qkdxmR7CNGA8" target="_blank">https://images.app.goo.gl/iVyk3qkdxmR7CNGA8</a></p>