MediaPipe

Mentors: Student: Pulkit Midha

<p>In this project, I will be building a unity plugin that runs the MediaPipe Holistic model to do animation of a rigged 3D model as shown on this link. The Holistic model is indeed a great model by MediaPipe. Over a period, I have noticed a sudden rise in its demand in Unity XR projects since it provides the accuracy, FPS and features that no other competitor possesses. As of current technology, this can be still implemented with unity barracuda or TensorFlow or posenet. But none of these technologies is having support for cross-platform and mobile development having a decent FPS on an average device.</p>

Mentors: Student: Daniil Pakhomov

<p>In our work, we aim to research, implement and integrate a method that is capable of real-time manipulation of faces on mobile devices like changing the style of the hair, color of the lipstick or eyes, increasing the length of the hair, adding or removing facial hair, adding or removing smile and similar. We aim to be able to perform these kind of transformations while also preserving the identity of the input face and only changing local part of the input image of face while keeping other parts untouched.</p>

Mentors: Student: Ahmed Moorsy

<p>People with special-needs face a variety of different challenges and barriers that isolate them from their surroundings. Nowadays, several assistive technologies have been developed to reduce many of these barriers and simplify the communication between special-needs persons and the surrounding environment. However, few frameworks are presented to support them in the Arabic region either due to the lack of resources or the complexity of the Arabic language. The main goal of this work is to present a mobile-based framework that will help Arabic deaf people to communicate ‘on the go’ easily with virtually any one without the need of any specific devices or support from other people.</p>

Mentors: Student: Jiyeon Lee

<p>Build a offline TFLite GPU inference latency estimator with ensemble of regression models</p>

Mentors: Student: Pinku Deb Nath

<p>MediaPipe fills the gap of handling the implementation of models for different systems and allows the developers to focus more on experimentation than on system architectures. WebAssembly has become prominent in recent years as it can port C++ applications onto web browsers, making code more portable and utilizing the efficiencies of C++. In my GSoC internship, I will combine MediaPipe and WebAssembly by building an open-source realtime web video effects app. I will create a complete workflow of wrapping models in MediaPipe to be used in a webapp for running different ML-based video effects live in the browser using WebAssembly. The workflow will include wrapping the model binaries in calculators, create calculator graphs to use the model binaries, set up Bazel for WebAssembly compilation, write Emscripten bindings to access the models, and create a Typescript webapp to demonstrate how to access the WebAssembly binaries in realtime. I have extensive experience in C++ development, along with web development using Javascript. I have also been using MediaPipe for almost a year and am accustomed to the codebase. Hence, I am confident that I will be successful in my internship.</p>