Society for Arts and Technology (SAT)

Mentors: Student: Lejie LIU

This project aims to enhance fulldome content rendering in ossia score by integrating advanced ray-tracing render pipelines. Traditional fulldome rendering requires multiple render passes, which leads to high computational overhead. By leveraging ray-tracing techniques on modern graphics APIs (e.g. DirectX, Vulkan), this project will enable single-pass fulldome rendering, significantly improving efficiency and render quality. The project will explore and conduct comparative analysis on various rendering pipelines and graphics APIs, including HDR, meshlet-based, volumetric, and SDF-based techniques, with a focus on their suitability and performance for fulldome rendering within ossia score. Final Deliverables: [1] Integration of the ray-tracing fulldome rendering pipeline into ossia score for better performance and quality. [2] Systematic comparison and analysis of different ray-tracing fulldome rendering pipelines. [3] Open-source release of the codebase and benchmarking results of Deliverable [2]. [4] Documentation of rendering pipeline design, challenges faced, and solutions implemented. Mentors: Jean-Michaël Celerier and Manuel Bolduc

Mentors: Student: Eddie Liu

This project comprises two parts: prototyping a real-time Python pipeline for object detection on point cloud data, and integrating it into LivePose. The goal is to build a functional pipeline that reads streaming point cloud data from various sources (e.g., Orbbec Femto Megas, Intel RealSense, stereo cameras), runs deep learning inference using models like PointNet++, applies filters for smoothing and tracking, and streams the results via OSC. During prototyping, the system will be evaluated across multiple models, input types, and hardware configurations to assess real-time performance and reliability. Once validated, the pipeline will be integrated into LivePose, leveraging its modular infrastructure for input handling, filtering, and output. By the end of the project, LivePose will support end-to-end processing of streaming 3D point cloud data in at least one working configuration, with clear documentation and performance benchmarks to guide future development and extension.

Mentors: Student: yashtiwari9182

This project addresses the limited accessibility of the powerful Puara-gestures library for creating high-level gestural descriptors within modern audiovisual workflows. The goal is to bridge this gap by porting Puara-gestures to the Avendish framework, enabling its integration with the visual programming environment ossia score. This will be achieved by wrapping each Puara-gestures class in Avendish to create standardized objects that can be compiled into native ossia score processes. These objects will allow artists and designers to seamlessly incorporate sophisticated gestural analysis into interactive multimedia environments using ossia score's visual interface and timeline-based orchestration. Key deliverables include Avendish wrappers for various Puara-gestures descriptors (motion, orientation, etc.), corresponding native ossia score objects with user-friendly controls, composite objects for complex gesture recognition, comprehensive documentation and examples, and performance optimizations. Optionally, the project will explore the creation of VST/LV2 plugins for wider compatibility with Digital Audio Workstations. This integration will democratize access to advanced gestural control, benefiting artists, performers, instrument designers, and the academic research community.

Mentors: Student: zepredos

Managing large libraries of media files efficiently is challenging, especially when dealing with diverse formats, extracting metadata, and performing fast searches. This project aims to develop a lightweight CLI-based Library Content Manager that automatically scans directories, extracts metadata using FFmpeg/TagLib, and organises media files in an SQLite database for efficient querying. To ensure speed, this tool will monitor directories in real-time (using inotify/ReadDirectoryChangesW), and to encourage personalisation, custom user-defined tagging, and fast search/filtering via structured queries will be implemented. Users can interact through a Command Line Interface or an IPC-based API, enabling integration with external tools. A dedicated plug-in will be developed to allow a seamless and intuitive integration with ossia score. Performance is crucial and thus multithreading, batch processing, and optimised indexing strategies will be implemented to ensure responsiveness even when handling large libraries. Existing solutions are either GUI-heavy or lack structured querying but this tool offers the best of both worlds. Therefore, this tool will provide a robust and extensible solution for organisation of media files, making it a great help for artists, developers and creative professionals.

Mentors: Student: Fanny

Live performances and concerts promote a sense of uniqueness. It happens for two main reasons: the relationship between the artist and the public; and the relationship between the audience itself. It is a space in time where artists respond to the public as much as the public responds to the artists. Also, every person in the audience is surrounded by other people having similar feelings and sensations. Searching to create related energy and experience via live performances in Extended Reality, this proposal aims to develop a platform prototype to generate a fluid XR experience for users by reliving live performances through motion capture. The key technologies for this goal are: MediaPipe Pose, open source software for pose detection in a live streaming context, and WebXR for the development and hosting of virtual reality and augmented reality experiences on the web. Live performances have no analog due to their unique energy and feeling of shared experience. XR's experience with concerts and live performances does not necessarily wants to exist as a copy of performances in real-life, but as a proposal of different interactions. Therefore Extended Reality can create its paradigm in art and technology.