Fetching the latest programs, projects, and workspace data.
Find open source projects actively accepting contributors. Search repositories, filter by program milestones, difficulty tags, or tech stack.
Use our Orbit AI Matcher to find out! Get instant matching scores based on your developer skills, preferred frameworks, and contribution experience.
Convert your selected open-source project into a winning GSoC, LFX, or Outreachy application using Proposal Studio.
<p>In many pairs it is useful to know in addition to the morphological tags of a word, syntactic function tags in order to make an adequate translation.</p> <p>The shallow syntactic function labeller is a tool which takes a string in Apertium stream format, parses it into a sequence of morphological tags and gives it to a classifier. The classifier is a seq2seq model trained on prepared datasets, which were made from parsed syntax-labelled corpora (for instance, UD-treebanks).</p> <p>The dataset for an encoder contains sequences of morphological tags, the dataset for a decoder contains sequences of labels, in both cases one sequence is a one sentence. The classifier analyzes the given sequence of morphological tags, gives a sequence of labels as an output and the labeller applies these labels to the original string.</p> <p>So, in the end of the work there will be:</p> <ol> <li>The labeller itself, which parses the string, restores a model for a needed language from a file, gives a sequence of tags to the model, gets a sequence of labels as an output and applies these labels to the original string</li> <li>Files with trained models, which are saved in a suitable format (it could be, for example, JSON)</li> </ol>
This project is an enhancement to the previous work that has been done. Its goal is to make annotation simple and easy while also providing a well-defined manager-nnotator-reviewer framework. The goal of this project is to use Transfer Learning (TL) approaches to make the annotation process easier for the user by offering label predictions. This way, it will be possible to accomplish more with less data and effort. - An analysis of pros and cons of existing work such as BAT, YAMNet and MusicNN. This task will help us understand the drawbacks of the existing implementations and try to avoid them while keeping all the already made advantages. - Enhance the already made database which will be used to store the corrections in order to refine the model. - Research and design of the transfer learning model and an explanation for the architecture. - Implementation of the aforementioned model. - Enhancements of the already made Graphical User Interface providing Mock-up screens. This task includes the design of every addition on every screen in our tool and it should be discussed with the mentors before the implementation. - Implementation of the changes on User Interface and system for the use of the AI-Assistance. - Implementation of the whole AI-Assisting model. - Perform Django lazy loading, testing and Dockerization - Documentation
<p>High Level Trigger 1(HLT1) is the first and critical stage in software reconstruction of collisions at the LHCb experiment in the Large Hadron Collider at CERN. Allen aims to do full software reconstruction on GPUs.</p> <p>However the reconstruction must also be able to run on the LHCb baseline x86 architecture. Since Allen's algorithm are designed to be efficient on SIMD architectures, a natural translation to support x86 architectures is possible. The SPMD programming model bears resemblance to the SIMT programming model of CUDA, and is a natural target for code translation. Such an automated conversion would be beneficial not only for Allen, but for any CUDA projects seeking cross-architecture support.</p> <p>Exploiting vectorization on modern CPU is a hard task . Manual vectorization requires lot of developer time and maintenance can be hard. ISPC raises abstraction of SIMD via the SPMD programming model. Since SIMT programming model on GPUs closely resemble SIMD, the code conversion can be automated using an intermediate translation engine. The project aims to convert from CUDA to ISPC using translation engine based on LLVM.</p>
Strong gravitational lensing is a powerful tool for studying dark matter, but machine learning models trained on simulated data often fail to generalize to real observations due to noise, domain shift, and lack of physical constraints. This project addresses this gap by developing a physics-informed deep learning framework that incorporates the gravitational lensing equation directly into the training process. The proposed approach augments a deep neural network with physically derived deflection field channels and introduces a physics-consistency loss that enforces agreement between learned and predicted physical parameters (e.g., Einstein radius). This enables the model to learn representations that are both data-driven and physically grounded, improving robustness, interpretability, and cross-domain generalization. The project will deliver a modular, open-source library for physics-informed lensing analysis, including baseline and physics-augmented models, reusable physics layers (e.g., SIS/NFW profiles), training and evaluation pipelines, and reproducible benchmarking tools. Additional deliverables include pretrained models, documentation with tutorials, and a systematic evaluation of performance on both simulated and real lensing datasets.
<p>The <em>Transaction Cost Analysis</em> (TCA) of an investment program is a critical framework to pursue its best execution, as costs minimization is a necessary condition to achieving it, given investors' objective in terms of return and risk tolerance. This is paramount especially for institutional investors. Financial industry practitioners individuate three major time-based segments: pre-trade, intraday and post-trade analyses. <em>Pre-trade analysis</em> relies on modeling and forecasting the set of variables that may influence a financial asset price dynamics, it ultimately seeks the best execution schedule. During <em>intraday analyses</em> such investment decisions are monitored and potentially adapts depending on real-time observed market conditions. In <em>post-trade analysis</em>, the occurred transaction cost is measured and hence orders execution performance is evaluated. The core aim of our project is to develop the <code>blotter</code> package by implementing two base sets of mathematical models in the algorithmic trading toolbox: (i) <em>Market Impact</em> models, mainly of interest in pre-trade and intraday settings; (ii) <em>Algorithmic TCA</em> post-trade benchmarking models and statistical testing methodologies for comparisons.</p>
Monumento App Summary: Monumento, an integrated social app for sharing landmarks, visiting places, and visualizing their 3D models right from a mobile device, aims to enhance user experiences through the following key features: Desktop and Web Apps: Develop a cross-platform desktop app using Flutter. Create a web version for users to access accounts, view landmarks, and engage with the community. Ensure responsive designs and consistent interactions across devices. Social and Community Features: Implement Firebase Authentication for secure user sign-up and login. Use Firestore for real-time storage of user-generated content (posts, images, etc.). Enable user interactions such as likes, comments, and profile management. Utilize Firebase Cloud Functions for real-time notifications. 3D Visualization: Integrate 3D models of landmarks. Convert models to GLTF format for efficient loading. Augmented Reality (AR) mode allows users to see 3D models superimposed on the real world. Interactive exploration with zoom and rotation. Deployment and Scalability: Package Monumento into Docker containers for consistency. Set up CI/CD pipelines for automated deployment. Choose deployment platforms (Heroku, Firebase Hosting, or GCP) and monitor performance post-launch.
The `loo` package is a widely used R package in the Stan ecosystem which exposes a clean API to cross-validate (CV) Bayesian models. As new CV methods and diagnostics are created and adopted by the field, the `loo` package must be updated to support them. Currently different predictive performance measures have different APIs and varying level of supported features. This project is focused on refactoring the API to unify the support for different predictive measures to make all useful features available for all predictive measures to make it easier for the modellers to use the most appropriate measures for their applications. Key technical components will include creating a flexible object which can report multiple metrics and scores. Additionally, we need to create functions to support model comparisons for all these scores and metrics. We will also allow for measures besides log score for LOO-CV to be used under the same, unified interface as existing options and create a consistent loo object for all measures. We will further unify the interface by allowing for non-log score measures for in-sample, test data, and K-fold-CV use cases. We will also be extending the model comparison functions to carry forward information on what measure is being compared, diagnostic data, and information on how to calculate the standard error (SE) of the differences of various measures.
<p>This project aims to build a ROS gazebo simulation framework to display robot navigation using OMPL. The framework include OMPL GUI, navigation node, obstacle plugin and robot model.</p> <p>OMPL GUI should manage the navigation’s start or stop and set all the parameters such as robot type, start and goal pose and different OMPL planner. The tool can publish relevant message or subscribe service to change parameter.</p> <p>Navigation receives configure from GUI node and obstacle information from gazebo. It calls the OMPL api and publish ROS path message. If the robot model is devoted to route, this node should update the new planning path based on new robot pose.</p> <p>Although it just needs a known scenario in claim, I think it is convenience that we can change obstacle position freely. The gazebo plugin for obstacle can publish self-information or subscribe navigation’s service to change obstacle information.</p> <p>When navigation node publish the Path message, a robot model and its path following function is needed to complete this motion. Conveniently, I will use robot models in the JdeRobot such as turtlebot and adrone.</p> <p>Integrate all the above and implement the integral ROS gazebo simulation framework.</p>
WasmEdge is a cross-platform and lightweight runtime for AI models. It can run a variety of GenAI models, such as LLM (https://llamaedge.com/docs/user-guide/llm/get-started-with-llamaedge/), whisper (voice to text)(https://llamaedge.com/docs/user-guide/speech-to-text/quick-start-whisper/) , and GPT-SoVITS (text to voice) (https://llamaedge.com/docs/user-guide/text-to-speech/gpt-sovits/) on your own computers. By combining those 3 models together, developers in the WasmEdge community has created “video translation” applications that can translate video and audio content into another language. One such application is VideoLangua.com In this mentorship, we would like to build a Japanese translator agent that are specifically tailored to CNCF technical content. Expected Outcome: 1. Use whisper to extract a time-stamped English transcript from a sample of CNCF videos. Develop whisper prompt that are suitable for CNCF technical content. 2. Evaluate and select LLMs that are good at English to Japanese translation. 3. Develop LLM prompts that are suitable for CNCF technical content. 4. Train Japanese TTS actor models for GPT-SoVITS using PyTorch. 5. Create dictionaries for how to pronounce CNCF technical words in Japanese. 6. Evaluate the synthesized Japanese voice.
We developed the first algorithm selector, Btor2-Select, for the word-level hardware model-checking problem described in the Btor2 language. Given a Btor2 instance, Btor2-Select selects and applies the expected best verifier(s) from a pool of hardware and software tools. These decisions are based on various machine learning (ML) models, trained upon historical performance data, mapping from instance features to algorithm selections. We proposed two embeddings for Btor2 instances: bag of keywords and bit-width aggregation. Two traditional algorithm-selection ML models, i.e., empirical hardness model and pairwise classifiers, were implemented in Btor2-Select. More importantly, we developed a novel algorithm selection and scheduling framework based on deep reinforcement learning (RL). This RL framework allows for adaptive algorithm selection throughout the solving process, leveraging dynamic information such as previous attempts and elapsed time. Upon evaluation, the adaptive algorithm selector outperformed the best non-portfolio solver (SBS) by 24.4%. Moreover, it closed 6.69% more SBS-VBS gaps than the best non-adaptive algorithm-selection method, a common performance indicator for algorithm selectors. All the codes, experiments, and data can be found at: https://gitlab.com/sosy-lab/software/btor2-select.
Scientific discovery fundamentally relies on integrating and reasoning over multimodal information—text, diagrams, plots, spectra, microscopy images, and experimental observations. While multimodal Language Models like Gemini show promise as potential AI research assistants, their capabilities in nuanced scientific reasoning remain largely unevaluated, particularly across diverse scientific domains. Existing benchmarks often focus on general knowledge, single modalities, or lack the depth needed to probe scientific reasoning. This project proposes the creation of SciResearchBench, a novel, open-source multimodal benchmark specifically designed to evaluate the scientific reasoning and discovery capabilities of multimodal across key research domains like Chemistry, Materials Science, Biology, and Physics. SciResearchBench will mirror real-world research workflows, encompassing tasks in multimodal data integration, experimental understanding, results interpretation, and hypothesis generation. A core focus will be extensive ablation studies to systematically analyze model sensitivities (prompting, cross-modal integration, grounding, reasoning complexity, context length), providing crucial insights into model failure modes and avenues for improvement. The project deliverables include the curated/generated dataset, robust evaluation code, comprehensive evaluation results for Gemini models, detailed analysis, and the full open-source release of all components to the research community.
Neural weather prediction models like neural-lam rely on graph neural networks, but the current graph construction pipeline is locked to rectangular grids — blocking support for irregular data sources like weather stations, satellite swaths, and icosahedral model grids (ICON, MPAS). This project replaces the hardcoded pipeline with a flexible, topology-agnostic architecture spanning two repositories (weather-model-graphs and neural-lam). The core work delivers three things: (1) a two-step mesh architecture that decouples node placement from connectivity, enabling any layout (triangular, prebuilt, density-adaptive) to combine with any connectivity mode; (2) a bridge layer that eliminates 600+ lines of duplicated code by making neural-lam call weather-model-graphs directly, validated through a shared schema contract; and (3) migration to PyTorch Geometric's HeteroData for type-safe, self-documenting graph representation. As stretch goals, the project introduces a graph quality metrics framework for comparing mesh topologies without training, density-adaptive meshing driven by data or prediction error, and spherical coordinate support to fix systematic distortion at high latitudes. I have already contributed the foundational PRs (#81, #91, #92, #123, #258) and authored the two strategic issues (#384, #385) that define this project's architecture.
<p><a href="https://github.com/KhronosGroup/SPIRV-Tools" target="_blank">SPIRV-Tools</a> is a collection of tools to work on <em>SPIR-V</em> modules. One of these tools is <em>spirv-fuzz</em>. It applies <em>semantics-preserving transformations</em> to a <em>SPIR-V</em> module, that we call the <em>reference</em> module, to obtain a modified <em>variant</em> module. Theoretically, a <em>semantics-preserving transformation</em> should not change the behaviour of the code. In this way, the <em>reference</em> and <em>variant</em> modules can be run, and the outputs compared. The output is usually an image. If the images differ, this probably indicates a shader compiler bug. This is a specialised form of <em>fuzzing</em> that uses a technique called <em>metamorphic testing</em>.</p> <p>This project involved using the metamorphic testing approach to extend <em>spirv-fuzz</em> by adding new transformations to cover instructions and patterns of <em>SPIR-V</em> that were not previously covered by existing <em>spirv-fuzz</em> transformations. Also, the project used tools such as <a href="https://github.com/ChrisLidbury/CLSmith" target="_blank">CLSmith</a> and <a href="https://github.com/google/clspv" target="_blank">clspv</a> to improve <em>spirv-fuzz</em>.</p>
I propose to integrate native text tokenization support into OpenCV’s DNN module so that Large Language Models (LLMs) can be run end-to-end directly in OpenCV. Currently, to feed text into models like GPT-2 or GPT-3, users must rely on external libraries (e.g. tiktoken) for tokenization. My project will create a built-in tokenizer utility—implemented in C++ with Python bindings—that can convert raw text into token IDs and decode model outputs back into text. The core deliverable is a new cv::dnn::Tokenizer class supporting Byte Pair Encoding (BPE), along with the ability to load typical vocabulary/merges files and handle special tokens. Once implemented, this will allow a seamless pipeline for text-based or multimodal models within OpenCV: raw text -> tokens -> DNN inference -> tokens -> decoded text. I will also write comprehensive documentation and test this functionality against known tokenizer outputs (e.g., Hugging Face), ensuring correctness and good performance. By adding native tokenization, OpenCV’s DNN module becomes more versatile for cutting-edge AI tasks involving both vision and language. This reduces external dependencies, makes it easier to deploy LLMs in pure C++ or Python environments, and helps unify image and text processing in a single library—further strengthening OpenCV as a comprehensive open-source toolkit.
This project evaluates large language models on frontend coding and UI generation tasks by extending and integrating ideas from open-source coding benchmarks such as Multi-SWE-bench and WebDev Arena. The evaluation framework assesses coding and UI generation capabilities across various dimensions, with particular emphasis on multimodal features for frontend development. It leverages the models’ ability to generate, render, and iteratively refine solutions using visual feedback from rendered UIs, and supports multiple frontend frameworks (React, Vue, Angular, Svelte, Next.js) alongside single-file HTML tasks. The solution involves building a modular pipeline with typed configuration, provider adapters (Ollama, vLLM, OpenRouter), and a model manager; implementing multimodal evaluation loops that incorporate screenshots and structured judging; supporting iterative evaluation (allowing models to refine solutions over multiple attempts); expanding the task taxonomy with increasingly complex frontend challenges (calculators, dashboards, interactive apps, framework-based projects); and automating the entire evaluation pipeline with reproducible runs and detailed artifact logging. Deliverables include the enhanced evaluation framework with new benchmark tasks, structured outputs and reports (JSON, screenshots, summaries), automation scripts for reproducible runs, and thorough documentation. The framework provides a transparent and extensible way to compare advanced AI coding capabilities in frontend development without relying on a leaderboard, focusing instead on rigorous, reproducible benchmarking.
Goal:- a)Creation of several realistic car models with Ackermann control some of them including a 3D LIDAR. b)Improved Realistic and Lightweight Environments for Autonomous Driving Exercises in RoboticsAcademy. c)Migration of Existing Autonomous Driving Exercises into ROS2 at RoboticsAcademy. Plan:- The detailed plan is included in the proposal, this is my plan of action before the official coding period starts. Before May 20: ● Study of the Car Sensor plugins and their implementation in ROS2. ● Explore the new Ignition Gazebo 11 simulator. ● Explore Sensors, Noise modules, command-line tools, and custom plugins in the Ignition Gazebo 11 simulator. ● To familiarize myself completely with RoboticsAcademy’s functionality and architecture. ● Study of Ackermann control and its implementation. ● Explore ROS2’s Architecture and Communication channels required for RoboticsAcademy Exercises. May 20 - June 12(Before the official coding time): ● To do self coding with Autonomous Driving Exercises to improve my further understanding. ● During this period I will remain in constant touch with my mentor and the JdeRobot community. I will remain active on Mailing lists to discuss and finalize the modifications that need to be on existing car models and the design of new models or environments. ● Thus with the help of my mentor, I will become absolutely clear about my future goals, the final exercise implementations that need to be done as well as the approach that I will follow to convert current exercises to support ROS2. Deliverables:- ●Several realistic car models with Ackermann control some of them including a 3D LIDAR. ●Improved Realistic and Lightweight Environments for Autonomous Driving Exercises in RoboticsAcademy. ●Migration of Existing Autonomous Driving Exercises into ROS2 at RoboticsAcademy.
In this project, I want to create Jupyter notebooks for a real-world machine learning use case, especially image processing using publicly accessible datasets. The goal is to create a reference guide that others can use to build ML pipelines for image processing or computer vision problems. I already have experience working with image processing to detect breast cancer cells in histopathological images and want to leverage that to build a similar pipeline using Apache Beam for image processing use cases. This can also be expanded to other computer vision problems like object detection, facial recognition, optical character recognition, and hand gesture recognition for disabled people. Following are some ideas on notebooks that I can build and contribute to : 1. Image Data Preprocessing: Apache Beam can be used to create a pipeline for different image pre-processing tasks like resizing, cropping, normalizing and filtering for different image file formats like TIFF, PNG and converting it into a more standardized format like JSON. 2. Model Inference: We can showcase how trained models can be used to make real time inferences in a Beam pipeline. 3. Model Evaluation: We can evaluate the model using Apache Beam, utilizing various metrics such as accuracy, precision, recall, and F1 score. 4. Stretch Goal: If time permits, we can do a similar process for video datasets and show how to preprocess videos using Beam. Deliverables Apache Beam notebook with working code to deploy a pipeline for image processing/video processing use case and supporting documentation.
Graph Neural Networks for End-to-End Particle Identification with the CMS Experiment: The goal of the project would be to develop, train, test, and benchmark GNNs (Graph Neural Networks) for low momentum tau identification. The GNN (Graph Neural Networks) would work with an E2E framework and work on raw collider data. Tau leptons play a crucial role in the search for new Physics, and in the search for new particles. Tau leptons decay into hadrons and lighter leptons, these analyses require efficient tagging of tau leptons for obtaining good signal to background ratios for discovery/exclusion. There has been recent work by CMS that incorporates DNNs (Deep Neural Networks) for performing tau identification that shows encouraging results. An E2E GNN working directly on the raw collider data may be able to provide better results. The goal may be tackled in the following steps: 1. Exploration and analysis of different strategies for graph construction: This would include the node construction and edge construction, which would further include pruning, preprocessing, and input representations. 2. Developing different graph architectures: This step would firstly include exploratory testing between different GNN architectures. Then taking ideas from the exploratory analysis novel architectures can be experimented with. 3. Testing and benchmarking GNN performance: Once a model architecture with hyperparameters is decided, a sweep may be performed to obtain the best performance of the model. Further testing and benchmarking will be performed in terms of the inference speeds, compute requirements, and performance on different cases (and possibly robustness). Deliverables: 1. Trained Model for the E2E low momentum tau identification 2. Analysis of the design choices 3. Analysis and Benchmark of the performance of the model
The Ziff–Gulari–Barshad (ZGB) model describes how CO and O₂ molecules adsorb and react on a metal surface represented as a square lattice. CO occupies a single site, while O₂ dissociates into two oxygen atoms placed on adjacent sites. When a CO and an O atom meet, they form CO₂, which leaves the surface and frees up space. As the CO partial pressure (yCO) changes, the surface transitions from oxygen-poisoned (no CO₂ production) to reactive (CO₂ formation) and eventually to CO-poisoned. Apothesis is a kinetic Monte Carlo simulator that models these surface reactions with high precision. Building on the ZGB model and the Apothesis simulator, PyApothesis provides a clean, Python-first workflow for running, analyzing, and sharing Kinetic Monte Carlo experiments. From a single script you can change simulation parameters (temperature, pressure, lattice size, seed), define custom adsorption/reaction/desorption processes, and run reproducible simulations — then immediately visualize results with publication-ready plots. PyApothesis includes time-series coverage plots, stacked-area composition views, process-rate and event-count analyses, spatial coverage maps and profiles, and real-time playback of the lattice (color-coded by species) to inspect microscopic behavior as it unfolds. It automatically produces a comprehensive analysis report (plots, logs, and statistical summaries) so users don’t have to dig through raw files, and it exposes hooks to incorporate lightweight ML models for surrogate predictions, parameter scans, or pattern detection directly from Python. Altogether, PyApothesis turns complex surface-kinetics simulations into an interactive, interpretable, and reproducible toolkit ideal for both researchers and students.
<p>Training of High end Deep Learning Models takes a Long time to Train, Hours, Days and Even Weeks. So, as a Developer/researchers it slows down the creativity and implementation cycle. The Whole Idea of Transfer Learning provides a nice solution where a model trained on a similar dataset can be trained partially or fully, on the custom dataset, yielding similar result. TFHub is a platform by Tensorflow where pretrained models are shared, so that developers and researchers can use transfer learning on these models. In this Project I'll be working on the following tasks:</p> <ul> <li>Implement Tensorflow Hub sample for Cloud TPU</li> <li>Convert ShuffleNet from ONNX to TF Hub Module</li> <li>Implement and Train ESRGAN(Enhanced Super Resolution Generative Adversarial Network) from Scratch (<a href="https://arxiv.org/pdf/1809.00219.pdf" target="_blank">https://arxiv.org/pdf/1809.00219.pdf</a>)</li> <li>Implement GAN Distillation to compress ESRGAN (<a href="https://arxiv.org/pdf/1902.00159.pdf" target="_blank">https://arxiv.org/pdf/1902.00159.pdf</a>)</li> <li>Add Support for SavedModel 2.0 Functions in saved_model_cli</li> <li>Add Sample Notebook demonstrating usage of ESRGAN TF Hub Module</li> <li>Create proof of concept Video player for real time video frame Super Resolution (using "distilled" ESRGAN)</li> </ul> <p>Github: <a href="https://github.com/captain-pool/GSOC" target="_blank">https://github.com/captain-pool/GSOC</a></p>
High-fidelity detector simulations are critical for accurate analysis in particle physics, but traditional Monte Carlo-based methods are computationally expensive. DeepFalcon introduced a promising approach using Graph Variational Autoencoders (GVAE) that treat particle hits as graph nodes, enabling faster inference while maintaining high accuracy. This implementation used CMS Open Data of boosted top quark jets and considered only the ECAL layer of the detector, achieving a 400× speedup in reconstructing jets compared to traditional simulations. The model architecture employed GraphSAGE layers and mincut pooling, while training used Adam optimizer with a hybrid loss function combining MSE and KL divergence. However, the current approach is limited to single-layer detector responses and doesn't fully capture the correlations between multiple detector components. This project aims to extend DeepFalcon’s GVAE architecture to support multi-layer detector systems, specifically incorporating Tracker, ECAL, and HCAL layers simultaneously. The proposal is structured into two phases: Phase I will focus on enhancing the graph data representation by including all three detector layers and designing optimized connectivity patterns that capture spatial and inter-layer dependencies. Phase II will explore advanced pooling techniques and alternative message-passing strategies to improve latent representation quality and model generalization, while maintaining fast inference speeds. All components will be implemented using Python and relevant machine learning libraries such as PyTorch Geometric. Expected deliverables include a multi-layer GVAE model, benchmark comparisons against existing results, and detailed analysis of the architectural choices. This project contributes toward advancing fast simulation techniques in high-energy physics by enabling more comprehensive and scalable detector modeling.
The End-to-End Deep Learning project within the CMS experiment at the Large Hadron Collider plays a critical role in identifying and reconstructing single particles, jets, and event topologies in collision events. This proposal aims to leverage state-of-the-art Self-Supervised Learning (SSL) techniques to develop robust models that outperform existing supervised baselines. By exploiting unlabelled data, these SSL approaches can allow models to learn rich representations enhancing the efficiency and accuracy of particle reconstruction and identification tasks at fine-tuning stage with labelled data. I-JEPA (Joint-Embedding Predictive Architecture) is a new approach, which aims to improve the semantic level of self-supervised representations without relying on extra prior knowledge encoded through image trans- formations. Given a context block, I-JEPA is trained predicting missing information in an abstract representation space. The targets are representation blocks obtained by dividing the input into non-overlapping patches and encoding them with a target-encoder. A subset of these representations is randomly sampled for loss computation. The context block, on the other hand, is derived from a single block randomly sampled from the input and processed through a context encoder. Overlapping regions between context and target blocks are removed to maintain a non-trivial prediction task. This approach represents a novel direction in self-supervised learning, addressing limitations of previous methods in producing high-level semantic representations suitable for various downstream tasks. I-JEPA has shown to be computationally efficient and outstanding results in image classification by linear probing its learnt representations. Deliverables: -Implemented SSL training pipeline. -SSL pre-trained model. -Fine tuned models on specific tasks. -Preprocessed datasets ready for training and evaluation. -Documentation covering implementation details, and experimental results
The aim of this project is to bring State of the Art methods for lightweight Visual Language models into Kornia in Rust. Recently, Hugging Face released a family of small visual language models (SmolVLM) which can be a game changer for the industry to build applications in embedded devices using such AI models.
Pytorch Lightning provides framework and functionalities for building and training PyTorch models. If integrated with deepchem, PyTorch-lightning will reduce the workload of implementing ML model functionalities for the deepchem library and will also enable useful functionalities like distributed training, easy model configuration and experimentation.