Explore Projects

Integrate automated 4-bit and 8-bit model weight quantization with fused FlashAttention-2 operators for efficient inference.

<p>Model zoo is a great compilation of deep learning and reinforcement learning algorithms. Currently state of the art baselines in terms of reinforcement learning and generative models are lacking in this package, which are present in tensorflow and pytorch. This project aims to add state of the art reinforcement learning algorithms like Proximal Policy Optimization and Trust Region Policy Optimization along with multi-modal translation and image captioning networks. These models are complex to implement and thus most users resort to standard tensorflow/pytorch implementations. Adding these to model zoo would attract a lot of researchers.</p>

<p>The aim of this project is to enrich Flux Model Zoo with unsupervised deep learning models, in particular variants of Generative Adversarial Networks. I propose to add the following models : 1) Spatial Transformer Networks 2) StarGAN for facial expression synthesis 3) VAE-GAN 4) Energy Based GAN 5) Gated Recurrent Convolutional Neural Network</p>

This project aims to improve INT8 inference performance for quantized YOLO26 models on ARM devices using OpenVINO. Currently, several transformation gaps prevent convolution layers from being executed in INT8, forcing fallback to FP16 kernels and increasing latency. Two key issues were identified. First, patterns such as Conv → Multiply → Add → Swish → FakeQuantize prevent convolution post-op fusion. During the optimization pipeline, the Snippets pass collapses this chain into a subgraph before the graph optimizer can fuse Swish and FakeQuantize into the convolution, preventing INT8 execution. Second, activation zero-points represented as Subtract operations are not correctly folded into quantization parameters, which disrupts precision alignment and contributes to FP16 fallback. This project proposes to address these issues by enabling fusion of Swish and FakeQuantize as sequential convolution post-operations and by refining transformation handling of activation zero-points. Additionally, the optimization pipeline will be adjusted to prevent premature subgraph formation for quantized patterns, allowing graph optimizations to correctly fuse operations. The goal is to maximize INT8 convolution coverage on ARM while preserving numerical correctness.

Optical Character Recognition (OCR) technology has revolutionised document digitization, converting printed and handwritten text into machine-readable formats. However, recognizing text from centuries-old works remains challenging due to the complexity of early non-standard print forms, handwritten texts, and manuscripts. Existing OCR tools like Adobe Acrobat and Google’s Pytesseract OCR often struggle with historical documents due to variations in font styles, image quality, degradation, and layouts. This project aims to develop a hybrid end-to-end Transformer model capable of accurately recognizing text from non-standard Spanish printed sources from the 16th and 17th centuries.

This project addresses the challenge of making advanced Transformer-based OCR technology for historical documents accessible to non-technical users on standard hardware. Despite significant advances in OCR accuracy for complex historical texts, current solutions remain inaccessible to many historians, archivists, and researchers due to technical barriers and hardware requirements. The solution involves developing an optimized, user-friendly OCR system specifically for historical documents through three main approaches: (1) implementing advanced model optimization techniques for efficient CPU inference, including kernel fusion, weight pruning, and knowledge distillation; (2) creating cross-platform deployment options for mobile devices and web browsers; and (3) developing an intuitive application interface with preprocessing controls and interactive correction capabilities. Deliverables will include: 1) A quantized OCR model achieving at least 4x speedup on CPU with <1% accuracy loss. 2) Browser-based inference capability: An enhanced Streamlit application with batch processing, export options and 1 click OCR. 3) A benchmark suite comparing performance across hardware configurations 4) Comprehensive documentation and tutorials for non-technical users This project will democratize access to cutting-edge OCR technology, enabling broader participation in historical document digitization and analysis without specialized technical expertise.

<p>The field of deep learning has had numerous impressive advancements over the last few years. Music generation has always been a difficult application of deep learning to make sufficient progress in. Recently, with the advent of attention-based Transformer networks, there have been substantive improvements in the automated generation of polyphonic music. With these advancements in mind, it calls for a need to enhance the support for Music generation and related research in the Julia language. I propose to implement the Music Transformer (Huang et al.) and provide an additional package NoteSequences.jl which would play a crucial role in speeding up the development of further music related projects in Julia.</p>

The project is divided into two parts. The goal for the first part is to train a lightweight Deep Learning model to generate a dataset of synthetic images. I propose a class-conditional Generative Adversarial Network to generate images for the 10 categories of the CIFAR-10 dataset, given the class label as input. The model is trained using a knowledge distillation framework, in an attempt to compress the StyleGAN2-ADA network. For the second part, the pre-trained model of the first part is used to generate a dataset of synthetic images for CIFAR-10. Subsequently, this dataset is used for model optimization using OpenVINO's Post-training Optimization Tool. We evaluate the performance of the 8-bit post-training quantization method on a range of Computer Vision models.

<p>Spatial interaction modeling involves the analysis of flows from an origin to a destination either over physical space (i.e., migration) or through abstract space (i.e., telecommunication). There is little to no software avaialble to carry out spatial interaction modeling and the analysis of flow data. This is especially true in the case of open source software and within the python ecosystem. Therefore, a comprehensive python package, which draws on existing PySAL infrastrucutre and extends it, would fill an important gap within the current set of avialable spatial analysis tools.</p> <p>PySAL currenty provides a rich set of tools for modeling spatial effects within a regression framework, which is typically applied to areal units. While it is possible to extend some of these models to the case of spatial interaction data, new spatial weight structures will be necessary to capture the unique spatial dependence that occurs between a data point that has both an origin and a destination. Furthermore, the existing spatial regression models are specifically designed for continuous data, whereas many spatial interaction phenomena are more properly modeled as counts (i.e., commuting).</p>

<p>While pre-trained language models like GPT-2 provided with <a href="https://github.com/chengchingwen/Transformers.jl" target="_blank">Transformers.jl</a> can generate coherent text, controlling and steering of text towards the desired attribute is not directly possible with these pre-trained models as it is (particularly without fine-tuning). So the goal of this project to implement <a href="https://github.com/uber-research/PPLM" target="_blank">Plug and Play Language Model</a> in Julia Language, which will facilitate the users working with Julia for NLP to generate Controllable text based on Transformers.jl package, which can be used in deployable Chatbots and further research in the field of Text Generation with Julia. Apart from the model, the final product of this project will also include an end-user interface to facilitate the integration of the PPLM model with few lines of code.</p> <p>Some of the common applications of the model will include Sentiment Control and detoxification of the generated text.</p>

Optical Character Recognition (OCR) technology has revolutionized document digitization. However, existing OCR tools like Adobe Acrobat and Google’s Pytesseract, they struggle with historical documents due to variations in font styles, image quality, degradation, and layouts. Text recognition from centuries-old works poses a significant challenge due to the complexity of early non-standard print forms, handwritten text and manuscripts. Over the past year, transformer-based models have been fine-tuned to improve transcription accuracy, particularly for degraded and complex historical texts. Training on a diverse dataset that combines expert transcriptions and synthetic data has enabled better generalization across various typographical styles. Building on this progress, we aim to expand the dataset, to help the model finetune to handle handwritten documents as well. Fine-tuning on larger datasets incorporating diverse typographical styles both printed and handwritten and also implementing Language Modeling & Contextual Understanding for post-processing, allowing for contextual corrections based on 17th-century grammar to further enhancing the OCR accuracy. Additionally, we plan to develop and deploy a web or mobile-based annotation tool for historians, researchers, and institutions to validate and refine OCR outputs

In particle physics, a cross section is a measure of the likelihood that particles will interact or scatter with one another when they collide. It is a fundamental quantity and its computation can become time intensive. This project aims to apply state-of-the-art machine learning techniques, particularly transformer-based models, for the generation of symbolic representations of the squared amplitudes in HEP. The proposed sequence-to-sequence system aims to improve upon the existing time-intensive processes of calculations of the squared amplitude . This project builds upon the previous work which includes vanilla models, and extends it further to advanced transformer models.

In particle physics, a cross section is a measure of the likelihood that particles will interact or scatter with one another when they collide. It is a fundamental quantity and its computation can become time intensive. This project aims to apply state-of-the-art machine learning techniques, particularly transformer-based models, for the generation of symbolic representations of the squared amplitudes in HEP. The proposed sequence-to-sequence system aims to improve upon the existing time-intensive processes of calculations of the squared amplitude . By harnessing the power of transformers, this project seeks to optimize the process of converting symbolic amplitudes to squared representations.

Symbolic regression (SR) aims to discover human-interpretable mathematical expressions from numerical data. While recent transformer models achieve high token-level accuracy on physics datasets, they struggle to recover exact, functionally correct equations. Conversely, generative techniques like Genetic Programming (GP) excel at numeric fitting but explore expression space blindly. This project solves this structural gap by combining modern transformer priors with generative search. I will build upon previous ML4SCI projects to implement a hybrid pipeline: an xVal-encoded seq2seq transformer generates structurally sound candidate expressions, which seed a GP population. The GP outputs are then ranked and used to iteratively fine-tune the transformer via Direct Preference Optimization (DPO).

<p>Brian 2 is the new version of Brian software. In 2010 the ModelFitting toolbox was developed for the original version (doi.org/10.3389/neuro.11.002.2010). The main aim of this project is to adapt the ModelFitting to the new version of the software. Access to the module will allow computational neuroscientists to fit the experimental data to the neuron models.</p> <p>The main goal of the module is to find the best mechanistic representation of the recorded spike trains. It has to be re-implemented for the needs of the new version of Brian and requires access to a new optimization library.</p>

Symbolic Regression refers to discovering a function that accurately fits a given dataset. Evolutionary/genetic algorithms have been dominating this field since the beginning and much recently deep learning is being used to solve the problem. However, there is a significant lack of collaboration between the GP and ML communities with reference to symbolic regression. I propose to develop a hybrid method which combines evolutionary algorithm with transformers to solve this contemporary problem. During the coding period I will also develop a library (compilation of various algorithms) which can be used as a ready-to-use tool for symbolic regression. And, I will also benchmark the methods on a high energy physics dataset.

This project addresses the challenge of accurately digitizing degraded 17th-century Spanish manuscripts, which suffer from irregular layouts, archaic grammar, and low scan quality. The solution integrates a hybrid OCR pipeline: a Mask R-CNN model segments text blocks into lines, a fine-tuned TrOCR transformer recognizes printed/handwritten text, and a T5 + LLM hybrid system corrects errors while preserving historical grammar. An open-source annotation tool enables historians to validate outputs and expand datasets. Deliverables include: (1) a production-ready OCR pipeline, (2) post-correction module with fallback strategies, (3) annotation tool for dataset refinement, and (4) comprehensive documentation. This tool will enable scalable, accurate digitization of cultural heritage texts.

<p>The OpenWorm project is building a simulation of the C. Elegans in an open science fashion. The model completion dashboard is a web-based visualization of the digital versions of biological entities that are currently captured within OpenWorm’s database API, PyOpenWorm. This interface is designed to display the results of the unifying modeling activity, and should be coordinated with the crowdsourcing platform for C. Elegans ion channels known as ChannelWorm. This interface allows a user to drill down into OpenWorms' model, and view the states of completion of modeled components at each level.</p>

This project aims to implement a framework that leverages Large Foundation Models (LFMs) for automated dataset annotation. It also aims to distill their knowledge into lightweight models suitable for deployment on edge devices. Initially, LFMs assign labels to training datasets, that are then refined through human-in-the-loop verification. During the training of lightweight models, human annotators improve the quality of the training data. Active learning is employed to identify informative samples to minimize human effort for label correction. Through this iterative process, the lightweight models gradually acquire task capability with minimal human effort, whose knowledge is distilled from LFMs.

Demucs v4 is a state-of-the-art open-source music source separation model developed by Antoine Défossez. While it provides exceptional quality in separating audio into stems, it is currently implemented in PyTorch and therefore cannot be used directly in C++ applications or run efficiently on hardware accelerators through ONNX Runtime. This project aims to bridge that gap. Exporting the model to ONNX will allow Mixxx to integrate stem separation directly into its platform, opening up use cases like live stem manipulation, AI-assisted mixing, and educational visualization tools. Moreover, embedding the model in a minimal C++ command-line application will serve as a reference for further integration into audio applications.

<p>Agent-based models (ABMs) are simulations in which autonomous entities (known as agents) react to their environment (which includes other agents) based on a predefined set of rules. The objective is to analyze the effect of agent behavior on the model as a whole. Agents.jl is a pure Julia library that assists with the creation of performant ABMs. It provides various tools and utilities to simulate complex agent-model behavior with little effort. My proposal is to add the following two features that are currently not present in Agents.jl:</p> <ol> <li>Serialization and deserialization of model state</li> <li>Extending pathfinding functionality<ul> <li>Pathfinding in continuous space</li> <li>Low-level control over the pathfinding API</li> </ul> </li> </ol>

<p>The <strong>Taylor Models</strong> are mathematical tools that enable the rigorous representation of functions by a polynomial expansion and a remainder that encloses all sources of errors. These kinds of models define it own arithmetic (and composition) of functions in a validated manner, this means that the represented function is guaranteed to be enclosed in some interval.</p> <p>Several methods can be benefited from these models, of particular interest are the interval optimization problems and the validated solution to ordinary differential equations.</p> <p>My project consists of improving the existent methods in <a href="https://github.com/JuliaIntervals/TaylorModels.jl" target="_blank">TaylorModels.jl</a>, a Julia package that implements these models. The improvements include:</p> <ul> <li>Algorithms to tightening the enclosure of the represented functions and its applications to interval optimization. </li> <li>A new approach to rigorously solving ordinary differential equations.</li> </ul>

<p><a href="https://github.com/JuliaText" target="_blank">JuliaText</a> is a JuliaLang organization that provides packages to work with text. The aim of this project is to enrich TextAnalysis.jl with Framework for Statistical Language models and Practical implementation of Pretrained ALBERT.</p> <p><a href="https://arxiv.org/abs/1909.11942" target="_blank">ALBERT</a> is a deep-learning natural language processing model that uses far fewer parameters than BERT (<a href="https://arxiv.org/abs/1810.04805" target="_blank">Devlin et al., 2019</a>) without sacrificing accuracy.</p>