Explore Projects

This project aims to build a multimodal foundation model for exoplanet characterization by addressing the challenge of heterogeneous astronomical data and instrument-specific biases. It combines transformer-based models for images (protoplanetary disks) and sequential data (transit light curves), trained using knowledge distillation, masked autoencoding, and contrastive learning on both real and simulated datasets. The deliverables include pretrained multimodal models, a curated data pipeline, evaluation on real-world tasks, and a well-documented open-source codebase with reproducible workflows.

Deep learning is now used to solve many different CV problems. A typical approach is to train a separate network for each specific task. However, there is a need for models that are closer to the human brain and can solve multiple tasks. Multi-task learning (MTL) is a branch of machine learning, in which multiple tasks learn simultaneously through a shared model. It has the following advantages: improving data efficiency, reducing overfitting through shared representation, and using auxiliary information to learn quickly. The project has four objectives: 1. One or more multi-task CV models trained (or borrowed, if the license is appropriate), and submitted to OpenCV model zoo. Specifically, at least MCN or networks with similar functions should be successfully transplanted or improved for OpenCV, and the accuracy and less computation should be provided, which is not lower than that of the existing single OpenCV model. 2. quantize the model on the condition that the accuracy does not decrease too much. At least try to quantize the model and make a trade-off between performance and effectiveness at the end. 3. Necessary patches, if any, for OpenCV DNN to support the provided multi-task models. Ideally, there should be an API introduced to optionally disable some parts of the multi-task CV model and save time by not computing unnecessary parts/heads of the model. 4. Using at least one example will demonstrate how to use this multi-task model.

<p>Connecting the encoder and decoder through an attention mechanism has made some of the best translation models. The encoder encodes input sentences into some fixed-length vector. The decoder then translates the encoded vector. Attention Mechanism allows the decoder to attend to different parts of the source sentence at each step of the output generation.</p> <p>The BERT is designed to pre-train deep bidirectional representations from unlabeled tests by jointly conditioning on both left and right context in all layers. The project will facilitate us in providing the Transfer Learning method in Natural language Processing tasks using the BERT model. After the pre-training, the BERT model can be finetuned with adding some output layer to create models that can be helpful in translation, next word prediction or next sentence prediction, question-answering, etc.</p>

This proposal sets out to implement several kinds of Quantum Vision Transformers (QViT) for High Energy Physics (HEP) analysis at the Large Hadron Collider (LHC), inspired by the seminal work of Cherrat et al., which introduces three innovative quantum transformer architectures: Orthogonal Patch-wise Neural Network, Quantum Orthogonal Transformer, and Quantum Compound Transformer. Despite the groundbreaking nature of their research, the absence of released source code has left a significant gap in practical application and reproducibility. Furthermore, attempts by last year's contributors, Marçal Comajoan Cara and Eyup B. Unlu, to implement the Quantum Orthogonal Transformer faced challenges; Marçal encountered training issues due to NaN values, and Eyup's implementation did not proceed to the training phase of transformers, highlighting the difficulties in realizing these models. This project aims to address these gaps by developing QViT models within the TensorFlow Quantum framework, leveraging this technology to bridge theoretical insights with practical application, thereby marking a potential first in applying these models for the analysis of particle jet images in HEP.

TensorFlow, Keras, and PyTorch are the most prominent libraries used for building DL and ML models. The use of PyTorch, as the primary backend for building algorithms, is growing due to its sustainability. Lately, DeepChem has decided to port some TensorFlow models to PyTorch. This project aims to successfully migrate one of the organization's 20 models. This model will perform a mix of invertible transformations between the base and target distribution. In order to optimize the results, a Normalizing Flow object initialized with the flow layers and base distribution would be iterated computing the loss for each epoch. For this reason, transformations and flows layers will be created with their respective unittests. Finally, the implementation will include the respective documentation and a tutorial, if applicable, following DeepChem's API.

The project aims to integrate a protein language model (PLM) into the DeepChem library through HuggingFace APIs and PyTorch. The project would involve wrapping the chosen PLM via HuggingFace APIs into DeepChem's workflow, allowing users to access the PLM via DeepChem. One of the main goals of the project is to establish the necessary skeleton structure in DeepChem to support future protein language models in DeepChem. The project would involve building on top of the existing DeepChem model class used to package HuggingFace LLMs by extending support to PLMs.

<p>If Musescore was being built today using web technologies like React etc., it would have definitely been built with a tree-like model for the score with each element in the score containing the smaller elements, like Score -&gt; Staff -&gt; Measure -&gt; Note -&gt; Accidental. In the case of Musescore, the code was written manually for QT, with all the drawing and file handling code written manually. So the classes in libmscore follow this tree-like model somewhat informally, but not explicitly. I will be refactoring the code, such that the classes in libmscore follow this model exactly by adding a "children" list to the ScoreElement class. Many operations can then be written as tree traversals. This would prevent a lot of duplication in code, and simplify the code in a lot of places.</p>

This project aims to develop quantum transformer architectures for high energy physics (HEP) analysis at the Large Hadron Collider (LHC). The focus will be on creating hybrid and fully quantum transformer and vision transformer models. Transformers have proven to be extremely powerful, revolutionizing the field of artificial intelligence and significantly impacting the world with their wide range of applications. By merging the remarkable capabilities of transformers with the emerging field of quantum machine learning, this project aims to make significant advancements in both areas. To ensure accessibility, an engaging visual guide will be provided, helping a wider audience understand the subject and fostering interest in quantum machine learning. The outcomes of the project are expected to benefit the open-source community and contribute to scientific advancements, inspiring further research and innovation in these exciting fields.

<p>I propose to add a wide variety of models in the model zoo of Flux. Currently, the model zoo lacks the latest architectures and algorithms in Reinforcement Learning. I will fill this gap by including:</p> <ul> <li>Dueling Deep Q Network</li> <li>Actor-Critic algorithm</li> <li>AlphaGo </li> </ul> <p>Apart from these, I will implement some interesting research papers which will cover below models:</p> <ul> <li>DCGAN</li> <li>Decoupling Neural Interfaces</li> <li>Spatial Transformer Networks</li> </ul>

DeepChem supports transformer-based foundation models for chemistry (ChemBERTa, MolFormer) but has no equivalent for genomic data. This project fills that gap by integrating DNABERT-2 into DeepChem, following the same design pattern used by ChemBERTa and MolFormer. The implementation consists of two components: (1) a DNABert model class that wraps HuggingFace's DNABERT-2 inside DeepChem's HuggingFaceModel/TorchModel hierarchy, supporting masked language modeling pretraining and fine-tuning for classification, regression, and multitask regression; and (2) a DNABertTokenizer that serves as both a HuggingFace tokenizer and a DeepChem featurizer, making it compatible with DeepChem's data loading pipeline. Deliverables include the fully integrated model and tokenizer classes, tests across all task modes, and benchmarks on a suitable genomic dataset.

We introduce Lensiformer, a state-of-the-art transformer architecture that incorporates the principles of relativistic physics for the classification of dark matter morphologies through gravitational lensing observations. This unique model combines the strengths of Transformer models in NLP with the analytical rigor of Physics-Informed Neural Networks (PINNs). Leveraging relativistic equations and Singular Isothermal Spherical models, Lensiformer accurately approximates the gravitational potential exerted by lensing galaxies and dark matter. These physics-based features are integrated into a Vision Transformer (ViT) neural network, providing a nuanced understanding of different dark matter structures like Cold Dark Matter and No-Substructure Dark Matter, and their associated gravitational imprints on light paths and galaxy distortions. To validate the model, we generate a specialized dataset using real galaxy images as source objects and benchmark Lensiformer against other leading models, confirming its superior performance in discerning different kinds of dark matter morphologies.

<p>Stitching pipeline is a well established code in OpenCV. It provides good results for creating panoramas from camera captured images. Main limitation of stitching pipeline is it expected camera model (perspective transformation). Although this model is fine for many applications working with camera captured images, there are applications which aren't covered by current stitching pipeline.</p> <p>Due to physical constraints it is possible for some applications to expect much simpler transform with less degrees of freedom. Those are situations when input data are not subject to perspective transform. The transformation can be much simpler, such as affine tranformation. Datasets considered here includes images captured by special hardware such as book scanners, maps from laser scanning, preprocessed images.</p>

<p>For this project, I would like to work with new models for random forests and gradient boosted trees. It would be interesting to then use this on a Kaggle competition and try to create materials to help students get started with these new models. Ideally, I would try to publish this on kaggle.com or tensorflow.org/tutorials in order to reach as large of an audience as possible.</p>

The goal is to design and implement the solution to the Airavata Data Catalog to record the metadata extracted from the Literature, Experimental and Computational data in support of Small Molecule Iconic Isolation Lattices (SMILES) Data. More precisely, this includes the data synchronization with the end-users. Currently, Airavata is extending an open-source project called “open-inventory” with additional features. Airavata uses three individual and unrelated databases to save the Literature data, Experimental data, and Computational data. Having these in usage, the chemical compounds are represented with ambiguous keys and missing data. Therefore, the scientific representation of the compounds with a specific structure; scientific data model (SDM), and related ontology (SDMO) is much better to analyze the behavior of a compound. In particular, a common data model should be used to synchronize the data efficiently.

My project focuses on developing diffusion models for gravitational lensing simulations. I will implement and evaluate various diffusion-based architectures, including Diffusion Transformers (DiT) and Latent Diffusion Models (LDM). Conditional modeling techniques such as Classifier-Free Guidance and Conditional Flow Matching will also be explored. The project involves creating a diverse dataset of simulated gravitational lensing images under various astrophysical conditions for downstream tasks. All implementations, findings, and documentation will be open-sourced through a GitHub repository.

<p>A compact Convolutional Neural Networks based model will be developed to detect known objects in an image in real time taking inspiration from SqueezeNet, SqueezeDet and using tiny-dnn deep learning library. Regression based model of YOLO (You Only Look Once) is incorporated to localize for objects in the image and generate bounding boxes. Fully connected layers at the end of the pipeline of YOLO are replaced with a single convolutional layer "ConvDet layer" for generating bounding boxes. Further Deep Compression will be implemented which includes methods like Network Pruning, Quantization and Huffman Coding to reduce the size of the model even further while still maintaining the baseline performance.</p>

Transformer-based models are gaining more and more traction in many fields, including physics. However, they are particularly known to require enormous computational resources. Moreover, as the data and the complexity of models grow, so does the computational resources required to operate and train. Supplying this increasing demand might be unsustainable in future. Large Hadron Collider (LHC) data is a good example of this phenomenon in physics with the large amount of data it produces especially in the high luminosity phase of the LHC. Quantum machine learning might be a viable solution to this problem by decreasing the time complexity of the computation significantly. I propose implementing a proof of concept Quantum Visual Transformer model to classify a synthetic dataset of jets produced via Pythia8.

The problem involves creating a seq2seq model for mapping histograms to empirical function sequences. I propose to tackle this in two steps. Step 1 involves creating random functions and normalizing them to get a set of function sequences, then generating histogram data and best-fit parameters for the function. This dataset is proposed to have a million examples. Step 2 involves experimenting with different seq2seq models like LSTMs, Transformers, and Pre-trained Transformers and training/evaluating their performance for our task. It also involves tuning hyperparameters and exploring the use of other transformer architectures like TabTransformer for this task. I also propose of stretch goal - checking the robustness of our model to distribution shift in the data.

Since Dark Matter was discovered, physicists have been trying to understand its composition. In practice, the best method to detect substructure is from strong gravitational lensing images. Given the huge success of deep learning algorithms in computer vision across various fields, it seems a promising path to use these algorithms on gravitational lensing images to classify the Dark Matter substructure. In recent years, attention-based networks like Vision Transformers have shown immense promise in Computer Vision tasks. Hence, the intention is to use them for classifying Dark Matter substructure from gravitational lensing images. DeepLense is a deep learning pipeline for particle dark matter searches with strong gravitational lensing. This project will focus on implementing Vision Transformers in the DeepLense pipeline, which is expected to immensely boost the current performance of the system. The overall deliverables of this project are: - A Python module implementing Vision Transformers (ViT). - Training/evaluation scripts to train this ViT model on gravitational lensing dataset. - Pre-trained, ready-to-use ViT models for other pipelines using lensing datasets.

<p>Dynamic Topic Models are used to model the evolution of topics in a corpus, over time. The Dynamic Topic Model is part of a class of probabilistic topic models, and unlike the previous models developed, takes into account time-series data. The idea behind this project proposal is to implement Dynamic Topic Models while taking care of Gensim's philosophy of being memory-independent and robust.</p>

One of the most important physical quantities in particle physics is the cross section, or a probability that a particular process takes place in the interaction of elementary particles. Its measure provides a testable link between theory and experiment. It is obtained theoretically mainly by calculating the squared amplitude. While the calculation of amplitudes for a particular expression is trivial, mapping it to a squared amplitude is computationally expensive. Previous works demonstrated that transformer-based (vanilla and KAN-infused) models can map amplitudes to squared amplitudes substantially faster and with high accuracy. However, transformers suffer from quadratic complexity due to self-attention, making them less efficient when handling extremely long sequences. Additionally, they may struggle with structured, hierarchical dependencies that naturally arise in amplitude computations. Inspired by Classical State Space theory, State Space Sequence Models (SSMs) emerged as a promising class of architectures for sequence modelling. These models can be interpreted as a combination of RNNs and CNNs. These models have near-linear scaling with sequence length, making them effective for long-range dependencies. This project will explore using SSMs for squared amplitude calculations. There are 2 major aims of this project 1. To develop novel hybrid models that combine state-space models (SSMs) with attention mechanisms for squared amplitude calculations. 2. A systematic evaluation of already existing models and SSMs on feynman interaction datasets and their scalability with respect to dataset size, maximum sequence length etc.

<p>The goal of this project is to create a user friendly API for an integrated workflow to perform typical text mining, natural language processing, and topic modelling tasks. This would include complete process of topic modelling:</p> <ul> <li>Loading data, including loading text files from a local filesystem, as well as harvesting texts from the internet (via the package stylo/tm)</li> <li>Data transformation, including calculating word frequencies (stylo/tm)</li> <li>Text stemming and tagging (koRpus/snowballC)</li> <li>Data subsampling (stylo)</li> <li>Topic modelling (mallet/LDA/topicmodelling)</li> <li>Visualizations (wordcloud/networkD3/ggplot2)</li> </ul> <p>In the first stage, I plan to integrate a few packages as mentioned above. Future development assumes construction of a package integrating more tools in the similar fashion as caret for predictive modelling. The Google Summer of Code is planned to be just outset of a bigger project.</p>

sktime provides a unified Python API for time series forecasting, but its support for modern foundation models, large pretrained models capable of zero-shot forecasting, is still maturing. Several high-value models, such as Google TimesFM and IBM TinyTimeMixer, are not yet interfaced in sktime; existing foundation model adapters lack probabilistic output (predict_interval, predict_quantiles), and there is no shared wrapper to reduce the boilerplate cost of adding new models from the Hugging Face Hub. This project directly addresses those gaps. The primary deliverable is one production-ready foundation model adapter fully integrated into sktime's BaseForecaster API, with fit, predict, predict_interval, and predict_quantiles support, CI-passing tests, soft-dependency handling, and a usage tutorial.