Software Heritage

Mentors: Student: Satvik Vemuganti

Software Heritage is an ambitious project that aims to preserve and share the source code of publicly available software. In preserving source code, it becomes important to maintain the metadata of the project in a standard format. This is supported, in Software Heritage, for only 4 package managers as of yet. In this project, I plan on extending the support to 6 more package managers, which together manage over 1.3 million packages. A more detailed description of my project may be found in the proposal under the implementation plan section.

Mentors: Student: Kumar Shivendu

<p>Software Heritage is on a mission to collect, preserve, and share all the publicly available software with its source code and development history. The archive periodically crawls GitHub, GitLab, Debian, PyPI, etc. It has preserved more than 11 billion unique source code files with 2.3 billion commits covering more than 165 million software projects.</p> <p>The archive has a search feature to find repositories based on the repository URL or the metadata. This metadata includes the package name, description, license, etc.</p> <p><strong>TLDR:</strong></p> <p>I made this search more expressive with the help of advanced search features like filters, sorting options, and search query language (DSL) with autocompletion (optional)</p> <p><strong>Overview:</strong></p> <ul> <li>Introduced new fields in the search service (based on Elasticsearch), ingested data from other swh services through their RPC APIs or the journal service (Kafka), and built filters/sorting features.</li> <li>Designed a grammar, built a parser and a translator (with TreeSitter) that traverses the AST to translate the custom query language (DSL) queries into Elasticsearch queries.</li> <li>Implemented autocomplete features for the query language in the Web UI (using a wasm version of the same parser)</li> </ul>

Mentors: Student: danseraf

<p>Software Heritage has the biggest open archive of the source code publicly available, it captures software projects from various forges and all of them are stored inside a (giant) Merkle DAG. Currently Software Heritage has an experimental tool to check, from a given code base, which part of the source code is already stored inside the archive. The main idea for this proposal is to enhance the Software Heritage Code Scanner to make it usable in real production use cases and lead the scanner to be on par with the actual state of art industrial scanner tools.</p>

Mentors: Student: Archit Agrawal

<h3>Increase archive coverage</h3> <p>As Software Heritage works on archiving and sharing source code, one of the major tasks is to ingest the latest source code available from all the possible sources(forges and package managers). The goal of this project is to increase the archive coverage by making listers and loaders for different forges and package managers. Listers are components that crawl the APIs of software forges (e.g., Bitbucket, Gitorious, Sourceforge, ...) and return a list of the software available in it whereas Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model.</p>

Mentors: Student: Thibault Allançon

<p><a href="https://www.softwareheritage.org/" target="_blank">Software Heritage</a> is an ambitious research project whose goal is to collect, preserve in the very long term, and share the whole publicly accessible Free/Open Source Software (FOSS) in source code form.</p> <p>The Software Heritage <a href="https://docs.softwareheritage.org/devel/swh-model/data-model.html" target="_blank">data model</a> is a big <a href="https://en.wikipedia.org/wiki/Merkle_tree" target="_blank">Merkle</a> DAG made of nodes like revisions, releases, directories, etc. It is a very big graph, with ~10 B nodes and ~160 B edges, which makes it hard to fit in memory using naive approaches. Graph compression techniques have been successfully used to compress the Web graph (which is slightly larger than the Software Heritage one) and make it fit in memory. The goal of this GSoC is review existing graph compression techniques and apply the most appropriate one to the Software Heritage case, enabling in-memory processing of its Merkle DAG.</p>