Xapian Search Engine Library

Mentors: Student: Parth Kapadia

<h3>Project: Text-Extraction Libraries</h3> <p>Currently, Omega has support for various file formats such as .htm, .html, .pdf, .csv etc. This project will focus on adding functionality for various other file formats to Omega.</p> <ul> <li>The project is based on adding support for extracting data from various file formats using some external filters or shared libraries. </li> <li>There has already been a <a href="https://trac.xapian.org/wiki/GSoC2019/TextExtraction" target="_blank">GSoC 2019 project</a>, which defined a way to safely use external libraries.</li> <li>This project will be built on top of this and proposes to add support for various other file formats.</li> </ul>

Mentors: Student: Dipanshu Garg

<p>Xapian currently supports Tf-Idf weighting scheme. It has some normalisations (described by SMART) already implemented. More normalisations can be added if we make some extra stats available to the weighting scheme. The new normalisations have shown to outperform others in certain cases.</p> <p>We will also compare the effectiveness of different normalisations. Xapian-evaluation will be used for this.</p>

Mentors: Student: Vaibhav Kansagara

<p>This project revolves around making the xapian-letor and xapian-evaluation module releasable. It includes writing extensive tests for various High-level API's and low-level pieces, the addition of new rankers and scorer metrics and adding binding support for letor module in various languages.</p>

Mentors: Student: Bruno Baruffaldi

<p>Currently Omega uses external filters to extract text from a wide range of file formats. However, it is possible to improve it replacing this filter with libraries and using multi-threading and resource constraints to isolate library bugs in a subprocess in order to avoid them from crashing omindex.</p>

Mentors: Student: Tanmay Sachan

<p>Currently the xapian tests run one after the other. Since most machines these days use more than 1 processor, the tests can be sped up by running them parallely. This project attempts to accomplish the same by implementing a test scheduler which can run tests in parallel. It also attempts some other improvements on the test suite such as the implementation of a test timer to track slow test cases, and speeding up certain slow test cases by using generated databases for them instead of having them recreate one every time.</p>