PRHLT PrIx search engine. Guidelines for the IBEM Mathematical Expression Search in Images

  1. Overview
  2. Examples of queries
  3. Confidence level, hierarchy, etc.
  4. Viewing search results
  5. Starting a new search
  6. Advanced searching

Overview

This interface allows users to search a collection of more than 11,000 pages from 1,098 scientific documents containing mathematical expressions. The collection has been automatically processed from document images, without requiring access to the original source files or manual transcription of the mathematical content.

The resulting search engine enables efficient search of mathematical expressions and sub-expressions directly from document images. Users can perform exact or partial-expression searches and inspect the corresponding occurrences within the original document pages.

The PRHLT research center has processed these documents using advanced mathematical expression recognition and probabilistic indexing technologies (PrIx). Rather than relying on a single transcription for each expression, PrIx preserves recognition uncertainty and indexes multiple competing hypotheses, improving the recoverability of mathematical content affected by recognition ambiguities.

One of the main scientific objectives of this work was to investigate how recognition uncertainty can be preserved and exploited for indexing and search of mathematical expressions in document image collections. To this end, the project combines mathematical expression recognition, uncertainty-aware indexing, structure-aware segmentation of expressions and sub-expressions, and efficient matching techniques that enable search directly over document images without requiring manually verified transcriptions.

This demonstrator is the result of research conducted at the PRHLT research center of the Universitat Politècnica de València on mathematical expression recognition, probabilistic indexing, and search in document image collections. The technologies presented here build upon research developed within the IBEM, DeepPattern, SimancasSearch, and EUDEEP projects.

This work was partially supported by Generalitat Valenciana under predoctoral grant CIACIF/2021/313, by grant PID2020-116813RB-I00 funded by MCIN/AEI/10.13039/501100011033, and by grant PID2024-161104OB-C21 funded by MICIU/AEI/10.13039/501100011033 and by the European Regional Development Fund (ERDF/FEDER), European Union.

Happy searching!

Examples of simple and advanced queries are given first, followed by a detailed explanation of all the search modes and operators used in the examples.

Searching and Viewing Search Results

Mathematical queries can be entered directly using standard LaTeX notation or generated automatically from uploaded or cropped images of mathematical expressions. When an image-based input method is used, the recognized LaTeX expression is automatically inserted into the search box, where it can be inspected and edited before searching.

Before executing the search, the interface displays both a rendered visualization of the query and, when applicable, its normalized representation. These previews allow the query to be verified before searching.

After selecting the desired search options, pressing the Search button executes the query within the current browsing context. Searches are contextual: when initiated from the home page, the query is performed over the entire collection; when a document is open, the search is restricted to that document; and when viewing an individual page, only that page is searched. Returning to a higher level of the hierarchy broadens the search scope accordingly.

Collection-Level Results

Collection-wide searches return the list of matching collections. Each collection entry displays a representative thumbnail together with the number of documents containing one or more matches. Selecting a collection opens the corresponding document-level results.

Document-Level Results

Document-level results list all documents within the selected collection that contain matches. Each document entry displays a thumbnail of the document together with the number of pages with associated matching expressions. Selecting a document opens the corresponding page-level results.

Page-Level Results

The page-level view displays page images together with the number of matching expressions found on that page. Selecting a page allows inspection of the matching expression within the original document image.

Page Context

The page context view displays the original document page with the selected mathematical expression highlighted. The colour of the box surrounding the matched expressions indicates the confidence level, with green being the highest and red the lowest. Navigation controls allow users to browse between multiple matches on the same page while preserving the surrounding textual and mathematical context.

Starting a New Search

A new search can be started at any time by entering a new query or modifying the current one. Searches are always performed within the current browsing context. Consequently, if a document or an individual page is currently open, the search is restricted to that document or page, respectively.

To perform a collection-wide search, return to the home page before executing the query. Clicking Home in the navigation path preserves the current query while broadening the search scope to the collection level. Clicking IBEM Mathematical Expression Search in Images in the upper-left corner also returns to the home page, but additionally clears the current query and restores the default search interface.

Searches can likewise be focused on a particular document by first opening that document from the collection-level results. Subsequent queries will then be evaluated only within the selected document until the search scope is changed again.

Search Controls

The interface provides several controls that modify how queries are processed and matched.

The Normalize option converts the query into the canonical representation used by the search engine before matching. This option is recommended for most searches, as it allows users to write familiar LaTeX expressions without following the internal indexing format.

The Segment option automatically decomposes the normalized query into structurally meaningful sub-expressions. Mathematical expressions are interpreted as hierarchical structures, where delimiters (such as {, }, {@, and }@) define nested branches, while relational, binary, and punctuation operators provide additional segmentation points within these nesting scopes. The resulting sub-expressions are searched independently and combined during matching. When enabled, Min.Tokens specifies the minimum number of tokens required for a generated segment, while Segm.Coverage defines the minimum percentage of query segments that must be matched. Lower coverage values allow more partial matches, whereas higher values require a larger fraction of the query structure to be present in the matched expression.

The Confidence control is provided as both a numeric input and a slider. It specifies the minimum confidence score required for a match to be returned. Higher confidence thresholds produce fewer but generally more reliable matches, whereas lower thresholds allow additional lower-scoring recognition hypotheses to be considered, increasing recall at the expense of potentially introducing less reliable matches.

The Max.Results control specifies the maximum number of matches displayed for a query.

Expression Search

The following examples illustrate searches for mathematical expressions using standard LaTeX notation. Depending on the query and search settings, matches may correspond to complete expressions or to structurally meaningful parts of larger expressions.

Segmentation-Based Search

These examples demonstrate searches that use the Segment option to automatically decompose the query into structurally meaningful sub-expressions. When enabled, lower coverage values increase recall by accepting partial matches, whereas higher values require a closer correspondence to the original query.

Boolean Queries: OR

The OR operator returns pages matching any of the specified query expressions. When Segment and Segm.Coverage are used with OR queries, coverage is computed across the combined decomposed segments but requires those matched segments to occur within the same expression. This can be useful, but the interpretation may be ambiguous because matching segments may come from different OR branches.

Boolean Queries: AND

Match pages containing all specified query components. When Segment and Segm.Coverage are used with AND queries, coverage is computed across the combined decomposed segments but requires those matched segments to occur within the same expression. This can be useful, but the interpretation may be ambiguous because matching segments may come from different AND branches.

Boolean Queries: NOT

The NOT operator returns pages matching the expression on the left, while reducing or excluding pages that also match the expression on the right. This exclusion is confidence-based: a weak match on the right may only lower the score and may not remove the page. The right-hand expression must match strongly enough, relative to the selected Confidence threshold, to act as an effective veto. Segm.Coverage is currently disabled for NOT queries because coverage mode does not yet support exclusion branches.

Grouping and Nested Queries

Parentheses allow multiple Boolean conditions to be combined into a single query. Grouped expressions are evaluated at the page level, making it possible to search for pages satisfying complex logical combinations of mathematical expressions.

Sequence Constraints (SEQ)

Search for expressions whose components appear in a specified symbolic order. SEQ is a more restrictive form of the AND operator. All query expressions or segments must be matched within the same mathematical expression and must appear in the order specified by the query. The ordering is determined by the canonical reading order of the mathematical expression rather than by their visual layout on the page. Sequence queries must be enclosed within $ACOR and $CCOR delimiters, which can be inserted using the corresponding bracket buttons in the query editor.

Query-by-Image

Search using a cropped image of a mathematical expression instead of typing LaTeX.

Indexed Segments

The frequency list shows expression segments that are present in the index, together with their accumulated score frequency. The list is filtered to exclude segments with an accumulated score frequency of 1 or less, reducing noise from one-off fragments while keeping commonly indexed segments inspectable. This can help inspect which formula fragments are available for direct search, segmentation, and coverage-based queries.

Open the indexed segments list

Additional Help and Feedback

Questions, bug reports, and feedback about the demonstrator are welcome. For assistance or to report issues, please contact danitei at prhlt dot upv dot es.