Invisible East documents now searchable on OpenITI / KITAB

Around 1,300 documents and fragments from the Invisible East Digital Corpus (IEDC) are now interoperable with the OpenITI/KITAB corpus, which contains more than 1 billion words of long-form texts in Arabic and Persian. Researchers can now triangulate our data with these texts, tracing passages, phrases, and formulas shared between documents and literary works.

The IEDC adds documentary sources to a corpus that consists largely of literary works: records of governance, administration and the economy in action, including texts in Bactrian and Middle Persian, languages not otherwise represented on KITAB. Setting them against the wider body of Islamicate literature lets researchers look for continuities between everyday documentary writing and the formulas, phrases and passages of the long-form texts.

KITAB is a platform for exploring and comparing digitised historical texts. It is particularly useful for finding passages that appear in more than one text. It works with the OpenITI corpus, which is stored in plain text with a simple system of markup, which makes it suitable for computational analysis. The corpus is available on GitHub, which hosts the continuously updated working version, and on Zenodo, which preserves fixed versions for academic citation. Citing a Zenodo version lets you refer to the exact version of a text used in your work.

How to find IEDC documents on KITAB

Open the KITAB corpus explorer
Search for a collection. Enter one of the following terms in the search bar (or use the links):
- Firuzkuh Papers: 'Firuzkuh'
- National Library of Israel (Bamiyan Papers): 'Ms.Heb.8333.'
- Khalili Collection: 'Khalili collection'

Filter by language: select Filters, then choose the relevant language from the dropdown menu (see images). The languages are identified by three-letter codes:
- per – Persian
- ara – Arabic
- bac – Bactrian
- mpp – Middle Persian (Pahlavi)

The data

KITAB brings together expertise from computer science, history, linguistics and the study of written texts. Consequently, some of the information displayed in its tables may be unfamiliar to users without a background in digital humanities. The following explanations introduce the main headings (see underlined in the image, from right to left):

Version ID

Each digital text has a unique identifier, displayed in the first column. This identifies the particular digitised version of a text within the OpenITI corpus.

Below the identifier are several icons that allow users to interact with the entry. These include options for reporting a problem, copying the text's identifier to the clipboard, and downloading the digital text (see red box in the image above).

The entry also indicates the language of the text, using the three-letter codes explained above, and provides information about its annotation status.

Annotation refers to the addition of structural information to a digital text, such as marking headings, sections or other textual features. This makes it easier for researchers and computer programs to navigate and analyse its contents.

For further information about downloaded OpenITI texts and their markup, consult the OpenITI documentation.

Shelfmark / book title

The shelfmark or name under which the document / book was incorporated into KITAB.

Author / holding

For literary works, this column normally identifies the author. For archival documents, it indicates the holding institution or collection.

For example, documents from the Firuzkuh Papers may be associated with the Afghanistan National Archive, while documents from the Bamiyan Papers may be associated with the National Library of Israel.

Author death date

For literary works, this column records the author's death date, which can help place the work in its historical context.

This field is generally not applicable to the IEDC archival documents discussed here.

Token count

This indicates the approximate length of the digitised text.

Computers measure text in 'tokens', which are usually individual words or punctuation marks. The token count can be taken as a roughly accurate word count.

Text Reuse

This column provides information about passages that KITAB has identified as appearing in other texts from the OpenITI corpus.

Text reuse means that two or more texts contain identical or sufficiently similar passages. KITAB identifies these connections using a computational method that searches for shared sequences of text.

Selecting an entry with reuse data opens a visualisation that allows users to investigate these connections in greater detail.

Importantly, textual similarity does not necessarily mean that one document was copied directly from another. Shared passages may also reflect common sources, conventional expressions, or documentary formulas.

Text reuse visualisation graph

To see the visualisation, click on your entry's magnifying glass icon (see image). This opens an interactive graph showing where KITAB has identified passages shared between the selected text and other texts in the corpus.

This graph provides a visual overview of which parts of the document have parallels elsewhere and which other works contain these parallels (see image).

Reading the graph

The horizontal axis shows other texts in which KITAB found related passages, in chronological order, with earlier texts on the left. Each dot marks a match.

The vertical axis represents your selected text, divided into consecutive units called milestones, each normally containing 300 words. The first milestone is the beginning of the text and appears at the top. For short documents, the whole text may fall within a single milestone. The colour of each dot reflects how many characters of the selected passage are included in the match. In the comparison viewer, coloured highlighting separates wording shared by both passages from wording found in only one.

Examining the matching passages

Double-clicking a dot allows users to examine the corresponding passages in more detail.

The comparison viewer places the two passages alongside one another and uses coloured highlighting to identify textual similarities and differences.

Depending on the viewer's display settings, users can distinguish words or passages shared by the two texts from wording that appears in only one of them. This makes it possible to identify both close textual correspondences and small variations in wording. Such comparisons can help researchers investigate whether two documents reproduce the same formula, share a particular phrase, or contain other noteworthy textual correspondences.

Interpreting the results

KITAB's automated comparisons are a starting point for further investigation, rather than definitive evidence of a historical relationship between two documents. A detected correspondence should be examined in context to establish whether it reflects meaningful textual borrowing, shared documentary conventions, or a more coincidental similarity.

Equally, the absence of identified reuse does not necessarily mean that no relationship exists: very short passages, paraphrases, spelling differences, or material outside the corpus may escape automated detection.