autopsy-flatpak/docs/doxygen-user/keyword_search.dox

/*! \page keyword_search_page Keyword Search Module

[TOC]


\section keyword_module_overview What Does It Do

The Keyword Search module facilitates both the \ref ingest_page "ingest" portion of searching and also supports manual text searching after ingest has completed (see \ref ad_hoc_keyword_search_page). It extracts text from files being ingested, selected reports generated by other modules, and results generated by other modules. 

Autopsy tries its best to extract the maximum amount of text from the files being indexed. First, it will try to extract text from supported file formats, such as pure text file format, MS Office Documents, PDF files, Email, and many others. If the file is not supported by the standard text extractor, Autopsy will fall back to a string extraction algorithm. String extraction on unknown file formats or arbitrary binary files can often extract a sizeable amount of text from a file, often enough to provide additional clues to reviewers. String extraction will not extract text strings from encrypted files.

Autopsy ships with some built-in lists that define regular expressions and enable the user to search for Phone Numbers, IP addresses, URLs and E-mail addresses. However, enabling some of these very general lists can produce a very large number of hits, and many of them can be false-positives. Regular expressions can potentially take a long time to complete.

Refer to \ref ad_hoc_keyword_search_page for more details on specifying regular expressions and other types of searches. 

As of Autopsy 4.21.0 release, two types of keyword searching are supported - Solr search with full text indexing, or an built-in Autopsy "In-Line" Keyword Search. See \ref keyword_ingest_settings on details regarding search type configuraiton. 

\subsection keyword_SolrSearch Solr Search With Indexing

Full text indexing with Solr allows user the flexibility to run ad-hoc manual text searching after ingest has completed (see \ref ad_hoc_keyword_search_page). However, the process of full text indexing can greately slow down ingest speed for large datasources and/or cases. Once files are placed in the Solr index, they can be searched quickly for specific keywords, regular expressions, or keyword search lists that can contain a mixture of keywords and regular expressions.  

\subsection keyword_InlineSearch In-Line Keyword Search

The In-Line Keyword Search performs the keyword searching during ingest at the time of text extraction and only indexes small sections of the files that have keyword hits for display purposes. Our profiling runs show that in most cases this has reduced data source ingest time in half, meaning that using In-Line Keyword Search a data source ingest is completed in roughly half the time that it takes to ingest and search the same data source using Solr indexing. The downside is that all of the search terms must be specified ahead of the ingest, and there is no way to run ad-hoc search on the entire extracted text after ingest has completed. 

\section keyword_search_configuration_dialog Keyword Search Configuration Dialog

The keyword search configuration dialog has three tabs, each with its own purpose:
\li The \ref keyword_keywordListsTab is used to add, remove, and modify keyword search lists.
\li The \ref keyword_stringExtractionTab is used to enable language scripts and extraction type.
\li The \ref keyword_generalSettingsTab is used to configure display information.

\subsection keyword_keywordListsTab Lists tab

The Lists tab is used to create/import and add content to keyword lists. To create a list, select the 'New List' button and choose a name for the new Keyword List. Once the list has been created, keywords can be added to it (see \ref ad_hoc_kw_types_section for more information on keyword types). Lists can be added to the keyword search ingest process; searches will happen at regular intervals as content is added to the index.

\image html keyword-search-configuration-dialog.PNG

The lists of keywords can be found on the left side of the panel. New lists can be created, existing lists can be renamed, copied, exported, or deleted, and lists can be imported. Autopsy supports importing Encase tab-delimited lists as well as lists created previously with Autopsy. For Encase lists, folder structure and hierarchy is ignored. There is currently no way to export lists for use with Encase, but lists can be exported to share between Autopsy users.

Once a keyword list is selected all keywords in that list will be displayed on the right side of the tab. The "New Keywords" button can be used to add one or more entries to the list, and the "Edit keyword" and "Delete keywords" buttons can alter the existing entries. 

\image html keyword-search-configuration-new-keywords.PNG

New entries can be typed into the dialog or pasted from the clipboard. All entries added at once must be the same type of match (exact, substring, or regex), but the dialog can be used multiple times to add keywords to the keyword list.  Refer to the \ref ad_hoc_kw_types_section section for an explanation of each keyword type.

Under the Keyword list is the option to send ingest inbox messages for each hit. If this is enabled, each keyword hit for that list will be accessible through the yellow triangle next to the Keyword Lists button. This feature gives you a quick way to view your most important keyword search results.

\image html keyword-search-inbox.PNG

\subsection keyword_stringExtractionTab String Extraction tab 
The string extraction setting defines how strings are extracted from files from which text cannot be extracted normally because their file formats are not supported. This is the case with arbitrary binary files (such as the page file) and chunks of unallocated space that represent deleted files.
When we extract strings from binary files we need to interpret sequences of bytes as text differently, depending on the possible text encoding and script/language used. In many cases we don't know in advance what the specific encoding/language the text is encoded in. However, it helps if the investigator is looking for a specific language, because by selecting less languages the indexing performance will be improved and the number of false positives will be reduced.

\image html keyword-search-configuration-dialog-string-extraction.PNG

The default setting is to search for English strings only, encoded as either UTF8 or UTF16. This setting has the best performance (shortest ingest time).
The user can also use the String Viewer first and try different script/language settings, and see which settings give satisfactory results for the type of text relevant to the investigation. Then the same setting that works for the investigation can be applied to the keyword search ingest.


\subsection keyword_generalSettingsTab General Settings tab

\image html keyword-search-configuration-dialog-general.PNG

\subsubsection keyword_nsrl NIST NSRL Support
The hash lookup ingest service can be configured to use the NIST NSRL hash set of known files. The keyword search advanced configuration dialog "General" tab contains an option to skip keyword indexing and search on files that have previously marked as "known" and uninteresting files. Selecting this option can greatly reduce size of the index and improve ingest performance. In most cases, user does not need to keyword search for "known" files.

\section keyword_usage Using the Module

After the ingest has completed, \ref ad_hoc_keyword_search_page will be available for manual search. The amount of files/text available for Ad Hoc Search depends on the Keyword Search module settings at the time of the ingest. See section \ref adhoc_limitations for details. 

\subsection keyword_ingest_settings Ingest Settings

The Ingest Settings for the Keyword Search module allow the user to enable or disable the specific built-in search expressions, Phone Numbers, IP Addresses, Email Addresses, and URLs. Using the Advanced button (covered below), one can add custom keyword groups.

As of Autopsy 4.21.0 release, two types of keyword searching are supported - Solr search with full text indexing, and/or an built-in Autopsy "In-Line" Keyword Search. See \ref keyword_ingest_settings on details regarding search type configuraiton. See sections \ref keyword_SolrSearch and \ref keyword_InlineSearch for details of each search type.

The keyword searh type selection is accomplished via "Add text to Solr Index" checkbox. If the checkbox is unchecked, Autopsy will perform "In-Line" Keyword Search during ingest but most of the extracted text will not be indexed by Solr, effectively disabling \ref ad_hoc_keyword_search_page functionality. If the checkbox is selected, Autopsy will perform "In-Line" Keyword Search during ingest, as well as add all of the extracted text to Solr index so that it can be searched later using \ref ad_hoc_keyword_search_page .

\image html keyword-search-ingest-settings.PNG

\subsubsection keyword_ocr Optical Character Recognition
\anchor keyword_search_ocr_config

There is also a setting to enable Optical Character Recognition (OCR). If enabled, text may be extracted from supported image types. Enabling this feature will make the keyword search module take longer to run, and the results are not perfect.

The following shows a sample image containing text:

\image html keyword-search-ocr-image.png

The "Indexed Text" tab shows the results when running the keyword search module with the OCR option enabled. If we were to use Keyword Search to look for the word "forensics", this file would be a match.

\image html keyword-search-ocr-indexed-text.png

The two options to related to OCR are the following:
<ul>
<li>Only index text extracted from an image. This will prevent keyword search from indexing text found in text files, docs, etc.
<li>Only run on large images and documents and extracted files. With this selected, OCR will only be performed on images over 100KB and PDFs/Office docs. It will also run on images of any size that were extracted from another file.
</ul>

By default, OCR is only configured for English text. Its configuration depends on the presence of language files (called "traineddata" files) 
that exist in a location that Autopsy can understand. To add support for more languages, you will need to download additional "traineddata" 
and move them to the right location. The following steps breakdown this process for you:

<ol>
<li>Navigate to https://tesseract-ocr.github.io/tessdoc/Data-Files.
<li>Under the section titled "Data Files for Version 4.00 (November 29, 2016)" you will find a table containing files that represent each language. These files have the extension ".traineddata".
<li>To download the desired language, click on the links in the far right-hand column of the table. You may download as many as you like. Note that you must only choose from this table. Language files under any other sections are not guaranteed to work in Autopsy.
<li>Once you've downloaded your language files, simply drag and drop them into the "AppData\Roaming\autopsy\ocr_language_packs" folder under your user folder.
<li>Start up Autopsy and you will be all set. If Autopsy was running, this will require a restart to take effect.
</ol>

The language files will now be supported when OCR is enabled in the Keyword Search Settings.


\section keyword_results Seeing Results

The Keyword Search module will save the search results regardless whether the search is performed by the ingest process, or manually by the user. The saved results are available in the Directory Tree in the left hand side panel.

The keyword results will appear in the tree under "Keyword Hits". Each keyword search term will display the number of matches, and can be expanded to show the matches. From here, clicking on one of the matches will show a list of files on the right side of the screen. Select a file and go to the Indexed Text tab to see exactly where the matches occurred in the file.

\image html keyword_results.PNG

 
*/