/*! \page ingest_page Ingest Modules
[TOC]
Ingest modules analyze the data in a data source. They perform all of the analysis of the files and parse their contents. Examples include \ref hash_db_page "hash calculation and lookup", \ref keyword_search_page "keyword searching", and \ref recent_activity_page "web artifact extraction".
Immediately after you add a data source to a case (see \ref ds_page), you will be presented with a dialog to configure the ingest modules to run on it. Once configured, they will run in the background and provide you real-time results when they find relevant information.
This page covers the use of ingest modules. Specific pages will cover the configuration of specific modules. See \ref module_install_page for details on installing 3rd-party ingest modules.
\section ingest_performance Multi-threaded and Priority
Ingest modules are configured to find user content quickly. The ingest modules are grouped into pipelines and each file goes down the pipeline, module by module. A pipeline may have modules in the following order:
\image html ingest_pipeline.PNG
Multiple pipelines may be running at the same time. By default, two pipelines are running, but you can add more depending on how many cores you have on your system. You can configure the number of pipelines to make in the "Tools", "Options", "General" area.
Autopsy prioritizes user content over other types of files and will send data from the "Documents and Settings" folder or "Users" folder into the pipelines before the "Windows" folder. It prioritizes each folder in the system to ensure that user content is analyzed before other content.
\section ingest_running Running Ingest Modules
There are two ways to start ingest modules:
-# Immediately after you add a data source
-# By right-clicking on a data source from the tree in the main interface and choosing "Run Ingest Modules"
Once ingest is started, you can review the currently running ingest tasks in the task bar on the bottom-right corner of the main window. The ingest tasks can be cancelled by the user if so desired.
Note: sometimes the cancellation process may take several seconds or more to complete cleanly, depending on what the ingest module was currently doing.
\section ingest_configure Configuring Ingest Modules
You will be presented with an interface to configure the ingest modules. From here, you can choose which type of files to analyze and enable or disable each module. Some modules will have further configuration settings.
\image html select-ingest-modules.PNG
The selection box at the top controls which files the ingest modules will run on. The two built-in options are "All files, directories, and unallocated space" and "All Files and Directories." The \ref file_filters section describes how to create custom file filters. The chosen filter applies to all ingest modules.
There are two places to configure ingest modules. When you select the module name, you may have some "run time" options to configure in the panel to the right. These are generally settings that you may want to change from image to image.
There may also be an "Advanced" button that is enabled in the lower corner. Pressing this button allows you to change global settings that are not specific to a single image. This advanced configuration panel can often be found in the "Tools", "Options" menu too.
As an example, the hash lookup module will allow you to enable or disable hash sets in the "run time" options panel, but requires you to go to the "Advanced" dialog to add or remove hash sets from the Autopsy configuration.
\section file_filters Custom File Filters
The file filters panel can be opened from the ingest module selection panel or through the Ingest tab on the main options panel. File filters allow ingest modules to be run on only a subset of the files. In the example below, a filter has been set up to only run on files with a "png" extension.
\image html ingest-file-filters.PNG
Each filter contains one or more rules for selecting files based on a combination of file name, path, file size, and how recently the file was modified. A file will match a rule set based on these conventions:
- If there are no rules in a rule set, no file will match the set.
- If there are only rules that exclude files, as long as the file doesn’t match one of the exclusive rules, it is included in the set.
- If both inclusive rules and exclusive rules are present, a file must match one of the inclusive rules and not match any of the exclusive rules to be included in the set.
- If there are only inclusive rules, a file must match one of the rules to be included in the set.
Additionally, you can enter multiple comma-separated file extensions. All files will still be displayed in the tree view, but the ingest modules will only run on a subset. If we use the previous example and run the hash module, only files ending in .png will have their hash computed.
\section ingest_profiles Using Ingest Profiles
Ingest profiles allow you to quickly choose a defined set of ingest modules to run. This can be useful if you run different sets of ingest modules (or different configurations of those ingest modules) on different types of data. Ingest profiles can be configured through the Ingest tab on the options panel.
\image html ingest-profiles.PNG
Each profile can specify different per-run settings for each ingest module, and you can choose use either a predefined or custom file filter (see \ref file_filters).
\image html ingest-profile-create.PNG
If any custom profiles are present, there will be a new screen in the add data source wizard.
\image html profile-data-source-panel.PNG
If you choose custom settings it will bring up the normal ingest module selection panel. If you choose a user-defined profile the ingest module screen will be skipped entirely and the ingest modules from that profile will be run on the data source. The profile selection panel will also appear when running ingest by right-clicking on a data source from the tree.
\section ingest_already_run Notification of Ingest Already Run
If an ingest module has already been run for a particular data source, you will see a triangular yellow icon with an exclaimation point next to the module in the "Run Ingest Modules" dialog, as shown in the screenshot below.
\image html ingest-already-run.PNG
If an older version of an ingest module has been run for a particular data source, you will see a round blue icon with an "i" next to the module in the "Run Ingest Modules" dialog, as shown in the screenshot below.
\image html previous-version-already-run.PNG
Clicking "View Ingest History" will show you the ingest history in tabular form, allowing you to see which modules were run on which data sources and when, as shown in the screenshot below.
\image html ingest-history.PNG
\section ingest_results Viewing Ingest Module Results
Ingest modules run in the background. An ingest module can provide you results in a variety of ways, but we recommend specific methods:
-# If they post results to the Blackboard, then you will find them in the "Results" area of the tree in the main interface.
-# They can send a message to the Ingest Inbox so that you get a message each time something really important is found.
\image html inbox-button.PNG
\image html inbox-main-screen.PNG
-# If the module is a wrapper around another forensics tool, they may simply provide a link to the output of that tool, in which case you will see a new entry in the "Reports" area of the tree.
All of the official Autopsy modules send results to the blackboard, but if you install third-party apps, then they may choose any approach -- including a pop-up window each time they find something.
\section ingest_monitoring Viewing Ongoing Ingest Activity
While Ingest is running, one can use the "Ingest Progress Snapshot" tool to see what activity is going on at the moment. Click on "Help", "Get Ingest Progress Snapshot" to view the dialog shown in the screenshot below.
\image html ingest_progress_snapshot.PNG
To refresh the view, use the "Refresh" button.
*/