autopsy-flatpak/docs/doxygen-user/central_repo.dox

/*! \page central_repo_page Central Repository

\section cr_overview Overview

The central repository allows a user to find matching artifacts both across cases and across data sources in the same case. 
It is a combination of an ingest module that extracts, stores, and compares properties against lists of notable 
properties, a database that stores these properties, and an additional panel in Autopsy to display other instances of each 
property. The central repository database can either be SQLite or PostgreSQL.

The following are some use cases for the Central Repository:
- <b>Finding Other Instances of a Property</b>
 - If you find a file or Autopsy artifact (such as a Web History item), there is a content viewer in the bottom right that will show you other cases that had this same file or that had items with the same feature (such as Domain name).   You will also be able to see what other data sources in the same case had this feature. 
- <b>Alerting When Previously Notable Properties Occur</b>
 - You can use the Central Repository to record which properties were associated with files and artifacts that were evidence (or notable).  Once these properties have been tagged as notable they will be added to the Interesting Items section of the tree when seen again in any future cases.
- <b>Storing Hash Sets</b>
 - You can create and import hash sets into the central repository instead of using local copies in the \ref hash_db_page "Hash Lookup module". These hash sets are functionally equivalent to local hash sets but can be shared among multiple analysts (when using a PostgreSQL central repository).

\section cr_terms Terms and Concepts

- <b>Central Repository</b> - The Autopsy feature containing the central repository database and Correlation Engine Ingest Module. Also responsible for displaying correlated properties to the user
- <b>Central Repository Database</b> - the SQLite or PostgreSQL database that holds all the data
- <b>Correlation Engine Ingest Module</b> - The ingest module responsible for adding new properties to the database and comparing these properties against existing notable properties
- <b>Property</b> - The data being stored/correlated. These can be file paths/MD5 hashes, email addresses, phone numbers, etc.

\section cr_setup Setup

To start, open the main options panel and select the Central Repository icon. 

\image html central_repo_options.png

\subsection cr_db_setup Setting up the Database

On the Central Repository options panel, check the 'Use a Central Repository' option and then click the Configure button to set up a database. There are two options here:
- <b>SQLite</b> - This option stores the database in a file. It should only be used when a single client will be accessing the database.
- <b>PostgreSQL</b> - This option uses a database server running either on the user's host or a remote server. This option must be used if multiple users will be using the same database.

Once a database has been configured, the lower two buttons on the main panel will be enabled, which will be described below.

<b>Setting up PostgreSQL Deployment</b>

If needed, see the \ref install_postgresql for help setting up your PostgreSQL server.

For PostgreSQL all values are required, but some defaults are provided for convenience.

\image html central_repo_postgres.png

- Host Name/IP is the hostname or IP of your PostgreSQL server.
- Port is the port that the PostgreSQL server is listening on; default is 5432.
- User Name is a PostgreSQL user that can create and modify databases
- User Password is the password for the user.

If the database does not exist, you wll be prompted to create it.


<b>Setting Up SQLite Deployment</b>

Select SQLite in the Database Type to set up a SQLite database. SQLite databases should not be used if more than one client will be accessing the central repository.

\image html central_repo_sqlite.png

Enter or browse to a folder for the database. If the database file does not exist in that folder, you will be prompted to create it. 

\subsection cr_manage_properties Manage Correlation Properties

The Correlation Engine ingest module can save different types of properties to the database. By default all properties are recorded, but 
this setting can be changed on the options panel through the Manage Correlation Properties button. Note that these settings 
are saved to the database, so in a multi-user setting any changes will affect all users.

\image html central_repo_types.png

Descriptions of the property types:
- <b>Files</b>
 - Files are correlated based on MD5 hash and file path and name. The Hash Lookup ingest module must be enabled.
- <b>Domains</b>
 - Domains are extracted from the various web artifacts, which primarily come from the Recent Activity module
- <b>Email Addresses</b>
 - Email addresses are pulled from Email Address hits from the Keyword Search module.
- <b>Phone Numbers</b>
 - Phone numbers are currently only extracted from call logs, contact lists and message, which come from the Android Analyzer module.
- <b>USB Devices</b>
 - USB device properties come from the registry parsing in the Recent Activity Module.

\subsection cr_manage_orgs Manage Organizations

Organizations are stored in the central repository and contain contact information for the given organization. Organizations are used for Hash Sets saved in the central repository, and can also be associated with Autopsy cases. 

\image html central_repo_orgs.png

One default org, "Not Specified" will always be present in the list. New organizations can be created, edited, and deleted through the appropriate buttons. Note that any organization that is currently in use by a case or hash set can not be deleted. All fields apart from the organization name are optional.

\image html central_repo_new_org.png
 
\section cr_using_repo Using the Central Repository

\subsection cr_ingest_module Correlation Engine Module

The Correlation Engine ingest module is responsible for adding properties to the database and comparing each property 
against the list of notable properties. It is best to run all ingest modules to get the most out of the Correlation 
Engine. For example, if Hash Lookup is not run then the Correlation Engine module will not put any files into the 
database. If the Correlation Engine module is not run on a particular case but a central repository is enabled, 
there will still be some limited functionality. The Content Viewer will still display matching properties from 
other cases/data sources where the Correlation Engine was run.

\subsection cr_tagging Tagging Files and Artifacts

Any file or artifact that a user tags with a tag with notable set will be added 
to the database as a file or artifact of interest. By default, there will be a tag named "Notable Item" that can be used for this purpose. See the \ref tagging_page "Tagging page" for more information on creating additional tags with notable status. Any future data source ingest (where this module is enabled) 
will use those notable files or artifacts in a similar manner as a Known Bad hash set, causing matching files from that 
ingest to be added to the Interesting Artifacts list in that currently open case.

\image html central_repo_tag_file.png

If a tag is accidentally added to a file or artifact, it can be removed though the context menu. This will remove its
notable status in the Central Repository.

If you would like to prevent the Interesting Items from being created in a particular case, you can disable the flagging through the run time ingest properties. Note that this only disables the Interesting Item results - all files and artifacts are still added to the central repository.

\image html central_repo_disable_flagging.png

\section cr_viewing_results Viewing Results

Results from enabling a central repository and running the Correlation Engine Ingest Module can be seen in two places:
- The Content Viewer for each file or artifact will display all matching properties from other cases/data sources
- The Interesting Files node of the result tree will contain any files or results that matched properties previously marked as notable

\subsection cr_content_viewer Content Viewer

The \ref content_viewer_page panel is where previous instances of properties are displayed. This module adds a new tab to the Content Viewer. The tab for this module is called "Other Occurrences". It can display data that is found in other cases, other data sources for the same case, or imported global artifacts.

If at least one other case or data source has been ingested with this module enabled, there is a potential that data will be displayed in the Other Occurrences content viewer. Note that the Correlation Engine Ingest Module does not have to have been run on the current data source to see correlated files from other cases/data sources. If the selected file or artifact is associated by one of the supported Correlation Types, to one or more file(s) or artifact(s) in the database, the associated files/artifacts will be displayed. Note: the Content Viewer will display ALL associated files and artifacts available in the database. It ignores the user's enabled/disabled Correlation Properties.

By default, the rows in the content viewer will have background colors to indicate if they are known to be of interest. Files/artifacts that are notable will have a Red background, all others will have a White background.

\image html central_repo_content_viewer.png

The user can click on any column heading to sort by the values in that column.

If the user right-clicks on a row, a menu will be displayed.
This menu has several options.
-# Select All
-# Export Selected Rows to CSV
-# Show Case Details
-# Show Frequency

<b>Select All</b>

This option will select all rows in the Content Viewer table.

<b>Export Selected Rows to CSV</b>

This option will save ALL SELECTED rows in the Content Viewer table to a CSV file.
By default, the CSV file is saved into the Export directory inside the currently open Autopsy case,
but the user is free to select a different location.

Note: if you want to copy/paste rows, it is usually possible to use CTRL+C to copy the
selected rows and then CTRL+V to paste them into a file, but it will not be CSV formatted.

<b>Show Case Details</b>

This option will open a dialog that displays all of the relevant details for the selected case. The 
details will include:
- Case UUID
- Case Name
- Case Creation Date
- Case Examiner contact information
- Case Examiner's notes

These details would have been entered by the examiner of the selected case, when creating the case or later by visiting
the Case -> Case Properties menu.

<b>Show Frequency</b>

This shows how common the selected file is. The value is the percentage of case/data source tuples that have the selected file or artifact.

\subsection cr_interesting_items Interesting Items

In the Results tree of an open case is an entry called Interesting Items. When this module is enabled, all of the enabled Correlatable Properties will cause matching files to be added to this Interesting Items tree during ingest.

\image html central_repo_interesting_items.png

As an example, if the Files Correlatable Property is enabled, and the ingest is currently processing a file, for example "badfile.exe", and the MD5 hash for that file already exists in the database as a notable file, then an entry in the Interesting Items tree will be added for the current instance of "badfile.exe" in the data source currently being ingested.

The same type of thing will happen for each enabled Correlatable Property.

In the case of the phone number correlatable type, the Interesting Items tree will start a sub-tree for each phone number. The sub-tree will then contain each instance of that notable phone number.




*/