autopsy-flatpak/docs/doxygen-user/central_repo.dox

/*! \page central_repo_page Central Repository

\section cr_overview Overview

The central repository allows a user to find matching artifacts both across cases and across data sources in the same case. 
It is a combination of an ingest module that extracts, stores, and compares properties against lists of notable 
properties, a database that stores these properties, and an additional panel in Autopsy to display other instances of each 
property. The central repository database can either be SQLite or PostgreSQL.

The following are some use cases for the central repository:
- <b>Finding Other Instances of a Property</b>
 - If you navigate to a file or Autopsy artifact (such as a Web History item), there is a content viewer in the bottom right that will show you other instances of this property across the data stored in the central repository. 
- <b>Alerting When Previously Notable Properties Occur</b>
 - You can use the central repository to record which properties were associated with files and artifacts that were evidence (or notable).  Once these properties have been tagged as notable they will be added to the Interesting Items section of the tree when seen again in any future cases.
- <b>Storing Hash Sets</b>
 - You can create and import hash sets into the central repository instead of using local copies in the \ref hash_db_page "Hash Lookup module". These hash sets are functionally equivalent to local hash sets but can be shared among multiple analysts (when using a PostgreSQL central repository).

\section cr_terms Terms and Concepts

- <b>Central Repository</b> - The Autopsy feature containing the central repository database and Central Repository Ingest Module. Also responsible for displaying correlated properties to the user
- <b>Central Repository Database</b> - the SQLite or PostgreSQL database that holds all the data
- <b>Central Repository Ingest Module</b> - The ingest module responsible for adding new properties to the database and comparing these properties against existing notable properties
- <b>Property</b> - The data being stored/correlated. These can be file paths/MD5 hashes, email addresses, phone numbers, etc.

\section cr_setup Setup

The central repository settings are found on the main options panel (Tools->Options) on the "Central Repository" tab. 

\image html central_repo_options.png

\subsection cr_db_setup Database Configuration

There are two types of central repository databases:
- <b>SQLite</b> - This type of database is stored in a file. It should only be used when a single client will be accessing the database. You can not use this option with \ref multiuser_page "multi-user cases".
- <b>PostgreSQL</b> - This type of database is stored on a server running either on the user's host or a remote server. This option must be used if multiple users will be using the same database.

\subsubsection cr_db_setup_auto Automatic Database Configuration

Starting with Autopsy 4.15, when you load Autopsy and the central repository is not enabled you will be 
asked if you want to enable it. Doing so will create a SQLite database in your Autopsy user folder (on Windows this will be in AppData). You will only be prompted to do this once. Whichever option you select, you can change your central repository settings later as described below.

Since a SQLite database can't be used for multi-user cases, you are also given the option to switch to a PostgreSQL database when you \ref multiuser_install_clients "enable multi-user cases". If you are currently using a SQLite database, when you enable multi-user cases you will be asked if you want to switch to a PostgreSQL database on the same server. Note that the contents of your SQLite database will not be copied over.

\subsubsection cr_db_setup_manual Manual Database Configuration

On the central repository options panel, check the 'Use a Central Repository' option and then click the Configure button to set up a database. There are three options here:
- <b>SQLite</b> - This option stores the database in a file. It should only be used when a single client will be accessing the database.
- <b>PostgreSQL using multi-user settings</b> - This option uses a central repository on the same PostgreSQL server that has been configured for \ref multiuser_page "multi-user cases". This option can not be selected if multi-user cases are not enabled. This is one of the options if multiple users will be using the same database.
- <b>Custom PostgreSQL</b> - This option uses a database server running either on the user's host or a remote server, where the server is specified in the central repository settings. This is one of the options if multiple users will be using the same database.

Once a database has been configured, the lower two buttons on the main panel will be enabled, which will be described below.

<b>Setting up a PostgreSQL deployment using the multi-user settings</b>

See the \ref install_multiuser_page page for instructions on configuring a multi-user environment. Once done, you can select the "PostgreSQL using multi-user settings" option to create/use a central repository on that PostgreSQL server.

<b>Setting up a custom PostgreSQL deployment</b>

If needed, see the \ref install_postgresql_page page for help setting up your PostgreSQL server.

For PostgreSQL all values are required, but some defaults are provided for convenience.

\image html central_repo_postgres.png

- Host Name/IP is the hostname or IP of your PostgreSQL server.
- Port is the port that the PostgreSQL server is listening on; default is 5432.
- User Name is a PostgreSQL user that can create and modify databases
- User Password is the password for the user.

If the database does not exist, you wll be prompted to create it.


<b>Setting Up SQLite Deployment</b>

Select SQLite in the Database Type to set up a SQLite database. SQLite databases should not be used if more than one client will be accessing the central repository.

\image html central_repo_sqlite.png

Enter or browse to a folder for the database. If the database file does not exist in that folder, you will be prompted to create it. 

\subsection cr_manage_properties Manage Correlation Properties

The Central Repository ingest module can save different types of properties to the database. By default all properties are recorded, but 
this setting can be changed on the options panel through the Manage Correlation Properties button. Note that these settings 
are saved to the database, so in a multi-user setting any changes will affect all users.

\image html central_repo_types.png

Descriptions of the property types:
- <b>Files</b>
 - Files are correlated based on MD5 hash and file path and name. The \ref hash_db_page must be enabled.
- <b>Domains</b>
 - Domains are extracted from the various web artifacts, which primarily come from the \ref recent_activity_page.
- <b>Email Addresses</b>
 - Email addresses are created by modules such as the \ref email_parser_page.
- <b>Phone Numbers</b>
 - Phone numbers are currently only extracted from call logs, contact lists and message, which come from the \ref android_analyzer_page.
- <b>USB Devices</b>
 - USB device properties come from the registry parsing in the \ref recent_activity_page.
- <b>Wireless Networks</b>
 - Wireless networks are correlated on SSIDs, and come from the registry parsing in the \ref recent_activity_page.
- <b>MAC Addresses</b>
 - MAC address properties are currently only created by custom Autopsy modules.
- <b>IMEI Number</b>
 - IMEIs properties are currently only created by custom Autopsy modules.
- <b>IMSI Number</b>
 - IMSI properties are currently only created by custom Autopsy modules.
- <b>ICCID Number</b>
 - ICCID properties are currently only created by custom Autopsy modules.
- <b>Credit Card</b>
 - Credid Card properties are created by the \ref keyword_search_page.
- <b> App-specific Accounts (Facebook, Twitter, etc)</b>
 - These properties primarily come from the \ref android_analyzer_page.

\subsection cr_manage_orgs Manage Organizations

Organizations are stored in the central repository and contain contact information for the given organization. Organizations are used for Hash Sets saved in the central repository, and can also be associated with Autopsy cases. 

\image html central_repo_orgs.png

One default org, "Not Specified" will always be present in the list. New organizations can be created, edited, and deleted through the appropriate buttons. Note that any organization that is currently in use by a case or hash set can not be deleted. All fields apart from the organization name are optional.

\image html central_repo_new_org.png

\subsection cr_show_cases Manage Cases

Displays a list of all cases that are in the central repository database and details about each case.

\image html central_repo_details.png

\section cr_using_repo Using the Central Repository

\subsection cr_ingest_module Central Repository Module

The Central Repository ingest module is responsible for adding properties to the database and comparing each property 
against the list of notable properties. It is best to run all ingest modules to get the most out of the Correlation 
Engine. For example, if Hash Lookup is not run then the Central Repository module will not put any files into the 
database. If the Central Repository module is not run on a particular case but a central repository is enabled, 
there will still be some limited functionality. The Content Viewer will still display matching properties from 
other cases/data sources where the Central Repository was run.

\image html central_repo_ingest_settings.png

There are three settings for the Central Repository ingest module:
<ul>
<li><b>Save items to the Central Repository</b> - This should only be unselected in the rare case that you don't want to add any properties from the current data source to the central repository, but still want to flag past occurrences.
<li><b>Flag items previously tagged as notable</b> - Enabling this causes Interesting Item/File artifacts to be created when properties matching those previously flagged are found. See the next section \ref cr_tagging for details.
<li><b>Flag previously seen devices</b> - When this is enabled, an Interesting Item artifact will be created if any device-related property (USB, MAC Address, IMSI, IMEI, ICCID) is found that is already in the central repository, regardless of whether they have been flagged.
</li>

\subsection cr_tagging Tagging Files and Artifacts

Tagging a file or artifact with a "notable" tag will change its associated property in the central repository to notable as well. 
By default, there will be a tag named "Notable Item" that can be used for this purpose. See the \ref tagging_page "Tagging page" for more information on creating additional tags with notable status. 
Any future data source ingest (where this module is enabled) 
will use those notable properties in a similar manner as a Known Bad hash set, causing matching files and artifacts from that 
ingest to be added to the Interesting Items list in that currently open case.

\image html central_repo_tag_file.png

If a tag is accidentally added to a file or artifact, it can be removed though the context menu. This will remove its property's
notable status in the central repository.

If you would like to prevent the Interesting Items from being created in a particular case, you can disable the flagging 
through the run time ingest properties. Note that this only disables the Interesting Item results - all properties
are still added to the central repository.

\image html central_repo_ingest_settings.png

\section cr_viewing_results Viewing Results

Results from enabling a central repository and running the Central Repository Ingest Module can be seen in two places:
- The Content Viewer for each file or artifact will display all matching properties from other cases/data sources
- The Interesting Files node of the result tree will contain any files or results that matched properties previously marked as notable

\subsection cr_content_viewer Content Viewer

The \ref content_viewer_page panel is where previous instances of properties are displayed. Without a central repository enabled, 
this "Other Occurrences" panel will show files with hashes matching the selected file within the current case. Enabling a central 
repository allows this panel to also display matching properties stored in the database, and adds some functionality to the row. 
Note that the Central Repository Ingest Module does not have to have been run on the current data source to see correlated 
properties from the central repository. If the selected file or artifact is associated by one of the supported Correlation Types, 
to one or more properties in the database, the associated properties will be displayed. Note: the Content 
Viewer will display ALL associated properties available in the database. It ignores the user's enabled/disabled Correlation Properties.

The other occurrences are grouped by case and then data source. Selecting one of the results brings up information on it in the right column. If a file or artifact was previously marked as notable, you will see "notable" in red next to "Known Status".

\image html central_repo_content_viewer.png

The user can click on any column heading to sort by the values in that column.

If the user selects an entry in the third column and then right-clicks, a menu will be displayed.
This menu has several options.
-# Export All Other Occurrences to CSV
-# Show Case Details
-# Show Frequency

<b>Export All Other Occurrences to CSV</b>

This option will save every other occurrence in the Content Viewer table to a CSV file.
By default, the CSV file is saved into the Export directory inside the currently open Autopsy case,
but the user is free to select a different location.

<b>Show Case Details</b>

This option will open a dialog that displays all of the relevant details for the selected case. The 
details will include:
- Case UUID
- Case Name
- Case Creation Date
- Case Examiner contact information
- Case Examiner's notes

These details would have been entered by the examiner of the selected case, when creating the case or later by visiting
the Case -> Case Properties menu.

<b>Show Frequency</b>

This shows how common the selected file is. The value is the percentage of case/data source tuples that have the selected property.

\subsection cr_interesting_items Interesting Items

In the Results tree of an open case is an entry called Interesting Items. When this module is enabled, all of the enabled 
Correlatable Properties will cause matching files and artifacts to be added to this Interesting Items tree during ingest.

\image html central_repo_interesting_items.png

As an example, suppose the Files Correlatable Property is enabled and the ingest is currently processing a file "badfile.exe", and the MD5 hash 
for that file already exists in the database as a notable file property. In this case an entry in the Interesting Items tree will be added for 
the current instance of "badfile.exe" in the data source currently being ingested.

The same type of thing will happen for each enabled Correlatable Property.

In the case of the phone number correlatable type, the Interesting Items tree will start a sub-tree for each phone number. The sub-tree will 
then contain each instance of that notable phone number.




*/