mstoeck3/autopsy-flatpak
1
0
Fork 0
mirror of https://github.com/overcuriousity/autopsy-flatpak.git synced 2025-07-06 21:00:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
autopsy-flatpak/docs/doxygen/modIngest.dox
Richard Cordovano 1ca7d7dea1 Updated ingest module developer guide documentation
2014-04-01 16:14:57 -04:00

492 lines
26 KiB
Plaintext
Raw Blame History

/*! \page mod_ingest_page Developing Ingest Modules
\section ingest_modules_getting_started Getting Started
This page describes how to develop ingest modules. It assumes you have
already set up your development environment as described in \ref mod_dev_page.
Ingest modules analyze data from a data source (e.g., a disk image or a folder
of logical files). Autopsy organizes ingest modules into sequences known as
ingest pipelines. Autopsy may start up multiple pipelines for each ingest job.
An ingest job is what Autopsy calls the processing of a single data source and
the files it contains. There are two types of ingest modules:
- Data-source-level ingest modules
- File-level ingest modules
Each ingest module typically focuses on a single, specific type
of analysis. Here are some guidelines for choosing the type of your ingest module:
- Your module should be a data-source-level ingest module if it only needs to
retrieve and analyze a small subset of the files present in a data source.
For example, a Windows registry analysis module that only processes
registry hive files should be implemented as a data-source-level ingest module.
- Your module should be a file-level ingest module if it analyzes most or all of
the files from a data source, one file at a time. For example, a hash look up
module might process every file system file by looking up its hash in one or
more known file and known bad files hash sets (hash databases).
As you will learn a little later in this guide, it is possible to package a
data-source-level ingest module and a file-level ingest module together. You
would do this when you need to work at both levels to get all of your analysis
done. The modules in such a pair will be enabled or disabled together and will
have common per ingest job and global settings.
The following sections of this page delve into what you need to know to develop
your own ingest modules:
- \ref ingest_modules_implementing_ingestmodule
- \ref ingest_modules_implementing_datasourceingestmodule
- \ref ingest_modules_implementing_fileingestmodule
- \ref ingest_modules_services
- \ref ingest_modules_implementing_ingestmodulefactory
- \ref ingest_modules_pipeline_configuration
- \ref ingest_modules_api_migration
You may also want to look at the org.sleuthkit.autopsy.ingest.example package to
see a sample of each type of module. The sample modules don't do anything
particularly useful, but they can serve as templates for developing your own
ingest modules.
\section ingest_modules_implementing_ingestmodule Implementing the IngestModule Interface
All ingest modules, whether they are data source or file ingest modules, must
implement the two methods defined by the org.sleuthkit.autopsy.ingest.IngestModule
interface:
- org.sleuthkit.autopsy.ingest.IngestModule.startUp()
- org.sleuthkit.autopsy.ingest.IngestModule.shutDown()
The startUp() method is invoked by Autopsy when it starts up the ingest pipeline
of which the module instance is a part. This gives your ingest module instance an
opportunity to set up any internal data structures and acquire any private
resources it will need while doing its part of the ingest job. The module
instance probably needs to store a reference to the
org.sleuthkit.autopsy.ingest.IngestJobContext object that is passed to startUp().
The job context provides data and services specific to the ingest job and the
pipeline. If an error occurs during startUp(), the module should throw an
org.sleuthkit.autopsy.ingest.IngestModule.IngestModuleException object. If a
module instance throws an exception, the module will be immediately discarded, so clean
up for exceptional conditions should occur within startUp().
The shutDown() method is invoked by Autopsy when an ingest job is completed or
canceled and it is shutting down the pipeline of which the module instance is a
part. The module should respond by doing things like releasing private resources, and if the job was not
canceled, posting final results to the blackboard and perhaps submitting a final
message to the user's ingest messages inbox (see \ref ingest_modules_making_results).
As a module developer, it is important for you to realize that Autopsy will
generally use several instances of an ingest module for each ingest job it
performs. In fact, an ingest job may be processed by multiple pipelines using
multiple worker threads. However, you are guaranteed that there will be exactly
one thread executing code in any module instance, so you may freely use
unsynchronized, non-volatile instance variables. On the other hand, if your
module instances must share resources through static class variables or other means,
you are responsible for synchronizing access to the shared resources
and doing reference counting as required to release those resources correctly.
Also, more than one ingest job may be in progress at any given time. This must
be taken into consideration when sharing resources or data that may be specific
to a particular ingest job. You may want to look at the sample ingest modules
in the org.sleuthkit.autopsy.ingest.example package to see a simple example of
sharing per ingest job state between module instances.
The org.sleuthkit.autopsy.ingest.DataSourceIngestModule and org.sleuthkit.autopsy.ingest.FileIngestModule
interfaces both extend org.sleuthkit.autopsy.ingest.IngestModule.
For your convenience, an ingest module that does not require
initialization and/or clean up may extend the abstract
org.sleuthkit.autopsy.ingest.IngestModuleAdapter class to get default
"do nothing" implementations of these methods.
\section ingest_modules_implementing_datasourceingestmodule Creating a Data Source Ingest Module
To create a data source ingest module, make a new Java class either manually or
using the NetBeans wizards. Make the class implement
org.sleuthkit.autopsy.ingest.DataSourceIngestModule and optionally make it
extend org.sleuthkit.autopsy.ingest.IngestModuleAdapter. The NetBeans IDE
will complain that you have not implemented one or more of the required methods.
You can use its "hints" to automatically generate stubs for the missing methods. Use this page and the
documentation for the org.sleuthkit.autopsy.ingest.IngestModule and
org.sleuthkit.autopsy.ingest.DataSourceIngestModule interfaces for guidance on
what each method needs to do. Or you can copy the code from
org.sleuthkit.autopsy.examples.SampleDataSourceIngestModule and use it as a
template for your module. The sample module does not do anything particularly
useful, but it should provide a skeleton for you to flesh out with your own code.
All data source ingest modules must implement the single method defined by the
org.sleuthkit.autopsy.ingest.DataSourceIngestModule interface:
- org.sleuthkit.autopsy.ingest.DataSourceIngestModule.process()
The process() method is where all of the work of a data source ingest module is
done. It will be called exactly once between startUp() and shutDown(). The
process() method receives a reference to an org.sleuthkit.datamodel.Content object
and an org.sleuthkit.autopsy.ingest.DataSourceIngestModuleStatusHelper object.
The former is a representation of the data source. The latter should be used
by the module instance to be a good citizen within Autopsy as it does its
potentially long-running processing. Here is a code snippet showing the
skeleton of a well-behaved process() method from the sample module:
\code
@Override
public ProcessResult process(Content dataSource, DataSourceIngestModuleStatusHelper statusHelper) {
// There are two tasks to do. Use the status helper to set the the
// progress bar to determinate and to set the remaining number of work
// units to be completed.
statusHelper.switchToDeterminate(2);
Case autopsyCase = Case.getCurrentCase();
SleuthkitCase sleuthkitCase = autopsyCase.getSleuthkitCase();
Services services = new Services(sleuthkitCase);
FileManager fileManager = services.getFileManager();
try {
// Get count of files with .doc extension.
long fileCount = 0;
List<AbstractFile> docFiles = fileManager.findFiles(dataSource, "%.doc");
for (AbstractFile docFile : docFiles) {
if (!skipKnownFiles || docFile.getKnown() != TskData.FileKnown.KNOWN) {
++fileCount;
}
}
statusHelper.progress(1);
// Get files by creation time.
long currentTime = System.currentTimeMillis() / 1000;
long minTime = currentTime - (14 * 24 * 60 * 60); // Go back two weeks.
List<FsContent> otherFiles = sleuthkitCase.findFilesWhere("crtime > " + minTime);
for (FsContent otherFile : otherFiles) {
if (!skipKnownFiles || otherFile.getKnown() != TskData.FileKnown.KNOWN) {
++fileCount;
}
}
// This method is thread-safe and keeps per ingest job counters.
addToFileCount(context.getJobId(), fileCount);
statusHelper.progress(1);
} catch (TskCoreException ex) {
IngestServices ingestServices = IngestServices.getInstance();
Logger logger = ingestServices.getLogger(SampleIngestModuleFactory.getModuleName());
logger.log(Level.SEVERE, "File query failed", ex);
return IngestModule.ProcessResult.ERROR;
}
return IngestModule.ProcessResult.OK;
}
\endcode
Note that data source ingest modules must find the files that they want to analyze.
The best way to do that is using one of the findFiles() methods of the
org.sleuthkit.autopsy.casemodule.services.FileManager class, as demonstrated
above. See
\ref mod_dev_other_services for more details.
\section ingest_modules_implementing_fileingestmodule Creating a File Ingest Module
To create a file ingest module, make a new Java class either manually or
using the NetBeans wizards. Make the class implement
org.sleuthkit.autopsy.ingest.FileIngestModule and optionally make it
extend org.sleuthkit.autopsy.ingest.IngestModuleAdapter. The NetBeans IDE
will complain that you have not implemented one or more of the required methods.
You can use its "hints" to automatically generate stubs for the missing methods. Use this page and the
documentation for the org.sleuthkit.autopsy.ingest.IngestModule and
org.sleuthkit.autopsy.ingest.FileIngestModule interfaces for guidance on what
each method needs to do. Or you can copy the code from
org.sleuthkit.autopsy.examples.SampleFileIngestModule and use it as a
template for your module. The sample module does not do anything particularly
useful, but it should provide a skeleton for you to flesh out with your own code.
All file ingest modules must implement the single method defined by the
org.sleuthkit.autopsy.ingest.FileIngestModule interface:
- org.sleuthkit.autopsy.ingest.FileIngestModule.process()
The process() method is where all of the work of a file ingest module is
done. It will be called repeatedly between startUp() and shutDown(), once for
each file Autopsy feeds into the pipeline of which the module instance is a part. The
process() method receives a reference to a org.sleuthkit.datamodel.AbstractFile
object. Here is a code snippet showing the
skeleton of a well-behaved process() method from the sample module:
\code
@Override
public IngestModule.ProcessResult process(AbstractFile file) {
if (attrId != -1) {
return IngestModule.ProcessResult.ERROR;
}
// Skip anything other than actual file system files.
if ((file.getType() == TskData.TSK_DB_FILES_TYPE_ENUM.UNALLOC_BLOCKS)
|| (file.getType() == TskData.TSK_DB_FILES_TYPE_ENUM.UNUSED_BLOCKS)) {
return IngestModule.ProcessResult.OK;
}
// Skip NSRL / known files.
if (skipKnownFiles && file.getKnown() == TskData.FileKnown.KNOWN) {
return IngestModule.ProcessResult.OK;
}
// Do a nonsensical calculation of the number of 0x00 bytes
// in the first 1024-bytes of the file. This is for demo
// purposes only.
try {
byte buffer[] = new byte[1024];
int len = file.read(buffer, 0, 1024);
int count = 0;
for (int i = 0; i < len; i++) {
if (buffer[i] == 0x00) {
count++;
}
}
// Make an attribute using the ID for the attribute type that
// was previously created.
BlackboardAttribute attr = new BlackboardAttribute(attrId, SampleIngestModuleFactory.getModuleName(), count);
// Add the to the general info artifact for the file. In a
// real module, you would likely have more complex data types
// and be making more specific artifacts.
BlackboardArtifact art = file.getGenInfoArtifact();
art.addAttribute(attr);
// Thread-safe.
addToBlackboardPostCount(context.getJobId(), 1L);
// Fire an event to notify any listeners for blackboard postings.
ModuleDataEvent event = new ModuleDataEvent(SampleIngestModuleFactory.getModuleName(), ARTIFACT_TYPE.TSK_GEN_INFO);
IngestServices.getInstance().fireModuleDataEvent(event);
return IngestModule.ProcessResult.OK;
} catch (TskCoreException ex) {
IngestServices ingestServices = IngestServices.getInstance();
Logger logger = ingestServices.getLogger(SampleIngestModuleFactory.getModuleName());
logger.log(Level.SEVERE, "Error processing file (id = " + file.getId() + ")", ex);
return IngestModule.ProcessResult.ERROR;
}
}
\endcode
\section ingest_modules_services Using Ingest Services
The singleton instance of the org.sleuthkit.autopsy.ingest.IngestServices class
provides services tailored to the needs of ingest modules, and a module developer
should use these utilities to log errors, send messages, get the current case,
fire events, persist simple global settings, etc. Refer to the documentation
of the IngestServices class for method details.
\section ingest_modules_making_results Posting Ingest Module Results
Ingest modules run in the background. There are three ways to send messages and
save results so that the user can see them:
- Use the blackboard for long-term storage of analysis results. These results
will be displayed in the results tree.
- Use the ingest messages inbox to notify users of high-value analysis results
that were also posted to the blackboard.
- Use the logging and/or message box utilities for error messages.
\subsection ingest_modules_making_results_bb Posting Results to the Blackboard
The blackboard is used to store results so that they are displayed in the results tree.
See \ref platform_blackboard for details on posting results to it.
The blackboard defines artifacts for specific data types (such as web bookmarks).
You can use one of the standard artifact types, create your own, or simply post text
as a org.sleuthkit.datamodel.BlackboardArtifact.ARTIFACT_TYPE.TSK_TOOL_OUTPUT artifact.
The latter is much easier (for example, you can simply copy in the output from
an existing tool), but it forces the user to parse the output themselves.
When modules add data to the blackboard, they should notify listeners of the new
data by invoking the org.sleuthkit.autopsy.ingest.IngestServices.fireModuleDataEvent() method.
Do so as soon as you have added an artifact to the blackboard.
This allows other modules (and the main UI) to know when to query the blackboard
for the latest data. However, if you are writing a large number of blackboard
artifacts in a loop, it is better to invoke org.sleuthkit.autopsy.ingest.IngestServices.fireModuleDataEvent()
only once after the bulk write, so as not to flood the system with events.
\subsection ingest_modules_making_results_inbox Posting Results to the Message Inbox
Modules should post messages to the inbox when interesting data is found.
Of course, such data should also be posted to the blackboard as described above. The idea behind
the ingest messages is that they are presented in chronological order so that
users can see what was found while they were focusing on something else.
Inbox messages should only be sent if the result has a low false positive rate
and will likely be relevant. For example, the core Autopsy hash lookup module
sends messages if known bad (notable) files are found, but not if known good
(NSRL) files are found. This module also provides a global setting
(using its global settings panel) that allows a user to turn these messages on
or off.
Messages are created using the org.sleuthkit.autopsy.ingest.IngestMessage class
and posted to the inbox using the org.sleuthkit.autopsy.ingest.IngestServices.postMessage()
method.
\subsection ingest_modules_making_results_error Reporting Errors
When an error occurs, you should write an error message to the Autopsy logs, using a
logger obtained from org.sleuthkit.autopsy.ingest.IngestServices.getLogger().
You could also send an error message to the ingest inbox. The
downside of this is that the ingest inbox was not really designed for this
purpose and it is easy for the user to miss these messages. Therefore, it is
preferable to post a pop-up message that is displayed in the lower right hand
corner of the main window by calling
org.sleuthkit.autopsy.coreutils.MessageNotifyUtil.Notify.show().
\section ingest_modules_implementing_ingestmodulefactory Creating an Ingest Module Factory
When Autopsy needs an instance of an ingest module to put in a pipeline for an
ingest job, it turns to the ingest module factories registered as providers of
the IngestModuleFactory service.
Each of these ingest module factories may provide global and per ingest job
settings user interface panels. The global
settings should apply to all module instances. The per ingest job settings
should apply to all module instances working on a particular ingest job. Autopsy
supports context-sensitive and persistent per ingest job settings, so these
settings must be serializable.
During ingest job configuration, Autopsy bundles the ingest module factory with
the ingest job settings specified by the user and expects the ingest factory to
be able to create any number of module instances using those settings. This
implies that the constructors of ingest modules that have per ingest job settings
must accept settings arguments. You must also provide a mechanism for your ingest
module instances to access global settings, should you choose to have them. For
example, the Autopsy core hash look up module comes with a singleton hash databases
manager. Users import and create hash databases using the global settings panel.
Then they select which hash databases to use for a particular job using the
ingest job settings panel. When a module instance runs, it gets the relevant
databases from the hash databases manager.
An ingest module factory is responsible for persisting global settings and may use the module
settings methods provided by org.sleuthkit.autopsy.ingest.IngestServices for
saving simple properties, or the facilities of classes such as
org.sleuthkit.autopsy.coreutils.PlatformUtil and org.sleuthkit.autopsy.coreutils.XMLUtil
for more sophisticated approaches.
To be discovered at runtime by the ingest framework, IngestModuleFactory
implementations must be marked with the following NetBeans Service provider
annotation:
\code
@ServiceProvider(service = IngestModuleFactory.class)
\endcode
The following Java package import is required for the ServiceProvider annotation:
\code
import org.openide.util.lookup.ServiceProvider
\endcode
To use this import, you will also need to add a dependency on the NetBeans Lookup
API module to the NetBeans module that contains your ingest module.
Compared to the DataSourceIngestModule and FileIngestModule interfaces, the
IngestModuleFactory is richer, but also more complex. For your convenience, an
ingest module factory that does not require a full-implementation of all of the
factory features may extend the abstract
org.sleuthkit.autopsy.ingest.IngestModuleFactoryAdapter class to get default
"do nothing" implementations of most of the methods in the IngestModuleFactory
interface. If you do need to implement the full interface, use the documentation
for the following classes as a guide:
- org.sleuthkit.autopsy.ingest.IngestModuleFactory
- org.sleuthkit.autopsy.ingest.IngestModuleGlobalSetttingsPanel
- org.sleuthkit.autopsy.ingest.IngestModuleIngestJobSettings
- org.sleuthkit.autopsy.ingest.IngestModuleIngestJobSettingsPanel
You can also refer to sample implementations of the interfaces and abstract
classes in the org.sleuthkit.autopsy.examples package, although you should note
that the samples do not do anything particularly useful.
\section ingest_modules_pipeline_configuration Controlling the Ordering of Ingest Modules in Ingest Pipelines
By default, ingest modules that are not part of the standard Autopsy
installation will run after the core ingest modules. No order is implied. This
will likely change in the future, but currently manual configuration is needed
to enforce sequencing of ingest modules.
There is an ingest pipeline configuration XML file that specifies the order for
running the core ingest modules. If you need to insert your ingest modules in
the sequence of core modules or control the ordering of non-core modules, you
must edit this file by hand. You will find it in the config directory of your
Autopsy installation, typically something like "C:\Users\yourUserName\AppData\Roaming\.autopsy\dev\config\pipeline_config.xml"
on a Microsoft Windows platform. Check the Userdir listed in the Autopsy About
dialog.
Autopsy will provide tools for reconfiguring the ingest pipeline in the near
future. Until that time, there is no guarantee that the schema of this file will
remain fixed and that it will not be overwritten when upgrading your Autopsy
installation.
\section ingest_modules_api_migration Migrating Ingest Modules to the Current API
Previous versions of ingest modules needed to be implemented as singletons that
extended either the abstract class IngestModuleDataSource or the abstract class
IngestModuleAbstractFile, both of which extended the abstract class
IngestModuleAbstract. With the current ingest module API, ingest modules are no
longer singletons and the creation and configuration of module instances has
been separated from their execution. As discussed in the previous sections of
this page, an ingest module implements one of two interfaces:
- org.sleuthkit.autopsy.ingest.DataSourceIngestModule
- org.sleuthkit.autopsy.ingest.FileIngestModule
Both of these interfaces extend org.sleuthkit.autopsy.ingest.IngestModule.
The ingest module developer must also provide a factory for his or her modules.
The factory must implement the following interface:
- org.sleuthkit.autopsy.ingest.IngestModuleFactory
The following tables provide a mapping of the methods of the old abstract classes to
the new interfaces:
Old method | New Method |
---------- | ---------- |
IngestModuleDataSource.process() | DataSourceIngestModule.process() |
IngestModuleAbstractFile.process | FileIngestModule.process() |
IngestModuleAbstract.getType() | N/A |
IngestModuleAbstract.init() | IngestModule.startUp() |
IngestModuleAbstract.getName() | IngestModuleFactory.getModuleName() |
IngestModuleAbstract.getDescription() | IngestModuleFactory.getModuleDescription() |
IngestModuleAbstract.getVersion() | IngestModuleFactory.getModuleVersion() |
IngestModuleAbstract.hasBackgroundJobsRunning | N/A |
IngestModuleAbstract.complete() | IngestModule.shutDown() |
IngestModuleAbstract.hasAdvancedConfiguration() | IngestModuleFactory.hasGlobalSettingsPanel() |
IngestModuleAbstract.getAdvancedConfiguration() | IngestModuleFactory.getGlobalSettingsPanel() |
IngestModuleAbstract.saveAdvancedConfiguration() | IngestModuleGlobalSetttingsPanel.saveSettings() |
N/A | IngestModuleFactory.getDefaultIngestJobSettings() |
IngestModuleAbstract.hasSimpleConfiguration() | IngestModuleFactory.hasIngestJobSettingsPanel() |
IngestModuleAbstract.getSimpleConfiguration() | IngestModuleFactory.getIngestJobSettingsPanel() |
IngestModuleAbstract.saveSimpleConfiguration() | N/A |
N/A | IngestModuleIngestJobSettingsPanel.getSettings() |
N/A | IngestModuleFactory.isDataSourceIngestModuleFactory() |
N/A | IngestModuleFactory.createDataSourceIngestModule() |
N/A | IngestModuleFactory.isFileIngestModuleFactory() |
N/A | IngestModuleFactory.createFileIngestModule() |
Notes:
- IngestModuleFactory.getModuleName() should delegate to a static class method
that can also be called by ingest module instances.
- Autopsy passes a flag to IngestModule.shutDown() indicating whether the ingest
job completed or was cancelled.
- The global settings panel (formerly "advanced") for a module must implement
IngestModuleGlobalSettingsPanel which extends JPanel. Global settings are those
that affect all modules, regardless of ingest job and pipeline.
- The per ingest job settings panel (formerly "simple") for a module must implement
IngestModuleIngestJobSettingsPanel which extends JPanel. It takes the settings
for the current context as a serializable IngestModuleIngestJobSettings object
and its getSettings() methods returns a serializable IngestModuleIngestJobSettings object.
The IngestModuleIngestJobSettingsPanel.getSettings() method replaces the saveSimpleSettings() method,
except that now Autopsy persists the settings in a context-sensitive fashion.
- The IngestModuleFactory creation methods replace the getInstance() methods of
the former singletons and receive a IngestModuleIngestJobSettings object that should be
passed to the constructors of the module instances the factory creates.
*/
Reference in New Issue View Git Blame Copy Permalink