Docs update

Ingest/src/org/sleuthkit/autopsy/ingest/example/ExampleAbstractFileIngestService.java

@@ -39,6 +39,12 @@ public class ExampleAbstractFileIngestService implements IngestServiceAbstractFi
     private IngestManagerProxy managerProxy;
     private static int messageId = 0;
 
+    //file ingest services require a private constructor
+    //to ensure singleton instances
+    private ExampleAbstractFileIngestService() {
+        
+    }
+    
     public static synchronized ExampleAbstractFileIngestService getDefault() {
         if (instance == null) {
             instance = new ExampleAbstractFileIngestService();
@@ -62,7 +68,7 @@ public class ExampleAbstractFileIngestService implements IngestServiceAbstractFi
     @Override
     public void complete() {
         logger.log(Level.INFO, "complete()");
-        managerProxy.postMessage(IngestMessage.createMessage(++messageId, MessageType.INFO, this, "COMPLETE"));
+        managerProxy.postMessage(IngestMessage.createMessage(++messageId, MessageType.INFO, this, "Complete"));
 
         //service specific cleanup due completion here
     }
@@ -90,6 +96,7 @@ public class ExampleAbstractFileIngestService implements IngestServiceAbstractFi
     @Override
     public void stop() {
         logger.log(Level.INFO, "stop()");
+        managerProxy.postMessage(IngestMessage.createMessage(++messageId, MessageType.INFO, this, "Stopped"));
 
         //service specific cleanup due interruption here
     }

Ingest/src/org/sleuthkit/autopsy/ingest/example/ExampleImageIngestService.java

@@ -91,7 +91,7 @@ public final class ExampleImageIngestService implements IngestServiceImage {
     public void complete() {
         logger.log(Level.INFO, "complete() " + this.toString());
 
-        final IngestMessage msg = IngestMessage.createMessage(++messageId, MessageType.INFO, this, "completed image processing");
+        final IngestMessage msg = IngestMessage.createMessage(++messageId, MessageType.INFO, this, "Complete");
         managerProxy.postMessage(msg);
 
         //service specific cleanup due to completion here
@@ -119,6 +119,7 @@ public final class ExampleImageIngestService implements IngestServiceImage {
     @Override
     public void stop() {
         logger.log(Level.INFO, "stop()");
+        managerProxy.postMessage(IngestMessage.createMessage(++messageId, MessageType.INFO, this, "Stopped"));
 
         //service specific cleanup due to interruption here
     }

Ingest/src/org/sleuthkit/autopsy/ingest/package.dox

@@ -13,8 +13,8 @@
  * For instance, processing of known files can be skipped by subsequent modules in the pipeline (if chosen so), for performance reasons.
  * 
  * The framework provides interfaces every ingest module needs to implement:
- *  - org.sleuthkit.autopsy.ingest.IngestServiceImage (for modules that are interested in the image as a whole, or query and analyze specific data from the image) 
- *  - org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile (for modules that should process every file).
+ * - org.sleuthkit.autopsy.ingest.IngestServiceImage (for modules that are interested in the image as a whole, or modules that selectively pick and analyze data from the image) 
+ * - org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile (for modules that should process every file in the image).
  * 
  * The interfaces define methods to initialize, process passed in data, configure the ingest service, 
  * query the service state and finalize the service.
@@ -26,18 +26,12 @@
  * - additional classes to support threading, sending messages, ingest monitoring, ingest cancellation, progress bars,
  * - a user interface component (Ingest Inbox) used to display interesting messages posted by ingest modules to the user,
  * 
- * Most ingest modules typically require configuration before they are executed.
- * The configuration methods are defined in the ingest modules interfaces.  
- * Module configuration is decentralized and module-specific; every module maintains its
- * own configuration state and is responsible for implementing its own JPanels to render
- * and present the configuration to the user.  There are method hooks defined in the ingest service interface 
- * that are used to hint the module when the configuration should be saved internally by the module.
  * 
  * Ingest modules run in background threads. There is a single background thread for file-level ingest modules, within which every file ingest module runs series for every file.
  * Image ingest modules run each in their own thread and thus can run in parallel (TODO we will change this in the future for performance reasons, and support image ingest module dependencies).  
  * Every ingest thread is presented with a progress bar and can be cancelled by a user, or by the framework, in case of a critical event (such as Autopsy is terminating, or a system error).
  *
- * Ingest module can also implement its own internal threads for any special-purpose processing that can occur in parallel.
+ * Ingest module can maintain internal threads for any special processing that can occur in parallel.
  * However, the module is then responsible for creating, managing and tearing down the internal threads and to implement locking to protect critical sections internal to the module.
  * An example of a module that maintains its own threads is the KeywordSearch module.
  * 

docs/doxygen/ingest.dox

@@ -11,21 +11,115 @@ This document outlines steps necessary to implement a functional ingest module.
 
 \subsection ingest_interface Interfaces
 
-\subsection ingest_interface Required methods
+Implement one of the interfaces:
+
+- org.sleuthkit.autopsy.ingest.IngestServiceImage (for modules that are interested in the image as a whole, or modules that selectively pick and analyze data from the image) 
+- org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile (for modules that should process every file in the image).
+
+Another difference between the two is that org.sleuthkit.autopsy.ingest.IngestServiceImage services run each in their own threads, and can run in parallel to other image services.
+File services execute within the same worker thread and they run in series, every service is called for every file.
+
+For this reason, org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile services are singleton instances
+and org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile are not singletons -- there could be multiple instances of 
+an image based service, because multiple images can be analyzed at the same time by multiple instances of the same image service class
+(NOTE: this design might change in the future to limit number of image services executing at the same time and to introduce a better service dependency system).
+
+org.sleuthkit.autopsy.ingest.IngestServiceAbstract defines common methods for both types of services, please refer to API documentation for details on the required methods.
+
+\subsection ingest_interface Implementation details.
+
+org.sleuthkit.autopsy.ingest.example package for sample service skeleton code.
+
+There is a static getDefault() method that are not part of the interface, that every module (whether an image or a file service)
+needs to implement to return the registered static instance of the service.
+Refer to example code in org.sleuthkit.autopsy.ingest.example.ExampleAbstractFileIngestService.getDefault()
+
+In case of file ingest service, there needs to be a private constructor to ensure only and only (singleton) instance.
+Ensure the default public file service constructor is overridden with the private one.
+
+In case of the image ingest service, the constructor should be public.
+
+The most important method is the process() method, which takes either a file or an image as an argument (depending on the type of the service).
+A service either performs work in the process method, 
+or enqueues the work for later processing (more common if the service manages its own internal threads).
+
+init() method is invoked on a service (by ingest manager) every time ingest starts, and the service is not already running.
+A service should support multiple invocations of init() throughout the application life-cycle.
+
+complete() method is invoked on a service when ingest completed.  The service should perform any resource (file, handles, caches)
+cleanup here.
+
+stop() method is invoked on a service when ingest is interrupted (by the user or by the system).
+The method is similar to complete(), in that the service should perform any cleanup work.
+The difference is if there is pending data to be processed or pending results to be reported by the service; 
+the results should be rejected and ignored if stop() is invoked and the service should terminate as early as possible.
+
+Services should post inbox messages to the user when stop() or complete() is invoked (refer to the examples).
+
+Every service should support multiple init() - process() - complete(), or init() - process() - stop() invocations.
+The services should also support multiple init() - complete() and init() - stop() invocations, 
+which can occur if ingest is started but no work is enqueued for the particular service.
+
+
+Module developers are encouraged to use the standard java.util.logging.Logger infrastructure to log errors to the Autopsy log.
+
 
 \subsection ingest_registration Service Registration
 
-To implement an ingest module it is required to implement one of the interfaces (for file or image ingest) 
-and to have the module register itself using Netbeans Lookup infrastructure in layer.xml file.
+Ingest service class / module should register itself using Netbeans Lookup infrastructure 
+in layer.xml file in the same package where the ingest module is located.
+
+Example image ingest service registration:
+
+<file name="org-sleuthkit-autopsy-ingest-example-ExampleImageIngestService.instance">
+<attr name="instanceOf" stringvalue="org.sleuthkit.autopsy.ingest.IngestServiceImage"/>
+<attr name="instanceCreate" methodvalue="org.sleuthkit.autopsy.ingest.example.ExampleImageIngestService.getDefault"/>
+<attr name="position" intvalue="1000"/>
+</file>
+      
+File image ingest service registration:
+  
+<file name="org-sleuthkit-autopsy-ingest-example-ExampleAbstractFileIngestService.instance">
+<attr name="instanceOf" stringvalue="org.sleuthkit.autopsy.ingest.IngestServiceAbstractFile"/>
+<attr name="instanceCreate" methodvalue="org.sleuthkit.autopsy.ingest.example.ExampleAbstractFileIngestService.getDefault"/>
+<attr name="position" intvalue="1100"/>
+</file>
+
+Note the "position" attribute.  The attribute determines the ordering of the module in the ingest pipeline.
+Services with lower position attribute will execute first.
+Use high numbers (higher than 1000) or non-core services.
+If your module depends on results from another module, use a higher position attribute to enforce the dependency.
+Note: we plan to implement a more flexible and robust module dependency system in future versions of the Autopsy ingest framework.
 
 \subsection ingest_configuration Service Configuration
 
+Ingest modules typically require configuration before they are executed.
+There are 2 levels of configuration: simple and advanced.
+
+
+Module configuration is decentralized and module-specific; every module maintains its
+own configuration state and is responsible for implementing its own JPanels to render
+and present the configuration to the user. 
+
+The configuration methods are defined in the ingest modules interfaces.
+For example, to implement simple configuration, module should return true in:
+
+org.sleuthkit.autopsy.ingest.IngestServiceAbstract.hasSimpleConfiguration()
+
+org.sleuthkit.autopsy.ingest.IngestServiceAbstract.getSimpleConfiguration()
+should then return javax.swing.JPanel instance.
+ 
+There are method hooks defined in the ingest service interface that are used to hint the module when the configuration should be saved internally by the module.
+For example, to save the simple configuration state, the module should implement
+org.sleuthkit.autopsy.ingest.IngestServiceAbstract.saveSimpleConfiguration()
+
+
 \subsection ingest_events Sending Service Events
 
-Service should notify listeners of new data available periodically by invoking fireServiceDataEvent() method in org.sleuthkit.autopsy.ingest.IngestManagerProxy class.
+Service should notify listeners of new data available periodically by invoking org.sleuthkit.autopsy.ingest.IngestManagerProxy.fireServiceDataEvent() method.
 The method accepts org.sleuthkit.autopsy.ingest.ServiceDataEvent parameter.  
-The artifacts passed in a single event should be of the same type, which is enforced by the org.sleuthkit.autopsy.ingest.ServiceDataEvent constructor.
-
+The artifacts passed in a single event should be of the same type, 
+which is enforced by the org.sleuthkit.autopsy.ingest.ServiceDataEvent constructor.
 
 \subsection ingest_events Data Posting Intervals