mstoeck3/autopsy-flatpak
1
0
Fork 0
mirror of https://github.com/overcuriousity/autopsy-flatpak.git synced 2025-07-06 21:00:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #6391 from eugene7646/ZkNodeMigration_6706

Browse Source 
ZK node migration 6706
This commit is contained in:
Richard Cordovano 2020-10-19 13:29:00 -04:00 committed by GitHub
commit 697379ba8e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 3247 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
ZookeeperNodeMigration/.gitignore vendored Executable file
View File

@ -0,0 +1,3 @@
/nbproject/private/
/build/
/dist/

73
ZookeeperNodeMigration/build.xml Executable file
View File

@ -0,0 +1,73 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- You may freely edit this file. See commented blocks below for -->
<!-- some examples of how to customize the build. -->
<!-- (If you delete it and reopen the project it will be recreated.) -->
<!-- By default, only the Clean and Build commands use this build script. -->
<!-- Commands such as Run, Debug, and Test only use this build script if -->
<!-- the Compile on Save feature is turned off for the project. -->
<!-- You can turn off the Compile on Save (or Deploy on Save) setting -->
<!-- in the project's Project Properties dialog box.-->
<project name="ZookeeperNodeMigration" default="default" basedir=".">
<description>Builds, tests, and runs the project ZookeeperNodeMigration.</description>
<import file="nbproject/build-impl.xml"/>
<!--
There exist several targets which are by default empty and which can be
used for execution of your tasks. These targets are usually executed
before and after some main targets. They are:
-pre-init: called before initialization of project properties
-post-init: called after initialization of project properties
-pre-compile: called before javac compilation
-post-compile: called after javac compilation
-pre-compile-single: called before javac compilation of single file
-post-compile-single: called after javac compilation of single file
-pre-compile-test: called before javac compilation of JUnit tests
-post-compile-test: called after javac compilation of JUnit tests
-pre-compile-test-single: called before javac compilation of single JUnit test
-post-compile-test-single: called after javac compilation of single JUunit test
-pre-jar: called before JAR building
-post-jar: called after JAR building
-post-clean: called after cleaning build products
(Targets beginning with '-' are not intended to be called on their own.)
Example of inserting an obfuscator after compilation could look like this:
<target name="-post-compile">
<obfuscate>
<fileset dir="${build.classes.dir}"/>
</obfuscate>
</target>
For list of available properties check the imported
nbproject/build-impl.xml file.
Another way to customize the build is by overriding existing main targets.
The targets of interest are:
-init-macrodef-javac: defines macro for javac compilation
-init-macrodef-junit: defines macro for junit execution
-init-macrodef-debug: defines macro for class debugging
-init-macrodef-java: defines macro for class execution
-do-jar: JAR building
run: execution of project
-javadoc-build: Javadoc generation
test-report: JUnit report generation
An example of overriding the target for project execution could look like this:
<target name="run" depends="ZookeeperNodeMigration-impl.jar">
<exec dir="bin" executable="launcher.exe">
<arg file="${dist.jar}"/>
</exec>
</target>
Notice that the overridden target depends on the jar target and not only on
the compile target as the regular run target does. Again, for a list of available
properties which you can use, check the target you are overriding in the
nbproject/build-impl.xml file.
-->
</project>

23
ZookeeperNodeMigration/docs/README.TXT Executable file
View File

@ -0,0 +1,23 @@
To run the project from the command line, go to the folder that contains "ZookeeperNodeMigration.jar" and
type the following:
java -jar ZookeeperNodeMigration.jar
To distribute this project, zip up the dist folder (including the lib folder)
and distribute the ZIP file.
Usage:
ZookeeperNodeMigration input needs to be:
[Input Zookeeper IP Address or Hostname] [Input Zookeeper Port Number] [Output Zookeeper IP Address or Hostname] [Output Zookeeper Port Number]
For example, if you execute the following command from command line line, the Zookeeper
nodes will get copied from Zookeeper server on localhost:9983 to Zookeeper server on localhost:19983 :
java -jar ZookeeperNodeMigration.jar localhost 9983 localhost 19983
If you do not have Java installed on the machine, you can use the packaged version of Java that is distributed along
with Autopsy. For example:
"C:\Program Files\Autopsy-4.16.0\jre\bin\java.exe" -jar ZookeeperNodeMigration.jar localhost 9983 localhost 19983

3
ZookeeperNodeMigration/manifest.mf Executable file
View File

@ -0,0 +1,3 @@
Manifest-Version: 1.0
X-COMMENT: Main-Class will be added automatically by build

1770
ZookeeperNodeMigration/nbproject/build-impl.xml Executable file
View File

File diff suppressed because it is too large Load Diff

107
ZookeeperNodeMigration/nbproject/project.properties Executable file
View File

@ -0,0 +1,107 @@
annotation.processing.enabled=true
annotation.processing.enabled.in.editor=false
annotation.processing.processors.list=
annotation.processing.run.all.processors=true
annotation.processing.source.output=${build.generated.sources.dir}/ap-source-output
application.title=ZookeeperNodeMigration
application.vendor=elivis
build.classes.dir=${build.dir}/classes
build.classes.excludes=**/*.java,**/*.form
# This directory is removed when the project is cleaned:
build.dir=build
build.generated.dir=${build.dir}/generated
build.generated.sources.dir=${build.dir}/generated-sources
# Only compile against the classpath explicitly listed here:
build.sysclasspath=ignore
build.test.classes.dir=${build.dir}/test/classes
build.test.results.dir=${build.dir}/test/results
# Uncomment to specify the preferred debugger connection transport:
#debug.transport=dt_socket
debug.classpath=\
${run.classpath}
debug.modulepath=\
${run.modulepath}
debug.test.classpath=\
${run.test.classpath}
debug.test.modulepath=\
${run.test.modulepath}
# Files in build.classes.dir which should be excluded from distribution jar
dist.archive.excludes=
# This directory is removed when the project is cleaned:
dist.dir=dist
dist.jar=${dist.dir}/ZookeeperNodeMigration.jar
dist.javadoc.dir=${dist.dir}/javadoc
endorsed.classpath=
excludes=
file.reference.curator-client-2.8.0.jar=release/curator-client-2.8.0.jar
file.reference.curator-framework-2.8.0.jar=release/curator-framework-2.8.0.jar
file.reference.curator-recipes-2.8.0.jar=release/curator-recipes-2.8.0.jar
file.reference.guava-17.0.jar=release/guava-17.0.jar
file.reference.log4j-1.2.17.jar=release/log4j-1.2.17.jar
file.reference.slf4j-api-1.7.24.jar=release/slf4j-api-1.7.24.jar
file.reference.slf4j-log4j12-1.7.6.jar=release/slf4j-log4j12-1.7.6.jar
file.reference.zookeeper-3.4.6.jar=release/zookeeper-3.4.6.jar
includes=**
jar.compress=false
javac.classpath=\
${file.reference.curator-client-2.8.0.jar}:\
${file.reference.curator-framework-2.8.0.jar}:\
${file.reference.curator-recipes-2.8.0.jar}:\
${file.reference.zookeeper-3.4.6.jar}:\
${file.reference.slf4j-api-1.7.24.jar}:\
${file.reference.slf4j-log4j12-1.7.6.jar}:\
${file.reference.log4j-1.2.17.jar}:\
${file.reference.guava-17.0.jar}
# Space-separated list of extra javac options
javac.compilerargs=
javac.deprecation=false
javac.external.vm=true
javac.modulepath=
javac.processormodulepath=
javac.processorpath=\
${javac.classpath}
javac.source=1.8
javac.target=1.8
javac.test.classpath=\
${javac.classpath}:\
${build.classes.dir}
javac.test.modulepath=\
${javac.modulepath}
javac.test.processorpath=\
${javac.test.classpath}
javadoc.additionalparam=
javadoc.author=false
javadoc.encoding=${source.encoding}
javadoc.html5=false
javadoc.noindex=false
javadoc.nonavbar=false
javadoc.notree=false
javadoc.private=false
javadoc.splitindex=true
javadoc.use=true
javadoc.version=false
javadoc.windowtitle=
jlink.launcher=false
jlink.launcher.name=ZookeeperNodeMigration
main.class=zookeepernodemigration.ZookeeperNodeMigration
manifest.file=manifest.mf
meta.inf.dir=${src.dir}/META-INF
mkdist.disabled=false
platform.active=default_platform
run.classpath=\
${javac.classpath}:\
${build.classes.dir}
# Space-separated list of JVM arguments used when running the project.
# You may also define separate properties like run-sys-prop.name=value instead of -Dname=value.
# To set system properties for unit tests define test-sys-prop.name=value:
run.jvmargs=
run.modulepath=\
${javac.modulepath}
run.test.classpath=\
${javac.test.classpath}:\
${build.test.classes.dir}
run.test.modulepath=\
${javac.test.modulepath}
source.encoding=UTF-8
src.dir=src
test.src.dir=test

15
ZookeeperNodeMigration/nbproject/project.xml Executable file
View File

@ -0,0 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://www.netbeans.org/ns/project/1">
<type>org.netbeans.modules.java.j2seproject</type>
<configuration>
<data xmlns="http://www.netbeans.org/ns/j2se-project/3">
<name>ZookeeperNodeMigration</name>
<source-roots>
<root id="src.dir"/>
</source-roots>
<test-roots>
<root id="test.src.dir"/>
</test-roots>
</data>
</configuration>
</project>

BIN
ZookeeperNodeMigration/release/curator-client-2.8.0.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/curator-framework-2.8.0.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/curator-recipes-2.8.0.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/guava-17.0.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/log4j-1.2.17.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/slf4j-api-1.7.24.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/slf4j-log4j12-1.7.6.jar Executable file
View File

Binary file not shown.

BIN
ZookeeperNodeMigration/release/zookeeper-3.4.6.jar Executable file
View File

Binary file not shown.

687
ZookeeperNodeMigration/src/zookeepernodemigration/AutoIngestJobNodeData.java Executable file
View File

@ -0,0 +1,687 @@
/*
* Autopsy Forensic Browser
*
* Copyright 2020 Basis Technology Corp.
* Contact: carrier <at> sleuthkit <dot> org
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package zookeepernodemigration;
import java.io.Serializable;
import java.nio.BufferUnderflowException;
import java.nio.ByteBuffer;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Date;
import javax.lang.model.type.TypeKind;
/**
* An object that converts auto ingest job data for an auto ingest job
* coordination service node to and from byte arrays.
*/
final class AutoIngestJobNodeData {
private static final int CURRENT_VERSION = 2;
private static final int DEFAULT_PRIORITY = 0;
/*
* This number is the sum of each piece of data, based on it's type. For the
* types boolean, int, and long, values 1, 4, and 8 will be added
* respectively. For String objects, the length of the string, plus either a
* byte or short respesenting the length of the string, will be added.
*
* This field is used to set the size of the buffer during the byte array
* creation in the 'toArray()' method. Since the final size of the array
* isn't immediately known at the time of creation, this number is used to
* create an array as large as could possibly be needed to store all the
* data. This avoids the need to continuously enlarge the buffer. Once the
* buffer has all the necessary data, it will be resized as appropriate.
*/
private static final int MAX_POSSIBLE_NODE_DATA_SIZE = 131637;
/*
* Version 0 fields.
*/
private int processingStatus;
private int priority;
private int numberOfCrashes;
private long completedDate;
private boolean errorsOccurred;
/*
* Version 1 fields.
*/
private int version;
private String manifestFilePath; // 'short' length used in byte array
private long manifestFileDate;
private String caseName; // 'byte' length used in byte array
private String deviceId; // 'byte' length used in byte array
private String dataSourcePath; // 'short' length used in byte array
private String caseDirectoryPath; // 'short' length used in byte array
private String processingHostName; // 'short' length used in byte array
private byte processingStage;
private long processingStageStartDate;
private String processingStageDetailsDescription; // 'byte' length used in byte array
private long processingStageDetailsStartDate;
/*
* Version 2 fields.
*/
private long dataSourceSize;
/**
* Processing statuses for an auto ingest job.
*/
enum ProcessingStatus {
PENDING,
PROCESSING,
COMPLETED,
DELETED
}
/**
* Processing stages for an auto ingest job.
*/
enum Stage {
PENDING("Pending"),
STARTING("Starting"),
UPDATING_SHARED_CONFIG("Updating shared configuration"),
CHECKING_SERVICES("Checking services"),
OPENING_CASE("Opening case"),
IDENTIFYING_DATA_SOURCE("Identifying data source type"),
ADDING_DATA_SOURCE("Adding data source"),
ANALYZING_DATA_SOURCE("Analyzing data source"),
ANALYZING_FILES("Analyzing files"),
EXPORTING_FILES("Exporting files"),
CANCELLING_MODULE("Cancelling module"),
CANCELLING("Cancelling"),
COMPLETED("Completed");
private final String displayText;
private Stage(String displayText) {
this.displayText = displayText;
}
String getDisplayText() {
return displayText;
}
}
/**
* Processing stage details for an auto ingest job.
*/
static final class StageDetails implements Serializable {
private static final long serialVersionUID = 1L;
private final String description;
private final Date startDate;
StageDetails(String description, Date startDate) {
this.description = description;
this.startDate = startDate;
}
String getDescription() {
return this.description;
}
Date getStartDate() {
return new Date(this.startDate.getTime());
}
}
/**
* Gets the current version of the auto ingest job coordination service node
* data.
*
* @return The version number.
*/
static int getCurrentVersion() {
return AutoIngestJobNodeData.CURRENT_VERSION;
}
/**
* Uses a coordination service node data to construct an object that
* converts auto ingest job data for an auto ingest job coordination service
* node to and from byte arrays.
*
* @param nodeData The raw bytes received from the coordination service.
*/
AutoIngestJobNodeData(byte[] nodeData) throws InvalidDataException {
if (null == nodeData || nodeData.length == 0) {
throw new InvalidDataException(null == nodeData ? "Null nodeData byte array" : "Zero-length nodeData byte array");
}
/*
* Set default values for all fields.
*/
this.processingStatus = ProcessingStatus.PENDING.ordinal();
this.priority = DEFAULT_PRIORITY;
this.numberOfCrashes = 0;
this.completedDate = 0L;
this.errorsOccurred = false;
this.version = 0;
this.manifestFilePath = "";
this.manifestFileDate = 0L;
this.caseName = "";
this.deviceId = "";
this.dataSourcePath = "";
this.caseDirectoryPath = "";
this.processingHostName = "";
this.processingStage = (byte) Stage.PENDING.ordinal();
this.processingStageStartDate = 0L;
this.processingStageDetailsDescription = "";
this.processingStageDetailsStartDate = 0L;
this.dataSourceSize = 0L;
/*
* Get fields from node data.
*/
ByteBuffer buffer = ByteBuffer.wrap(nodeData);
try {
if (buffer.hasRemaining()) {
/*
* Get version 0 fields.
*/
this.processingStatus = buffer.getInt();
this.priority = buffer.getInt();
this.numberOfCrashes = buffer.getInt();
this.completedDate = buffer.getLong();
int errorFlag = buffer.getInt();
this.errorsOccurred = (1 == errorFlag);
}
if (buffer.hasRemaining()) {
/*
* Get version 1 fields.
*/
this.version = buffer.getInt();
this.deviceId = getStringFromBuffer(buffer, TypeKind.BYTE);
this.caseName = getStringFromBuffer(buffer, TypeKind.BYTE);
this.caseDirectoryPath = getStringFromBuffer(buffer, TypeKind.SHORT);
this.manifestFileDate = buffer.getLong();
this.manifestFilePath = getStringFromBuffer(buffer, TypeKind.SHORT);
this.dataSourcePath = getStringFromBuffer(buffer, TypeKind.SHORT);
this.processingStage = buffer.get();
this.processingStageStartDate = buffer.getLong();
this.processingStageDetailsDescription = getStringFromBuffer(buffer, TypeKind.BYTE);
this.processingStageDetailsStartDate = buffer.getLong();
this.processingHostName = getStringFromBuffer(buffer, TypeKind.SHORT);
}
if (buffer.hasRemaining()) {
/*
* Get version 2 fields.
*/
this.dataSourceSize = buffer.getLong();
}
} catch (BufferUnderflowException ex) {
throw new InvalidDataException("Node data is incomplete", ex);
}
}
/**
* Gets the processing status of the job.
*
* @return The processing status.
*/
ProcessingStatus getProcessingStatus() {
return ProcessingStatus.values()[this.processingStatus];
}
/**
* Sets the processing status of the job.
*
* @param processingSatus The processing status.
*/
void setProcessingStatus(ProcessingStatus processingStatus) {
this.processingStatus = processingStatus.ordinal();
}
/**
* Gets the priority of the job.
*
* @return The priority.
*/
int getPriority() {
return this.priority;
}
/**
* Sets the priority of the job. A higher number indicates a higheer
* priority.
*
* @param priority The priority.
*/
void setPriority(int priority) {
this.priority = priority;
}
/**
* Gets the number of times the job has crashed during processing.
*
* @return The number of crashes.
*/
int getNumberOfCrashes() {
return this.numberOfCrashes;
}
/**
* Sets the number of times the job has crashed during processing.
*
* @param numberOfCrashes The number of crashes.
*/
void setNumberOfCrashes(int numberOfCrashes) {
this.numberOfCrashes = numberOfCrashes;
}
/**
* Gets the date the job was completed. A completion date equal to the epoch
* (January 1, 1970, 00:00:00 GMT), i.e., Date.getTime() returns 0L,
* indicates the job has not been completed.
*
* @return The job completion date.
*/
Date getCompletedDate() {
return new Date(this.completedDate);
}
/**
* Sets the date the job was completed. A completion date equal to the epoch
* (January 1, 1970, 00:00:00 GMT), i.e., Date.getTime() returns 0L,
* indicates the job has not been completed.
*
* @param completedDate The job completion date.
*/
void setCompletedDate(Date completedDate) {
this.completedDate = completedDate.getTime();
}
/**
* Gets whether or not any errors occurred during the processing of the job.
*
* @return True or false.
*/
boolean getErrorsOccurred() {
return this.errorsOccurred;
}
/**
* Sets whether or not any errors occurred during the processing of job.
*
* @param errorsOccurred True or false.
*/
void setErrorsOccurred(boolean errorsOccurred) {
this.errorsOccurred = errorsOccurred;
}
/**
* Gets the node data version number.
*
* @return The version number.
*/
int getVersion() {
return this.version;
}
/**
* Gets the device ID of the device associated with the data source for the
* job.
*
* @return The device ID.
*/
String getDeviceId() {
return this.deviceId;
}
/**
* Sets the device ID of the device associated with the data source for the
* job.
*
* @param deviceId The device ID.
*/
void setDeviceId(String deviceId) {
this.deviceId = deviceId;
}
/**
* Gets the case name.
*
* @return The case name.
*/
String getCaseName() {
return this.caseName;
}
/**
* Sets the case name.
*
* @param caseName The case name.
*/
void setCaseName(String caseName) {
this.caseName = caseName;
}
/**
* Sets the path to the case directory of the case associated with the job.
*
* @param caseDirectoryPath The path to the case directory.
*/
synchronized void setCaseDirectoryPath(Path caseDirectoryPath) {
if (caseDirectoryPath == null) {
this.caseDirectoryPath = "";
} else {
this.caseDirectoryPath = caseDirectoryPath.toString();
}
}
/**
* Gets the path to the case directory of the case associated with the job.
*
* @return The case directory path or an empty string path if the case
* directory has not been created yet.
*/
synchronized Path getCaseDirectoryPath() {
if (!caseDirectoryPath.isEmpty()) {
return Paths.get(caseDirectoryPath);
} else {
return Paths.get("");
}
}
/**
* Gets the date the manifest was created.
*
* @return The date the manifest was created.
*/
Date getManifestFileDate() {
return new Date(this.manifestFileDate);
}
/**
* Sets the date the manifest was created.
*
* @param manifestFileDate The date the manifest was created.
*/
void setManifestFileDate(Date manifestFileDate) {
this.manifestFileDate = manifestFileDate.getTime();
}
/**
* Gets the manifest file path.
*
* @return The manifest file path.
*/
Path getManifestFilePath() {
return Paths.get(this.manifestFilePath);
}
/**
* Sets the manifest file path.
*
* @param manifestFilePath The manifest file path.
*/
void setManifestFilePath(Path manifestFilePath) {
if (manifestFilePath != null) {
this.manifestFilePath = manifestFilePath.toString();
} else {
this.manifestFilePath = "";
}
}
/**
* Gets the path of the data source for the job.
*
* @return The data source path.
*/
Path getDataSourcePath() {
return Paths.get(dataSourcePath);
}
/**
* Get the file name portion of the path of the data source for the job.
*
* @return The data source file name.
*/
public String getDataSourceFileName() {
return Paths.get(dataSourcePath).getFileName().toString();
}
/**
* Sets the path of the data source for the job.
*
* @param dataSourcePath The data source path.
*/
void setDataSourcePath(Path dataSourcePath) {
if (dataSourcePath != null) {
this.dataSourcePath = dataSourcePath.toString();
} else {
this.dataSourcePath = "";
}
}
/**
* Get the processing stage of the job.
*
* @return The processing stage.
*/
Stage getProcessingStage() {
return Stage.values()[this.processingStage];
}
/**
* Sets the processing stage job.
*
* @param processingStage The processing stage.
*/
void setProcessingStage(Stage processingStage) {
this.processingStage = (byte) processingStage.ordinal();
}
/**
* Gets the processing stage start date.
*
* @return The processing stage start date.
*/
Date getProcessingStageStartDate() {
return new Date(this.processingStageStartDate);
}
/**
* Sets the processing stage start date.
*
* @param processingStageStartDate The processing stage start date.
*/
void setProcessingStageStartDate(Date processingStageStartDate) {
this.processingStageStartDate = processingStageStartDate.getTime();
}
/**
* Get the processing stage details.
*
* @return A processing stage details object.
*/
StageDetails getProcessingStageDetails() {
return new StageDetails(this.processingStageDetailsDescription, new Date(this.processingStageDetailsStartDate));
}
/**
* Sets the details of the current processing stage.
*
* @param stageDetails A stage details object.
*/
void setProcessingStageDetails(StageDetails stageDetails) {
this.processingStageDetailsDescription = stageDetails.getDescription();
this.processingStageDetailsStartDate = stageDetails.getStartDate().getTime();
}
/**
* Gets the processing host name, may be the empty string.
*
* @return The processing host. The empty string if the job is not currently
* being processed.
*/
String getProcessingHostName() {
return this.processingHostName;
}
/**
* Sets the processing host name. May be the empty string.
*
* @param processingHost The processing host name. The empty string if the
* job is not currently being processed.
*/
void setProcessingHostName(String processingHost) {
this.processingHostName = processingHost;
}
/**
* Gets the total size of the data source.
*
* @return The data source size.
*/
long getDataSourceSize() {
return this.dataSourceSize;
}
/**
* Sets the total size of the data source.
*
* @param dataSourceSize The data source size.
*/
void setDataSourceSize(long dataSourceSize) {
this.dataSourceSize = dataSourceSize;
}
/**
* Gets the node data as a byte array that can be sent to the coordination
* service.
*
* @return The node data as a byte array.
*/
byte[] toArray() {
ByteBuffer buffer = ByteBuffer.allocate(MAX_POSSIBLE_NODE_DATA_SIZE);
// Write data (compatible with version 0)
buffer.putInt(this.processingStatus);
buffer.putInt(this.priority);
buffer.putInt(this.numberOfCrashes);
buffer.putLong(this.completedDate);
buffer.putInt(this.errorsOccurred ? 1 : 0);
if (this.version >= 1) {
// Write version
buffer.putInt(this.version);
// Write data
putStringIntoBuffer(deviceId, buffer, TypeKind.BYTE);
putStringIntoBuffer(caseName, buffer, TypeKind.BYTE);
putStringIntoBuffer(caseDirectoryPath, buffer, TypeKind.SHORT);
buffer.putLong(this.manifestFileDate);
putStringIntoBuffer(manifestFilePath, buffer, TypeKind.SHORT);
putStringIntoBuffer(dataSourcePath, buffer, TypeKind.SHORT);
buffer.put(this.processingStage);
buffer.putLong(this.processingStageStartDate);
putStringIntoBuffer(this.processingStageDetailsDescription, buffer, TypeKind.BYTE);
buffer.putLong(this.processingStageDetailsStartDate);
putStringIntoBuffer(processingHostName, buffer, TypeKind.SHORT);
if (this.version >= 2) {
buffer.putLong(this.dataSourceSize);
}
}
// Prepare the array
byte[] array = new byte[buffer.position()];
buffer.rewind();
buffer.get(array, 0, array.length);
return array;
}
/**
* This method retrieves a string from a given buffer. Depending on the type
* specified, either a 'byte' or a 'short' will first be read out of the
* buffer which gives the length of the string so it can be properly parsed.
*
* @param buffer The buffer from which the string will be read.
* @param lengthType The size of the length data.
*
* @return The string read from the buffer.
*/
private String getStringFromBuffer(ByteBuffer buffer, TypeKind lengthType) {
int length = 0;
String output = "";
switch (lengthType) {
case BYTE:
length = buffer.get();
break;
case SHORT:
length = buffer.getShort();
break;
}
if (length > 0) {
byte[] array = new byte[length];
buffer.get(array, 0, length);
output = new String(array);
}
return output;
}
/**
* This method puts a given string into a given buffer. Depending on the
* type specified, either a 'byte' or a 'short' will be inserted prior to
* the string which gives the length of the string so it can be properly
* parsed.
*
* @param stringValue The string to write to the buffer.
* @param buffer The buffer to which the string will be written.
* @param lengthType The size of the length data.
*/
private void putStringIntoBuffer(String stringValue, ByteBuffer buffer, TypeKind lengthType) {
switch (lengthType) {
case BYTE:
buffer.put((byte) stringValue.length());
break;
case SHORT:
buffer.putShort((short) stringValue.length());
break;
}
buffer.put(stringValue.getBytes());
}
final static class InvalidDataException extends Exception {
private static final long serialVersionUID = 1L;
private InvalidDataException(String message) {
super(message);
}
private InvalidDataException(String message, Throwable cause) {
super(message, cause);
}
}
}

93
ZookeeperNodeMigration/src/zookeepernodemigration/TimeStampUtils.java Executable file
View File

@ -0,0 +1,93 @@
/*
* Autopsy Forensic Browser
*
* Copyright 2020 Basis Technology Corp.
* Contact: carrier <at> sleuthkit <dot> org
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package zookeepernodemigration;
import java.text.SimpleDateFormat;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Utility methods for working with time stamps of the form
* 'yyyy_MM_dd_HH_mm_ss'.
*/
final class TimeStampUtils {
/*
* Sample time stamp suffix: 2015_02_02_12_10_31
*/
private static final Pattern TIME_STAMP_PATTERN = Pattern.compile("\\d{4}_\\d{2}_\\d{2}_\\d{2}_\\d{2}_\\d{2}$");
private static final int LENGTH_OF_DATE_TIME_STAMP = 20; // length of the above time stamp
/**
* Checks whether a string ends with a time stamp.
*
* @param inputString The string to check.
*
* @return True or false.
*/
static boolean endsWithTimeStamp(String inputString) {
Matcher m = TIME_STAMP_PATTERN.matcher(inputString);
return m.find();
}
/**
* Gets the fixed length of the time stamp suffix.
*
* @return The length.
*/
static int getTimeStampLength() {
return LENGTH_OF_DATE_TIME_STAMP;
}
/**
* Removes the time stamp suffix from a string, if present.
*
* @param inputString The string to trim.
*
* @return The trimmed string.
*/
static String removeTimeStamp(String inputString) {
String trimmedString = inputString;
if (inputString != null && endsWithTimeStamp(inputString)) {
trimmedString = inputString.substring(0, inputString.length() - getTimeStampLength());
}
return trimmedString;
}
/**
* Gets the time stamp suffix from a string, if present.
*
* @param inputString the name to check for a timestamp
*
* @return The time stamp, may be the empty.
*/
static String getTimeStampOnly(String inputString) {
String timeStamp = "";
if (inputString != null && endsWithTimeStamp(inputString)) {
timeStamp = inputString.substring(inputString.length() - getTimeStampLength(), inputString.length());
}
return timeStamp;
}
/*
* Private contructor to prevent instantiation.
*/
private TimeStampUtils() {
}
}

464
ZookeeperNodeMigration/src/zookeepernodemigration/ZookeeperNodeMigration.java Executable file
View File

@ -0,0 +1,464 @@
/*
* Autopsy Forensic Browser
*
* Copyright 2020 Basis Technology Corp.
* Contact: carrier <at> sleuthkit <dot> org
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package zookeepernodemigration;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.apache.curator.framework.recipes.locks.InterProcessReadWriteLock;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.KeeperException.NoNodeException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
/**
* Utility to migrate Autopsy coordination service data from one ZK database to
* another.
*/
public class ZookeeperNodeMigration {
private static final Logger LOGGER = Logger.getLogger(ZookeeperNodeMigration.class.getName());
private static final int SESSION_TIMEOUT_MILLISECONDS = 300000;
private static final int CONNECTION_TIMEOUT_MILLISECONDS = 300000;
private static final int ZOOKEEPER_SESSION_TIMEOUT_MILLIS = 3000;
private static final int ZOOKEEPER_CONNECTION_TIMEOUT_MILLIS = 15000;
private static final String DEFAULT_NAMESPACE_ROOT = "autopsy";
private static CuratorFramework inputCurator;
private static CuratorFramework outputCurator;
private static Map<String, String> categoryNodeToPath;
private ZookeeperNodeMigration(){
}
/**
* Main method.
*
* @param args the command line arguments
*/
public static void main(String[] args) {
String inputZkIpAddr, inputZkPort, outputZkIpAddr, outputZkPort;
if (args.length == 4) {
inputZkIpAddr = args[0];
inputZkPort = args[1];
outputZkIpAddr = args[2];
outputZkPort = args[3];
} else {
System.out.println("Input needs to be [Input Zookeeper IP Address] [Input Zookeeper Port Number] [Output Zookeeper IP Address] [Output Zookeeper Port Number]");
LOGGER.log(Level.SEVERE, "Input needs to be [Input Zookeeper IP Address] [Input Zookeeper Port Number] [Output Zookeeper IP Address] [Output Zookeeper Port Number]");
return;
}
if (inputZkIpAddr.isEmpty() || inputZkPort.isEmpty() || outputZkIpAddr.isEmpty() || outputZkPort.isEmpty()) {
System.out.println("Input needs to be [Input Zookeeper IP Address] [Input Zookeeper Port Number] [Output Zookeeper IP Address] [Output Zookeeper Port Number]");
LOGGER.log(Level.SEVERE, "Input needs to be [Input Zookeeper IP Address] [Input Zookeeper Port Number] [Output Zookeeper IP Address] [Output Zookeeper Port Number]");
return;
}
inputCurator = initializeCurator(inputZkIpAddr, inputZkPort);
if (inputCurator == null) {
System.out.println("Unable to initialize Zookeeper or Curator: " + inputZkIpAddr + ":" + inputZkPort);
LOGGER.log(Level.SEVERE, "Unable to initialize Zookeeper or Curator: {0}:{1}", new Object[]{inputZkIpAddr, inputZkPort});
return;
}
try {
categoryNodeToPath = populateCategoryNodes(inputCurator);
} catch (KeeperException | CoordinationServiceException ex) {
System.out.println("Unable to initialize Curator: " + inputZkIpAddr + ":" + inputZkPort);
LOGGER.log(Level.SEVERE, "Unable to initialize Curator: {0}:{1}", new Object[]{inputZkIpAddr, inputZkPort});
return;
}
outputCurator = initializeCurator(outputZkIpAddr, outputZkPort);
if (outputCurator == null) {
System.out.println("Unable to initialize Zookeeper or Curator: " + outputZkIpAddr + ":" + outputZkPort);
LOGGER.log(Level.SEVERE, "Unable to initialize Zookeeper or Curator: {0}:{1}", new Object[]{outputZkIpAddr, outputZkPort});
return;
}
try {
// if output ZK database is new, we may have to ceate root "autopsy" node and it's sub-nodes
populateCategoryNodes(outputCurator);
} catch (KeeperException | CoordinationServiceException ex) {
System.out.println("Unable to initialize Curator: " + outputZkIpAddr + ":" + outputZkPort);
LOGGER.log(Level.SEVERE, "Unable to initialize Curator: {0}:{1}", new Object[]{outputZkIpAddr, outputZkPort});
return;
}
copyAllCategoryNodes();
System.out.println("Done...");
}
/**
* Copy all Autopsy coordination service nodes from one ZK database to
* another.
*/
private static void copyAllCategoryNodes() {
for (CategoryNode category : CategoryNode.values()) {
List<String> inputNodeList = Collections.EMPTY_LIST;
try {
inputNodeList = getNodeList(category);
} catch (CoordinationServiceException ex) {
System.out.println("Unable to get ZK nodes for category: " + category.getDisplayName());
LOGGER.log(Level.SEVERE, "Unable to get ZK nodes for category: " + category.getDisplayName(), ex);
continue;
}
for (String zkNode : inputNodeList) {
try {
final byte[] nodeBytes = getNodeData(category, zkNode);
try (Lock manifestLock = tryGetExclusiveLock(outputCurator, category, zkNode)) {
setNodeData(outputCurator, category, zkNode, nodeBytes);
}
} catch (CoordinationServiceException | InterruptedException ex) {
System.out.println("Unable to write ZK node data for node: " + zkNode);
LOGGER.log(Level.SEVERE, "Unable to write ZK node data for node: " + zkNode, ex);
continue;
}
}
}
}
/**
* Initialize Curator framework.
*
* @param zkIpAddr Zookeeper server IP address.
* @param zookeeperPort Zookeeper server port number.
*
* @return CuratorFramework object
*/
private static CuratorFramework initializeCurator(String zkIpAddr, String zookeeperPort) {
try {
if (!isZooKeeperAccessible(zkIpAddr, zookeeperPort)) {
System.out.println("Unable to connect to Zookeeper");
LOGGER.log(Level.SEVERE, "Unable to connect to Zookeeper");
return null;
}
} catch (InterruptedException | IOException ex) {
System.out.println("Unable to connect to Zookeeper");
LOGGER.log(Level.SEVERE, "Unable to connect to Zookeeper", ex);
return null;
}
/*
* Connect to ZooKeeper via Curator.
*/
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
String connectString = zkIpAddr + ":" + zookeeperPort;
CuratorFramework curator = CuratorFrameworkFactory.newClient(connectString, SESSION_TIMEOUT_MILLISECONDS, CONNECTION_TIMEOUT_MILLISECONDS, retryPolicy);
curator.start();
return curator;
}
/*
* Creates Autopsy coordination service root ZK nodes.
*/
private static Map<String, String> populateCategoryNodes(CuratorFramework curator) throws KeeperException, CoordinationServiceException {
/*
* Create the top-level root and category nodes.
*/
String rootNodeName = DEFAULT_NAMESPACE_ROOT;
String rootNode = rootNodeName;
if (!rootNode.startsWith("/")) {
rootNode = "/" + rootNode;
}
Map<String, String> categoryPaths = new ConcurrentHashMap<>();
for (CategoryNode node : CategoryNode.values()) {
String nodePath = rootNode + "/" + node.getDisplayName();
try {
curator.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT).withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE).forPath(nodePath);
} catch (KeeperException ex) {
if (ex.code() != KeeperException.Code.NODEEXISTS) {
throw ex;
}
} catch (Exception ex) {
throw new CoordinationServiceException("Curator experienced an error", ex);
}
categoryPaths.put(node.getDisplayName(), nodePath);
}
return categoryPaths;
}
/**
* Determines if ZooKeeper is accessible with the current settings. Closes
* the connection prior to returning.
*
* @return true if a connection was achieved, false otherwise
*
* @throws InterruptedException
* @throws IOException
*/
private static boolean isZooKeeperAccessible(String solrIpAddr, String zookeeperPort) throws InterruptedException, IOException {
boolean result = false;
Object workerThreadWaitNotifyLock = new Object();
String connectString = solrIpAddr + ":" + zookeeperPort;
ZooKeeper zooKeeper = new ZooKeeper(connectString, ZOOKEEPER_SESSION_TIMEOUT_MILLIS,
(WatchedEvent event) -> {
synchronized (workerThreadWaitNotifyLock) {
workerThreadWaitNotifyLock.notifyAll();
}
});
synchronized (workerThreadWaitNotifyLock) {
workerThreadWaitNotifyLock.wait(ZOOKEEPER_CONNECTION_TIMEOUT_MILLIS);
}
ZooKeeper.States state = zooKeeper.getState();
if (state == ZooKeeper.States.CONNECTED || state == ZooKeeper.States.CONNECTEDREADONLY) {
result = true;
}
zooKeeper.close();
return result;
}
/**
* Tries to get an exclusive lock on a node path appended to a category path
* in the namespace managed by this coordination service. Returns
* immediately if the lock can not be acquired.
*
* IMPORTANT: The lock needs to be released in the same thread in which it
* is acquired.
*
* @param category The desired category in the namespace.
* @param nodePath The node path to use as the basis for the lock.
*
* @return The lock, or null if the lock could not be obtained.
*
* @throws CoordinationServiceException If there is an error during lock
* acquisition.
*/
private static Lock tryGetExclusiveLock(CuratorFramework curator, CategoryNode category, String nodePath) throws CoordinationServiceException {
String fullNodePath = getFullyQualifiedNodePath(category, nodePath);
try {
InterProcessReadWriteLock lock = new InterProcessReadWriteLock(curator, fullNodePath);
if (!lock.writeLock().acquire(0, TimeUnit.SECONDS)) {
return null;
}
return new Lock(nodePath, lock.writeLock());
} catch (Exception ex) {
throw new CoordinationServiceException(String.format("Failed to get exclusive lock for %s", fullNodePath), ex);
}
}
/**
* Retrieve the data associated with the specified node.
*
* @param category The desired category in the namespace.
* @param nodePath The node to retrieve the data for.
*
* @return The data associated with the node, if any, or null if the node
* has not been created yet.
*
* @throws CoordinationServiceException If there is an error setting the
* node data.
* @throws InterruptedException If interrupted while blocked during
* setting of node data.
*/
private static byte[] getNodeData(CategoryNode category, String nodePath) throws CoordinationServiceException, InterruptedException {
String fullNodePath = getFullyQualifiedNodePath(category, nodePath);
try {
return inputCurator.getData().forPath(fullNodePath);
} catch (NoNodeException ex) {
return null;
} catch (Exception ex) {
if (ex instanceof InterruptedException) {
throw (InterruptedException) ex;
} else {
throw new CoordinationServiceException(String.format("Failed to get data for %s", fullNodePath), ex);
}
}
}
/**
* Store the given data with the specified node.
*
* @param category The desired category in the namespace.
* @param nodePath The node to associate the data with.
* @param data The data to store with the node.
*
* @throws CoordinationServiceException If there is an error setting the
* node data.
* @throws InterruptedException If interrupted while blocked during
* setting of node data.
*/
private static void setNodeData(CuratorFramework curator, CategoryNode category, String nodePath, byte[] data) throws CoordinationServiceException, InterruptedException {
String fullNodePath = getFullyQualifiedNodePath(category, nodePath);
try {
curator.setData().forPath(fullNodePath, data);
} catch (Exception ex) {
if (ex instanceof InterruptedException) {
throw (InterruptedException) ex;
} else {
throw new CoordinationServiceException(String.format("Failed to set data for %s", fullNodePath), ex);
}
}
}
/**
* Delete the specified node.
*
* @param category The desired category in the namespace.
* @param nodePath The node to delete.
*
* @throws CoordinationServiceException If there is an error setting the
* node data.
* @throws InterruptedException If interrupted while blocked during
* setting of node data.
*/
private static void deleteNode(CategoryNode category, String nodePath) throws CoordinationServiceException, InterruptedException {
String fullNodePath = getFullyQualifiedNodePath(category, nodePath);
try {
inputCurator.delete().forPath(fullNodePath);
} catch (Exception ex) {
if (ex instanceof InterruptedException) {
throw (InterruptedException) ex;
} else {
throw new CoordinationServiceException(String.format("Failed to set data for %s", fullNodePath), ex);
}
}
}
/**
* Gets a list of the child nodes of a category in the namespace.
*
* @param category The desired category in the namespace.
*
* @return A list of child node names.
*
* @throws CoordinationServiceException If there is an error getting the
* node list.
*/
private static List<String> getNodeList(CategoryNode category) throws CoordinationServiceException {
try {
List<String> list = inputCurator.getChildren().forPath(categoryNodeToPath.get(category.getDisplayName()));
return list;
} catch (Exception ex) {
throw new CoordinationServiceException(String.format("Failed to get node list for %s", category.getDisplayName()), ex);
}
}
/**
* Creates a node path within a given category.
*
* @param category A category node.
* @param nodePath A node path relative to a category node path.
*
* @return
*/
private static String getFullyQualifiedNodePath(CategoryNode category, String nodePath) {
// nodePath on Unix systems starts with a "/" and ZooKeeper doesn't like two slashes in a row
if (nodePath.startsWith("/")) {
return categoryNodeToPath.get(category.getDisplayName()) + nodePath.toUpperCase();
} else {
return categoryNodeToPath.get(category.getDisplayName()) + "/" + nodePath.toUpperCase();
}
}
/**
* Exception type thrown by the coordination service.
*/
private final static class CoordinationServiceException extends Exception {
private static final long serialVersionUID = 1L;
private CoordinationServiceException(String message) {
super(message);
}
private CoordinationServiceException(String message, Throwable cause) {
super(message, cause);
}
}
/**
* An opaque encapsulation of a lock for use in distributed synchronization.
* Instances are obtained by calling a get lock method and must be passed to
* a release lock method.
*/
private static class Lock implements AutoCloseable {
/**
* This implementation uses the Curator read/write lock. see
* http://curator.apache.org/curator-recipes/shared-reentrant-read-write-lock.html
*/
private final InterProcessMutex interProcessLock;
private final String nodePath;
private Lock(String nodePath, InterProcessMutex lock) {
this.nodePath = nodePath;
this.interProcessLock = lock;
}
public String getNodePath() {
return nodePath;
}
public void release() throws CoordinationServiceException {
try {
this.interProcessLock.release();
} catch (Exception ex) {
throw new CoordinationServiceException(String.format("Failed to release the lock on %s", nodePath), ex);
}
}
@Override
public void close() throws CoordinationServiceException {
release();
}
}
/**
* Category nodes are the immediate children of the root node of a shared
* hierarchical namespace managed by a coordination service.
*/
public enum CategoryNode {
CASES("cases"),
MANIFESTS("manifests"),
CONFIG("config"),
CENTRAL_REPO("centralRepository"),
HEALTH_MONITOR("healthMonitor");
private final String displayName;
private CategoryNode(String displayName) {
this.displayName = displayName;
}
public String getDisplayName() {
return displayName;
}
}
}

11
build.xml
View File

@ -106,8 +106,7 @@
<copy file="${basedir}/NEWS.txt" tofile="${zip-tmp}/${app.name}/NEWS.txt"/> <copy file="${basedir}/NEWS.txt" tofile="${zip-tmp}/${app.name}/NEWS.txt"/>
<copy file="${basedir}/Running_Linux_OSX.txt" tofile="${zip-tmp}/${app.name}/Running_Linux_OSX.txt"/> <copy file="${basedir}/Running_Linux_OSX.txt" tofile="${zip-tmp}/${app.name}/Running_Linux_OSX.txt"/>
<copy file="${basedir}/unix_setup.sh" tofile="${zip-tmp}/${app.name}/unix_setup.sh"/> <copy file="${basedir}/unix_setup.sh" tofile="${zip-tmp}/${app.name}/unix_setup.sh"/>
<copy file="${basedir}/ManifestTool/ManifestTool.exe" todir="${zip-tmp}/${app.name}/bin"/> <copy file="${basedir}/ManifestTool/ManifestTool.exe" todir="${zip-tmp}/${app.name}/bin"/>
<copy file="${basedir}/icons/icon.ico" tofile="${zip-tmp}/${app.name}/icon.ico" overwrite="true"/> <copy file="${basedir}/icons/icon.ico" tofile="${zip-tmp}/${app.name}/icon.ico" overwrite="true"/>
@ -115,6 +114,14 @@
<copy flatten="true" todir="${zip-tmp}/${app.name}/docs"> <copy flatten="true" todir="${zip-tmp}/${app.name}/docs">
<fileset dir="${basedir}/docs/doxygen-user/user-docs"/> <fileset dir="${basedir}/docs/doxygen-user/user-docs"/>
</copy> </copy>
<!-- Copy the ZooKeeper migration tool, it's JAR files, and documentation -->
<copy flatten="false" todir="${zip-tmp}/${app.name}/autopsy/ZookeeperNodeMigration">
<fileset dir="${basedir}/ZookeeperNodeMigration/dist"/>
</copy>
<copy flatten="false" todir="${zip-tmp}/${app.name}/autopsy/ZookeeperNodeMigration" overwrite="true">
<fileset dir="${basedir}/ZookeeperNodeMigration/docs"/>
</copy>
<property name="app.property.file" value="${zip-tmp}/${app.name}/etc/${app.name}.conf" /> <property name="app.property.file" value="${zip-tmp}/${app.name}/etc/${app.name}.conf" />
<var name="jvm-value" value="--branding ${app.name} -J-Xms24m -J-Xmx4G -J-Xverify:none -J-XX:+UseG1GC -J-XX:+UseStringDeduplication"/> <var name="jvm-value" value="--branding ${app.name} -J-Xms24m -J-Xmx4G -J-Xverify:none -J-XX:+UseG1GC -J-XX:+UseStringDeduplication"/>