Merge pull request #6618 from APriestman/7122_solr

7122 Update doc for solr 8

docs/doxygen-user/multi-user/installMultiUserClient.dox

@@ -3,20 +3,19 @@
 
 \section multiuser_install_clients Overview
 
-Once the infrastructure is in place, you can configure Autopsy clients to use them. 
-- Install Autopsy on each client system.  Use the normal installer and pick the defaults.
-- Test that the user has access to the shared storage by opening the shared storage folders using Windows Explorer.  If a password prompt is given, then enter the password and store the credentials (see \ref multiuser_users_store).
-- Start Autopsy and open the multi-user settings panel from "Tools", "Options", "Multi-user".  As shown in the screenshot below, you can then enter all of the address and authentication information for the network-based services. Note that in order to create or open Multi-user cases, "Enable Multi-user cases" must be checked and the settings below must be correct.
-
-\image html multiuser_settings.PNG
-
-- For each setting, press the "Test" button to ensure that Autopsy can communicate with each service.  If any fail, then refer to the specific setup page for testing options.  Also check that a firewall is not blocking the communications. 
-
- - NOTE: None of these tests are for permissions on the shared storage because Autopsy does not know about the shared storage.   It can't test that until you make a case.
-
-- Make a test case (see \ref creating_multi_user_cases). You can add a single file in as a logical data source.  The key concept is to look for errors. 
- - If you find errors, look for errors in the log file on the Autopsy client. 
- - If you followed all of the previous steps in all of the previous pages, then a common error at this point is that Solr cannot access the shared storage and it is running as a Service account.  When this happens, you'll see an error message about Solr not being able to create or access a "core".  If this happens, review what user Solr should be running as (see \ref multiuser_users_solr) and change the shared storage configuration or ensure that credentials are stored.    
+Once the infrastructure is in place, you can configure Autopsy clients to use them.
+<ol>
+<li>Install Autopsy on each client system. Use the normal installer and pick the defaults.
+<li>Test that the user has access to the shared storage by opening the shared storage folders using Windows Explorer. If a password prompt is given, then enter the password and store the credentials (see \ref multiuser_users_store).
+<li>Start Autopsy and open the multi-user settings panel from "Tools", "Options", "Multi-user". As shown in the screenshot below, you can then enter all of the address and authentication information for the network-based services. Note that in order to create or open Multi-user cases, "Enable Multi-user cases" must be checked and the settings below must be correct.
 
+\image html solr_autopsy.png
 
+<li>For each setting, press the "Test Connection" button to ensure that Autopsy can communicate with each service. If any fail, then refer to the specific setup page for testing options. Also check that a firewall is not blocking the communications.
+<ul><li>NOTE: None of these tests are for permissions on the shared storage because Autopsy does not know about the shared storage. It can't test that until you make a case.</ul>
+<li>Make a test case (see \ref creating_multi_user_cases). You can add a single file in as a logical data source. The key concept is to look for errors.
+<ul>
+<li>If you find errors, look for errors in the log file on the Autopsy client.
+<li>If you followed all of the previous steps in all of the previous pages, then a common error at this point is that Solr cannot access the shared storage and it is running as a Service account. When this happens, you'll see an error message about Solr not being able to create or access a "core". If this happens, review what user Solr should be running as (see the \ref multiuser_users_solr section) and change the shared storage configuration or ensure that credentials are stored.</ul>
+</ol>
 */

docs/doxygen-user/multi-user/installSolr.dox

@@ -1,149 +1,211 @@
 /*! \page install_solr_page Install and Configure Solr
 
+[TOC]
+
 \section install_solr_overview Overview
+Autopsy uses Apache Solr to store keyword text indexes. A central server is needed in a multi-user cluster to maintain and search the indexes.
 
-Autopsy uses Apache Solr to store keyword text indexes.  A central server is needed in a multi-user cluster to maintain and search the indexes.  
+A new text index is created for each case. The index can be stored either on shared storage or on the local drive of the Solr server(s) (large amount of local storage is required).
 
-A new text index is created for each case and is stored in the case folder on shared storage (not on the local drive of the Solr server). 
+Solr's embedded ZooKeeper is also used as a coordination service for Autopsy.
 
-Solr's embedded ZooKeeper is also used as a coordination service for Autopsy. 
+If you have already installed Solr 4 with a previous version of Autopsy, please see the \ref upgrade_solr_page page for information on how open older cases after the upgrade and migrate data.
 
 \section install_solr_prereq Prerequisites
 
-We use Bitnami Solr, which packages Solr as a Windows service. 
+We have prepared a pre-packaged version of Solr which packages Solr as Windows service.
 
 You will need:
-- A 64-bit version of the Java 8 Runtime Environment (JRE) from https://github.com/ojdkbuild/ojdkbuild. (<a href="https://github.com/ojdkbuild/ojdkbuild/releases/download/java-1.8.0-openjdk-1.8.0.242-1.b08/java-1.8.0-openjdk-1.8.0.242-1.b08.ojdkbuild.windows.x86_64.msi"> Link to installer</a>)
-- The Apache Solr 4.10.3-0 installation package.  This is no longer available from its original source, but you can find it on our site: https://sourceforge.net/projects/autopsy/files/CollaborativeServices/Solr.
--- NOTE: We tested Solr 6 at one point, but ran into stability problems when loading and unloading cores.  For now, you need to use Solr 4. 
-- An installed version of Autopsy so that you can copy files from it.   You can install Autopsy on one of the planned client systems.  You do not need to install it on the Solr server. 
-- A network-accessible machine to install Solr on. Note that the Solr process will need to write data out to the main shared storage drive, and needs adequate permissions to write to this location, which may be across a network.
-
-
+<ul>
+<li>A 64-bit version of the Java 8 Runtime Environment (JRE) from <a href="https://github.com/ojdkbuild/ojdkbuild">https://github.com/ojdkbuild/ojdkbuild</a>. (<a href="https://github.com/ojdkbuild/ojdkbuild/blob/master/README.md">Download links</a>)
+<li>Pre-packaged Autopsy version of Solr from https://sourceforge.net/projects/autopsy/files/CollaborativeServices/Solr/bitnami-solr-4.10.3-0-windows-installer.exe/download
+<li>A network-accessible machine to install Solr on. Note that the Solr process will need to write data out to the main shared storage drive, and needs adequate permissions to write to this location, which may be across a network.
 
 \section install_solr_install Installation
 
-\subsection install_solr_install_java JRE Installation
-1.  Install the Java JRE if needed. You can test this by running _where java_ from the command line. If you see output similar to the results below, you have a JRE.
-<br><br>
+\subsection install_solr_jre JRE Installation
+
+Solr requires a Java Runtime Environment (JRE), which may already be installed. You can test this by running \c "where java" from the command line. If you see output similar to the results below, you have a JRE.
+
 \image html wherejava.PNG
-<br><br>   
-If you need the JRE, install it with the default settings.
-
-\subsection install_solr_install_solr Solr Installation
-
-The following steps will configure Solr to run using an account that will have access to the network storage.
-
-1.	Run the Bitnami installer, <i>"bitnami-solr-4.10.3-0-windows-installer.exe"</i>
-2.	If Windows prompts with User Account Control, click _Yes_
-3.	Follow the prompts through to completion. You do not need to <i>"Learn more about Bitnami cloud hosting"</i> so you can clear the check box.
-4.	If you see an error dialog like the following, you may safely ignore it.
-<br><br>
-\image html apachebadmessage.PNG
-<br>
-5.	When the installation completes, clear the <i>"Launch Bitnami Apache Solr Stack Now?"</i> checkbox and click _Finish_.
-
 
+If you need the JRE, use the link in the \ref install_solr_prereq section above to download an installer. Accept the default settings during installation.
 
 \subsection install_solr_config Solr Configuration
 
-1. Stop the _solrJetty_ service by pressing _Start_, typing _services.msc_, pressing _Enter_, and locating the _solrJetty_ Windows service. Select the service and press _Stop the service_. If the service is already stopped and there is no _Stop the service_ available, this is okay.
-2. <b>Service Configuration</b>: Edit the <i>"C:\Bitnami\solr-4.10.3-0\apache-solr\scripts\serviceinstall.bat"</i> script. You need administrator rights to change this file. The easiest way around this is to save a copy on the Desktop, edit the Desktop version, and copy the new one back over the top of the old. Windows will ask for permission to overwrite the old file; allow it. You should make the following changes to this file:
-    <br>
-    <br>
-    - Add the following options in the line that begins with <i>"C:\Bitnami\solr-4.10.3-0/apache-solr\scripts\prunsrv.exe"</i> :
-        + <i>++JvmOptions=-Dcollection.configName=AutopsyConfig</i>
-	+ <i>++JvmOptions=-Dbootstrap_confdir="C:\Bitnami\solr-4.10.3-0\apache-solr\solr\configsets\AutopsyConfig\conf"</i>
-	+ <i>++JvmOptions=-DzkRun </i>
-               <br>
-    - Replace the path to JavaHome with the path to your 64-bit version of the JRE. If you do not know the path, the correct JavaHome path can be obtained by running the command "where java" from the Windows command line. An example is shown below. The text in yellow is what we are interested in. Do not include the "bin" folder in the path you place into the JavaHome variable. A correct example of the final result will look something like this: <i>–-JavaHome="C:\Program Files\ojdkbuild\java-1.8.0-openjdk-1.8.0.222-1"</i>
-	<br><br>
-	A portion of an updated _serviceinstall.bat_ is shown below, with the changes marked in yellow.
-     <br><br>
-     \image html serviceinstall.PNG
-	     <br><br>
-3. <b>Solr Configuration</b>: Edit <i>"C:\Bitnami\solr-4.10.3-0\apache-solr\solr\solr.xml"</i> to set the _transientCacheSize_ to the maximum number of cases expected to be open concurrently. If you expect ten concurrent cases, the text to add is
- <i>\<int name="transientCacheSize">10\</int></i>
- <br><br>
- The added part is highlighted in yellow below. Ensure that it is inside the <i>\<solr></i> tag as follows:
- <br>
- \image html transientcache.PNG
- <br><br>
-4. <b>Log Configuration</b>: Edit <i>"C:\Bitnami\solr-4.10.3-0\apache-solr\resources/log4j.properties"</i> to configure Solr log settings:
+Follow these steps to configure Solr:
+
 <ol>
-<li>Increase the log rotation size threshold (_log4j\.appender\.file\.MaxFileSize_) from 4MB to 100MB.
-<li>Remove the _CONSOLE_ appender from the _log4j\.rootLogger_ line.
-<li>Add the line "log4j.logger.org.apache.solr.update.processor.LogUpdateProcessor=WARN".
+<li>Extract the solr-8.6.3.zip archive from the location given in the \ref install_solr_prereq section into a directory of your choice. The rest of this document assumes that the archive is extracted into \c "C:\solr-8.6.3" directory.
+<li>Go to the \c "C:\solr-8.6.3\bin" directory and open the \c "solr.in.cmd" file in a text editor.
+\image html solr_config_folder.png
+<li>Search for each "TODO" and specify a valid path for each of the required configuration parameters. These parameters will be described in detail \ref install_solr_params "below".
+
+\image html solr_config_todo.png
+<br>
+\image html solr_config_param.png
 </ol>
-The log file should end up looking like this (modified lines are highlighted in yellow
 
-\image html log4j.PNG
+\subsubsection install_solr_params Solr Configuration Parameters
 
- 
-5. <b>Schema Configuration</b>: From an Autopsy installation, copy the following into <i>"C:\Bitnami\solr-4.10.3-0\apache-solr\solr"</i>:
-- The folder <i>"C:\Program Files\Autopsy-XXX(current version)\autopsy\solr\solr\configsets"</i>
-- The folder <i>"C:\Program Files\Autopsy-XXX(current version)\autopsy\solr\solr\lib"</i>  
-- The file <i>"C:\Program Files\Autopsy-XXX(current version)\autopsy\solr\solr\zoo.cfg"</i>
+Required Solr Configuration Parameters:
+<ul>
+<li><b>JAVA_HOME</b> – path to 64-bit JRE installation. For example \c "JAVA_HOME=C:\Program Files\Java\jre1.8.0_151" or \c "JAVA_HOME=C:\Program Files\ojdkbuild\java-1.8.0-openjdk-1.8.0.222-1"
+<li><b>DEFAULT_CONFDIR</b> – path to Autopsy configuration directory. If the Solr archive was extracted into \c "C:\solr-8.6.3" directory, then this path will be \c "C:\ solr-8.6.3\server\solr\configsets\AutopsyConfig\conf".
+<li><b>Dbootstrap_confdir</b> – same path as <b>DEFAULT_CONFDIR</b>
+<li><b>SOLR_JAVA_MEM</b> - Solr JVM heap size should be somewhere between one third and one half of the total RAM available on the machine. A rule of thumb would be use \c "set SOLR_JAVA_MEM=-Xms2G -Xmx14G" for a machine with 32GB of RAM or more, and \c "set SOLR_JAVA_MEM=-Xms2G -Xmx8G" for a machine with 16GB of RAM.
+<li><b>SOLR_DATA_HOME</b> – location where Solr indexes will be stored. If this is not configured, the indexes will be stored in the \c "C:\solr-8.6.3\server\solr" directory. NOTE: for Autopsy cases consisting of large number of data sources, Solr indexes can get very large (hundreds of GBs, or TBs) so they should probably be stored on a larger network share.
+</ul>
 
+Optional Solr Configuration Parameters:
+<ul>
+<li><b>SOLR_HOST</b> – by default, the Solr node name is "localhost". If multiple Solr nodes are going to be used as part of Solr Cloud, then specify the current computer’s host name in the SOLR_HOST variable. 
+</ul>
 
-\subsection install_solr_reinstall Reinstall Service
+\subsubsection install_sorl_index_file_loc Solr Text Index File Location
 
-Because we made changes to the service configuration, we need to reinstall it. 
+<b>Important note:</b> previous versions of Autopsy (Autopsy 4.17.0 and earlier) stored the Solr text indexes in the case output directory. As a result, the Solr indexes would get deleted if a user deleted the case output directory. Solr 8 (i.e. Autpsy 4.18.0 and later) no longer stores the Solr text index files in the case output directory but instead stores them in location defined by the <b>SOLR_DATA_HOME</b> parameter. As a consequence, if a user choses to manually delete case output directories (for example, to free up disk space), the Solr index directories located in <b>SOLR_DATA_HOME</b> need to be manually deleted as well.
 
-1. Start a Windows command prompt as administrator by pressing Start, typing <i>command</i>, right clicking on <i>Command Prompt</i>, and clicking on <i>Run as administrator</i>. Then run the following command to uninstall the solrJetty service:
+Text index for an Autopsy case will follow a naming structure according to following rules: \c "[Autopsy case name] [Case creation time stamp] [Text index creation time stamp] [shardX_replica_nY]". For example, the text index for an Autopsy case "Test Case" will be located in the following directory inside <b>SOLR_DATA_HOME</b>:
 
-              cmd /c C:\Bitnami\solr-4.10.3-0\apache-solr\scripts\serviceinstall.bat UNINSTALL
+\image html solr_config_case.png
 
-       You will very likely see a result that says "The solrJetty service is not started." This is okay.
-2. In the same prompt, run the following command to install the solrJetty service:
+\subsection install_solr_service Solr Service Installation
 
-               cmd /c C:\Bitnami\solr-4.10.3-0\apache-solr\scripts\serviceinstall.bat INSTALL
-<br> Note the argument "INSTALL" is case sensitive. Your command prompt should look like the screenshot below. Very likely your command prompt will say "The solrJetty service could not be started." This is okay.
- <br><br>
- \image html solrinstall1.PNG
- <br><br>
- 
+At this point Solr has been configured and ready to be installed as Windows service. Open a command line console as Administrator and navigate to the \c "C:\solr-8.6.3\bin" directory. From there, run the following command: \c "nssm install Solr_8.6.3".
 
-At this point you should be able to access the Solr admin panel in a web browser via the URL http://localhost:8983/solr/#/
+\image html solr_install_1.png
 
+An NSSM UI window will appear. Click the "Path" navigation button:
+
+\image html solr_install_2.png
+
+Select the \c "C:\solr-8.6.3\bin\solr.cmd" file. NOTE: Make sure you don’t select the \c "solr.in.cmd" file by accident. In the "Arguments" parameter, type in \c "start –f –c":
+
+\image html solr_install_3.png
+
+Optionally, configure service’s display name, startup type, and account info: 
+
+\image html solr_install_4.png
 
 \subsection install_solr_service_user Configure Service User
 
-Back in \ref install_multiuseruser_page, you should have decided what user to run Solr as.   To configure Solr to run as that user, you'll use Windows Service Manager. 
+In the \ref install_multiuseruser_page section, you should have decided what user to run Solr as. To configure Solr to run as that user, you'll use Windows Service Manager.
 
+Switch to the "Log On" tab to change the logon credentials to the chosen user who will have access to the shared storage.
+<ul><li>If you specify a domain account, the account name will be in the form of \c "DOMAINNAME\username" as shown in the example below</ul>
 
-- Press _Start_, type _services.msc_, and press _Enter_. 
-- Find _solrJetty_. If the service is running, press _Stop the service_, 
-- Double click the service and switch to the _Log On_ tab to change the logon credentials to the chosen user who will have access to the shared storage. 
- - If you specify a domain account, the account name will be in the form of _DOMAINNAME\\username_ as shown in the example below
+\image html solr_user_1.png
 
-\image html solrinstall2.PNG
+Click \c "Install Service". You should see the following UI window appear:
 
+\image html solr_user_2.png
 
-- Start the service again. 
+\section install_solr_start Start Solr Service
 
-\section install_solr_test Testing
+At this point the Solr service has been configured and installed. You can verify this by opening Windows "Services" window:
 
-There are two tests that you should perform to confirm that the Solr machine is configured correctly. 
+\image html solr_start_1.png
 
-- <b>Web Interface</b>: You should attempt to access the Solr admin panel in a web browser from another machine on the network. Replace the IP address in the following URL with the IP address or the host name that the Solr service is running on: <i>http://172.16.12.61:8983/solr/#/</i>.
- <br><br>
- \image html solrinstall3.PNG
- <br><br>
+Start the "Solr_8.6.3" service, and verify that the service status changes to "Running".
 
-If the service is appropriately started but you are unable to see the screenshot above, then it could be that port 8983 for Solr and port 9983 for ZooKeeper are blocked by your firewall. Contact your network administrator to open these ports. 
+\image html solr_start_2.png
 
-- <b>Shared Storage</b>: Log into the Solr computer as the user you decided to run the Solr service as and attempt to access the shared storage paths. Ensure that you can access the UNC paths (or drive letters if you have hardware NAS).  If everything is configured correctly you should be able to access the storage paths without having to provide credentials. 
-If you are prompted for a password to access the shared storage, then either enter the password and choose to save the credentials or reconfigure the setup so that the same passwords are used, etc. See \ref multiuser_users_store for steps on storing credentials.  If you needed to store the credentials, then you should restart the service or reboot the computer (we have observed that a running service does not get the updated credentials). 
+\section install_solr_testing Testing
 
-NOTE: You can not do a full test of permissions until you make a test case after all of the services are configured.
+There are two tests that you should perform to confirm that the Solr machine is configured correctly.
+
+<ul>
+<li><b>Web Interface:</b> You should attempt to access the Solr admin panel in a web browser. On the Solr machine, navigate to http://localhost:8983/solr/#/ and verify that the Solr admin console gets displayed. You should also attempt to access the Solr admin panel in a web browser from another machine on the network. Replace "localhost" in the previous URL with the IP address or the host name that the Solr service is running on.
+
+\image html solr_testing_1.png
+
+If the service is appropriately started but you are unable to see the screenshot above, then it could be that port 8983 for Solr and port 9983 for ZooKeeper are blocked by your firewall. Contact your network administrator to open these ports.
+
+<li><b>Shared Storage:</b> Log in to the Solr computer as the user you decided to run the Solr service as and attempt to access the shared storage paths. Ensure that you can access the UNC paths (or drive letters if you have hardware NAS). If everything is configured correctly you should be able to access the storage paths without having to provide credentials. If you are prompted for a password to access the shared storage, then either enter the password and choose to save the credentials or reconfigure the setup so that the same passwords are used. See the \ref multiuser_users_store section for steps on storing credentials. If you needed to store the credentials, then you should restart the service or reboot the computer (we have observed that a running service does not get the updated credentials).
+</ul>
+
+\section install_solr_autopsy Configuring Autopsy
+
+Once the rest of the services are configured you will \ref install_multiuserclient_page "configure Autopsy to enable multi-user cases". For the Solr 8 server, configure the Solr 8 Service and the ZooKeeper service connection info. ZooKeeper connection info is required. The ZooKeeper port number is 1000 higher than Solr service port number. By default, Solr service port is 8983 making the embedded ZooKeeper port 9983. You may also use a \ref install_solr_standalone_zk "standalone ZooKeeper service".
+
+\section install_sorl_adding_nodes Adding More Solr Nodes
+
+Solr 8 has ability for multiple Solr nodes to work together as a Solr cluster. In this mode (SolrCloud mode) each Solr collection/index is split across all of the available Solr nodes. This is called sharding. For example, if there are 4 Solr nodes in a SolrCloud cluster, then the text index will be split across the 4 Solr nodes, thus greatly reducing the load on each individual Solr server and improving Solr indexing and searching performance.
+ 
+To create a Solr cluster, the following steps need to be taken:
+<ol>
+<li>Follow steps in the \ref install_solr_config and \ref install_solr_service sections to create a Solr node (e.g. "Solr1"). Start the Solr service on the Solr1 machine. This machine will host the ZooKeeper service for the SolrCloud cluster. 
+<li>To add an additional Solr node (e.g. "Solr2") to the SolrCloud cluster, follow the steps in the \ref install_solr_config and \ref install_solr_service sections on the Solr2 machine. Do not start the Solr service on Solr2 yet.
+<li>Solr uses ZooKeeper for its internal coordination, so all of the Solr nodes in SolrCloud need to be pointed at the same ZooKeeper service instance. Therefore in order for Solr2 node to be part of SolrCloud, it needs to use the ZooKeeper service that all the other Solr nodes in the SolrCloud cluster are using. In step 1 we have configured Solr1 node to start its embedded ZooKeeper service (this is default Solr behavior). To achieve that, <b>ZK_HOST</b> setting on Solr2 needs to be changed to point at the ZooKeeper service that is running on Solr1 node. The ZooKeeper port number is 1000 higher than <b>SOLR_PORT</b>. By default, <b>SOLR_PORT</b> is 8983 so the embedded ZooKeeper port is 9983. Therefore the <b>ZK_HOST</b> setting in \c "C:\solr-8.6.3\bin\solr.in.cmd" file (assuming that the Solr package ZIP was extracted into \c "C:\solr-8.6.3\" directory) on Solr2 machine needs to be modified to use ZooKeeper service running on Solr1:9983.
+
+\image html solr_adding_nodes_1.png
+
+<li>Start Solr service on Solr2 machine.
+<li>When you log into a Solr admin console on either Solr1 or Solr2 (via either going to http://localhost:8983/solr/#/ on the machine, or via http://solr1:8983/solr/#/), and then navigate to "Cloud" -> "Nodes" section of the admin tree, you should see all of the Solr nodes that are part of the SolrCloud:
+
+\image html solr_adding_nodes_2.png
+
+<li>Additional Solr nodes can be added to the SolrCloud by repeating the previous steps.
+</ol>
+
+\section install_solr_autopsy_zk Autopsy’s Use of ZooKeeper Service
+
+Autopsy uses ZooKeeper service for multi-user coordination purposes. Autopsy uses ZooKeeper to obtain locks on case level resources before modifying them. Most importantly, Autopsy stores some of its internal state data in ZooKeeper – which cases have been created, their processing state (pending, processing, or completed), as well as other case and job level state data. This is especially important if you are running Autopsy in Auto Ingest Mode, as auto ingest needs to know which jobs have already been processed. 
+
+In the screen shot below, for coordination purposes Autopsy will be using the ZooKeeper server that is running on the Solr 8 server ("Solr1" machine). 
+
+\image html solr_autopsy_zk.png
+
+\subsection install_solr_standalone_zk Standalone ZooKeeper Server
+
+In our testing, for Autopsy purposes it is not necessary to have a standalone ZooKeeper server. For the regular Autopsy use case it is sufficient to use the "embedded" ZooKeeper service that is started by Solr service (on port 9983). However, Apache Solr documentation recommends that a standalone ZooKeeper service (running on separate a machine) is used in production environments. Below are instructions on how to setup a standalone ZooKeeper server and how to configure Solr & Autopsy to use it. 
+
+General Solr-related steps for this process are outlined in the Solr user guide below, in section "SolrCloud Configuration and Parameters":
+
+https://lucene.apache.org/solr/guide/8_6/solrcloud-configuration-and-parameters.html
+
+<ol>
+<li>Download the appropriate Zookeeper installation from http://zookeeper.apache.org/releases.html . Solr 8.6.3 is integrated with Zookeeper 3.5.7. There are several options for download – binaries or source code. The file that you are looking for is \c "apache-zookeeper-3.5.7-bin.tar.gz":
+https://archive.apache.org/dist/zookeeper/zookeeper-3.5.7/
+<li>Extract the downloaded tar file containing Zookeeper installation
+<li>Create/edit the \c "/conf/zoo.cfg" file to have the following: 
+<ul>
+<li>Specify "dataDir" directory – this is where ZK database will be stored.
+<li>Specify "clientPort". For example, "clientPort=9983". 
+<li>"Four letter commands" need to be enabled in ZK config file: https://lucene.apache.org/solr/guide/8_6/setting-up-an-external-zookeeper-ensemble.html#configuration-for-a-zookeeper-ensemble
+</ul>
+<li>There are Windows and Linux Zookeeeper startup scripts. For Windows, open a command prompt (admin NOT required), go to the directory where the tar file was extracted (e.g. \c "C:\Bitnami\zookeeper-3.5.7"), and type in \c "bin\zkServer.cmd". We have been using Cygwin in our testing and therefore using Linux commands in our examples. For Linux/CygWin, go to the same directory (e.g. \c "C:\Bitnami\zookeeper-3.5.7"), and type in \c "bin/zkServer.sh start".
+<li>To verify that Zookeeper is running, in command prompt one can type in \c "bin/zkServer.sh status" (or equivalent Windows command). 
+
+\image html solr_standalone_zk_1.png
+
+<li>To make Solr use the external ZooKeeper, the following needs to be done. Navigate to the directory where Solr startup scripts are located (usually \c "C:\solr-8.6.3\apache-solr\bin"). Open the \c "solr.in.cmd" file in text editor. If standalone ZooKeeper service is running on the same machine (not recommended), edit the <b>ZK_HOST</b> variable to be \c "set ZK_HOST=localhost:9983". If Zookeeper is running a different machine (e.g. "Solr5"), then enter the Zookeeper machine's host name or IP address instead of "localhost" (e.g. \c "set ZK_HOST=Solr5:9983").
+
+\image html solr_standalone_zk_2.png
+
+<li>Re-installation of Solr service is not necessary. Simply stop the Solr service and re-start it.
+<li>Once the Solr service has been restarted, you can navigate to Solr admin console (Cloud -> ZK Status) and verify that Solr is using the correct Zookeeper and that the Zookeeper is running. 
+
+\image html solr_standalone_zk_3.png
+
+<li>Configure Autopsy Multi-User panel to use the standalone ZooKeeper server. Start Autopsy and open the multi-user settings panel from "Tools", "Options", "Multi-user". Note that in order to create or open Multi-user cases, "Enable Multi-user cases" must be checked and the settings below must be correct.
+
+\image html solr_standalone_zk_4.png
+</ol>
 
 \section install_solr_backup Backing Up
 
 Solr creates two types of data that need to be backed up:
-- <b>Text Indexes</b>: These are stored in the case folder on the shared storage.
-- <b>ZooKeeper Data</b>: Autopsy uses a service called ZooKeeper embedded in Solr that stores data about what cases exist and who has them open. This data needs to be backed up so that you can have a list of all available multi-user cases. 
- - In a default installation that data is stored in C:\\Bitnami\\solr-4.10.3-0\\apache-solr\\solr\\zoo_data.
 
+<ul>
+<li><b>Text Indexes:</b> These are stored in directory that was specified in <b>SOLR_DATA_HOME</b> parameter (see \ref install_solr_params).
+<li><b>ZooKeeper Data:</b> Autopsy uses a service called ZooKeeper embedded in Solr that stores data about what cases exist and who has them open. This data needs to be backed up so that you can have a list of all available multi-user cases.
+<ol><li>In a default installation that data is stored in \c "C:\solr-8.6.3\server\solr zoo_data" (assuming that the Solr package ZIP was extracted into \c "C:\solr-8.6.3" directory).</ol>
+</ul>
+
+\section install_solr_delayed_start Delayed Start Problems With Large Number Of Solr Collections
+
+In our testing, we have encountered an issue when a very large number (thousands) of Autopsy multi-user cases was created. Each new Autopsy multi-user case creates a Solr "collection" that contains the Solr text index. With 2,000 existing collections, when Solr service is restarted, Solr appears to internally be "loading" roughly 250 collections per minute (in chronological order, starting with oldest collections). After 4 minutes roughly half of the 2,000 collections were loaded. Users are able to search the collections that have been loaded, but they are unable to open or search the collections that have not yet been internally loaded by Solr. After 7-8 minutes all collections were loaded. These numbers will vary depending on the specific cluster configuration, text index file location (network or local storage), network throughput, number of Solr servers, etc. 
 
 */

docs/doxygen-user/multi-user/upgradeSolr.dox

@@ -0,0 +1,76 @@
+/*! \page upgrade_solr_page Upgrading Solr
+
+[TOC]
+
+\section solr_upgrade_solr4 Upgrading from Solr 4 from Solr 8
+
+To upgrade from Solr 4 to Solr 8, first follow the instructions on the \ref install_solr_page page. At the end of that process, all new cases will be created using Solr 8. If you need to open older cases that were created using Solr 4, please see the following sections which describe how to set up Solr 4 and Solr 8 in parallel and how to migrate Zookeeper data.
+
+\subsection install_solr_solr_4 Legacy (Solr 4) Autopsy Cases and Text Indexes
+
+Things to keep in mind regarding backwards compatibility with existing Solr 4 multi-user cases (cases created with Autopsy versions 4.17.0 and earlier):
+
+<ul>
+<li>New text indexes can only be created using Solr 8 (Autopsy version 4.18.0 and later). 
+<li>Autopsy is able to open and search existing Solr 4 single-user cases. This functionality comes "out of the box" and nothing needs to be configured or installed to achieve this.
+<li>Solr 8 multi-user server is unable to open existing legacy Solr 4 multi-user cases because Solr indexes are only backward compatible one major version back (i.e. Solr 8 is only able to open indexes created using Solr 8 or Solr 7).
+</ul>
+
+To overcome this limitation on older multi-user cases, there is an option of configuring Autopsy to be able to connect to both Solr 8 and Solr 4 multi-user servers. If a multi-user Solr 4 server is configured, you will be able to open and search existing Solr 4 cases, as well as add data sources to the existing Solr 4 cases, thus adding more data to the existing legacy Solr 4 text indexes. However, new text indexes can only be created using Solr 8. Note that a Solr 4 server is unable to open text indexes created using a Solr 8 server.
+
+\subsection install_solr_parallel Running Solr 8 and Solr 4 Multi-user Servers in Parallel 
+
+It is possible to configure Autopsy to be able to connect to both Solr 8 and Solr 4 multi-user servers. Start Autopsy and open the multi-user settings panel from "Tools", "Options", "Multi-user". Note that in order to create or open Multi-user cases, "Enable Multi-user cases" must be checked and the settings below must be correct. 
+
+It is recommended that you run Solr 8 and Solr 4 servers on separate machines. In the example below, Solr 8 server is running on a machine with host name "Solr1" and Solr 4 server is running on a machine with host name "Solr6". 
+
+\image html solr_running_in_parallel.png
+
+Once both the Solr 8 and Solr 4 multi-user server connection info is entered and saved, Autopsy will be able to open both Solr 8 multi-user cases (Autopsy version 4.18.0 and later), as well as legacy existing Solr 4 multi-user cases (cases created with Autopsy versions 4.17.0 and earlier).
+
+<b>IMPORTANT</b>: The "Test Connection" button does not verify which version of Solr Autopsy is connecting to. It only verifies that Autopsy is connecting to a Solr server and is able to receive a response. Therefore it is important that the user enters correct server connection info in appropriate fields. 
+
+If you intend to run Solr 4 and Solr 8 servers on the same machine at the same time, you need to change the port of the Solr 8 service using the setting <b>SOLR_PORT</b> in \c "C:\solr-8.6.3\bin\solr.in.cmd" file (assuming that the Solr package ZIP was extracted into \c "C:\solr-8.6.3\" directory). By default the Solr service starts on port 8983. 
+
+\subsection install_solr_zk_migration Migrating existing ZooKeeper data to Solr 8 server
+
+If you have an existing Solr 4 server that you have been using in Multi-User (MU) mode and you wish to migrate this data onto your new Solr 8 server, we have developed a migration utility that performs this task. Note that the utility is able to migrate ZooKeeper data from any one ZooKeeper server to another ZooKeeper server, and is not tied to migrating Solr 4 ZooKeeper data to a Sol 8 ZooKeeper server. 
+
+In our example we will be migrating ZooKeeper data from a ZooKeeper server running on a Solr 4 server (on machine "Solr6") to a brand new ZooKeeper server running on a Solr 8 server (on machine "Solr1"). 
+
+You can browse the existing ZooKeeper data if you go to Solr6 machine and open the Solr admin console (http://localhost:8983/solr/#/). In the Solr admin console, navigate to "Cloud"-> "Tree", and expand the "autopsy" section of the tree:
+
+\image html solr_zk_migration_1.png
+
+You can follow the same steps to browse the ZooKeeper data on the new Solr 8 server (on "Solr1" machine). If Autopsy has not been used with this server yet, the "autopsy" folder will be missing, as in the example below:
+
+\image html solr_zk_migration_2.png
+
+The ZooKeeper migration utility (ZookeeperNodeMigration.jar) is located in \c "C:\Program Files\(current version of Autopsy)\autopsy\ZookeeperNodeMigration" directory:
+
+\image html solr_zk_migration_3.png
+
+ZookeeperNodeMigration utility requires the following inputs: 
+<ul>
+<li>Input Zookeeper IP Address or Hostname
+<li>Input Zookeeper Port Number
+<li>Output Zookeeper IP Address or Hostname
+<li>Output Zookeeper Port Number
+</ul>
+
+For example, if you execute the following command from command line, the Zookeeper nodes will get copied from Zookeeper server on Solr6:9983 to Zookeeper server on Solr1:9983 :
+
+> java -jar ZookeeperNodeMigration.jar Solr6 9983 Solr1 9983
+
+\image html solr_zk_migration_4.png
+
+If you do not have Java installed on the machine, you can use the packaged version of Java that is distributed along with Autopsy. For example:
+
+> \c "C:\Program Files\Autopsy-4.18.0\jre\bin\java.exe" -jar ZookeeperNodeMigration.jar Solr6 9983 Solr1 9983 
+
+To verify that the ZooKeeper data has been copied from the Solr6 server to the Solr1 server, refresh the Solr admin console on the Solr1 machine. You should now see the "autopsy" directory, along with its contents, when you go to the "Cloud" -> "Tree" section of the Solr admin console:
+
+\image html solr_zk_migration_5.png
+
+
+*/