readme file & some ux improvements

README.md

@@ -1,14 +1,16 @@
 # DNSRecon - Passive Infrastructure Reconnaissance Tool
 
-DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys.
+DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
 
-**Current Status: Phase 2 Implementation**
+**Repo Link:** [https://git.cc24.dev/mstoeck3/dnsrecon](https://git.cc24.dev/mstoeck3/dnsrecon)
 
-  * ✅ Core infrastructure and graph engine
+-----
-  * ✅ Multi-provider support (crt.sh, DNS, Shodan)
+
-  * ✅ Session-based multi-user support
+## Concept and Philosophy
-  * ✅ Real-time web interface with interactive visualization
+
-  * ✅ Forensic logging system and JSON export
+The core philosophy of DNSRecon is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNSRecon will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
+
+For power users who require more in-depth information, DNSRecon can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
 
 -----
 
@@ -20,21 +22,45 @@ DNSRecon is an interactive, passive reconnaissance tool designed to map adversar
   * **Forensic Logging**: A complete audit trail of all reconnaissance activities is maintained.
   * **Confidence Scoring**: Relationships are weighted based on the reliability of the data source.
   * **Session Management**: Supports concurrent user sessions with isolated scanner instances.
+  * **Extensible Provider Architecture**: Easily add new data sources to expand the tool's capabilities.
+  * **Web-Based UI**: An intuitive and interactive web interface for managing scans and visualizing results.
 
 -----
 
-## Installation
+## Technical Architecture
+
+DNSRecon is a web-based application built with a modern technology stack:
+
+  * **Backend**: The backend is a **Flask** application that provides a REST API for the frontend and manages the scanning process.
+  * **Scanning Engine**: The core scanning engine is a multi-threaded Python application that uses a provider-based architecture to query different data sources.
+  * **Session Management**: **Redis** is used for session management, allowing for concurrent user sessions with isolated scanner instances.
+  * **Data Storage**: The application uses an in-memory graph to store and analyze the relationships between different pieces of information. The graph is built using the **NetworkX** library.
+  * **Frontend**: The frontend is a single-page application that uses JavaScript to interact with the backend API and visualize the graph.
+
+-----
+
+## Data Sources
+
+DNSRecon queries the following data sources:
+
+  * **DNS**: Standard DNS lookups (A, AAAA, CNAME, MX, NS, SOA, TXT).
+  * **crt.sh**: A certificate transparency log that provides information about SSL/TLS certificates.
+  * **Shodan**: A search engine for internet-connected devices (requires an API key).
+
+-----
+
+## Installation and Setup
 
 ### Prerequisites
 
   * Python 3.8 or higher
   * A modern web browser with JavaScript enabled
-  * (Recommended) A Linux host for running the application and the optional DNS cache.
+  * A Linux host for running the application
 
 ### 1\. Clone the Project
 
 ```bash
-git clone https://github.com/your-repo/dnsrecon.git
+git clone https://git.cc24.dev/mstoeck3/dnsrecon
 cd dnsrecon
 ```
 
@@ -50,20 +76,18 @@ pip install -r requirements.txt
 
 The `requirements.txt` file contains the following dependencies:
 
-  * Flask\>=2.3.3
+  * Flask
-  * networkx\>=3.1
+  * networkx
-  * requests\>=2.31.0
+  * requests
-  * python-dateutil\>=2.8.2
+  * python-dateutil
-  * Werkzeug\>=2.3.7
+  * Werkzeug
-  * urllib3\>=2.0.0
+  * urllib3
-  * dnspython\>=2.4.2
+  * dnspython
   * gunicorn
   * redis
   * python-dotenv
 
------
+### 3\. Configure the Application
-
-## Configuration
 
 DNSRecon is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
 
@@ -91,6 +115,22 @@ The following environment variables are available for configuration:
 
 -----
 
+## Running the Application
+
+For development, you can run the application using the following command:
+
+```bash
+python app.py
+```
+
+For production, it is recommended to use a more robust server, such as Gunicorn:
+
+```bash
+gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+```
+
+-----
+
 ## Systemd Service
 
 To run DNSRecon as a service that starts automatically on boot, you can use `systemd`.
@@ -145,6 +185,65 @@ sudo systemctl status dnsrecon.service
 
 -----
 
+## Updating the Application
+
+To update the application, you should first pull the latest changes from the git repository. Then, you will need to wipe the Redis database and the local cache to ensure that you are using the latest data.
+
+### 1\. Update the Code
+
+```bash
+git pull
+```
+
+### 2\. Wipe the Redis Database
+
+```bash
+redis-cli FLUSHALL
+```
+
+### 3\. Wipe the Local Cache
+
+```bash
+rm -rf cache/*
+```
+
+### 4\. Restart the Service
+
+```bash
+sudo systemctl restart dnsrecon.service
+```
+
+-----
+
+## Extensibility
+
+DNSRecon is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
+
+  * `get_name()`: Return the name of the provider.
+  * `get_display_name()`: Return a display-friendly name for the provider.
+  * `requires_api_key()`: Return `True` if the provider requires an API key.
+  * `get_eligibility()`: Return a dictionary indicating whether the provider can query domains and/or IPs.
+  * `is_available()`: Return `True` if the provider is available (e.g., if an API key is configured).
+  * `query_domain(domain)`: Query the provider for information about a domain.
+  * `query_ip(ip)`: Query the provider for information about an IP address.
+
+-----
+
+## Unique Capabilities and Limitations
+
+### Unique Capabilities
+
+  * **Graph-Based Analysis**: The use of a graph-based data model allows for a more intuitive and powerful analysis of the relationships between different pieces of information.
+  * **Real-Time Visualization**: The real-time visualization of the graph provides immediate feedback and allows for a more interactive and engaging analysis experience.
+  * **Session Management**: The session management feature allows multiple users to use the application concurrently without interfering with each other's work.
+
+### Limitations
+
+  * **Passive-Only by Default**: While the passive-only approach is a key feature of the tool, it also means that the information it can gather is limited to what is publicly available.
+  * **No Active Scanning**: The tool does not perform any active scanning, such as port scanning or vulnerability scanning.
+
+-----
+
 ## License
 
 This project is licensed under the terms of the **BSD-3-Clause** license.

core/graph_manager.py

@@ -471,7 +471,8 @@ class GraphManager:
                 'attributes': attrs.get('attributes', []), # Raw attributes list
                 'description': attrs.get('description', ''),
                 'metadata': attrs.get('metadata', {}),
-                'added_timestamp': attrs.get('added_timestamp')
+                'added_timestamp': attrs.get('added_timestamp'),
+                'max_depth_reached': attrs.get('metadata', {}).get('max_depth_reached', False)
             }
             
             # Add incoming and outgoing edges to node data

core/scanner.py

@@ -608,9 +608,13 @@ class Scanner:
             else:
                 target_type = NodeType.DOMAIN
             
+            # Add max_depth_reached flag
+            max_depth_reached = current_depth >= self.max_depth
+
             # Create or update nodes with proper types
             self.graph.add_node(source_node, source_type)
-            self.graph.add_node(target_node, target_type)
+            self.graph.add_node(target_node, target_type, metadata={'max_depth_reached': max_depth_reached})
+
 
             # Add the relationship edge
             if self.graph.add_edge(
@@ -623,7 +627,7 @@ class Scanner:
                 pass  # Edge was successfully added
 
             # Add target to discovered nodes for further processing
-            if _is_valid_domain(target_node) or _is_valid_ip(target_node):
+            if (_is_valid_domain(target_node) or _is_valid_ip(target_node)) and not max_depth_reached:
                 discovered_targets.add(target_node)
 
         return discovered_targets, False

static/js/graph.js

@@ -559,6 +559,11 @@ class GraphManager {
             outgoing_edges: node.outgoing_edges || []
         };
 
+        if (node.max_depth_reached) {
+            processedNode.borderColor = '#ff0000'; // Red border for max depth
+        }
+
+
         // Add confidence-based styling
         if (node.confidence) {
             processedNode.borderWidth = Math.max(2, Math.floor(node.confidence * 5));