attempt bugfix

Reviewed-on: mstoeck3/dnsrecon#4

.env.example

@@ -0,0 +1,36 @@
+# ===============================================
+# DNScope Environment Variables
+# ===============================================
+# Copy this file to .env and fill in your values.
+
+# --- API Keys ---
+# Add your Shodan API key for the Shodan provider to be enabled.
+SHODAN_API_KEY=
+
+# --- Flask & Session Settings ---
+# A strong, random secret key is crucial for session security.
+FLASK_SECRET_KEY=your-very-secret-and-random-key-here
+FLASK_HOST=127.0.0.1
+FLASK_PORT=5000
+FLASK_DEBUG=True
+# How long a user's session in the browser lasts (in hours).
+FLASK_PERMANENT_SESSION_LIFETIME_HOURS=2
+# How long inactive scanner data is stored in Redis (in minutes).
+SESSION_TIMEOUT_MINUTES=60
+
+
+# --- Application Core Settings ---
+# The default number of levels to recurse when scanning.
+DEFAULT_RECURSION_DEPTH=2
+# Default timeout for provider API requests in seconds.
+DEFAULT_TIMEOUT=30
+# The number of concurrent provider requests to make.
+MAX_CONCURRENT_REQUESTS=1
+# The number of results from a provider that triggers the "large entity" grouping.
+LARGE_ENTITY_THRESHOLD=100
+# The number of times to retry a target if a provider fails.
+MAX_RETRIES_PER_TARGET=8
+# How long cached provider responses are stored (in hours).
+CACHE_TIMEOUT_HOURS=12
+
+GRAPH_POLLING_NODE_THRESHOLD=100

.gitignore

@@ -168,3 +168,5 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
 
+dump.rdb
+cache/

README.md

@@ -1,107 +1,292 @@
-# DNS Reconnaissance Tool
+# DNScope - Passive Infrastructure Reconnaissance Tool
 
-A comprehensive DNS reconnaissance tool designed for investigators to gather intelligence on hostnames and IP addresses through multiple data sources.
+DNScope is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
+
+**Repo Link:** [https://github.com/overcuriousity/DNScope](https://github.com/overcuriousity/DNScope)
+
+-----
+
+## Concept and Philosophy
+
+The core philosophy of DNScope is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNScope will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
+
+For power users who require more in-depth information, DNScope can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
+
+-----
+
+-----
 
 ## Features
 
-- **DNS Resolution**: Query multiple DNS servers (1.1.1.1, 8.8.8.8, 9.9.9.9)
-- **TLD Expansion**: Automatically try all IANA TLDs for hostname-only inputs
-- **Certificate Transparency**: Query crt.sh for SSL certificate information
-- **Recursive Discovery**: Automatically discover and analyze subdomains
-- **External Intelligence**: Optional Shodan and VirusTotal integration
-- **Multiple Interfaces**: Both CLI and web interface available
-- **Comprehensive Reports**: JSON and text output formats
+  * **Passive Reconnaissance**: Gathers data without direct contact with target infrastructure.
+  * **In-Memory Graph Analysis**: Uses NetworkX for efficient relationship mapping.
+  * **Real-Time Visualization**: The graph updates dynamically as the scan progresses.
+  * **Forensic Logging**: A complete audit trail of all reconnaissance activities is maintained.
+  * **Session Management**: Supports concurrent user sessions with isolated scanner instances.
+  * **Extensible Provider Architecture**: Easily add new data sources to expand the tool's capabilities.
+  * **Web-Based UI**: An intuitive and interactive web interface for managing scans and visualizing results.
+  * **Export Options**: Export scan results to JSON, a list of targets to a text file, or an executive summary.
+  * **API Key Management**: Securely manage API keys for various providers through the web interface.
+  * **Provider Management**: Enable or disable providers for the current session.
 
-## Installation
+-----
+
+## Technical Architecture
+
+DNScope is a web-based application built with a modern technology stack:
+
+  * **Backend**: The backend is a **Flask** application that provides a REST API for the frontend and manages the scanning process.
+  * **Scanning Engine**: The core scanning engine is a multi-threaded Python application that uses a provider-based architecture to query different data sources.
+  * **Session Management**: **Redis** is used for session management, allowing for concurrent user sessions with isolated scanner instances.
+  * **Data Storage**: The application uses an in-memory graph to store and analyze the relationships between different pieces of information. The graph is built using the **NetworkX** library.
+  * **Frontend**: The frontend is a single-page application that uses JavaScript to interact with the backend API and visualize the graph.
+
+-----
+
+## Data Sources
+
+DNScope queries the following data sources:
+
+  * **DNS**: Standard DNS lookups (A, AAAA, CNAME, MX, NS, SOA, TXT).
+  * **crt.sh**: A certificate transparency log that provides information about SSL/TLS certificates.
+  * **Shodan**: A search engine for internet-connected devices (requires an API key).
+
+-----
+
+## Installation and Setup
+
+### Prerequisites
+
+  * Python 3.8 or higher
+  * A modern web browser with JavaScript enabled
+  * A Linux host for running the application
+  * Redis Server
+
+### 1\. Install Redis
+
+It is recommended to install Redis from the official repositories.
+
+**On Debian/Ubuntu:**
 
 ```bash
-# Clone or create the project structure
-mkdir dns-recon-tool && cd dns-recon-tool
+sudo apt-get update
+sudo apt-get install redis-server
+```
 
-# Install dependencies
+**On CentOS/RHEL:**
+
+```bash
+sudo yum install redis
+sudo systemctl start redis
+sudo systemctl enable redis
+```
+
+You can verify that Redis is running with the following command:
+
+```bash
+redis-cli ping
+```
+
+You should see `PONG` as the response.
+
+### 2\. Clone the Project
+
+```bash
+git clone https://github.com/overcuriousity/DNScope
+cd DNScope
+```
+
+### 3\. Install Python Dependencies
+
+It is highly recommended to use a virtual environment:
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
 pip install -r requirements.txt
 ```
 
-## Usage
+The `requirements.txt` file contains the following dependencies:
 
-### Command Line Interface
+  * Flask
+  * networkx
+  * requests
+  * python-dateutil
+  * Werkzeug
+  * urllib3
+  * dnspython
+  * gunicorn
+  * redis
+  * python-dotenv
+  * psycopg2-binary
+
+### 4\. Configure the Application
+
+DNScope is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
 
 ```bash
-# Basic domain scan
-python -m src.main example.com
-
-# Try all TLDs for hostname
-python -m src.main example
-
-# With API keys and custom depth
-python -m src.main example.com --shodan-key YOUR_KEY --virustotal-key YOUR_KEY --max-depth 3
-
-# Save reports
-python -m src.main example.com --output results
-
-# JSON only output
-python -m src.main example.com --json-only
+cp .env.example .env
 ```
 
-### Web Interface
+The following environment variables are available for configuration:
+
+| Variable | Description | Default |
+| :--- | :--- | :--- |
+| `SHODAN_API_KEY` | Your Shodan API key. | |
+| `FLASK_SECRET_KEY`| A strong, random secret key for session security. | `your-very-secret-and-random-key-here` |
+| `FLASK_HOST` | The host address for the Flask application. | `127.0.0.1` |
+| `FLASK_PORT` | The port for the Flask application. | `5000` |
+| `FLASK_DEBUG` | Enable or disable Flask's debug mode. | `True` |
+| `FLASK_PERMANENT_SESSION_LIFETIME_HOURS`| How long a user's session in the browser lasts (in hours). | `2` |
+| `SESSION_TIMEOUT_MINUTES` | How long inactive scanner data is stored in Redis (in minutes). | `60` |
+| `DEFAULT_RECURSION_DEPTH` | The default number of levels to recurse when scanning. | `2` |
+| `DEFAULT_TIMEOUT` | Default timeout for provider API requests in seconds. | `30` |
+| `MAX_CONCURRENT_REQUESTS`| The number of concurrent provider requests to make. | `5` |
+| `LARGE_ENTITY_THRESHOLD`| The number of results from a provider that triggers the "large entity" grouping. | `100` |
+| `MAX_RETRIES_PER_TARGET`| The number of times to retry a target if a provider fails. | `8` |
+| `CACHE_EXPIRY_HOURS`| How long cached provider responses are stored (in hours). | `12` |
+
+-----
+
+## Running the Application
+
+For development, you can run the application using the following command:
 
 ```bash
-# Start web server
-python -m src.main --web
-
-# Custom port
-python -m src.main --web --port 8080
+python app.py
 ```
 
-Then open http://localhost:5000 in your browser.
-
-## Configuration
-
-The tool uses the following default settings:
-- DNS Servers: 1.1.1.1, 8.8.8.8, 9.9.9.9
-- Max Recursion Depth: 2
-- Rate Limits: DNS (10/s), crt.sh (2/s), Shodan (0.5/s), VirusTotal (0.25/s)
-
-## API Keys
-
-For enhanced reconnaissance, obtain API keys from:
-- [Shodan](https://shodan.io) - Port scanning and service detection
-- [VirusTotal](https://virustotal.com) - Security analysis and reputation
-
-## Output
-
-The tool generates two types of reports:
-
-### JSON Report
-Complete machine-readable data including:
-- All discovered hostnames and IPs
-- DNS records by type
-- Certificate information
-- External service results
-- Metadata and timing
-
-### Text Report
-Human-readable summary with:
-- Executive summary
-- Hostnames by discovery depth
-- IP address analysis
-- DNS record details
-- Certificate analysis
-- Security findings
-
-## Architecture
+For production, it is recommended to use a more robust server, such as Gunicorn:
 
+```bash
+gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
 ```
-src/
-├── main.py              # CLI entry point
-├── web_app.py           # Flask web interface  
-├── config.py            # Configuration management
-├── data_structures.py   # Data models
-├── dns_resolver.py      # DNS functionality
-├── certificate_checker.py # crt.sh integration
-├── shodan_client.py     # Shodan API
-├── virustotal_client.py # VirusTotal API
-├── tld_fetcher.py       # IANA TLD handling
-├── reconnaissance.py    # Main logic
-└── report_generator.py  # Report generation
+
+-----
+
+## Systemd Service
+
+To run DNScope as a service that starts automatically on boot, you can use `systemd`.
+
+### 1\. Create a `.service` file
+
+Create a new service file in `/etc/systemd/system/`:
+
+```bash
+sudo nano /etc/systemd/system/DNScope.service
 ```
+
+### 2\. Add the Service Configuration
+
+Paste the following configuration into the file. **Remember to replace `/path/to/your/DNScope` and `your_user` with your actual project path and username.**
+
+```ini
+[Unit]
+Description=DNScope Application
+After=network.target
+
+[Service]
+User=your_user
+Group=your_user
+WorkingDirectory=/path/to/your/DNScope
+ExecStart=/path/to/your/DNScope/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+Restart=always
+Environment="SECRET_KEY=your-super-secret-and-random-key"
+Environment="FLASK_ENV=production"
+Environment="FLASK_DEBUG=False"
+Environment="SHODAN_API_KEY=your_shodan_key"
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### 3\. Enable and Start the Service
+
+Reload the `systemd` daemon, enable the service to start on boot, and then start it immediately:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable DNScope.service
+sudo systemctl start DNScope.service
+```
+
+You can check the status of the service at any time with:
+
+```bash
+sudo systemctl status DNScope.service
+```
+
+-----
+
+## Updating the Application
+
+To update the application, you should first pull the latest changes from the git repository. Then, you will need to wipe the Redis database and the local cache to ensure that you are using the latest data.
+
+### 1\. Update the Code
+
+```bash
+git pull
+```
+
+### 2\. Wipe the Redis Database
+
+```bash
+redis-cli FLUSHALL
+```
+
+### 3\. Wipe the Local Cache
+
+```bash
+rm -rf cache/*
+```
+
+### 4\. Restart the Service
+
+```bash
+sudo systemctl restart DNScope.service
+```
+
+-----
+
+## Extensibility
+
+DNScope is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
+
+  * `get_name()`: Return the name of the provider.
+  * `get_display_name()`: Return a display-friendly name for the provider.
+  * `requires_api_key()`: Return `True` if the provider requires an API key.
+  * `get_eligibility()`: Return a dictionary indicating whether the provider can query domains and/or IPs.
+  * `is_available()`: Return `True` if the provider is available (e.g., if an API key is configured).
+  * `query_domain(domain)`: Query the provider for information about a domain.
+  * `query_ip(ip)`: Query the provider for information about an IP address.
+
+-----
+
+## Unique Capabilities and Limitations
+
+### Unique Capabilities
+
+  * **Graph-Based Analysis**: The use of a graph-based data model allows for a more intuitive and powerful analysis of the relationships between different pieces of information.
+  * **Real-Time Visualization**: The real-time visualization of the graph provides immediate feedback and allows for a more interactive and engaging analysis experience.
+  * **Session Management**: The session management feature allows multiple users to use the application concurrently without interfering with each other's work.
+
+### Limitations
+
+  * **Passive-Only by Default**: While the passive-only approach is a key feature of the tool, it also means that the information it can gather is limited to what is publicly available.
+  * **No Active Scanning**: The tool does not perform any active scanning, such as port scanning or vulnerability scanning.
+
+-----
+
+## License
+
+This project is licensed under the terms of the **BSD-3-Clause** license.
+
+Copyright (c) 2025 mstoeck3.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1.  Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2.  Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+3.  Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

app.py

@@ -0,0 +1,616 @@
+# DNScope-reduced/app.py
+
+"""
+Flask application entry point for DNScope web interface.
+Provides REST API endpoints and serves the web interface with user session support.
+UPDATED: Added /api/config endpoint for graph polling optimization settings.
+"""
+
+import json
+import traceback
+from flask import Flask, render_template, request, jsonify, send_file, session
+from datetime import datetime, timezone, timedelta
+import io
+import os
+
+from core.session_manager import session_manager
+from config import config
+from core.graph_manager import NodeType
+from utils.helpers import is_valid_target
+from utils.export_manager import export_manager
+from decimal import Decimal
+
+
+app = Flask(__name__)
+app.config['SECRET_KEY'] = config.flask_secret_key
+app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=config.flask_permanent_session_lifetime_hours)
+
+def get_user_scanner():
+    """
+    Retrieves the scanner for the current session, or creates a new one if none exists.
+    """
+    current_flask_session_id = session.get('DNScope_session_id')
+    
+    if current_flask_session_id:
+        existing_scanner = session_manager.get_session(current_flask_session_id)
+        if existing_scanner:
+            return current_flask_session_id, existing_scanner
+    
+    new_session_id = session_manager.create_session()
+    new_scanner = session_manager.get_session(new_session_id)
+    
+    if not new_scanner:
+        raise Exception("Failed to create new scanner session")
+    
+    session['DNScope_session_id'] = new_session_id
+    session.permanent = True
+    
+    return new_session_id, new_scanner
+
+
+@app.route('/')
+def index():
+    """Serve the main web interface."""
+    return render_template('index.html')
+
+
+@app.route('/api/config', methods=['GET'])
+def get_config():
+    """Get configuration settings for frontend."""
+    try:
+        return jsonify({
+            'success': True,
+            'config': {
+                'graph_polling_node_threshold': config.graph_polling_node_threshold
+            }
+        })
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+@app.route('/api/scan/start', methods=['POST'])
+def start_scan():
+    """
+    Starts a new reconnaissance scan.
+    """
+    try:
+        data = request.get_json()
+        if not data or 'target' not in data:
+            return jsonify({'success': False, 'error': 'Missing target in request'}), 400
+        
+        target = data['target'].strip()
+        max_depth = data.get('max_depth', config.default_recursion_depth)
+        clear_graph = data.get('clear_graph', True)
+        force_rescan_target = data.get('force_rescan_target', None)
+        
+        if not target:
+            return jsonify({'success': False, 'error': 'Target cannot be empty'}), 400
+        if not is_valid_target(target):
+            return jsonify({'success': False, 'error': 'Invalid target format.'}), 400
+        if not isinstance(max_depth, int) or not 1 <= max_depth <= 5:
+            return jsonify({'success': False, 'error': 'Max depth must be an integer between 1 and 5'}), 400
+        
+        user_session_id, scanner = get_user_scanner()
+        
+        if not scanner:
+            return jsonify({'success': False, 'error': 'Failed to get scanner instance.'}), 500
+        
+        success = scanner.start_scan(target, max_depth, clear_graph=clear_graph, force_rescan_target=force_rescan_target)
+        
+        if success:
+            return jsonify({
+                'success': True,
+                'message': 'Reconnaissance scan started successfully',
+                'scan_id': scanner.logger.session_id,
+                'user_session_id': user_session_id
+            })
+        else:
+            return jsonify({
+                'success': False,
+                'error': f'Failed to start scan (scanner status: {scanner.status})',
+            }), 409
+        
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+@app.route('/api/scan/stop', methods=['POST'])
+def stop_scan():
+    """Stop the current scan."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No scanner found for session'}), 404
+        
+        if not scanner.session_id:
+            scanner.session_id = user_session_id
+        
+        scanner.stop_scan()
+        session_manager.set_stop_signal(user_session_id)
+        session_manager.update_scanner_status(user_session_id, 'stopped')
+        session_manager.update_session_scanner(user_session_id, scanner)
+        
+        return jsonify({
+            'success': True,
+            'message': 'Scan stop requested',
+            'user_session_id': user_session_id
+        })
+        
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+@app.route('/api/scan/status', methods=['GET'])
+def get_scan_status():
+    """Get current scan status."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        
+        if not scanner:
+            return jsonify({
+                'success': True,
+                'status': {
+                    'status': 'idle', 'target_domain': None, 'current_depth': 0,
+                    'max_depth': 0, 'progress_percentage': 0.0,
+                    'user_session_id': user_session_id
+                }
+            })
+        
+        if not scanner.session_id:
+            scanner.session_id = user_session_id
+        
+        status = scanner.get_scan_status()
+        status['user_session_id'] = user_session_id
+        
+        return jsonify({'success': True, 'status': status})
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({
+            'success': False, 'error': f'Internal server error: {str(e)}',
+            'fallback_status': {'status': 'error', 'progress_percentage': 0.0}
+        }), 500
+
+
+
+@app.route('/api/graph', methods=['GET'])
+def get_graph_data():
+    """Get current graph data."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        
+        empty_graph = {
+            'nodes': [], 'edges': [],
+            'statistics': {'node_count': 0, 'edge_count': 0}
+        }
+        
+        if not scanner:
+            return jsonify({'success': True, 'graph': empty_graph, 'user_session_id': user_session_id})
+        
+        graph_data = scanner.get_graph_data() or empty_graph
+        
+        return jsonify({'success': True, 'graph': graph_data, 'user_session_id': user_session_id})
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({
+            'success': False, 'error': f'Internal server error: {str(e)}',
+            'fallback_graph': {'nodes': [], 'edges': [], 'statistics': {}}
+        }), 500
+
+@app.route('/api/graph/large-entity/extract', methods=['POST'])
+def extract_from_large_entity():
+    """
+    FIXED: Extract a node from a large entity with proper error handling.
+    """
+    try:
+        data = request.get_json()
+        large_entity_id = data.get('large_entity_id')
+        node_id = data.get('node_id')
+
+        if not large_entity_id or not node_id:
+            return jsonify({'success': False, 'error': 'Missing required parameters'}), 400
+
+        user_session_id, scanner = get_user_scanner()
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active session found'}), 404
+
+        # FIXED: Check if node exists and provide better error messages
+        if not scanner.graph.graph.has_node(node_id):
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} not found in graph'
+            }), 404
+
+        # FIXED: Check if node is actually part of the large entity
+        node_data = scanner.graph.graph.nodes[node_id]
+        metadata = node_data.get('metadata', {})
+        current_large_entity = metadata.get('large_entity_id')
+        
+        if not current_large_entity:
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} is not part of any large entity'
+            }), 400
+            
+        if current_large_entity != large_entity_id:
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} belongs to large entity {current_large_entity}, not {large_entity_id}'
+            }), 400
+
+        # FIXED: Check if large entity exists
+        if not scanner.graph.graph.has_node(large_entity_id):
+            return jsonify({
+                'success': False, 
+                'error': f'Large entity {large_entity_id} not found'
+            }), 404
+
+        # Perform the extraction
+        success = scanner.extract_node_from_large_entity(large_entity_id, node_id)
+        
+        if success:
+            # Force immediate session state update
+            session_manager.update_session_scanner(user_session_id, scanner)
+            
+            return jsonify({
+                'success': True, 
+                'message': f'Node {node_id} extracted successfully from {large_entity_id}.',
+                'extracted_node': node_id,
+                'large_entity': large_entity_id
+            })
+        else:
+            # This should not happen with the improved checks above, but handle it gracefully
+            return jsonify({
+                'success': False, 
+                'error': f'Failed to extract node {node_id} from {large_entity_id}. Node may have already been extracted.'
+            }), 409
+
+    except json.JSONDecodeError:
+        return jsonify({'success': False, 'error': 'Invalid JSON in request body'}), 400
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({
+            'success': False, 
+            'error': f'Internal server error: {str(e)}',
+            'error_type': type(e).__name__
+        }), 500
+
+@app.route('/api/graph/node/<node_id>', methods=['DELETE'])
+def delete_graph_node(node_id):
+    """Delete a node from the graph."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active session found'}), 404
+
+        success = scanner.graph.remove_node(node_id)
+        
+        if success:
+            session_manager.update_session_scanner(user_session_id, scanner)
+            return jsonify({'success': True, 'message': f'Node {node_id} deleted successfully.'})
+        else:
+            return jsonify({'success': False, 'error': f'Node {node_id} not found.'}), 404
+
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+@app.route('/api/graph/revert', methods=['POST'])
+def revert_graph_action():
+    """Reverts a graph action, such as re-adding a deleted node."""
+    try:
+        data = request.get_json()
+        if not data or 'type' not in data or 'data' not in data:
+            return jsonify({'success': False, 'error': 'Invalid revert request format'}), 400
+
+        user_session_id, scanner = get_user_scanner()
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active session found'}), 404
+
+        action_type = data['type']
+        action_data = data['data']
+        
+        if action_type == 'delete':
+            node_to_add = action_data.get('node')
+            if node_to_add:
+                scanner.graph.add_node(
+                    node_id=node_to_add['id'],
+                    node_type=NodeType(node_to_add['type']),
+                    attributes=node_to_add.get('attributes'),
+                    description=node_to_add.get('description'),
+                    metadata=node_to_add.get('metadata')
+                )
+            
+            edges_to_add = action_data.get('edges', [])
+            for edge in edges_to_add:
+                if scanner.graph.graph.has_node(edge['from']) and scanner.graph.graph.has_node(edge['to']):
+                    scanner.graph.add_edge(
+                        source_id=edge['from'], target_id=edge['to'],
+                        relationship_type=edge['metadata']['relationship_type'],
+                        source_provider=edge['metadata']['source_provider'],
+                        raw_data=edge.get('raw_data', {})
+                    )
+            
+            session_manager.update_session_scanner(user_session_id, scanner)
+            return jsonify({'success': True, 'message': 'Delete action reverted successfully.'})
+
+        return jsonify({'success': False, 'error': f'Unknown revert action type: {action_type}'}), 400
+
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+@app.route('/api/export', methods=['GET'])
+def export_results():
+    """Export scan results as a JSON file with improved error handling."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
+        
+        # Get export data using the new export manager
+        try:
+            results = export_manager.export_scan_results(scanner)
+        except Exception as e:
+            return jsonify({'success': False, 'error': f'Failed to gather export data: {str(e)}'}), 500
+        
+        # Add user session metadata
+        results['export_metadata']['user_session_id'] = user_session_id
+        results['export_metadata']['forensic_integrity'] = 'maintained'
+        
+        # Generate filename
+        filename = export_manager.generate_filename(
+            target=scanner.current_target or 'unknown',
+            export_type='json'
+        )
+        
+        # Serialize with export manager
+        try:
+            json_data = export_manager.serialize_to_json(results)
+        except Exception as e:
+            return jsonify({
+                'success': False, 
+                'error': f'JSON serialization failed: {str(e)}'
+            }), 500
+        
+        # Create file object
+        file_obj = io.BytesIO(json_data.encode('utf-8'))
+        
+        return send_file(
+            file_obj, 
+            as_attachment=True,
+            download_name=filename, 
+            mimetype='application/json'
+        )
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({
+            'success': False, 
+            'error': f'Export failed: {str(e)}',
+            'error_type': type(e).__name__
+        }), 500
+
+@app.route('/api/export/targets', methods=['GET'])
+def export_targets():
+    """Export all discovered targets as a TXT file."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
+
+        # Use export manager for targets export
+        targets_txt = export_manager.export_targets_list(scanner)
+        
+        # Generate filename using export manager
+        filename = export_manager.generate_filename(
+            target=scanner.current_target or 'unknown',
+            export_type='targets'
+        )
+        
+        file_obj = io.BytesIO(targets_txt.encode('utf-8'))
+        
+        return send_file(
+            file_obj,
+            as_attachment=True,
+            download_name=filename,
+            mimetype='text/plain'
+        )
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Export failed: {str(e)}'}), 500
+
+
+@app.route('/api/export/summary', methods=['GET'])
+def export_summary():
+    """Export an executive summary as a TXT file."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        if not scanner:
+            return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
+
+        # Use export manager for summary generation
+        summary_txt = export_manager.generate_executive_summary(scanner)
+
+        # Generate filename using export manager
+        filename = export_manager.generate_filename(
+            target=scanner.current_target or 'unknown',
+            export_type='summary'
+        )
+        
+        file_obj = io.BytesIO(summary_txt.encode('utf-8'))
+        
+        return send_file(
+            file_obj,
+            as_attachment=True,
+            download_name=filename,
+            mimetype='text/plain'
+        )
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Export failed: {str(e)}'}), 500
+
+@app.route('/api/config/api-keys', methods=['POST'])
+def set_api_keys():
+    """Set API keys for the current session."""
+    try:
+        data = request.get_json()
+        if data is None:
+            return jsonify({'success': False, 'error': 'No API keys provided'}), 400
+        
+        user_session_id, scanner = get_user_scanner()
+        session_config = scanner.config
+        
+        updated_providers = []
+        
+        for provider_name, api_key in data.items():
+            api_key_value = str(api_key or '').strip()
+            success = session_config.set_api_key(provider_name.lower(), api_key_value)
+            
+            if success:
+                updated_providers.append(provider_name)
+        
+        if updated_providers:
+            scanner._initialize_providers()
+            session_manager.update_session_scanner(user_session_id, scanner)
+            
+            return jsonify({
+                'success': True,
+                'message': f'API keys updated for: {", ".join(updated_providers)}',
+                'user_session_id': user_session_id
+            })
+        else:
+            return jsonify({'success': False, 'error': 'No valid API keys were provided.'}), 400
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+@app.route('/api/providers', methods=['GET'])
+def get_providers():
+    """Get enhanced information about available providers including API key sources."""
+    try:
+        user_session_id, scanner = get_user_scanner()
+        base_provider_info = scanner.get_provider_info()
+        
+        # Enhance provider info with API key source information
+        enhanced_provider_info = {}
+        
+        for provider_name, info in base_provider_info.items():
+            enhanced_info = dict(info)  # Copy base info
+            
+            if info['requires_api_key']:
+                # Determine API key source and configuration status
+                api_key = scanner.config.get_api_key(provider_name)
+                backend_api_key = os.getenv(f'{provider_name.upper()}_API_KEY')
+                
+                if backend_api_key:
+                    # API key configured via backend/environment
+                    enhanced_info.update({
+                        'api_key_configured': True,
+                        'api_key_source': 'backend',
+                        'api_key_help': f'API key configured via environment variable {provider_name.upper()}_API_KEY'
+                    })
+                elif api_key:
+                    # API key configured via web interface
+                    enhanced_info.update({
+                        'api_key_configured': True,
+                        'api_key_source': 'frontend',
+                        'api_key_help': f'API key set via web interface (session-only)'
+                    })
+                else:
+                    # No API key configured
+                    enhanced_info.update({
+                        'api_key_configured': False,
+                        'api_key_source': None,
+                        'api_key_help': f'Requires API key to enable {info["display_name"]} integration'
+                    })
+            else:
+                # Provider doesn't require API key
+                enhanced_info.update({
+                    'api_key_configured': True,  # Always "configured" for non-API providers
+                    'api_key_source': None,
+                    'api_key_help': None
+                })
+            
+            enhanced_provider_info[provider_name] = enhanced_info
+        
+        return jsonify({
+            'success': True, 
+            'providers': enhanced_provider_info, 
+            'user_session_id': user_session_id
+        })
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+@app.route('/api/config/providers', methods=['POST'])
+def configure_providers():
+    """Configure provider settings (enable/disable)."""
+    try:
+        data = request.get_json()
+        if data is None:
+            return jsonify({'success': False, 'error': 'No provider settings provided'}), 400
+        
+        user_session_id, scanner = get_user_scanner()
+        session_config = scanner.config
+        
+        updated_providers = []
+        
+        for provider_name, settings in data.items():
+            provider_name_clean = provider_name.lower().strip()
+            
+            if 'enabled' in settings:
+                # Update the enabled state in session config
+                session_config.enabled_providers[provider_name_clean] = settings['enabled']
+                updated_providers.append(provider_name_clean)
+        
+        if updated_providers:
+            # Reinitialize providers with new settings
+            scanner._initialize_providers()
+            session_manager.update_session_scanner(user_session_id, scanner)
+            
+            return jsonify({
+                'success': True,
+                'message': f'Provider settings updated for: {", ".join(updated_providers)}',
+                'user_session_id': user_session_id
+            })
+        else:
+            return jsonify({'success': False, 'error': 'No valid provider settings were provided.'}), 400
+    
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
+
+@app.errorhandler(404)
+def not_found(error):
+    """Handle 404 errors."""
+    return jsonify({'success': False, 'error': 'Endpoint not found'}), 404
+
+
+@app.errorhandler(500)
+def internal_error(error):
+    """Handle 500 errors."""
+    traceback.print_exc()
+    return jsonify({'success': False, 'error': 'Internal server error'}), 500
+
+
+if __name__ == '__main__':
+    config.load_from_env()
+    app.run(
+        host=config.flask_host,
+        port=config.flask_port,
+        debug=config.flask_debug,
+        threaded=True
+    )

config.py

@@ -0,0 +1,163 @@
+# DNScope-reduced/config.py
+
+"""
+Configuration management for DNScope tool.
+Handles API key storage, rate limiting, and default settings.
+"""
+
+import os
+from typing import Dict, Optional
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
+
+class Config:
+    """Configuration manager for DNScope application."""
+    
+    def __init__(self):
+        """Initialize configuration with default values."""
+        self.api_keys: Dict[str, Optional[str]] = {}
+        
+        # --- General Settings ---
+        self.default_recursion_depth = 2
+        self.default_timeout = 60
+        self.max_concurrent_requests = 1
+        self.large_entity_threshold = 100
+        self.max_retries_per_target = 8
+        
+        # --- Graph Polling Performance Settings ---
+        self.graph_polling_node_threshold = 100  # Stop graph auto-polling above this many nodes
+        
+        # --- Provider Caching Settings ---
+        self.cache_timeout_hours = 6  # Provider-specific cache timeout
+        
+        # --- Rate Limiting (requests per minute) ---
+        self.rate_limits = {
+            'crtsh': 5,
+            'shodan': 60,
+            'dns': 100,
+            'correlation': 0 # Set to 0 to make sure correlations run last
+        }
+        
+        # --- Provider Settings ---
+        self.enabled_providers = {
+            'crtsh': True,
+            'dns': True,
+            'shodan': False,
+            'correlation': True # Enable the new provider by default
+        }
+        
+        # --- Logging ---
+        self.log_level = 'INFO'
+        self.log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        
+        # --- Flask & Session Settings ---
+        self.flask_host = '127.0.0.1'
+        self.flask_port = 5000
+        self.flask_debug = True
+        self.flask_secret_key = 'default-secret-key-change-me'
+        self.flask_permanent_session_lifetime_hours = 2
+        self.session_timeout_minutes = 60
+        
+        # Load environment variables to override defaults
+        self.load_from_env()
+        
+    def load_from_env(self):
+        """Load configuration from environment variables."""
+        self.set_api_key('shodan', os.getenv('SHODAN_API_KEY'))
+
+        # Override settings from environment
+        self.default_recursion_depth = int(os.getenv('DEFAULT_RECURSION_DEPTH', self.default_recursion_depth))
+        self.default_timeout = int(os.getenv('DEFAULT_TIMEOUT', self.default_timeout))
+        self.max_concurrent_requests = int(os.getenv('MAX_CONCURRENT_REQUESTS', self.max_concurrent_requests))
+        self.large_entity_threshold = int(os.getenv('LARGE_ENTITY_THRESHOLD', self.large_entity_threshold))
+        self.max_retries_per_target = int(os.getenv('MAX_RETRIES_PER_TARGET', self.max_retries_per_target))
+        self.cache_timeout_hours = int(os.getenv('CACHE_TIMEOUT_HOURS', self.cache_timeout_hours))
+        
+        # Override graph polling threshold from environment
+        self.graph_polling_node_threshold = int(os.getenv('GRAPH_POLLING_NODE_THRESHOLD', self.graph_polling_node_threshold))
+        
+        # Override Flask and session settings
+        self.flask_host = os.getenv('FLASK_HOST', self.flask_host)
+        self.flask_port = int(os.getenv('FLASK_PORT', self.flask_port))
+        self.flask_debug = os.getenv('FLASK_DEBUG', str(self.flask_debug)).lower() == 'true'
+        self.flask_secret_key = os.getenv('FLASK_SECRET_KEY', self.flask_secret_key)
+        self.flask_permanent_session_lifetime_hours = int(os.getenv('FLASK_PERMANENT_SESSION_LIFETIME_HOURS', self.flask_permanent_session_lifetime_hours))
+        self.session_timeout_minutes = int(os.getenv('SESSION_TIMEOUT_MINUTES', self.session_timeout_minutes))
+
+    def set_api_key(self, provider: str, api_key: Optional[str]) -> bool:
+        """Set API key for a provider."""
+        self.api_keys[provider] = api_key
+        if api_key:
+            self.enabled_providers[provider] = True
+        return True
+    
+    def set_provider_enabled(self, provider: str, enabled: bool) -> bool:
+        """
+        Set provider enabled status for the session.
+        
+        Args:
+            provider: Provider name
+            enabled: Whether the provider should be enabled
+            
+        Returns:
+            True if the setting was applied successfully
+        """
+        provider_key = provider.lower()
+        self.enabled_providers[provider_key] = enabled
+        return True
+
+    def get_provider_enabled(self, provider: str) -> bool:
+        """
+        Get provider enabled status.
+        
+        Args:
+            provider: Provider name
+            
+        Returns:
+            True if the provider is enabled
+        """
+        provider_key = provider.lower()
+        return self.enabled_providers.get(provider_key, True)  # Default to enabled
+
+    def bulk_set_provider_settings(self, provider_settings: dict) -> dict:
+        """
+        Set multiple provider settings at once.
+        
+        Args:
+            provider_settings: Dict of provider_name -> {'enabled': bool, ...}
+            
+        Returns:
+            Dict with results for each provider
+        """
+        results = {}
+        
+        for provider_name, settings in provider_settings.items():
+            provider_key = provider_name.lower()
+            
+            try:
+                if 'enabled' in settings:
+                    self.enabled_providers[provider_key] = settings['enabled']
+                    results[provider_key] = {'success': True, 'enabled': settings['enabled']}
+                else:
+                    results[provider_key] = {'success': False, 'error': 'No enabled setting provided'}
+            except Exception as e:
+                results[provider_key] = {'success': False, 'error': str(e)}
+        
+        return results
+    
+    def get_api_key(self, provider: str) -> Optional[str]:
+        """Get API key for a provider."""
+        return self.api_keys.get(provider)
+    
+    def is_provider_enabled(self, provider: str) -> bool:
+        """Check if a provider is enabled."""
+        return self.enabled_providers.get(provider, False)
+    
+    def get_rate_limit(self, provider: str) -> int:
+        """Get rate limit for a provider."""
+        return self.rate_limits.get(provider, 60)
+
+# Global configuration instance
+config = Config()

core/__init__.py

@@ -0,0 +1,25 @@
+"""
+Core modules for DNScope passive reconnaissance tool.
+Contains graph management, scanning orchestration, and forensic logging.
+"""
+
+from .graph_manager import GraphManager, NodeType
+from .scanner import Scanner, ScanStatus  
+from .logger import ForensicLogger, get_forensic_logger, new_session
+from .session_manager import session_manager 
+from .session_config import SessionConfig, create_session_config 
+
+__all__ = [
+    'GraphManager',
+    'NodeType', 
+    'Scanner',
+    'ScanStatus',
+    'ForensicLogger',
+    'get_forensic_logger',
+    'new_session',
+    'session_manager', 
+    'SessionConfig',
+    'create_session_config'
+]
+
+__version__ = "1.0.0-phase2"

core/graph_manager.py

@@ -0,0 +1,218 @@
+# DNScope-reduced/core/graph_manager.py
+
+"""
+Graph data model for DNScope using NetworkX.
+Manages in-memory graph storage with forensic metadata.
+Now fully compatible with the unified ProviderResult data model.
+UPDATED: Fixed correlation exclusion keys to match actual attribute names.
+UPDATED: Removed export_json() method - now handled by ExportManager.
+"""
+import re
+from datetime import datetime, timezone
+from enum import Enum
+from typing import Dict, List, Any, Optional, Tuple
+
+import networkx as nx
+
+
+class NodeType(Enum):
+    """Enumeration of supported node types."""
+    DOMAIN = "domain"
+    IP = "ip"
+    ISP = "isp"
+    CA = "ca"
+    LARGE_ENTITY = "large_entity"
+    CORRELATION_OBJECT = "correlation_object"
+
+    def __repr__(self):
+        return self.value
+
+
+class GraphManager:
+    """
+    Thread-safe graph manager for DNScope infrastructure mapping.
+    Uses NetworkX for in-memory graph storage.
+    Compatible with unified ProviderResult data model.
+    """
+
+    def __init__(self):
+        """Initialize empty directed graph."""
+        self.graph = nx.DiGraph()
+        self.creation_time = datetime.now(timezone.utc).isoformat()
+        self.last_modified = self.creation_time
+        
+    def add_node(self, node_id: str, node_type: NodeType, attributes: Optional[List[Dict[str, Any]]] = None,
+                description: str = "", metadata: Optional[Dict[str, Any]] = None) -> bool:
+        """
+        Add a node to the graph, update attributes, and process correlations.
+        Now compatible with unified data model - attributes are dictionaries from converted StandardAttribute objects.
+        """
+        is_new_node = not self.graph.has_node(node_id)
+        if is_new_node:
+            self.graph.add_node(node_id, type=node_type.value,
+                                added_timestamp=datetime.now(timezone.utc).isoformat(),
+                                attributes=attributes or [],  # Store as a list from the start
+                                description=description,
+                                metadata=metadata or {})
+        else:
+            # Safely merge new attributes into the existing list of attributes
+            if attributes:
+                existing_attributes = self.graph.nodes[node_id].get('attributes', [])
+                
+                # Handle cases where old data might still be in dictionary format
+                if not isinstance(existing_attributes, list):
+                    existing_attributes = []
+
+                # Create a set of existing attribute names for efficient duplicate checking
+                existing_attr_names = {attr['name'] for attr in existing_attributes}
+
+                for new_attr in attributes:
+                    if new_attr['name'] not in existing_attr_names:
+                        existing_attributes.append(new_attr)
+                        existing_attr_names.add(new_attr['name'])
+                
+                self.graph.nodes[node_id]['attributes'] = existing_attributes
+            if description:
+                self.graph.nodes[node_id]['description'] = description
+            if metadata:
+                existing_metadata = self.graph.nodes[node_id].get('metadata', {})
+                existing_metadata.update(metadata)
+                self.graph.nodes[node_id]['metadata'] = existing_metadata
+
+        self.last_modified = datetime.now(timezone.utc).isoformat()
+        return is_new_node
+
+    def add_edge(self, source_id: str, target_id: str, relationship_type: str,
+                source_provider: str = "unknown",
+                raw_data: Optional[Dict[str, Any]] = None) -> bool:
+        """
+        UPDATED: Add or update an edge between two nodes with raw relationship labels.
+        """
+        if not self.graph.has_node(source_id) or not self.graph.has_node(target_id):
+            return False
+
+        
+        # UPDATED: Use raw relationship type - no formatting
+        edge_label = relationship_type
+
+        # Add a new edge with raw attributes
+        self.graph.add_edge(source_id, target_id,
+                            relationship_type=edge_label,
+                            source_provider=source_provider,
+                            discovery_timestamp=datetime.now(timezone.utc).isoformat(),
+                            raw_data=raw_data or {})
+        self.last_modified = datetime.now(timezone.utc).isoformat()
+        return True
+
+    def remove_node(self, node_id: str) -> bool:
+        """Remove a node and its connected edges from the graph."""
+        if not self.graph.has_node(node_id):
+            return False
+
+        # Remove node from the graph (NetworkX handles removing connected edges)
+        self.graph.remove_node(node_id)
+        
+        self.last_modified = datetime.now(timezone.utc).isoformat()
+        return True
+
+    def get_node_count(self) -> int:
+        """Get total number of nodes in the graph."""
+        return self.graph.number_of_nodes()
+
+    def get_edge_count(self) -> int:
+        """Get total number of edges in the graph."""
+        return self.graph.number_of_edges()
+
+    def get_nodes_by_type(self, node_type: NodeType) -> List[str]:
+        """Get all nodes of a specific type."""
+        return [n for n, d in self.graph.nodes(data=True) if d.get('type') == node_type.value]
+
+    def get_graph_data(self) -> Dict[str, Any]:
+        """
+        Export graph data formatted for frontend visualization.
+        SIMPLIFIED: No certificate styling - frontend handles all visual styling.
+        """
+        nodes = []
+        for node_id, attrs in self.graph.nodes(data=True):
+            node_data = {
+                'id': node_id, 
+                'label': node_id, 
+                'type': attrs.get('type', 'unknown'),
+                'attributes': attrs.get('attributes', []), # Raw attributes list
+                'description': attrs.get('description', ''),
+                'metadata': attrs.get('metadata', {}),
+                'added_timestamp': attrs.get('added_timestamp'),
+                'max_depth_reached': attrs.get('metadata', {}).get('max_depth_reached', False)
+            }
+            
+            # Add incoming and outgoing edges to node data
+            if self.graph.has_node(node_id):
+                node_data['incoming_edges'] = [
+                    {'from': u, 'data': d} for u, _, d in self.graph.in_edges(node_id, data=True)
+                ]
+                node_data['outgoing_edges'] = [
+                    {'to': v, 'data': d} for _, v, d in self.graph.out_edges(node_id, data=True)
+                ]
+            
+            nodes.append(node_data)
+
+        edges = []
+        for source, target, attrs in self.graph.edges(data=True):
+            edges.append({
+                'from': source, 
+                'to': target,
+                'label': attrs.get('relationship_type', ''),
+                'source_provider': attrs.get('source_provider', ''),
+                'discovery_timestamp': attrs.get('discovery_timestamp'),
+                'raw_data': attrs.get('raw_data', {})
+            })
+        
+        return {
+            'nodes': nodes, 
+            'edges': edges,
+            'statistics': self.get_statistics()['basic_metrics']
+        }
+
+    def get_statistics(self) -> Dict[str, Any]:
+        """Get comprehensive statistics about the graph with proper empty graph handling."""
+        
+        # FIXED: Handle empty graph case properly
+        node_count = self.get_node_count()
+        edge_count = self.get_edge_count()
+        
+        stats = {
+            'basic_metrics': {
+                'total_nodes': node_count,
+                'total_edges': edge_count,
+                'creation_time': self.creation_time,
+                'last_modified': self.last_modified
+            },
+            'node_type_distribution': {}, 
+            'relationship_type_distribution': {},
+            'provider_distribution': {}
+        }
+        
+        # FIXED: Only calculate distributions if we have data
+        if node_count > 0:
+            # Calculate node type distributions
+            for node_type in NodeType:
+                count = len(self.get_nodes_by_type(node_type))
+                if count > 0:  # Only include types that exist
+                    stats['node_type_distribution'][node_type.value] = count
+        
+        if edge_count > 0:
+            # Calculate edge distributions
+            for _, _, data in self.graph.edges(data=True):
+                rel_type = data.get('relationship_type', 'unknown')
+                stats['relationship_type_distribution'][rel_type] = stats['relationship_type_distribution'].get(rel_type, 0) + 1
+                
+                provider = data.get('source_provider', 'unknown')
+                stats['provider_distribution'][provider] = stats['provider_distribution'].get(provider, 0) + 1
+        
+        return stats
+
+    def clear(self) -> None:
+        """Clear all nodes and edges from the graph."""
+        self.graph.clear()
+        self.creation_time = datetime.now(timezone.utc).isoformat()
+        self.last_modified = self.creation_time

core/logger.py

@@ -0,0 +1,282 @@
+# DNScope/core/logger.py
+
+import logging
+import threading
+from datetime import datetime
+from typing import Dict, Any, Optional, List
+from dataclasses import dataclass, asdict
+from datetime import timezone
+
+
+@dataclass
+class APIRequest:
+    """Structured representation of an API request for forensic logging."""
+    timestamp: str
+    provider: str
+    url: str
+    method: str
+    status_code: Optional[int]
+    response_size: Optional[int]
+    duration_ms: Optional[float]
+    error: Optional[str]
+    target_indicator: str
+    discovery_context: Optional[str]
+
+
+@dataclass
+class RelationshipDiscovery:
+    """Structured representation of a discovered relationship."""
+    timestamp: str
+    source_node: str
+    target_node: str
+    relationship_type: str
+    provider: str
+    raw_data: Dict[str, Any]
+    discovery_method: str
+
+
+class ForensicLogger:
+    """
+    Thread-safe forensic logging system for DNScope.
+    Maintains detailed audit trail of all reconnaissance activities.
+    """
+    
+    def __init__(self, session_id: str = ""):
+        """
+        Initialize forensic logger.
+        
+        Args:
+            session_id: Unique identifier for this reconnaissance session
+        """
+        self.session_id = session_id or self._generate_session_id()
+        self.lock = threading.Lock()
+        
+        # Initialize audit trail storage
+        self.api_requests: List[APIRequest] = []
+        self.relationships: List[RelationshipDiscovery] = []
+        self.session_metadata = {
+            'session_id': self.session_id,
+            'start_time': datetime.now(timezone.utc).isoformat(),
+            'end_time': None,
+            'total_requests': 0,
+            'total_relationships': 0,
+            'providers_used': set(),
+            'target_domains': set()
+        }
+        
+        # Configure standard logger
+        self.logger = logging.getLogger(f'DNScope.{self.session_id}')
+        self.logger.setLevel(logging.INFO)
+        
+        # Create formatter for structured logging
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        )
+        
+        # Add console handler if not already present
+        if not self.logger.handlers:
+            console_handler = logging.StreamHandler()
+            console_handler.setFormatter(formatter)
+            self.logger.addHandler(console_handler)
+
+    def __getstate__(self):
+        """Prepare ForensicLogger for pickling by excluding unpicklable objects."""
+        state = self.__dict__.copy()
+        # Remove the unpickleable 'logger' attribute
+        if 'logger' in state:
+            del state['logger']
+        if 'lock' in state:
+            del state['lock']
+        return state
+
+    def __setstate__(self, state):
+        """Restore ForensicLogger after unpickling by reconstructing logger."""
+        self.__dict__.update(state)
+        # Re-initialize the 'logger' attribute
+        self.logger = logging.getLogger(f'DNScope.{self.session_id}')
+        self.logger.setLevel(logging.INFO)
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        )
+        if not self.logger.handlers:
+            console_handler = logging.StreamHandler()
+            console_handler.setFormatter(formatter)
+            self.logger.addHandler(console_handler)
+        self.lock = threading.Lock()
+
+    def _generate_session_id(self) -> str:
+        """Generate unique session identifier."""
+        return f"DNScope_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
+    
+    def log_api_request(self, provider: str, url: str, method: str = "GET", 
+                       status_code: Optional[int] = None, 
+                       response_size: Optional[int] = None,
+                       duration_ms: Optional[float] = None,
+                       error: Optional[str] = None,
+                       target_indicator: str = "",
+                       discovery_context: Optional[str] = None) -> None:
+        """
+        Log an API request for forensic audit trail.
+        
+        Args:
+            provider: Name of the data provider
+            url: Request URL
+            method: HTTP method
+            status_code: HTTP response status code
+            response_size: Size of response in bytes
+            duration_ms: Request duration in milliseconds
+            error: Error message if request failed
+            target_indicator: The indicator being investigated
+            discovery_context: Context of how this indicator was discovered
+        """
+        api_request = APIRequest(
+            timestamp=datetime.now(timezone.utc).isoformat(),
+            provider=provider,
+            url=url,
+            method=method,
+            status_code=status_code,
+            response_size=response_size,
+            duration_ms=duration_ms,
+            error=error,
+            target_indicator=target_indicator,
+            discovery_context=discovery_context
+        )
+        
+        self.api_requests.append(api_request)
+        self.session_metadata['total_requests'] += 1
+        self.session_metadata['providers_used'].add(provider)
+        
+        if target_indicator:
+            self.session_metadata['target_domains'].add(target_indicator)
+        
+        # Log to standard logger
+        if error:
+            self.logger.error(f"API Request Failed.")
+        else:
+            self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
+    
+    def log_relationship_discovery(self, source_node: str, target_node: str,
+                                 relationship_type: str,
+                                 provider: str, raw_data: Dict[str, Any],
+                                 discovery_method: str) -> None:
+        """
+        Log discovery of a new relationship between indicators.
+        
+        Args:
+            source_node: Source node identifier
+            target_node: Target node identifier
+            relationship_type: Type of relationship (e.g., 'SAN', 'A_Record')
+            provider: Provider that discovered this relationship
+            raw_data: Raw data from provider response
+            discovery_method: Method used to discover relationship
+        """
+        relationship = RelationshipDiscovery(
+            timestamp=datetime.now(timezone.utc).isoformat(),
+            source_node=source_node,
+            target_node=target_node,
+            relationship_type=relationship_type,
+            provider=provider,
+            raw_data=raw_data,
+            discovery_method=discovery_method
+        )
+        
+        self.relationships.append(relationship)
+        self.session_metadata['total_relationships'] += 1
+        
+        self.logger.info(
+            f"Relationship Discovered - {source_node} -> {target_node} "
+            f"({relationship_type}) - Provider: {provider}"
+        )
+    
+    def log_scan_start(self, target_domain: str, recursion_depth: int, 
+                      enabled_providers: List[str]) -> None:
+        """Log the start of a reconnaissance scan."""
+        self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
+        self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
+        
+        self.session_metadata['target_domains'].add(target_domain)
+    
+    def log_scan_complete(self) -> None:
+        """Log the completion of a reconnaissance scan."""
+        self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
+        
+        self.logger.info(f"Scan Complete - Session: {self.session_id}")
+    
+    def export_audit_trail(self) -> Dict[str, Any]:
+        """
+        Export complete audit trail for forensic analysis.
+        
+        Returns:
+            Dictionary containing complete session audit trail
+        """
+        session_metadata_export = self.session_metadata.copy()
+        session_metadata_export['providers_used'] = list(session_metadata_export['providers_used'])
+        session_metadata_export['target_domains'] = list(session_metadata_export['target_domains'])
+        
+        return {
+            'session_metadata': session_metadata_export,
+            'api_requests': [asdict(req) for req in self.api_requests],
+            'relationships': [asdict(rel) for rel in self.relationships],
+            'export_timestamp': datetime.now(timezone.utc).isoformat()
+        }
+    
+    def get_forensic_summary(self) -> Dict[str, Any]:
+        """
+        Get summary statistics for forensic reporting.
+        
+        Returns:
+            Dictionary containing summary statistics
+        """
+        provider_stats = {}
+        for provider in self.session_metadata['providers_used']:
+            provider_requests = [req for req in self.api_requests if req.provider == provider]
+            provider_relationships = [rel for rel in self.relationships if rel.provider == provider]
+            
+            provider_stats[provider] = {
+                'total_requests': len(provider_requests),
+                'successful_requests': len([req for req in provider_requests if req.error is None]),
+                'failed_requests': len([req for req in provider_requests if req.error is not None]),
+                'relationships_discovered': len(provider_relationships),
+            }
+        
+        return {
+            'session_id': self.session_id,
+            'duration_minutes': self._calculate_session_duration(),
+            'total_requests': self.session_metadata['total_requests'],
+            'total_relationships': self.session_metadata['total_relationships'],
+            'unique_indicators': len(set([rel.source_node for rel in self.relationships] + [rel.target_node for rel in self.relationships])),
+            'provider_statistics': provider_stats
+        }
+    
+    def _calculate_session_duration(self) -> float:
+        """Calculate session duration in minutes."""
+        if not self.session_metadata['end_time']:
+            end_time = datetime.now(timezone.utc)
+        else:
+            end_time = datetime.fromisoformat(self.session_metadata['end_time'])
+        
+        start_time = datetime.fromisoformat(self.session_metadata['start_time'])
+        duration = (end_time - start_time).total_seconds() / 60
+        return round(duration, 2)
+
+
+# Global logger instance for the current session
+_current_logger: Optional[ForensicLogger] = None
+_logger_lock = threading.Lock()
+
+
+def get_forensic_logger() -> ForensicLogger:
+    """Get or create the current forensic logger instance."""
+    global _current_logger
+    with _logger_lock:
+        if _current_logger is None:
+            _current_logger = ForensicLogger()
+        return _current_logger
+
+
+def new_session() -> ForensicLogger:
+    """Start a new forensic logging session."""
+    global _current_logger
+    with _logger_lock:
+        _current_logger = ForensicLogger()
+        return _current_logger

core/provider_result.py

@@ -0,0 +1,89 @@
+# DNScope-reduced/core/provider_result.py
+
+"""
+Unified data model for DNScope passive reconnaissance.
+Standardizes the data structure across all providers to ensure consistent processing.
+"""
+
+from typing import Any, Optional, List, Dict
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+
+
+@dataclass
+class StandardAttribute:
+    """A unified data structure for a single piece of information about a node."""
+    target_node: str
+    name: str
+    value: Any
+    type: str
+    provider: str
+    timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+    metadata: Optional[Dict[str, Any]] = field(default_factory=dict)
+
+@dataclass
+class Relationship:
+    """A unified data structure for a directional link between two nodes."""
+    source_node: str
+    target_node: str
+    relationship_type: str
+    provider: str
+    timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+    raw_data: Optional[Dict[str, Any]] = field(default_factory=dict)
+
+@dataclass
+class ProviderResult:
+    """A container for all data returned by a provider from a single query."""
+    attributes: List[StandardAttribute] = field(default_factory=list)
+    relationships: List[Relationship] = field(default_factory=list)
+
+    def add_attribute(self, target_node: str, name: str, value: Any, attr_type: str, 
+                     provider: str, metadata: Optional[Dict[str, Any]] = None) -> None:
+        """Helper method to add an attribute to the result."""
+        self.attributes.append(StandardAttribute(
+            target_node=target_node,
+            name=name,
+            value=value,
+            type=attr_type,
+            provider=provider,
+            metadata=metadata or {}
+        ))
+
+    def add_relationship(self, source_node: str, target_node: str, relationship_type: str,
+                        provider: str, raw_data: Optional[Dict[str, Any]] = None) -> None:
+        """Helper method to add a relationship to the result."""
+        self.relationships.append(Relationship(
+            source_node=source_node,
+            target_node=target_node,
+            relationship_type=relationship_type,
+            provider=provider,
+            raw_data=raw_data or {}
+        ))
+
+    def get_discovered_nodes(self) -> set:
+        """Get all unique node identifiers discovered in this result."""
+        nodes = set()
+        
+        # Add nodes from relationships
+        for rel in self.relationships:
+            nodes.add(rel.source_node)
+            nodes.add(rel.target_node)
+            
+        # Add nodes from attributes
+        for attr in self.attributes:
+            nodes.add(attr.target_node)
+            
+        return nodes
+
+    def get_relationship_count(self) -> int:
+        """Get the total number of relationships in this result."""
+        return len(self.relationships)
+
+    def get_attribute_count(self) -> int:
+        """Get the total number of attributes in this result."""
+        return len(self.attributes)
+
+    ##TODO
+    #def is_large_entity(self, threshold: int) -> bool:
+    #    """Check if this result qualifies as a large entity based on relationship count."""
+    #    return self.get_relationship_count() > threshold

core/rate_limiter.py

@@ -0,0 +1,145 @@
+# DNScope-reduced/core/rate_limiter.py
+
+import time
+import logging
+
+class GlobalRateLimiter:
+    """
+    FIXED: Improved rate limiter with better cleanup and error handling.
+    Prevents accumulation of stale entries that cause infinite retry loops.
+    """
+    
+    def __init__(self, redis_client):
+        self.redis = redis_client
+        self.logger = logging.getLogger('DNScope.rate_limiter')
+        # Track last cleanup times to avoid excessive Redis operations
+        self._last_cleanup = {}
+
+    def is_rate_limited(self, key, limit, period):
+        """
+        FIXED: Check if a key is rate-limited with improved cleanup and error handling.
+        
+        Args:
+            key: Rate limit key (e.g., provider name)
+            limit: Maximum requests allowed
+            period: Time period in seconds (60 for per-minute)
+            
+        Returns:
+            bool: True if rate limited, False otherwise
+        """
+        if limit <= 0:
+            # Rate limit of 0 or negative means no limiting
+            return False
+            
+        now = time.time()
+        rate_key = f"rate_limit:{key}"
+        
+        try:
+            # FIXED: More aggressive cleanup to prevent accumulation
+            # Only clean up if we haven't cleaned recently (every 10 seconds max)
+            should_cleanup = (
+                rate_key not in self._last_cleanup or 
+                now - self._last_cleanup.get(rate_key, 0) > 10
+            )
+            
+            if should_cleanup:
+                # Remove entries older than the period
+                removed_count = self.redis.zremrangebyscore(rate_key, 0, now - period)
+                self._last_cleanup[rate_key] = now
+                
+                if removed_count > 0:
+                    self.logger.debug(f"Rate limiter cleaned up {removed_count} old entries for {key}")
+            
+            # Get current count
+            current_count = self.redis.zcard(rate_key)
+            
+            if current_count >= limit:
+                self.logger.debug(f"Rate limited: {key} has {current_count}/{limit} requests in period")
+                return True
+            
+            # Add new timestamp with error handling
+            try:
+                # Use pipeline for atomic operations
+                pipe = self.redis.pipeline()
+                pipe.zadd(rate_key, {str(now): now})
+                pipe.expire(rate_key, int(period * 2))  # Set TTL to 2x period for safety
+                pipe.execute()
+            except Exception as e:
+                self.logger.warning(f"Failed to record rate limit entry for {key}: {e}")
+                # Don't block the request if we can't record it
+                return False
+            
+            return False
+            
+        except Exception as e:
+            self.logger.error(f"Rate limiter error for {key}: {e}")
+            # FIXED: On Redis errors, don't block requests to avoid infinite loops
+            return False
+    
+    def get_rate_limit_status(self, key, limit, period):
+        """
+        Get detailed rate limit status for debugging.
+        
+        Returns:
+            dict: Status information including current count, limit, and time to reset
+        """
+        now = time.time()
+        rate_key = f"rate_limit:{key}"
+        
+        try:
+            current_count = self.redis.zcard(rate_key)
+            
+            # Get oldest entry to calculate reset time
+            oldest_entries = self.redis.zrange(rate_key, 0, 0, withscores=True)
+            time_to_reset = 0
+            if oldest_entries:
+                oldest_time = oldest_entries[0][1]
+                time_to_reset = max(0, period - (now - oldest_time))
+            
+            return {
+                'key': key,
+                'current_count': current_count,
+                'limit': limit,
+                'period': period,
+                'is_limited': current_count >= limit,
+                'time_to_reset': time_to_reset
+            }
+        except Exception as e:
+            self.logger.error(f"Failed to get rate limit status for {key}: {e}")
+            return {
+                'key': key,
+                'current_count': 0,
+                'limit': limit,
+                'period': period,
+                'is_limited': False,
+                'time_to_reset': 0,
+                'error': str(e)
+            }
+    
+    def reset_rate_limit(self, key):
+        """
+        ADDED: Reset rate limit for a specific key (useful for debugging).
+        """
+        rate_key = f"rate_limit:{key}"
+        try:
+            deleted = self.redis.delete(rate_key)
+            self.logger.info(f"Reset rate limit for {key} (deleted: {deleted})")
+            return True
+        except Exception as e:
+            self.logger.error(f"Failed to reset rate limit for {key}: {e}")
+            return False
+    
+    def cleanup_all_rate_limits(self):
+        """
+        ADDED: Clean up all rate limit entries (useful for maintenance).
+        """
+        try:
+            keys = self.redis.keys("rate_limit:*")
+            if keys:
+                deleted = self.redis.delete(*keys)
+                self.logger.info(f"Cleaned up {deleted} rate limit keys")
+                return deleted
+            return 0
+        except Exception as e:
+            self.logger.error(f"Failed to cleanup rate limits: {e}")
+            return 0

core/session_config.py

@@ -0,0 +1,20 @@
+"""
+Per-session configuration management for DNScope.
+Provides isolated configuration instances for each user session.
+"""
+
+from config import Config
+
+class SessionConfig(Config):
+    """
+    Session-specific configuration that inherits from global config
+    but maintains isolated API keys and provider settings.
+    """
+    
+    def __init__(self):
+        """Initialize session config with global defaults."""
+        super().__init__()
+
+def create_session_config() -> 'SessionConfig':
+    """Create a new session configuration instance."""
+    return SessionConfig()

core/session_manager.py

@@ -0,0 +1,401 @@
+# DNScope/core/session_manager.py
+
+import threading
+import time
+import uuid
+import redis
+import pickle
+from typing import Dict, Optional, Any
+
+from core.scanner import Scanner
+from config import config
+
+class SessionManager:
+    """
+    FIXED: Manages multiple scanner instances for concurrent user sessions using Redis.
+    Now more conservative about session creation to preserve API keys and configuration.
+    """
+    
+    def __init__(self, session_timeout_minutes: int = 0):
+        """
+        Initialize session manager with a Redis backend.
+        """
+        if session_timeout_minutes is None:
+            session_timeout_minutes = config.session_timeout_minutes
+
+        self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
+        self.session_timeout = session_timeout_minutes * 60  # Convert to seconds
+        self.lock = threading.Lock()
+        
+        # FIXED: Add a creation lock to prevent race conditions
+        self.creation_lock = threading.Lock()
+        
+        # Start cleanup thread
+        self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
+        self.cleanup_thread.start()
+        
+        print(f"SessionManager initialized with Redis backend and {session_timeout_minutes}min timeout")
+    
+    def __getstate__(self):
+        """Prepare SessionManager for pickling."""
+        state = self.__dict__.copy()
+        # Exclude unpickleable attributes - Redis client and threading objects
+        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock']
+        for attr in unpicklable_attrs:
+            if attr in state:
+                del state[attr]
+        return state
+
+    def __setstate__(self, state):
+        """Restore SessionManager after unpickling."""
+        self.__dict__.update(state)
+        # Re-initialize unpickleable attributes
+        self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
+        self.lock = threading.Lock()
+        self.creation_lock = threading.Lock()
+        self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
+        self.cleanup_thread.start()
+
+    def _get_session_key(self, session_id: str) -> str:
+        """Generates the Redis key for a session."""
+        return f"DNScope:session:{session_id}"
+
+    def _get_stop_signal_key(self, session_id: str) -> str:
+        """Generates the Redis key for a session's stop signal."""
+        return f"DNScope:stop:{session_id}"
+
+    def create_session(self) -> str:
+        """
+        FIXED: Create a new user session with thread-safe creation to prevent duplicates.
+        """
+        # FIXED: Use creation lock to prevent race conditions
+        with self.creation_lock:
+            session_id = str(uuid.uuid4())
+            print(f"=== CREATING SESSION {session_id} IN REDIS ===")
+            
+            try:
+                from core.session_config import create_session_config
+                session_config = create_session_config()
+                scanner_instance = Scanner(session_config=session_config)
+                
+                # Set the session ID on the scanner for cross-process stop signal management
+                scanner_instance.session_id = session_id
+                
+                session_data = {
+                    'scanner': scanner_instance,
+                    'config': session_config,
+                    'created_at': time.time(),
+                    'last_activity': time.time(),
+                    'status': 'active'
+                }
+                
+                # Serialize the entire session data dictionary using pickle
+                serialized_data = pickle.dumps(session_data)
+                
+                # Store in Redis
+                session_key = self._get_session_key(session_id)
+                self.redis_client.setex(session_key, self.session_timeout, serialized_data)
+                
+                # Initialize stop signal as False
+                stop_key = self._get_stop_signal_key(session_id)
+                self.redis_client.setex(stop_key, self.session_timeout, b'0')
+                
+                print(f"Session {session_id} stored in Redis with stop signal initialized")
+                print(f"Session has {len(scanner_instance.providers)} providers: {[p.get_name() for p in scanner_instance.providers]}")
+                return session_id
+                
+            except Exception as e:
+                print(f"ERROR: Failed to create session {session_id}: {e}")
+                raise
+
+    def set_stop_signal(self, session_id: str) -> bool:
+        """
+        Set the stop signal for a session (cross-process safe).
+        
+        Args:
+            session_id: Session identifier
+            
+        Returns:
+            bool: True if signal was set successfully
+        """
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            # Set stop signal to '1' with the same TTL as the session
+            self.redis_client.setex(stop_key, self.session_timeout, b'1')
+            print(f"Stop signal set for session {session_id}")
+            return True
+        except Exception as e:
+            print(f"ERROR: Failed to set stop signal for session {session_id}: {e}")
+            return False
+
+    def is_stop_requested(self, session_id: str) -> bool:
+        """
+        Check if stop is requested for a session (cross-process safe).
+        
+        Args:
+            session_id: Session identifier
+            
+        Returns:
+            bool: True if stop is requested
+        """
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            value = self.redis_client.get(stop_key)
+            return value == b'1' if value is not None else False
+        except Exception as e:
+            print(f"ERROR: Failed to check stop signal for session {session_id}: {e}")
+            return False
+
+    def clear_stop_signal(self, session_id: str) -> bool:
+        """
+        Clear the stop signal for a session.
+        
+        Args:
+            session_id: Session identifier
+            
+        Returns:
+            bool: True if signal was cleared successfully
+        """
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            self.redis_client.setex(stop_key, self.session_timeout, b'0')
+            print(f"Stop signal cleared for session {session_id}")
+            return True
+        except Exception as e:
+            print(f"ERROR: Failed to clear stop signal for session {session_id}: {e}")
+            return False
+
+    def _get_session_data(self, session_id: str) -> Optional[Dict[str, Any]]:
+        """Retrieves and deserializes session data from Redis."""
+        try:
+            session_key = self._get_session_key(session_id)
+            serialized_data = self.redis_client.get(session_key)
+            if serialized_data:
+                session_data = pickle.loads(serialized_data)
+                # Ensure the scanner has the correct session ID for stop signal checking
+                if 'scanner' in session_data and session_data['scanner']:
+                    session_data['scanner'].session_id = session_id
+                return session_data
+            return None
+        except Exception as e:
+            print(f"ERROR: Failed to get session data for {session_id}: {e}")
+            return None
+
+    def _save_session_data(self, session_id: str, session_data: Dict[str, Any]) -> bool:
+        """
+        Serializes and saves session data back to Redis with updated TTL.
+        
+        Returns:
+            bool: True if save was successful
+        """
+        try:
+            session_key = self._get_session_key(session_id)
+            serialized_data = pickle.dumps(session_data)
+            result = self.redis_client.setex(session_key, self.session_timeout, serialized_data)
+            return result
+        except Exception as e:
+            print(f"ERROR: Failed to save session data for {session_id}: {e}")
+            return False
+
+    def update_session_scanner(self, session_id: str, scanner: 'Scanner') -> bool:
+        """
+        Updates just the scanner object in a session with immediate persistence.
+        
+        Returns:
+            bool: True if update was successful
+        """
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data:
+                # Ensure scanner has the session ID
+                scanner.session_id = session_id
+                session_data['scanner'] = scanner
+                session_data['last_activity'] = time.time()
+                
+                # Immediately save to Redis for GUI updates
+                success = self._save_session_data(session_id, session_data)
+                if success:
+                    # Only log occasionally to reduce noise
+                    if hasattr(self, '_last_update_log'):
+                        if time.time() - self._last_update_log > 5:  # Log every 5 seconds max
+                            #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
+                            self._last_update_log = time.time()
+                    else:
+                        #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
+                        self._last_update_log = time.time()
+                else:
+                    print(f"WARNING: Failed to save scanner state for session {session_id}")
+                return success
+            else:
+                print(f"WARNING: Session {session_id} not found for scanner update")
+                return False
+        except Exception as e:
+            print(f"ERROR: Failed to update scanner for session {session_id}: {e}")
+            return False
+
+    def update_scanner_status(self, session_id: str, status: str) -> bool:
+        """
+        Quickly update just the scanner status for immediate GUI feedback.
+        
+        Args:
+            session_id: Session identifier
+            status: New scanner status
+            
+        Returns:
+            bool: True if update was successful
+        """
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data and 'scanner' in session_data:
+                session_data['scanner'].status = status
+                session_data['last_activity'] = time.time()
+                
+                success = self._save_session_data(session_id, session_data)
+                if success:
+                    print(f"Scanner status updated to '{status}' for session {session_id}")
+                else:
+                    print(f"WARNING: Failed to save status update for session {session_id}")
+                return success
+            return False
+        except Exception as e:
+            print(f"ERROR: Failed to update scanner status for session {session_id}: {e}")
+            return False
+
+    def get_session(self, session_id: str) -> Optional[Scanner]:
+        """
+        Get scanner instance for a session from Redis with session ID management.
+        """
+        if not session_id:
+            return None
+            
+        session_data = self._get_session_data(session_id)
+        
+        if not session_data or session_data.get('status') != 'active':
+            return None
+        
+        # Update last activity and save back to Redis
+        session_data['last_activity'] = time.time()
+        self._save_session_data(session_id, session_data)
+        
+        scanner = session_data.get('scanner')
+        if scanner:
+            # Ensure the scanner can check the Redis-based stop signal
+            scanner.session_id = session_id
+        
+        return scanner
+
+    def get_session_status_only(self, session_id: str) -> Optional[str]:
+        """
+        Get just the scanner status without full session retrieval (for performance).
+        
+        Args:
+            session_id: Session identifier
+            
+        Returns:
+            Scanner status string or None if not found
+        """
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data and 'scanner' in session_data:
+                return session_data['scanner'].status
+            return None
+        except Exception as e:
+            print(f"ERROR: Failed to get session status for {session_id}: {e}")
+            return None
+
+    def terminate_session(self, session_id: str) -> bool:
+        """
+        Terminate a specific session in Redis with reliable stop signal and immediate status update.
+        """
+        print(f"=== TERMINATING SESSION {session_id} ===")
+        
+        try:
+            # First, set the stop signal
+            self.set_stop_signal(session_id)
+            
+            # Update scanner status to stopped immediately for GUI feedback
+            self.update_scanner_status(session_id, 'stopped')
+            
+            session_data = self._get_session_data(session_id)
+            if not session_data:
+                print(f"Session {session_id} not found")
+                return False
+
+            scanner = session_data.get('scanner')
+            if scanner and scanner.status == 'running':
+                print(f"Stopping scan for session: {session_id}")
+                # The scanner will check the Redis stop signal
+                scanner.stop_scan()
+                
+                # Update the scanner state immediately
+                self.update_session_scanner(session_id, scanner)
+            
+            # Wait a moment for graceful shutdown
+            time.sleep(0.5)
+            
+            # Delete session data and stop signal from Redis
+            session_key = self._get_session_key(session_id)
+            stop_key = self._get_stop_signal_key(session_id)
+            self.redis_client.delete(session_key)
+            self.redis_client.delete(stop_key)
+            
+            print(f"Terminated and removed session from Redis: {session_id}")
+            return True
+            
+        except Exception as e:
+            print(f"ERROR: Failed to terminate session {session_id}: {e}")
+            return False
+
+    def _cleanup_loop(self) -> None:
+        """
+        Background thread to cleanup inactive sessions and orphaned stop signals.
+        """
+        while True:
+            try:
+                # Clean up orphaned stop signals
+                stop_keys = self.redis_client.keys("DNScope:stop:*")
+                for stop_key in stop_keys:
+                    # Extract session ID from stop key
+                    session_id = stop_key.decode('utf-8').split(':')[-1]
+                    session_key = self._get_session_key(session_id)
+                    
+                    # If session doesn't exist but stop signal does, clean it up
+                    if not self.redis_client.exists(session_key):
+                        self.redis_client.delete(stop_key)
+                        print(f"Cleaned up orphaned stop signal for session {session_id}")
+                        
+            except Exception as e:
+                print(f"Error in cleanup loop: {e}")
+            
+            time.sleep(300)  # Sleep for 5 minutes
+
+    def get_statistics(self) -> Dict[str, Any]:
+        """Get session manager statistics."""
+        try:
+            session_keys = self.redis_client.keys("DNScope:session:*")
+            stop_keys = self.redis_client.keys("DNScope:stop:*")
+            
+            active_sessions = len(session_keys)
+            running_scans = 0
+            
+            for session_key in session_keys:
+                session_id = session_key.decode('utf-8').split(':')[-1]
+                status = self.get_session_status_only(session_id)
+                if status == 'running':
+                    running_scans += 1
+            
+            return {
+                'total_active_sessions': active_sessions,
+                'running_scans': running_scans,
+                'total_stop_signals': len(stop_keys)
+            }
+        except Exception as e:
+            print(f"ERROR: Failed to get statistics: {e}")
+            return {
+                'total_active_sessions': 0,
+                'running_scans': 0,
+                'total_stop_signals': 0
+            }
+
+# Global session manager instance
+session_manager = SessionManager(session_timeout_minutes=60)

dnsrecon.py

@@ -1,976 +0,0 @@
-#!/usr/bin/env python3
-"""
-Enhanced DNS Reconnaissance Tool with Recursive Analysis
-
-Copyright (c) 2025 mstoeck3. 
-
-Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
-1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
-2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
-
-3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-"""
-
-import subprocess
-import json
-import requests
-import argparse
-import sys
-import time
-import os
-import re
-import ipaddress
-from datetime import datetime
-from typing import Dict, List, Optional, Any, Set
-from urllib.parse import urlparse
-import threading
-from queue import Queue, Empty
-
-class EnhancedDNSReconTool:
-    def __init__(self, shodan_api_key: Optional[str] = None, virustotal_api_key: Optional[str] = None):
-        self.shodan_api_key = shodan_api_key
-        self.virustotal_api_key = virustotal_api_key
-        self.output_dir = "dns_recon_results"
-        self.session = requests.Session()
-        self.session.headers.update({
-            'User-Agent': 'EnhancedDNSReconTool/2.0 (Educational/Research Purpose)'
-        })
-        
-        # Track processed items to avoid infinite recursion
-        self.processed_domains: Set[str] = set()
-        self.processed_ips: Set[str] = set()
-        
-        # Results storage for recursive analysis
-        self.all_results: Dict[str, Any] = {}
-        
-        # Rate limiting
-        self.last_vt_request = 0
-        self.last_shodan_request = 0
-        self.vt_rate_limit = 4  # 4 requests per minute for free tier
-        self.shodan_rate_limit = 1  # 1 request per second for free tier
-        
-    def check_dependencies(self) -> bool:
-        """Check if required system tools are available."""
-        required_tools = ['dig', 'whois']
-        missing_tools = []
-        
-        for tool in required_tools:
-            try:
-                subprocess.run([tool, '--help'], 
-                             capture_output=True, check=False, timeout=5)
-            except (subprocess.TimeoutExpired, FileNotFoundError):
-                missing_tools.append(tool)
-        
-        if missing_tools:
-            print(f"❌ Missing required tools: {', '.join(missing_tools)}")
-            print("Install with: apt install dnsutils whois (Ubuntu/Debian)")
-            return False
-        return True
-    
-    def run_command(self, cmd: str, timeout: int = 30) -> str:
-        """Run shell command with timeout and error handling."""
-        try:
-            result = subprocess.run(
-                cmd, shell=True, capture_output=True, 
-                text=True, timeout=timeout
-            )
-            return result.stdout.strip() if result.stdout else result.stderr.strip()
-        except subprocess.TimeoutExpired:
-            return "Error: Command timed out"
-        except Exception as e:
-            return f"Error: {str(e)}"
-    
-    def rate_limit_virustotal(self):
-        """Implement rate limiting for VirusTotal API."""
-        current_time = time.time()
-        time_since_last = current_time - self.last_vt_request
-        min_interval = 60 / self.vt_rate_limit  # seconds between requests
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            print(f"  Rate limiting: waiting {sleep_time:.1f}s for VirusTotal...")
-            time.sleep(sleep_time)
-        
-        self.last_vt_request = time.time()
-    
-    def rate_limit_shodan(self):
-        """Implement rate limiting for Shodan API."""
-        current_time = time.time()
-        time_since_last = current_time - self.last_shodan_request
-        min_interval = 1 / self.shodan_rate_limit  # seconds between requests
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            time.sleep(sleep_time)
-        
-        self.last_shodan_request = time.time()
-
-    def query_virustotal_domain(self, domain: str) -> Dict[str, Any]:
-        """Query VirusTotal API for domain information."""
-        if not self.virustotal_api_key:
-            return {
-                'success': False,
-                'message': 'No VirusTotal API key provided'
-            }
-        
-        print(f"🔍 Querying VirusTotal for domain: {domain}")
-        
-        try:
-            self.rate_limit_virustotal()
-            
-            url = f"https://www.virustotal.com/vtapi/v2/domain/report"
-            params = {
-                'apikey': self.virustotal_api_key,
-                'domain': domain
-            }
-            
-            response = self.session.get(url, params=params, timeout=30)
-            
-            if response.status_code == 200:
-                data = response.json()
-                
-                # Extract key information
-                result = {
-                    'success': True,
-                    'domain': domain,
-                    'response_code': data.get('response_code', 0),
-                    'verbose_msg': data.get('verbose_msg', ''),
-                    'detection_ratio': f"{data.get('positives', 0)}/{data.get('total', 0)}"
-                }
-                
-                # Add scan results if available
-                if 'scans' in data:
-                    result['scan_engines'] = len(data['scans'])
-                    result['malicious_engines'] = sum(1 for scan in data['scans'].values() if scan.get('detected', False))
-                    result['scan_summary'] = {}
-                    
-                    # Categorize detections
-                    for engine, scan_result in data['scans'].items():
-                        if scan_result.get('detected', False):
-                            category = scan_result.get('result', 'malicious')
-                            if category not in result['scan_summary']:
-                                result['scan_summary'][category] = []
-                            result['scan_summary'][category].append(engine)
-                
-                # Add additional data if available
-                for key in ['subdomains', 'detected_urls', 'undetected_urls', 'resolutions']:
-                    if key in data:
-                        result[key] = data[key]
-                
-                return result
-            else:
-                return {
-                    'success': False,
-                    'error': f"HTTP {response.status_code}",
-                    'message': response.text[:200]
-                }
-                
-        except Exception as e:
-            return {
-                'success': False,
-                'error': str(e),
-                'message': 'VirusTotal domain query failed'
-            }
-
-    def query_virustotal_ip(self, ip: str) -> Dict[str, Any]:
-        """Query VirusTotal API for IP information."""
-        if not self.virustotal_api_key:
-            return {
-                'success': False,
-                'message': 'No VirusTotal API key provided'
-            }
-        
-        print(f"🔍 Querying VirusTotal for IP: {ip}")
-        
-        try:
-            self.rate_limit_virustotal()
-            
-            url = f"https://www.virustotal.com/vtapi/v2/ip-address/report"
-            params = {
-                'apikey': self.virustotal_api_key,
-                'ip': ip
-            }
-            
-            response = self.session.get(url, params=params, timeout=30)
-            
-            if response.status_code == 200:
-                data = response.json()
-                
-                result = {
-                    'success': True,
-                    'ip': ip,
-                    'response_code': data.get('response_code', 0),
-                    'verbose_msg': data.get('verbose_msg', ''),
-                    'detection_ratio': f"{data.get('positives', 0)}/{data.get('total', 0)}"
-                }
-                
-                # Add scan results if available
-                if 'scans' in data:
-                    result['scan_engines'] = len(data['scans'])
-                    result['malicious_engines'] = sum(1 for scan in data['scans'].values() if scan.get('detected', False))
-                
-                # Add additional data
-                for key in ['detected_urls', 'undetected_urls', 'resolutions', 'asn', 'country']:
-                    if key in data:
-                        result[key] = data[key]
-                
-                return result
-            else:
-                return {
-                    'success': False,
-                    'error': f"HTTP {response.status_code}",
-                    'message': response.text[:200]
-                }
-                
-        except Exception as e:
-            return {
-                'success': False,
-                'error': str(e),
-                'message': 'VirusTotal IP query failed'
-            }
-    
-    def get_dns_records(self, domain: str, record_type: str, 
-                       server: Optional[str] = None) -> Dict[str, Any]:
-        """Fetch DNS records with comprehensive error handling and proper parsing."""
-        server_flag = f"@{server}" if server else ""
-        cmd = f"dig {domain} {record_type} {server_flag} +noall +answer"
-        
-        output = self.run_command(cmd)
-        
-        # Parse the output into structured data
-        records = []
-        if output and not output.startswith("Error:"):
-            for line in output.split('\n'):
-                line = line.strip()
-                if line and not line.startswith(';') and not line.startswith('>>'):
-                    # Split on any whitespace (handles both tabs and spaces)
-                    parts = line.split()
-                    
-                    if len(parts) >= 4:
-                        name = parts[0].rstrip('.')
-                        
-                        # Check if second field is numeric (TTL)
-                        if len(parts) >= 5 and parts[1].isdigit():
-                            # Format: name TTL class type data
-                            ttl = parts[1]
-                            dns_class = parts[2]
-                            dns_type = parts[3]
-                            data = ' '.join(parts[4:])
-                        else:
-                            # Format: name class type data (no TTL shown)
-                            ttl = ''
-                            dns_class = parts[1]
-                            dns_type = parts[2]
-                            data = ' '.join(parts[3:]) if len(parts) > 3 else ''
-                        
-                        # Validate that we have the expected record type
-                        if dns_type.upper() == record_type.upper():
-                            records.append({
-                                'name': name,
-                                'ttl': ttl,
-                                'class': dns_class,
-                                'type': dns_type,
-                                'data': data
-                            })
-        
-        return {
-            'query': f"{domain} {record_type}",
-            'server': server or 'system',
-            'raw_output': output,
-            'records': records,
-            'record_count': len(records)
-        }
-    
-    def get_comprehensive_dns(self, domain: str) -> Dict[str, Any]:
-        """Get comprehensive DNS information."""
-        print(f"🔍 Gathering DNS records for {domain}...")
-        
-        # Standard record types
-        record_types = ['A', 'AAAA', 'MX', 'NS', 'SOA', 'TXT', 'CNAME', 
-                       'CAA', 'SRV', 'PTR']
-        
-        # DNS servers to query
-        dns_servers = [
-            None,           # System default
-            '1.1.1.1',      # Cloudflare
-            '8.8.8.8',      # Google
-            '9.9.9.9',      # Quad9
-        ]
-        
-        dns_results = {}
-        
-        for record_type in record_types:
-            dns_results[record_type] = {}
-            for server in dns_servers:
-                server_name = server or 'system'
-                result = self.get_dns_records(domain, record_type, server)
-                dns_results[record_type][server_name] = result
-                
-                time.sleep(0.1)  # Rate limiting
-        
-        # Try DNSSEC validation
-        dnssec_cmd = f"dig {domain} +dnssec +noall +answer"
-        dns_results['DNSSEC'] = {
-            'system': {
-                'query': f"{domain} +dnssec",
-                'raw_output': self.run_command(dnssec_cmd),
-                'records': [],
-                'record_count': 0
-            }
-        }
-        
-        return dns_results
-    
-    def perform_reverse_dns(self, ip: str) -> Dict[str, Any]:
-        """Perform reverse DNS lookup on IP address."""
-        print(f"🔄 Reverse DNS lookup for {ip}")
-        
-        try:
-            # Validate IP address
-            ipaddress.ip_address(ip)
-            
-            # Perform reverse DNS lookup
-            cmd = f"dig -x {ip} +short"
-            output = self.run_command(cmd)
-            
-            hostnames = []
-            if output and not output.startswith("Error:"):
-                hostnames = [line.strip().rstrip('.') for line in output.split('\n') if line.strip()]
-            
-            return {
-                'success': True,
-                'ip': ip,
-                'hostnames': hostnames,
-                'hostname_count': len(hostnames),
-                'raw_output': output
-            }
-            
-        except Exception as e:
-            return {
-                'success': False,
-                'ip': ip,
-                'error': str(e),
-                'hostnames': [],
-                'hostname_count': 0
-            }
-    
-    def extract_subdomains_from_certificates(self, domain: str) -> Set[str]:
-        """Extract subdomains from certificate transparency logs."""
-        print(f"📋 Extracting subdomains from certificates for {domain}")
-        
-        try:
-            url = f"https://crt.sh/?q=%.{domain}&output=json"
-            response = self.session.get(url, timeout=30)
-            
-            subdomains = set()
-            
-            if response.status_code == 200:
-                cert_data = response.json()
-                
-                for cert in cert_data:
-                    name_value = cert.get('name_value', '')
-                    if name_value:
-                        # Handle multiple domains in one certificate
-                        domains_in_cert = [d.strip() for d in name_value.split('\n')]
-                        for subdomain in domains_in_cert:
-                            # Clean up the subdomain
-                            subdomain = subdomain.lower().strip()
-                            if subdomain and '.' in subdomain:
-                                # Only include subdomains of the target domain
-                                if subdomain.endswith(f".{domain}") or subdomain == domain:
-                                    subdomains.add(subdomain)
-                                elif subdomain.startswith("*."):
-                                    # Handle wildcard certificates
-                                    clean_subdomain = subdomain[2:]
-                                    if clean_subdomain.endswith(f".{domain}") or clean_subdomain == domain:
-                                        subdomains.add(clean_subdomain)
-            
-            return subdomains
-            
-        except Exception as e:
-            print(f"  Error extracting subdomains: {e}")
-            return set()
-    
-    def extract_ips_from_dns(self, dns_data: Dict[str, Any]) -> Set[str]:
-        """Extract IP addresses from DNS records."""
-        ips = set()
-        
-        # Extract from A records
-        for server_data in dns_data.get('A', {}).values():
-            for record in server_data.get('records', []):
-                ip = record.get('data', '')
-                if ip and self.is_valid_ip(ip):
-                    ips.add(ip)
-        
-        # Extract from AAAA records
-        for server_data in dns_data.get('AAAA', {}).values():
-            for record in server_data.get('records', []):
-                ipv6 = record.get('data', '')
-                if ipv6 and self.is_valid_ip(ipv6):
-                    ips.add(ipv6)
-        
-        return ips
-    
-    def is_valid_ip(self, ip: str) -> bool:
-        """Check if string is a valid IP address."""
-        try:
-            ipaddress.ip_address(ip)
-            return True
-        except ValueError:
-            return False
-    
-    def get_whois_data(self, domain: str) -> Dict[str, Any]:
-        """Fetch and parse WHOIS data with improved parsing."""
-        print(f"📋 Fetching WHOIS data for {domain}...")
-        
-        raw_whois = self.run_command(f"whois {domain}")
-        
-        # Basic parsing of common WHOIS fields
-        whois_data = {
-            'raw': raw_whois,
-            'parsed': {}
-        }
-        
-        if not raw_whois.startswith("Error:"):
-            lines = raw_whois.split('\n')
-            for line in lines:
-                line = line.strip()
-                if ':' in line and not line.startswith('%') and not line.startswith('#') and not line.startswith('>>>'):
-                    # Handle different WHOIS formats
-                    if line.count(':') == 1:
-                        key, value = line.split(':', 1)
-                    else:
-                        # Multiple colons - take first as key, rest as value
-                        parts = line.split(':', 2)
-                        key, value = parts[0], ':'.join(parts[1:])
-                    
-                    key = key.strip().lower().replace(' ', '_').replace('-', '_')
-                    value = value.strip()
-                    if value and key:
-                        # Handle multiple values for same key (like name servers)
-                        if key in whois_data['parsed']:
-                            # Convert to list if not already
-                            if not isinstance(whois_data['parsed'][key], list):
-                                whois_data['parsed'][key] = [whois_data['parsed'][key]]
-                            whois_data['parsed'][key].append(value)
-                        else:
-                            whois_data['parsed'][key] = value
-        
-        return whois_data
-    
-    def get_certificate_transparency(self, domain: str) -> Dict[str, Any]:
-        """Query certificate transparency logs via crt.sh."""
-        print(f"🔐 Querying certificate transparency logs for {domain}...")
-        
-        try:
-            # Query crt.sh API
-            url = f"https://crt.sh/?q=%.{domain}&output=json"
-            response = self.session.get(url, timeout=30)
-            
-            if response.status_code == 200:
-                cert_data = response.json()
-                
-                # Extract unique subdomains
-                subdomains = set()
-                cert_details = []
-                
-                for cert in cert_data:
-                    # Extract subdomains from name_value
-                    name_value = cert.get('name_value', '')
-                    if name_value:
-                        # Handle multiple domains in one certificate
-                        domains_in_cert = [d.strip() for d in name_value.split('\n')]
-                        subdomains.update(domains_in_cert)
-                    
-                    cert_details.append({
-                        'id': cert.get('id'),
-                        'issuer': cert.get('issuer_name'),
-                        'common_name': cert.get('common_name'),
-                        'name_value': cert.get('name_value'),
-                        'not_before': cert.get('not_before'),
-                        'not_after': cert.get('not_after'),
-                        'serial_number': cert.get('serial_number')
-                    })
-                
-                return {
-                    'success': True,
-                    'total_certificates': len(cert_data),
-                    'unique_subdomains': sorted(list(subdomains)),
-                    'subdomain_count': len(subdomains),
-                    'certificates': cert_details[:50]  # Limit for output size
-                }
-            else:
-                return {
-                    'success': False,
-                    'error': f"HTTP {response.status_code}",
-                    'message': 'Failed to fetch certificate data'
-                }
-                
-        except Exception as e:
-            return {
-                'success': False,
-                'error': str(e),
-                'message': 'Request to crt.sh failed'
-            }
-    
-    def query_shodan(self, domain: str) -> Dict[str, Any]:
-        """Query Shodan API for domain information."""
-        if not self.shodan_api_key:
-            return {
-                'success': False,
-                'message': 'No Shodan API key provided'
-            }
-        
-        print(f"🔎 Querying Shodan for {domain}...")
-        
-        try:
-            self.rate_limit_shodan()
-            
-            # Search for the domain
-            url = f"https://api.shodan.io/shodan/host/search"
-            params = {
-                'key': self.shodan_api_key,
-                'query': f'hostname:{domain}'
-            }
-            
-            response = self.session.get(url, params=params, timeout=30)
-            
-            if response.status_code == 200:
-                data = response.json()
-                return {
-                    'success': True,
-                    'total_results': data.get('total', 0),
-                    'matches': data.get('matches', [])[:10],  # Limit results
-                    'facets': data.get('facets', {})
-                }
-            else:
-                return {
-                    'success': False,
-                    'error': f"HTTP {response.status_code}",
-                    'message': response.text[:200]
-                }
-                
-        except Exception as e:
-            return {
-                'success': False,
-                'error': str(e),
-                'message': 'Shodan query failed'
-            }
-
-    def query_shodan_ip(self, ip: str) -> Dict[str, Any]:
-        """Query Shodan API for IP information."""
-        if not self.shodan_api_key:
-            return {
-                'success': False,
-                'message': 'No Shodan API key provided'
-            }
-        
-        print(f"🔎 Querying Shodan for IP {ip}...")
-        
-        try:
-            self.rate_limit_shodan()
-            
-            url = f"https://api.shodan.io/shodan/host/{ip}"
-            params = {'key': self.shodan_api_key}
-            
-            response = self.session.get(url, params=params, timeout=30)
-            
-            if response.status_code == 200:
-                data = response.json()
-                return {
-                    'success': True,
-                    'ip': ip,
-                    'data': data
-                }
-            else:
-                return {
-                    'success': False,
-                    'error': f"HTTP {response.status_code}",
-                    'message': response.text[:200]
-                }
-                
-        except Exception as e:
-            return {
-                'success': False,
-                'error': str(e),
-                'message': 'Shodan IP query failed'
-            }
-    
-    def analyze_domain_recursively(self, domain: str, depth: int = 0, max_depth: int = 2) -> Dict[str, Any]:
-        """Perform comprehensive analysis on a domain with recursive subdomain discovery."""
-        if domain in self.processed_domains or depth > max_depth:
-            return {}
-        
-        self.processed_domains.add(domain)
-        
-        print(f"\n{'  ' * depth}🎯 Analyzing domain: {domain} (depth {depth})")
-        
-        results = {
-            'domain': domain,
-            'timestamp': datetime.now().isoformat(),
-            'depth': depth,
-            'dns_records': {},
-            'whois': {},
-            'certificate_transparency': {},
-            'virustotal_domain': {},
-            'shodan': {},
-            'discovered_ips': {},
-            'discovered_subdomains': {}
-        }
-        
-        # DNS Records
-        results['dns_records'] = self.get_comprehensive_dns(domain)
-        
-        # Extract IP addresses from DNS records
-        discovered_ips = self.extract_ips_from_dns(results['dns_records'])
-        
-        # WHOIS (only for primary domain to avoid rate limiting)
-        if depth == 0:
-            results['whois'] = self.get_whois_data(domain)
-        
-        # Certificate Transparency
-        results['certificate_transparency'] = self.get_certificate_transparency(domain)
-        
-        # VirusTotal Domain Analysis
-        results['virustotal_domain'] = self.query_virustotal_domain(domain)
-        
-        # Shodan Domain Analysis
-        results['shodan'] = self.query_shodan(domain)
-        
-        # Extract subdomains from certificate transparency
-        if depth < max_depth:
-            subdomains = self.extract_subdomains_from_certificates(domain)
-            
-            # Filter out already processed subdomains
-            new_subdomains = subdomains - self.processed_domains
-            new_subdomains.discard(domain)  # Remove the current domain itself
-            
-            print(f"{'  ' * depth}📋 Found {len(new_subdomains)} new subdomains to analyze")
-            
-            # Recursively analyze subdomains (limit to prevent excessive recursion)
-            for subdomain in list(new_subdomains)[:20]:  # Limit to 20 subdomains per domain
-                if subdomain not in self.processed_domains:
-                    subdomain_results = self.analyze_domain_recursively(subdomain, depth + 1, max_depth)
-                    if subdomain_results:
-                        results['discovered_subdomains'][subdomain] = subdomain_results
-        
-        # Analyze discovered IP addresses
-        for ip in discovered_ips:
-            if ip not in self.processed_ips:
-                ip_results = self.analyze_ip_recursively(ip, depth)
-                if ip_results:
-                    results['discovered_ips'][ip] = ip_results
-        
-        # Store in global results
-        self.all_results[domain] = results
-        
-        return results
-    
-    def analyze_ip_recursively(self, ip: str, depth: int = 0) -> Dict[str, Any]:
-        """Perform comprehensive analysis on an IP address."""
-        if ip in self.processed_ips:
-            return {}
-        
-        self.processed_ips.add(ip)
-        
-        print(f"{'  ' * depth}🌐 Analyzing IP: {ip}")
-        
-        results = {
-            'ip': ip,
-            'timestamp': datetime.now().isoformat(),
-            'reverse_dns': {},
-            'virustotal_ip': {},
-            'shodan_ip': {},
-            'discovered_domains': {}
-        }
-        
-        # Reverse DNS lookup
-        results['reverse_dns'] = self.perform_reverse_dns(ip)
-        
-        # VirusTotal IP Analysis
-        results['virustotal_ip'] = self.query_virustotal_ip(ip)
-        
-        # Shodan IP Analysis
-        results['shodan_ip'] = self.query_shodan_ip(ip)
-        
-        # Analyze discovered domains from reverse DNS
-        reverse_dns = results['reverse_dns']
-        if reverse_dns.get('success') and reverse_dns.get('hostnames'):
-            for hostname in reverse_dns['hostnames'][:5]:  # Limit to 5 hostnames
-                if hostname not in self.processed_domains and hostname.count('.') >= 1:
-                    # Only analyze if it's a reasonable hostname and not already processed
-                    domain_results = self.analyze_domain_recursively(hostname, depth + 1, max_depth=1)
-                    if domain_results:
-                        results['discovered_domains'][hostname] = domain_results
-        
-        return results
-    
-    def create_comprehensive_summary(self, filename: str) -> None:
-        """Create comprehensive summary report with recursive analysis results."""
-        with open(filename, 'w', encoding='utf-8') as f:
-            f.write("Enhanced DNS Reconnaissance Report with Recursive Analysis\n")
-            f.write("=" * 65 + "\n")
-            f.write(f"Analysis completed at: {datetime.now().isoformat()}\n")
-            f.write(f"Total domains analyzed: {len(self.processed_domains)}\n")
-            f.write(f"Total IP addresses analyzed: {len(self.processed_ips)}\n\n")
-            
-            # Executive Summary
-            f.write("EXECUTIVE SUMMARY\n")
-            f.write("-" * 17 + "\n")
-            
-            total_threats = 0
-            domains_with_issues = []
-            ips_with_issues = []
-            
-            # Count threats across all analyzed domains and IPs
-            for domain, domain_data in self.all_results.items():
-                # Check VirusTotal results for domain
-                vt_domain = domain_data.get('virustotal_domain', {})
-                if vt_domain.get('success') and vt_domain.get('malicious_engines', 0) > 0:
-                    total_threats += 1
-                    domains_with_issues.append(domain)
-                
-                # Check discovered IPs
-                for ip, ip_data in domain_data.get('discovered_ips', {}).items():
-                    vt_ip = ip_data.get('virustotal_ip', {})
-                    if vt_ip.get('success') and vt_ip.get('malicious_engines', 0) > 0:
-                        total_threats += 1
-                        ips_with_issues.append(ip)
-            
-            f.write(f"Security Status: {'⚠️  THREATS DETECTED' if total_threats > 0 else '✅ NO THREATS DETECTED'}\n")
-            f.write(f"Total Security Issues: {total_threats}\n")
-            if domains_with_issues:
-                f.write(f"Domains with issues: {', '.join(domains_with_issues[:5])}\n")
-            if ips_with_issues:
-                f.write(f"IPs with issues: {', '.join(ips_with_issues[:5])}\n")
-            f.write("\n")
-            
-            # Process each domain in detail
-            for domain, domain_data in self.all_results.items():
-                if domain_data.get('depth', 0) == 0:  # Only show primary domains in detail
-                    self._write_domain_analysis(f, domain, domain_data)
-            
-            # Summary of all discovered assets
-            f.write("\nASSET DISCOVERY SUMMARY\n")
-            f.write("-" * 23 + "\n")
-            f.write(f"All Discovered Domains ({len(self.processed_domains)}):\n")
-            for domain in sorted(self.processed_domains):
-                f.write(f"  {domain}\n")
-            
-            f.write(f"\nAll Discovered IP Addresses ({len(self.processed_ips)}):\n")
-            for ip in sorted(self.processed_ips, key=ipaddress.IPv4Address):
-                f.write(f"  {ip}\n")
-            
-            f.write(f"\n{'=' * 65}\n")
-            f.write("Report Generation Complete\n")
-
-    def _write_domain_analysis(self, f, domain: str, domain_data: Dict[str, Any]) -> None:
-        """Write detailed domain analysis to file."""
-        f.write(f"\nDETAILED ANALYSIS: {domain.upper()}\n")
-        f.write("=" * (20 + len(domain)) + "\n")
-        
-        # DNS Records Summary
-        dns_data = domain_data.get('dns_records', {})
-        f.write("DNS Records Summary:\n")
-        for record_type in ['A', 'AAAA', 'MX', 'NS', 'TXT']:
-            system_records = dns_data.get(record_type, {}).get('system', {}).get('records', [])
-            f.write(f"  {record_type}: {len(system_records)} records\n")
-        
-        # Security Analysis
-        f.write(f"\nSecurity Analysis:\n")
-        
-        # VirusTotal Domain Results
-        vt_domain = domain_data.get('virustotal_domain', {})
-        if vt_domain.get('success'):
-            detection_ratio = vt_domain.get('detection_ratio', '0/0')
-            malicious_engines = vt_domain.get('malicious_engines', 0)
-            f.write(f"  VirusTotal Domain: {detection_ratio} ({malicious_engines} flagged as malicious)\n")
-            
-            if malicious_engines > 0:
-                f.write(f"  ⚠️  SECURITY ALERT: Domain flagged by {malicious_engines} security engines\n")
-                scan_summary = vt_domain.get('scan_summary', {})
-                for category, engines in scan_summary.items():
-                    f.write(f"    {category}: {', '.join(engines[:3])}\n")
-        else:
-            f.write(f"  VirusTotal Domain: {vt_domain.get('message', 'Not available')}\n")
-        
-        # Certificate Information
-        cert_data = domain_data.get('certificate_transparency', {})
-        if cert_data.get('success'):
-            f.write(f"  SSL Certificates: {cert_data.get('total_certificates', 0)} found\n")
-            f.write(f"  Subdomains from Certificates: {cert_data.get('subdomain_count', 0)}\n")
-        
-        # Discovered Assets
-        discovered_ips = domain_data.get('discovered_ips', {})
-        discovered_subdomains = domain_data.get('discovered_subdomains', {})
-        
-        if discovered_ips:
-            f.write(f"\nDiscovered IP Addresses ({len(discovered_ips)}):\n")
-            for ip, ip_data in discovered_ips.items():
-                vt_ip = ip_data.get('virustotal_ip', {})
-                reverse_dns = ip_data.get('reverse_dns', {})
-                
-                f.write(f"  {ip}:\n")
-                
-                # Reverse DNS
-                if reverse_dns.get('success') and reverse_dns.get('hostnames'):
-                    f.write(f"    Reverse DNS: {', '.join(reverse_dns['hostnames'][:3])}\n")
-                
-                # VirusTotal IP results
-                if vt_ip.get('success'):
-                    detection_ratio = vt_ip.get('detection_ratio', '0/0')
-                    malicious_engines = vt_ip.get('malicious_engines', 0)
-                    f.write(f"    VirusTotal: {detection_ratio}")
-                    if malicious_engines > 0:
-                        f.write(f" ⚠️  FLAGGED BY {malicious_engines} ENGINES")
-                    f.write("\n")
-                
-                # Shodan IP results
-                shodan_ip = ip_data.get('shodan_ip', {})
-                if shodan_ip.get('success'):
-                    shodan_data = shodan_ip.get('data', {})
-                    ports = shodan_data.get('ports', [])
-                    if ports:
-                        f.write(f"    Shodan Ports: {', '.join(map(str, ports[:10]))}\n")
-                
-                f.write("\n")
-        
-        if discovered_subdomains:
-            f.write(f"Discovered Subdomains ({len(discovered_subdomains)}):\n")
-            for subdomain, subdomain_data in discovered_subdomains.items():
-                f.write(f"  {subdomain}\n")
-                
-                # Quick security check for subdomain
-                vt_subdomain = subdomain_data.get('virustotal_domain', {})
-                if vt_subdomain.get('success') and vt_subdomain.get('malicious_engines', 0) > 0:
-                    f.write(f"    ⚠️  Security Issue: Flagged by VirusTotal\n")
-                
-                subdomain_ips = subdomain_data.get('discovered_ips', {})
-                if subdomain_ips:
-                    f.write(f"    IPs: {', '.join(list(subdomain_ips.keys())[:3])}\n")
-        
-        f.write("\n")
-
-    def save_results(self, domain: str) -> None:
-        """Save results in multiple formats."""
-        if not os.path.exists(self.output_dir):
-            os.makedirs(self.output_dir)
-        
-        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
-        base_filename = f"{self.output_dir}/{domain}_{timestamp}"
-        
-        # Save complete JSON (all recursive data)
-        json_file = f"{base_filename}_complete.json"
-        with open(json_file, 'w', encoding='utf-8') as f:
-            json.dump(self.all_results, f, indent=2, ensure_ascii=False, default=str)
-        
-        # Save comprehensive summary
-        summary_file = f"{base_filename}_analysis.txt"
-        self.create_comprehensive_summary(summary_file)
-        
-        # Save asset list (domains and IPs)
-        assets_file = f"{base_filename}_assets.txt"
-        with open(assets_file, 'w', encoding='utf-8') as f:
-            f.write("Discovered Assets Summary\n")
-            f.write("=" * 25 + "\n\n")
-            
-            f.write(f"Domains ({len(self.processed_domains)}):\n")
-            for domain in sorted(self.processed_domains):
-                f.write(f"{domain}\n")
-            
-            f.write(f"\nIP Addresses ({len(self.processed_ips)}):\n")
-            for ip in sorted(self.processed_ips, key=lambda x: ipaddress.IPv4Address(x)):
-                f.write(f"{ip}\n")
-        
-        print(f"\n📄 Results saved:")
-        print(f"   Complete JSON: {json_file}")
-        print(f"   Analysis Report: {summary_file}")
-        print(f"   Asset List: {assets_file}")
-
-    def run_enhanced_reconnaissance(self, domain: str, max_depth: int = 2) -> Dict[str, Any]:
-        """Run enhanced recursive DNS reconnaissance."""
-        print(f"\n🚀 Starting enhanced DNS reconnaissance for: {domain}")
-        print(f"   Max recursion depth: {max_depth}")
-        print(f"   APIs enabled: VirusTotal={bool(self.virustotal_api_key)}, Shodan={bool(self.shodan_api_key)}")
-        
-        start_time = time.time()
-        
-        # Clear previous results
-        self.processed_domains.clear()
-        self.processed_ips.clear()
-        self.all_results.clear()
-        
-        # Start recursive analysis
-        results = self.analyze_domain_recursively(domain, depth=0, max_depth=max_depth)
-        
-        end_time = time.time()
-        duration = end_time - start_time
-        
-        print(f"\n✅ Enhanced reconnaissance completed in {duration:.1f} seconds")
-        print(f"   Domains analyzed: {len(self.processed_domains)}")
-        print(f"   IP addresses analyzed: {len(self.processed_ips)}")
-        
-        return results
-
-def main():
-    parser = argparse.ArgumentParser(
-        description="Enhanced DNS Reconnaissance Tool with Recursive Analysis - Use only on domains you own or have permission to test",
-        epilog="LEGAL NOTICE: Unauthorized reconnaissance may violate applicable laws. Use responsibly."
-    )
-    parser.add_argument('domain', help='Target domain (e.g., example.com)')
-    parser.add_argument('--shodan-key', help='Shodan API key for additional reconnaissance')
-    parser.add_argument('--virustotal-key', help='VirusTotal API key for threat intelligence')
-    parser.add_argument('--max-depth', type=int, default=2, 
-                       help='Maximum recursion depth for subdomain analysis (default: 2)')
-    parser.add_argument('--output-dir', default='dns_recon_results', 
-                       help='Output directory for results')
-    
-    args = parser.parse_args()
-    
-    # Validate domain format
-    if not re.match(r'^[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', args.domain):
-        print("❌ Invalid domain format. Please provide a valid domain (e.g., example.com)")
-        sys.exit(1)
-    
-    # Initialize tool
-    tool = EnhancedDNSReconTool(
-        shodan_api_key=args.shodan_key,
-        virustotal_api_key=args.virustotal_key
-    )
-    tool.output_dir = args.output_dir
-    
-    # Check dependencies
-    if not tool.check_dependencies():
-        sys.exit(1)
-    
-    # Warn about API keys
-    if not args.virustotal_key:
-        print("⚠️  No VirusTotal API key provided. Threat intelligence will be limited.")
-    if not args.shodan_key:
-        print("⚠️  No Shodan API key provided. Host intelligence will be limited.")
-    
-    try:
-        # Run enhanced reconnaissance
-        results = tool.run_enhanced_reconnaissance(args.domain, args.max_depth)
-        
-        # Save results
-        tool.save_results(args.domain)
-        
-        print(f"\n🎯 Enhanced reconnaissance completed for {args.domain}")
-        
-    except KeyboardInterrupt:
-        print("\n⏹️  Reconnaissance interrupted by user")
-        sys.exit(0)
-    except Exception as e:
-        print(f"❌ Error during reconnaissance: {e}")
-        import traceback
-        traceback.print_exc()
-        sys.exit(1)
-
-if __name__ == "__main__":
-    main()

providers/__init__.py

@@ -0,0 +1,22 @@
+"""
+Data provider modules for DNScope.
+Contains implementations for various reconnaissance data sources.
+"""
+
+from .base_provider import BaseProvider
+from .crtsh_provider import CrtShProvider
+from .dns_provider import DNSProvider
+from .shodan_provider import ShodanProvider
+from .correlation_provider import CorrelationProvider
+from core.rate_limiter import GlobalRateLimiter
+
+__all__ = [
+    'BaseProvider',
+    'GlobalRateLimiter',
+    'CrtShProvider',
+    'DNSProvider',
+    'ShodanProvider',
+    'CorrelationProvider'
+]
+
+__version__ = "0.0.0-rc"

providers/base_provider.py

@@ -0,0 +1,274 @@
+# DNScope/providers/base_provider.py
+
+import time
+import requests
+import threading
+from abc import ABC, abstractmethod
+from typing import Dict, Any, Optional
+
+from core.logger import get_forensic_logger  # Ensure this import is present
+from core.rate_limiter import GlobalRateLimiter
+from core.provider_result import ProviderResult
+
+
+class BaseProvider(ABC):
+    """
+    Abstract base class for all DNScope data providers.
+    Now supports session-specific configuration and returns standardized ProviderResult objects.
+    """
+
+    def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30, session_config=None):
+        """
+        Initialize base provider with session-specific configuration.
+
+        Args:
+            name: Provider name for logging
+            rate_limit: Requests per minute limit (default override)
+            timeout: Request timeout in seconds
+            session_config: Session-specific configuration
+        """
+        # Use session config if provided, otherwise fall back to global config
+        if session_config is not None:
+            self.config = session_config
+            actual_rate_limit = self.config.get_rate_limit(name)
+            actual_timeout = self.config.default_timeout
+        else:
+            # Fallback to global config for backwards compatibility
+            from config import config as global_config
+            self.config = global_config
+            actual_timeout = timeout
+
+        self.name = name
+        self.timeout = actual_timeout
+        self._local = threading.local()
+        self._stop_event = None
+
+        # Statistics (per provider instance)
+        self.total_requests = 0
+        self.successful_requests = 0
+        self.failed_requests = 0
+        self.total_relationships_found = 0
+
+    def __getstate__(self):
+        """Prepare BaseProvider for pickling by excluding unpicklable objects."""
+        state = self.__dict__.copy()
+        # Exclude the unpickleable '_local' attribute and stop event
+        unpicklable_attrs = ['_local', '_stop_event']
+        for attr in unpicklable_attrs:
+            if attr in state:
+                del state[attr]
+        return state
+
+    def __setstate__(self, state):
+        """Restore BaseProvider after unpickling by reconstructing threading objects."""
+        self.__dict__.update(state)
+        # Re-initialize the '_local' attribute and stop event
+        self._local = threading.local()
+        self._stop_event = None
+
+    @property
+    def session(self):
+        if not hasattr(self._local, 'session'):
+            self._local.session = requests.Session()
+            self._local.session.headers.update({
+                'User-Agent': 'DNScope/1.0 (Passive Reconnaissance Tool)'
+            })
+        return self._local.session
+
+    @property
+    def logger(self):
+        """Get the current forensic logger instance."""
+        return get_forensic_logger()
+
+    @abstractmethod
+    def get_name(self) -> str:
+        """Return the provider name."""
+        pass
+
+    @abstractmethod
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        pass
+
+    @abstractmethod
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        pass
+
+    @abstractmethod
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        pass
+
+    @abstractmethod
+    def is_available(self) -> bool:
+        """Check if the provider is available and properly configured."""
+        pass
+
+    @abstractmethod
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        Query the provider for information about a domain.
+
+        Args:
+            domain: Domain to investigate
+
+        Returns:
+            ProviderResult containing standardized attributes and relationships
+        """
+        pass
+
+    @abstractmethod
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        Query the provider for information about an IP address.
+
+        Args:
+            ip: IP address to investigate
+
+        Returns:
+            ProviderResult containing standardized attributes and relationships
+        """
+        pass
+
+    def make_request(self, url: str, method: str = "GET",
+                    params: Optional[Dict[str, Any]] = None,
+                    headers: Optional[Dict[str, str]] = None,
+                    target_indicator: str = "") -> Optional[requests.Response]:
+        """
+        Make a rate-limited HTTP request.
+        FIXED: Returns response without automatically raising HTTPError exceptions.
+        Individual providers should handle status codes appropriately.
+        """
+        if self._is_stop_requested():
+            print(f"Request cancelled before start: {url}")
+            return None
+
+        start_time = time.time()
+        response = None
+        error = None
+
+        try:
+            self.total_requests += 1
+
+            request_headers = dict(self.session.headers).copy()
+            if headers:
+                request_headers.update(headers)
+
+            print(f"Making {method} request to: {url}")
+
+            if method.upper() == "GET":
+                response = self.session.get(
+                    url,
+                    params=params,
+                    headers=request_headers,
+                    timeout=self.timeout
+                )
+            elif method.upper() == "POST":
+                response = self.session.post(
+                    url,
+                    json=params,
+                    headers=request_headers,
+                    timeout=self.timeout
+                )
+            else:
+                raise ValueError(f"Unsupported HTTP method: {method}")
+
+            print(f"Response status: {response.status_code}")
+            
+            # FIXED: Don't automatically raise for HTTP error status codes
+            # Let individual providers handle status codes appropriately
+            # Only count 2xx responses as successful
+            if 200 <= response.status_code < 300:
+                self.successful_requests += 1
+            else:
+                self.failed_requests += 1
+            
+            duration_ms = (time.time() - start_time) * 1000
+            self.logger.log_api_request(
+                provider=self.name,
+                url=url,
+                method=method.upper(),
+                status_code=response.status_code,
+                response_size=len(response.content),
+                duration_ms=duration_ms,
+                error=None,
+                target_indicator=target_indicator
+            )
+            
+            return response
+
+        except requests.exceptions.RequestException as e:
+            error = str(e)
+            self.failed_requests += 1
+            duration_ms = (time.time() - start_time) * 1000
+            self.logger.log_api_request(
+                provider=self.name,
+                url=url,
+                method=method.upper(),
+                status_code=response.status_code if response else None,
+                response_size=len(response.content) if response else None,
+                duration_ms=duration_ms,
+                error=error,
+                target_indicator=target_indicator
+            )
+            raise e
+
+    def _is_stop_requested(self) -> bool:
+        """
+        Enhanced stop signal checking that handles both local and Redis-based signals.
+        """
+        if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
+            return True
+        return False
+
+    def set_stop_event(self, stop_event: threading.Event) -> None:
+        """
+        Set the stop event for this provider to enable cancellation.
+        
+        Args:
+            stop_event: Threading event to signal cancellation
+        """
+        self._stop_event = stop_event
+
+    def log_relationship_discovery(self, source_node: str, target_node: str,
+                                 relationship_type: str,
+                                 raw_data: Dict[str, Any],
+                                 discovery_method: str) -> None:
+        """
+        Log discovery of a new relationship.
+
+        Args:
+            source_node: Source node identifier
+            target_node: Target node identifier
+            relationship_type: Type of relationship
+            raw_data: Raw data from provider
+            discovery_method: Method used for discovery
+        """
+        self.total_relationships_found += 1
+
+        self.logger.log_relationship_discovery(
+            source_node=source_node,
+            target_node=target_node,
+            relationship_type=relationship_type,
+            provider=self.name,
+            raw_data=raw_data,
+            discovery_method=discovery_method
+        )
+
+    def get_statistics(self) -> Dict[str, Any]:
+        """
+        Get provider statistics.
+
+        Returns:
+            Dictionary containing provider performance metrics
+        """
+        return {
+            'name': self.name,
+            'total_requests': self.total_requests,
+            'successful_requests': self.successful_requests,
+            'failed_requests': self.failed_requests,
+            'success_rate': (self.successful_requests / self.total_requests * 100) if self.total_requests > 0 else 0,
+            'relationships_found': self.total_relationships_found,
+            'rate_limit': self.config.get_rate_limit(self.name)
+        }

providers/correlation_provider.py

@@ -0,0 +1,283 @@
+# dnsrecon-reduced/providers/correlation_provider.py
+
+import re
+from typing import Dict, Any, List
+from datetime import datetime, timezone
+
+from .base_provider import BaseProvider
+from core.provider_result import ProviderResult
+from core.graph_manager import NodeType, GraphManager
+
+class CorrelationProvider(BaseProvider):
+    """
+    A provider that finds correlations between nodes in the graph.
+    UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+    """
+
+    def __init__(self, name: str = "correlation", session_config=None):
+        """
+        Initialize the correlation provider.
+        """
+        super().__init__(name, session_config=session_config)
+        self.graph: GraphManager | None = None
+        self.correlation_index = {}
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
+        self.EXCLUDED_KEYS = [
+            'cert_source',
+            'a_records',
+            'mx_records',
+            'ns_records',
+            'ptr_records',
+            'cert_issuer_ca_id',
+            'cert_common_name',
+            'cert_validity_period_days',
+            'cert_issuer_name',
+            'cert_entry_timestamp',
+            'cert_serial_number', # useless
+            'cert_not_before',
+            'cert_not_after',
+            'dns_ttl',
+            'timestamp',
+            'last_update',
+            'updated_timestamp',
+            'discovery_timestamp',
+            'query_timestamp',
+            'shodan_ip_str',
+            'shodan_a_record',
+        ]
+
+    def get_name(self) -> str:
+        """Return the provider name."""
+        return "correlation"
+
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "Correlation Engine"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': True}
+
+    def is_available(self) -> bool:
+        """Check if the provider is available and properly configured."""
+        return True
+
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        Query the provider for information about a domain.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+        """
+        return self._find_correlations(domain)
+
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        Query the provider for information about an IP address.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+        """
+        return self._find_correlations(ip)
+
+    def set_graph_manager(self, graph_manager: GraphManager):
+        """
+        Set the graph manager for the provider to use.
+        """
+        self.graph = graph_manager
+
+    def _find_correlations(self, node_id: str) -> ProviderResult:
+        """
+        Find correlations for a given node with enhanced filtering and error handling.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring and list value processing.
+        """
+        result = ProviderResult()
+        discovery_time = datetime.now(timezone.utc)
+        
+        # Enhanced safety checks
+        if not self.graph or not self.graph.graph.has_node(node_id):
+            return result
+
+        try:
+            node_attributes = self.graph.graph.nodes[node_id].get('attributes', [])
+            
+            # Ensure attributes is a list (handle legacy data)
+            if not isinstance(node_attributes, list):
+                return result
+                
+            correlations_found = 0
+            
+            for attr in node_attributes:
+                if not isinstance(attr, dict):
+                    continue
+                    
+                attr_name = attr.get('name', '')
+                attr_value = attr.get('value')
+                attr_provider = attr.get('provider', 'unknown')
+
+                # Prepare a list of values to iterate over
+                values_to_process = []
+                if isinstance(attr_value, list):
+                    values_to_process.extend(attr_value)
+                else:
+                    values_to_process.append(attr_value)
+
+                for value_item in values_to_process:
+                    # Enhanced filtering logic
+                    should_exclude = self._should_exclude_attribute(attr_name, value_item)
+                    
+                    if should_exclude:
+                        continue
+
+                    # Build correlation index
+                    if value_item not in self.correlation_index:
+                        self.correlation_index[value_item] = {
+                            'nodes': set(),
+                            'sources': []
+                        }
+
+                    self.correlation_index[value_item]['nodes'].add(node_id)
+
+                    source_info = {
+                        'node_id': node_id,
+                        'provider': attr_provider,
+                        'attribute': attr_name,
+                        'path': f"{attr_provider}_{attr_name}"
+                    }
+
+                    # Avoid duplicate sources
+                    existing_sources = [s for s in self.correlation_index[value_item]['sources']
+                                    if s['node_id'] == node_id and s['path'] == source_info['path']]
+                    if not existing_sources:
+                        self.correlation_index[value_item]['sources'].append(source_info)
+
+                    # Create correlation if we have multiple nodes with this value
+                    if len(self.correlation_index[value_item]['nodes']) > 1:
+                        self._create_correlation_relationships(value_item, self.correlation_index[value_item], result, discovery_time)
+                        correlations_found += 1
+                    
+            # Log correlation results
+            if correlations_found > 0:
+                self.logger.logger.info(f"Found {correlations_found} correlations for node {node_id}")
+                
+        except Exception as e:
+            self.logger.logger.error(f"Error finding correlations for {node_id}: {e}")
+            
+        return result
+    
+    def _should_exclude_attribute(self, attr_name: str, attr_value: Any) -> bool:
+        """
+        Enhanced logic to determine if an attribute should be excluded from correlation.
+        """
+        # Check against excluded keys (exact match or substring)
+        if any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS):
+            return True
+        
+        # Value type filtering
+        if not isinstance(attr_value, (str, int, float, bool)) or attr_value is None:
+            return True
+        
+        # Boolean values are not useful for correlation
+        if isinstance(attr_value, bool):
+            return True
+            
+        # String value filtering
+        if isinstance(attr_value, str):                
+            # Date/timestamp strings
+            if self.date_pattern.match(attr_value):
+                return True
+                
+            # Common non-useful values
+            if attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']:
+                return True
+                
+            # Very long strings that are likely unique (> 100 chars)
+            if len(attr_value) > 100:
+                return True
+        
+        # Numeric value filtering  
+        if isinstance(attr_value, (int, float)):
+            # Very common values
+            if attr_value in [0, 1]:
+                return True
+                
+            # Very large numbers (likely timestamps or unique IDs)
+            if abs(attr_value) > 1000000:
+                return True
+        
+        return False
+    
+    def _create_correlation_relationships(self, value: Any, correlation_data: Dict[str, Any], 
+                                        result: ProviderResult, discovery_time: datetime):
+        """
+        Create correlation relationships with enhanced deduplication and validation.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+        """
+        correlation_node_id = f"corr_{hash(str(value)) & 0x7FFFFFFF}"
+        nodes = correlation_data['nodes']
+        sources = correlation_data['sources']
+        
+        # Only create correlations if we have meaningful nodes (more than 1)
+        if len(nodes) < 2:
+            return
+            
+        # Limit correlation size to prevent overly large correlation objects
+        MAX_CORRELATION_SIZE = 50
+        if len(nodes) > MAX_CORRELATION_SIZE:
+            # Sample the nodes to keep correlation manageable
+            import random
+            sampled_nodes = random.sample(list(nodes), MAX_CORRELATION_SIZE)
+            nodes = set(sampled_nodes)
+            # Filter sources to match sampled nodes
+            sources = [s for s in sources if s['node_id'] in nodes]
+
+        # Add the correlation node as an attribute to the result
+        result.add_attribute(
+            target_node=correlation_node_id,
+            name="correlation_value",
+            value=value,
+            attr_type=str(type(value).__name__),
+            provider=self.name,
+            metadata={
+                'correlated_nodes': list(nodes),
+                'sources': sources,
+                'correlation_size': len(nodes),
+                'value_type': type(value).__name__
+            }
+        )
+
+        # Create relationships with source validation and enhanced timestamps
+        created_relationships = set()
+        
+        for source in sources:
+            node_id = source['node_id']
+            provider = source['provider']
+            attribute = source['attribute']
+            
+            # Skip if we've already created this relationship
+            relationship_key = (node_id, correlation_node_id)
+            if relationship_key in created_relationships:
+                continue
+                
+            relationship_label = f"corr_{provider}_{attribute}"
+
+            # Enhanced raw_data with discovery timestamp for time-based edge coloring
+            raw_data = {
+                'correlation_value': value,
+                'original_attribute': attribute,
+                'correlation_type': 'attribute_matching',
+                'correlation_size': len(nodes),
+                'discovery_timestamp': discovery_time.isoformat(),
+                'relevance_timestamp': discovery_time.isoformat()  # Correlation data is "fresh" when discovered
+            }
+
+            # Add the relationship to the result
+            result.add_relationship(
+                source_node=node_id,
+                target_node=correlation_node_id,
+                relationship_type=relationship_label,
+                provider=self.name,
+                raw_data=raw_data
+            )
+            
+            created_relationships.add(relationship_key)

providers/crtsh_provider.py

@@ -0,0 +1,746 @@
+# DNScope/providers/crtsh_provider.py
+
+import json
+import re
+from pathlib import Path
+from typing import List, Dict, Any, Set, Optional
+from urllib.parse import quote
+from datetime import datetime, timezone
+import requests
+
+from .base_provider import BaseProvider
+from core.provider_result import ProviderResult
+from utils.helpers import _is_valid_domain
+from core.logger import get_forensic_logger
+
+class CrtShProvider(BaseProvider):
+    """
+    Provider for querying crt.sh certificate transparency database.
+    FIXED: Improved caching logic and error handling to prevent infinite retry loops.
+    Returns standardized ProviderResult objects with caching support.
+    UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
+    """
+    
+    def __init__(self, name=None, session_config=None):
+        """Initialize CrtSh provider with session-specific configuration."""
+        super().__init__(
+            name="crtsh",
+            rate_limit=60,
+            timeout=15,
+            session_config=session_config
+        )
+        self.base_url = "https://crt.sh/"
+        self._stop_event = None
+        
+        # Initialize cache directory
+        self.domain_cache_dir = Path('cache') / 'crtsh'
+        self.domain_cache_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Compile regex for date filtering for efficiency
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
+
+    def get_name(self) -> str:
+        """Return the provider name."""
+        return "crtsh"
+    
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "crt.sh"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': False}
+
+    def is_available(self) -> bool:
+        """Check if the provider is configured to be used."""
+        return True
+    
+    def _get_cache_file_path(self, domain: str) -> Path:
+        """Generate cache file path for a domain."""
+        safe_domain = domain.replace('.', '_').replace('/', '_').replace('\\', '_')
+        return self.domain_cache_dir / f"{safe_domain}.json"
+    
+    def _get_cache_status(self, cache_file_path: Path) -> str:
+        """
+        FIXED: More robust cache status checking with better error handling.
+        Returns: 'not_found', 'fresh', or 'stale'
+        """
+        if not cache_file_path.exists():
+            return "not_found"
+        
+        try:
+            # Check if file is readable and not corrupted
+            if cache_file_path.stat().st_size == 0:
+                self.logger.logger.warning(f"Empty cache file: {cache_file_path}")
+                return "stale"
+                
+            with open(cache_file_path, 'r', encoding='utf-8') as f:
+                cache_data = json.load(f)
+            
+            # Validate cache structure
+            if not isinstance(cache_data, dict):
+                self.logger.logger.warning(f"Invalid cache structure: {cache_file_path}")
+                return "stale"
+            
+            last_query_str = cache_data.get("last_upstream_query")
+            if not last_query_str or not isinstance(last_query_str, str):
+                self.logger.logger.warning(f"Missing or invalid last_upstream_query: {cache_file_path}")
+                return "stale"
+            
+            try:
+                # More robust datetime parsing
+                if last_query_str.endswith('Z'):
+                    last_query = datetime.fromisoformat(last_query_str.replace('Z', '+00:00'))
+                elif '+' in last_query_str or last_query_str.endswith('UTC'):
+                    # Handle various timezone formats
+                    clean_time = last_query_str.replace('UTC', '').strip()
+                    if '+' in clean_time:
+                        clean_time = clean_time.split('+')[0]
+                    last_query = datetime.fromisoformat(clean_time).replace(tzinfo=timezone.utc)
+                else:
+                    last_query = datetime.fromisoformat(last_query_str).replace(tzinfo=timezone.utc)
+                    
+            except (ValueError, AttributeError) as e:
+                self.logger.logger.warning(f"Failed to parse timestamp in cache {cache_file_path}: {e}")
+                return "stale"
+            
+            hours_since_query = (datetime.now(timezone.utc) - last_query).total_seconds() / 3600
+            cache_timeout = self.config.cache_timeout_hours
+            
+            if hours_since_query < cache_timeout:
+                return "fresh"
+            else:
+                return "stale"
+                
+        except (json.JSONDecodeError, OSError, PermissionError) as e:
+            self.logger.logger.warning(f"Cache file error for {cache_file_path}: {e}")
+            # FIXED: Try to remove corrupted cache file
+            try:
+                cache_file_path.unlink()
+                self.logger.logger.info(f"Removed corrupted cache file: {cache_file_path}")
+            except Exception:
+                pass
+            return "not_found"
+        except Exception as e:
+            self.logger.logger.error(f"Unexpected error checking cache status for {cache_file_path}: {e}")
+            return "stale"
+
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        FIXED: Simplified and more robust domain querying with better error handling.
+        UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
+        """
+        if not _is_valid_domain(domain):
+            return ProviderResult()
+        
+        if self._stop_event and self._stop_event.is_set():
+            return ProviderResult()
+
+        cache_file = self._get_cache_file_path(domain)
+        result = ProviderResult()
+
+        try:
+            cache_status = self._get_cache_status(cache_file)
+            
+            if cache_status == "fresh":
+                # Load from cache
+                result = self._load_from_cache(cache_file)
+                if result and (result.relationships or result.attributes):
+                    self.logger.logger.debug(f"Using fresh cached crt.sh data for {domain}")
+                    return result
+                else:
+                    # Cache exists but is empty, treat as stale
+                    cache_status = "stale"
+            
+            # Need to query API (either no cache, stale cache, or empty cache)
+            self.logger.logger.debug(f"Querying crt.sh API for {domain} (cache status: {cache_status})")
+            new_raw_certs = self._query_crtsh_api(domain)
+            
+            if self._stop_event and self._stop_event.is_set():
+                return ProviderResult()
+
+            # FIXED: Simplified processing - just process the new data
+            # Don't try to merge with stale cache as it can cause corruption
+            raw_certificates_to_process = new_raw_certs
+            
+            if cache_status == "stale":
+                self.logger.logger.info(f"Refreshed stale cache for {domain} with {len(raw_certificates_to_process)} certs")
+            else:
+                self.logger.logger.info(f"Created fresh cache for {domain} with {len(raw_certificates_to_process)} certs")
+            
+            result = self._process_certificates_to_result_fixed(domain, raw_certificates_to_process)
+            
+            # Save the result to cache
+            self._save_result_to_cache(cache_file, result, raw_certificates_to_process, domain)
+            
+            return result
+
+        except requests.exceptions.RequestException as e:
+            # FIXED: Don't re-raise network errors after long idle periods
+            # Instead return empty result and log the issue
+            self.logger.logger.warning(f"Network error querying crt.sh for {domain}: {e}")
+            
+            # Try to use stale cache if available
+            if cache_status == "stale":
+                try:
+                    stale_result = self._load_from_cache(cache_file)
+                    if stale_result and (stale_result.relationships or stale_result.attributes):
+                        self.logger.logger.info(f"Using stale cache for {domain} due to network error")
+                        return stale_result
+                except Exception as cache_error:
+                    self.logger.logger.warning(f"Failed to load stale cache for {domain}: {cache_error}")
+            
+            # Return empty result instead of raising - this prevents infinite retries
+            return ProviderResult()
+            
+        except Exception as e:
+            # FIXED: Handle any other exceptions gracefully
+            self.logger.logger.error(f"Unexpected error querying crt.sh for {domain}: {e}")
+            
+            # Try stale cache as fallback
+            try:
+                if cache_file.exists():
+                    fallback_result = self._load_from_cache(cache_file)
+                    if fallback_result and (fallback_result.relationships or fallback_result.attributes):
+                        self.logger.logger.info(f"Using cached data for {domain} due to processing error")
+                        return fallback_result
+            except Exception:
+                pass
+            
+            # Return empty result to prevent retries
+            return ProviderResult()
+
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        crt.sh does not support IP-based certificate queries effectively via its API.
+        """
+        return ProviderResult()
+
+    def _load_from_cache(self, cache_file_path: Path) -> ProviderResult:
+        """FIXED: More robust cache loading with better validation."""
+        try:
+            if not cache_file_path.exists() or cache_file_path.stat().st_size == 0:
+                return ProviderResult()
+                
+            with open(cache_file_path, 'r', encoding='utf-8') as f:
+                cache_content = json.load(f)
+            
+            if not isinstance(cache_content, dict):
+                self.logger.logger.warning(f"Invalid cache format in {cache_file_path}")
+                return ProviderResult()
+            
+            result = ProviderResult()
+            
+            # Reconstruct relationships with validation
+            relationships = cache_content.get("relationships", [])
+            if isinstance(relationships, list):
+                for rel_data in relationships:
+                    if not isinstance(rel_data, dict):
+                        continue
+                    try:
+                        result.add_relationship(
+                            source_node=rel_data.get("source_node", ""),
+                            target_node=rel_data.get("target_node", ""),
+                            relationship_type=rel_data.get("relationship_type", ""),
+                            provider=rel_data.get("provider", self.name),
+                            raw_data=rel_data.get("raw_data", {})
+                        )
+                    except (ValueError, TypeError) as e:
+                        self.logger.logger.warning(f"Skipping invalid relationship in cache: {e}")
+                        continue
+            
+            # Reconstruct attributes with validation
+            attributes = cache_content.get("attributes", [])
+            if isinstance(attributes, list):
+                for attr_data in attributes:
+                    if not isinstance(attr_data, dict):
+                        continue
+                    try:
+                        result.add_attribute(
+                            target_node=attr_data.get("target_node", ""),
+                            name=attr_data.get("name", ""),
+                            value=attr_data.get("value"),
+                            attr_type=attr_data.get("type", "unknown"),
+                            provider=attr_data.get("provider", self.name),
+                            metadata=attr_data.get("metadata", {})
+                        )
+                    except (ValueError, TypeError) as e:
+                        self.logger.logger.warning(f"Skipping invalid attribute in cache: {e}")
+                        continue
+            
+            return result
+            
+        except (json.JSONDecodeError, OSError, PermissionError) as e:
+            self.logger.logger.warning(f"Failed to load cache from {cache_file_path}: {e}")
+            return ProviderResult()
+        except Exception as e:
+            self.logger.logger.error(f"Unexpected error loading cache from {cache_file_path}: {e}")
+            return ProviderResult()
+
+    def _save_result_to_cache(self, cache_file_path: Path, result: ProviderResult, raw_certificates: List[Dict[str, Any]], domain: str) -> None:
+        """FIXED: More robust cache saving with atomic writes."""
+        try:
+            cache_data = {
+                "domain": domain,
+                "last_upstream_query": datetime.now(timezone.utc).isoformat(),
+                "raw_certificates": raw_certificates,
+                "relationships": [
+                    {
+                        "source_node": rel.source_node,
+                        "target_node": rel.target_node,
+                        "relationship_type": rel.relationship_type,
+                        "provider": rel.provider,
+                        "raw_data": rel.raw_data
+                    } for rel in result.relationships
+                ],
+                "attributes": [
+                    {
+                        "target_node": attr.target_node,
+                        "name": attr.name,
+                        "value": attr.value,
+                        "type": attr.type,
+                        "provider": attr.provider,
+                        "metadata": attr.metadata
+                    } for attr in result.attributes
+                ]
+            }
+            
+            cache_file_path.parent.mkdir(parents=True, exist_ok=True)
+            
+            # FIXED: Atomic write using temporary file
+            temp_file = cache_file_path.with_suffix('.tmp')
+            try:
+                with open(temp_file, 'w', encoding='utf-8') as f:
+                    json.dump(cache_data, f, separators=(',', ':'), default=str, ensure_ascii=False)
+                
+                # Atomic rename
+                temp_file.replace(cache_file_path)
+                self.logger.logger.debug(f"Saved cache for {domain} ({len(result.relationships)} relationships)")
+                
+            except Exception as e:
+                # Clean up temp file on error
+                if temp_file.exists():
+                    try:
+                        temp_file.unlink()
+                    except Exception:
+                        pass
+                raise e
+                
+        except Exception as e:
+            self.logger.logger.warning(f"Failed to save cache file for {domain}: {e}")
+    
+    def _query_crtsh_api(self, domain: str) -> List[Dict[str, Any]]:
+        """FIXED: More robust API querying with better error handling."""
+        url = f"{self.base_url}?q={quote(domain)}&output=json"
+        
+        try:
+            response = self.make_request(url, target_indicator=domain)
+            
+            if not response:
+                self.logger.logger.warning(f"No response from crt.sh for {domain}")
+                return []
+            
+            if response.status_code != 200:
+                self.logger.logger.warning(f"crt.sh returned status {response.status_code} for {domain}")
+                return []
+            
+            # FIXED: Better JSON parsing with error handling
+            try:
+                certificates = response.json()
+            except json.JSONDecodeError as e:
+                self.logger.logger.error(f"crt.sh returned invalid JSON for {domain}: {e}")
+                return []
+
+            if not certificates or not isinstance(certificates, list):
+                self.logger.logger.debug(f"crt.sh returned no certificates for {domain}")
+                return []
+            
+            self.logger.logger.debug(f"crt.sh returned {len(certificates)} certificates for {domain}")
+            return certificates
+            
+        except Exception as e:
+            self.logger.logger.error(f"Error querying crt.sh API for {domain}: {e}")
+            raise e
+
+    def _process_certificates_to_result_fixed(self, query_domain: str, certificates: List[Dict[str, Any]]) -> ProviderResult:
+        """
+        Process certificates to create proper domain and CA nodes.
+        FIXED: Better error handling and progress tracking.
+        UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
+        """
+        result = ProviderResult()
+
+        if self._stop_event and self._stop_event.is_set():
+            self.logger.logger.info(f"CrtSh processing cancelled before processing for domain: {query_domain}")
+            return result
+        
+        if not certificates:
+            self.logger.logger.debug(f"No certificates to process for {query_domain}")
+            return result
+        
+        # Check for incomplete data warning
+        incompleteness_warning = self._check_for_incomplete_data(query_domain, certificates)
+        if incompleteness_warning:
+            result.add_attribute(
+                target_node=query_domain,
+                name="crtsh_data_warning",
+                value=incompleteness_warning,
+                attr_type='metadata',
+                provider=self.name
+            )
+
+        all_discovered_domains = set()
+        processed_issuers = set()
+        processed_certs = 0
+
+        for i, cert_data in enumerate(certificates):
+            # FIXED: More frequent stop checks and progress logging
+            if i % 5 == 0:
+                if self._stop_event and self._stop_event.is_set():
+                    self.logger.logger.info(f"CrtSh processing cancelled at certificate {i}/{len(certificates)} for domain: {query_domain}")
+                    break
+                
+                if i > 0 and i % 100 == 0:
+                    self.logger.logger.debug(f"Processed {i}/{len(certificates)} certificates for {query_domain}")
+
+            try:
+                # Extract all domains from this certificate
+                cert_domains = self._extract_domains_from_certificate(cert_data)
+                if cert_domains:
+                    all_discovered_domains.update(cert_domains)
+
+                # Create CA nodes for certificate issuers with timestamp
+                issuer_name = self._parse_issuer_organization(cert_data.get('issuer_name', ''))
+                if issuer_name and issuer_name not in processed_issuers:
+                    # Enhanced raw_data with certificate timestamp for time-based edge coloring
+                    issuer_raw_data = {'issuer_dn': cert_data.get('issuer_name', '')}
+                    
+                    # Add certificate issue date (not_before) as relevance timestamp
+                    not_before = cert_data.get('not_before')
+                    if not_before:
+                        try:
+                            not_before_date = self._parse_certificate_date(not_before)
+                            issuer_raw_data['cert_not_before'] = not_before_date.isoformat()
+                            issuer_raw_data['relevance_timestamp'] = not_before_date.isoformat()  # Standardized field
+                        except Exception as e:
+                            self.logger.logger.debug(f"Failed to parse not_before date for issuer: {e}")
+                    
+                    result.add_relationship(
+                        source_node=query_domain,
+                        target_node=issuer_name,
+                        relationship_type='crtsh_cert_issuer',
+                        provider=self.name,
+                        raw_data=issuer_raw_data
+                    )
+                    processed_issuers.add(issuer_name)
+
+                # Add certificate metadata to each domain in this certificate
+                cert_metadata = self._extract_certificate_metadata(cert_data)
+                for cert_domain in cert_domains:
+                    if not _is_valid_domain(cert_domain):
+                        continue
+
+                    for key, value in cert_metadata.items():
+                        if value is not None:
+                            result.add_attribute(
+                                target_node=cert_domain,
+                                name=f"cert_{key}",
+                                value=value,
+                                attr_type='certificate_data',
+                                provider=self.name,
+                                metadata={'certificate_id': cert_data.get('id')}
+                            )
+                
+                processed_certs += 1
+                
+            except Exception as e:
+                self.logger.logger.warning(f"Error processing certificate {i} for {query_domain}: {e}")
+                continue
+        
+        # Check for stop event before creating final relationships
+        if self._stop_event and self._stop_event.is_set():
+            self.logger.logger.info(f"CrtSh query cancelled before relationship creation for domain: {query_domain}")
+            return result
+
+        # Create selective relationships to avoid large entities with enhanced timestamps
+        relationships_created = 0
+        for discovered_domain in all_discovered_domains:
+            if discovered_domain == query_domain:
+                continue
+
+            if not _is_valid_domain(discovered_domain):
+                continue
+
+            if self._should_create_relationship(query_domain, discovered_domain):
+                # Enhanced raw_data with certificate timestamp for domain relationships
+                domain_raw_data = {'relationship_type': 'certificate_discovery'}
+                
+                # Find the most recent certificate for this domain pair to use as timestamp
+                most_recent_cert = self._find_most_recent_cert_for_domains(
+                    certificates, query_domain, discovered_domain
+                )
+                if most_recent_cert:
+                    not_before = most_recent_cert.get('not_before')
+                    if not_before:
+                        try:
+                            not_before_date = self._parse_certificate_date(not_before)
+                            domain_raw_data['cert_not_before'] = not_before_date.isoformat()
+                            domain_raw_data['relevance_timestamp'] = not_before_date.isoformat()
+                        except Exception as e:
+                            self.logger.logger.debug(f"Failed to parse not_before date for domain relationship: {e}")
+
+                result.add_relationship(
+                    source_node=query_domain,
+                    target_node=discovered_domain,
+                    relationship_type='crtsh_san_certificate',
+                    provider=self.name,
+                    raw_data=domain_raw_data
+                )
+
+                self.log_relationship_discovery(
+                    source_node=query_domain,
+                    target_node=discovered_domain,
+                    relationship_type='crtsh_san_certificate',
+                    raw_data=domain_raw_data,
+                    discovery_method="certificate_transparency_analysis"
+                )
+                relationships_created += 1
+
+        self.logger.logger.info(f"CrtSh processing completed for {query_domain}: processed {processed_certs}/{len(certificates)} certificates, {len(all_discovered_domains)} domains, {relationships_created} relationships")
+        return result
+
+    def _find_most_recent_cert_for_domains(self, certificates: List[Dict[str, Any]], 
+                                         domain1: str, domain2: str) -> Optional[Dict[str, Any]]:
+        """
+        Find the most recent certificate that contains both domains.
+        Used for determining the relevance timestamp for domain relationships.
+        """
+        most_recent_cert = None
+        most_recent_date = None
+        
+        for cert in certificates:
+            # Check if this certificate contains both domains
+            cert_domains = self._extract_domains_from_certificate(cert)
+            if domain1 in cert_domains and domain2 in cert_domains:
+                not_before = cert.get('not_before')
+                if not_before:
+                    try:
+                        cert_date = self._parse_certificate_date(not_before)
+                        if most_recent_date is None or cert_date > most_recent_date:
+                            most_recent_date = cert_date
+                            most_recent_cert = cert
+                    except Exception:
+                        continue
+        
+        return most_recent_cert
+
+    # [Rest of the methods remain the same as in the original file]
+    def _should_create_relationship(self, source_domain: str, target_domain: str) -> bool:
+        """
+        Determine if a relationship should be created between two domains.
+        """
+        if target_domain.endswith(f'.{source_domain}') or source_domain.endswith(f'.{target_domain}'):
+            return True
+        
+        source_parts = source_domain.split('.')
+        target_parts = target_domain.split('.')
+        
+        if len(source_parts) >= 2 and len(target_parts) >= 2:
+            source_root = '.'.join(source_parts[-2:])
+            target_root = '.'.join(target_parts[-2:])
+            return source_root == target_root
+        
+        return False
+
+    def _extract_certificate_metadata(self, cert_data: Dict[str, Any]) -> Dict[str, Any]:
+        """Extract comprehensive metadata from certificate data."""
+        raw_issuer_name = cert_data.get('issuer_name', '')
+        parsed_issuer_name = self._parse_issuer_organization(raw_issuer_name)
+        
+        metadata = {
+            'certificate_id': cert_data.get('id'),
+            'serial_number': cert_data.get('serial_number'),
+            'issuer_name': parsed_issuer_name,
+            'issuer_ca_id': cert_data.get('issuer_ca_id'),
+            'common_name': cert_data.get('common_name'),
+            'not_before': cert_data.get('not_before'),
+            'not_after': cert_data.get('not_after'),
+            'entry_timestamp': cert_data.get('entry_timestamp'),
+            'source': 'crtsh'
+        }
+        
+        try:
+            if metadata['not_before'] and metadata['not_after']:
+                not_before = self._parse_certificate_date(metadata['not_before'])
+                not_after = self._parse_certificate_date(metadata['not_after'])
+                
+                metadata['validity_period_days'] = (not_after - not_before).days
+                metadata['is_currently_valid'] = self._is_cert_valid(cert_data)
+                metadata['expires_soon'] = (not_after - datetime.now(timezone.utc)).days <= 30
+                
+                metadata['not_before'] = not_before.isoformat()
+                metadata['not_after'] = not_after.isoformat()
+                
+        except Exception as e:
+            self.logger.logger.debug(f"Error computing certificate metadata: {e}")
+            metadata['is_currently_valid'] = False
+            metadata['expires_soon'] = False
+        
+        return metadata
+
+    def _parse_issuer_organization(self, issuer_dn: str) -> str:
+        """Parse the issuer Distinguished Name to extract just the organization name."""
+        if not issuer_dn:
+            return issuer_dn
+        
+        try:
+            components = [comp.strip() for comp in issuer_dn.split(',')]
+            
+            for component in components:
+                if component.startswith('O='):
+                    org_name = component[2:].strip()
+                    if org_name.startswith('"') and org_name.endswith('"'):
+                        org_name = org_name[1:-1]
+                    return org_name
+            
+            return issuer_dn
+            
+        except Exception as e:
+            self.logger.logger.debug(f"Failed to parse issuer DN '{issuer_dn}': {e}")
+            return issuer_dn
+
+    def _parse_certificate_date(self, date_string: str) -> datetime:
+        """Parse certificate date from crt.sh format."""
+        if not date_string:
+            raise ValueError("Empty date string")
+
+        try:
+            if isinstance(date_string, datetime):
+                return date_string.replace(tzinfo=timezone.utc)
+            if date_string.endswith('Z'):
+                return datetime.fromisoformat(date_string[:-1]).replace(tzinfo=timezone.utc)
+            elif '+' in date_string or date_string.endswith('UTC'):
+                date_string = date_string.replace('UTC', '').strip()
+                if '+' in date_string:
+                    date_string = date_string.split('+')[0]
+                return datetime.fromisoformat(date_string).replace(tzinfo=timezone.utc)
+            else:
+                return datetime.fromisoformat(date_string).replace(tzinfo=timezone.utc)
+        except Exception as e:
+            try:
+                return datetime.strptime(date_string[:19], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=timezone.utc)
+            except Exception:
+                raise ValueError(f"Unable to parse date: {date_string}") from e
+
+    def _is_cert_valid(self, cert_data: Dict[str, Any]) -> bool:
+        """Check if a certificate is currently valid based on its expiry date."""
+        try:
+            not_after_str = cert_data.get('not_after')
+            if not not_after_str:
+                return False
+
+            not_after_date = self._parse_certificate_date(not_after_str)
+            not_before_str = cert_data.get('not_before')
+
+            now = datetime.now(timezone.utc)
+            is_not_expired = not_after_date > now
+
+            if not_before_str:
+                not_before_date = self._parse_certificate_date(not_before_str)
+                is_not_before_valid = not_before_date <= now
+                return is_not_expired and is_not_before_valid
+
+            return is_not_expired
+
+        except Exception as e:
+            return False
+
+    def _extract_domains_from_certificate(self, cert_data: Dict[str, Any]) -> Set[str]:
+        """Extract all domains from certificate data."""
+        domains = set()
+        
+        common_name = cert_data.get('common_name', '')
+        if common_name:
+            cleaned_cn = self._clean_domain_name(common_name)
+            if cleaned_cn:
+                domains.update(cleaned_cn)
+        
+        name_value = cert_data.get('name_value', '')
+        if name_value:
+            for line in name_value.split('\n'):
+                cleaned_domains = self._clean_domain_name(line.strip())
+                if cleaned_domains:
+                    domains.update(cleaned_domains)
+        
+        return domains
+
+    def _clean_domain_name(self, domain_name: str) -> List[str]:
+        """Clean and normalize domain name from certificate data."""
+        if not domain_name:
+            return []
+
+        domain = domain_name.strip().lower()
+
+        if domain.startswith(('http://', 'https://')):
+            domain = domain.split('://', 1)[1]
+
+        if '/' in domain:
+            domain = domain.split('/', 1)[0]
+
+        if ':' in domain and not domain.count(':') > 1:
+            domain = domain.split(':', 1)[0]
+
+        cleaned_domains = []
+        if domain.startswith('*.'):
+            cleaned_domains.append(domain)
+            cleaned_domains.append(domain[2:])
+        else:
+            cleaned_domains.append(domain)
+
+        final_domains = []
+        for d in cleaned_domains:
+            d = re.sub(r'[^\w\-\.]', '', d)
+            if d and not d.startswith(('.', '-')) and not d.endswith(('.', '-')):
+                final_domains.append(d)
+
+        return [d for d in final_domains if _is_valid_domain(d)]
+
+
+    def _determine_relationship_context(self, cert_domain: str, query_domain: str) -> str:
+        """Determine the context of the relationship between certificate domain and query domain."""
+        if cert_domain == query_domain:
+            return 'exact_match'
+        elif cert_domain.endswith(f'.{query_domain}'):
+            return 'subdomain'
+        elif query_domain.endswith(f'.{cert_domain}'):
+            return 'parent_domain'
+        else:
+            return 'related_domain'
+        
+    def _check_for_incomplete_data(self, domain: str, certificates: List[Dict[str, Any]]) -> Optional[str]:
+        """
+        Analyzes the certificate list to heuristically detect if the data from crt.sh is incomplete.
+        """
+        cert_count = len(certificates)
+
+        if cert_count >= 10000:
+            return f"Result likely truncated; received {cert_count} certificates, which may be the maximum limit."
+
+        if cert_count > 1000: 
+            latest_expiry = None
+            for cert in certificates:
+                try:
+                    not_after = self._parse_certificate_date(cert.get('not_after'))
+                    if latest_expiry is None or not_after > latest_expiry:
+                        latest_expiry = not_after
+                except (ValueError, TypeError):
+                    continue
+
+            if latest_expiry and (datetime.now(timezone.utc) - latest_expiry).days > 365:
+                 return f"Incomplete data suspected: The latest certificate expired more than a year ago ({latest_expiry.strftime('%Y-%m-%d')})."
+
+        return None

providers/dns_provider.py

@@ -0,0 +1,288 @@
+# DNScope/providers/dns_provider.py
+
+from dns import resolver, reversename
+from typing import Dict
+from datetime import datetime, timezone
+from .base_provider import BaseProvider
+from core.provider_result import ProviderResult
+from utils.helpers import _is_valid_ip, _is_valid_domain, get_ip_version
+
+
+class DNSProvider(BaseProvider):
+    """
+    Provider for standard DNS resolution and reverse DNS lookups.
+    Now returns standardized ProviderResult objects with IPv4 and IPv6 support.
+    UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+    """
+
+    def __init__(self, name=None, session_config=None):
+        """Initialize DNS provider with session-specific configuration."""
+        super().__init__(
+            name="dns",
+            rate_limit=100,
+            timeout=10,
+            session_config=session_config
+        )
+
+        # Configure DNS resolver
+        self.resolver = resolver.Resolver()
+        self.resolver.timeout = 5
+        self.resolver.lifetime = 10
+
+    def get_name(self) -> str:
+        """Return the provider name."""
+        return "dns"
+
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "DNS"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': True}
+
+    def is_available(self) -> bool:
+        """DNS is always available - no API key required."""
+        return True
+
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        Query DNS records for the domain to discover relationships and attributes.
+        FIXED: Now creates separate attributes for each DNS record type.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+        
+        Args:
+            domain: Domain to investigate
+            
+        Returns:
+            ProviderResult containing discovered relationships and attributes
+        """
+        if not _is_valid_domain(domain):
+            return ProviderResult()
+
+        result = ProviderResult()
+        discovery_time = datetime.now(timezone.utc)
+
+        # Query all record types - each gets its own attribute
+        for record_type in ['A', 'AAAA', 'CNAME', 'MX', 'NS', 'SOA', 'TXT', 'SRV', 'CAA']:
+            try:
+                self._query_record(domain, record_type, result, discovery_time)
+            #except resolver.NoAnswer:
+                # This is not an error, just a confirmation that the record doesn't exist.
+                #self.logger.logger.debug(f"No {record_type} record found for {domain}")
+            except Exception as e:
+                self.failed_requests += 1
+                self.logger.logger.debug(f"{record_type} record query failed for {domain}: {e}")
+
+        return result
+
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        Query reverse DNS for the IP address (supports both IPv4 and IPv6).
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+
+        Args:
+            ip: IP address to investigate (IPv4 or IPv6)
+
+        Returns:
+            ProviderResult containing discovered relationships and attributes
+        """
+        if not _is_valid_ip(ip):
+            return ProviderResult()
+
+        result = ProviderResult()
+        ip_version = get_ip_version(ip)
+        discovery_time = datetime.now(timezone.utc)
+
+        try:
+            # Perform reverse DNS lookup (works for both IPv4 and IPv6)
+            self.total_requests += 1
+            reverse_name = reversename.from_address(ip)
+            response = self.resolver.resolve(reverse_name, 'PTR')
+            self.successful_requests += 1
+
+            ptr_records = []
+            for ptr_record in response:
+                hostname = str(ptr_record).rstrip('.')
+
+                if _is_valid_domain(hostname):
+                    # Determine appropriate forward relationship type based on IP version
+                    if ip_version == 6:
+                        relationship_type = 'dns_aaaa_record'
+                        record_prefix = 'AAAA'
+                    else:
+                        relationship_type = 'dns_a_record'
+                        record_prefix = 'A'
+                    
+                    # Enhanced raw_data with discovery timestamp for time-based edge coloring
+                    raw_data = {
+                        'query_type': 'PTR',
+                        'ip_address': ip,
+                        'ip_version': ip_version,
+                        'hostname': hostname,
+                        'ttl': response.ttl,
+                        'discovery_timestamp': discovery_time.isoformat(),
+                        'relevance_timestamp': discovery_time.isoformat()  # DNS data is "fresh" when discovered
+                    }
+                    
+                    # Add the relationship
+                    result.add_relationship(
+                        source_node=ip,
+                        target_node=hostname,
+                        relationship_type='dns_ptr_record',
+                        provider=self.name,
+                        raw_data=raw_data
+                    )
+
+                    # Add to PTR records list
+                    ptr_records.append(f"PTR: {hostname}")
+
+                    # Log the relationship discovery
+                    self.log_relationship_discovery(
+                        source_node=ip,
+                        target_node=hostname,
+                        relationship_type='dns_ptr_record',
+                        raw_data=raw_data,
+                        discovery_method=f"reverse_dns_lookup_ipv{ip_version}"
+                    )
+
+            # Add PTR records as separate attribute
+            if ptr_records:
+                result.add_attribute(
+                    target_node=ip,
+                    name='ptr_records',  # Specific name for PTR records
+                    value=ptr_records,
+                    attr_type='dns_record',
+                    provider=self.name,
+                    metadata={'ttl': response.ttl, 'ip_version': ip_version}
+                )
+
+        except resolver.NXDOMAIN:
+            self.failed_requests += 1
+            self.logger.logger.debug(f"Reverse DNS lookup failed for {ip}: NXDOMAIN")
+        except Exception as e:
+            self.failed_requests += 1
+            self.logger.logger.debug(f"Reverse DNS lookup failed for {ip}: {e}")
+            # Re-raise the exception so the scanner can handle the failure
+            raise e
+
+        return result
+
+    def _query_record(self, domain: str, record_type: str, result: ProviderResult, discovery_time: datetime) -> None:
+        """
+        FIXED: Query DNS records with unique attribute names for each record type.
+        Enhanced to better handle IPv6 AAAA records.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
+        """
+        try:
+            self.total_requests += 1
+            response = self.resolver.resolve(domain, record_type)
+            self.successful_requests += 1
+
+            dns_records = []
+            
+            for record in response:
+                target = ""
+                if record_type in ['A', 'AAAA']:
+                    target = str(record)
+                    # Validate that the IP address is properly formed
+                    if not _is_valid_ip(target):
+                        self.logger.logger.debug(f"Invalid IP address in {record_type} record: {target}")
+                        continue
+                elif record_type in ['CNAME', 'NS', 'PTR']:
+                    target = str(record.target).rstrip('.')
+                elif record_type == 'MX':
+                    target = str(record.exchange).rstrip('.')
+                elif record_type == 'SOA':
+                    target = str(record.mname).rstrip('.')
+                elif record_type in ['TXT']:
+                    # Keep raw TXT record value
+                    txt_value = str(record).strip('"')
+                    dns_records.append(txt_value)  # Just the value for TXT
+                    continue
+                elif record_type == 'SRV':
+                    target = str(record.target).rstrip('.')
+                elif record_type == 'CAA':
+                    # Keep raw CAA record format
+                    caa_value = f"{record.flags} {record.tag.decode('utf-8')} \"{record.value.decode('utf-8')}\""
+                    dns_records.append(caa_value)  # Just the value for CAA
+                    continue
+                else:
+                    target = str(record)
+
+                if target:
+                    # Determine IP version for metadata if this is an IP record
+                    ip_version = None
+                    if record_type in ['A', 'AAAA'] and _is_valid_ip(target):
+                        ip_version = get_ip_version(target)
+                    
+                    # Enhanced raw_data with discovery timestamp for time-based edge coloring
+                    raw_data = {
+                        'query_type': record_type,
+                        'domain': domain,
+                        'value': target,
+                        'ttl': response.ttl,
+                        'discovery_timestamp': discovery_time.isoformat(),
+                        'relevance_timestamp': discovery_time.isoformat()  # DNS data is "fresh" when discovered
+                    }
+                    
+                    if ip_version:
+                        raw_data['ip_version'] = ip_version
+                    
+                    relationship_type = f"dns_{record_type.lower()}_record"
+
+                    # Add relationship
+                    result.add_relationship(
+                        source_node=domain,
+                        target_node=target,
+                        relationship_type=relationship_type,
+                        provider=self.name,
+                        raw_data=raw_data
+                    )
+
+                    # Add target to records list
+                    dns_records.append(target)
+
+                    # Log relationship discovery with IP version info
+                    discovery_method = f"dns_{record_type.lower()}_record"
+                    if ip_version:
+                        discovery_method += f"_ipv{ip_version}"
+                    
+                    self.log_relationship_discovery(
+                        source_node=domain,
+                        target_node=target,
+                        relationship_type=relationship_type,
+                        raw_data=raw_data,
+                        discovery_method=discovery_method
+                    )
+
+            # FIXED: Create attribute with specific name for each record type
+            if dns_records:
+                # Use record type specific attribute name (e.g., 'a_records', 'mx_records', etc.)
+                attribute_name = f"{record_type.lower()}_records"
+                
+                metadata = {'record_type': record_type, 'ttl': response.ttl}
+                
+                # Add IP version info for A/AAAA records
+                if record_type in ['A', 'AAAA'] and dns_records:
+                    first_ip_version = get_ip_version(dns_records[0])
+                    if first_ip_version:
+                        metadata['ip_version'] = first_ip_version
+                
+                result.add_attribute(
+                    target_node=domain,
+                    name=attribute_name,  # UNIQUE name for each record type!
+                    value=dns_records,
+                    attr_type='dns_record_list',
+                    provider=self.name,
+                    metadata=metadata
+                )
+
+        except Exception as e:
+            self.failed_requests += 1
+            self.logger.logger.debug(f"{record_type} record query failed for {domain}: {e}")
+            raise e

providers/shodan_provider.py

@@ -0,0 +1,472 @@
+# DNScope/providers/shodan_provider.py
+
+import json
+from pathlib import Path
+from typing import Dict, Any
+from datetime import datetime, timezone
+import requests
+
+from .base_provider import BaseProvider
+from core.provider_result import ProviderResult
+from utils.helpers import _is_valid_ip, _is_valid_domain, get_ip_version, normalize_ip
+
+
+class ShodanProvider(BaseProvider):
+    """
+    Provider for querying Shodan API for IP address information.
+    Now returns standardized ProviderResult objects with caching support for IPv4 and IPv6.
+    UPDATED: Enhanced with last_seen timestamp for time-based edge coloring.
+    """
+
+    def __init__(self, name=None, session_config=None):
+        """Initialize Shodan provider with session-specific configuration."""
+        super().__init__(
+            name="shodan",
+            rate_limit=60,
+            timeout=30,
+            session_config=session_config
+        )
+        self.base_url = "https://api.shodan.io"
+        self.api_key = self.config.get_api_key('shodan')
+        
+        # FIXED: Don't fail initialization on connection issues - defer to actual usage
+        self._connection_tested = False
+        self._connection_works = False
+
+        # Initialize cache directory
+        self.cache_dir = Path('cache') / 'shodan'
+        self.cache_dir.mkdir(parents=True, exist_ok=True)
+
+    def _check_api_connection(self) -> bool:
+        """
+        FIXED: Lazy connection checking - only test when actually needed.
+        Don't block provider initialization on network issues.
+        """
+        if self._connection_tested:
+            return self._connection_works
+            
+        if not self.api_key:
+            self._connection_tested = True
+            self._connection_works = False
+            return False
+            
+        try:
+            print(f"Testing Shodan API connection with key: {self.api_key[:8]}...")
+            response = self.session.get(f"{self.base_url}/api-info?key={self.api_key}", timeout=5)
+            self._connection_works = response.status_code == 200
+            print(f"Shodan API test result: {response.status_code} - {'Success' if self._connection_works else 'Failed'}")
+        except requests.exceptions.RequestException as e:
+            print(f"Shodan API connection test failed: {e}")
+            self._connection_works = False
+        finally:
+            self._connection_tested = True
+            
+        return self._connection_works
+
+    def is_available(self) -> bool:
+        """
+        FIXED: Check if Shodan provider is available based on API key presence.
+        Don't require successful connection test during initialization.
+        """
+        has_api_key = self.api_key is not None and len(self.api_key.strip()) > 0
+        
+        if not has_api_key:
+            return False
+            
+        # FIXED: Only test connection on first actual usage, not during initialization
+        return True
+
+    def get_name(self) -> str:
+        """Return the provider name."""
+        return "shodan"
+
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "Shodan"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return True
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': False, 'ips': True}
+
+    def _get_cache_file_path(self, ip: str) -> Path:
+        """
+        Generate cache file path for an IP address (IPv4 or IPv6).
+        IPv6 addresses contain colons which are replaced with underscores for filesystem safety.
+        """
+        # Normalize the IP address first to ensure consistent caching
+        normalized_ip = normalize_ip(ip)
+        if not normalized_ip:
+            # Fallback for invalid IPs
+            safe_ip = ip.replace('.', '_').replace(':', '_')
+        else:
+            # Replace problematic characters for both IPv4 and IPv6
+            safe_ip = normalized_ip.replace('.', '_').replace(':', '_')
+        
+        return self.cache_dir / f"{safe_ip}.json"
+
+    def _get_cache_status(self, cache_file_path: Path) -> str:
+        """
+        Check cache status for an IP.
+        Returns: 'not_found', 'fresh', or 'stale'
+        """
+        if not cache_file_path.exists():
+            return "not_found"
+        
+        try:
+            with open(cache_file_path, 'r') as f:
+                cache_data = json.load(f)
+            
+            last_query_str = cache_data.get("last_upstream_query")
+            if not last_query_str:
+                return "stale"
+            
+            last_query = datetime.fromisoformat(last_query_str.replace('Z', '+00:00'))
+            hours_since_query = (datetime.now(timezone.utc) - last_query).total_seconds() / 3600
+            
+            cache_timeout = self.config.cache_timeout_hours
+            if hours_since_query < cache_timeout:
+                return "fresh"
+            else:
+                return "stale"
+                
+        except (json.JSONDecodeError, ValueError, KeyError):
+            return "stale"
+
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        Shodan does not support domain queries. This method returns an empty result.
+        """
+        return ProviderResult()
+
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        Query Shodan for information about an IP address (IPv4 or IPv6), with caching of processed data.
+        FIXED: Proper 404 handling to prevent unnecessary retries.
+        UPDATED: Enhanced with last_seen timestamp extraction for time-based edge coloring.
+        
+        Args:
+            ip: IP address to investigate (IPv4 or IPv6)
+            
+        Returns:
+            ProviderResult containing discovered relationships and attributes
+            
+        Raises:
+            Exception: For temporary failures that should be retried (timeouts, 502/503 errors, connection issues)
+        """
+        if not _is_valid_ip(ip):
+            return ProviderResult()
+            
+        # Test connection only when actually making requests
+        if not self._check_api_connection():
+            print(f"Shodan API not available for {ip} - API key: {'present' if self.api_key else 'missing'}")
+            return ProviderResult()
+        
+        # Normalize IP address for consistent processing
+        normalized_ip = normalize_ip(ip)
+        if not normalized_ip:
+            return ProviderResult()
+        
+        cache_file = self._get_cache_file_path(normalized_ip)
+        cache_status = self._get_cache_status(cache_file)
+        
+        if cache_status == "fresh":
+            self.logger.logger.debug(f"Using fresh cache for Shodan query: {normalized_ip}")
+            return self._load_from_cache(cache_file)
+        
+        # Need to query API
+        self.logger.logger.debug(f"Querying Shodan API for: {normalized_ip}")
+        url = f"{self.base_url}/shodan/host/{normalized_ip}"
+        params = {'key': self.api_key}
+        
+        try:
+            response = self.make_request(url, method="GET", params=params, target_indicator=normalized_ip)
+            
+            if not response:
+                self.logger.logger.warning(f"Shodan API unreachable for {normalized_ip} - network failure")
+                if cache_status == "stale":
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to network failure")
+                    return self._load_from_cache(cache_file)
+                else:
+                    # FIXED: Treat network failures as "no information" rather than retryable errors
+                    self.logger.logger.info(f"No Shodan data available for {normalized_ip} due to network failure")
+                    result = ProviderResult()  # Empty result
+                    network_failure_data = {'shodan_status': 'network_unreachable', 'error': 'API unreachable'}
+                    self._save_to_cache(cache_file, result, network_failure_data)
+                    return result
+            
+            # FIXED: Handle different status codes more precisely
+            if response.status_code == 200:
+                self.logger.logger.debug(f"Shodan returned data for {normalized_ip}")
+                try:
+                    data = response.json()
+                    result = self._process_shodan_data(normalized_ip, data)
+                    self._save_to_cache(cache_file, result, data)
+                    return result
+                except json.JSONDecodeError as e:
+                    self.logger.logger.error(f"Invalid JSON response from Shodan for {normalized_ip}: {e}")
+                    if cache_status == "stale":
+                        return self._load_from_cache(cache_file)
+                    else:
+                        raise requests.exceptions.RequestException("Invalid JSON response from Shodan - should retry")
+            
+            elif response.status_code == 404:
+                # FIXED: 404 = "no information available" - successful but empty result, don't retry
+                self.logger.logger.debug(f"Shodan has no information for {normalized_ip} (404)")
+                result = ProviderResult()  # Empty but successful result
+                # Cache the empty result to avoid repeated queries
+                empty_data = {'shodan_status': 'no_information', 'status_code': 404}
+                self._save_to_cache(cache_file, result, empty_data)
+                return result
+            
+            elif response.status_code in [401, 403]:
+                # Authentication/authorization errors - permanent failures, don't retry
+                self.logger.logger.error(f"Shodan API authentication failed for {normalized_ip} (HTTP {response.status_code})")
+                return ProviderResult()  # Empty result, don't retry
+            
+            elif response.status_code == 429:
+                # Rate limiting - should be handled by rate limiter, but if we get here, retry
+                self.logger.logger.warning(f"Shodan API rate limited for {normalized_ip} (HTTP {response.status_code})")
+                if cache_status == "stale":
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to rate limiting")
+                    return self._load_from_cache(cache_file)
+                else:
+                    raise requests.exceptions.RequestException(f"Shodan API rate limited (HTTP {response.status_code}) - should retry")
+            
+            elif response.status_code in [500, 502, 503, 504]:
+                # Server errors - temporary failures that should be retried
+                self.logger.logger.warning(f"Shodan API server error for {normalized_ip} (HTTP {response.status_code})")
+                if cache_status == "stale":
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to server error")
+                    return self._load_from_cache(cache_file)
+                else:
+                    raise requests.exceptions.RequestException(f"Shodan API server error (HTTP {response.status_code}) - should retry")
+            
+            else:
+                # FIXED: Other HTTP status codes - treat as no information available, don't retry
+                self.logger.logger.info(f"Shodan returned status {response.status_code} for {normalized_ip} - treating as no information")
+                result = ProviderResult()  # Empty result
+                no_info_data = {'shodan_status': 'no_information', 'status_code': response.status_code}
+                self._save_to_cache(cache_file, result, no_info_data)
+                return result
+                
+        except requests.exceptions.Timeout:
+            # Timeout errors - should be retried
+            self.logger.logger.warning(f"Shodan API timeout for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to timeout")
+                return self._load_from_cache(cache_file)
+            else:
+                raise  # Re-raise timeout for retry
+        
+        except requests.exceptions.ConnectionError:
+            # Connection errors - should be retried
+            self.logger.logger.warning(f"Shodan API connection error for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to connection error")
+                return self._load_from_cache(cache_file)
+            else:
+                raise  # Re-raise connection error for retry
+        
+        except json.JSONDecodeError:
+            # JSON parsing error - treat as temporary failure
+            self.logger.logger.error(f"Invalid JSON response from Shodan for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to JSON parsing error")
+                return self._load_from_cache(cache_file)
+            else:
+                raise requests.exceptions.RequestException("Invalid JSON response from Shodan - should retry")
+        
+        # FIXED: Remove the generic RequestException handler that was causing 404s to retry
+        # Now only specific exceptions that should be retried are re-raised
+        
+        except Exception as e:
+            # FIXED: Unexpected exceptions - log but treat as no information available, don't retry
+            self.logger.logger.warning(f"Unexpected exception in Shodan query for {normalized_ip}: {e}")
+            result = ProviderResult()  # Empty result
+            error_data = {'shodan_status': 'error', 'error': str(e)}
+            self._save_to_cache(cache_file, result, error_data)
+            return result
+
+    def _load_from_cache(self, cache_file_path: Path) -> ProviderResult:
+        """Load processed Shodan data from a cache file."""
+        try:
+            with open(cache_file_path, 'r') as f:
+                cache_content = json.load(f)
+            
+            result = ProviderResult()
+            
+            # Reconstruct relationships
+            for rel_data in cache_content.get("relationships", []):
+                result.add_relationship(
+                    source_node=rel_data["source_node"],
+                    target_node=rel_data["target_node"],
+                    relationship_type=rel_data["relationship_type"],
+                    provider=rel_data["provider"],
+                    raw_data=rel_data.get("raw_data", {})
+                )
+            
+            # Reconstruct attributes
+            for attr_data in cache_content.get("attributes", []):
+                result.add_attribute(
+                    target_node=attr_data["target_node"],
+                    name=attr_data["name"],
+                    value=attr_data["value"],
+                    attr_type=attr_data["type"],
+                    provider=attr_data["provider"],
+                    metadata=attr_data.get("metadata", {})
+                )
+            
+            return result
+            
+        except (json.JSONDecodeError, FileNotFoundError, KeyError):
+            return ProviderResult()
+
+    def _save_to_cache(self, cache_file_path: Path, result: ProviderResult, raw_data: Dict[str, Any]) -> None:
+        """Save processed Shodan data to a cache file."""
+        try:
+            cache_data = {
+                "last_upstream_query": datetime.now(timezone.utc).isoformat(),
+                "raw_data": raw_data,  # Preserve original for forensic purposes
+                "relationships": [
+                    {
+                        "source_node": rel.source_node,
+                        "target_node": rel.target_node,
+                        "relationship_type": rel.relationship_type,
+                        "provider": rel.provider,
+                        "raw_data": rel.raw_data
+                    } for rel in result.relationships
+                ],
+                "attributes": [
+                    {
+                        "target_node": attr.target_node,
+                        "name": attr.name,
+                        "value": attr.value,
+                        "type": attr.type,
+                        "provider": attr.provider,
+                        "metadata": attr.metadata
+                    } for attr in result.attributes
+                ]
+            }
+            with open(cache_file_path, 'w') as f:
+                json.dump(cache_data, f, separators=(',', ':'), default=str)
+        except Exception as e:
+            self.logger.logger.warning(f"Failed to save Shodan cache for {cache_file_path.name}: {e}")
+
+    def _process_shodan_data(self, ip: str, data: Dict[str, Any]) -> ProviderResult:
+        """
+        VERIFIED: Process Shodan data creating ISP nodes with ASN attributes and proper relationships.
+        Enhanced to include IP version information for IPv6 addresses.
+        UPDATED: Enhanced with last_seen timestamp for time-based edge coloring.
+        """
+        result = ProviderResult()
+        
+        # Determine IP version for metadata
+        ip_version = get_ip_version(ip)
+
+        # Extract last_seen timestamp for time-based edge coloring
+        last_seen = data.get('last_seen')
+        
+        # VERIFIED: Extract ISP information and create proper ISP node with ASN
+        isp_name = data.get('org')
+        asn_value = data.get('asn')
+
+        if isp_name and asn_value:
+            # Enhanced raw_data with last_seen timestamp
+            raw_data = {
+                'asn': asn_value, 
+                'shodan_org': isp_name, 
+                'ip_version': ip_version
+            }
+            
+            # Add last_seen timestamp if available
+            if last_seen:
+                raw_data['last_seen'] = last_seen
+                raw_data['relevance_timestamp'] = last_seen  # Standardized field for time-based coloring
+            
+            # Create relationship from IP to ISP
+            result.add_relationship(
+                source_node=ip,
+                target_node=isp_name,
+                relationship_type='shodan_isp',
+                provider=self.name,
+                raw_data=raw_data
+            )
+            
+            # Add ASN as attribute to the ISP node
+            result.add_attribute(
+                target_node=isp_name,
+                name='asn',
+                value=asn_value,
+                attr_type='isp_info',
+                provider=self.name,
+                metadata={'description': 'Autonomous System Number from Shodan', 'ip_version': ip_version}
+            )
+            
+            # Also add organization name as attribute to ISP node for completeness
+            result.add_attribute(
+                target_node=isp_name,
+                name='organization_name',
+                value=isp_name,
+                attr_type='isp_info',
+                provider=self.name,
+                metadata={'description': 'Organization name from Shodan', 'ip_version': ip_version}
+            )
+
+        # Process hostnames (reverse DNS)
+        for key, value in data.items():
+            if key == 'hostnames':
+                for hostname in value:
+                    if _is_valid_domain(hostname):
+                        # Use appropriate relationship type based on IP version
+                        if ip_version == 6:
+                            relationship_type = 'shodan_aaaa_record'
+                        else:
+                            relationship_type = 'shodan_a_record'
+                        
+                        # Enhanced raw_data with last_seen timestamp
+                        hostname_raw_data = {**data, 'ip_version': ip_version}
+                        if last_seen:
+                            hostname_raw_data['last_seen'] = last_seen
+                            hostname_raw_data['relevance_timestamp'] = last_seen
+                        
+                        result.add_relationship(
+                            source_node=ip,
+                            target_node=hostname,
+                            relationship_type=relationship_type,
+                            provider=self.name,
+                            raw_data=hostname_raw_data
+                        )
+                        self.log_relationship_discovery(
+                            source_node=ip,
+                            target_node=hostname,
+                            relationship_type=relationship_type,
+                            raw_data=hostname_raw_data,
+                            discovery_method=f"shodan_host_lookup_ipv{ip_version}"
+                        )
+            elif key == 'ports':
+                # Add open ports as attributes to the IP
+                for port in value:
+                    result.add_attribute(
+                        target_node=ip,
+                        name='shodan_open_port',
+                        value=port,
+                        attr_type='shodan_network_info',
+                        provider=self.name,
+                        metadata={'ip_version': ip_version}
+                    )
+            elif isinstance(value, (str, int, float, bool)) and value is not None:
+                # Add other Shodan fields as IP attributes (keep raw field names)
+                result.add_attribute(
+                    target_node=ip,
+                    name=key,  # Raw field name from Shodan API
+                    value=value,
+                    attr_type='shodan_info',
+                    provider=self.name,
+                    metadata={'ip_version': ip_version}
+                )
+
+        return result

requirements.txt

@@ -1,4 +1,11 @@
-requests>=2.31.0
-flask>=2.3.0
-dnspython>=2.4.0
-click>=8.1.0
+Flask
+networkx
+requests
+python-dateutil
+Werkzeug
+urllib3
+dnspython
+gunicorn
+redis
+python-dotenv
+psycopg2-binary

src/__init__.py

@@ -1,20 +0,0 @@
-# File: src/__init__.py
-"""DNS Reconnaissance Tool Package."""
-
-__version__ = "1.0.0"
-__author__ = "DNS Recon Tool"
-__email__ = ""
-__description__ = "A comprehensive DNS reconnaissance tool for investigators"
-
-from .main import main
-from .config import Config
-from .reconnaissance import ReconnaissanceEngine
-from .data_structures import ReconData
-
-__all__ = [
-    'main',
-    'Config', 
-    'ReconnaissanceEngine',
-    'ReconData'
-]
-

src/certificate_checker.py

@@ -1,324 +0,0 @@
-# File: src/certificate_checker.py
-"""Certificate transparency log checker using crt.sh."""
-
-import requests
-import json
-import time
-import logging
-import socket
-from datetime import datetime
-from typing import List, Optional, Set
-from .data_structures import Certificate
-from .config import Config
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-class CertificateChecker:
-    """Check certificates using crt.sh."""
-    
-    CRT_SH_URL = "https://crt.sh/"
-    
-    def __init__(self, config: Config):
-        self.config = config
-        self.last_request = 0
-        self.query_count = 0
-        self.connection_failures = 0
-        self.max_connection_failures = 3  # Stop trying after 3 consecutive failures
-        
-        logger.info("🔐 Certificate checker initialized")
-        
-        # Test connectivity to crt.sh on initialization
-        self._test_connectivity()
-    
-    def _test_connectivity(self):
-        """Test if we can reach crt.sh."""
-        try:
-            logger.info("🔗 Testing connectivity to crt.sh...")
-            
-            # First test DNS resolution
-            try:
-                socket.gethostbyname('crt.sh')
-                logger.debug("✅ DNS resolution for crt.sh successful")
-            except socket.gaierror as e:
-                logger.warning(f"⚠️ DNS resolution failed for crt.sh: {e}")
-                return False
-            
-            # Test HTTP connection with a simple request
-            response = requests.get(
-                self.CRT_SH_URL,
-                params={'q': 'example.com', 'output': 'json'},
-                timeout=10,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            
-            if response.status_code in [200, 404]:  # 404 is also acceptable (no results)
-                logger.info("✅ crt.sh connectivity test successful")
-                return True
-            else:
-                logger.warning(f"⚠️ crt.sh returned status {response.status_code}")
-                return False
-                
-        except requests.exceptions.ConnectionError as e:
-            logger.warning(f"⚠️ Cannot reach crt.sh: {e}")
-            return False
-        except requests.exceptions.Timeout:
-            logger.warning("⚠️ crt.sh connectivity test timed out")
-            return False
-        except Exception as e:
-            logger.warning(f"⚠️ Unexpected error testing crt.sh connectivity: {e}")
-            return False
-    
-    def _rate_limit(self):
-        """Apply rate limiting for crt.sh."""
-        now = time.time()
-        time_since_last = now - self.last_request
-        min_interval = 1.0 / self.config.CRT_SH_RATE_LIMIT
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            logger.debug(f"⏸️ crt.sh rate limiting: sleeping for {sleep_time:.2f}s")
-            time.sleep(sleep_time)
-        
-        self.last_request = time.time()
-        self.query_count += 1
-    
-    def get_certificates(self, domain: str) -> List[Certificate]:
-        """Get certificates for a domain from crt.sh."""
-        logger.debug(f"🔍 Getting certificates for domain: {domain}")
-        
-        # Skip if we've had too many connection failures
-        if self.connection_failures >= self.max_connection_failures:
-            logger.warning(f"⚠️ Skipping certificate lookup for {domain} due to repeated connection failures")
-            return []
-        
-        certificates = []
-        
-        # Query for the domain
-        domain_certs = self._query_crt_sh(domain)
-        certificates.extend(domain_certs)
-        
-        # Also query for wildcard certificates (if the main query succeeded)
-        if domain_certs or self.connection_failures < self.max_connection_failures:
-            wildcard_certs = self._query_crt_sh(f"%.{domain}")
-            certificates.extend(wildcard_certs)
-        
-        # Remove duplicates based on certificate ID
-        unique_certs = {cert.id: cert for cert in certificates}
-        final_certs = list(unique_certs.values())
-        
-        if final_certs:
-            logger.info(f"📜 Found {len(final_certs)} unique certificates for {domain}")
-        else:
-            logger.debug(f"❌ No certificates found for {domain}")
-        
-        return final_certs
-    
-    def _query_crt_sh(self, query: str) -> List[Certificate]:
-        """Query crt.sh API with retry logic and better error handling."""
-        certificates = []
-        self._rate_limit()
-        
-        logger.debug(f"📡 Querying crt.sh for: {query}")
-        
-        max_retries = 2  # Reduced retries for faster failure
-        backoff_delays = [1, 3]  # Shorter delays
-        
-        for attempt in range(max_retries):
-            try:
-                params = {
-                    'q': query,
-                    'output': 'json'
-                }
-                
-                response = requests.get(
-                    self.CRT_SH_URL,
-                    params=params,
-                    timeout=self.config.HTTP_TIMEOUT,
-                    headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-                )
-                
-                logger.debug(f"📡 crt.sh API response for {query}: {response.status_code}")
-                
-                if response.status_code == 200:
-                    try:
-                        data = response.json()
-                        logger.debug(f"📊 crt.sh returned {len(data)} certificate entries for {query}")
-                        
-                        for cert_data in data:
-                            try:
-                                # Parse dates with better error handling
-                                not_before = self._parse_date(cert_data.get('not_before'))
-                                not_after = self._parse_date(cert_data.get('not_after'))
-                                
-                                if not_before and not_after:
-                                    certificate = Certificate(
-                                        id=cert_data.get('id'),
-                                        issuer=cert_data.get('issuer_name', ''),
-                                        subject=cert_data.get('name_value', ''),
-                                        not_before=not_before,
-                                        not_after=not_after,
-                                        is_wildcard='*.' in cert_data.get('name_value', '')
-                                    )
-                                    certificates.append(certificate)
-                                    logger.debug(f"✅ Parsed certificate ID {certificate.id} for {query}")
-                                else:
-                                    logger.debug(f"⚠️ Skipped certificate with invalid dates: {cert_data.get('id')}")
-                                    
-                            except (ValueError, TypeError, KeyError) as e:
-                                logger.debug(f"⚠️ Error parsing certificate data: {e}")
-                                continue  # Skip malformed certificate data
-                        
-                        # Success! Reset connection failure counter
-                        self.connection_failures = 0
-                        logger.info(f"✅ Successfully processed {len(certificates)} certificates from crt.sh for {query}")
-                        return certificates
-                    
-                    except json.JSONDecodeError as e:
-                        logger.warning(f"❌ Invalid JSON response from crt.sh for {query}: {e}")
-                        if attempt < max_retries - 1:
-                            time.sleep(backoff_delays[attempt])
-                            continue
-                        return certificates
-                
-                elif response.status_code == 404:
-                    # 404 is normal - no certificates found
-                    logger.debug(f"ℹ️ No certificates found for {query} (404)")
-                    self.connection_failures = 0  # Reset counter for successful connection
-                    return certificates
-                
-                elif response.status_code == 429:
-                    logger.warning(f"⚠️ crt.sh rate limit exceeded for {query}")
-                    if attempt < max_retries - 1:
-                        time.sleep(5)  # Wait longer for rate limits
-                        continue
-                    return certificates
-                    
-                else:
-                    logger.warning(f"⚠️ crt.sh HTTP error for {query}: {response.status_code}")
-                    if attempt < max_retries - 1:
-                        time.sleep(backoff_delays[attempt])
-                        continue
-                    return certificates
-                            
-            except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:
-                error_type = "Connection Error" if isinstance(e, requests.exceptions.ConnectionError) else "Timeout"
-                logger.warning(f"🌐 crt.sh {error_type} for {query} (attempt {attempt+1}/{max_retries}): {e}")
-                
-                # Track connection failures
-                if isinstance(e, requests.exceptions.ConnectionError):
-                    self.connection_failures += 1
-                
-                if attempt < max_retries - 1:
-                    time.sleep(backoff_delays[attempt])
-                    continue
-                    
-            except requests.exceptions.RequestException as e:
-                logger.warning(f"🌐 crt.sh network error for {query} (attempt {attempt+1}/{max_retries}): {e}")
-                if attempt < max_retries - 1:
-                    time.sleep(backoff_delays[attempt])
-                    continue
-            except Exception as e:
-                logger.error(f"❌ Unexpected error querying crt.sh for {query}: {e}")
-                if attempt < max_retries - 1:
-                    time.sleep(backoff_delays[attempt])
-                    continue
-        
-        # If we get here, all retries failed
-        logger.warning(f"❌ All {max_retries} attempts failed for crt.sh query: {query}")
-        return certificates
-    
-    def _parse_date(self, date_str: str) -> Optional[datetime]:
-        """Parse date string with multiple format support."""
-        if not date_str:
-            return None
-        
-        # Common date formats from crt.sh
-        date_formats = [
-            '%Y-%m-%dT%H:%M:%S',         # ISO format without timezone
-            '%Y-%m-%dT%H:%M:%SZ',        # ISO format with Z
-            '%Y-%m-%d %H:%M:%S',         # Space separated
-            '%Y-%m-%dT%H:%M:%S.%f',      # With microseconds
-            '%Y-%m-%dT%H:%M:%S.%fZ',     # With microseconds and Z
-        ]
-        
-        for fmt in date_formats:
-            try:
-                return datetime.strptime(date_str, fmt)
-            except ValueError:
-                continue
-        
-        # Try with timezone info
-        try:
-            return datetime.fromisoformat(date_str.replace('Z', '+00:00'))
-        except ValueError:
-            pass
-        
-        logger.debug(f"⚠️ Could not parse date: {date_str}")
-        return None
-    
-    def extract_subdomains_from_certificates(self, certificates: List[Certificate]) -> Set[str]:
-        """Extract subdomains from certificate subjects."""
-        subdomains = set()
-        
-        logger.debug(f"🌿 Extracting subdomains from {len(certificates)} certificates")
-        
-        for cert in certificates:
-            # Parse subject field for domain names
-            # Certificate subjects can be multi-line with multiple domains
-            subject_lines = cert.subject.split('\n')
-            
-            for line in subject_lines:
-                line = line.strip()
-                
-                # Skip wildcard domains for recursion (they don't resolve directly)
-                if line.startswith('*.'):
-                    logger.debug(f"🌿 Skipping wildcard domain: {line}")
-                    continue
-                
-                if self._is_valid_domain(line):
-                    subdomains.add(line.lower())
-                    logger.debug(f"🌿 Found subdomain from certificate: {line}")
-        
-        if subdomains:
-            logger.info(f"🌿 Extracted {len(subdomains)} subdomains from certificates")
-        else:
-            logger.debug("❌ No subdomains extracted from certificates")
-        
-        return subdomains
-    
-    def _is_valid_domain(self, domain: str) -> bool:
-        """Basic domain validation."""
-        if not domain or '.' not in domain:
-            return False
-        
-        # Remove common prefixes
-        domain = domain.lower().strip()
-        if domain.startswith('www.'):
-            domain = domain[4:]
-        
-        # Basic validation
-        if len(domain) < 3 or len(domain) > 255:
-            return False
-        
-        # Must not be an IP address
-        try:
-            import socket
-            socket.inet_aton(domain)
-            return False  # It's an IPv4 address
-        except socket.error:
-            pass
-        
-        # Check for reasonable domain structure
-        parts = domain.split('.')
-        if len(parts) < 2:
-            return False
-        
-        # Each part should be reasonable
-        for part in parts:
-            if len(part) < 1 or len(part) > 63:
-                return False
-            if not part.replace('-', '').replace('_', '').isalnum():
-                return False
-        
-        return True

src/config.py

@@ -1,80 +0,0 @@
-# File: src/config.py
-"""Configuration settings for the reconnaissance tool."""
-
-import os
-import logging
-from dataclasses import dataclass
-from typing import List, Optional
-
-@dataclass
-class Config:
-    """Configuration class for the reconnaissance tool."""
-    
-    # DNS servers to query
-    DNS_SERVERS: List[str] = None
-    
-    # API keys
-    shodan_key: Optional[str] = None
-    virustotal_key: Optional[str] = None
-    
-    # Rate limiting (requests per second)
-    # DNS servers are generally quite robust, increased from 10 to 50/s
-    DNS_RATE_LIMIT: float = 50.0
-    CRT_SH_RATE_LIMIT: float = 2.0
-    SHODAN_RATE_LIMIT: float = 0.5  # Shodan is more restrictive
-    VIRUSTOTAL_RATE_LIMIT: float = 0.25  # VirusTotal is very restrictive
-    
-    # Recursive depth
-    max_depth: int = 2
-    
-    # Timeouts
-    DNS_TIMEOUT: int = 5
-    HTTP_TIMEOUT: int = 20
-    
-    # Logging level
-    log_level: str = "INFO"
-    
-    def __post_init__(self):
-        if self.DNS_SERVERS is None:
-            # Use multiple reliable DNS servers
-            self.DNS_SERVERS = [
-                '1.1.1.1',      # Cloudflare
-                '8.8.8.8',      # Google
-                '9.9.9.9'      # Quad9
-            ]
-    
-    @classmethod
-    def from_args(cls, shodan_key: Optional[str] = None, 
-                  virustotal_key: Optional[str] = None,
-                  max_depth: int = 2,
-                  log_level: str = "INFO") -> 'Config':
-        """Create config from command line arguments."""
-        return cls(
-            shodan_key=shodan_key,
-            virustotal_key=virustotal_key,
-            max_depth=max_depth,
-            log_level=log_level.upper()
-        )
-    
-    def setup_logging(self, cli_mode: bool = True):
-        """Set up logging configuration."""
-        log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
-        
-        if cli_mode:
-            # For CLI, use a more readable format
-            log_format = '%(asctime)s [%(levelname)s] %(message)s'
-        
-        logging.basicConfig(
-            level=getattr(logging, self.log_level, logging.INFO),
-            format=log_format,
-            datefmt='%H:%M:%S'
-        )
-        
-        # Set specific loggers
-        logging.getLogger('urllib3').setLevel(logging.WARNING)  # Reduce HTTP noise
-        logging.getLogger('requests').setLevel(logging.WARNING)  # Reduce HTTP noise
-        
-        if self.log_level == "DEBUG":
-            logging.getLogger(__name__.split('.')[0]).setLevel(logging.DEBUG)
-        
-        return logging.getLogger(__name__)

src/data_structures.py

@@ -1,204 +0,0 @@
-# File: src/data_structures.py
-"""Data structures for storing reconnaissance results."""
-
-from dataclasses import dataclass, field
-from typing import Dict, List, Set, Optional, Any
-from datetime import datetime
-import json
-import logging
-
-# Set up logging for this module
-logger = logging.getLogger(__name__)
-
-@dataclass
-class DNSRecord:
-    """DNS record information."""
-    record_type: str
-    value: str
-    ttl: Optional[int] = None
-    
-    def to_dict(self) -> dict:
-        return {
-            'record_type': self.record_type,
-            'value': self.value,
-            'ttl': self.ttl
-        }
-
-@dataclass
-class Certificate:
-    """Certificate information from crt.sh."""
-    id: int
-    issuer: str
-    subject: str
-    not_before: datetime
-    not_after: datetime
-    is_wildcard: bool = False
-    
-    def to_dict(self) -> dict:
-        return {
-            'id': self.id,
-            'issuer': self.issuer,
-            'subject': self.subject,
-            'not_before': self.not_before.isoformat() if self.not_before else None,
-            'not_after': self.not_after.isoformat() if self.not_after else None,
-            'is_wildcard': self.is_wildcard
-        }
-
-@dataclass
-class ShodanResult:
-    """Shodan scan result."""
-    ip: str
-    ports: List[int]
-    services: Dict[str, Any]
-    organization: Optional[str] = None
-    country: Optional[str] = None
-    
-    def to_dict(self) -> dict:
-        return {
-            'ip': self.ip,
-            'ports': self.ports,
-            'services': self.services,
-            'organization': self.organization,
-            'country': self.country
-        }
-
-@dataclass
-class VirusTotalResult:
-    """VirusTotal scan result."""
-    resource: str  # IP or domain
-    positives: int
-    total: int
-    scan_date: datetime
-    permalink: str
-    
-    def to_dict(self) -> dict:
-        return {
-            'resource': self.resource,
-            'positives': self.positives,
-            'total': self.total,
-            'scan_date': self.scan_date.isoformat() if self.scan_date else None,
-            'permalink': self.permalink
-        }
-
-@dataclass
-class ReconData:
-    """Main data structure for reconnaissance results."""
-    
-    # Core data
-    hostnames: Set[str] = field(default_factory=set)
-    ip_addresses: Set[str] = field(default_factory=set)
-    
-    # DNS information
-    dns_records: Dict[str, List[DNSRecord]] = field(default_factory=dict)
-    reverse_dns: Dict[str, str] = field(default_factory=dict)
-    
-    # Certificate information
-    certificates: Dict[str, List[Certificate]] = field(default_factory=dict)
-    
-    # External service results
-    shodan_results: Dict[str, ShodanResult] = field(default_factory=dict)
-    virustotal_results: Dict[str, VirusTotalResult] = field(default_factory=dict)
-    
-    # Metadata
-    start_time: datetime = field(default_factory=datetime.now)
-    end_time: Optional[datetime] = None
-    depth_map: Dict[str, int] = field(default_factory=dict)  # Track recursion depth
-    
-    def add_hostname(self, hostname: str, depth: int = 0) -> None:
-        """Add a hostname to the dataset."""
-        hostname = hostname.lower()
-        self.hostnames.add(hostname)
-        self.depth_map[hostname] = depth
-        logger.info(f"Added hostname: {hostname} (depth: {depth})")
-    
-    def add_ip_address(self, ip: str) -> None:
-        """Add an IP address to the dataset."""
-        self.ip_addresses.add(ip)
-        logger.info(f"Added IP address: {ip}")
-    
-    def add_dns_record(self, hostname: str, record: DNSRecord) -> None:
-        """Add a DNS record for a hostname."""
-        hostname = hostname.lower()
-        if hostname not in self.dns_records:
-            self.dns_records[hostname] = []
-        self.dns_records[hostname].append(record)
-        logger.debug(f"Added DNS record for {hostname}: {record.record_type} -> {record.value}")
-    
-    def add_shodan_result(self, ip: str, result: ShodanResult) -> None:
-        """Add Shodan result."""
-        self.shodan_results[ip] = result
-        logger.info(f"Added Shodan result for {ip}: {len(result.ports)} ports, org: {result.organization}")
-    
-    def add_virustotal_result(self, resource: str, result: VirusTotalResult) -> None:
-        """Add VirusTotal result."""
-        self.virustotal_results[resource] = result
-        logger.info(f"Added VirusTotal result for {resource}: {result.positives}/{result.total} detections")
-    
-    def get_new_subdomains(self, max_depth: int) -> Set[str]:
-        """Get subdomains that haven't been processed yet and are within depth limit."""
-        new_domains = set()
-        for hostname in self.hostnames:
-            if (hostname not in self.dns_records and 
-                self.depth_map.get(hostname, 0) < max_depth):
-                new_domains.add(hostname)
-        return new_domains
-    
-    def get_stats(self) -> Dict[str, int]:
-        """Get current statistics."""
-        return {
-            'hostnames': len(self.hostnames),
-            'ip_addresses': len(self.ip_addresses),
-            'dns_records': sum(len(records) for records in self.dns_records.values()),
-            'certificates': sum(len(certs) for certs in self.certificates.values()),
-            'shodan_results': len(self.shodan_results),
-            'virustotal_results': len(self.virustotal_results)
-        }
-    
-    def to_dict(self) -> dict:
-        """Export data as a serializable dictionary."""
-        logger.debug(f"Serializing ReconData with stats: {self.get_stats()}")
-        
-        result = {
-            'hostnames': sorted(list(self.hostnames)),
-            'ip_addresses': sorted(list(self.ip_addresses)),
-            'dns_records': {
-                host: [record.to_dict() for record in records] 
-                for host, records in self.dns_records.items()
-            },
-            'reverse_dns': dict(self.reverse_dns),
-            'certificates': {
-                host: [cert.to_dict() for cert in certs]
-                for host, certs in self.certificates.items()
-            },
-            'shodan_results': {
-                ip: result.to_dict() for ip, result in self.shodan_results.items()
-            },
-            'virustotal_results': {
-                resource: result.to_dict() for resource, result in self.virustotal_results.items()
-            },
-            'depth_map': dict(self.depth_map),
-            'metadata': {
-                'start_time': self.start_time.isoformat() if self.start_time else None,
-                'end_time': self.end_time.isoformat() if self.end_time else None,
-                'stats': self.get_stats()
-            }
-        }
-        
-        logger.info(f"Serialized data contains: {len(result['hostnames'])} hostnames, "
-                   f"{len(result['ip_addresses'])} IPs, {len(result['shodan_results'])} Shodan results, "
-                   f"{len(result['virustotal_results'])} VirusTotal results")
-        
-        return result
-    
-    def to_json(self) -> str:
-        """Export data as JSON."""
-        try:
-            return json.dumps(self.to_dict(), indent=2, ensure_ascii=False)
-        except Exception as e:
-            logger.error(f"Failed to serialize to JSON: {e}")
-            # Return minimal JSON in case of error
-            return json.dumps({
-                'error': str(e),
-                'stats': self.get_stats(),
-                'timestamp': datetime.now().isoformat()
-            }, indent=2)

src/dns_resolver.py

@@ -1,312 +0,0 @@
-# File: src/dns_resolver.py
-"""DNS resolution functionality with enhanced TLD testing."""
-
-import dns.resolver
-import dns.reversename
-import dns.query
-import dns.zone
-from typing import List, Dict, Optional, Set
-import socket
-import time
-import logging
-from .data_structures import DNSRecord, ReconData
-from .config import Config
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-class DNSResolver:
-    """DNS resolution and record lookup with optimized TLD testing."""
-    
-    # All DNS record types to query
-    RECORD_TYPES = [
-        'A', 'AAAA', 'MX', 'NS', 'TXT', 'CNAME', 'SOA', 'PTR', 
-        'SRV', 'CAA', 'DNSKEY', 'DS', 'RRSIG', 'NSEC', 'NSEC3'
-    ]
-    
-    def __init__(self, config: Config):
-        self.config = config
-        self.last_request = 0
-        self.query_count = 0
-        
-        logger.info(f"🌐 DNS resolver initialized with {len(config.DNS_SERVERS)} servers: {config.DNS_SERVERS}")
-        logger.info(f"⚡ DNS rate limit: {config.DNS_RATE_LIMIT}/s, timeout: {config.DNS_TIMEOUT}s")
-    
-    def _rate_limit(self):
-        """Apply rate limiting - more graceful for DNS servers."""
-        now = time.time()
-        time_since_last = now - self.last_request
-        min_interval = 1.0 / self.config.DNS_RATE_LIMIT
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            # Only log if sleep is significant to reduce spam
-            if sleep_time > 0.1:
-                logger.debug(f"⏸️ DNS rate limiting: sleeping for {sleep_time:.2f}s")
-            time.sleep(sleep_time)
-        
-        self.last_request = time.time()
-        self.query_count += 1
-    
-    def resolve_hostname_fast(self, hostname: str) -> List[str]:
-        """Fast hostname resolution optimized for TLD testing."""
-        ips = []
-        
-        logger.debug(f"🚀 Fast resolving hostname: {hostname}")
-        
-        # Use only the first DNS server and shorter timeout for TLD testing
-        resolver = dns.resolver.Resolver()
-        resolver.nameservers = [self.config.DNS_SERVERS[0]]  # Use primary DNS only
-        resolver.timeout = 2  # Shorter timeout for TLD testing
-        resolver.lifetime = 2  # Total query time limit
-        
-        try:
-            # Try A records only for speed (most common)
-            answers = resolver.resolve(hostname, 'A')
-            for answer in answers:
-                ips.append(str(answer))
-                logger.debug(f"⚡ Fast A record for {hostname}: {answer}")
-        except dns.resolver.NXDOMAIN:
-            logger.debug(f"❌ NXDOMAIN for {hostname}")
-        except dns.resolver.NoAnswer:
-            logger.debug(f"⚠️ No A record for {hostname}")
-        except dns.resolver.Timeout:
-            logger.debug(f"⏱️ Timeout for {hostname}")
-        except Exception as e:
-            logger.debug(f"⚠️ Error fast resolving {hostname}: {e}")
-        
-        if ips:
-            logger.debug(f"⚡ Fast resolved {hostname} to {len(ips)} IPs: {ips}")
-        
-        return ips
-    
-    def resolve_hostname(self, hostname: str) -> List[str]:
-        """Resolve hostname to IP addresses (full resolution with retries)."""
-        ips = []
-        
-        logger.debug(f"🔍 Resolving hostname: {hostname}")
-        
-        for dns_server in self.config.DNS_SERVERS:
-            self._rate_limit()
-            resolver = dns.resolver.Resolver()
-            resolver.nameservers = [dns_server]
-            resolver.timeout = self.config.DNS_TIMEOUT
-            
-            try:
-                # Try A records
-                answers = resolver.resolve(hostname, 'A')
-                for answer in answers:
-                    ips.append(str(answer))
-                    logger.debug(f"✅ A record for {hostname}: {answer}")
-            except dns.resolver.NXDOMAIN:
-                logger.debug(f"❌ NXDOMAIN for {hostname} A record on {dns_server}")
-            except dns.resolver.NoAnswer:
-                logger.debug(f"⚠️ No A record for {hostname} on {dns_server}")
-            except Exception as e:
-                logger.debug(f"⚠️ Error resolving A record for {hostname} on {dns_server}: {e}")
-            
-            try:
-                # Try AAAA records (IPv6)
-                answers = resolver.resolve(hostname, 'AAAA')
-                for answer in answers:
-                    ips.append(str(answer))
-                    logger.debug(f"✅ AAAA record for {hostname}: {answer}")
-            except dns.resolver.NXDOMAIN:
-                logger.debug(f"❌ NXDOMAIN for {hostname} AAAA record on {dns_server}")
-            except dns.resolver.NoAnswer:
-                logger.debug(f"⚠️ No AAAA record for {hostname} on {dns_server}")
-            except Exception as e:
-                logger.debug(f"⚠️ Error resolving AAAA record for {hostname} on {dns_server}: {e}")
-        
-        unique_ips = list(set(ips))
-        if unique_ips:
-            logger.info(f"✅ Resolved {hostname} to {len(unique_ips)} unique IPs: {unique_ips}")
-        else:
-            logger.debug(f"❌ No IPs found for {hostname}")
-            
-        return unique_ips
-    
-    def get_all_dns_records(self, hostname: str) -> List[DNSRecord]:
-        """Get all DNS records for a hostname."""
-        records = []
-        successful_queries = 0
-        
-        logger.debug(f"📋 Getting all DNS records for: {hostname}")
-        
-        for record_type in self.RECORD_TYPES:
-            type_found = False
-            
-            for dns_server in self.config.DNS_SERVERS:
-                self._rate_limit()
-                resolver = dns.resolver.Resolver()
-                resolver.nameservers = [dns_server]
-                resolver.timeout = self.config.DNS_TIMEOUT
-                
-                try:
-                    answers = resolver.resolve(hostname, record_type)
-                    for answer in answers:
-                        records.append(DNSRecord(
-                            record_type=record_type,
-                            value=str(answer),
-                            ttl=answers.ttl
-                        ))
-                        if not type_found:
-                            logger.debug(f"✅ Found {record_type} record for {hostname}: {answer}")
-                            type_found = True
-                    
-                    if not type_found:
-                        successful_queries += 1
-                    break  # Found records, no need to query other DNS servers for this type
-                    
-                except dns.resolver.NXDOMAIN:
-                    logger.debug(f"❌ NXDOMAIN for {hostname} {record_type} on {dns_server}")
-                    break  # Domain doesn't exist, no point checking other servers
-                except dns.resolver.NoAnswer:
-                    logger.debug(f"⚠️ No {record_type} record for {hostname} on {dns_server}")
-                    continue  # Try next DNS server
-                except dns.resolver.Timeout:
-                    logger.debug(f"⏱️ Timeout for {hostname} {record_type} on {dns_server}")
-                    continue  # Try next DNS server
-                except Exception as e:
-                    logger.debug(f"⚠️ Error querying {record_type} for {hostname} on {dns_server}: {e}")
-                    continue  # Try next DNS server
-        
-        logger.info(f"📋 Found {len(records)} DNS records for {hostname} across {len(set(r.record_type for r in records))} record types")
-        
-        # Log query statistics every 100 queries
-        if self.query_count % 100 == 0:
-            logger.info(f"📊 DNS query statistics: {self.query_count} total queries performed")
-            
-        return records
-    
-    def reverse_dns_lookup(self, ip: str) -> Optional[str]:
-        """Perform reverse DNS lookup."""
-        logger.debug(f"🔍 Reverse DNS lookup for: {ip}")
-        
-        try:
-            self._rate_limit()
-            hostname = socket.gethostbyaddr(ip)[0]
-            logger.info(f"✅ Reverse DNS for {ip}: {hostname}")
-            return hostname
-        except socket.herror:
-            logger.debug(f"❌ No reverse DNS for {ip}")
-            return None
-        except Exception as e:
-            logger.debug(f"⚠️ Error in reverse DNS for {ip}: {e}")
-            return None
-    
-    def extract_subdomains_from_dns(self, records: List[DNSRecord]) -> Set[str]:
-        """Extract potential subdomains from DNS records."""
-        subdomains = set()
-        
-        logger.debug(f"🌿 Extracting subdomains from {len(records)} DNS records")
-        
-        for record in records:
-            value = record.value.lower()
-            
-            # Extract from different record types
-            try:
-                if record.record_type == 'MX':
-                    # MX record format: "priority hostname"
-                    parts = value.split()
-                    if len(parts) >= 2:
-                        hostname = parts[-1].rstrip('.')  # Take the last part (hostname)
-                        if self._is_valid_hostname(hostname):
-                            subdomains.add(hostname)
-                            logger.debug(f"🌿 Found subdomain from MX: {hostname}")
-                
-                elif record.record_type in ['CNAME', 'NS']:
-                    # Direct hostname records
-                    hostname = value.rstrip('.')
-                    if self._is_valid_hostname(hostname):
-                        subdomains.add(hostname)
-                        logger.debug(f"🌿 Found subdomain from {record.record_type}: {hostname}")
-                
-                elif record.record_type == 'TXT':
-                    # Search for domain-like strings in TXT records
-                    # Common patterns: include:example.com, v=spf1 include:_spf.google.com
-                    words = value.replace(',', ' ').replace(';', ' ').split()
-                    for word in words:
-                        # Look for include: patterns
-                        if word.startswith('include:'):
-                            hostname = word[8:].rstrip('.')
-                            if self._is_valid_hostname(hostname):
-                                subdomains.add(hostname)
-                                logger.debug(f"🌿 Found subdomain from TXT include: {hostname}")
-                        
-                        # Look for other domain patterns
-                        elif '.' in word and not word.startswith('http'):
-                            clean_word = word.strip('",\'()[]{}').rstrip('.')
-                            if self._is_valid_hostname(clean_word):
-                                subdomains.add(clean_word)
-                                logger.debug(f"🌿 Found subdomain from TXT: {clean_word}")
-                
-                elif record.record_type == 'SRV':
-                    # SRV record format: "priority weight port target"
-                    parts = value.split()
-                    if len(parts) >= 4:
-                        hostname = parts[-1].rstrip('.')  # Target hostname
-                        if self._is_valid_hostname(hostname):
-                            subdomains.add(hostname)
-                            logger.debug(f"🌿 Found subdomain from SRV: {hostname}")
-                        
-            except Exception as e:
-                logger.debug(f"⚠️ Error extracting subdomain from {record.record_type} record '{value}': {e}")
-                continue
-        
-        if subdomains:
-            logger.info(f"🌿 Extracted {len(subdomains)} potential subdomains")
-        else:
-            logger.debug("❌ No subdomains extracted from DNS records")
-        
-        return subdomains
-    
-    def _is_valid_hostname(self, hostname: str) -> bool:
-        """Basic hostname validation."""
-        if not hostname or len(hostname) > 255:
-            return False
-        
-        # Must contain at least one dot
-        if '.' not in hostname:
-            return False
-        
-        # Must not be an IP address
-        if self._looks_like_ip(hostname):
-            return False
-        
-        # Basic character check - allow international domains
-        # Remove overly restrictive character filtering
-        if not hostname.replace('-', '').replace('.', '').replace('_', '').isalnum():
-            # Allow some special cases for internationalized domains
-            try:
-                hostname.encode('ascii')
-            except UnicodeEncodeError:
-                return False  # Skip non-ASCII for now
-        
-        # Must have reasonable length parts
-        parts = hostname.split('.')
-        if len(parts) < 2:
-            return False
-        
-        # Each part should be reasonable length
-        for part in parts:
-            if len(part) < 1 or len(part) > 63:
-                return False
-        
-        return True
-    
-    def _looks_like_ip(self, text: str) -> bool:
-        """Check if text looks like an IP address."""
-        try:
-            socket.inet_aton(text)
-            return True
-        except socket.error:
-            pass
-        
-        try:
-            socket.inet_pton(socket.AF_INET6, text)
-            return True
-        except socket.error:
-            pass
-        
-        return False

src/main.py

@@ -1,195 +0,0 @@
-# File: src/main.py
-"""Main CLI interface for the reconnaissance tool."""
-
-import click
-import json
-import sys
-import logging
-from pathlib import Path
-from .config import Config
-from .reconnaissance import ReconnaissanceEngine
-from .report_generator import ReportGenerator
-from .web_app import create_app
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-@click.command()
-@click.argument('target', required=False)
-@click.option('--web', is_flag=True, help='Start web interface instead of CLI')
-@click.option('--shodan-key', help='Shodan API key')
-@click.option('--virustotal-key', help='VirusTotal API key')
-@click.option('--max-depth', default=2, help='Maximum recursion depth (default: 2)')
-@click.option('--output', '-o', help='Output file prefix (will create .json and .txt files)')
-@click.option('--json-only', is_flag=True, help='Only output JSON')
-@click.option('--text-only', is_flag=True, help='Only output text report')
-@click.option('--port', default=5000, help='Port for web interface (default: 5000)')
-@click.option('--verbose', '-v', is_flag=True, help='Enable verbose logging (DEBUG level)')
-@click.option('--quiet', '-q', is_flag=True, help='Quiet mode (WARNING level only)')
-def main(target, web, shodan_key, virustotal_key, max_depth, output, json_only, text_only, port, verbose, quiet):
-    """DNS Reconnaissance Tool
-    
-    Examples:
-        recon example.com                    # Scan example.com
-        recon example                        # Try example.* for all TLDs
-        recon example.com --max-depth 3      # Deeper recursion
-        recon example.com -v                 # Verbose logging
-        recon --web                          # Start web interface
-    """
-    
-    # Determine log level
-    if verbose:
-        log_level = "DEBUG"
-    elif quiet:
-        log_level = "WARNING"
-    else:
-        log_level = "INFO"
-    
-    # Create configuration and setup logging
-    config = Config.from_args(shodan_key, virustotal_key, max_depth, log_level)
-    config.setup_logging(cli_mode=True)
-    
-    if web:
-        # Start web interface
-        logger.info("🌐 Starting web interface...")
-        app = create_app(config)
-        logger.info(f"🚀 Web interface starting on http://0.0.0.0:{port}")
-        app.run(host='0.0.0.0', port=port, debug=False)  # Changed debug to False to reduce noise
-        return
-    
-    if not target:
-        click.echo("❌ Error: TARGET is required for CLI mode. Use --web for web interface.")
-        sys.exit(1)
-    
-    # Initialize reconnaissance engine
-    logger.info("🔧 Initializing reconnaissance engine...")
-    engine = ReconnaissanceEngine(config)
-    
-    # Set up progress callback for CLI
-    def progress_callback(message, percentage=None):
-        if percentage is not None:
-            click.echo(f"[{percentage:3d}%] {message}")
-        else:
-            click.echo(f"       {message}")
-    
-    engine.set_progress_callback(progress_callback)
-    
-    # Display startup information
-    click.echo("=" * 60)
-    click.echo("🔍 DNS RECONNAISSANCE TOOL")
-    click.echo("=" * 60)
-    click.echo(f"🎯 Target: {target}")
-    click.echo(f"📊 Max recursion depth: {max_depth}")
-    click.echo(f"🌐 DNS servers: {', '.join(config.DNS_SERVERS[:3])}{'...' if len(config.DNS_SERVERS) > 3 else ''}")
-    click.echo(f"⚡ DNS rate limit: {config.DNS_RATE_LIMIT}/s")
-    
-    if shodan_key:
-        click.echo("✅ Shodan integration enabled")
-        logger.info(f"🕵️  Shodan API key provided (ends with: ...{shodan_key[-4:] if len(shodan_key) > 4 else shodan_key})")
-    else:
-        click.echo("⚠️  Shodan integration disabled (no API key)")
-        
-    if virustotal_key:
-        click.echo("✅ VirusTotal integration enabled")
-        logger.info(f"🛡️  VirusTotal API key provided (ends with: ...{virustotal_key[-4:] if len(virustotal_key) > 4 else virustotal_key})")
-    else:
-        click.echo("⚠️  VirusTotal integration disabled (no API key)")
-    
-    click.echo("")
-    
-    # Run reconnaissance
-    try:
-        logger.info(f"🚀 Starting reconnaissance for target: {target}")
-        data = engine.run_reconnaissance(target)
-        
-        # Display final statistics
-        stats = data.get_stats()
-        click.echo("")
-        click.echo("=" * 60)
-        click.echo("📊 RECONNAISSANCE COMPLETE")
-        click.echo("=" * 60)
-        click.echo(f"🏠 Hostnames discovered: {stats['hostnames']}")
-        click.echo(f"🌐 IP addresses found: {stats['ip_addresses']}")
-        click.echo(f"📋 DNS records collected: {stats['dns_records']}")
-        click.echo(f"📜 Certificates found: {stats['certificates']}")
-        click.echo(f"🕵️  Shodan results: {stats['shodan_results']}")
-        click.echo(f"🛡️  VirusTotal results: {stats['virustotal_results']}")
-        
-        # Calculate and display timing
-        if data.end_time and data.start_time:
-            duration = data.end_time - data.start_time
-            click.echo(f"⏱️  Total time: {duration}")
-        
-        click.echo("")
-        
-        # Generate reports
-        logger.info("📄 Generating reports...")
-        report_gen = ReportGenerator(data)
-        
-        if output:
-            # Save to files
-            saved_files = []
-            
-            if not text_only:
-                json_file = f"{output}.json"
-                try:
-                    json_content = data.to_json()
-                    with open(json_file, 'w', encoding='utf-8') as f:
-                        f.write(json_content)
-                    saved_files.append(json_file)
-                    logger.info(f"💾 JSON report saved: {json_file}")
-                except Exception as e:
-                    logger.error(f"❌ Failed to save JSON report: {e}")
-            
-            if not json_only:
-                text_file = f"{output}.txt"
-                try:
-                    with open(text_file, 'w', encoding='utf-8') as f:
-                        f.write(report_gen.generate_text_report())
-                    saved_files.append(text_file)
-                    logger.info(f"💾 Text report saved: {text_file}")
-                except Exception as e:
-                    logger.error(f"❌ Failed to save text report: {e}")
-            
-            if saved_files:
-                click.echo(f"💾 Reports saved:")
-                for file in saved_files:
-                    click.echo(f"   📄 {file}")
-        
-        else:
-            # Output to stdout
-            if json_only:
-                try:
-                    click.echo(data.to_json())
-                except Exception as e:
-                    logger.error(f"❌ Failed to generate JSON output: {e}")
-                    click.echo(f"Error generating JSON: {e}")
-            elif text_only:
-                try:
-                    click.echo(report_gen.generate_text_report())
-                except Exception as e:
-                    logger.error(f"❌ Failed to generate text report: {e}")
-                    click.echo(f"Error generating text report: {e}")
-            else:
-                # Default: show text report
-                try:
-                    click.echo(report_gen.generate_text_report())
-                    click.echo(f"\n💡 To get JSON output, use: --json-only")
-                    click.echo(f"💡 To save reports, use: --output filename")
-                except Exception as e:
-                    logger.error(f"❌ Failed to generate report: {e}")
-                    click.echo(f"Error generating report: {e}")
-    
-    except KeyboardInterrupt:
-        logger.warning("⚠️  Reconnaissance interrupted by user")
-        click.echo("\n⚠️  Reconnaissance interrupted by user.")
-        sys.exit(1)
-    except Exception as e:
-        logger.error(f"❌ Error during reconnaissance: {e}", exc_info=True)
-        click.echo(f"❌ Error during reconnaissance: {e}")
-        if verbose:
-            raise  # Re-raise in verbose mode to show full traceback
-        sys.exit(1)
-
-if __name__ == '__main__':
-    main()

src/reconnaissance.py

@@ -1,400 +0,0 @@
-# File: src/reconnaissance.py
-"""Main reconnaissance logic with enhanced TLD expansion."""
-
-import threading
-import concurrent.futures
-import logging
-from datetime import datetime
-from typing import Set, List, Optional, Tuple
-from .data_structures import ReconData
-from .config import Config
-from .dns_resolver import DNSResolver
-from .certificate_checker import CertificateChecker
-from .shodan_client import ShodanClient
-from .virustotal_client import VirusTotalClient
-from .tld_fetcher import TLDFetcher
-
-# Set up logging for this module
-logger = logging.getLogger(__name__)
-
-class ReconnaissanceEngine:
-    """Main reconnaissance engine with smart TLD expansion."""
-    
-    def __init__(self, config: Config):
-        self.config = config
-        
-        # Initialize clients
-        self.dns_resolver = DNSResolver(config)
-        self.cert_checker = CertificateChecker(config)
-        self.tld_fetcher = TLDFetcher()
-        
-        # Optional clients
-        self.shodan_client = None
-        if config.shodan_key:
-            self.shodan_client = ShodanClient(config.shodan_key, config)
-            logger.info("✅ Shodan client initialized")
-        else:
-            logger.info("⚠️ Shodan API key not provided, skipping Shodan integration")
-        
-        self.virustotal_client = None
-        if config.virustotal_key:
-            self.virustotal_client = VirusTotalClient(config.virustotal_key, config)
-            logger.info("✅ VirusTotal client initialized")
-        else:
-            logger.info("⚠️ VirusTotal API key not provided, skipping VirusTotal integration")
-        
-        # Progress tracking
-        self.progress_callback = None
-        self._lock = threading.Lock()
-        
-        # Shared data object for live updates
-        self.shared_data = None
-    
-    def set_progress_callback(self, callback):
-        """Set callback for progress updates."""
-        self.progress_callback = callback
-    
-    def set_shared_data(self, shared_data: ReconData):
-        """Set shared data object for live updates during web interface usage."""
-        self.shared_data = shared_data
-        logger.info("📊 Using shared data object for live updates")
-    
-    def _update_progress(self, message: str, percentage: int = None):
-        """Update progress if callback is set."""
-        logger.info(f"Progress: {message} ({percentage}%)" if percentage else f"Progress: {message}")
-        if self.progress_callback:
-            self.progress_callback(message, percentage)
-    
-    def run_reconnaissance(self, target: str) -> ReconData:
-        """Run full reconnaissance on target."""
-        # Use shared data object if available, otherwise create new one
-        if self.shared_data is not None:
-            self.data = self.shared_data
-            logger.info("📊 Using shared data object for reconnaissance")
-        else:
-            self.data = ReconData()
-            logger.info("📊 Created new data object for reconnaissance")
-            
-        self.data.start_time = datetime.now()
-        
-        logger.info(f"🚀 Starting reconnaissance for target: {target}")
-        logger.info(f"📊 Configuration: max_depth={self.config.max_depth}, "
-                   f"DNS_rate={self.config.DNS_RATE_LIMIT}/s")
-        
-        try:
-            # Determine if target is hostname.tld or just hostname
-            if '.' in target:
-                logger.info(f"🎯 Target '{target}' appears to be a full domain name")
-                self._update_progress(f"Starting reconnaissance for {target}", 0)
-                self.data.add_hostname(target, 0)
-                initial_targets = {target}
-            else:
-                logger.info(f"🔍 Target '{target}' appears to be a hostname, expanding to all TLDs")
-                self._update_progress(f"Expanding {target} to all TLDs", 5)
-                initial_targets = self._expand_hostname_to_tlds_smart(target)
-                logger.info(f"📋 Found {len(initial_targets)} valid domains after TLD expansion")
-            
-            self._update_progress("Resolving initial targets", 10)
-            
-            # Process all targets recursively
-            self._process_targets_recursively(initial_targets)
-            
-            # Final external lookups
-            self._update_progress("Performing external service lookups", 90)
-            self._perform_external_lookups()
-            
-            # Log final statistics
-            stats = self.data.get_stats()
-            logger.info(f"📈 Final statistics: {stats}")
-            
-            self._update_progress("Reconnaissance complete", 100)
-            
-        except Exception as e:
-            logger.error(f"❌ Error during reconnaissance: {e}", exc_info=True)
-            raise
-        finally:
-            self.data.end_time = datetime.now()
-            duration = self.data.end_time - self.data.start_time
-            logger.info(f"⏱️ Total reconnaissance time: {duration}")
-        
-        return self.data
-    
-    def _expand_hostname_to_tlds_smart(self, hostname: str) -> Set[str]:
-        """Smart TLD expansion with prioritization and parallel processing."""
-        logger.info(f"🌐 Starting smart TLD expansion for hostname: {hostname}")
-        
-        # Get prioritized TLD lists
-        priority_tlds, normal_tlds, deprioritized_tlds = self.tld_fetcher.get_prioritized_tlds()
-        
-        logger.info(f"📊 TLD categories: {len(priority_tlds)} priority, "
-                   f"{len(normal_tlds)} normal, {len(deprioritized_tlds)} deprioritized")
-        
-        valid_domains = set()
-        
-        # Phase 1: Check priority TLDs first (parallel processing)
-        logger.info("🚀 Phase 1: Checking priority TLDs...")
-        priority_results = self._check_tlds_parallel(hostname, priority_tlds, "priority")
-        valid_domains.update(priority_results)
-        
-        self._update_progress(f"Phase 1 complete: {len(priority_results)} priority TLD matches", 6)
-        
-        # Phase 2: Check normal TLDs (if we found fewer than 5 results)
-        if len(valid_domains) < 5:
-            logger.info("🔍 Phase 2: Checking normal TLDs...")
-            normal_results = self._check_tlds_parallel(hostname, normal_tlds, "normal")
-            valid_domains.update(normal_results)
-            
-            self._update_progress(f"Phase 2 complete: {len(normal_results)} normal TLD matches", 8)
-        else:
-            logger.info(f"⏭️ Skipping normal TLDs (found {len(valid_domains)} matches in priority)")
-        
-        # Phase 3: Check deprioritized TLDs only if we found very few results
-        if len(valid_domains) < 2:
-            logger.info("🔍 Phase 3: Checking deprioritized TLDs (limited results so far)...")
-            depri_results = self._check_tlds_parallel(hostname, deprioritized_tlds, "deprioritized")
-            valid_domains.update(depri_results)
-            
-            self._update_progress(f"Phase 3 complete: {len(depri_results)} deprioritized TLD matches", 9)
-        else:
-            logger.info(f"⏭️ Skipping deprioritized TLDs (found {len(valid_domains)} matches already)")
-        
-        logger.info(f"🎯 Smart TLD expansion complete: found {len(valid_domains)} valid domains")
-        return valid_domains
-    
-    def _check_tlds_parallel(self, hostname: str, tlds: List[str], phase_name: str) -> Set[str]:
-        """Check TLDs in parallel with optimized settings."""
-        valid_domains = set()
-        tested_count = 0
-        wildcard_detected = set()
-        
-        # Use thread pool for parallel processing
-        max_workers = min(20, len(tlds))  # Limit concurrent requests
-        
-        logger.info(f"⚡ Starting parallel check of {len(tlds)} {phase_name} TLDs "
-                   f"with {max_workers} workers")
-        
-        with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
-            # Submit all tasks
-            future_to_tld = {
-                executor.submit(self._check_single_tld, hostname, tld): tld 
-                for tld in tlds
-            }
-            
-            # Process results as they complete
-            for future in concurrent.futures.as_completed(future_to_tld):
-                tld = future_to_tld[future]
-                tested_count += 1
-                
-                try:
-                    result = future.result(timeout=10)  # 10 second timeout per future
-                    
-                    if result:
-                        full_hostname, ips = result
-                        
-                        
-                        logger.info(f"✅ Valid domain found: {full_hostname} -> {ips}")
-                        self.data.add_hostname(full_hostname, 0)
-                        valid_domains.add(full_hostname)
-                        
-                        for ip in ips:
-                            self.data.add_ip_address(ip)
-                    
-                    # Progress update every 50 TLDs in this phase
-                    if tested_count % 50 == 0:
-                        logger.info(f"📊 {phase_name.title()} phase progress: "
-                                  f"{tested_count}/{len(tlds)} tested, "
-                                  f"{len(valid_domains)} found")
-                        
-                except concurrent.futures.TimeoutError:
-                    logger.debug(f"⏱️ Timeout checking {hostname}.{tld}")
-                except Exception as e:
-                    logger.debug(f"⚠️ Error checking {hostname}.{tld}: {e}")
-        
-        logger.info(f"📊 {phase_name.title()} phase complete: "
-                   f"tested {tested_count} TLDs, found {len(valid_domains)} valid domains, "
-                   f"detected {len(wildcard_detected)} wildcards")
-        
-        return valid_domains
-    
-    def _check_single_tld(self, hostname: str, tld: str) -> Optional[Tuple[str, List[str]]]:
-        """Check a single TLD combination with optimized DNS resolution."""
-        full_hostname = f"{hostname}.{tld}"
-        
-        # Use faster DNS resolution with shorter timeout for TLD testing
-        ips = self.dns_resolver.resolve_hostname_fast(full_hostname)
-        
-        if ips:
-            logger.debug(f"✅ {full_hostname} -> {ips}")
-            return (full_hostname, ips)
-        
-        return None
-    
-    def _process_targets_recursively(self, targets: Set[str]):
-        """Process targets with recursive subdomain discovery."""
-        current_depth = 0
-        
-        while current_depth <= self.config.max_depth and targets:
-            logger.info(f"🔄 Processing depth {current_depth} with {len(targets)} targets")
-            self._update_progress(f"Processing depth {current_depth} ({len(targets)} targets)", 15 + (current_depth * 25))
-            
-            new_targets = set()
-            
-            for target in targets:
-                logger.debug(f"🎯 Processing target: {target}")
-                
-                # DNS resolution and record gathering
-                self._process_single_target(target, current_depth)
-                
-                # Extract new subdomains
-                if current_depth < self.config.max_depth:
-                    new_subdomains = self._extract_new_subdomains(target)
-                    logger.debug(f"🌿 Found {len(new_subdomains)} new subdomains from {target}")
-                    
-                    for subdomain in new_subdomains:
-                        self.data.add_hostname(subdomain, current_depth + 1)
-                        new_targets.add(subdomain)
-            
-            logger.info(f"📊 Depth {current_depth} complete. Found {len(new_targets)} new targets for next depth")
-            targets = new_targets
-            current_depth += 1
-        
-        logger.info(f"🏁 Recursive processing complete after {current_depth} levels")
-    
-    def _process_single_target(self, hostname: str, depth: int):
-        """Process a single target hostname."""
-        logger.debug(f"🎯 Processing single target: {hostname} at depth {depth}")
-        
-        # Get all DNS records
-        dns_records = self.dns_resolver.get_all_dns_records(hostname)
-        logger.debug(f"📋 Found {len(dns_records)} DNS records for {hostname}")
-        
-        for record in dns_records:
-            self.data.add_dns_record(hostname, record)
-            
-            # Extract IP addresses from A and AAAA records
-            if record.record_type in ['A', 'AAAA']:
-                self.data.add_ip_address(record.value)
-        
-        # Get certificates
-        logger.debug(f"🔍 Checking certificates for {hostname}")
-        certificates = self.cert_checker.get_certificates(hostname)
-        if certificates:
-            self.data.certificates[hostname] = certificates
-            logger.info(f"📜 Found {len(certificates)} certificates for {hostname}")
-        else:
-            logger.debug(f"❌ No certificates found for {hostname}")
-    
-    def _extract_new_subdomains(self, hostname: str) -> Set[str]:
-        """Extract new subdomains from DNS records and certificates."""
-        new_subdomains = set()
-        
-        # From DNS records
-        if hostname in self.data.dns_records:
-            dns_subdomains = self.dns_resolver.extract_subdomains_from_dns(
-                self.data.dns_records[hostname]
-            )
-            new_subdomains.update(dns_subdomains)
-            logger.debug(f"🌐 Extracted {len(dns_subdomains)} subdomains from DNS records of {hostname}")
-        
-        # From certificates
-        if hostname in self.data.certificates:
-            cert_subdomains = self.cert_checker.extract_subdomains_from_certificates(
-                self.data.certificates[hostname]
-            )
-            new_subdomains.update(cert_subdomains)
-            logger.debug(f"🔍 Extracted {len(cert_subdomains)} subdomains from certificates of {hostname}")
-        
-        # Filter out already known hostnames
-        filtered_subdomains = new_subdomains - self.data.hostnames
-        logger.debug(f"🆕 {len(filtered_subdomains)} new subdomains after filtering")
-        
-        return filtered_subdomains
-    
-    def _perform_external_lookups(self):
-        """Perform Shodan and VirusTotal lookups."""
-        logger.info(f"🔍 Starting external lookups for {len(self.data.ip_addresses)} IPs and {len(self.data.hostnames)} hostnames")
-        
-        # Reverse DNS for all IPs
-        logger.info("🔄 Performing reverse DNS lookups")
-        reverse_dns_count = 0
-        for ip in self.data.ip_addresses:
-            reverse = self.dns_resolver.reverse_dns_lookup(ip)
-            if reverse:
-                self.data.reverse_dns[ip] = reverse
-                reverse_dns_count += 1
-                logger.debug(f"🔙 Reverse DNS for {ip}: {reverse}")
-        
-        logger.info(f"✅ Completed reverse DNS: {reverse_dns_count}/{len(self.data.ip_addresses)} successful")
-        
-        # Shodan lookups
-        if self.shodan_client:
-            logger.info(f"🕵️ Starting Shodan lookups for {len(self.data.ip_addresses)} IPs")
-            shodan_success_count = 0
-            
-            for ip in self.data.ip_addresses:
-                try:
-                    logger.debug(f"🔍 Querying Shodan for IP: {ip}")
-                    result = self.shodan_client.lookup_ip(ip)
-                    if result:
-                        self.data.add_shodan_result(ip, result)
-                        shodan_success_count += 1
-                        logger.info(f"✅ Shodan result for {ip}: {len(result.ports)} ports")
-                    else:
-                        logger.debug(f"❌ No Shodan data for {ip}")
-                except Exception as e:
-                    logger.warning(f"⚠️ Error querying Shodan for {ip}: {e}")
-            
-            logger.info(f"✅ Shodan lookups complete: {shodan_success_count}/{len(self.data.ip_addresses)} successful")
-        else:
-            logger.info("⚠️ Skipping Shodan lookups (no API key)")
-        
-        # VirusTotal lookups
-        if self.virustotal_client:
-            total_resources = len(self.data.ip_addresses) + len(self.data.hostnames)
-            logger.info(f"🛡️ Starting VirusTotal lookups for {total_resources} resources")
-            vt_success_count = 0
-            
-            # Check IPs
-            for ip in self.data.ip_addresses:
-                try:
-                    logger.debug(f"🔍 Querying VirusTotal for IP: {ip}")
-                    result = self.virustotal_client.lookup_ip(ip)
-                    if result:
-                        self.data.add_virustotal_result(ip, result)
-                        vt_success_count += 1
-                        logger.info(f"🛡️ VirusTotal result for {ip}: {result.positives}/{result.total} detections")
-                    else:
-                        logger.debug(f"❌ No VirusTotal data for {ip}")
-                except Exception as e:
-                    logger.warning(f"⚠️ Error querying VirusTotal for IP {ip}: {e}")
-            
-            # Check domains
-            for hostname in self.data.hostnames:
-                try:
-                    logger.debug(f"🔍 Querying VirusTotal for domain: {hostname}")
-                    result = self.virustotal_client.lookup_domain(hostname)
-                    if result:
-                        self.data.add_virustotal_result(hostname, result)
-                        vt_success_count += 1
-                        logger.info(f"🛡️ VirusTotal result for {hostname}: {result.positives}/{result.total} detections")
-                    else:
-                        logger.debug(f"❌ No VirusTotal data for {hostname}")
-                except Exception as e:
-                    logger.warning(f"⚠️ Error querying VirusTotal for domain {hostname}: {e}")
-            
-            logger.info(f"✅ VirusTotal lookups complete: {vt_success_count}/{total_resources} successful")
-        else:
-            logger.info("⚠️ Skipping VirusTotal lookups (no API key)")
-        
-        # Final external lookup summary
-        ext_stats = {
-            'reverse_dns': len(self.data.reverse_dns),
-            'shodan_results': len(self.data.shodan_results),
-            'virustotal_results': len(self.data.virustotal_results)
-        }
-        logger.info(f"📊 External lookups summary: {ext_stats}")
-    
-    # Keep the original method name for backward compatibility
-    def _expand_hostname_to_tlds(self, hostname: str) -> Set[str]:
-        """Legacy method - redirects to smart expansion."""
-        return self._expand_hostname_to_tlds_smart(hostname)

src/report_generator.py

@@ -1,111 +0,0 @@
-# File: src/report_generator.py
-"""Generate reports from reconnaissance data."""
-
-from datetime import datetime
-from typing import Dict, Any
-from .data_structures import ReconData
-
-class ReportGenerator:
-    """Generate various report formats."""
-    
-    def __init__(self, data: ReconData):
-        self.data = data
-    
-    def generate_text_report(self) -> str:
-        """Generate comprehensive text report."""
-        report = []
-        
-        # Header
-        report.append("="*80)
-        report.append("DNS RECONNAISSANCE REPORT")
-        report.append("="*80)
-        report.append(f"Start Time: {self.data.start_time}")
-        report.append(f"End Time: {self.data.end_time}")
-        if self.data.end_time:
-            duration = self.data.end_time - self.data.start_time
-            report.append(f"Duration: {duration}")
-        report.append("")
-        
-        # Summary
-        report.append("SUMMARY")
-        report.append("-" * 40)
-        report.append(f"Total Hostnames Discovered: {len(self.data.hostnames)}")
-        report.append(f"Total IP Addresses Found: {len(self.data.ip_addresses)}")
-        report.append(f"Total DNS Records: {sum(len(records) for records in self.data.dns_records.values())}")
-        report.append(f"Total Certificates Found: {sum(len(certs) for certs in self.data.certificates.values())}")
-        report.append("")
-        
-        # Hostnames by depth
-        report.append("HOSTNAMES BY DISCOVERY DEPTH")
-        report.append("-" * 40)
-        depth_groups = {}
-        for hostname, depth in self.data.depth_map.items():
-            if depth not in depth_groups:
-                depth_groups[depth] = []
-            depth_groups[depth].append(hostname)
-        
-        for depth in sorted(depth_groups.keys()):
-            report.append(f"Depth {depth}: {len(depth_groups[depth])} hostnames")
-            for hostname in sorted(depth_groups[depth]):
-                report.append(f"  - {hostname}")
-        report.append("")
-        
-        # IP Addresses
-        report.append("IP ADDRESSES")
-        report.append("-" * 40)
-        for ip in sorted(self.data.ip_addresses):
-            report.append(f"{ip}")
-            if ip in self.data.reverse_dns:
-                report.append(f"  Reverse DNS: {self.data.reverse_dns[ip]}")
-            if ip in self.data.shodan_results:
-                shodan = self.data.shodan_results[ip]
-                report.append(f"  Shodan: {len(shodan.ports)} open ports")
-                if shodan.organization:
-                    report.append(f"    Organization: {shodan.organization}")
-                if shodan.country:
-                    report.append(f"    Country: {shodan.country}")
-        report.append("")
-        
-        # DNS Records
-        report.append("DNS RECORDS")
-        report.append("-" * 40)
-        for hostname in sorted(self.data.dns_records.keys()):
-            report.append(f"{hostname}:")
-            records_by_type = {}
-            for record in self.data.dns_records[hostname]:
-                if record.record_type not in records_by_type:
-                    records_by_type[record.record_type] = []
-                records_by_type[record.record_type].append(record)
-            
-            for record_type in sorted(records_by_type.keys()):
-                report.append(f"  {record_type}:")
-                for record in records_by_type[record_type]:
-                    report.append(f"    {record.value}")
-        report.append("")
-        
-        # Certificates
-        if self.data.certificates:
-            report.append("CERTIFICATES")
-            report.append("-" * 40)
-            for hostname in sorted(self.data.certificates.keys()):
-                report.append(f"{hostname}:")
-                for cert in self.data.certificates[hostname]:
-                    report.append(f"  Certificate ID: {cert.id}")
-                    report.append(f"    Issuer: {cert.issuer}")
-                    report.append(f"    Valid From: {cert.not_before}")
-                    report.append(f"    Valid Until: {cert.not_after}")
-                    if cert.is_wildcard:
-                        report.append(f"    Type: Wildcard Certificate")
-                    report.append("")
-        
-        # Security Analysis
-        if self.data.virustotal_results:
-            report.append("SECURITY ANALYSIS")
-            report.append("-" * 40)
-            for resource, result in self.data.virustotal_results.items():
-                if result.positives > 0:
-                    report.append(f"⚠️  {resource}: {result.positives}/{result.total} detections")
-                    report.append(f"   Scan Date: {result.scan_date}")
-                    report.append(f"   Report: {result.permalink}")
-        
-        return "\n".join(report)

src/shodan_client.py

@@ -1,166 +0,0 @@
-# File: src/shodan_client.py
-"""Shodan API integration."""
-
-import requests
-import time
-import logging
-from typing import Optional, Dict, Any, List
-from .data_structures import ShodanResult
-from .config import Config
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-class ShodanClient:
-    """Shodan API client."""
-    
-    BASE_URL = "https://api.shodan.io"
-    
-    def __init__(self, api_key: str, config: Config):
-        self.api_key = api_key
-        self.config = config
-        self.last_request = 0
-        
-        logger.info(f"🕵️  Shodan client initialized with API key ending in: ...{api_key[-4:] if len(api_key) > 4 else api_key}")
-    
-    def _rate_limit(self):
-        """Apply rate limiting for Shodan."""
-        now = time.time()
-        time_since_last = now - self.last_request
-        min_interval = 1.0 / self.config.SHODAN_RATE_LIMIT
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            logger.debug(f"⏸️  Shodan rate limiting: sleeping for {sleep_time:.2f}s")
-            time.sleep(sleep_time)
-        
-        self.last_request = time.time()
-    
-    def lookup_ip(self, ip: str) -> Optional[ShodanResult]:
-        """Lookup IP address information."""
-        self._rate_limit()
-        
-        logger.debug(f"🔍 Querying Shodan for IP: {ip}")
-        
-        try:
-            url = f"{self.BASE_URL}/shodan/host/{ip}"
-            params = {'key': self.api_key}
-            
-            response = requests.get(
-                url, 
-                params=params, 
-                timeout=self.config.HTTP_TIMEOUT,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            
-            logger.debug(f"📡 Shodan API response for {ip}: {response.status_code}")
-            
-            if response.status_code == 200:
-                data = response.json()
-                
-                ports = []
-                services = {}
-                
-                for service in data.get('data', []):
-                    port = service.get('port')
-                    if port:
-                        ports.append(port)
-                        services[str(port)] = {
-                            'product': service.get('product', ''),
-                            'version': service.get('version', ''),
-                            'banner': service.get('data', '').strip()[:200] if service.get('data') else ''
-                        }
-                
-                result = ShodanResult(
-                    ip=ip,
-                    ports=sorted(list(set(ports))),
-                    services=services,
-                    organization=data.get('org'),
-                    country=data.get('country_name')
-                )
-                
-                logger.info(f"✅ Shodan result for {ip}: {len(result.ports)} ports, org: {result.organization}")
-                return result
-                
-            elif response.status_code == 404:
-                logger.debug(f"ℹ️  IP {ip} not found in Shodan database")
-                return None
-            elif response.status_code == 401:
-                logger.error("❌ Shodan API key is invalid or expired")
-                return None
-            elif response.status_code == 429:
-                logger.warning("⚠️  Shodan API rate limit exceeded")
-                return None
-            else:
-                logger.warning(f"⚠️  Shodan API error for {ip}: HTTP {response.status_code}")
-                try:
-                    error_data = response.json()
-                    logger.debug(f"Shodan error details: {error_data}")
-                except:
-                    pass
-                return None
-                
-        except requests.exceptions.Timeout:
-            logger.warning(f"⏱️  Shodan query timeout for {ip}")
-            return None
-        except requests.exceptions.RequestException as e:
-            logger.error(f"🌐 Shodan network error for {ip}: {e}")
-            return None
-        except Exception as e:
-            logger.error(f"❌ Unexpected error querying Shodan for {ip}: {e}")
-            return None
-    
-    def search_domain(self, domain: str) -> List[str]:
-        """Search for IPs associated with a domain."""
-        self._rate_limit()
-        
-        logger.debug(f"🔍 Searching Shodan for domain: {domain}")
-        
-        try:
-            url = f"{self.BASE_URL}/shodan/host/search"
-            params = {
-                'key': self.api_key,
-                'query': f'hostname:{domain}',
-                'limit': 100
-            }
-            
-            response = requests.get(
-                url, 
-                params=params, 
-                timeout=self.config.HTTP_TIMEOUT,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            
-            logger.debug(f"📡 Shodan search response for {domain}: {response.status_code}")
-            
-            if response.status_code == 200:
-                data = response.json()
-                ips = []
-                
-                for match in data.get('matches', []):
-                    ip = match.get('ip_str')
-                    if ip:
-                        ips.append(ip)
-                
-                unique_ips = list(set(ips))
-                logger.info(f"🔍 Shodan search for {domain} found {len(unique_ips)} unique IPs")
-                return unique_ips
-            elif response.status_code == 401:
-                logger.error("❌ Shodan API key is invalid for search")
-                return []
-            elif response.status_code == 429:
-                logger.warning("⚠️  Shodan search rate limit exceeded")
-                return []
-            else:
-                logger.warning(f"⚠️  Shodan search error for {domain}: HTTP {response.status_code}")
-                return []
-                
-        except requests.exceptions.Timeout:
-            logger.warning(f"⏱️  Shodan search timeout for {domain}")
-            return []
-        except requests.exceptions.RequestException as e:
-            logger.error(f"🌐 Shodan search network error for {domain}: {e}")
-            return []
-        except Exception as e:
-            logger.error(f"❌ Unexpected error searching Shodan for {domain}: {e}")
-            return []

src/tld_fetcher.py

@@ -1,213 +0,0 @@
-# File: src/tld_fetcher.py
-"""Fetch and cache IANA TLD list with smart prioritization."""
-
-import requests
-import logging
-from typing import List, Set, Optional, Tuple
-import os
-import time
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-class TLDFetcher:
-    """Fetches and caches IANA TLD list with smart prioritization."""
-    
-    IANA_TLD_URL = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
-    CACHE_FILE = "tlds_cache.txt"
-    CACHE_DURATION = 86400  # 24 hours in seconds
-    
-    # Common TLDs that should be checked first (high success rate)
-    PRIORITY_TLDS = {
-        # Generic top-level domains (most common)
-        'com', 'org', 'net', 'edu', 'gov', 'mil', 'int', 'info', 'biz', 'name',
-        'io', 'co', 'me', 'tv', 'cc', 'ly', 'to', 'us', 'uk', 'ca',
-        
-        # Major country codes (high usage)
-        'de', 'fr', 'it', 'es', 'nl', 'be', 'ch', 'at', 'se', 'no', 'dk', 'fi',
-        'au', 'nz', 'jp', 'kr', 'cn', 'hk', 'sg', 'my', 'th', 'in', 'br', 'mx',
-        'ru', 'pl', 'cz', 'hu', 'ro', 'bg', 'hr', 'si', 'sk', 'lt', 'lv', 'ee',
-        'ie', 'pt', 'gr', 'cy', 'mt', 'lu', 'is', 'tr', 'il', 'za', 'ng', 'eg',
-        
-        # Popular new gTLDs (established, not spam-prone)
-        'app', 'dev', 'tech', 'blog', 'news', 'shop', 'store', 'cloud', 'digital',
-        'website', 'site', 'online', 'world', 'global', 'international'
-    }
-    
-    # TLDs to deprioritize (often have wildcard DNS or low-quality domains)
-    DEPRIORITIZED_PATTERNS = [
-        'xn--',  # Internationalized domain names (often less common)
-        # These TLDs are known for high wildcard/parking rates
-        'tk', 'ml', 'ga', 'cf',  # Free TLDs often misused
-        'top', 'win', 'download', 'stream', 'science', 'click', 'link',
-        'loan', 'men', 'racing', 'review', 'party', 'trade', 'date',
-        'cricket', 'accountant', 'faith', 'gdn', 'realtor'
-    ]
-    
-    def __init__(self):
-        self._tlds: Optional[Set[str]] = None
-        self._prioritized_tlds: Optional[Tuple[List[str], List[str], List[str]]] = None
-        logger.info("🌐 TLD fetcher initialized with smart prioritization")
-    
-    def get_tlds(self) -> Set[str]:
-        """Get list of TLDs, using cache if available."""
-        if self._tlds is None:
-            logger.debug("🔍 Loading TLD list...")
-            self._tlds = self._load_tlds()
-            logger.info(f"✅ Loaded {len(self._tlds)} TLDs")
-        return self._tlds
-    
-    def get_prioritized_tlds(self) -> Tuple[List[str], List[str], List[str]]:
-        """Get TLDs sorted by priority: (priority, normal, deprioritized)."""
-        if self._prioritized_tlds is None:
-            all_tlds = self.get_tlds()
-            logger.debug("📊 Categorizing TLDs by priority...")
-            
-            priority_list = []
-            normal_list = []
-            deprioritized_list = []
-            
-            for tld in all_tlds:
-                tld_lower = tld.lower()
-                
-                if tld_lower in self.PRIORITY_TLDS:
-                    priority_list.append(tld_lower)
-                elif any(pattern in tld_lower for pattern in self.DEPRIORITIZED_PATTERNS):
-                    deprioritized_list.append(tld_lower)
-                else:
-                    normal_list.append(tld_lower)
-            
-            # Sort each category alphabetically for consistency
-            priority_list.sort()
-            normal_list.sort()
-            deprioritized_list.sort()
-            
-            self._prioritized_tlds = (priority_list, normal_list, deprioritized_list)
-            
-            logger.info(f"📊 TLD prioritization complete: "
-                       f"{len(priority_list)} priority, "
-                       f"{len(normal_list)} normal, "
-                       f"{len(deprioritized_list)} deprioritized")
-        
-        return self._prioritized_tlds
-    
-    def _load_tlds(self) -> Set[str]:
-        """Load TLDs from cache or fetch from IANA."""
-        if self._is_cache_valid():
-            logger.debug("📂 Loading TLDs from cache")
-            return self._load_from_cache()
-        else:
-            logger.info("🌐 Fetching fresh TLD list from IANA")
-            return self._fetch_and_cache()
-    
-    def _is_cache_valid(self) -> bool:
-        """Check if cache file exists and is recent."""
-        if not os.path.exists(self.CACHE_FILE):
-            logger.debug("❌ TLD cache file does not exist")
-            return False
-        
-        cache_age = time.time() - os.path.getmtime(self.CACHE_FILE)
-        is_valid = cache_age < self.CACHE_DURATION
-        
-        if is_valid:
-            logger.debug(f"✅ TLD cache is valid (age: {cache_age/3600:.1f} hours)")
-        else:
-            logger.debug(f"❌ TLD cache is expired (age: {cache_age/3600:.1f} hours)")
-        
-        return is_valid
-    
-    def _load_from_cache(self) -> Set[str]:
-        """Load TLDs from cache file."""
-        try:
-            with open(self.CACHE_FILE, 'r', encoding='utf-8') as f:
-                tlds = set()
-                for line in f:
-                    line = line.strip().lower()
-                    if line and not line.startswith('#'):
-                        tlds.add(line)
-                
-                logger.info(f"📂 Loaded {len(tlds)} TLDs from cache")
-                return tlds
-        except Exception as e:
-            logger.error(f"❌ Error loading TLD cache: {e}")
-            # Fall back to fetching fresh data
-            return self._fetch_and_cache()
-    
-    def _fetch_and_cache(self) -> Set[str]:
-        """Fetch TLDs from IANA and cache them."""
-        try:
-            logger.info(f"📡 Fetching TLD list from: {self.IANA_TLD_URL}")
-            
-            response = requests.get(
-                self.IANA_TLD_URL, 
-                timeout=30,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            response.raise_for_status()
-            
-            tlds = set()
-            lines_processed = 0
-            
-            for line in response.text.split('\n'):
-                line = line.strip().lower()
-                if line and not line.startswith('#'):
-                    tlds.add(line)
-                lines_processed += 1
-            
-            logger.info(f"✅ Fetched {len(tlds)} TLDs from IANA (processed {lines_processed} lines)")
-            
-            # Cache the results
-            try:
-                with open(self.CACHE_FILE, 'w', encoding='utf-8') as f:
-                    f.write(response.text)
-                logger.info(f"💾 TLD list cached to {self.CACHE_FILE}")
-            except Exception as cache_error:
-                logger.warning(f"⚠️ Could not cache TLD list: {cache_error}")
-            
-            return tlds
-            
-        except requests.exceptions.Timeout:
-            logger.error("⏱️ Timeout fetching TLD list from IANA")
-            return self._get_fallback_tlds()
-        except requests.exceptions.RequestException as e:
-            logger.error(f"🌐 Network error fetching TLD list: {e}")
-            return self._get_fallback_tlds()
-        except Exception as e:
-            logger.error(f"❌ Unexpected error fetching TLD list: {e}")
-            return self._get_fallback_tlds()
-    
-    def _get_fallback_tlds(self) -> Set[str]:
-        """Return a minimal set of short TLDs if fetch fails."""
-        logger.warning("⚠️ Using fallback TLD list")
-        
-        # Use only short, well-established TLDs as fallback
-        fallback_tlds = {
-            # 2-character TLDs (country codes - most established)
-            'ad', 'ae', 'af', 'ag', 'ai', 'al', 'am', 'ao', 'aq', 'ar', 'as', 'at',
-            'au', 'aw', 'ax', 'az', 'ba', 'bb', 'bd', 'be', 'bf', 'bg', 'bh', 'bi',
-            'bj', 'bl', 'bm', 'bn', 'bo', 'bq', 'br', 'bs', 'bt', 'bv', 'bw', 'by',
-            'bz', 'ca', 'cc', 'cd', 'cf', 'cg', 'ch', 'ci', 'ck', 'cl', 'cm', 'cn',
-            'co', 'cr', 'cu', 'cv', 'cw', 'cx', 'cy', 'cz', 'de', 'dj', 'dk', 'dm',
-            'do', 'dz', 'ec', 'ee', 'eg', 'eh', 'er', 'es', 'et', 'eu', 'fi', 'fj',
-            'fk', 'fm', 'fo', 'fr', 'ga', 'gb', 'gd', 'ge', 'gf', 'gg', 'gh', 'gi',
-            'gl', 'gm', 'gn', 'gp', 'gq', 'gr', 'gs', 'gt', 'gu', 'gw', 'gy', 'hk',
-            'hm', 'hn', 'hr', 'ht', 'hu', 'id', 'ie', 'il', 'im', 'in', 'io', 'iq',
-            'ir', 'is', 'it', 'je', 'jm', 'jo', 'jp', 'ke', 'kg', 'kh', 'ki', 'km',
-            'kn', 'kp', 'kr', 'kw', 'ky', 'kz', 'la', 'lb', 'lc', 'li', 'lk', 'lr',
-            'ls', 'lt', 'lu', 'lv', 'ly', 'ma', 'mc', 'md', 'me', 'mf', 'mg', 'mh',
-            'mk', 'ml', 'mm', 'mn', 'mo', 'mp', 'mq', 'mr', 'ms', 'mt', 'mu', 'mv',
-            'mw', 'mx', 'my', 'mz', 'na', 'nc', 'ne', 'nf', 'ng', 'ni', 'nl', 'no',
-            'np', 'nr', 'nu', 'nz', 'om', 'pa', 'pe', 'pf', 'pg', 'ph', 'pk', 'pl',
-            'pm', 'pn', 'pr', 'ps', 'pt', 'pw', 'py', 'qa', 're', 'ro', 'rs', 'ru',
-            'rw', 'sa', 'sb', 'sc', 'sd', 'se', 'sg', 'sh', 'si', 'sj', 'sk', 'sl',
-            'sm', 'sn', 'so', 'sr', 'ss', 'st', 'sv', 'sx', 'sy', 'sz', 'tc', 'td',
-            'tf', 'tg', 'th', 'tj', 'tk', 'tl', 'tm', 'tn', 'to', 'tr', 'tt', 'tv',
-            'tw', 'tz', 'ua', 'ug', 'uk', 'um', 'us', 'uy', 'uz', 'va', 'vc', 've',
-            'vg', 'vi', 'vn', 'vu', 'wf', 'ws', 'ye', 'yt', 'za', 'zm', 'zw',
-            
-            # 3-character TLDs (generic - most common)
-            'com', 'org', 'net', 'edu', 'gov', 'mil', 'int'
-        }
-        
-        logger.info(f"📋 Using {len(fallback_tlds)} fallback TLDs (≤3 characters)")
-        return fallback_tlds

src/virustotal_client.py

@@ -1,214 +0,0 @@
-# File: src/virustotal_client.py
-"""VirusTotal API integration."""
-
-import requests
-import time
-import logging
-from datetime import datetime
-from typing import Optional
-from .data_structures import VirusTotalResult
-from .config import Config
-
-# Module logger
-logger = logging.getLogger(__name__)
-
-class VirusTotalClient:
-    """VirusTotal API client."""
-    
-    BASE_URL = "https://www.virustotal.com/vtapi/v2"
-    
-    def __init__(self, api_key: str, config: Config):
-        self.api_key = api_key
-        self.config = config
-        self.last_request = 0
-        
-        logger.info(f"🛡️  VirusTotal client initialized with API key ending in: ...{api_key[-4:] if len(api_key) > 4 else api_key}")
-    
-    def _rate_limit(self):
-        """Apply rate limiting for VirusTotal."""
-        now = time.time()
-        time_since_last = now - self.last_request
-        min_interval = 1.0 / self.config.VIRUSTOTAL_RATE_LIMIT
-        
-        if time_since_last < min_interval:
-            sleep_time = min_interval - time_since_last
-            logger.debug(f"⏸️  VirusTotal rate limiting: sleeping for {sleep_time:.2f}s")
-            time.sleep(sleep_time)
-        
-        self.last_request = time.time()
-    
-    def lookup_ip(self, ip: str) -> Optional[VirusTotalResult]:
-        """Lookup IP address reputation."""
-        self._rate_limit()
-        
-        logger.debug(f"🔍 Querying VirusTotal for IP: {ip}")
-        
-        try:
-            url = f"{self.BASE_URL}/ip-address/report"
-            params = {
-                'apikey': self.api_key,
-                'ip': ip
-            }
-            
-            response = requests.get(
-                url, 
-                params=params, 
-                timeout=self.config.HTTP_TIMEOUT,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            
-            logger.debug(f"📡 VirusTotal API response for IP {ip}: {response.status_code}")
-            
-            if response.status_code == 200:
-                data = response.json()
-                
-                logger.debug(f"VirusTotal IP response data keys: {data.keys()}")
-                
-                if data.get('response_code') == 1:
-                    # Count detected URLs
-                    detected_urls = data.get('detected_urls', [])
-                    positives = sum(1 for url in detected_urls if url.get('positives', 0) > 0)
-                    total = len(detected_urls)
-                    
-                    # Parse scan date
-                    scan_date = datetime.now()
-                    if data.get('scan_date'):
-                        try:
-                            scan_date = datetime.fromisoformat(data['scan_date'].replace('Z', '+00:00'))
-                        except ValueError:
-                            try:
-                                scan_date = datetime.strptime(data['scan_date'], '%Y-%m-%d %H:%M:%S')
-                            except ValueError:
-                                logger.debug(f"Could not parse scan_date: {data.get('scan_date')}")
-                    
-                    result = VirusTotalResult(
-                        resource=ip,
-                        positives=positives,
-                        total=total,
-                        scan_date=scan_date,
-                        permalink=data.get('permalink', f'https://www.virustotal.com/gui/ip-address/{ip}')
-                    )
-                    
-                    logger.info(f"✅ VirusTotal result for IP {ip}: {result.positives}/{result.total} detections")
-                    return result
-                elif data.get('response_code') == 0:
-                    logger.debug(f"ℹ️  IP {ip} not found in VirusTotal database")
-                    return None
-                else:
-                    logger.debug(f"VirusTotal returned response_code: {data.get('response_code')}")
-                    return None
-            elif response.status_code == 204:
-                logger.warning("⚠️  VirusTotal API rate limit exceeded")
-                return None
-            elif response.status_code == 403:
-                logger.error("❌ VirusTotal API key is invalid or lacks permissions")
-                return None
-            else:
-                logger.warning(f"⚠️  VirusTotal API error for IP {ip}: HTTP {response.status_code}")
-                try:
-                    error_data = response.json()
-                    logger.debug(f"VirusTotal error details: {error_data}")
-                except:
-                    pass
-                return None
-                    
-        except requests.exceptions.Timeout:
-            logger.warning(f"⏱️  VirusTotal query timeout for IP {ip}")
-            return None
-        except requests.exceptions.RequestException as e:
-            logger.error(f"🌐 VirusTotal network error for IP {ip}: {e}")
-            return None
-        except Exception as e:
-            logger.error(f"❌ Unexpected error querying VirusTotal for IP {ip}: {e}")
-            return None
-    
-    def lookup_domain(self, domain: str) -> Optional[VirusTotalResult]:
-        """Lookup domain reputation."""
-        self._rate_limit()
-        
-        logger.debug(f"🔍 Querying VirusTotal for domain: {domain}")
-        
-        try:
-            url = f"{self.BASE_URL}/domain/report"
-            params = {
-                'apikey': self.api_key,
-                'domain': domain
-            }
-            
-            response = requests.get(
-                url, 
-                params=params, 
-                timeout=self.config.HTTP_TIMEOUT,
-                headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
-            )
-            
-            logger.debug(f"📡 VirusTotal API response for domain {domain}: {response.status_code}")
-            
-            if response.status_code == 200:
-                data = response.json()
-                
-                logger.debug(f"VirusTotal domain response data keys: {data.keys()}")
-                
-                if data.get('response_code') == 1:
-                    # Count detected URLs
-                    detected_urls = data.get('detected_urls', [])
-                    positives = sum(1 for url in detected_urls if url.get('positives', 0) > 0)
-                    total = len(detected_urls)
-                    
-                    # Also check for malicious/suspicious categories
-                    categories = data.get('categories', [])
-                    if any(cat in ['malicious', 'suspicious', 'phishing', 'malware'] 
-                          for cat in categories):
-                        positives += 1
-                    
-                    # Parse scan date
-                    scan_date = datetime.now()
-                    if data.get('scan_date'):
-                        try:
-                            scan_date = datetime.fromisoformat(data['scan_date'].replace('Z', '+00:00'))
-                        except ValueError:
-                            try:
-                                scan_date = datetime.strptime(data['scan_date'], '%Y-%m-%d %H:%M:%S')
-                            except ValueError:
-                                logger.debug(f"Could not parse scan_date: {data.get('scan_date')}")
-                    
-                    result = VirusTotalResult(
-                        resource=domain,
-                        positives=positives,
-                        total=max(total, 1),  # Ensure total is at least 1
-                        scan_date=scan_date,
-                        permalink=data.get('permalink', f'https://www.virustotal.com/gui/domain/{domain}')
-                    )
-                    
-                    logger.info(f"✅ VirusTotal result for domain {domain}: {result.positives}/{result.total} detections")
-                    return result
-                elif data.get('response_code') == 0:
-                    logger.debug(f"ℹ️  Domain {domain} not found in VirusTotal database")
-                    return None
-                else:
-                    logger.debug(f"VirusTotal returned response_code: {data.get('response_code')}")
-                    return None
-            elif response.status_code == 204:
-                logger.warning("⚠️  VirusTotal API rate limit exceeded")
-                return None
-            elif response.status_code == 403:
-                logger.error("❌ VirusTotal API key is invalid or lacks permissions")
-                return None
-            else:
-                logger.warning(f"⚠️  VirusTotal API error for domain {domain}: HTTP {response.status_code}")
-                try:
-                    error_data = response.json()
-                    logger.debug(f"VirusTotal error details: {error_data}")
-                except:
-                    pass
-                return None
-                    
-        except requests.exceptions.Timeout:
-            logger.warning(f"⏱️  VirusTotal query timeout for domain {domain}")
-            return None
-        except requests.exceptions.RequestException as e:
-            logger.error(f"🌐 VirusTotal network error for domain {domain}: {e}")
-            return None
-        except Exception as e:
-            logger.error(f"❌ Unexpected error querying VirusTotal for domain {domain}: {e}")
-            return None

src/web_app.py

@@ -1,231 +0,0 @@
-# File: src/web_app.py
-"""Flask web application for reconnaissance tool."""
-
-from flask import Flask, render_template, request, jsonify, send_from_directory
-import threading
-import time
-import logging
-from .config import Config
-from .reconnaissance import ReconnaissanceEngine
-from .report_generator import ReportGenerator
-from .data_structures import ReconData
-
-# Set up logging for this module
-logger = logging.getLogger(__name__)
-
-# Global variables for tracking ongoing scans
-active_scans = {}
-scan_lock = threading.Lock()
-
-def create_app(config: Config):
-    """Create Flask application."""
-    app = Flask(__name__, 
-                template_folder='../templates',
-                static_folder='../static')
-    
-    app.config['SECRET_KEY'] = 'recon-tool-secret-key'
-    
-    # Set up logging for web app
-    config.setup_logging(cli_mode=False)
-    logger.info("🌐 Web application initialized")
-    
-    @app.route('/')
-    def index():
-        """Main page."""
-        return render_template('index.html')
-    
-    @app.route('/api/scan', methods=['POST'])
-    def start_scan():
-        """Start a new reconnaissance scan."""
-        try:
-            data = request.get_json()
-            target = data.get('target')
-            scan_config = Config.from_args(
-                shodan_key=data.get('shodan_key'),
-                virustotal_key=data.get('virustotal_key'),
-                max_depth=data.get('max_depth', 2)
-            )
-            
-            if not target:
-                logger.warning("⚠️ Scan request missing target")
-                return jsonify({'error': 'Target is required'}), 400
-            
-            # Generate scan ID
-            scan_id = f"{target}_{int(time.time())}"
-            logger.info(f"🚀 Starting new scan: {scan_id} for target: {target}")
-            
-            # Create shared ReconData object for live updates
-            shared_data = ReconData()
-            
-            # Initialize scan data with the shared data object
-            with scan_lock:
-                active_scans[scan_id] = {
-                    'status': 'starting',
-                    'progress': 0,
-                    'message': 'Initializing...',
-                    'data': shared_data,  # Share the data object from the start!
-                    'error': None,
-                    'live_stats': {
-                        'hostnames': 0,
-                        'ip_addresses': 0,
-                        'dns_records': 0,
-                        'certificates': 0,
-                        'shodan_results': 0,
-                        'virustotal_results': 0
-                    },
-                    'latest_discoveries': []
-                }
-            
-            # Start reconnaissance in background thread
-            thread = threading.Thread(
-                target=run_reconnaissance_background,
-                args=(scan_id, target, scan_config, shared_data)
-            )
-            thread.daemon = True
-            thread.start()
-            
-            return jsonify({'scan_id': scan_id})
-        
-        except Exception as e:
-            logger.error(f"❌ Error starting scan: {e}", exc_info=True)
-            return jsonify({'error': str(e)}), 500
-    
-    @app.route('/api/scan/<scan_id>/status')
-    def get_scan_status(scan_id):
-        """Get scan status and progress with live discoveries."""
-        with scan_lock:
-            if scan_id not in active_scans:
-                return jsonify({'error': 'Scan not found'}), 404
-            
-            scan_data = active_scans[scan_id].copy()
-        
-        # Don't include the full data object in status (too large)
-        if 'data' in scan_data:
-            del scan_data['data']
-        
-        return jsonify(scan_data)
-    
-    @app.route('/api/scan/<scan_id>/report')
-    def get_scan_report(scan_id):
-        """Get scan report."""
-        with scan_lock:
-            if scan_id not in active_scans:
-                return jsonify({'error': 'Scan not found'}), 404
-            
-            scan_data = active_scans[scan_id]
-            
-            if scan_data['status'] != 'completed' or not scan_data['data']:
-                return jsonify({'error': 'Scan not completed'}), 400
-        
-        try:
-            # Generate report
-            report_gen = ReportGenerator(scan_data['data'])
-            
-            return jsonify({
-                'json_report': scan_data['data'].to_json(),
-                'text_report': report_gen.generate_text_report()
-            })
-        except Exception as e:
-            logger.error(f"❌ Error generating report for {scan_id}: {e}", exc_info=True)
-            return jsonify({'error': f'Failed to generate report: {str(e)}'}), 500
-    
-    @app.route('/api/scan/<scan_id>/live-data')
-    def get_live_scan_data(scan_id):
-        """Get current reconnaissance data (for real-time updates)."""
-        with scan_lock:
-            if scan_id not in active_scans:
-                return jsonify({'error': 'Scan not found'}), 404
-            
-            scan_data = active_scans[scan_id]
-            
-            # Now we always have a data object, even if it's empty initially
-            data_obj = scan_data['data']
-            
-            if not data_obj:
-                return jsonify({
-                    'hostnames': [],
-                    'ip_addresses': [],
-                    'stats': scan_data['live_stats'],
-                    'latest_discoveries': []
-                })
-            
-            # Return current discoveries from the shared data object
-            return jsonify({
-                'hostnames': sorted(list(data_obj.hostnames)),
-                'ip_addresses': sorted(list(data_obj.ip_addresses)),
-                'stats': data_obj.get_stats(),
-                'latest_discoveries': scan_data.get('latest_discoveries', [])
-            })
-    
-    return app
-
-def run_reconnaissance_background(scan_id: str, target: str, config: Config, shared_data: ReconData):
-    """Run reconnaissance in background thread with shared data object."""
-    
-    def update_progress(message: str, percentage: int = None):
-        """Update scan progress and live statistics."""
-        with scan_lock:
-            if scan_id in active_scans:
-                active_scans[scan_id]['message'] = message
-                if percentage is not None:
-                    active_scans[scan_id]['progress'] = percentage
-                
-                # Update live stats from the shared data object
-                if shared_data:
-                    active_scans[scan_id]['live_stats'] = shared_data.get_stats()
-                    
-                    # Add to latest discoveries (keep last 10)
-                    if 'latest_discoveries' not in active_scans[scan_id]:
-                        active_scans[scan_id]['latest_discoveries'] = []
-                    
-                    active_scans[scan_id]['latest_discoveries'].append({
-                        'timestamp': time.time(),
-                        'message': message
-                    })
-                    
-                    # Keep only last 10 discoveries
-                    active_scans[scan_id]['latest_discoveries'] = \
-                        active_scans[scan_id]['latest_discoveries'][-10:]
-        
-        logger.info(f"[{scan_id}] {message} ({percentage}%)" if percentage else f"[{scan_id}] {message}")
-    
-    try:
-        logger.info(f"🔧 Initializing reconnaissance engine for scan: {scan_id}")
-        
-        # Initialize engine
-        engine = ReconnaissanceEngine(config)
-        engine.set_progress_callback(update_progress)
-        
-        # IMPORTANT: Pass the shared data object to the engine
-        engine.set_shared_data(shared_data)
-        
-        # Update status
-        with scan_lock:
-            active_scans[scan_id]['status'] = 'running'
-        
-        logger.info(f"🚀 Starting reconnaissance for: {target}")
-        
-        # Run reconnaissance - this will populate the shared_data object incrementally
-        final_data = engine.run_reconnaissance(target)
-        
-        logger.info(f"✅ Reconnaissance completed for scan: {scan_id}")
-        
-        # Update with final results (the shared_data should already be populated)
-        with scan_lock:
-            active_scans[scan_id]['status'] = 'completed'
-            active_scans[scan_id]['progress'] = 100
-            active_scans[scan_id]['message'] = 'Reconnaissance completed'
-            active_scans[scan_id]['data'] = final_data  # This should be the same as shared_data
-            active_scans[scan_id]['live_stats'] = final_data.get_stats()
-        
-        # Log final statistics
-        final_stats = final_data.get_stats()
-        logger.info(f"📊 Final stats for {scan_id}: {final_stats}")
-    
-    except Exception as e:
-        logger.error(f"❌ Error in reconnaissance for {scan_id}: {e}", exc_info=True)
-        # Handle errors
-        with scan_lock:
-            active_scans[scan_id]['status'] = 'error'
-            active_scans[scan_id]['error'] = str(e)
-            active_scans[scan_id]['message'] = f'Error: {str(e)}'

static/script.js

@@ -1,555 +0,0 @@
-// DNS Reconnaissance Tool - Enhanced Frontend JavaScript with Debug Output
-
-class ReconTool {
-    constructor() {
-        this.currentScanId = null;
-        this.pollInterval = null;
-        this.liveDataInterval = null;
-        this.currentReport = null;
-        this.debugMode = true; // Enable debug logging
-        this.init();
-    }
-
-    debug(message, data = null) {
-        if (this.debugMode) {
-            if (data) {
-                console.log(`🔍 DEBUG: ${message}`, data);
-            } else {
-                console.log(`🔍 DEBUG: ${message}`);
-            }
-        }
-    }
-
-    init() {
-        this.bindEvents();
-        this.setupRealtimeElements();
-    }
-    
-    setupRealtimeElements() {
-        // Create live discovery container if it doesn't exist
-        if (!document.getElementById('liveDiscoveries')) {
-            const progressSection = document.getElementById('progressSection');
-            const liveDiv = document.createElement('div');
-            liveDiv.id = 'liveDiscoveries';
-            liveDiv.innerHTML = `
-                <div class="live-discoveries" style="display: none;">
-                    <h3>🔍 Live Discoveries</h3>
-                    <div class="stats-grid">
-                        <div class="stat-item">
-                            <span class="stat-label">Hostnames:</span>
-                            <span id="liveHostnames" class="stat-value">0</span>
-                        </div>
-                        <div class="stat-item">
-                            <span class="stat-label">IP Addresses:</span>
-                            <span id="liveIPs" class="stat-value">0</span>
-                        </div>
-                        <div class="stat-item">
-                            <span class="stat-label">DNS Records:</span>
-                            <span id="liveDNS" class="stat-value">0</span>
-                        </div>
-                        <div class="stat-item">
-                            <span class="stat-label">Certificates:</span>
-                            <span id="liveCerts" class="stat-value">0</span>
-                        </div>
-                        <div class="stat-item">
-                            <span class="stat-label">Shodan Results:</span>
-                            <span id="liveShodan" class="stat-value">0</span>
-                        </div>
-                        <div class="stat-item">
-                            <span class="stat-label">VirusTotal:</span>
-                            <span id="liveVT" class="stat-value">0</span>
-                        </div>
-                    </div>
-                    <div class="discoveries-list">
-                        <h4>📋 Recent Discoveries</h4>
-                        <div id="recentHostnames" class="discovery-section">
-                            <strong>Hostnames:</strong>
-                            <div class="hostname-list"></div>
-                        </div>
-                        <div id="recentIPs" class="discovery-section">
-                            <strong>IP Addresses:</strong>
-                            <div class="ip-list"></div>
-                        </div>
-                        <div id="activityLog" class="discovery-section">
-                            <strong>Activity Log:</strong>
-                            <div class="activity-list"></div>
-                        </div>
-                    </div>
-                </div>
-            `;
-            progressSection.appendChild(liveDiv);
-            this.debug("Live discoveries container created");
-        }
-    }
-
-    bindEvents() {
-        // Start scan button
-        document.getElementById('startScan').addEventListener('click', () => {
-            this.startScan();
-        });
-
-        // New scan button
-        document.getElementById('newScan').addEventListener('click', () => {
-            this.resetToForm();
-        });
-
-        // Report view toggles
-        document.getElementById('showJson').addEventListener('click', () => {
-            this.showReport('json');
-        });
-
-        document.getElementById('showText').addEventListener('click', () => {
-            this.showReport('text');
-        });
-
-        // Download buttons
-        document.getElementById('downloadJson').addEventListener('click', () => {
-            this.downloadReport('json');
-        });
-
-        document.getElementById('downloadText').addEventListener('click', () => {
-            this.downloadReport('text');
-        });
-
-        // Enter key in target field
-        document.getElementById('target').addEventListener('keypress', (e) => {
-            if (e.key === 'Enter') {
-                this.startScan();
-            }
-        });
-    }
-
-    async startScan() {
-        const target = document.getElementById('target').value.trim();
-        
-        if (!target) {
-            alert('Please enter a target domain or hostname');
-            return;
-        }
-
-        const scanData = {
-            target: target,
-            max_depth: parseInt(document.getElementById('maxDepth').value),
-            shodan_key: document.getElementById('shodanKey').value.trim() || null,
-            virustotal_key: document.getElementById('virustotalKey').value.trim() || null
-        };
-
-        try {
-            // Show progress section
-            this.showProgressSection();
-            this.updateProgress(0, 'Starting scan...');
-
-            this.debug('Starting scan with data:', scanData);
-
-            const response = await fetch('/api/scan', {
-                method: 'POST',
-                headers: {
-                    'Content-Type': 'application/json',
-                },
-                body: JSON.stringify(scanData)
-            });
-
-            if (!response.ok) {
-                throw new Error(`HTTP error! status: ${response.status}`);
-            }
-
-            const result = await response.json();
-            
-            if (result.error) {
-                throw new Error(result.error);
-            }
-
-            this.currentScanId = result.scan_id;
-            this.debug('Scan started with ID:', this.currentScanId);
-            
-            this.startPolling();
-            this.startLiveDataPolling();
-
-        } catch (error) {
-            console.error('❌ Failed to start scan:', error);
-            this.showError(`Failed to start scan: ${error.message}`);
-        }
-    }
-
-    startPolling() {
-        this.debug('Starting status polling...');
-        // Poll every 2 seconds for status updates
-        this.pollInterval = setInterval(() => {
-            this.checkScanStatus();
-        }, 2000);
-        
-        // Also check immediately
-        this.checkScanStatus();
-    }
-    
-    startLiveDataPolling() {
-        this.debug('Starting live data polling...');
-        // Poll every 3 seconds for live data updates
-        this.liveDataInterval = setInterval(() => {
-            this.updateLiveData();
-        }, 3000);
-        
-        // Show the live discoveries section
-        const liveSection = document.querySelector('.live-discoveries');
-        if (liveSection) {
-            liveSection.style.display = 'block';
-            this.debug('Live discoveries section made visible');
-        } else {
-            this.debug('ERROR: Live discoveries section not found!');
-        }
-        
-        // Also update immediately
-        this.updateLiveData();
-    }
-
-    async checkScanStatus() {
-        if (!this.currentScanId) {
-            return;
-        }
-
-        try {
-            const response = await fetch(`/api/scan/${this.currentScanId}/status`);
-            
-            if (!response.ok) {
-                throw new Error(`HTTP error! status: ${response.status}`);
-            }
-
-            const status = await response.json();
-            
-            if (status.error) {
-                throw new Error(status.error);
-            }
-
-            // Update progress
-            this.updateProgress(status.progress, status.message);
-            
-            // Update live stats
-            if (status.live_stats) {
-                this.debug('Received live stats:', status.live_stats);
-                this.updateLiveStats(status.live_stats);
-            }
-
-            // Check if completed
-            if (status.status === 'completed') {
-                this.debug('Scan completed, loading report...');
-                this.stopPolling();
-                await this.loadScanReport();
-            } else if (status.status === 'error') {
-                this.stopPolling();
-                throw new Error(status.error || 'Scan failed');
-            }
-
-        } catch (error) {
-            console.error('❌ Error checking scan status:', error);
-            this.stopPolling();
-            this.showError(`Error checking scan status: ${error.message}`);
-        }
-    }
-    
-    async updateLiveData() {
-        if (!this.currentScanId) {
-            return;
-        }
-        
-        this.debug(`Fetching live data for scan: ${this.currentScanId}`);
-        
-        try {
-            const response = await fetch(`/api/scan/${this.currentScanId}/live-data`);
-            
-            if (!response.ok) {
-                this.debug(`Live data request failed: HTTP ${response.status}`);
-                return; // Silently fail for live data
-            }
-            
-            const data = await response.json();
-            
-            if (data.error) {
-                this.debug('Live data error:', data.error);
-                return; // Silently fail for live data
-            }
-            
-            this.debug('Received live data:', data);
-            
-            // Update live discoveries
-            this.updateLiveDiscoveries(data);
-            
-        } catch (error) {
-            // Silently fail for live data updates
-            this.debug('Live data update failed:', error);
-        }
-    }
-    
-    updateLiveStats(stats) {
-        this.debug('Updating live stats:', stats);
-        
-        // Update the live statistics counters
-        const statElements = {
-            'liveHostnames': stats.hostnames || 0,
-            'liveIPs': stats.ip_addresses || 0,
-            'liveDNS': stats.dns_records || 0,
-            'liveCerts': stats.certificates || 0,
-            'liveShodan': stats.shodan_results || 0,
-            'liveVT': stats.virustotal_results || 0
-        };
-        
-        Object.entries(statElements).forEach(([elementId, value]) => {
-            const element = document.getElementById(elementId);
-            if (element) {
-                const currentValue = element.textContent;
-                element.textContent = value;
-                
-                if (currentValue !== value.toString()) {
-                    this.debug(`Updated ${elementId}: ${currentValue} -> ${value}`);
-                    // Add a brief highlight effect when value changes
-                    element.style.backgroundColor = '#ff9900';
-                    setTimeout(() => {
-                        element.style.backgroundColor = '';
-                    }, 1000);
-                }
-            } else {
-                this.debug(`ERROR: Element ${elementId} not found!`);
-            }
-        });
-    }
-    
-    updateLiveDiscoveries(data) {
-        this.debug('Updating live discoveries with data:', data);
-        
-        // Update hostnames list
-        const hostnameList = document.querySelector('#recentHostnames .hostname-list');
-        if (hostnameList && data.hostnames && data.hostnames.length > 0) {
-            // Show last 10 hostnames
-            const recentHostnames = data.hostnames;
-            hostnameList.innerHTML = recentHostnames.map(hostname => 
-                `<span class="discovery-item">${hostname}</span>`
-            ).join('');
-            this.debug(`Updated hostname list with ${recentHostnames.length} items`);
-        } else if (hostnameList) {
-            this.debug(`No hostnames to display (${data.hostnames ? data.hostnames.length : 0} total)`);
-        }
-        
-        // Update IP addresses list
-        const ipList = document.querySelector('#recentIPs .ip-list');
-        if (ipList && data.ip_addresses && data.ip_addresses.length > 0) {
-            // Show last 10 IPs
-            const recentIPs = data.ip_addresses;
-            ipList.innerHTML = recentIPs.map(ip => 
-                `<span class="discovery-item">${ip}</span>`
-            ).join('');
-            this.debug(`Updated IP list with ${recentIPs.length} items`);
-        } else if (ipList) {
-            this.debug(`No IPs to display (${data.ip_addresses ? data.ip_addresses.length : 0} total)`);
-        }
-        
-        // Update activity log
-        const activityList = document.querySelector('#activityLog .activity-list');
-        if (activityList && data.latest_discoveries && data.latest_discoveries.length > 0) {
-            const activities = data.latest_discoveries.slice(-5); // Last 5 activities
-            activityList.innerHTML = activities.map(activity => {
-                const time = new Date(activity.timestamp * 1000).toLocaleTimeString();
-                return `<div class="activity-item">[${time}] ${activity.message}</div>`;
-            }).join('');
-            this.debug(`Updated activity log with ${activities.length} items`);
-        } else if (activityList) {
-            this.debug(`No activities to display (${data.latest_discoveries ? data.latest_discoveries.length : 0} total)`);
-        }
-    }
-
-    async loadScanReport() {
-        try {
-            this.debug('Loading scan report...');
-            const response = await fetch(`/api/scan/${this.currentScanId}/report`);
-            
-            if (!response.ok) {
-                throw new Error(`HTTP error! status: ${response.status}`);
-            }
-
-            const report = await response.json();
-            
-            if (report.error) {
-                throw new Error(report.error);
-            }
-
-            this.currentReport = report;
-            this.debug('Report loaded successfully');
-            this.showResultsSection();
-            this.showReport('text'); // Default to text view
-
-        } catch (error) {
-            console.error('❌ Error loading report:', error);
-            this.showError(`Error loading report: ${error.message}`);
-        }
-    }
-
-    stopPolling() {
-        this.debug('Stopping polling intervals...');
-        if (this.pollInterval) {
-            clearInterval(this.pollInterval);
-            this.pollInterval = null;
-        }
-        if (this.liveDataInterval) {
-            clearInterval(this.liveDataInterval);
-            this.liveDataInterval = null;
-        }
-    }
-
-    showProgressSection() {
-        document.getElementById('scanForm').style.display = 'none';
-        document.getElementById('progressSection').style.display = 'block';
-        document.getElementById('resultsSection').style.display = 'none';
-        this.debug('Showing progress section');
-    }
-
-    showResultsSection() {
-        document.getElementById('scanForm').style.display = 'none';
-        document.getElementById('progressSection').style.display = 'block'; // Keep visible
-        document.getElementById('resultsSection').style.display = 'block';
-        
-        // Change the title to show it's the final summary
-        const liveSection = document.querySelector('.live-discoveries');
-        if (liveSection) {
-            const title = liveSection.querySelector('h3');
-            if (title) {
-                title.textContent = '📊 Final Discovery Summary';
-            }
-            liveSection.style.display = 'block';
-        }
-        
-        // Hide just the progress bar and scan controls
-        const progressBar = document.querySelector('.progress-bar');
-        const progressMessage = document.getElementById('progressMessage');
-        const scanControls = document.querySelector('.scan-controls');
-        
-        if (progressBar) progressBar.style.display = 'none';
-        if (progressMessage) progressMessage.style.display = 'none';
-        if (scanControls) scanControls.style.display = 'none';
-        
-        this.debug('Showing results section with live discoveries');
-    }
-
-    resetToForm() {
-        this.stopPolling();
-        this.currentScanId = null;
-        this.currentReport = null;
-        
-        document.getElementById('scanForm').style.display = 'block';
-        document.getElementById('progressSection').style.display = 'none';
-        document.getElementById('resultsSection').style.display = 'none';
-        
-        // Show progress elements again
-        const progressBar = document.querySelector('.progress-bar');
-        const progressMessage = document.getElementById('progressMessage');
-        const scanControls = document.querySelector('.scan-controls');
-        
-        if (progressBar) progressBar.style.display = 'block';
-        if (progressMessage) progressMessage.style.display = 'block';
-        if (scanControls) scanControls.style.display = 'block';
-        
-        // Hide live discoveries and reset title
-        const liveSection = document.querySelector('.live-discoveries');
-        if (liveSection) {
-            liveSection.style.display = 'none';
-            const title = liveSection.querySelector('h3');
-            if (title) {
-                title.textContent = '🔍 Live Discoveries';
-            }
-        }
-        
-        // Clear form
-        document.getElementById('target').value = '';
-        document.getElementById('shodanKey').value = '';
-        document.getElementById('virustotalKey').value = '';
-        document.getElementById('maxDepth').value = '2';
-        
-        this.debug('Reset to form view');
-    }
-
-    updateProgress(percentage, message) {
-        const progressFill = document.getElementById('progressFill');
-        const progressMessage = document.getElementById('progressMessage');
-        
-        progressFill.style.width = `${percentage || 0}%`;
-        progressMessage.textContent = message || 'Processing...';
-    }
-
-    showError(message) {
-        // Update progress section to show error
-        this.updateProgress(0, `Error: ${message}`);
-        
-        // Also alert the user
-        alert(`Error: ${message}`);
-    }
-
-    showReport(type) {
-        if (!this.currentReport) {
-            return;
-        }
-
-        const reportContent = document.getElementById('reportContent');
-        const showJsonBtn = document.getElementById('showJson');
-        const showTextBtn = document.getElementById('showText');
-
-        if (type === 'json') {
-            // Show JSON report
-            try {
-                // The json_report should already be a string from the server
-                let jsonData;
-                if (typeof this.currentReport.json_report === 'string') {
-                    jsonData = JSON.parse(this.currentReport.json_report);
-                } else {
-                    jsonData = this.currentReport.json_report;
-                }
-                reportContent.textContent = JSON.stringify(jsonData, null, 2);
-            } catch (e) {
-                console.error('Error parsing JSON report:', e);
-                reportContent.textContent = this.currentReport.json_report;
-            }
-            
-            showJsonBtn.classList.add('active');
-            showTextBtn.classList.remove('active');
-        } else {
-            // Show text report
-            reportContent.textContent = this.currentReport.text_report;
-            
-            showTextBtn.classList.add('active');
-            showJsonBtn.classList.remove('active');
-        }
-    }
-
-    downloadReport(type) {
-        if (!this.currentReport) {
-            return;
-        }
-
-        let content, filename, mimeType;
-
-        if (type === 'json') {
-            content = typeof this.currentReport.json_report === 'string' 
-                ? this.currentReport.json_report 
-                : JSON.stringify(this.currentReport.json_report, null, 2);
-            filename = `recon-report-${this.currentScanId}.json`;
-            mimeType = 'application/json';
-        } else {
-            content = this.currentReport.text_report;
-            filename = `recon-report-${this.currentScanId}.txt`;
-            mimeType = 'text/plain';
-        }
-
-        // Create download link
-        const blob = new Blob([content], { type: mimeType });
-        const url = window.URL.createObjectURL(blob);
-        const a = document.createElement('a');
-        a.href = url;
-        a.download = filename;
-        document.body.appendChild(a);
-        a.click();
-        window.URL.revokeObjectURL(url);
-        document.body.removeChild(a);
-    }
-}
-
-// Initialize the application when DOM is loaded
-document.addEventListener('DOMContentLoaded', () => {
-    console.log('🌐 DNS Reconnaissance Tool initialized with debug mode');
-    new ReconTool();
-});

static/style.css

@@ -1,439 +0,0 @@
-/*
-    ███████╗██████╗ ███████╗ ██████╗████████╗     ██████╗  ██████╗ ██╗  ██╗███████╗
-    ██╔════╝██╔══██╗██╔════╝██╔═══██╗╚══██╔══╝    ██╔═══██╗██╔═══██╗╚██╗██╔╝██╔════╝
-    ███████╗██████╔╝█████╗  ██║   ██║   ██║       ██║   ██║██║   ██║ ╚███╔╝ ███████╗
-    ╚════██║██╔══██╗██╔══╝  ██║   ██║   ██║       ██║   ██║██║   ██║ ██╔██╗ ╚════██║
-    ███████║██║  ██║███████╗╚██████╔╝   ██║       ╚██████╔╝╚██████╔╝██╔╝ ██╗███████║
-    ╚══════╝╚═╝  ╚═╝╚══════╝ ╚═════╝    ╚═╝        ╚═════╝  ╚═════╝ ╚═╝  ╚═╝╚══════╝
-    
-    TACTICAL THEME - DNS RECONNAISSANCE INTERFACE
-    STYLE OVERRIDE
-*/
-
-* {
-    margin: 0;
-    padding: 0;
-    box-sizing: border-box;
-}
-
-body {
-    font-family: 'Roboto Mono', 'Lucida Console', Monaco, monospace;
-    line-height: 1.6;
-    color: #c7c7c7; /* Light grey for readability */
-    /* Dark, textured background for a gritty feel */
-    background-color: #1a1a1a;
-    background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='4' height='4' viewBox='0 0 4 4'%3E%3Cpath fill='%23333333' fill-opacity='0.4' d='M1 3h1v1H1V3zm2-2h1v1H3V1z'%3E%3C/path%3E%3C/svg%3E");
-    min-height: 100vh;
-}
-
-.container {
-    max-width: 1200px;
-    margin: 0 auto;
-    padding: 20px;
-}
-
-header {
-    text-align: center;
-    color: #e0e0e0;
-    margin-bottom: 40px;
-    border-bottom: 1px solid #444;
-    padding-bottom: 20px;
-}
-
-header h1 {
-    font-family: 'Special Elite', 'Courier New', monospace; /* Stencil / Typewriter font */
-    font-size: 2.8rem;
-    color: #00ff41; /* Night-vision green */
-    text-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
-    margin-bottom: 10px;
-    letter-spacing: 2px;
-}
-
-header p {
-    font-size: 1.1rem;
-    color: #a0a0a0;
-}
-
-.scan-form, .progress-section, .results-section {
-    background: #2a2a2a; /* Dark charcoal */
-    border-radius: 4px; /* Sharper edges */
-    border: 1px solid #444;
-    box-shadow: inset 0 0 15px rgba(0,0,0,0.5);
-    padding: 30px;
-    margin-bottom: 25px;
-}
-
-.scan-form h2, .progress-section h2, .results-section h2 {
-    margin-bottom: 20px;
-    color: #e0e0e0;
-    border-bottom: 1px solid #555;
-    padding-bottom: 10px;
-    text-transform: uppercase; /* Military style */
-    letter-spacing: 1px;
-}
-
-.form-group {
-    margin-bottom: 20px;
-}
-
-.form-group label {
-    display: block;
-    margin-bottom: 8px;
-    font-weight: 600;
-    color: #b0b0b0;
-    text-transform: uppercase;
-    font-size: 0.9rem;
-}
-
-.form-group input, .form-group select {
-    width: 100%;
-    padding: 12px;
-    background: #1a1a1a;
-    border: 1px solid #555;
-    border-radius: 2px;
-    font-size: 16px;
-    color: #00ff41; /* Green text for input fields */
-    font-family: 'Roboto Mono', monospace;
-    transition: all 0.2s ease-in-out;
-}
-
-.form-group input:focus, .form-group select:focus {
-    outline: none;
-    border-color: #ff9900; /* Amber focus color */
-    box-shadow: 0 0 5px rgba(255, 153, 0, 0.5);
-}
-
-.api-keys {
-    background: rgba(0,0,0,0.3);
-    padding: 20px;
-    border-radius: 4px;
-    border: 1px solid #444;
-    margin: 20px 0;
-}
-
-.api-keys h3 {
-    margin-bottom: 15px;
-    color: #c7c7c7;
-}
-
-.btn-primary, .btn-secondary {
-    padding: 12px 24px;
-    border: 1px solid #666;
-    border-radius: 2px;
-    font-size: 16px;
-    font-weight: 600;
-    cursor: pointer;
-    transition: all 0.2s ease-in-out;
-    margin-right: 10px;
-    margin-bottom: 10px;
-    text-transform: uppercase;
-    letter-spacing: 1px;
-}
-
-.btn-primary {
-    background: #2c5c34; /* Dark military green */
-    color: #e0e0e0;
-    border-color: #3b7b46;
-}
-
-.btn-primary:hover {
-    background: #3b7b46; /* Lighter green on hover */
-    color: #fff;
-    border-color: #4cae5c;
-}
-
-.btn-secondary {
-    background: #4a4a4a; /* Dark grey */
-    color: #c7c7c7;
-    border-color: #666;
-}
-
-.btn-secondary:hover {
-    background: #5a5a5a;
-}
-
-.btn-secondary.active {
-    background: #6a4f2a; /* Amber/Brown for active state */
-    color: #fff;
-    border-color: #ff9900;
-}
-
-.progress-bar {
-    width: 100%;
-    height: 20px;
-    background: #1a1a1a;
-    border: 1px solid #555;
-    border-radius: 2px;
-    overflow: hidden;
-    margin-bottom: 15px;
-    padding: 2px;
-}
-
-.progress-fill {
-    height: 100%;
-    background: #ff9900; /* Solid amber progress fill */
-    width: 0%;
-    transition: width 0.3s ease;
-    border-radius: 0;
-}
-
-#progressMessage {
-    font-weight: 500;
-    color: #a0a0a0;
-    margin-bottom: 20px;
-}
-
-.scan-controls {
-    text-align: center;
-}
-
-.results-controls {
-    margin-bottom: 20px;
-    text-align: center;
-}
-
-.report-container {
-    background: #0a0a0a; /* Near-black terminal background */
-    border-radius: 4px;
-    border: 1px solid #333;
-    padding: 20px;
-    max-height: 600px;
-    overflow-y: auto;
-    box-shadow: inset 0 0 10px #000;
-}
-
-#reportContent {
-    color: #00ff41; /* Classic terminal green */
-    font-family: 'Courier New', monospace;
-    font-size: 14px;
-    line-height: 1.4;
-    white-space: pre-wrap;
-    word-wrap: break-word;
-}
-
-.hostname-list, .ip-list {
-    display: flex;
-    flex-wrap: wrap;
-    gap: 5px;
-}
-
-.discovery-item {
-    background: #2a2a2a;
-    color: #00ff41;
-    padding: 2px 6px;
-    border-radius: 2px;
-    font-family: 'Courier New', monospace;
-    font-size: 0.8rem;
-    border: 1px solid #444;
-}
-
-.activity-list {
-    max-height: 150px;
-    overflow-y: auto;
-}
-
-.activity-item {
-    color: #a0a0a0;
-    font-family: 'Courier New', monospace;
-    font-size: 0.8rem;
-    padding: 2px 0;
-    border-bottom: 1px solid #333;
-}
-
-.activity-item:last-child {
-    border-bottom: none;
-}
-
-/* Live Discoveries Base Styling */
-.live-discoveries {
-    background: rgba(0, 20, 0, 0.6);
-    border: 1px solid #00ff41;
-    border-radius: 4px;
-    padding: 20px;
-    margin-top: 20px;
-}
-
-.live-discoveries h3 {
-    color: #00ff41;
-    margin-bottom: 15px;
-    text-transform: uppercase;
-    letter-spacing: 1px;
-}
-
-/* Enhanced styling for live discoveries when shown in results view */
-.results-section .live-discoveries {
-    background: rgba(0, 40, 0, 0.8);
-    border: 2px solid #00ff41;
-    border-radius: 4px;
-    padding: 20px;
-    margin-bottom: 25px;
-    box-shadow: 0 0 10px rgba(0, 255, 65, 0.3);
-}
-
-.results-section .live-discoveries h3 {
-    color: #00ff41;
-    text-shadow: 0 0 3px rgba(0, 255, 65, 0.5);
-}
-
-/* Ensure the progress section flows nicely when showing both progress and results */
-.progress-section.with-results {
-    margin-bottom: 0;
-    border-bottom: none;
-}
-
-.results-section.with-live-data {
-    border-top: 1px solid #444;
-    padding-top: 20px;
-}
-
-/* Better spacing for the combined view */
-.progress-section + .results-section {
-    margin-top: 0;
-}
-
-/* Hide specific progress elements while keeping the section visible */
-.progress-section .progress-bar.hidden,
-.progress-section #progressMessage.hidden,
-.progress-section .scan-controls.hidden {
-    display: none !important;
-}
-
-.stats-grid {
-    display: grid;
-    grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
-    gap: 15px;
-    margin-bottom: 20px;
-}
-
-.stat-item {
-    display: flex;
-    justify-content: space-between;
-    align-items: center;
-    padding: 8px 12px;
-    background: rgba(0, 0, 0, 0.5);
-    border: 1px solid #333;
-    border-radius: 2px;
-}
-
-.stat-label {
-    color: #a0a0a0;
-    font-size: 0.9rem;
-}
-
-.stat-value {
-    color: #00ff41;
-    font-weight: bold;
-    font-family: 'Courier New', monospace;
-    transition: background-color 0.3s ease;
-}
-
-/* Animation for final stats highlight */
-@keyframes finalHighlight {
-    0% { background-color: #ff9900; }
-    100% { background-color: transparent; }
-}
-
-.stat-value.final {
-    animation: finalHighlight 2s ease-in-out;
-}
-
-.discoveries-list {
-    margin-top: 20px;
-}
-
-.discoveries-list h4 {
-    color: #ff9900;
-    margin-bottom: 15px;
-    border-bottom: 1px solid #444;
-    padding-bottom: 5px;
-}
-
-.discovery-section {
-    margin-bottom: 15px;
-    padding: 10px;
-    background: rgba(0, 0, 0, 0.3);
-    border: 1px solid #333;
-    border-radius: 2px;
-}
-
-.discovery-section strong {
-    color: #c7c7c7;
-    display: block;
-    margin-bottom: 8px;
-    font-size: 0.9rem;
-}
-
-/* Tactical loading spinner */
-.loading {
-    display: inline-block;
-    width: 20px;
-    height: 20px;
-    border: 3px solid rgba(199, 199, 199, 0.3);
-    border-radius: 50%;
-    border-top-color: #00ff41; /* Night-vision green spinner */
-    animation: spin 1s linear infinite;
-}
-
-@keyframes spin {
-    to { transform: rotate(360deg); }
-}
-
-/* Responsive design adjustments */
-@media (max-width: 768px) {
-    .container {
-        padding: 10px;
-    }
-    
-    header h1 {
-        font-size: 2.2rem;
-    }
-    
-    .scan-form, .progress-section, .results-section {
-        padding: 20px;
-    }
-    
-    .btn-primary, .btn-secondary {
-        width: 100%;
-        margin-right: 0;
-    }
-    
-    .results-controls {
-        display: flex;
-        flex-wrap: wrap;
-        justify-content: center;
-    }
-    
-    .results-controls button {
-        flex: 1;
-        min-width: 120px;
-    }
-    
-    .stats-grid {
-        grid-template-columns: repeat(2, 1fr);
-        gap: 10px;
-    }
-    
-    .stat-item {
-        padding: 6px 8px;
-    }
-    
-    .stat-label, .stat-value {
-        font-size: 0.8rem;
-    }
-    
-    .hostname-list, .ip-list {
-        flex-direction: column;
-        align-items: flex-start;
-    }
-
-    /* Responsive adjustments for the combined view */
-    .results-section .live-discoveries {
-        padding: 15px;
-        margin-bottom: 15px;
-    }
-    
-    .results-section .live-discoveries .stats-grid {
-        grid-template-columns: repeat(2, 1fr);
-        gap: 10px;
-    }
-}

templates/index.html

@@ -1,80 +1,321 @@
 <!DOCTYPE html>
 <html lang="en">
+
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>DNS Reconnaissance Tool</title>
-    <link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
+    <title>DNScope - Infrastructure Reconnaissance</title>
+    <link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
+    <link href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" rel="stylesheet" type="text/css">
+    <link
+        href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@300;400;500;700&family=Special+Elite&display=swap"
+        rel="stylesheet">
 </head>
+
 <body>
     <div class="container">
-        <header>
-            <h1>🔍 DNS Reconnaissance Tool</h1>
-            <p>Comprehensive domain and IP intelligence gathering</p>
+        <header class="header">
+            <div class="header-content">
+                <div class="logo">
+                    <span class="logo-icon">[DN]</span>
+                    <span class="logo-text">Scope</span>
+                </div>
+                <div class="status-indicator">
+                    <span id="connection-status" class="status-dot"></span>
+                    <span class="status-text">System Online</span>
+                </div>
+            </div>
         </header>
 
-        <div class="scan-form" id="scanForm">
-            <h2>Start New Scan</h2>
-            
-            <div class="form-group">
-                <label for="target">Target (domain.com or hostname):</label>
-                <input type="text" id="target" placeholder="example.com or example" required>
+        <main class="main-content">
+            <section class="control-panel">
+                <div class="panel-header">
+                    <h2>Target Configuration</h2>
                 </div>
 
-            <div class="form-group">
-                <label for="maxDepth">Max Recursion Depth:</label>
-                <select id="maxDepth">
-                    <option value="1">1</option>
-                    <option value="2" selected>2</option>
-                    <option value="3">3</option>
-                    <option value="4">4</option>
-                    <option value="5">5</option>
+                <div class="form-container">
+                    <div class="input-group">
+                        <label for="target-input">Target Domain or IP</label>
+                        <input type="text" id="target-input" placeholder="example.com or 8.8.8.8" autocomplete="off">
+                    </div>
+
+                    <div class="button-group">
+                        <button id="start-scan" class="btn btn-primary">
+                            <span class="btn-icon">[RUN]</span>
+                            <span>Start Reconnaissance</span>
+                        </button>
+                        <button id="add-to-graph" class="btn btn-primary">
+                            <span class="btn-icon">[ADD]</span>
+                            <span>Add to Graph</span>
+                        </button>
+                        <button id="stop-scan" class="btn btn-secondary" disabled>
+                            <span class="btn-icon">[STOP]</span>
+                            <span>Terminate Scan</span>
+                        </button>
+                        <button id="export-options" class="btn btn-secondary">
+                            <span class="btn-icon">[EXPORT]</span>
+                            <span>Export Options</span>
+                        </button>
+                        <button id="configure-settings" class="btn btn-secondary">
+                            <span class="btn-icon">[API]</span>
+                            <span>Settings</span>
+                        </button>
+                    </div>
+                </div>
+            </section>
+
+            <section class="status-panel">
+                <div class="panel-header">
+                    <h2>Reconnaissance Status</h2>
+                </div>
+
+                <div class="status-content">
+                    <div class="status-row">
+                        <span class="status-label">Current Status:</span>
+                        <span id="scan-status" class="status-value">Idle</span>
+                    </div>
+                    <div class="status-row">
+                        <span class="status-label">Target:</span>
+                        <span id="target-display" class="status-value">None</span>
+                    </div>
+                    <div class="status-row">
+                        <span class="status-label">Depth:</span>
+                        <span id="depth-display" class="status-value">0/0</span>
+                    </div>
+                    <div class="status-row">
+                        <span class="status-label">Relationships:</span>
+                        <span id="relationships-display" class="status-value">0</span>
+                    </div>
+                </div>
+
+                <div class="progress-container">
+                    <div class="progress-info">
+                        <span id="progress-label">Progress:</span>
+                        <span id="progress-compact">0/0</span>
+                    </div>
+                    <div class="progress-bar">
+                        <div id="progress-fill" class="progress-fill"></div>
+                    </div>
+                    <div class="progress-placeholder">
+                        <span class="status-label">
+                            ⚠️ <strong>Important:</strong> Scanning large public services (e.g., Google, Cloudflare,
+                            AWS) is
+                            <strong>discouraged</strong> due to rate limits (e.g., crt.sh).
+                            <br><br>
+                            Our task scheduler operates on a <strong>priority-based queue</strong>:
+                            Short, targeted tasks like DNS are processed first, while resource-intensive requests (e.g.,
+                            crt.sh)
+                            are <strong>automatically deprioritized</strong> and may be processed later.
+                        </span>
+                    </div>
+                </div>
+            </section>
+
+            <section class="visualization-panel">
+                <div class="panel-header">
+                    <h2>Infrastructure Map</h2>
+                </div>
+
+                <div id="network-graph" class="graph-container">
+                    <div class="graph-placeholder">
+                        <div class="placeholder-content">
+                            <div class="placeholder-icon">[◯]</div>
+                            <div class="placeholder-text">Infrastructure map will appear here</div>
+                            <div class="placeholder-subtext">Start a reconnaissance scan to visualize relationships
+                            </div>
+                        </div>
+                    </div>
+                </div>
+
+                <div class="legend">
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #00ff41;"></div>
+                        <span>Domains</span>
+                    </div>
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #c92f2f;"></div>
+                        <span>Domain (no valid cert)</span>
+                    </div>
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #c7c7c7;"></div>
+                        <span>Domain (never had cert)</span>
+                    </div>
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #ff9900;"></div>
+                        <span>IP Addresses</span>
+                    </div>
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #00aaff;"></div>
+                        <span>ISPs</span>
+                    </div>
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #ff6b6b;"></div>
+                        <span>Certificate Authorities</span>
+                    </div>
+
+                    <div class="legend-item">
+                        <div class="legend-color" style="background-color: #9d4edd;"></div>
+                        <span>Correlation Objects</span>
+                    </div>
+                </div>
+            </section>
+
+            <section class="provider-panel">
+                <div class="panel-header">
+                    <h2>Data Providers</h2>
+                </div>
+
+                <div id="provider-list" class="provider-list">
+                </div>
+            </section>
+        </main>
+
+        <footer class="footer">
+            <div class="footer-content">
+                <span>v0.0.0rc</span>
+                <span class="footer-separator">|</span>
+                <span>Passive Infrastructure Reconnaissance</span>
+                <span class="footer-separator">|</span>
+                <span id="session-id">Session: Loading...</span>
+            </div>
+        </footer>
+
+        <div id="node-modal" class="modal">
+            <div class="modal-content">
+                <div class="modal-header">
+                    <h3 id="modal-title">Node Details</h3>
+                    <button id="modal-close" class="modal-close">[×]</button>
+                </div>
+                <div class="modal-body">
+                    <div id="modal-details">
+                    </div>
+                </div>
+            </div>
+        </div>
+
+        <div id="settings-modal" class="modal">
+            <div class="modal-content">
+                <div class="modal-header">
+                    <h3>Scanner Configuration</h3>
+                    <button id="settings-modal-close" class="modal-close">[×]</button>
+                </div>
+                <div class="modal-body">
+                    <div class="modal-details">
+                        <section class="modal-section">
+                            <details open>
+                                <summary>
+                                    <span>⚙️ Scan Settings</span>
+                                </summary>
+                                <div class="modal-section-content">
+                                    <div class="input-group">
+                                        <label for="max-depth">Recursion Depth</label>
+                                        <select id="max-depth">
+                                            <option value="1">Depth 1 - Direct relationships</option>
+                                            <option value="2" selected>Depth 2 - Recommended</option>
+                                            <option value="3">Depth 3 - Extended analysis</option>
+                                            <option value="4">Depth 4 - Deep reconnaissance</option>
+                                            <option value="5">Depth 5 - Maximum depth</option>
                                         </select>
                                     </div>
+                                </div>
+                            </details>
+                        </section>
 
-            <div class="api-keys">
-                <h3>Optional API Keys</h3>
-                <div class="form-group">
-                    <label for="shodanKey">Shodan API Key:</label>
-                    <input type="password" id="shodanKey" placeholder="Optional - for port scanning data">
+                        <section class="modal-section">
+                            <details open>
+                                <summary>
+                                    <span>🔧 Provider Configuration</span>
+                                    <span class="merge-badge" id="provider-count">0</span>
+                                </summary>
+                                <div class="modal-section-content">
+                                    <div id="provider-config-list">
                                         </div>
+                                </div>
+                            </details>
+                        </section>
 
-                <div class="form-group">
-                    <label for="virustotalKey">VirusTotal API Key:</label>
-                    <input type="password" id="virustotalKey" placeholder="Optional - for security analysis">
+                        <section class="modal-section">
+                            <details>
+                                <summary>
+                                    <span>🔑 API Keys</span>
+                                    <span class="merge-badge" id="api-key-count">0</span>
+                                </summary>
+                                <div class="modal-section-content">
+                                    <p class="placeholder-subtext" style="margin-bottom: 1rem;">
+                                        ⚠️ API keys are stored in memory for the current session only.
+                                        Only provide API keys you don't use for anything else.
+                                    </p>
+                                    <div id="api-key-inputs">
                                         </div>
                                 </div>
+                            </details>
+                        </section>
 
-            <button id="startScan" class="btn-primary">Start Reconnaissance</button>
-        </div>
-
-        <div class="progress-section" id="progressSection" style="display: none;">
-            <h2>Scan Progress</h2>
-            <div class="progress-bar">
-                <div class="progress-fill" id="progressFill"></div>
-            </div>
-            <p id="progressMessage">Initializing...</p>
-            <div class="scan-controls">
-                <button id="newScan" class="btn-secondary">New Scan</button>
+                        <div class="button-group" style="margin-top: 1.5rem;">
+                            <button id="save-settings" class="btn btn-primary">
+                                <span class="btn-icon">[SAVE]</span>
+                                <span>Save Configuration</span>
+                            </button>
+                            <button id="reset-settings" class="btn btn-secondary">
+                                <span class="btn-icon">[RESET]</span>
+                                <span>Reset to Defaults</span>
+                            </button>
                         </div>
                     </div>
-
-        <div class="results-section" id="resultsSection" style="display: none;">
-            <h2>Reconnaissance Results</h2>
-            
-            <div class="results-controls">
-                <button id="showJson" class="btn-secondary">Show JSON</button>
-                <button id="showText" class="btn-secondary active">Show Text Report</button>
-                <button id="downloadJson" class="btn-secondary">Download JSON</button>
-                <button id="downloadText" class="btn-secondary">Download Text</button>
-            </div>
-            
-            <div class="report-container">
-                <pre id="reportContent"></pre>
                 </div>
             </div>
         </div>
 
-    <script src="{{ url_for('static', filename='script.js') }}"></script>
+        <div id="export-modal" class="modal">
+            <div class="modal-content">
+                <div class="modal-header">
+                    <h3>Export Options</h3>
+                    <button id="export-modal-close" class="modal-close">[×]</button>
+                </div>
+                <div class="modal-body">
+                    <div class="modal-details">
+                        <section class="modal-section">
+                            <details open>
+                                <summary>
+                                    <span>📊 Available Exports</span>
+                                </summary>
+                                <div class="modal-section-content">
+                                    <div class="button-group" style="margin-top: 1rem;">
+                                        <button id="export-graph-json" class="btn btn-primary">
+                                            <span class="btn-icon">[JSON]</span>
+                                            <span>Export Graph Data</span>
+                                        </button>
+                                        <div class="status-row" style="margin-top: 0.5rem;">
+                                            <span class="status-label">Complete graph data with forensic audit trail,
+                                                provider statistics, and scan metadata in JSON format for analysis and
+                                                archival.</span>
+                                        </div>
+                                        <button id="export-targets-txt" class="btn btn-primary" style="margin-top: 1rem;">
+                                            <span class="btn-icon">[TXT]</span>
+                                            <span>Export Targets</span>
+                                        </button>
+                                        <div class="status-row" style="margin-top: 0.5rem;">
+                                            <span class="status-label">A simple text file containing all discovered domains and IP addresses.</span>
+                                        </div>
+                                        <button id="export-executive-summary" class="btn btn-primary" style="margin-top: 1rem;">
+                                            <span class="btn-icon">[TXT]</span>
+                                            <span>Export Executive Summary</span>
+                                        </button>
+                                        <div class="status-row" style="margin-top: 0.5rem;">
+                                            <span class="status-label">A natural-language summary of the scan findings.</span>
+                                        </div>
+                                    </div>
+                                </div>
+                            </details>
+                        </section>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+    <script src="{{ url_for('static', filename='js/graph.js') }}"></script>
+    <script src="{{ url_for('static', filename='js/main.js') }}"></script>
 </body>
+
 </html>

utils/__init__.py

@@ -0,0 +1,22 @@
+# DNScope-reduced/utils/__init__.py
+
+"""
+Utility modules for DNScope.
+Contains helper functions, export management, and supporting utilities.
+"""
+
+from .helpers import is_valid_target, _is_valid_domain, _is_valid_ip, get_ip_version, normalize_ip
+from .export_manager import export_manager, ExportManager, CustomJSONEncoder
+
+__all__ = [
+    'is_valid_target',
+    '_is_valid_domain', 
+    '_is_valid_ip',
+    'get_ip_version',
+    'normalize_ip',
+    'export_manager',
+    'ExportManager',
+    'CustomJSONEncoder'
+]
+
+__version__ = "1.0.0"

utils/export_manager.py

@@ -0,0 +1,800 @@
+# DNScope-reduced/utils/export_manager.py
+
+"""
+Centralized export functionality for DNScope.
+Handles all data export operations with forensic integrity and proper formatting.
+ENHANCED: Professional forensic executive summary generation for court-ready documentation.
+"""
+
+import json
+from datetime import datetime, timezone
+from typing import Dict, Any, List, Optional, Set, Tuple
+from decimal import Decimal
+from collections import defaultdict, Counter
+import networkx as nx
+
+from utils.helpers import _is_valid_domain, _is_valid_ip
+
+
+class ExportManager:
+    """
+    Centralized manager for all DNScope export operations.
+    Maintains forensic integrity and provides consistent export formats.
+    ENHANCED: Advanced forensic analysis and professional reporting capabilities.
+    """
+    
+    def __init__(self):
+        """Initialize export manager."""
+        pass
+    
+    def export_scan_results(self, scanner) -> Dict[str, Any]:
+        """
+        Export complete scan results with forensic metadata.
+        
+        Args:
+            scanner: Scanner instance with completed scan data
+            
+        Returns:
+            Complete scan results dictionary
+        """
+        graph_data = self.export_graph_json(scanner.graph)
+        audit_trail = scanner.logger.export_audit_trail()
+        provider_stats = {}
+        
+        for provider in scanner.providers:
+            provider_stats[provider.get_name()] = provider.get_statistics()
+        
+        results = {
+            'scan_metadata': {
+                'target_domain': scanner.current_target,
+                'max_depth': scanner.max_depth,
+                'final_status': scanner.status,
+                'total_indicators_processed': scanner.indicators_processed,
+                'enabled_providers': list(provider_stats.keys()),
+                'session_id': scanner.session_id
+            },
+            'graph_data': graph_data,
+            'forensic_audit': audit_trail,
+            'provider_statistics': provider_stats,
+            'scan_summary': scanner.logger.get_forensic_summary()
+        }
+        
+        # Add export metadata
+        results['export_metadata'] = {
+            'export_timestamp': datetime.now(timezone.utc).isoformat(),
+            'export_version': '1.0.0',
+            'forensic_integrity': 'maintained'
+        }
+        
+        return results
+    
+    def export_targets_list(self, scanner) -> str:
+        """
+        Export all discovered domains and IPs as a text file.
+        
+        Args:
+            scanner: Scanner instance with graph data
+            
+        Returns:
+            Newline-separated list of targets
+        """
+        nodes = scanner.graph.get_graph_data().get('nodes', [])
+        targets = {
+            node['id'] for node in nodes 
+            if _is_valid_domain(node['id']) or _is_valid_ip(node['id'])
+        }
+        return "\n".join(sorted(list(targets)))
+    
+    def generate_executive_summary(self, scanner) -> str:
+        """
+        ENHANCED: Generate a comprehensive, court-ready forensic executive summary.
+        
+        Args:
+            scanner: Scanner instance with completed scan data
+            
+        Returns:
+            Professional forensic summary formatted for investigative use
+        """
+        report = []
+        now = datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
+        
+        # Get comprehensive data for analysis
+        graph_data = scanner.graph.get_graph_data()
+        nodes = graph_data.get('nodes', [])
+        edges = graph_data.get('edges', [])
+        audit_trail = scanner.logger.export_audit_trail()
+        
+        # Perform advanced analysis
+        infrastructure_analysis = self._analyze_infrastructure_patterns(nodes, edges)
+        
+        # === HEADER AND METADATA ===
+        report.extend([
+            "=" * 80,
+            "DIGITAL INFRASTRUCTURE RECONNAISSANCE REPORT",
+            "=" * 80,
+            "",
+            f"Report Generated: {now}",
+            f"Investigation Target: {scanner.current_target}",
+            f"Analysis Session: {scanner.session_id}",
+            f"Scan Depth: {scanner.max_depth} levels",
+            f"Final Status: {scanner.status.upper()}",
+            ""
+        ])
+        
+        # === EXECUTIVE SUMMARY ===
+        report.extend([
+            "EXECUTIVE SUMMARY",
+            "-" * 40,
+            "",
+            f"This report presents the findings of a comprehensive passive reconnaissance analysis "
+            f"conducted against the target '{scanner.current_target}'. The investigation employed "
+            f"multiple intelligence sources and discovered {len(nodes)} distinct digital entities "
+            f"connected through {len(edges)} verified relationships.",
+            "",
+            f"The analysis reveals a digital infrastructure comprising {infrastructure_analysis['domains']} "
+            f"domain names, {infrastructure_analysis['ips']} IP addresses, and {infrastructure_analysis['isps']} "
+            f"infrastructure service providers. Certificate transparency analysis identified "
+            f"{infrastructure_analysis['cas']} certificate authorities managing the cryptographic "
+            f"infrastructure for the investigated entities.",
+            "",
+        ])
+        
+        # === METHODOLOGY ===
+        report.extend([
+            "INVESTIGATIVE METHODOLOGY",
+            "-" * 40,
+            "",
+            "This analysis employed passive reconnaissance techniques using the following verified data sources:",
+            ""
+        ])
+        
+        provider_info = {
+            'dns': 'Standard DNS resolution and reverse DNS lookups',
+            'crtsh': 'Certificate Transparency database analysis via crt.sh',
+            'shodan': 'Internet-connected device intelligence via Shodan API'
+        }
+        
+        for provider in scanner.providers:
+            provider_name = provider.get_name()
+            stats = provider.get_statistics()
+            description = provider_info.get(provider_name, f'{provider_name} data provider')
+            
+            report.extend([
+                f"• {provider.get_display_name()}: {description}",
+                f"  - Total Requests: {stats['total_requests']}",
+                f"  - Success Rate: {stats['success_rate']:.1f}%",
+                f"  - Relationships Discovered: {stats['relationships_found']}",
+                ""
+            ])
+        
+        # === INFRASTRUCTURE ANALYSIS ===
+        report.extend([
+            "INFRASTRUCTURE ANALYSIS",
+            "-" * 40,
+            ""
+        ])
+        
+        # Domain Analysis
+        if infrastructure_analysis['domains'] > 0:
+            report.extend([
+                f"Domain Name Infrastructure ({infrastructure_analysis['domains']} entities):",
+                ""
+            ])
+            
+            domain_details = self._get_detailed_domain_analysis(nodes, edges)
+            for domain_info in domain_details[:10]:  # Top 10 domains
+                report.extend([
+                    f"• {domain_info['domain']}",
+                    f"  - Type: {domain_info['classification']}",
+                    f"  - Connected IPs: {len(domain_info['ips'])}",
+                    f"  - Certificate Status: {domain_info['cert_status']}",
+                ])
+                
+                if domain_info['security_notes']:
+                    report.extend([
+                        f"  - Security Notes: {', '.join(domain_info['security_notes'])}",
+                    ])
+                report.append("")
+        
+        # IP Address Analysis
+        if infrastructure_analysis['ips'] > 0:
+            report.extend([
+                f"IP Address Infrastructure ({infrastructure_analysis['ips']} entities):",
+                ""
+            ])
+            
+            ip_details = self._get_detailed_ip_analysis(nodes, edges)
+            for ip_info in ip_details[:8]:  # Top 8 IPs
+                report.extend([
+                    f"• {ip_info['ip']} ({ip_info['version']})",
+                    f"  - Associated Domains: {len(ip_info['domains'])}",
+                    f"  - ISP: {ip_info['isp'] or 'Unknown'}",
+                    f"  - Geographic Location: {ip_info['location'] or 'Not determined'}",
+                ])
+                
+                if ip_info['open_ports']:
+                    report.extend([
+                        f"  - Exposed Services: {', '.join(map(str, ip_info['open_ports'][:5]))}"
+                        + (f" (and {len(ip_info['open_ports']) - 5} more)" if len(ip_info['open_ports']) > 5 else ""),
+                    ])
+                report.append("")
+        
+        # === RELATIONSHIP ANALYSIS ===
+        report.extend([
+            "ENTITY RELATIONSHIP ANALYSIS",
+            "-" * 40,
+            ""
+        ])
+        
+        # Network topology insights
+        topology = self._analyze_network_topology(nodes, edges)
+        report.extend([
+            f"Network Topology Assessment:",
+            f"• Central Hubs: {len(topology['hubs'])} entities serve as primary connection points",
+            f"• Isolated Clusters: {len(topology['clusters'])} distinct groupings identified",
+            f"• Relationship Density: {topology['density']:.3f} (0=sparse, 1=fully connected)",
+            f"• Average Path Length: {topology['avg_path_length']:.2f} degrees of separation",
+            ""
+        ])
+        
+        # Key relationships
+        key_relationships = self._identify_key_relationships(edges)
+        if key_relationships:
+            report.extend([
+                "Critical Infrastructure Relationships:",
+                ""
+            ])
+            
+            for rel in key_relationships[:8]:  # Top 8 relationships
+                report.extend([
+                    f"• {rel['source']} → {rel['target']}",
+                    f"  - Relationship: {self._humanize_relationship_type(rel['type'])}",
+                    f"  - Discovery Method: {rel['provider']}",
+                    ""
+                ])
+        
+        # === CERTIFICATE ANALYSIS ===
+        cert_analysis = self._analyze_certificate_infrastructure(nodes)
+        if cert_analysis['total_certs'] > 0:
+            report.extend([
+                "CERTIFICATE INFRASTRUCTURE ANALYSIS",
+                "-" * 40,
+                "",
+                f"Certificate Status Overview:",
+                f"• Total Certificates Analyzed: {cert_analysis['total_certs']}",
+                f"• Valid Certificates: {cert_analysis['valid']}",
+                f"• Expired/Invalid: {cert_analysis['expired']}",
+                f"• Certificate Authorities: {len(cert_analysis['cas'])}",
+                ""
+            ])
+            
+            if cert_analysis['cas']:
+                report.extend([
+                    "Certificate Authority Distribution:",
+                    ""
+                ])
+                for ca, count in cert_analysis['cas'].most_common(5):
+                    report.extend([
+                        f"• {ca}: {count} certificate(s)",
+                    ])
+                report.append("")
+        
+        
+        # === TECHNICAL APPENDIX ===
+        report.extend([
+            "TECHNICAL APPENDIX",
+            "-" * 40,
+            "",
+            "Data Quality Assessment:",
+            f"• Total API Requests: {audit_trail.get('session_metadata', {}).get('total_requests', 0)}",
+            f"• Data Providers Used: {len(audit_trail.get('session_metadata', {}).get('providers_used', []))}",
+        ])
+        
+        correlation_provider = next((p for p in scanner.providers if p.get_name() == 'correlation'), None)
+        correlation_count = len(correlation_provider.correlation_index) if correlation_provider else 0
+        
+        report.extend([
+            "",
+            "Correlation Analysis:",
+            f"• Entity Correlations Identified: {correlation_count}",
+            f"• Cross-Reference Validation: {self._count_cross_validated_relationships(edges)} relationships verified by multiple sources",
+            ""
+        ])
+        
+        # === CONCLUSION ===
+        report.extend([
+            "CONCLUSION",
+            "-" * 40,
+            "",
+            self._generate_conclusion(scanner.current_target, infrastructure_analysis, 
+                                    len(edges)),
+            "",
+            "This analysis was conducted using passive reconnaissance techniques and represents "
+            "the digital infrastructure observable through public data sources at the time of investigation. "
+            "All findings are supported by verifiable technical evidence and documented through "
+            "a complete audit trail maintained for forensic integrity.",
+            "",
+            f"Investigation completed: {now}",
+            f"Report authenticated by: DNScope v{self._get_version()}",
+            "",
+            "=" * 80,
+            "END OF REPORT",
+            "=" * 80
+        ])
+        
+        return "\n".join(report)
+    
+    def _analyze_infrastructure_patterns(self, nodes: List[Dict], edges: List[Dict]) -> Dict[str, Any]:
+        """Analyze infrastructure patterns and classify entities."""
+        analysis = {
+            'domains': len([n for n in nodes if n['type'] == 'domain']),
+            'ips': len([n for n in nodes if n['type'] == 'ip']),
+            'isps': len([n for n in nodes if n['type'] == 'isp']),
+            'cas': len([n for n in nodes if n['type'] == 'ca']),
+            'correlations': len([n for n in nodes if n['type'] == 'correlation_object'])
+        }
+        return analysis
+    
+    def _get_detailed_domain_analysis(self, nodes: List[Dict], edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Generate detailed analysis for each domain."""
+        domain_nodes = [n for n in nodes if n['type'] == 'domain']
+        domain_analysis = []
+        
+        for domain in domain_nodes:
+            # Find connected IPs
+            connected_ips = [e['to'] for e in edges 
+                           if e['from'] == domain['id'] and _is_valid_ip(e['to'])]
+            
+            # Determine classification
+            classification = "Primary Domain"
+            if domain['id'].startswith('www.'):
+                classification = "Web Interface"
+            elif any(subdomain in domain['id'] for subdomain in ['api.', 'mail.', 'smtp.']):
+                classification = "Service Endpoint"
+            elif domain['id'].count('.') > 1:
+                classification = "Subdomain"
+            
+            # Certificate status
+            cert_status = self._determine_certificate_status(domain)
+            
+            # Security notes
+            security_notes = []
+            if cert_status == "Expired/Invalid":
+                security_notes.append("Certificate validation issues")
+            if len(connected_ips) == 0:
+                security_notes.append("No IP resolution found")
+            if len(connected_ips) > 5:
+                security_notes.append("Multiple IP endpoints")
+            
+            domain_edges = [e for e in edges if e['from'] == domain['id']]
+            
+            domain_analysis.append({
+                'domain': domain['id'],
+                'classification': classification,
+                'ips': connected_ips,
+                'cert_status': cert_status,
+                'security_notes': security_notes,
+            })
+        
+        # Sort by number of connections (most connected first)
+        return sorted(domain_analysis, key=lambda x: len(x['ips']), reverse=True)
+    
+    def _get_detailed_ip_analysis(self, nodes: List[Dict], edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Generate detailed analysis for each IP address."""
+        ip_nodes = [n for n in nodes if n['type'] == 'ip']
+        ip_analysis = []
+        
+        for ip in ip_nodes:
+            # Find connected domains
+            connected_domains = [e['from'] for e in edges 
+                               if e['to'] == ip['id'] and _is_valid_domain(e['from'])]
+            
+            # Extract metadata from attributes
+            ip_version = "IPv4"
+            location = None
+            isp = None
+            open_ports = []
+            
+            for attr in ip.get('attributes', []):
+                if attr.get('name') == 'country':
+                    location = attr.get('value')
+                elif attr.get('name') == 'org':
+                    isp = attr.get('value')
+                elif attr.get('name') == 'shodan_open_port':
+                    open_ports.append(attr.get('value'))
+                elif 'ipv6' in str(attr.get('metadata', {})).lower():
+                    ip_version = "IPv6"
+            
+            # Find ISP from relationships
+            if not isp:
+                isp_edges = [e for e in edges if e['from'] == ip['id'] and e['label'].endswith('_isp')]
+                isp = isp_edges[0]['to'] if isp_edges else None
+            
+            ip_analysis.append({
+                'ip': ip['id'],
+                'version': ip_version,
+                'domains': connected_domains,
+                'isp': isp,
+                'location': location,
+                'open_ports': open_ports
+            })
+        
+        # Sort by number of connected domains
+        return sorted(ip_analysis, key=lambda x: len(x['domains']), reverse=True)
+    
+    def _analyze_network_topology(self, nodes: List[Dict], edges: List[Dict]) -> Dict[str, Any]:
+        """Analyze network topology and identify key structural patterns."""
+        if not nodes or not edges:
+            return {'hubs': [], 'clusters': [], 'density': 0, 'avg_path_length': 0}
+        
+        # Create NetworkX graph
+        G = nx.DiGraph()
+        for node in nodes:
+            G.add_node(node['id'])
+        for edge in edges:
+            G.add_edge(edge['from'], edge['to'])
+        
+        # Convert to undirected for certain analyses
+        G_undirected = G.to_undirected()
+        
+        # Identify hubs (nodes with high degree centrality)
+        centrality = nx.degree_centrality(G_undirected)
+        hub_threshold = max(centrality.values()) * 0.7 if centrality else 0
+        hubs = [node for node, cent in centrality.items() if cent >= hub_threshold]
+        
+        # Find connected components (clusters)
+        clusters = list(nx.connected_components(G_undirected))
+        
+        # Calculate density
+        density = nx.density(G_undirected)
+        
+        # Calculate average path length (for largest component)
+        if G_undirected.number_of_nodes() > 1:
+            largest_cc = max(nx.connected_components(G_undirected), key=len)
+            subgraph = G_undirected.subgraph(largest_cc)
+            try:
+                avg_path_length = nx.average_shortest_path_length(subgraph)
+            except:
+                avg_path_length = 0
+        else:
+            avg_path_length = 0
+        
+        return {
+            'hubs': hubs,
+            'clusters': clusters,
+            'density': density,
+            'avg_path_length': avg_path_length
+        }
+    
+    def _identify_key_relationships(self, edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Identify the most significant relationships in the infrastructure."""
+        # Score relationships by type importance
+        relationship_importance = {
+            'dns_a_record': 0.9,
+            'dns_aaaa_record': 0.9,
+            'crtsh_cert_issuer': 0.8,
+            'shodan_isp': 0.8,
+            'crtsh_san_certificate': 0.7,
+            'dns_mx_record': 0.7,
+            'dns_ns_record': 0.7
+        }
+        
+        edges = []
+        for edge in edges:
+            type_weight = relationship_importance.get(edge.get('label', ''), 0.5)
+            
+            edges.append({
+                'source': edge['from'],
+                'target': edge['to'],
+                'type': edge.get('label', ''),
+                'provider': edge.get('source_provider', ''),
+            })
+        
+        # Return top relationships by score
+        return sorted(edges, key=lambda x: x['score'], reverse=True)
+    
+    def _analyze_certificate_infrastructure(self, nodes: List[Dict]) -> Dict[str, Any]:
+        """Analyze certificate infrastructure across all domains."""
+        domain_nodes = [n for n in nodes if n['type'] == 'domain']
+        ca_nodes = [n for n in nodes if n['type'] == 'ca']
+        
+        valid_certs = 0
+        expired_certs = 0
+        total_certs = 0
+        cas = Counter()
+        
+        for domain in domain_nodes:
+            for attr in domain.get('attributes', []):
+                if attr.get('name') == 'cert_is_currently_valid':
+                    total_certs += 1
+                    if attr.get('value') is True:
+                        valid_certs += 1
+                    else:
+                        expired_certs += 1
+                elif attr.get('name') == 'cert_issuer_name':
+                    issuer = attr.get('value')
+                    if issuer:
+                        cas[issuer] += 1
+        
+        return {
+            'total_certs': total_certs,
+            'valid': valid_certs,
+            'expired': expired_certs,
+            'cas': cas
+        }
+    
+    def _has_expired_certificates(self, domain_node: Dict) -> bool:
+        """Check if domain has expired certificates."""
+        for attr in domain_node.get('attributes', []):
+            if (attr.get('name') == 'cert_is_currently_valid' and 
+                attr.get('value') is False):
+                return True
+        return False
+    
+    def _determine_certificate_status(self, domain_node: Dict) -> str:
+        """Determine the certificate status for a domain."""
+        has_valid = False
+        has_expired = False
+        has_any = False
+        
+        for attr in domain_node.get('attributes', []):
+            if attr.get('name') == 'cert_is_currently_valid':
+                has_any = True
+                if attr.get('value') is True:
+                    has_valid = True
+                else:
+                    has_expired = True
+        
+        if not has_any:
+            return "No Certificate Data"
+        elif has_valid and not has_expired:
+            return "Valid"
+        elif has_expired and not has_valid:
+            return "Expired/Invalid"
+        else:
+            return "Mixed Status"
+    
+    def _humanize_relationship_type(self, rel_type: str) -> str:
+        """Convert technical relationship types to human-readable descriptions."""
+        type_map = {
+            'dns_a_record': 'DNS A Record Resolution',
+            'dns_aaaa_record': 'DNS AAAA Record (IPv6) Resolution',
+            'dns_mx_record': 'Email Server (MX) Configuration',
+            'dns_ns_record': 'Name Server Delegation',
+            'dns_cname_record': 'DNS Alias (CNAME) Resolution',
+            'crtsh_cert_issuer': 'SSL Certificate Issuer Relationship',
+            'crtsh_san_certificate': 'Shared SSL Certificate',
+            'shodan_isp': 'Internet Service Provider Assignment',
+            'shodan_a_record': 'IP-to-Domain Resolution (Shodan)',
+            'dns_ptr_record': 'Reverse DNS Resolution'
+        }
+        return type_map.get(rel_type, rel_type.replace('_', ' ').title())
+    
+    def _count_cross_validated_relationships(self, edges: List[Dict]) -> int:
+        """Count relationships verified by multiple providers."""
+        # Group edges by source-target pair
+        edge_pairs = defaultdict(list)
+        for edge in edges:
+            pair_key = f"{edge['from']}->{edge['to']}"
+            edge_pairs[pair_key].append(edge.get('source_provider', ''))
+        
+        # Count pairs with multiple providers
+        cross_validated = 0
+        for pair, providers in edge_pairs.items():
+            if len(set(providers)) > 1:  # Multiple unique providers
+                cross_validated += 1
+        
+        return cross_validated
+    
+    def _generate_security_recommendations(self, infrastructure_analysis: Dict) -> List[str]:
+        """Generate actionable security recommendations."""
+        recommendations = []
+        
+        # Check for complex infrastructure
+        if infrastructure_analysis['ips'] > 10:
+            recommendations.append(
+                "Document and validate the necessity of extensive IP address infrastructure"
+            )
+        
+        if infrastructure_analysis['correlations'] > 5:
+            recommendations.append(
+                "Investigate shared infrastructure components for operational security implications"
+            )
+        
+        if not recommendations:
+            recommendations.append(
+                "Continue monitoring for changes in the identified digital infrastructure"
+            )
+        
+        return recommendations
+    
+    def _generate_conclusion(self, target: str, infrastructure_analysis: Dict, total_relationships: int) -> str:
+        """Generate a professional conclusion for the report."""
+        conclusion_parts = [
+            f"The passive reconnaissance analysis of '{target}' has successfully mapped "
+            f"a digital infrastructure ecosystem consisting of {infrastructure_analysis['domains']} "
+            f"domain names, {infrastructure_analysis['ips']} IP addresses, and "
+            f"{total_relationships} verified inter-entity relationships."
+        ]
+        
+        conclusion_parts.append(
+            "All findings in this report are based on publicly available information and "
+            "passive reconnaissance techniques. The analysis maintains full forensic integrity "
+            "with complete audit trails for all data collection activities."
+        )
+        
+        return " ".join(conclusion_parts)
+    
+    def _count_bidirectional_relationships(self, graph) -> int:
+        """Count bidirectional relationships in the graph."""
+        count = 0
+        for u, v in graph.edges():
+            if graph.has_edge(v, u):
+                count += 1
+        return count // 2  # Each pair counted twice
+    
+    def _identify_hub_nodes(self, graph, nodes: List[Dict]) -> List[str]:
+        """Identify nodes that serve as major hubs in the network."""
+        if not graph.nodes():
+            return []
+        
+        degree_centrality = nx.degree_centrality(graph.to_undirected())
+        threshold = max(degree_centrality.values()) * 0.8 if degree_centrality else 0
+        
+        return [node for node, centrality in degree_centrality.items() 
+                if centrality >= threshold]
+    
+    def _get_version(self) -> str:
+        """Get DNScope version for report authentication."""
+        return "1.0.0-forensic"
+    
+    def export_graph_json(self, graph_manager) -> Dict[str, Any]:
+        """
+        Export complete graph data as a JSON-serializable dictionary.
+        Moved from GraphManager to centralize export functionality.
+        
+        Args:
+            graph_manager: GraphManager instance with graph data
+            
+        Returns:
+            Complete graph data with export metadata
+        """
+        graph_data = nx.node_link_data(graph_manager.graph, edges="edges")
+        
+        return {
+            'export_metadata': {
+                'export_timestamp': datetime.now(timezone.utc).isoformat(),
+                'graph_creation_time': graph_manager.creation_time,
+                'last_modified': graph_manager.last_modified,
+                'total_nodes': graph_manager.get_node_count(),
+                'total_edges': graph_manager.get_edge_count(),
+                'graph_format': 'DNScope_v1_unified_model'
+            },
+            'graph': graph_data,
+            'statistics': graph_manager.get_statistics()
+        }
+    
+    def serialize_to_json(self, data: Dict[str, Any], indent: int = 2) -> str:
+        """
+        Serialize data to JSON with custom handling for non-serializable objects.
+        
+        Args:
+            data: Data to serialize
+            indent: JSON indentation level
+            
+        Returns:
+            JSON string representation
+        """
+        try:
+            return json.dumps(data, indent=indent, cls=CustomJSONEncoder, ensure_ascii=False)
+        except Exception:
+            # Fallback to aggressive cleaning
+            cleaned_data = self._clean_for_json(data)
+            return json.dumps(cleaned_data, indent=indent, ensure_ascii=False)
+    
+    def _clean_for_json(self, obj, max_depth: int = 10, current_depth: int = 0) -> Any:
+        """
+        Recursively clean an object to make it JSON serializable.
+        Handles circular references and problematic object types.
+        
+        Args:
+            obj: Object to clean
+            max_depth: Maximum recursion depth
+            current_depth: Current recursion depth
+            
+        Returns:
+            JSON-serializable object
+        """
+        if current_depth > max_depth:
+            return f"<max_depth_exceeded_{type(obj).__name__}>"
+        
+        if obj is None or isinstance(obj, (bool, int, float, str)):
+            return obj
+        elif isinstance(obj, datetime):
+            return obj.isoformat()
+        elif isinstance(obj, (set, frozenset)):
+            return list(obj)
+        elif isinstance(obj, dict):
+            cleaned = {}
+            for key, value in obj.items():
+                try:
+                    # Ensure key is string
+                    clean_key = str(key) if not isinstance(key, str) else key
+                    cleaned[clean_key] = self._clean_for_json(value, max_depth, current_depth + 1)
+                except Exception:
+                    cleaned[str(key)] = f"<serialization_error_{type(value).__name__}>"
+            return cleaned
+        elif isinstance(obj, (list, tuple)):
+            cleaned = []
+            for item in obj:
+                try:
+                    cleaned.append(self._clean_for_json(item, max_depth, current_depth + 1))
+                except Exception:
+                    cleaned.append(f"<serialization_error_{type(item).__name__}>")
+            return cleaned
+        elif hasattr(obj, '__dict__'):
+            try:
+                return self._clean_for_json(obj.__dict__, max_depth, current_depth + 1)
+            except Exception:
+                return str(obj)
+        elif hasattr(obj, 'value'):
+            # For enum-like objects
+            return obj.value
+        else:
+            return str(obj)
+    
+    def generate_filename(self, target: str, export_type: str, timestamp: Optional[datetime] = None) -> str:
+        """
+        Generate standardized filename for exports.
+        
+        Args:
+            target: Target domain/IP being scanned
+            export_type: Type of export (json, txt, summary)
+            timestamp: Optional timestamp (defaults to now)
+            
+        Returns:
+            Formatted filename with forensic naming convention
+        """
+        if timestamp is None:
+            timestamp = datetime.now(timezone.utc)
+        
+        timestamp_str = timestamp.strftime('%Y%m%d_%H%M%S')
+        safe_target = "".join(c for c in target if c.isalnum() or c in ('-', '_', '.')).rstrip()
+        
+        extension_map = {
+            'json': 'json',
+            'txt': 'txt', 
+            'summary': 'txt',
+            'targets': 'txt'
+        }
+        
+        extension = extension_map.get(export_type, 'txt')
+        return f"DNScope_{export_type}_{safe_target}_{timestamp_str}.{extension}"
+
+
+class CustomJSONEncoder(json.JSONEncoder):
+    """Custom JSON encoder to handle non-serializable objects."""
+    
+    def default(self, obj):
+        if isinstance(obj, datetime):
+            return obj.isoformat()
+        elif isinstance(obj, set):
+            return list(obj)
+        elif isinstance(obj, Decimal):
+            return float(obj)
+        elif hasattr(obj, '__dict__'):
+            # For custom objects, try to serialize their dict representation
+            try:
+                return obj.__dict__
+            except:
+                return str(obj)
+        elif hasattr(obj, 'value') and hasattr(obj, 'name'):
+            # For enum objects
+            return obj.value
+        else:
+            # For any other non-serializable object, convert to string
+            return str(obj)
+
+
+# Global export manager instance
+export_manager = ExportManager()

utils/helpers.py

@@ -0,0 +1,94 @@
+# DNScope-reduced/utils/helpers.py
+
+import ipaddress
+from typing import Union
+
+def _is_valid_domain(domain: str) -> bool:
+    """
+    Basic domain validation.
+
+    Args:
+        domain: Domain string to validate
+
+    Returns:
+        True if domain appears valid
+    """
+    if not domain or len(domain) > 253:
+        return False
+
+    # Check for valid characters and structure
+    parts = domain.split('.')
+    if len(parts) < 2:
+        return False
+
+    for part in parts:
+        if not part or len(part) > 63:
+            return False
+        if not part.replace('-', '').replace('_', '').isalnum():
+            return False
+
+    return True
+
+def _is_valid_ip(ip: str) -> bool:
+    """
+    IP address validation supporting both IPv4 and IPv6.
+
+    Args:
+        ip: IP address string to validate
+
+    Returns:
+        True if IP appears valid (IPv4 or IPv6)
+    """
+    if not ip:
+        return False
+    
+    try:
+        # This handles both IPv4 and IPv6 validation
+        ipaddress.ip_address(ip.strip())
+        return True
+    except (ValueError, AttributeError):
+        return False
+
+def is_valid_target(target: str) -> bool:
+    """
+    Checks if the target is a valid domain or IP address (IPv4/IPv6).
+
+    Args:
+        target: The target string to validate.
+
+    Returns:
+        True if the target is a valid domain or IP, False otherwise.
+    """
+    return _is_valid_domain(target) or _is_valid_ip(target)
+
+def get_ip_version(ip: str) -> Union[int, None]:
+    """
+    Get the IP version (4 or 6) of a valid IP address.
+    
+    Args:
+        ip: IP address string
+        
+    Returns:
+        4 for IPv4, 6 for IPv6, None if invalid
+    """
+    try:
+        addr = ipaddress.ip_address(ip.strip())
+        return addr.version
+    except (ValueError, AttributeError):
+        return None
+
+def normalize_ip(ip: str) -> Union[str, None]:
+    """
+    Normalize an IP address to its canonical form.
+    
+    Args:
+        ip: IP address string
+        
+    Returns:
+        Normalized IP address string, None if invalid
+    """
+    try:
+        addr = ipaddress.ip_address(ip.strip())
+        return str(addr)
+    except (ValueError, AttributeError):
+        return None