iteration on ws implementation

Reviewed-on: mstoeck3/dnsrecon#3

README.md

@@ -1,14 +1,16 @@
 # DNSRecon - Passive Infrastructure Reconnaissance Tool
 
-DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys.
+DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
 
-**Current Status: Phase 2 Implementation**
+**Repo Link:** [https://git.cc24.dev/mstoeck3/dnsrecon](https://git.cc24.dev/mstoeck3/dnsrecon)
 
-  * ✅ Core infrastructure and graph engine
+-----
-  * ✅ Multi-provider support (crt.sh, DNS, Shodan)
+
-  * ✅ Session-based multi-user support
+## Concept and Philosophy
-  * ✅ Real-time web interface with interactive visualization
+
-  * ✅ Forensic logging system and JSON export
+The core philosophy of DNSRecon is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNSRecon will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
+
+For power users who require more in-depth information, DNSRecon can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
 
 -----
 
@@ -20,21 +22,45 @@ DNSRecon is an interactive, passive reconnaissance tool designed to map adversar
   * **Forensic Logging**: A complete audit trail of all reconnaissance activities is maintained.
   * **Confidence Scoring**: Relationships are weighted based on the reliability of the data source.
   * **Session Management**: Supports concurrent user sessions with isolated scanner instances.
+  * **Extensible Provider Architecture**: Easily add new data sources to expand the tool's capabilities.
+  * **Web-Based UI**: An intuitive and interactive web interface for managing scans and visualizing results.
 
 -----
 
-## Installation
+## Technical Architecture
+
+DNSRecon is a web-based application built with a modern technology stack:
+
+  * **Backend**: The backend is a **Flask** application that provides a REST API for the frontend and manages the scanning process.
+  * **Scanning Engine**: The core scanning engine is a multi-threaded Python application that uses a provider-based architecture to query different data sources.
+  * **Session Management**: **Redis** is used for session management, allowing for concurrent user sessions with isolated scanner instances.
+  * **Data Storage**: The application uses an in-memory graph to store and analyze the relationships between different pieces of information. The graph is built using the **NetworkX** library.
+  * **Frontend**: The frontend is a single-page application that uses JavaScript to interact with the backend API and visualize the graph.
+
+-----
+
+## Data Sources
+
+DNSRecon queries the following data sources:
+
+  * **DNS**: Standard DNS lookups (A, AAAA, CNAME, MX, NS, SOA, TXT).
+  * **crt.sh**: A certificate transparency log that provides information about SSL/TLS certificates.
+  * **Shodan**: A search engine for internet-connected devices (requires an API key).
+
+-----
+
+## Installation and Setup
 
 ### Prerequisites
 
   * Python 3.8 or higher
   * A modern web browser with JavaScript enabled
-  * (Recommended) A Linux host for running the application and the optional DNS cache.
+  * A Linux host for running the application
 
 ### 1\. Clone the Project
 
 ```bash
-git clone https://github.com/your-repo/dnsrecon.git
+git clone https://git.cc24.dev/mstoeck3/dnsrecon
 cd dnsrecon
 ```
 
@@ -50,20 +76,18 @@ pip install -r requirements.txt
 
 The `requirements.txt` file contains the following dependencies:
 
-  * Flask\>=2.3.3
+  * Flask
-  * networkx\>=3.1
+  * networkx
-  * requests\>=2.31.0
+  * requests
-  * python-dateutil\>=2.8.2
+  * python-dateutil
-  * Werkzeug\>=2.3.7
+  * Werkzeug
-  * urllib3\>=2.0.0
+  * urllib3
-  * dnspython\>=2.4.2
+  * dnspython
   * gunicorn
   * redis
   * python-dotenv
 
------
+### 3\. Configure the Application
-
-## Configuration
 
 DNSRecon is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
 
@@ -91,6 +115,22 @@ The following environment variables are available for configuration:
 
 -----
 
+## Running the Application
+
+For development, you can run the application using the following command:
+
+```bash
+python app.py
+```
+
+For production, it is recommended to use a more robust server, such as Gunicorn:
+
+```bash
+gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+```
+
+-----
+
 ## Systemd Service
 
 To run DNSRecon as a service that starts automatically on boot, you can use `systemd`.
@@ -145,6 +185,65 @@ sudo systemctl status dnsrecon.service
 
 -----
 
+## Updating the Application
+
+To update the application, you should first pull the latest changes from the git repository. Then, you will need to wipe the Redis database and the local cache to ensure that you are using the latest data.
+
+### 1\. Update the Code
+
+```bash
+git pull
+```
+
+### 2\. Wipe the Redis Database
+
+```bash
+redis-cli FLUSHALL
+```
+
+### 3\. Wipe the Local Cache
+
+```bash
+rm -rf cache/*
+```
+
+### 4\. Restart the Service
+
+```bash
+sudo systemctl restart dnsrecon.service
+```
+
+-----
+
+## Extensibility
+
+DNSRecon is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
+
+  * `get_name()`: Return the name of the provider.
+  * `get_display_name()`: Return a display-friendly name for the provider.
+  * `requires_api_key()`: Return `True` if the provider requires an API key.
+  * `get_eligibility()`: Return a dictionary indicating whether the provider can query domains and/or IPs.
+  * `is_available()`: Return `True` if the provider is available (e.g., if an API key is configured).
+  * `query_domain(domain)`: Query the provider for information about a domain.
+  * `query_ip(ip)`: Query the provider for information about an IP address.
+
+-----
+
+## Unique Capabilities and Limitations
+
+### Unique Capabilities
+
+  * **Graph-Based Analysis**: The use of a graph-based data model allows for a more intuitive and powerful analysis of the relationships between different pieces of information.
+  * **Real-Time Visualization**: The real-time visualization of the graph provides immediate feedback and allows for a more interactive and engaging analysis experience.
+  * **Session Management**: The session management feature allows multiple users to use the application concurrently without interfering with each other's work.
+
+### Limitations
+
+  * **Passive-Only by Default**: While the passive-only approach is a key feature of the tool, it also means that the information it can gather is limited to what is publicly available.
+  * **No Active Scanning**: The tool does not perform any active scanning, such as port scanning or vulnerability scanning.
+
+-----
+
 ## License
 
 This project is licensed under the terms of the **BSD-3-Clause** license.

app.py

@@ -3,9 +3,9 @@
 """
 Flask application entry point for DNSRecon web interface.
 Provides REST API endpoints and serves the web interface with user session support.
+FIXED: Enhanced WebSocket integration with proper connection management.
 """
 
-import json
 import traceback
 from flask import Flask, render_template, request, jsonify, send_file, session
 from datetime import datetime, timezone, timedelta
@@ -13,60 +13,50 @@ import io
 import os
 
 from core.session_manager import session_manager
+from flask_socketio import SocketIO
 from config import config
 from core.graph_manager import NodeType
 from utils.helpers import is_valid_target
+from utils.export_manager import export_manager
 from decimal import Decimal
 
 
 app = Flask(__name__)
+socketio = SocketIO(app, cors_allowed_origins="*")
 app.config['SECRET_KEY'] = config.flask_secret_key
 app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=config.flask_permanent_session_lifetime_hours)
 
 def get_user_scanner():
     """
-    Retrieves the scanner for the current session, or creates a new one if none exists.
+    FIXED: Retrieves the scanner for the current session with proper socketio management.
     """
     current_flask_session_id = session.get('dnsrecon_session_id')
     
     if current_flask_session_id:
         existing_scanner = session_manager.get_session(current_flask_session_id)
         if existing_scanner:
+            # FIXED: Ensure socketio is properly maintained
+            existing_scanner.socketio = socketio
+            print(f"✓ Retrieved existing scanner for session {current_flask_session_id[:8]}... with socketio restored")
             return current_flask_session_id, existing_scanner
     
-    new_session_id = session_manager.create_session()
+    # FIXED: Register socketio connection when creating new session
+    new_session_id = session_manager.create_session(socketio)
     new_scanner = session_manager.get_session(new_session_id)
     
     if not new_scanner:
         raise Exception("Failed to create new scanner session")
     
+    # FIXED: Ensure new scanner has socketio reference and register the connection
+    new_scanner.socketio = socketio
+    session_manager.register_socketio_connection(new_session_id, socketio)
     session['dnsrecon_session_id'] = new_session_id
     session.permanent = True
     
+    print(f"✓ Created new scanner for session {new_session_id[:8]}... with socketio registered")
     return new_session_id, new_scanner
 
-class CustomJSONEncoder(json.JSONEncoder):
+
-    """Custom JSON encoder to handle non-serializable objects."""
-    
-    def default(self, obj):
-        if isinstance(obj, datetime):
-            return obj.isoformat()
-        elif isinstance(obj, set):
-            return list(obj)
-        elif isinstance(obj, Decimal):
-            return float(obj)
-        elif hasattr(obj, '__dict__'):
-            # For custom objects, try to serialize their dict representation
-            try:
-                return obj.__dict__
-            except:
-                return str(obj)
-        elif hasattr(obj, 'value') and hasattr(obj, 'name'):
-            # For enum objects
-            return obj.value
-        else:
-            # For any other non-serializable object, convert to string
-            return str(obj)
 @app.route('/')
 def index():
     """Serve the main web interface."""
@@ -76,7 +66,7 @@ def index():
 @app.route('/api/scan/start', methods=['POST'])
 def start_scan():
     """
-    Starts a new reconnaissance scan.
+    FIXED: Starts a new reconnaissance scan with proper socketio management.
     """
     try:
         data = request.get_json()
@@ -100,12 +90,20 @@ def start_scan():
         if not scanner:
             return jsonify({'success': False, 'error': 'Failed to get scanner instance.'}), 500
         
+        # FIXED: Ensure scanner has socketio reference and is registered
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        print(f"🚀 Starting scan for {target} with socketio enabled and registered")
+        
         success = scanner.start_scan(target, max_depth, clear_graph=clear_graph, force_rescan_target=force_rescan_target)
         
         if success:
+            # Update session with socketio-enabled scanner
+            session_manager.update_session_scanner(user_session_id, scanner)
+            
             return jsonify({
                 'success': True,
-                'message': 'Scan started successfully',
+                'message': 'Reconnaissance scan started successfully',
                 'scan_id': scanner.logger.session_id,
                 'user_session_id': user_session_id
             })
@@ -131,6 +129,10 @@ def stop_scan():
         if not scanner.session_id:
             scanner.session_id = user_session_id
         
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         scanner.stop_scan()
         session_manager.set_stop_signal(user_session_id)
         session_manager.update_scanner_status(user_session_id, 'stopped')
@@ -147,37 +149,83 @@ def stop_scan():
         return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
 
 
-@app.route('/api/scan/status', methods=['GET'])
+@socketio.on('connect')
+def handle_connect():
+    """
+    FIXED: Handle WebSocket connection with proper session management.
+    """
+    print(f'✓ WebSocket client connected: {request.sid}')
+    
+    # Try to restore existing session connection
+    current_flask_session_id = session.get('dnsrecon_session_id')
+    if current_flask_session_id:
+        # Register this socketio connection for the existing session
+        session_manager.register_socketio_connection(current_flask_session_id, socketio)
+        print(f'✓ Registered WebSocket for existing session: {current_flask_session_id[:8]}...')
+    
+    # Immediately send current status to new connection
+    get_scan_status()
+
+
+@socketio.on('disconnect')
+def handle_disconnect():
+    """
+    FIXED: Handle WebSocket disconnection gracefully.
+    """
+    print(f'✗ WebSocket client disconnected: {request.sid}')
+    
+    # Note: We don't immediately remove the socketio connection from session_manager
+    # because the user might reconnect. The cleanup will happen during session cleanup.
+
+
+@socketio.on('get_status')
 def get_scan_status():
-    """Get current scan status."""
+    """
+    FIXED: Get current scan status and emit real-time update with proper error handling.
+    """
     try:
         user_session_id, scanner = get_user_scanner()
         
         if not scanner:
-            return jsonify({
+            status = {
-                'success': True,
+                'status': 'idle', 
-                'status': {
+                'target_domain': None, 
-                    'status': 'idle', 'target_domain': None, 'current_depth': 0,
+                'current_depth': 0,
-                    'max_depth': 0, 'progress_percentage': 0.0,
+                'max_depth': 0, 
-                    'user_session_id': user_session_id
+                'progress_percentage': 0.0,
-                }
+                'user_session_id': user_session_id,
-            })
+                'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
+            }
+            print(f"📡 Emitting idle status for session {user_session_id[:8] if user_session_id else 'none'}...")
+        else:
+            if not scanner.session_id:
+                scanner.session_id = user_session_id
+            
+            # FIXED: Ensure scanner has socketio reference for future updates
+            scanner.socketio = socketio
+            session_manager.register_socketio_connection(user_session_id, socketio)
+            
+            status = scanner.get_scan_status()
+            status['user_session_id'] = user_session_id
+            
+            print(f"📡 Emitting status update: {status['status']} - "
+                  f"Nodes: {len(status.get('graph', {}).get('nodes', []))}, "
+                  f"Edges: {len(status.get('graph', {}).get('edges', []))}")
+            
+            # Update session with socketio-enabled scanner
+            session_manager.update_session_scanner(user_session_id, scanner)
         
-        if not scanner.session_id:
+        socketio.emit('scan_update', status)
-            scanner.session_id = user_session_id
-        
-        status = scanner.get_scan_status()
-        status['user_session_id'] = user_session_id
-        
-        return jsonify({'success': True, 'status': status})
     
     except Exception as e:
         traceback.print_exc()
-        return jsonify({
+        error_status = {
-            'success': False, 'error': f'Internal server error: {str(e)}',
+            'status': 'error', 
-            'fallback_status': {'status': 'error', 'progress_percentage': 0.0}
+            'message': 'Failed to get status',
-        }), 500
+            'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
-
+        }
+        print(f"⚠️ Error getting status, emitting error status")
+        socketio.emit('scan_update', error_status)
 
 
 @app.route('/api/graph', methods=['GET'])
@@ -194,6 +242,10 @@ def get_graph_data():
         if not scanner:
             return jsonify({'success': True, 'graph': empty_graph, 'user_session_id': user_session_id})
         
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         graph_data = scanner.get_graph_data() or empty_graph
         
         return jsonify({'success': True, 'graph': graph_data, 'user_session_id': user_session_id})
@@ -220,6 +272,10 @@ def extract_from_large_entity():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         success = scanner.extract_node_from_large_entity(large_entity_id, node_id)
         
         if success:
@@ -240,6 +296,10 @@ def delete_graph_node(node_id):
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
+        # FIXED: Ensure scanner has socketio reference  
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         success = scanner.graph.remove_node(node_id)
         
         if success:
@@ -265,6 +325,10 @@ def revert_graph_action():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+
         action_type = data['type']
         action_data = data['data']
         
@@ -309,41 +373,34 @@ def export_results():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
         
-        # Get export data with error handling
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
+        # Get export data using the new export manager
         try:
-            results = scanner.export_results()
+            results = export_manager.export_scan_results(scanner)
         except Exception as e:
             return jsonify({'success': False, 'error': f'Failed to gather export data: {str(e)}'}), 500
         
-        # Add export metadata
+        # Add user session metadata
-        results['export_metadata'] = {
+        results['export_metadata']['user_session_id'] = user_session_id
-            'user_session_id': user_session_id,
+        results['export_metadata']['forensic_integrity'] = 'maintained'
-            'export_timestamp': datetime.now(timezone.utc).isoformat(),
-            'export_version': '1.0.0',
-            'forensic_integrity': 'maintained'
-        }
         
-        # Generate filename with forensic naming convention
+        # Generate filename
-        timestamp = datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')
+        filename = export_manager.generate_filename(
-        target = scanner.current_target or 'unknown'
+            target=scanner.current_target or 'unknown',
-        # Sanitize target for filename
+            export_type='json'
-        safe_target = "".join(c for c in target if c.isalnum() or c in ('-', '_', '.')).rstrip()
+        )
-        filename = f"dnsrecon_{safe_target}_{timestamp}.json"
         
-        # Serialize with custom encoder and error handling
+        # Serialize with export manager
         try:
-            json_data = json.dumps(results, indent=2, cls=CustomJSONEncoder, ensure_ascii=False)
+            json_data = export_manager.serialize_to_json(results)
         except Exception as e:
-            # If custom encoder fails, try a more aggressive approach
+            return jsonify({
-            try:
+                'success': False, 
-                # Convert problematic objects to strings recursively
+                'error': f'JSON serialization failed: {str(e)}'
-                cleaned_results = _clean_for_json(results)
+            }), 500
-                json_data = json.dumps(cleaned_results, indent=2, ensure_ascii=False)
-            except Exception as e2:
-                return jsonify({
-                    'success': False, 
-                    'error': f'JSON serialization failed: {str(e2)}'
-                }), 500
         
         # Create file object
         file_obj = io.BytesIO(json_data.encode('utf-8'))
@@ -363,48 +420,72 @@ def export_results():
             'error_type': type(e).__name__
         }), 500
 
-def _clean_for_json(obj, max_depth=10, current_depth=0):
+@app.route('/api/export/targets', methods=['GET'])
-    """
+def export_targets():
-    Recursively clean an object to make it JSON serializable.
+    """Export all discovered targets as a TXT file."""
-    Handles circular references and problematic object types.
+    try:
-    """
+        user_session_id, scanner = get_user_scanner()
-    if current_depth > max_depth:
+        if not scanner:
-        return f"<max_depth_exceeded_{type(obj).__name__}>"
+            return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
-    
+
-    if obj is None or isinstance(obj, (bool, int, float, str)):
+        # FIXED: Ensure scanner has socketio reference
-        return obj
+        scanner.socketio = socketio
-    elif isinstance(obj, datetime):
+        session_manager.register_socketio_connection(user_session_id, socketio)
-        return obj.isoformat()
+
-    elif isinstance(obj, (set, frozenset)):
+        # Use export manager for targets export
-        return list(obj)
+        targets_txt = export_manager.export_targets_list(scanner)
-    elif isinstance(obj, dict):
+        
-        cleaned = {}
+        # Generate filename using export manager
-        for key, value in obj.items():
+        filename = export_manager.generate_filename(
-            try:
+            target=scanner.current_target or 'unknown',
-                # Ensure key is string
+            export_type='targets'
-                clean_key = str(key) if not isinstance(key, str) else key
+        )
-                cleaned[clean_key] = _clean_for_json(value, max_depth, current_depth + 1)
+        
-            except Exception:
+        file_obj = io.BytesIO(targets_txt.encode('utf-8'))
-                cleaned[str(key)] = f"<serialization_error_{type(value).__name__}>"
+        
-        return cleaned
+        return send_file(
-    elif isinstance(obj, (list, tuple)):
+            file_obj,
-        cleaned = []
+            as_attachment=True,
-        for item in obj:
+            download_name=filename,
-            try:
+            mimetype='text/plain'
-                cleaned.append(_clean_for_json(item, max_depth, current_depth + 1))
+        )
-            except Exception:
+    except Exception as e:
-                cleaned.append(f"<serialization_error_{type(item).__name__}>")
+        traceback.print_exc()
-        return cleaned
+        return jsonify({'success': False, 'error': f'Export failed: {str(e)}'}), 500
-    elif hasattr(obj, '__dict__'):
+
-        try:
+
-            return _clean_for_json(obj.__dict__, max_depth, current_depth + 1)
+@app.route('/api/export/summary', methods=['GET'])
-        except Exception:
+def export_summary():
-            return str(obj)
+    """Export an executive summary as a TXT file."""
-    elif hasattr(obj, 'value'):
+    try:
-        # For enum-like objects
+        user_session_id, scanner = get_user_scanner()
-        return obj.value
+        if not scanner:
-    else:
+            return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
-        return str(obj)
+
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+
+        # Use export manager for summary generation
+        summary_txt = export_manager.generate_executive_summary(scanner)
+
+        # Generate filename using export manager
+        filename = export_manager.generate_filename(
+            target=scanner.current_target or 'unknown',
+            export_type='summary'
+        )
+        
+        file_obj = io.BytesIO(summary_txt.encode('utf-8'))
+        
+        return send_file(
+            file_obj,
+            as_attachment=True,
+            download_name=filename,
+            mimetype='text/plain'
+        )
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Export failed: {str(e)}'}), 500
 
 @app.route('/api/config/api-keys', methods=['POST'])
 def set_api_keys():
@@ -417,6 +498,10 @@ def set_api_keys():
         user_session_id, scanner = get_user_scanner()
         session_config = scanner.config
         
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         updated_providers = []
         
         for provider_name, api_key in data.items():
@@ -449,6 +534,10 @@ def get_providers():
         user_session_id, scanner = get_user_scanner()
         base_provider_info = scanner.get_provider_info()
         
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         # Enhance provider info with API key source information
         enhanced_provider_info = {}
         
@@ -513,6 +602,10 @@ def configure_providers():
         user_session_id, scanner = get_user_scanner()
         session_config = scanner.config
         
+        # FIXED: Ensure scanner has socketio reference
+        scanner.socketio = socketio
+        session_manager.register_socketio_connection(user_session_id, socketio)
+        
         updated_providers = []
         
         for provider_name, settings in data.items():
@@ -541,7 +634,6 @@ def configure_providers():
         return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
 
 
-
 @app.errorhandler(404)
 def not_found(error):
     """Handle 404 errors."""
@@ -557,9 +649,9 @@ def internal_error(error):
 
 if __name__ == '__main__':
     config.load_from_env()
-    app.run(
+    print("🚀 Starting DNSRecon with enhanced WebSocket support...")
-        host=config.flask_host,
+    print(f"   Host: {config.flask_host}")
-        port=config.flask_port,
+    print(f"   Port: {config.flask_port}")
-        debug=config.flask_debug,
+    print(f"   Debug: {config.flask_debug}")
-        threaded=True
+    print("   WebSocket: Enhanced connection management enabled")
-    )
+    socketio.run(app, host=config.flask_host, port=config.flask_port, debug=config.flask_debug)

config.py

@@ -33,14 +33,16 @@ class Config:
         self.rate_limits = {
             'crtsh': 5,
             'shodan': 60,
-            'dns': 100
+            'dns': 100,
+            'correlation': 0 # Set to 0 to make sure correlations run last
         }
         
         # --- Provider Settings ---
         self.enabled_providers = {
             'crtsh': True,
             'dns': True,
-            'shodan': False
+            'shodan': False,
+            'correlation': True # Enable the new provider by default
         }
         
         # --- Logging ---

core/graph_manager.py

@@ -4,7 +4,7 @@
 Graph data model for DNSRecon using NetworkX.
 Manages in-memory graph storage with confidence scoring and forensic metadata.
 Now fully compatible with the unified ProviderResult data model.
-UPDATED: Fixed correlation exclusion keys to match actual attribute names.
+FIXED: Added proper pickle support to prevent weakref serialization errors.
 """
 import re
 from datetime import datetime, timezone
@@ -32,6 +32,7 @@ class GraphManager:
     Thread-safe graph manager for DNSRecon infrastructure mapping.
     Uses NetworkX for in-memory graph storage with confidence scoring.
     Compatible with unified ProviderResult data model.
+    FIXED: Added proper pickle support to handle NetworkX graph serialization.
     """
 
     def __init__(self):
@@ -39,270 +40,57 @@ class GraphManager:
         self.graph = nx.DiGraph()
         self.creation_time = datetime.now(timezone.utc).isoformat()
         self.last_modified = self.creation_time
-        self.correlation_index = {}
-        # Compile regex for date filtering for efficiency
-        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
-        
-        # FIXED: Exclude cert_issuer_name since we already create proper CA relationships
-        self.EXCLUDED_KEYS = [
-            # Certificate metadata that creates noise or has dedicated node types
-            'cert_source',                    # Always 'crtsh' for crtsh provider
-            'cert_common_name',
-            'cert_validity_period_days',      # Numerical, not useful for correlation
-            'cert_issuer_name',              # FIXED: Has dedicated CA nodes, don't correlate
-            #'cert_certificate_id',            # Unique per certificate
-            #'cert_serial_number',            # Unique per certificate
-            'cert_entry_timestamp',          # Timestamp, filtered by date regex anyway
-            'cert_not_before',               # Date, filtered by date regex anyway
-            'cert_not_after',                # Date, filtered by date regex anyway
-            # DNS metadata that creates noise
-            'dns_ttl',                       # TTL values are not meaningful for correlation
-            # Shodan metadata that might create noise
-            'timestamp',                     # Generic timestamp fields
-            'last_update',                   # Generic timestamp fields
-            #'org',                          # Too generic, causes false correlations
-            #'isp',                          # Too generic, causes false correlations
-            # Generic noisy attributes
-            'updated_timestamp',             # Any timestamp field
-            'discovery_timestamp',           # Any timestamp field
-            'query_timestamp',               # Any timestamp field
-        ]
 
     def __getstate__(self):
-        """Prepare GraphManager for pickling, excluding compiled regex."""
+        """Prepare GraphManager for pickling by converting NetworkX graph to serializable format."""
         state = self.__dict__.copy()
-        # Compiled regex patterns are not always picklable
+        
-        if 'date_pattern' in state:
+        # Convert NetworkX graph to a serializable format
-            del state['date_pattern']
+        if hasattr(self, 'graph') and self.graph:
+            # Extract all nodes with their data
+            nodes_data = {}
+            for node_id, attrs in self.graph.nodes(data=True):
+                nodes_data[node_id] = dict(attrs)
+            
+            # Extract all edges with their data
+            edges_data = []
+            for source, target, attrs in self.graph.edges(data=True):
+                edges_data.append({
+                    'source': source,
+                    'target': target,
+                    'attributes': dict(attrs)
+                })
+            
+            # Replace the NetworkX graph with serializable data
+            state['_graph_nodes'] = nodes_data
+            state['_graph_edges'] = edges_data
+            del state['graph']
+        
         return state
 
     def __setstate__(self, state):
-        """Restore GraphManager state and recompile regex."""
+        """Restore GraphManager after unpickling by reconstructing NetworkX graph."""
+        # Restore basic attributes
         self.__dict__.update(state)
-        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
-
-    def process_correlations_for_node(self, node_id: str):
-        """
-        UPDATED: Process correlations for a given node with enhanced tracking.
-        Now properly tracks which attribute/provider created each correlation.
-        """
-        if not self.graph.has_node(node_id):
-            return
-
-        node_attributes = self.graph.nodes[node_id].get('attributes', [])
         
-        # Process each attribute for potential correlations
+        # Reconstruct NetworkX graph from serializable data
-        for attr in node_attributes:
+        self.graph = nx.DiGraph()
-            attr_name = attr.get('name')
-            attr_value = attr.get('value')
-            attr_provider = attr.get('provider', 'unknown')
-
-            # IMPROVED: More comprehensive exclusion logic
-            should_exclude = (
-                # Check against excluded keys (exact match or substring)
-                any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS) or
-                # Invalid value types
-                not isinstance(attr_value, (str, int, float, bool)) or 
-                attr_value is None or
-                # Boolean values are not useful for correlation
-                isinstance(attr_value, bool) or
-                # String values that are too short or are dates
-                (isinstance(attr_value, str) and (
-                    len(attr_value) < 4 or 
-                    self.date_pattern.match(attr_value) or
-                    # Exclude common generic values that create noise
-                    attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']
-                )) or
-                # Numerical values that are likely to be unique identifiers
-                (isinstance(attr_value, (int, float)) and (
-                    attr_value == 0 or  # Zero values are not meaningful
-                    attr_value == 1 or  # One values are too common
-                    abs(attr_value) > 1000000  # Very large numbers are likely IDs
-                ))
-            )
-
-            if should_exclude:
-                continue
-
-            # Initialize correlation tracking for this value
-            if attr_value not in self.correlation_index:
-                self.correlation_index[attr_value] = {
-                    'nodes': set(),
-                    'sources': []  # Track which provider/attribute combinations contributed
-                }
-
-            # Add this node and source information
-            self.correlation_index[attr_value]['nodes'].add(node_id)
-            
-            # Track the source of this correlation value
-            source_info = {
-                'node_id': node_id,
-                'provider': attr_provider,
-                'attribute': attr_name,
-                'path': f"{attr_provider}_{attr_name}"
-            }
-            
-            # Add source if not already present (avoid duplicates)
-            existing_sources = [s for s in self.correlation_index[attr_value]['sources'] 
-                              if s['node_id'] == node_id and s['path'] == source_info['path']]
-            if not existing_sources:
-                self.correlation_index[attr_value]['sources'].append(source_info)
-
-            # Create correlation node if we have multiple nodes with this value
-            if len(self.correlation_index[attr_value]['nodes']) > 1:
-                self._create_enhanced_correlation_node_and_edges(attr_value, self.correlation_index[attr_value])
-
-    def _create_enhanced_correlation_node_and_edges(self, value, correlation_data):
-        """
-        UPDATED: Create correlation node and edges with raw provider data (no formatting).
-        """
-        correlation_node_id = f"corr_{hash(str(value)) & 0x7FFFFFFF}"
-        nodes = correlation_data['nodes']
-        sources = correlation_data['sources']
         
-        # Create or update correlation node
+        # Restore nodes
-        if not self.graph.has_node(correlation_node_id):
+        if hasattr(self, '_graph_nodes'):
-            # Use raw provider/attribute data - no formatting
+            for node_id, attrs in self._graph_nodes.items():
-            provider_counts = {}
+                self.graph.add_node(node_id, **attrs)
-            for source in sources:
+            del self._graph_nodes
-                # Keep original provider and attribute names
+        
-                key = f"{source['provider']}_{source['attribute']}"
+        # Restore edges
-                provider_counts[key] = provider_counts.get(key, 0) + 1
+        if hasattr(self, '_graph_edges'):
-            
+            for edge_data in self._graph_edges:
-            # Use the most common provider/attribute as the primary label (raw)
+                self.graph.add_edge(
-            primary_source = max(provider_counts.items(), key=lambda x: x[1])[0] if provider_counts else "unknown_correlation"
+                    edge_data['source'], 
-            
+                    edge_data['target'], 
-            metadata = {
+                    **edge_data['attributes']
-                'value': value,
-                'correlated_nodes': list(nodes),
-                'sources': sources,
-                'primary_source': primary_source,
-                'correlation_count': len(nodes)
-            }
-            
-            self.add_node(correlation_node_id, NodeType.CORRELATION_OBJECT, metadata=metadata)
-            #print(f"Created correlation node {correlation_node_id} for value '{value}' with {len(nodes)} nodes")
-
-        # Create edges from each node to the correlation node
-        for source in sources:
-            node_id = source['node_id']
-            provider = source['provider']
-            attribute = source['attribute']
-            
-            if self.graph.has_node(node_id) and not self.graph.has_edge(node_id, correlation_node_id):
-                # Format relationship label as "corr_provider_attribute"
-                relationship_label = f"corr_{provider}_{attribute}"
-                
-                self.add_edge(
-                    source_id=node_id,
-                    target_id=correlation_node_id,
-                    relationship_type=relationship_label,
-                    confidence_score=0.9,
-                    source_provider=provider,
-                    raw_data={
-                        'correlation_value': value,
-                        'original_attribute': attribute,
-                        'correlation_type': 'attribute_matching'
-                    }
                 )
-                
+            del self._graph_edges
-                #print(f"Added correlation edge: {node_id} -> {correlation_node_id} ({relationship_label})")
-
-
-    def _has_direct_edge_bidirectional(self, node_a: str, node_b: str) -> bool:
-        """
-        Check if there's a direct edge between two nodes in either direction.
-        Returns True if node_aâ†'node_b OR node_bâ†'node_a exists.
-        """
-        return (self.graph.has_edge(node_a, node_b) or 
-                self.graph.has_edge(node_b, node_a))
-
-    def _correlation_value_matches_existing_node(self, correlation_value: str) -> bool:
-        """
-        Check if correlation value contains any existing node ID as substring.
-        Returns True if match found (correlation node should NOT be created).
-        """
-        correlation_str = str(correlation_value).lower()
-        
-        # Check against all existing nodes
-        for existing_node_id in self.graph.nodes():
-            if existing_node_id.lower() in correlation_str:
-                return True
-        
-        return False
-
-    def _find_correlation_nodes_with_same_pattern(self, node_set: set) -> List[str]:
-        """
-        Find existing correlation nodes that have the exact same pattern of connected nodes.
-        Returns list of correlation node IDs with matching patterns.
-        """
-        correlation_nodes = self.get_nodes_by_type(NodeType.CORRELATION_OBJECT)
-        matching_nodes = []
-        
-        for corr_node_id in correlation_nodes:
-            # Get all nodes connected to this correlation node
-            connected_nodes = set()
-            
-            # Add all predecessors (nodes pointing TO the correlation node)
-            connected_nodes.update(self.graph.predecessors(corr_node_id))
-            
-            # Add all successors (nodes pointed TO by the correlation node)  
-            connected_nodes.update(self.graph.successors(corr_node_id))
-            
-            # Check if the pattern matches exactly
-            if connected_nodes == node_set:
-                matching_nodes.append(corr_node_id)
-        
-        return matching_nodes
-
-    def _merge_correlation_values(self, target_node_id: str, new_value: Any, corr_data: Dict) -> None:
-        """
-        Merge a new correlation value into an existing correlation node.
-        Uses same logic as large entity merging.
-        """
-        if not self.graph.has_node(target_node_id):
-            return
-            
-        target_metadata = self.graph.nodes[target_node_id]['metadata']
-        
-        # Get existing values (ensure it's a list)
-        existing_values = target_metadata.get('values', [])
-        if not isinstance(existing_values, list):
-            existing_values = [existing_values]
-        
-        # Add new value if not already present
-        if new_value not in existing_values:
-            existing_values.append(new_value)
-            
-        # Merge sources
-        existing_sources = target_metadata.get('sources', [])
-        new_sources = corr_data.get('sources', [])
-        
-        # Create set of unique sources based on (node_id, path) tuples
-        source_set = set()
-        for source in existing_sources + new_sources:
-            source_tuple = (source['node_id'], source.get('path', ''))
-            source_set.add(source_tuple)
-        
-        # Convert back to list of dictionaries
-        merged_sources = [{'node_id': nid, 'path': path} for nid, path in source_set]
-        
-        # Update metadata
-        target_metadata.update({
-            'values': existing_values,
-            'sources': merged_sources,
-            'correlated_nodes': list(set(target_metadata.get('correlated_nodes', []) + corr_data.get('nodes', []))),
-            'merge_count': len(existing_values),
-            'last_merge_timestamp': datetime.now(timezone.utc).isoformat()
-        })
-        
-        # Update description to reflect merged nature
-        value_count = len(existing_values)
-        node_count = len(target_metadata['correlated_nodes'])
-        self.graph.nodes[target_node_id]['description'] = (
-            f"Correlation container with {value_count} merged values "
-            f"across {node_count} nodes"
-        )
         
     def add_node(self, node_id: str, node_type: NodeType, attributes: Optional[List[Dict[str, Any]]] = None,
                 description: str = "", metadata: Optional[Dict[str, Any]] = None) -> bool:
@@ -377,36 +165,6 @@ class GraphManager:
         self.last_modified = datetime.now(timezone.utc).isoformat()
         return True
 
-    def extract_node_from_large_entity(self, large_entity_id: str, node_id_to_extract: str) -> bool:
-        """
-        Removes a node from a large entity's internal lists and updates its count.
-        This prepares the large entity for the node's promotion to a regular node.
-        """
-        if not self.graph.has_node(large_entity_id):
-            return False
-            
-        node_data = self.graph.nodes[large_entity_id]
-        attributes = node_data.get('attributes', [])
-        
-        # Find the 'nodes' attribute dictionary in the list
-        nodes_attr = next((attr for attr in attributes if attr.get('name') == 'nodes'), None)
-        
-        # Remove from the list of member nodes
-        if nodes_attr and 'value' in nodes_attr and isinstance(nodes_attr['value'], list) and node_id_to_extract in nodes_attr['value']:
-            nodes_attr['value'].remove(node_id_to_extract)
-            
-            # Find the 'count' attribute and update it
-            count_attr = next((attr for attr in attributes if attr.get('name') == 'count'), None)
-            if count_attr:
-                count_attr['value'] = len(nodes_attr['value'])
-        else:
-            # This can happen if the node was already extracted, which is not an error.
-            print(f"Warning: Node {node_id_to_extract} not found in the 'nodes' list of {large_entity_id}.")
-            return True # Proceed as if successful
-            
-        self.last_modified = datetime.now(timezone.utc).isoformat()
-        return True
-
     def remove_node(self, node_id: str) -> bool:
         """Remove a node and its connected edges from the graph."""
         if not self.graph.has_node(node_id):
@@ -414,29 +172,7 @@ class GraphManager:
 
         # Remove node from the graph (NetworkX handles removing connected edges)
         self.graph.remove_node(node_id)
-
-        # Clean up the correlation index
-        keys_to_delete = []
-        for value, data in self.correlation_index.items():
-            if isinstance(data, dict) and 'nodes' in data:
-                # Updated correlation structure
-                if node_id in data['nodes']:
-                    data['nodes'].discard(node_id)
-                    # Remove sources for this node
-                    data['sources'] = [s for s in data['sources'] if s['node_id'] != node_id]
-                if not data['nodes']:  # If no other nodes are associated, remove it
-                    keys_to_delete.append(value)
-            else:
-                # Legacy correlation structure (fallback)
-                if isinstance(data, set) and node_id in data:
-                    data.discard(node_id)
-                if not data:
-                    keys_to_delete.append(value)
         
-        for key in keys_to_delete:
-            if key in self.correlation_index:
-                del self.correlation_index[key]
-
         self.last_modified = datetime.now(timezone.utc).isoformat()
         return True
 
@@ -471,7 +207,8 @@ class GraphManager:
                 'attributes': attrs.get('attributes', []), # Raw attributes list
                 'description': attrs.get('description', ''),
                 'metadata': attrs.get('metadata', {}),
-                'added_timestamp': attrs.get('added_timestamp')
+                'added_timestamp': attrs.get('added_timestamp'),
+                'max_depth_reached': attrs.get('metadata', {}).get('max_depth_reached', False)
             }
             
             # Add incoming and outgoing edges to node data
@@ -502,22 +239,6 @@ class GraphManager:
             'statistics': self.get_statistics()['basic_metrics']
         }
 
-    def export_json(self) -> Dict[str, Any]:
-        """Export complete graph data as a JSON-serializable dictionary."""
-        graph_data = nx.node_link_data(self.graph, edges="edges")
-        return {
-            'export_metadata': {
-                'export_timestamp': datetime.now(timezone.utc).isoformat(),
-                'graph_creation_time': self.creation_time,
-                'last_modified': self.last_modified,
-                'total_nodes': self.get_node_count(),
-                'total_edges': self.get_edge_count(),
-                'graph_format': 'dnsrecon_v1_unified_model'
-            },
-            'graph': graph_data,
-            'statistics': self.get_statistics()
-        }
-
     def _get_confidence_distribution(self) -> Dict[str, int]:
         """Get distribution of edge confidence scores with empty graph handling."""
         distribution = {'high': 0, 'medium': 0, 'low': 0}
@@ -576,8 +297,7 @@ class GraphManager:
         return stats
 
     def clear(self) -> None:
-        """Clear all nodes, edges, and indices from the graph."""
+        """Clear all nodes and edges from the graph."""
         self.graph.clear()
-        self.correlation_index.clear()
         self.creation_time = datetime.now(timezone.utc).isoformat()
         self.last_modified = self.creation_time

core/logger.py

@@ -40,6 +40,7 @@ class ForensicLogger:
     """
     Thread-safe forensic logging system for DNSRecon.
     Maintains detailed audit trail of all reconnaissance activities.
+    FIXED: Enhanced pickle support to prevent weakref issues in logging handlers.
     """
     
     def __init__(self, session_id: str = ""):
@@ -65,45 +66,74 @@ class ForensicLogger:
             'target_domains': set()
         }
         
-        # Configure standard logger
+        # Configure standard logger with simple setup to avoid weakrefs
         self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
         self.logger.setLevel(logging.INFO)
         
-        # Create formatter for structured logging
+        # Create minimal formatter
         formatter = logging.Formatter(
             '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         )
         
-        # Add console handler if not already present
+        # Add console handler only if not already present (avoid duplicate handlers)
         if not self.logger.handlers:
             console_handler = logging.StreamHandler()
             console_handler.setFormatter(formatter)
             self.logger.addHandler(console_handler)
 
     def __getstate__(self):
-        """Prepare ForensicLogger for pickling by excluding unpicklable objects."""
+        """
+        FIXED: Prepare ForensicLogger for pickling by excluding problematic objects.
+        """
         state = self.__dict__.copy()
-        # Remove the unpickleable 'logger' attribute
+        
-        if 'logger' in state:
+        # Remove potentially unpickleable attributes that may contain weakrefs
-            del state['logger']
+        unpicklable_attrs = ['logger', 'lock']
-        if 'lock' in state:
+        for attr in unpicklable_attrs:
-            del state['lock']
+            if attr in state:
+                del state[attr]
+        
+        # Convert sets to lists for JSON serialization compatibility
+        if 'session_metadata' in state:
+            metadata = state['session_metadata'].copy()
+            if 'providers_used' in metadata and isinstance(metadata['providers_used'], set):
+                metadata['providers_used'] = list(metadata['providers_used'])
+            if 'target_domains' in metadata and isinstance(metadata['target_domains'], set):
+                metadata['target_domains'] = list(metadata['target_domains'])
+            state['session_metadata'] = metadata
+        
         return state
 
     def __setstate__(self, state):
-        """Restore ForensicLogger after unpickling by reconstructing logger."""
+        """
+        FIXED: Restore ForensicLogger after unpickling by reconstructing components.
+        """
         self.__dict__.update(state)
-        # Re-initialize the 'logger' attribute
+        
+        # Re-initialize threading lock
+        self.lock = threading.Lock()
+        
+        # Re-initialize logger with minimal setup
         self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
         self.logger.setLevel(logging.INFO)
+        
         formatter = logging.Formatter(
             '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         )
+        
+        # Only add handler if not already present
         if not self.logger.handlers:
             console_handler = logging.StreamHandler()
             console_handler.setFormatter(formatter)
             self.logger.addHandler(console_handler)
-        self.lock = threading.Lock()
+        
+        # Convert lists back to sets if needed
+        if 'session_metadata' in self.__dict__:
+            metadata = self.session_metadata
+            if 'providers_used' in metadata and isinstance(metadata['providers_used'], list):
+                metadata['providers_used'] = set(metadata['providers_used'])
+            if 'target_domains' in metadata and isinstance(metadata['target_domains'], list):
+                metadata['target_domains'] = set(metadata['target_domains'])
 
     def _generate_session_id(self) -> str:
         """Generate unique session identifier."""
@@ -143,18 +173,23 @@ class ForensicLogger:
             discovery_context=discovery_context
         )
         
-        self.api_requests.append(api_request)
+        with self.lock:
-        self.session_metadata['total_requests'] += 1
+            self.api_requests.append(api_request)
-        self.session_metadata['providers_used'].add(provider)
+            self.session_metadata['total_requests'] += 1
+            self.session_metadata['providers_used'].add(provider)
+            
+            if target_indicator:
+                self.session_metadata['target_domains'].add(target_indicator)
         
-        if target_indicator:
+        # Log to standard logger with error handling
-            self.session_metadata['target_domains'].add(target_indicator)
+        try:
-        
+            if error:
-        # Log to standard logger
+                self.logger.error(f"API Request Failed - {provider}: {url}")
-        if error:
+            else:
-            self.logger.error(f"API Request Failed.")
+                self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
-        else:
+        except Exception:
-            self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
+            # If logging fails, continue without breaking the application
+            pass
     
     def log_relationship_discovery(self, source_node: str, target_node: str,
                                  relationship_type: str, confidence_score: float,
@@ -183,29 +218,44 @@ class ForensicLogger:
             discovery_method=discovery_method
         )
         
-        self.relationships.append(relationship)
+        with self.lock:
-        self.session_metadata['total_relationships'] += 1
+            self.relationships.append(relationship)
+            self.session_metadata['total_relationships'] += 1
         
-        self.logger.info(
+        # Log to standard logger with error handling
-            f"Relationship Discovered - {source_node} -> {target_node} "
+        try:
-            f"({relationship_type}) - Confidence: {confidence_score:.2f} - Provider: {provider}"
+            self.logger.info(
-        )
+                f"Relationship Discovered - {source_node} -> {target_node} "
+                f"({relationship_type}) - Confidence: {confidence_score:.2f} - Provider: {provider}"
+            )
+        except Exception:
+            # If logging fails, continue without breaking the application
+            pass
     
     def log_scan_start(self, target_domain: str, recursion_depth: int, 
                       enabled_providers: List[str]) -> None:
         """Log the start of a reconnaissance scan."""
-        self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
+        try:
-        self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
+            self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
-        
+            self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
-        self.session_metadata['target_domains'].update(target_domain)
+            
+            with self.lock:
+                self.session_metadata['target_domains'].add(target_domain)
+        except Exception:
+            pass
     
     def log_scan_complete(self) -> None:
         """Log the completion of a reconnaissance scan."""
-        self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
+        with self.lock:
-        self.session_metadata['providers_used'] = list(self.session_metadata['providers_used'])
+            self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
-        self.session_metadata['target_domains'] = list(self.session_metadata['target_domains'])
+            # Convert sets to lists for serialization
+            self.session_metadata['providers_used'] = list(self.session_metadata['providers_used'])
+            self.session_metadata['target_domains'] = list(self.session_metadata['target_domains'])
         
-        self.logger.info(f"Scan Complete - Session: {self.session_id}")
+        try:
+            self.logger.info(f"Scan Complete - Session: {self.session_id}")
+        except Exception:
+            pass
     
     def export_audit_trail(self) -> Dict[str, Any]:
         """
@@ -214,12 +264,13 @@ class ForensicLogger:
         Returns:
             Dictionary containing complete session audit trail
         """
-        return {
+        with self.lock:
-            'session_metadata': self.session_metadata.copy(),
+            return {
-            'api_requests': [asdict(req) for req in self.api_requests],
+                'session_metadata': self.session_metadata.copy(),
-            'relationships': [asdict(rel) for rel in self.relationships],
+                'api_requests': [asdict(req) for req in self.api_requests],
-            'export_timestamp': datetime.now(timezone.utc).isoformat()
+                'relationships': [asdict(rel) for rel in self.relationships],
-        }
+                'export_timestamp': datetime.now(timezone.utc).isoformat()
+            }
     
     def get_forensic_summary(self) -> Dict[str, Any]:
         """
@@ -229,7 +280,13 @@ class ForensicLogger:
             Dictionary containing summary statistics
         """
         provider_stats = {}
-        for provider in self.session_metadata['providers_used']:
+        
+        # Ensure providers_used is a set for iteration
+        providers_used = self.session_metadata['providers_used']
+        if isinstance(providers_used, list):
+            providers_used = set(providers_used)
+        
+        for provider in providers_used:
             provider_requests = [req for req in self.api_requests if req.provider == provider]
             provider_relationships = [rel for rel in self.relationships if rel.provider == provider]
             

core/session_manager.py

@@ -6,6 +6,7 @@ import uuid
 import redis
 import pickle
 from typing import Dict, Optional, Any
+import copy
 
 from core.scanner import Scanner
 from config import config
@@ -13,7 +14,7 @@ from config import config
 class SessionManager:
     """
     FIXED: Manages multiple scanner instances for concurrent user sessions using Redis.
-    Now more conservative about session creation to preserve API keys and configuration.
+    Enhanced to properly maintain WebSocket connections throughout scan lifecycle.
     """
     
     def __init__(self, session_timeout_minutes: int = 0):
@@ -30,6 +31,9 @@ class SessionManager:
         # FIXED: Add a creation lock to prevent race conditions
         self.creation_lock = threading.Lock()
         
+        # Track active socketio connections per session
+        self.active_socketio_connections = {}
+        
         # Start cleanup thread
         self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
         self.cleanup_thread.start()
@@ -40,7 +44,7 @@ class SessionManager:
         """Prepare SessionManager for pickling."""
         state = self.__dict__.copy()
         # Exclude unpickleable attributes - Redis client and threading objects
-        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock']
+        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock', 'active_socketio_connections']
         for attr in unpicklable_attrs:
             if attr in state:
                 del state[attr]
@@ -53,6 +57,7 @@ class SessionManager:
         self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
         self.lock = threading.Lock()
         self.creation_lock = threading.Lock()
+        self.active_socketio_connections = {}
         self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
         self.cleanup_thread.start()
 
@@ -64,22 +69,70 @@ class SessionManager:
         """Generates the Redis key for a session's stop signal."""
         return f"dnsrecon:stop:{session_id}"
 
-    def create_session(self) -> str:
+    def register_socketio_connection(self, session_id: str, socketio) -> None:
         """
-        FIXED: Create a new user session with thread-safe creation to prevent duplicates.
+        FIXED: Register a socketio connection for a session.
+        This ensures the connection is maintained throughout the session lifecycle.
+        """
+        with self.lock:
+            self.active_socketio_connections[session_id] = socketio
+            print(f"Registered socketio connection for session {session_id}")
+
+    def get_socketio_connection(self, session_id: str):
+        """
+        FIXED: Get the active socketio connection for a session.
+        """
+        with self.lock:
+            return self.active_socketio_connections.get(session_id)
+
+    def _prepare_scanner_for_storage(self, scanner: Scanner, session_id: str) -> Scanner:
+        """
+        FIXED: Prepare scanner for storage by ensuring proper cleanup of unpicklable objects.
+        Now preserves socketio connection info for restoration.
+        """
+        # Set the session ID on the scanner for cross-process stop signal management
+        scanner.session_id = session_id
+        
+        # FIXED: Don't set socketio to None if we want to preserve real-time updates
+        # Instead, we'll restore it when loading the scanner
+        scanner.socketio = None
+        
+        # Force cleanup of any threading objects that might cause issues
+        if hasattr(scanner, 'stop_event'):
+            scanner.stop_event = None
+        if hasattr(scanner, 'scan_thread'):
+            scanner.scan_thread = None
+        if hasattr(scanner, 'executor'):
+            scanner.executor = None
+        if hasattr(scanner, 'status_logger_thread'):
+            scanner.status_logger_thread = None
+        if hasattr(scanner, 'status_logger_stop_event'):
+            scanner.status_logger_stop_event = None
+        
+        return scanner
+
+    def create_session(self, socketio=None) -> str:
+        """
+        FIXED: Create a new user session with enhanced WebSocket management.
         """
         # FIXED: Use creation lock to prevent race conditions
         with self.creation_lock:
             session_id = str(uuid.uuid4())
             print(f"=== CREATING SESSION {session_id} IN REDIS ===")
             
+            # FIXED: Register socketio connection first
+            if socketio:
+                self.register_socketio_connection(session_id, socketio)
+            
             try:
                 from core.session_config import create_session_config
                 session_config = create_session_config()
-                scanner_instance = Scanner(session_config=session_config)
                 
-                # Set the session ID on the scanner for cross-process stop signal management
+                # Create scanner WITHOUT socketio to avoid weakref issues
-                scanner_instance.session_id = session_id
+                scanner_instance = Scanner(session_config=session_config, socketio=None)
+                
+                # Prepare scanner for storage (removes problematic objects)
+                scanner_instance = self._prepare_scanner_for_storage(scanner_instance, session_id)
                 
                 session_data = {
                     'scanner': scanner_instance,
@@ -89,12 +142,24 @@ class SessionManager:
                     'status': 'active'
                 }
                 
-                # Serialize the entire session data dictionary using pickle
+                # Test serialization before storing to catch issues early
-                serialized_data = pickle.dumps(session_data)
+                try:
+                    test_serialization = pickle.dumps(session_data)
+                    print(f"Session serialization test successful ({len(test_serialization)} bytes)")
+                except Exception as pickle_error:
+                    print(f"PICKLE TEST FAILED: {pickle_error}")
+                    # Try to identify the problematic object
+                    for key, value in session_data.items():
+                        try:
+                            pickle.dumps(value)
+                            print(f"  {key}: OK")
+                        except Exception as item_error:
+                            print(f"  {key}: FAILED - {item_error}")
+                    raise pickle_error
                 
                 # Store in Redis
                 session_key = self._get_session_key(session_id)
-                self.redis_client.setex(session_key, self.session_timeout, serialized_data)
+                self.redis_client.setex(session_key, self.session_timeout, test_serialization)
                 
                 # Initialize stop signal as False
                 stop_key = self._get_stop_signal_key(session_id)
@@ -106,6 +171,8 @@ class SessionManager:
                 
             except Exception as e:
                 print(f"ERROR: Failed to create session {session_id}: {e}")
+                import traceback
+                traceback.print_exc()
                 raise
 
     def set_stop_signal(self, session_id: str) -> bool:
@@ -175,31 +242,63 @@ class SessionManager:
                 # Ensure the scanner has the correct session ID for stop signal checking
                 if 'scanner' in session_data and session_data['scanner']:
                     session_data['scanner'].session_id = session_id
+                    # FIXED: Restore socketio connection from our registry
+                    socketio_conn = self.get_socketio_connection(session_id)
+                    if socketio_conn:
+                        session_data['scanner'].socketio = socketio_conn
+                        print(f"Restored socketio connection for session {session_id}")
+                    else:
+                        print(f"No socketio connection found for session {session_id}")
+                        session_data['scanner'].socketio = None
                 return session_data
             return None
         except Exception as e:
             print(f"ERROR: Failed to get session data for {session_id}: {e}")
+            import traceback
+            traceback.print_exc()
             return None
 
     def _save_session_data(self, session_id: str, session_data: Dict[str, Any]) -> bool:
         """
         Serializes and saves session data back to Redis with updated TTL.
+        FIXED: Now preserves socketio connection during storage.
         
         Returns:
             bool: True if save was successful
         """
         try:
             session_key = self._get_session_key(session_id)
-            serialized_data = pickle.dumps(session_data)
+            
+            # Create a deep copy to avoid modifying the original scanner object
+            session_data_to_save = copy.deepcopy(session_data)
+
+            # Prepare scanner for storage if it exists
+            if 'scanner' in session_data_to_save and session_data_to_save['scanner']:
+                # FIXED: Preserve the original socketio connection before preparing for storage
+                original_socketio = session_data_to_save['scanner'].socketio
+                
+                session_data_to_save['scanner'] = self._prepare_scanner_for_storage(
+                    session_data_to_save['scanner'], 
+                    session_id
+                )
+                
+                # FIXED: If we had a socketio connection, make sure it's registered
+                if original_socketio and session_id not in self.active_socketio_connections:
+                    self.register_socketio_connection(session_id, original_socketio)
+            
+            serialized_data = pickle.dumps(session_data_to_save)
             result = self.redis_client.setex(session_key, self.session_timeout, serialized_data)
             return result
         except Exception as e:
             print(f"ERROR: Failed to save session data for {session_id}: {e}")
+            import traceback
+            traceback.print_exc()
             return False
 
     def update_session_scanner(self, session_id: str, scanner: 'Scanner') -> bool:
         """
-        Updates just the scanner object in a session with immediate persistence.
+        FIXED: Updates just the scanner object in a session with immediate persistence.
+        Now maintains socketio connection throughout the update process.
         
         Returns:
             bool: True if update was successful
@@ -207,21 +306,27 @@ class SessionManager:
         try:
             session_data = self._get_session_data(session_id)
             if session_data:
-                # Ensure scanner has the session ID
+                # FIXED: Preserve socketio connection before preparing for storage
-                scanner.session_id = session_id
+                original_socketio = scanner.socketio
+                
+                # Prepare scanner for storage
+                scanner = self._prepare_scanner_for_storage(scanner, session_id)
                 session_data['scanner'] = scanner
                 session_data['last_activity'] = time.time()
                 
+                # FIXED: Restore socketio connection after preparation
+                if original_socketio:
+                    self.register_socketio_connection(session_id, original_socketio)
+                    session_data['scanner'].socketio = original_socketio
+                
                 # Immediately save to Redis for GUI updates
                 success = self._save_session_data(session_id, session_data)
                 if success:
                     # Only log occasionally to reduce noise
                     if hasattr(self, '_last_update_log'):
                         if time.time() - self._last_update_log > 5:  # Log every 5 seconds max
-                            #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
                             self._last_update_log = time.time()
                     else:
-                        #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
                         self._last_update_log = time.time()
                 else:
                     print(f"WARNING: Failed to save scanner state for session {session_id}")
@@ -231,6 +336,8 @@ class SessionManager:
                 return False
         except Exception as e:
             print(f"ERROR: Failed to update scanner for session {session_id}: {e}")
+            import traceback
+            traceback.print_exc()
             return False
 
     def update_scanner_status(self, session_id: str, status: str) -> bool:
@@ -263,7 +370,7 @@ class SessionManager:
 
     def get_session(self, session_id: str) -> Optional[Scanner]:
         """
-        Get scanner instance for a session from Redis with session ID management.
+        FIXED: Get scanner instance for a session from Redis with proper socketio restoration.
         """
         if not session_id:
             return None
@@ -281,6 +388,15 @@ class SessionManager:
         if scanner:
             # Ensure the scanner can check the Redis-based stop signal
             scanner.session_id = session_id
+            
+            # FIXED: Restore socketio connection from our registry
+            socketio_conn = self.get_socketio_connection(session_id)
+            if socketio_conn:
+                scanner.socketio = socketio_conn
+                print(f"✓ Restored socketio connection for session {session_id}")
+            else:
+                scanner.socketio = None
+                print(f"⚠️ No socketio connection found for session {session_id}")
         
         return scanner
 
@@ -333,6 +449,12 @@ class SessionManager:
             # Wait a moment for graceful shutdown
             time.sleep(0.5)
             
+            # FIXED: Clean up socketio connection
+            with self.lock:
+                if session_id in self.active_socketio_connections:
+                    del self.active_socketio_connections[session_id]
+                    print(f"Cleaned up socketio connection for session {session_id}")
+            
             # Delete session data and stop signal from Redis
             session_key = self._get_session_key(session_id)
             stop_key = self._get_stop_signal_key(session_id)
@@ -344,6 +466,8 @@ class SessionManager:
             
         except Exception as e:
             print(f"ERROR: Failed to terminate session {session_id}: {e}")
+            import traceback
+            traceback.print_exc()
             return False
 
     def _cleanup_loop(self) -> None:
@@ -364,6 +488,12 @@ class SessionManager:
                         self.redis_client.delete(stop_key)
                         print(f"Cleaned up orphaned stop signal for session {session_id}")
                         
+                        # Also clean up socketio connection
+                        with self.lock:
+                            if session_id in self.active_socketio_connections:
+                                del self.active_socketio_connections[session_id]
+                                print(f"Cleaned up orphaned socketio for session {session_id}")
+                        
             except Exception as e:
                 print(f"Error in cleanup loop: {e}")
             
@@ -387,14 +517,16 @@ class SessionManager:
             return {
                 'total_active_sessions': active_sessions,
                 'running_scans': running_scans,
-                'total_stop_signals': len(stop_keys)
+                'total_stop_signals': len(stop_keys),
+                'active_socketio_connections': len(self.active_socketio_connections)
             }
         except Exception as e:
             print(f"ERROR: Failed to get statistics: {e}")
             return {
                 'total_active_sessions': 0,
                 'running_scans': 0,
-                'total_stop_signals': 0
+                'total_stop_signals': 0,
+                'active_socketio_connections': 0
             }
 
 # Global session manager instance

providers/__init__.py

@@ -7,14 +7,16 @@ from .base_provider import BaseProvider
 from .crtsh_provider import CrtShProvider
 from .dns_provider import DNSProvider
 from .shodan_provider import ShodanProvider
+from .correlation_provider import CorrelationProvider
 from core.rate_limiter import GlobalRateLimiter
 
 __all__ = [
     'BaseProvider',
-    'GlobalRateLimiter', 
+    'GlobalRateLimiter',
     'CrtShProvider',
     'DNSProvider',
-    'ShodanProvider'
+    'ShodanProvider',
+    'CorrelationProvider'
 ]
 
 __version__ = "0.0.0-rc"

providers/base_provider.py

@@ -15,6 +15,7 @@ class BaseProvider(ABC):
     """
     Abstract base class for all DNSRecon data providers.
     Now supports session-specific configuration and returns standardized ProviderResult objects.
+    FIXED: Enhanced pickle support to prevent weakref serialization errors.
     """
 
     def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30, session_config=None):
@@ -53,22 +54,57 @@ class BaseProvider(ABC):
     def __getstate__(self):
         """Prepare BaseProvider for pickling by excluding unpicklable objects."""
         state = self.__dict__.copy()
-        # Exclude the unpickleable '_local' attribute and stop event
+        
-        unpicklable_attrs = ['_local', '_stop_event']
+        # Exclude unpickleable attributes that may contain weakrefs
+        unpicklable_attrs = [
+            '_local',           # Thread-local storage (contains requests.Session)
+            '_stop_event',      # Threading event
+            'logger',           # Logger may contain weakrefs in handlers
+        ]
+        
         for attr in unpicklable_attrs:
             if attr in state:
                 del state[attr]
+        
+        # Also handle any potential weakrefs in the config object
+        if 'config' in state and hasattr(state['config'], '__getstate__'):
+            # If config has its own pickle support, let it handle itself
+            pass
+        elif 'config' in state:
+            # Otherwise, ensure config doesn't contain unpicklable objects
+            try:
+                # Test if config can be pickled
+                import pickle
+                pickle.dumps(state['config'])
+            except (TypeError, AttributeError):
+                # If config can't be pickled, we'll recreate it during unpickling
+                state['_config_class'] = type(state['config']).__name__
+                del state['config']
+        
         return state
 
     def __setstate__(self, state):
         """Restore BaseProvider after unpickling by reconstructing threading objects."""
         self.__dict__.update(state)
-        # Re-initialize the '_local' attribute and stop event
+        
+        # Re-initialize unpickleable attributes
         self._local = threading.local()
         self._stop_event = None
+        self.logger = get_forensic_logger()
+        
+        # Recreate config if it was removed during pickling
+        if not hasattr(self, 'config') and hasattr(self, '_config_class'):
+            if self._config_class == 'Config':
+                from config import config as global_config
+                self.config = global_config
+            elif self._config_class == 'SessionConfig':
+                from core.session_config import create_session_config
+                self.config = create_session_config()
+            del self._config_class
 
     @property
     def session(self):
+        """Get or create thread-local requests session."""
         if not hasattr(self._local, 'session'):
             self._local.session = requests.Session()
             self._local.session.headers.update({

providers/correlation_provider.py

@@ -0,0 +1,220 @@
+# dnsrecon/providers/correlation_provider.py
+
+import re
+from typing import Dict, Any, List
+
+from .base_provider import BaseProvider
+from core.provider_result import ProviderResult
+from core.graph_manager import NodeType, GraphManager
+
+class CorrelationProvider(BaseProvider):
+    """
+    A provider that finds correlations between nodes in the graph.
+    FIXED: Enhanced pickle support to prevent weakref issues with graph references.
+    """
+
+    def __init__(self, name: str = "correlation", session_config=None):
+        """
+        Initialize the correlation provider.
+        """
+        super().__init__(name, session_config=session_config)
+        self.graph: GraphManager | None = None
+        self.correlation_index = {}
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
+        self.EXCLUDED_KEYS = [
+            'cert_source',
+            'cert_issuer_ca_id',
+            'cert_common_name',
+            'cert_validity_period_days',
+            'cert_issuer_name',
+            'cert_serial_number',
+            'cert_entry_timestamp',
+            'cert_not_before',
+            'cert_not_after',
+            'dns_ttl',
+            'timestamp',
+            'last_update',
+            'updated_timestamp',
+            'discovery_timestamp',
+            'query_timestamp',
+        ]
+
+    def __getstate__(self):
+        """
+        FIXED: Prepare CorrelationProvider for pickling by excluding graph reference.
+        """
+        state = super().__getstate__()
+        
+        # Remove graph reference to prevent circular dependencies and weakrefs
+        if 'graph' in state:
+            del state['graph']
+        
+        # Also handle correlation_index which might contain complex objects
+        if 'correlation_index' in state:
+            # Clear correlation index as it will be rebuilt when needed
+            state['correlation_index'] = {}
+        
+        return state
+
+    def __setstate__(self, state):
+        """
+        FIXED: Restore CorrelationProvider after unpickling.
+        """
+        super().__setstate__(state)
+        
+        # Re-initialize graph reference (will be set by scanner)
+        self.graph = None
+        
+        # Re-initialize correlation index
+        self.correlation_index = {}
+        
+        # Re-compile regex pattern
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
+
+    def get_name(self) -> str:
+        """Return the provider name."""
+        return "correlation"
+
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "Correlation Engine"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': True}
+
+    def is_available(self) -> bool:
+        """Check if the provider is available and properly configured."""
+        return True
+
+    def query_domain(self, domain: str) -> ProviderResult:
+        """
+        Query the provider for information about a domain.
+        """
+        return self._find_correlations(domain)
+
+    def query_ip(self, ip: str) -> ProviderResult:
+        """
+        Query the provider for information about an IP address.
+        """
+        return self._find_correlations(ip)
+
+    def set_graph_manager(self, graph_manager: GraphManager):
+        """
+        Set the graph manager for the provider to use.
+        """
+        self.graph = graph_manager
+
+    def _find_correlations(self, node_id: str) -> ProviderResult:
+        """
+        Find correlations for a given node.
+        FIXED: Added safety checks to prevent issues when graph is None.
+        """
+        result = ProviderResult()
+        
+        # FIXED: Ensure self.graph is not None before proceeding
+        if not self.graph or not self.graph.graph.has_node(node_id):
+            return result
+
+        try:
+            node_attributes = self.graph.graph.nodes[node_id].get('attributes', [])
+        except Exception as e:
+            # If there's any issue accessing the graph, return empty result
+            print(f"Warning: Could not access graph for correlation analysis: {e}")
+            return result
+
+        for attr in node_attributes:
+            attr_name = attr.get('name')
+            attr_value = attr.get('value')
+            attr_provider = attr.get('provider', 'unknown')
+
+            should_exclude = (
+                any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS) or
+                not isinstance(attr_value, (str, int, float, bool)) or
+                attr_value is None or
+                isinstance(attr_value, bool) or
+                (isinstance(attr_value, str) and (
+                    len(attr_value) < 4 or
+                    self.date_pattern.match(attr_value) or
+                    attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']
+                )) or
+                (isinstance(attr_value, (int, float)) and (
+                    attr_value == 0 or
+                    attr_value == 1 or
+                    abs(attr_value) > 1000000
+                ))
+            )
+
+            if should_exclude:
+                continue
+
+            if attr_value not in self.correlation_index:
+                self.correlation_index[attr_value] = {
+                    'nodes': set(),
+                    'sources': []
+                }
+
+            self.correlation_index[attr_value]['nodes'].add(node_id)
+
+            source_info = {
+                'node_id': node_id,
+                'provider': attr_provider,
+                'attribute': attr_name,
+                'path': f"{attr_provider}_{attr_name}"
+            }
+
+            existing_sources = [s for s in self.correlation_index[attr_value]['sources']
+                              if s['node_id'] == node_id and s['path'] == source_info['path']]
+            if not existing_sources:
+                self.correlation_index[attr_value]['sources'].append(source_info)
+
+            if len(self.correlation_index[attr_value]['nodes']) > 1:
+                self._create_correlation_relationships(attr_value, self.correlation_index[attr_value], result)
+                
+        return result
+
+    def _create_correlation_relationships(self, value: Any, correlation_data: Dict[str, Any], result: ProviderResult):
+        """
+        Create correlation relationships and add them to the provider result.
+        """
+        correlation_node_id = f"corr_{hash(str(value)) & 0x7FFFFFFF}"
+        nodes = correlation_data['nodes']
+        sources = correlation_data['sources']
+
+        # Add the correlation node as an attribute to the result
+        result.add_attribute(
+            target_node=correlation_node_id,
+            name="correlation_value",
+            value=value,
+            attr_type=str(type(value)),
+            provider=self.name,
+            confidence=0.9,
+            metadata={
+                'correlated_nodes': list(nodes),
+                'sources': sources,
+            }
+        )
+
+        for source in sources:
+            node_id = source['node_id']
+            provider = source['provider']
+            attribute = source['attribute']
+            relationship_label = f"corr_{provider}_{attribute}"
+
+            # Add the relationship to the result
+            result.add_relationship(
+                source_node=node_id,
+                target_node=correlation_node_id,
+                relationship_type=relationship_label,
+                provider=self.name,
+                confidence=0.9,
+                raw_data={
+                    'correlation_value': value,
+                    'original_attribute': attribute,
+                    'correlation_type': 'attribute_matching'
+                }
+            )

providers/crtsh_provider.py

@@ -3,7 +3,7 @@
 import json
 import re
 from pathlib import Path
-from typing import List, Dict, Any, Set
+from typing import List, Dict, Any, Set, Optional
 from urllib.parse import quote
 from datetime import datetime, timezone
 import requests
@@ -11,6 +11,7 @@ import requests
 from .base_provider import BaseProvider
 from core.provider_result import ProviderResult
 from utils.helpers import _is_valid_domain
+from core.logger import get_forensic_logger
 
 
 class CrtShProvider(BaseProvider):
@@ -37,7 +38,7 @@ class CrtShProvider(BaseProvider):
         
         # Compile regex for date filtering for efficiency
         self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
-    
+
     def get_name(self) -> str:
         """Return the provider name."""
         return "crtsh"
@@ -114,51 +115,42 @@ class CrtShProvider(BaseProvider):
         
         result = ProviderResult()
 
-        try:
+        if cache_status == "fresh":
-            if cache_status == "fresh":
+            result = self._load_from_cache(cache_file)
-                result = self._load_from_cache(cache_file)
+            self.logger.logger.info(f"Using fresh cached crt.sh data for {domain}")
-                self.logger.logger.info(f"Using fresh cached crt.sh data for {domain}")
+        
+        else:  # "stale" or "not_found"
+            # Query the API for the latest certificates
+            new_raw_certs = self._query_crtsh_api(domain)
             
-            else:  # "stale" or "not_found"
+            if self._stop_event and self._stop_event.is_set():
-                # Query the API for the latest certificates
+                return ProviderResult()
-                new_raw_certs = self._query_crtsh_api(domain)
+
+            # Combine with old data if cache is stale
+            if cache_status == "stale":
+                old_raw_certs = self._load_raw_data_from_cache(cache_file)
+                combined_certs = old_raw_certs + new_raw_certs
                 
-                if self._stop_event and self._stop_event.is_set():
+                # Deduplicate the combined list
-                    return ProviderResult()
+                seen_ids = set()
-
+                unique_certs = []
-                # Combine with old data if cache is stale
+                for cert in combined_certs:
-                if cache_status == "stale":
+                    cert_id = cert.get('id')
-                    old_raw_certs = self._load_raw_data_from_cache(cache_file)
+                    if cert_id not in seen_ids:
-                    combined_certs = old_raw_certs + new_raw_certs
+                        unique_certs.append(cert)
-                    
+                        seen_ids.add(cert_id)
-                    # Deduplicate the combined list
-                    seen_ids = set()
-                    unique_certs = []
-                    for cert in combined_certs:
-                        cert_id = cert.get('id')
-                        if cert_id not in seen_ids:
-                            unique_certs.append(cert)
-                            seen_ids.add(cert_id)
-                    
-                    raw_certificates_to_process = unique_certs
-                    self.logger.logger.info(f"Refreshed and merged cache for {domain}. Total unique certs: {len(raw_certificates_to_process)}")
-                else:  # "not_found"
-                    raw_certificates_to_process = new_raw_certs
                 
-                # FIXED: Process certificates to create proper domain and CA nodes
+                raw_certificates_to_process = unique_certs
-                result = self._process_certificates_to_result_fixed(domain, raw_certificates_to_process)
+                self.logger.logger.info(f"Refreshed and merged cache for {domain}. Total unique certs: {len(raw_certificates_to_process)}")
-                self.logger.logger.info(f"Created fresh result for {domain} ({result.get_relationship_count()} relationships)")
+            else:  # "not_found"
+                raw_certificates_to_process = new_raw_certs
+            
+            # FIXED: Process certificates to create proper domain and CA nodes
+            result = self._process_certificates_to_result_fixed(domain, raw_certificates_to_process)
+            self.logger.logger.info(f"Created fresh result for {domain} ({result.get_relationship_count()} relationships)")
 
-                # Save the new result and the raw data to the cache
+            # Save the new result and the raw data to the cache
-                self._save_result_to_cache(cache_file, result, raw_certificates_to_process, domain)
+            self._save_result_to_cache(cache_file, result, raw_certificates_to_process, domain)
-
-        except requests.exceptions.RequestException as e:
-            self.logger.logger.error(f"API query failed for {domain}: {e}")
-            if cache_status != "not_found":
-                result = self._load_from_cache(cache_file)
-                self.logger.logger.warning(f"Using stale cache for {domain} due to API failure.")
-            else:
-                raise e  # Re-raise if there's no cache to fall back on
 
         return result
 
@@ -285,6 +277,17 @@ class CrtShProvider(BaseProvider):
         if self._stop_event and self._stop_event.is_set():
             self.logger.logger.info(f"CrtSh processing cancelled before processing for domain: {query_domain}")
             return result
+        
+        incompleteness_warning = self._check_for_incomplete_data(query_domain, certificates)
+        if incompleteness_warning:
+            result.add_attribute(
+                target_node=query_domain,
+                name="crtsh_data_warning",
+                value=incompleteness_warning,
+                attr_type='metadata',
+                provider=self.name,
+                confidence=1.0
+            )
 
         all_discovered_domains = set()
         processed_issuers = set()
@@ -457,6 +460,8 @@ class CrtShProvider(BaseProvider):
             raise ValueError("Empty date string")
 
         try:
+            if isinstance(date_string, datetime):
+                return date_string.replace(tzinfo=timezone.utc)
             if date_string.endswith('Z'):
                 return datetime.fromisoformat(date_string[:-1]).replace(tzinfo=timezone.utc)
             elif '+' in date_string or date_string.endswith('UTC'):
@@ -577,4 +582,30 @@ class CrtShProvider(BaseProvider):
         elif query_domain.endswith(f'.{cert_domain}'):
             return 'parent_domain'
         else:
-            return 'related_domain'
+            return 'related_domain'
+        
+    def _check_for_incomplete_data(self, domain: str, certificates: List[Dict[str, Any]]) -> Optional[str]:
+        """
+        Analyzes the certificate list to heuristically detect if the data from crt.sh is incomplete.
+        """
+        cert_count = len(certificates)
+
+        # Heuristic 1: Check if the number of certs hits a known hard limit.
+        if cert_count >= 10000:
+            return f"Result likely truncated; received {cert_count} certificates, which may be the maximum limit."
+
+        # Heuristic 2: Check if all returned certificates are old.
+        if cert_count > 1000: # Only apply this for a reasonable number of certs
+            latest_expiry = None
+            for cert in certificates:
+                try:
+                    not_after = self._parse_certificate_date(cert.get('not_after'))
+                    if latest_expiry is None or not_after > latest_expiry:
+                        latest_expiry = not_after
+                except (ValueError, TypeError):
+                    continue
+
+            if latest_expiry and (datetime.now(timezone.utc) - latest_expiry).days > 365:
+                 return f"Incomplete data suspected: The latest certificate expired more than a year ago ({latest_expiry.strftime('%Y-%m-%d')})."
+
+        return None

providers/dns_provider.py

@@ -11,6 +11,7 @@ class DNSProvider(BaseProvider):
     """
     Provider for standard DNS resolution and reverse DNS lookups.
     Now returns standardized ProviderResult objects with IPv4 and IPv6 support.
+    FIXED: Enhanced pickle support to prevent resolver serialization issues.
     """
 
     def __init__(self, name=None, session_config=None):
@@ -27,6 +28,22 @@ class DNSProvider(BaseProvider):
         self.resolver.timeout = 5
         self.resolver.lifetime = 10
 
+    def __getstate__(self):
+        """Prepare the object for pickling by excluding resolver."""
+        state = super().__getstate__()
+        # Remove the unpickleable 'resolver' attribute
+        if 'resolver' in state:
+            del state['resolver']
+        return state
+
+    def __setstate__(self, state):
+        """Restore the object after unpickling by reconstructing resolver."""
+        super().__setstate__(state)
+        # Re-initialize the 'resolver' attribute
+        self.resolver = resolver.Resolver()
+        self.resolver.timeout = 5
+        self.resolver.lifetime = 10
+
     def get_name(self) -> str:
         """Return the provider name."""
         return "dns"
@@ -106,10 +123,10 @@ class DNSProvider(BaseProvider):
                 if _is_valid_domain(hostname):
                     # Determine appropriate forward relationship type based on IP version
                     if ip_version == 6:
-                        relationship_type = 'dns_aaaa_record'
+                        relationship_type = 'shodan_aaaa_record'
                         record_prefix = 'AAAA'
                     else:
-                        relationship_type = 'dns_a_record'
+                        relationship_type = 'shodan_a_record'
                         record_prefix = 'A'
                     
                     # Add the relationship

providers/shodan_provider.py

@@ -28,13 +28,61 @@ class ShodanProvider(BaseProvider):
         self.base_url = "https://api.shodan.io"
         self.api_key = self.config.get_api_key('shodan')
         
+        # FIXED: Don't fail initialization on connection issues - defer to actual usage
+        self._connection_tested = False
+        self._connection_works = False
+
         # Initialize cache directory
         self.cache_dir = Path('cache') / 'shodan'
         self.cache_dir.mkdir(parents=True, exist_ok=True)
 
+    def __getstate__(self):
+        """Prepare the object for pickling."""
+        state = super().__getstate__()
+        return state
+
+    def __setstate__(self, state):
+        """Restore the object after unpickling."""
+        super().__setstate__(state)
+
+    def _check_api_connection(self) -> bool:
+        """
+        FIXED: Lazy connection checking - only test when actually needed.
+        Don't block provider initialization on network issues.
+        """
+        if self._connection_tested:
+            return self._connection_works
+            
+        if not self.api_key:
+            self._connection_tested = True
+            self._connection_works = False
+            return False
+            
+        try:
+            print(f"Testing Shodan API connection with key: {self.api_key[:8]}...")
+            response = self.session.get(f"{self.base_url}/api-info?key={self.api_key}", timeout=5)
+            self._connection_works = response.status_code == 200
+            print(f"Shodan API test result: {response.status_code} - {'Success' if self._connection_works else 'Failed'}")
+        except requests.exceptions.RequestException as e:
+            print(f"Shodan API connection test failed: {e}")
+            self._connection_works = False
+        finally:
+            self._connection_tested = True
+            
+        return self._connection_works
+
     def is_available(self) -> bool:
-        """Check if Shodan provider is available (has valid API key in this session)."""
+        """
-        return self.api_key is not None and len(self.api_key.strip()) > 0
+        FIXED: Check if Shodan provider is available based on API key presence.
+        Don't require successful connection test during initialization.
+        """
+        has_api_key = self.api_key is not None and len(self.api_key.strip()) > 0
+        
+        if not has_api_key:
+            return False
+            
+        # FIXED: Only test connection on first actual usage, not during initialization
+        return True
 
     def get_name(self) -> str:
         """Return the provider name."""
@@ -98,27 +146,30 @@ class ShodanProvider(BaseProvider):
 
     def query_domain(self, domain: str) -> ProviderResult:
         """
-        Domain queries are no longer supported for the Shodan provider.
+        Shodan does not support domain queries. This method returns an empty result.
-        
-        Args:
-            domain: Domain to investigate
-            
-        Returns:
-            Empty ProviderResult
         """
         return ProviderResult()
 
     def query_ip(self, ip: str) -> ProviderResult:
         """
         Query Shodan for information about an IP address (IPv4 or IPv6), with caching of processed data.
+        FIXED: Proper 404 handling to prevent unnecessary retries.
         
         Args:
             ip: IP address to investigate (IPv4 or IPv6)
             
         Returns:
             ProviderResult containing discovered relationships and attributes
+            
+        Raises:
+            Exception: For temporary failures that should be retried (timeouts, 502/503 errors, connection issues)
         """
-        if not _is_valid_ip(ip) or not self.is_available():
+        if not _is_valid_ip(ip):
+            return ProviderResult()
+            
+        # Test connection only when actually making requests
+        if not self._check_api_connection():
+            print(f"Shodan API not available for {ip} - API key: {'present' if self.api_key else 'missing'}")
             return ProviderResult()
         
         # Normalize IP address for consistent processing
@@ -129,55 +180,123 @@ class ShodanProvider(BaseProvider):
         cache_file = self._get_cache_file_path(normalized_ip)
         cache_status = self._get_cache_status(cache_file)
         
-        result = ProviderResult()
+        if cache_status == "fresh":
+            self.logger.logger.debug(f"Using fresh cache for Shodan query: {normalized_ip}")
+            return self._load_from_cache(cache_file)
+        
+        # Need to query API
+        self.logger.logger.debug(f"Querying Shodan API for: {normalized_ip}")
+        url = f"{self.base_url}/shodan/host/{normalized_ip}"
+        params = {'key': self.api_key}
         
         try:
-            if cache_status == "fresh":
+            response = self.make_request(url, method="GET", params=params, target_indicator=normalized_ip)
-                result = self._load_from_cache(cache_file)
+            
-                self.logger.logger.info(f"Using cached Shodan data for {normalized_ip}")
+            if not response:
-            else:  # "stale" or "not_found"
+                self.logger.logger.warning(f"Shodan API unreachable for {normalized_ip} - network failure")
-                url = f"{self.base_url}/shodan/host/{normalized_ip}"
+                if cache_status == "stale":
-                params = {'key': self.api_key}
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to network failure")
-                response = self.make_request(url, method="GET", params=params, target_indicator=normalized_ip)
+                    return self._load_from_cache(cache_file)
-                
-                if response and response.status_code == 200:
-                    data = response.json()
-                    # Process the data into ProviderResult BEFORE caching
-                    result = self._process_shodan_data(normalized_ip, data)
-                    self._save_to_cache(cache_file, result, data)  # Save both result and raw data
-                elif response and response.status_code == 404:
-                    # Handle 404 "No information available" as successful empty result
-                    try:
-                        error_data = response.json()
-                        if "No information available" in error_data.get('error', ''):
-                            # This is a successful query - Shodan just has no data
-                            self.logger.logger.debug(f"Shodan has no information for {normalized_ip}")
-                            result = ProviderResult()  # Empty but successful result
-                            # Cache the empty result to avoid repeated queries
-                            self._save_to_cache(cache_file, result, {'error': 'No information available'})
-                        else:
-                            # Some other 404 error - treat as failure
-                            raise requests.exceptions.RequestException(f"Shodan API returned 404: {error_data}")
-                    except (ValueError, KeyError):
-                        # Could not parse JSON response - treat as failure
-                        raise requests.exceptions.RequestException(f"Shodan API returned 404 with unparseable response")
-                elif cache_status == "stale":
-                    # If API fails on a stale cache, use the old data
-                    result = self._load_from_cache(cache_file)
                 else:
-                    # Other HTTP error codes should be treated as failures
+                    # FIXED: Treat network failures as "no information" rather than retryable errors
-                    status_code = response.status_code if response else "No response"
+                    self.logger.logger.info(f"No Shodan data available for {normalized_ip} due to network failure")
-                    raise requests.exceptions.RequestException(f"Shodan API returned HTTP {status_code}")
+                    result = ProviderResult()  # Empty result
-
+                    network_failure_data = {'shodan_status': 'network_unreachable', 'error': 'API unreachable'}
-        except requests.exceptions.RequestException as e:
+                    self._save_to_cache(cache_file, result, network_failure_data)
-            self.logger.logger.info(f"Shodan API query returned no info for {normalized_ip}: {e}")
+                    return result
-            if cache_status == "stale":
+            
-                result = self._load_from_cache(cache_file)
+            # FIXED: Handle different status codes more precisely
+            if response.status_code == 200:
+                self.logger.logger.debug(f"Shodan returned data for {normalized_ip}")
+                try:
+                    data = response.json()
+                    result = self._process_shodan_data(normalized_ip, data)
+                    self._save_to_cache(cache_file, result, data)
+                    return result
+                except json.JSONDecodeError as e:
+                    self.logger.logger.error(f"Invalid JSON response from Shodan for {normalized_ip}: {e}")
+                    if cache_status == "stale":
+                        return self._load_from_cache(cache_file)
+                    else:
+                        raise requests.exceptions.RequestException("Invalid JSON response from Shodan - should retry")
+            
+            elif response.status_code == 404:
+                # FIXED: 404 = "no information available" - successful but empty result, don't retry
+                self.logger.logger.debug(f"Shodan has no information for {normalized_ip} (404)")
+                result = ProviderResult()  # Empty but successful result
+                # Cache the empty result to avoid repeated queries
+                empty_data = {'shodan_status': 'no_information', 'status_code': 404}
+                self._save_to_cache(cache_file, result, empty_data)
+                return result
+            
+            elif response.status_code in [401, 403]:
+                # Authentication/authorization errors - permanent failures, don't retry
+                self.logger.logger.error(f"Shodan API authentication failed for {normalized_ip} (HTTP {response.status_code})")
+                return ProviderResult()  # Empty result, don't retry
+            
+            elif response.status_code == 429:
+                # Rate limiting - should be handled by rate limiter, but if we get here, retry
+                self.logger.logger.warning(f"Shodan API rate limited for {normalized_ip} (HTTP {response.status_code})")
+                if cache_status == "stale":
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to rate limiting")
+                    return self._load_from_cache(cache_file)
+                else:
+                    raise requests.exceptions.RequestException(f"Shodan API rate limited (HTTP {response.status_code}) - should retry")
+            
+            elif response.status_code in [500, 502, 503, 504]:
+                # Server errors - temporary failures that should be retried
+                self.logger.logger.warning(f"Shodan API server error for {normalized_ip} (HTTP {response.status_code})")
+                if cache_status == "stale":
+                    self.logger.logger.info(f"Using stale cache for {normalized_ip} due to server error")
+                    return self._load_from_cache(cache_file)
+                else:
+                    raise requests.exceptions.RequestException(f"Shodan API server error (HTTP {response.status_code}) - should retry")
+            
             else:
-                # Re-raise for retry scheduling - but only for actual failures
+                # FIXED: Other HTTP status codes - treat as no information available, don't retry
-                raise e
+                self.logger.logger.info(f"Shodan returned status {response.status_code} for {normalized_ip} - treating as no information")
+                result = ProviderResult()  # Empty result
+                no_info_data = {'shodan_status': 'no_information', 'status_code': response.status_code}
+                self._save_to_cache(cache_file, result, no_info_data)
+                return result
+                
+        except requests.exceptions.Timeout:
+            # Timeout errors - should be retried
+            self.logger.logger.warning(f"Shodan API timeout for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to timeout")
+                return self._load_from_cache(cache_file)
+            else:
+                raise  # Re-raise timeout for retry
         
-        return result
+        except requests.exceptions.ConnectionError:
+            # Connection errors - should be retried
+            self.logger.logger.warning(f"Shodan API connection error for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to connection error")
+                return self._load_from_cache(cache_file)
+            else:
+                raise  # Re-raise connection error for retry
+        
+        except json.JSONDecodeError:
+            # JSON parsing error - treat as temporary failure
+            self.logger.logger.error(f"Invalid JSON response from Shodan for {normalized_ip}")
+            if cache_status == "stale":
+                self.logger.logger.info(f"Using stale cache for {normalized_ip} due to JSON parsing error")
+                return self._load_from_cache(cache_file)
+            else:
+                raise requests.exceptions.RequestException("Invalid JSON response from Shodan - should retry")
+        
+        # FIXED: Remove the generic RequestException handler that was causing 404s to retry
+        # Now only specific exceptions that should be retried are re-raised
+        
+        except Exception as e:
+            # FIXED: Unexpected exceptions - log but treat as no information available, don't retry
+            self.logger.logger.warning(f"Unexpected exception in Shodan query for {normalized_ip}: {e}")
+            result = ProviderResult()  # Empty result
+            error_data = {'shodan_status': 'error', 'error': str(e)}
+            self._save_to_cache(cache_file, result, error_data)
+            return result
 
     def _load_from_cache(self, cache_file_path: Path) -> ProviderResult:
         """Load processed Shodan data from a cache file."""

requirements.txt

@@ -7,4 +7,7 @@ urllib3
 dnspython
 gunicorn
 redis
-python-dotenv
+python-dotenv
+psycopg2-binary
+Flask-SocketIO
+eventlet

static/js/graph.js

@@ -1,3 +1,4 @@
+// dnsrecon-reduced/static/js/graph.js
 /**
  * Graph visualization module for DNSRecon
  * Handles network graph rendering using vis.js with proper large entity node hiding
@@ -362,100 +363,60 @@ class GraphManager {
         }
 
         try {
-            // Initialize if not already done
             if (!this.isInitialized) {
                 this.initialize();
             }
 
             this.initialTargetIds = new Set(graphData.initial_targets || []);
-            // Check if we have actual data to display
             const hasData = graphData.nodes.length > 0 || graphData.edges.length > 0;
             
-            // Handle placeholder visibility
             const placeholder = this.container.querySelector('.graph-placeholder');
             if (placeholder) {
-                if (hasData) {
+                placeholder.style.display = hasData ? 'none' : 'flex';
-                    placeholder.style.display = 'none';
+            }
-                } else {
+            if (!hasData) {
-                    placeholder.style.display = 'flex';
+                this.nodes.clear();
-                    // Early return if no data to process
+                this.edges.clear();
-                    return;
+                return;
-                }
             }
 
-            this.largeEntityMembers.clear();
+            const nodeMap = new Map(graphData.nodes.map(node => [node.id, node]));
-            const largeEntityMap = new Map();
-            
-            graphData.nodes.forEach(node => {
-                if (node.type === 'large_entity' && node.attributes) {
-                    const nodesAttribute = this.findAttributeByName(node.attributes, 'nodes');
-                    if (nodesAttribute && Array.isArray(nodesAttribute.value)) {
-                        nodesAttribute.value.forEach(nodeId => {
-                            largeEntityMap.set(nodeId, node.id);
-                            this.largeEntityMembers.add(nodeId);
-                        });
-                    }
-                }
-            });
 
-            const filteredNodes = graphData.nodes.filter(node => {
+            // Filter out hidden nodes before processing for rendering
-                return !this.largeEntityMembers.has(node.id) || node.type === 'large_entity';
+            const filteredNodes = graphData.nodes.filter(node => 
-            });
+                !(node.metadata && node.metadata.large_entity_id)
+            );
 
-            console.log(`Filtered ${graphData.nodes.length - filteredNodes.length} large entity member nodes from visualization`);
+            const processedNodes = graphData.nodes.map(node => {
-
-            // Process nodes with proper certificate coloring
-            const processedNodes = filteredNodes.map(node => {
                 const processed = this.processNode(node);
-                
+                if (node.metadata && node.metadata.large_entity_id) {
-                // Apply certificate-based coloring here in frontend
+                    processed.hidden = true;
-                if (node.type === 'domain' && Array.isArray(node.attributes)) {
-                    const certInfo = this.analyzeCertificateInfo(node.attributes);
-                    
-                    if (certInfo.hasExpiredOnly) {
-                        // Red for domains with only expired/invalid certificates
-                        processed.color = { background: '#ff6b6b', border: '#cc5555' };
-                    } else if (!certInfo.hasCertificates) {
-                        // Grey for domains with no certificates
-                        processed.color = { background: '#c7c7c7', border: '#999999' };
-                    }
-                    // Valid certificates use default green (handled by processNode)
-                }
-                
-                return processed;
-            });
-
-            const mergedEdges = {};
-            graphData.edges.forEach(edge => {
-                const fromNode = largeEntityMap.has(edge.from) ? largeEntityMap.get(edge.from) : edge.from;
-                const toNode = largeEntityMap.has(edge.to) ? largeEntityMap.get(edge.to) : edge.to;
-                const mergeKey = `${fromNode}-${toNode}-${edge.label}`;
-
-                if (!mergedEdges[mergeKey]) {
-                    mergedEdges[mergeKey] = {
-                        ...edge,
-                        from: fromNode,
-                        to: toNode,
-                        count: 0,
-                        confidence_score: 0
-                    };
-                }
-
-                mergedEdges[mergeKey].count++;
-                if (edge.confidence_score > mergedEdges[mergeKey].confidence_score) {
-                    mergedEdges[mergeKey].confidence_score = edge.confidence_score;
-                }
-            });
-
-            const processedEdges = Object.values(mergedEdges).map(edge => {
-                const processed = this.processEdge(edge);
-                if (edge.count > 1) {
-                    processed.label = `${edge.label} (${edge.count})`;
                 }
                 return processed;
             });
+            
+            const processedEdges = graphData.edges.map(edge => {
+                let fromNode = nodeMap.get(edge.from);
+                let toNode = nodeMap.get(edge.to);
+                let fromId = edge.from;
+                let toId = edge.to;
+
+                if (fromNode && fromNode.metadata && fromNode.metadata.large_entity_id) {
+                    fromId = fromNode.metadata.large_entity_id;
+                }
+                if (toNode && toNode.metadata && toNode.metadata.large_entity_id) {
+                    toId = toNode.metadata.large_entity_id;
+                }
+                
+                // Avoid self-referencing edges from re-routing
+                if (fromId === toId) {
+                    return null; 
+                }
+
+                const reRoutedEdge = { ...edge, from: fromId, to: toId };
+                return this.processEdge(reRoutedEdge);
+            }).filter(Boolean); // Remove nulls from self-referencing edges
 
-            // Update datasets with animation
             const existingNodeIds = this.nodes.getIds();
             const existingEdgeIds = this.edges.getIds();
 
@@ -472,12 +433,9 @@ class GraphManager {
                 setTimeout(() => this.highlightNewElements(newNodes, newEdges), 100);
             }
 
-            if (processedNodes.length <= 10 || existingNodeIds.length === 0) {
+            if (this.nodes.length <= 10 || existingNodeIds.length === 0) {
                 setTimeout(() => this.fitView(), 800);
             }
-
-            console.log(`Graph updated: ${processedNodes.length} nodes, ${processedEdges.length} edges (${newNodes.length} new nodes, ${newEdges.length} new edges)`);
-            console.log(`Large entity members hidden: ${this.largeEntityMembers.size}`);
             
         } catch (error) {
             console.error('Failed to update graph:', error);
@@ -559,6 +517,11 @@ class GraphManager {
             outgoing_edges: node.outgoing_edges || []
         };
 
+        if (node.max_depth_reached) {
+            processedNode.borderColor = '#ff0000'; // Red border for max depth
+        }
+
+
         // Add confidence-based styling
         if (node.confidence) {
             processedNode.borderWidth = Math.max(2, Math.floor(node.confidence * 5));
@@ -582,26 +545,17 @@ class GraphManager {
         
         // Handle merged correlation objects
         if (node.type === 'correlation_object') {
-            const metadata = node.metadata || {};
+            const correlationValueAttr = this.findAttributeByName(node.attributes, 'correlation_value');
-            const values = metadata.values || [];
+            const value = correlationValueAttr ? correlationValueAttr.value : 'Unknown';
-            const mergeCount = metadata.merge_count || 1;
+            const displayValue = typeof value === 'string' && value.length > 20 ? value.substring(0, 17) + '...' : value;
-            
+
-            if (mergeCount > 1) {
+            processedNode.label = `${displayValue}`;
-                processedNode.label = `Correlations (${mergeCount})`;
+            processedNode.title = `Correlation: ${value}`;
-                processedNode.title = `Merged correlation container with ${mergeCount} values: ${values.slice(0, 3).join(', ')}${values.length > 3 ? '...' : ''}`;
-                processedNode.borderWidth = 3;
-            } else {
-                const value = Array.isArray(values) && values.length > 0 ? values[0] : (metadata.value || 'Unknown');
-                const displayValue = typeof value === 'string' && value.length > 20 ? value.substring(0, 17) + '...' : value;
-                processedNode.label = `${displayValue}`;
-                processedNode.title = `Correlation: ${value}`;
-            }
         }
 
         return processedNode;
     }
 
-
     /**
      * Process edge data with styling and metadata
      * @param {Object} edge - Raw edge data
@@ -610,7 +564,7 @@ class GraphManager {
     processEdge(edge) {
         const confidence = edge.confidence_score || 0;
         const processedEdge = {
-            id: `${edge.from}-${edge.to}`,
+            id: `${edge.from}-${edge.to}-${edge.label}`,
             from: edge.from,
             to: edge.to,
             label: this.formatEdgeLabel(edge.label, confidence),
@@ -1057,7 +1011,7 @@ class GraphManager {
         this.nodes.clear();
         this.edges.clear();
         this.history = [];
-        this.largeEntityMembers.clear(); // Clear large entity tracking
+        this.largeEntityMembers.clear();
         this.initialTargetIds.clear();
 
         // Show placeholder
@@ -1215,7 +1169,6 @@ class GraphManager {
         const basicStats = {
             nodeCount: this.nodes.length,
             edgeCount: this.edges.length,
-            largeEntityMembersHidden: this.largeEntityMembers.size
         };
 
         // Add forensic statistics
@@ -1612,14 +1565,43 @@ class GraphManager {
     }
 
     /**
-     * Unhide all hidden nodes
+     * FIXED: Unhide all hidden nodes, excluding large entity members and disconnected nodes.
+     * This prevents orphaned large entity members from appearing as free-floating nodes.
      */
     unhideAll() {
-        const allNodes = this.nodes.get({
+        const allHiddenNodes = this.nodes.get({
-            filter: (node) => node.hidden === true
+            filter: (node) => {
+                // Skip nodes that are part of a large entity
+                if (node.metadata && node.metadata.large_entity_id) {
+                    return false;
+                }
+                
+                // Skip nodes that are not hidden
+                if (node.hidden !== true) {
+                    return false;
+                }
+                
+                // Skip nodes that have no edges (would appear disconnected)
+                const nodeId = node.id;
+                const hasIncomingEdges = this.edges.get().some(edge => edge.to === nodeId && !edge.hidden);
+                const hasOutgoingEdges = this.edges.get().some(edge => edge.from === nodeId && !edge.hidden);
+                
+                if (!hasIncomingEdges && !hasOutgoingEdges) {
+                    console.log(`Skipping disconnected node ${nodeId} from unhide`);
+                    return false;
+                }
+                
+                return true;
+            }
         });
-        const updates = allNodes.map(node => ({ id: node.id, hidden: false }));
+        
-        this.nodes.update(updates);
+        if (allHiddenNodes.length > 0) {
+            console.log(`Unhiding ${allHiddenNodes.length} nodes with valid connections`);
+            const updates = allHiddenNodes.map(node => ({ id: node.id, hidden: false }));
+            this.nodes.update(updates);
+        } else {
+            console.log('No eligible nodes to unhide');
+        }
     }
     
 }

static/js/main.js

@@ -1,15 +1,15 @@
 /**
  * Main application logic for DNSRecon web interface
  * Handles UI interactions, API communication, and data flow
- * UPDATED: Now compatible with a strictly flat, unified data model for attributes.
+ * FIXED: Enhanced real-time WebSocket graph updates
  */
 
 class DNSReconApp {
     constructor() {
         console.log('DNSReconApp constructor called');
         this.graphManager = null;
+        this.socket = null;
         this.scanStatus = 'idle';
-        this.pollInterval = null;
         this.currentSessionId = null;
         
         this.elements = {};
@@ -17,6 +17,14 @@ class DNSReconApp {
         this.isScanning = false;
         this.lastGraphUpdate = null;
         
+        // FIXED: Add connection state tracking
+        this.isConnected = false;
+        this.reconnectAttempts = 0;
+        this.maxReconnectAttempts = 5;
+        
+        // FIXED: Track last graph data for debugging
+        this.lastGraphData = null;
+        
         this.init();
     }
     
@@ -31,13 +39,11 @@ class DNSReconApp {
                 this.initializeElements();
                 this.setupEventHandlers();
                 this.initializeGraph();
-                this.updateStatus();
+                this.initializeSocket();
                 this.loadProviders();
                 this.initializeEnhancedModals();
                 this.addCheckboxStyling();
                 
-                this.updateGraph();
-                
                 console.log('DNSRecon application initialized successfully');
             } catch (error) {
                 console.error('Failed to initialize DNSRecon application:', error);
@@ -45,6 +51,162 @@ class DNSReconApp {
             }
         });
     }
+
+    initializeSocket() {
+        console.log('🔌 Initializing WebSocket connection...');
+        
+        try {
+            this.socket = io({
+                transports: ['websocket', 'polling'],
+                timeout: 10000,
+                reconnection: true,
+                reconnectionAttempts: 5,
+                reconnectionDelay: 2000
+            });
+
+            this.socket.on('connect', () => {
+                console.log('✅ WebSocket connected successfully');
+                this.isConnected = true;
+                this.reconnectAttempts = 0;
+                this.updateConnectionStatus('idle');
+                
+                console.log('📡 Requesting initial status...');
+                this.socket.emit('get_status');
+            });
+
+            this.socket.on('disconnect', (reason) => {
+                console.log('❌ WebSocket disconnected:', reason);
+                this.isConnected = false;
+                this.updateConnectionStatus('error');
+            });
+
+            this.socket.on('connect_error', (error) => {
+                console.error('❌ WebSocket connection error:', error);
+                this.reconnectAttempts++;
+                this.updateConnectionStatus('error');
+                
+                if (this.reconnectAttempts >= 5) {
+                    this.showError('WebSocket connection failed. Please refresh the page.');
+                }
+            });
+
+            this.socket.on('reconnect', (attemptNumber) => {
+                console.log('✅ WebSocket reconnected after', attemptNumber, 'attempts');
+                this.isConnected = true;
+                this.reconnectAttempts = 0;
+                this.updateConnectionStatus('idle');
+                this.socket.emit('get_status');
+            });
+
+            // FIXED: Enhanced scan_update handler with detailed graph processing and debugging
+            this.socket.on('scan_update', (data) => {
+                console.log('📨 WebSocket update received:', {
+                    status: data.status,
+                    target: data.target_domain,
+                    progress: data.progress_percentage,
+                    graphNodes: data.graph?.nodes?.length || 0,
+                    graphEdges: data.graph?.edges?.length || 0,
+                    timestamp: new Date().toISOString()
+                });
+                
+                try {
+                    // Handle status change
+                    if (data.status !== this.scanStatus) {
+                        console.log(`📄 Status change: ${this.scanStatus} → ${data.status}`);
+                        this.handleStatusChange(data.status, data.task_queue_size);
+                    }
+                    this.scanStatus = data.status;
+                    
+                    // Update status display
+                    this.updateStatusDisplay(data);
+                    
+                    // FIXED: Always update graph if data is present and graph manager exists
+                    if (data.graph && this.graphManager) {
+                        console.log('📊 Processing graph update:', {
+                            nodes: data.graph.nodes?.length || 0,
+                            edges: data.graph.edges?.length || 0,
+                            hasNodes: Array.isArray(data.graph.nodes),
+                            hasEdges: Array.isArray(data.graph.edges),
+                            isInitialized: this.graphManager.isInitialized
+                        });
+                        
+                        // FIXED: Initialize graph manager if not already done
+                        if (!this.graphManager.isInitialized) {
+                            console.log('🎯 Initializing graph manager...');
+                            this.graphManager.initialize();
+                        }
+                        
+                        // FIXED: Force graph update and verify it worked
+                        const previousNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
+                        const previousEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
+                        
+                        console.log('🔄 Before update - Nodes:', previousNodeCount, 'Edges:', previousEdgeCount);
+                        
+                        // Store the data for debugging
+                        this.lastGraphData = data.graph;
+                        
+                        // Update the graph
+                        this.graphManager.updateGraph(data.graph);
+                        this.lastGraphUpdate = Date.now();
+                        
+                        // Verify the update worked
+                        const newNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
+                        const newEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
+                        
+                        console.log('🔄 After update - Nodes:', newNodeCount, 'Edges:', newEdgeCount);
+                        
+                        if (newNodeCount !== data.graph.nodes.length || newEdgeCount !== data.graph.edges.length) {
+                            console.warn('⚠️ Graph update mismatch!', {
+                                expectedNodes: data.graph.nodes.length,
+                                actualNodes: newNodeCount,
+                                expectedEdges: data.graph.edges.length,
+                                actualEdges: newEdgeCount
+                            });
+                            
+                            // Force a complete rebuild if there's a mismatch
+                            console.log('🔧 Force rebuilding graph...');
+                            this.graphManager.clear();
+                            this.graphManager.updateGraph(data.graph);
+                        }
+                        
+                        console.log('✅ Graph updated successfully');
+                        
+                        // FIXED: Force network redraw if we're using vis.js
+                        if (this.graphManager.network) {
+                            try {
+                                this.graphManager.network.redraw();
+                                console.log('🎨 Network redrawn');
+                            } catch (redrawError) {
+                                console.warn('⚠️ Network redraw failed:', redrawError);
+                            }
+                        }
+                        
+                    } else {
+                        if (!data.graph) {
+                            console.log('⚠️ No graph data in WebSocket update');
+                        }
+                        if (!this.graphManager) {
+                            console.log('⚠️ Graph manager not available');
+                        }
+                    }
+                    
+                } catch (error) {
+                    console.error('❌ Error processing WebSocket update:', error);
+                    console.error('Update data:', data);
+                    console.error('Stack trace:', error.stack);
+                }
+            });
+
+            this.socket.on('error', (error) => {
+                console.error('❌ WebSocket error:', error);
+                this.showError('WebSocket communication error');
+            });
+
+        } catch (error) {
+            console.error('❌ Failed to initialize WebSocket:', error);
+            this.showError('Failed to establish real-time connection');
+        }
+    }
         
     /**
      * Initialize DOM element references
@@ -62,6 +224,8 @@ class DNSReconApp {
             exportModal: document.getElementById('export-modal'),
             exportModalClose: document.getElementById('export-modal-close'),
             exportGraphJson: document.getElementById('export-graph-json'),
+            exportTargetsTxt: document.getElementById('export-targets-txt'),
+            exportExecutiveSummary: document.getElementById('export-executive-summary'),
             configureSettings: document.getElementById('configure-settings'),
             
             // Status elements
@@ -165,6 +329,12 @@ class DNSReconApp {
             if (this.elements.exportGraphJson) {
                 this.elements.exportGraphJson.addEventListener('click', () => this.exportGraphJson());
             }
+            if (this.elements.exportTargetsTxt) {
+                this.elements.exportTargetsTxt.addEventListener('click', () => this.exportTargetsTxt());
+            }
+            if (this.elements.exportExecutiveSummary) {
+                this.elements.exportExecutiveSummary.addEventListener('click', () => this.exportExecutiveSummary());
+            }
 
 
             this.elements.configureSettings.addEventListener('click', () => this.showSettingsModal());
@@ -255,12 +425,36 @@ class DNSReconApp {
     }
     
     /**
-     * Initialize graph visualization
+     * FIXED: Initialize graph visualization with enhanced debugging
      */
     initializeGraph() {
         try {
             console.log('Initializing graph manager...');
             this.graphManager = new GraphManager('network-graph');
+            
+            // FIXED: Add debugging hooks to graph manager
+            if (this.graphManager) {
+                // Override updateGraph to add debugging
+                const originalUpdateGraph = this.graphManager.updateGraph.bind(this.graphManager);
+                this.graphManager.updateGraph = (graphData) => {
+                    console.log('🔧 GraphManager.updateGraph called with:', {
+                        nodes: graphData?.nodes?.length || 0,
+                        edges: graphData?.edges?.length || 0,
+                        timestamp: new Date().toISOString()
+                    });
+                    
+                    const result = originalUpdateGraph(graphData);
+                    
+                    console.log('🔧 GraphManager.updateGraph completed, network state:', {
+                        networkExists: !!this.graphManager.network,
+                        nodeDataSetLength: this.graphManager.nodes?.length || 0,
+                        edgeDataSetLength: this.graphManager.edges?.length || 0
+                    });
+                    
+                    return result;
+                };
+            }
+            
             console.log('Graph manager initialized successfully');
         } catch (error) {
             console.error('Failed to initialize graph manager:', error);
@@ -280,7 +474,6 @@ class DNSReconApp {
             
             console.log(`Target: "${target}", Max depth: ${maxDepth}`);
             
-            // Validation
             if (!target) {
                 console.log('Validation failed: empty target');
                 this.showError('Please enter a target domain or IP');
@@ -295,6 +488,19 @@ class DNSReconApp {
                 return;
             }
             
+            // FIXED: Ensure WebSocket connection before starting scan
+            if (!this.isConnected) {
+                console.log('WebSocket not connected, attempting to connect...');
+                this.socket.connect();
+                
+                // Wait a moment for connection
+                await new Promise(resolve => setTimeout(resolve, 1000));
+                
+                if (!this.isConnected) {
+                    this.showWarning('WebSocket connection not established. Updates may be delayed.');
+                }
+            }
+            
             console.log('Validation passed, setting UI state to scanning...');
             this.setUIState('scanning');
             this.showInfo('Starting reconnaissance scan...');
@@ -312,23 +518,28 @@ class DNSReconApp {
                         
             if (response.success) {
                 this.currentSessionId = response.scan_id;
-                this.showSuccess('Reconnaissance scan started successfully');
+                this.showSuccess('Reconnaissance scan started - watching for real-time updates');
 
-                if (clearGraph) {
+                if (clearGraph && this.graphManager) {
+                    console.log('🧹 Clearing graph for new scan');
                     this.graphManager.clear();
                 }
                 
-                console.log(`Scan started for ${target} with depth ${maxDepth}`);
+                console.log(`✅ Scan started for ${target} with depth ${maxDepth}`);
                 
-                // Start polling immediately with faster interval for responsiveness
+                // FIXED: Immediately start listening for updates
-                this.startPolling(1000);
+                if (this.socket && this.isConnected) {
-                
+                    console.log('📡 Requesting initial status update...');
-                // Force an immediate status update
+                    this.socket.emit('get_status');
-                console.log('Forcing immediate status update...');
+                    
-                setTimeout(() => {
+                    // Set up periodic status requests as backup (every 5 seconds during scan)
-                    this.updateStatus();
+                    /*this.statusRequestInterval = setInterval(() => {
-                    this.updateGraph();
+                        if (this.isScanning && this.socket && this.isConnected) {
-                }, 100);
+                            console.log('📡 Periodic status request...');
+                            this.socket.emit('get_status');
+                        }
+                    }, 5000);*/
+                }
                 
             } else {
                 throw new Error(response.error || 'Failed to start scan');
@@ -340,20 +551,23 @@ class DNSReconApp {
             this.setUIState('idle');
         }
     }
-    /**
+
-     * Scan stop with immediate UI feedback
+    // FIXED: Enhanced stop scan with interval cleanup
-     */
     async stopScan() {
         try {
             console.log('Stopping scan...');
             
-            // Immediately disable stop button and show stopping state
+            // Clear status request interval
+            /*if (this.statusRequestInterval) {
+                clearInterval(this.statusRequestInterval);
+                this.statusRequestInterval = null;
+            }*/
+            
             if (this.elements.stopScan) {
                 this.elements.stopScan.disabled = true;
                 this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOPPING]</span><span>Stopping...</span>';
             }
             
-            // Show immediate feedback
             this.showInfo('Stopping scan...');
             
             const response = await this.apiCall('/api/scan/stop', 'POST');
@@ -361,21 +575,10 @@ class DNSReconApp {
             if (response.success) {
                 this.showSuccess('Scan stop requested');
                 
-                // Force immediate status update
+                // Request final status update
-                setTimeout(() => {
+                if (this.socket && this.isConnected) {
-                    this.updateStatus();
+                    setTimeout(() => this.socket.emit('get_status'), 500);
-                }, 100);
+                }
-                
-                // Continue polling for a bit to catch the status change
-                this.startPolling(500); // Fast polling to catch status change
-                
-                // Stop fast polling after 10 seconds
-                setTimeout(() => {
-                    if (this.scanStatus === 'stopped' || this.scanStatus === 'idle') {
-                        this.stopPolling();
-                    }
-                }, 10000);
-                
             } else {
                 throw new Error(response.error || 'Failed to stop scan');
             }
@@ -384,7 +587,6 @@ class DNSReconApp {
             console.error('Failed to stop scan:', error);
             this.showError(`Failed to stop scan: ${error.message}`);
             
-            // Re-enable stop button on error
             if (this.elements.stopScan) {
                 this.elements.stopScan.disabled = false;
                 this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
@@ -485,87 +687,80 @@ class DNSReconApp {
             }
         }
     }
-    
+
-    /**
+    async exportTargetsTxt() {
-     * Start polling for scan updates with configurable interval
+        await this.exportFile('/api/export/targets', this.elements.exportTargetsTxt, 'Exporting Targets...');
-     */
-    startPolling(interval = 2000) {
-        console.log('=== STARTING POLLING ===');
-        
-        if (this.pollInterval) {
-            console.log('Clearing existing poll interval');
-            clearInterval(this.pollInterval);
-        }
-        
-        this.pollInterval = setInterval(() => {
-            this.updateStatus();
-            this.updateGraph();
-            this.loadProviders();
-        }, interval);
-        
-        console.log(`Polling started with ${interval}ms interval`);
     }
-    
+
-    /**
+    async exportExecutiveSummary() {
-     * Stop polling for updates
+        await this.exportFile('/api/export/summary', this.elements.exportExecutiveSummary, 'Generating Summary...');
-     */
-    stopPolling() {
-        console.log('=== STOPPING POLLING ===');
-        if (this.pollInterval) {
-            clearInterval(this.pollInterval);
-            this.pollInterval = null;
-        }
     }
-    
+
-    /**
+    async exportFile(endpoint, buttonElement, loadingMessage) {
-     * Status update with better error handling
-     */
-    async updateStatus() {
         try {
-            const response = await this.apiCall('/api/scan/status');
+            console.log(`Exporting from ${endpoint}...`);
-            
+
-            
+            const originalContent = buttonElement.innerHTML;
-            if (response.success && response.status) {
+            buttonElement.innerHTML = `<span class="btn-icon">[...]</span><span>${loadingMessage}</span>`;
-                const status = response.status;
+            buttonElement.disabled = true;
-                
+
-                this.updateStatusDisplay(status);
+            const response = await fetch(endpoint, { method: 'GET' });
-                
+
-                // Handle status changes
+            if (!response.ok) {
-                if (status.status !== this.scanStatus) {
+                const errorData = await response.json().catch(() => ({}));
-                    console.log(`*** STATUS CHANGED: ${this.scanStatus} -> ${status.status} ***`);
+                throw new Error(errorData.error || `HTTP ${response.status}: ${response.statusText}`);
-                    this.handleStatusChange(status.status, status.task_queue_size);
+            }
+
+            const contentDisposition = response.headers.get('content-disposition');
+            let filename = 'export.txt';
+            if (contentDisposition) {
+                const filenameMatch = contentDisposition.match(/filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/);
+                if (filenameMatch) {
+                    filename = filenameMatch[1].replace(/['"]/g, '');
                 }
-                
-                this.scanStatus = status.status;
-            } else {
-                console.error('Status update failed:', response);
-                // Don't show error for status updates to avoid spam
             }
             
+            const blob = await response.blob();
+            const url = window.URL.createObjectURL(blob);
+            const link = document.createElement('a');
+            link.href = url;
+            link.download = filename;
+            document.body.appendChild(link);
+            link.click();
+            document.body.removeChild(link);
+            window.URL.revokeObjectURL(url);
+            
+            this.showSuccess('File exported successfully');
+            this.hideExportModal();
         } catch (error) {
-            console.error('Failed to update status:', error);
+            console.error(`Failed to export from ${endpoint}:`, error);
-            this.showConnectionError();
+            this.showError(`Export failed: ${error.message}`);
+        } finally {
+            const originalContent = buttonElement._originalContent || buttonElement.innerHTML;
+            buttonElement.innerHTML = originalContent;
+            buttonElement.disabled = false;
         }
     }
     
     /**
-     * Update graph from server
+     * FIXED: Update graph from server with enhanced debugging
      */
     async updateGraph() {
         try {
-            console.log('Updating graph...');
+            console.log('Updating graph via API call...');
             const response = await this.apiCall('/api/graph');
             
             
             if (response.success) {
                 const graphData = response.graph;
                 
-                console.log('Graph data received:');
+                console.log('Graph data received from API:');
                 console.log('- Nodes:', graphData.nodes ? graphData.nodes.length : 0);
                 console.log('- Edges:', graphData.edges ? graphData.edges.length : 0);
                 
                 // FIXED: Always update graph, even if empty - let GraphManager handle placeholder
                 if (this.graphManager) {
+                    console.log('🔧 Calling GraphManager.updateGraph from API response...');
                     this.graphManager.updateGraph(graphData);
                     this.lastGraphUpdate = Date.now();
                     
@@ -574,6 +769,8 @@ class DNSReconApp {
                     if (this.elements.relationshipsDisplay) {
                         this.elements.relationshipsDisplay.textContent = edgeCount;
                     }
+                    
+                    console.log('✅ Manual graph update completed');
                 }
             } else {
                 console.error('Graph update failed:', response);
@@ -669,48 +866,70 @@ class DNSReconApp {
      * @param {string} newStatus - New scan status
      */
     handleStatusChange(newStatus, task_queue_size) {
-        console.log(`=== STATUS CHANGE: ${this.scanStatus} -> ${newStatus} ===`);
+        console.log(`📄 Status change handler: ${this.scanStatus} → ${newStatus}`);
         
         switch (newStatus) {
             case 'running':
                 this.setUIState('scanning', task_queue_size);
-                this.showSuccess('Scan is running');
+                this.showSuccess('Scan is running - updates in real-time');
-                // Increase polling frequency for active scans
-                this.startPolling(1000); // Poll every 1 second for running scans
                 this.updateConnectionStatus('active');
                 break;
                 
             case 'completed':
                 this.setUIState('completed', task_queue_size);
-                this.stopPolling();
                 this.showSuccess('Scan completed successfully');
                 this.updateConnectionStatus('completed');
                 this.loadProviders();
-                // Force a final graph update
+                console.log('✅ Scan completed - requesting final graph update');
-                console.log('Scan completed - forcing final graph update');
+                // Request final status to ensure we have the complete graph
-                setTimeout(() => this.updateGraph(), 100);
+                setTimeout(() => {
+                    if (this.socket && this.isConnected) {
+                        this.socket.emit('get_status');
+                    }
+                }, 1000);
+                
+                // Clear status request interval
+                /*if (this.statusRequestInterval) {
+                    clearInterval(this.statusRequestInterval);
+                    this.statusRequestInterval = null;
+                }*/
                 break;
                 
             case 'failed':
                 this.setUIState('failed', task_queue_size);
-                this.stopPolling();
                 this.showError('Scan failed');
                 this.updateConnectionStatus('error');
                 this.loadProviders();
+                
+                // Clear status request interval
+                /*if (this.statusRequestInterval) {
+                    clearInterval(this.statusRequestInterval);
+                    this.statusRequestInterval = null;
+                }*/
                 break;
                 
             case 'stopped':
                 this.setUIState('stopped', task_queue_size);
-                this.stopPolling();
                 this.showSuccess('Scan stopped');
                 this.updateConnectionStatus('stopped');
                 this.loadProviders();
+                
+                // Clear status request interval
+                if (this.statusRequestInterval) {
+                    clearInterval(this.statusRequestInterval);
+                    this.statusRequestInterval = null;
+                }
                 break;
                 
             case 'idle':
                 this.setUIState('idle', task_queue_size);
-                this.stopPolling();
                 this.updateConnectionStatus('idle');
+                
+                // Clear status request interval
+                /*if (this.statusRequestInterval) {
+                    clearInterval(this.statusRequestInterval);
+                    this.statusRequestInterval = null;
+                }*/
                 break;
                 
             default:
@@ -762,6 +981,7 @@ class DNSReconApp {
                 if (this.graphManager) {
                     this.graphManager.isScanning = true;
                 }
+                
                 if (this.elements.startScan) {
                     this.elements.startScan.disabled = true;
                     this.elements.startScan.classList.add('loading');
@@ -789,6 +1009,7 @@ class DNSReconApp {
                 if (this.graphManager) {
                     this.graphManager.isScanning = false;
                 }
+                
                 if (this.elements.startScan) {
                     this.elements.startScan.disabled = !isQueueEmpty;
                     this.elements.startScan.classList.remove('loading');
@@ -1031,7 +1252,7 @@ class DNSReconApp {
                 } else {
                     // API key not configured - ALWAYS show input field
                     const statusClass = info.enabled ? 'enabled' : 'api-key-required';
-                    const statusText = info.enabled ? '○ Ready for API Key' : '⚠️ API Key Required';
+                    const statusText = info.enabled ? '◯ Ready for API Key' : '⚠️ API Key Required';
                     
                     inputGroup.innerHTML = `
                         <div class="provider-header">
@@ -1335,28 +1556,62 @@ class DNSReconApp {
     }
 
     /**
-     * UPDATED: Generate details for standard nodes with organized attribute grouping
+     * UPDATED: Generate details for standard nodes with organized attribute grouping and data warnings
      */
     generateStandardNodeDetails(node) {
         let html = '';
-        
+
+        // Check for and display a crt.sh data warning if it exists
+        const crtshWarningAttr = this.findAttributeByName(node.attributes, 'crtsh_data_warning');
+        if (crtshWarningAttr) {
+            html += `
+                <div class="modal-section" style="border-left: 3px solid #ff9900; background: rgba(255, 153, 0, 0.05);">
+                    <details open>
+                        <summary style="color: #ff9900;">
+                            <span>⚠️ Data Integrity Warning</span>
+                        </summary>
+                        <div class="modal-section-content">
+                            <p class="placeholder-subtext" style="color: #e0e0e0; font-size: 0.8rem; line-height: 1.5;">
+                                ${this.escapeHtml(crtshWarningAttr.value)}
+                                <br><br>
+                                This can occur for very large domains (e.g., google.com) where crt.sh may return a limited subset of all available certificates. As a result, the certificate status may not be fully representative.
+                            </p>
+                        </div>
+                    </details>
+                </div>
+            `;
+        }
+
         // Relationships sections
         html += this.generateRelationshipsSection(node);
-        
+
         // UPDATED: Enhanced attributes section with intelligent grouping (no formatting)
         if (node.attributes && Array.isArray(node.attributes) && node.attributes.length > 0) {
             html += this.generateOrganizedAttributesSection(node.attributes, node.type);
         }
-        
+
         // Description section
         html += this.generateDescriptionSection(node);
-        
+
         // Metadata section (collapsed by default)
         html += this.generateMetadataSection(node);
-        
+
         return html;
     }
 
+    /**
+     * Helper method to find an attribute by name in the standardized attributes list
+     * @param {Array} attributes - List of StandardAttribute objects
+     * @param {string} name - Attribute name to find
+     * @returns {Object|null} The attribute object if found, null otherwise
+     */
+    findAttributeByName(attributes, name) {
+        if (!Array.isArray(attributes)) {
+            return null;
+        }
+        return attributes.find(attr => attr.name === name) || null;
+    }
+
     generateOrganizedAttributesSection(attributes, nodeType) {
         if (!Array.isArray(attributes) || attributes.length === 0) {
             return '';
@@ -1547,15 +1802,19 @@ class DNSReconApp {
      * UPDATED: Enhanced correlation details showing the correlated attribute clearly (no formatting)
      */
     generateCorrelationDetails(node) {
-        const metadata = node.metadata || {};
+        const attributes = node.attributes || [];
-        const value = metadata.value;
+        const correlationValueAttr = attributes.find(attr => attr.name === 'correlation_value');
+        const value = correlationValueAttr ? correlationValueAttr.value : 'Unknown';
+
+        const metadataAttr = attributes.find(attr => attr.name === 'correlation_value');
+        const metadata = metadataAttr ? metadataAttr.metadata : {};
         const correlatedNodes = metadata.correlated_nodes || [];
         const sources = metadata.sources || [];
         
         let html = '';
         
         // Show what attribute is being correlated (raw names)
-        const primarySource = metadata.primary_source || 'unknown';
+        const primarySource = sources.length > 0 ? sources[0].attribute : 'unknown';
         
         html += `
             <div class="modal-section">
@@ -1931,14 +2190,12 @@ class DNSReconApp {
             if (response.success) {
                 this.showSuccess(response.message);
                 
-                this.hideModal();
-
                 // If the scanner was idle, it's now running. Start polling to see the new node appear.
                 if (this.scanStatus === 'idle') {
-                    this.startPolling(1000);
+                    this.socket.emit('get_status');
                 } else {
                     // If already scanning, force a quick graph update to see the change sooner.
-                    setTimeout(() => this.updateGraph(), 500);
+                    setTimeout(() => this.socket.emit('get_status'), 500);
                 }
 
             } else {
@@ -1977,8 +2234,8 @@ class DNSReconApp {
      */
     getNodeTypeIcon(nodeType) {
         const icons = {
-            'domain': '🌍',
+            'domain': '🌐',
-            'ip': '📍',
+            'ip': '🔢',
             'asn': '🏢',
             'large_entity': '📦',
             'correlation_object': '🔗'

templates/index.html

@@ -7,6 +7,7 @@
     <title>DNSRecon - Infrastructure Reconnaissance</title>
     <link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
     <script src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.7.2/socket.io.js"></script>
     <link href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" rel="stylesheet" type="text/css">
     <link
         href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@300;400;500;700&family=Special+Elite&display=swap"
@@ -194,7 +195,6 @@
             </div>
         </div>
 
-        <!-- Settings Modal -->
         <div id="settings-modal" class="modal">
             <div class="modal-content">
                 <div class="modal-header">
@@ -203,7 +203,6 @@
                 </div>
                 <div class="modal-body">
                     <div class="modal-details">
-                        <!-- Scan Settings Section -->
                         <section class="modal-section">
                             <details open>
                                 <summary>
@@ -224,7 +223,6 @@
                             </details>
                         </section>
 
-                        <!-- Provider Configuration Section -->
                         <section class="modal-section">
                             <details open>
                                 <summary>
@@ -233,13 +231,11 @@
                                 </summary>
                                 <div class="modal-section-content">
                                     <div id="provider-config-list">
-                                        <!-- Dynamically populated -->
+                                        </div>
-                                    </div>
                                 </div>
                             </details>
                         </section>
 
-                        <!-- API Keys Section -->
                         <section class="modal-section">
                             <details>
                                 <summary>
@@ -252,13 +248,11 @@
                                         Only provide API keys you don't use for anything else.
                                     </p>
                                     <div id="api-key-inputs">
-                                        <!-- Dynamically populated -->
+                                        </div>
-                                    </div>
                                 </div>
                             </details>
                         </section>
 
-                        <!-- Action Buttons -->
                         <div class="button-group" style="margin-top: 1.5rem;">
                             <button id="save-settings" class="btn btn-primary">
                                 <span class="btn-icon">[SAVE]</span>
@@ -274,7 +268,6 @@
             </div>
         </div>
 
-        <!-- Export Modal -->
         <div id="export-modal" class="modal">
             <div class="modal-content">
                 <div class="modal-header">
@@ -299,6 +292,20 @@
                                                 provider statistics, and scan metadata in JSON format for analysis and
                                                 archival.</span>
                                         </div>
+                                        <button id="export-targets-txt" class="btn btn-primary" style="margin-top: 1rem;">
+                                            <span class="btn-icon">[TXT]</span>
+                                            <span>Export Targets</span>
+                                        </button>
+                                        <div class="status-row" style="margin-top: 0.5rem;">
+                                            <span class="status-label">A simple text file containing all discovered domains and IP addresses.</span>
+                                        </div>
+                                        <button id="export-executive-summary" class="btn btn-primary" style="margin-top: 1rem;">
+                                            <span class="btn-icon">[TXT]</span>
+                                            <span>Export Executive Summary</span>
+                                        </button>
+                                        <div class="status-row" style="margin-top: 0.5rem;">
+                                            <span class="status-label">A natural-language summary of the scan findings.</span>
+                                        </div>
                                     </div>
                                 </div>
                             </details>

utils/__init__.py

@@ -0,0 +1,22 @@
+# dnsrecon-reduced/utils/__init__.py
+
+"""
+Utility modules for DNSRecon.
+Contains helper functions, export management, and supporting utilities.
+"""
+
+from .helpers import is_valid_target, _is_valid_domain, _is_valid_ip, get_ip_version, normalize_ip
+from .export_manager import export_manager, ExportManager, CustomJSONEncoder
+
+__all__ = [
+    'is_valid_target',
+    '_is_valid_domain', 
+    '_is_valid_ip',
+    'get_ip_version',
+    'normalize_ip',
+    'export_manager',
+    'ExportManager',
+    'CustomJSONEncoder'
+]
+
+__version__ = "1.0.0"

utils/export_manager.py

@@ -0,0 +1,849 @@
+# dnsrecon-reduced/utils/export_manager.py
+
+"""
+Centralized export functionality for DNSRecon.
+Handles all data export operations with forensic integrity and proper formatting.
+ENHANCED: Professional forensic executive summary generation for court-ready documentation.
+"""
+
+import json
+from datetime import datetime, timezone
+from typing import Dict, Any, List, Optional, Set, Tuple
+from decimal import Decimal
+from collections import defaultdict, Counter
+import networkx as nx
+
+from utils.helpers import _is_valid_domain, _is_valid_ip
+
+
+class ExportManager:
+    """
+    Centralized manager for all DNSRecon export operations.
+    Maintains forensic integrity and provides consistent export formats.
+    ENHANCED: Advanced forensic analysis and professional reporting capabilities.
+    """
+    
+    def __init__(self):
+        """Initialize export manager."""
+        pass
+    
+    def export_scan_results(self, scanner) -> Dict[str, Any]:
+        """
+        Export complete scan results with forensic metadata.
+        
+        Args:
+            scanner: Scanner instance with completed scan data
+            
+        Returns:
+            Complete scan results dictionary
+        """
+        graph_data = self.export_graph_json(scanner.graph)
+        audit_trail = scanner.logger.export_audit_trail()
+        provider_stats = {}
+        
+        for provider in scanner.providers:
+            provider_stats[provider.get_name()] = provider.get_statistics()
+        
+        results = {
+            'scan_metadata': {
+                'target_domain': scanner.current_target,
+                'max_depth': scanner.max_depth,
+                'final_status': scanner.status,
+                'total_indicators_processed': scanner.indicators_processed,
+                'enabled_providers': list(provider_stats.keys()),
+                'session_id': scanner.session_id
+            },
+            'graph_data': graph_data,
+            'forensic_audit': audit_trail,
+            'provider_statistics': provider_stats,
+            'scan_summary': scanner.logger.get_forensic_summary()
+        }
+        
+        # Add export metadata
+        results['export_metadata'] = {
+            'export_timestamp': datetime.now(timezone.utc).isoformat(),
+            'export_version': '1.0.0',
+            'forensic_integrity': 'maintained'
+        }
+        
+        return results
+    
+    def export_targets_list(self, scanner) -> str:
+        """
+        Export all discovered domains and IPs as a text file.
+        
+        Args:
+            scanner: Scanner instance with graph data
+            
+        Returns:
+            Newline-separated list of targets
+        """
+        nodes = scanner.graph.get_graph_data().get('nodes', [])
+        targets = {
+            node['id'] for node in nodes 
+            if _is_valid_domain(node['id']) or _is_valid_ip(node['id'])
+        }
+        return "\n".join(sorted(list(targets)))
+    
+    def generate_executive_summary(self, scanner) -> str:
+        """
+        ENHANCED: Generate a comprehensive, court-ready forensic executive summary.
+        
+        Args:
+            scanner: Scanner instance with completed scan data
+            
+        Returns:
+            Professional forensic summary formatted for investigative use
+        """
+        report = []
+        now = datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
+        
+        # Get comprehensive data for analysis
+        graph_data = scanner.graph.get_graph_data()
+        nodes = graph_data.get('nodes', [])
+        edges = graph_data.get('edges', [])
+        audit_trail = scanner.logger.export_audit_trail()
+        
+        # Perform advanced analysis
+        infrastructure_analysis = self._analyze_infrastructure_patterns(nodes, edges)
+        
+        # === HEADER AND METADATA ===
+        report.extend([
+            "=" * 80,
+            "DIGITAL INFRASTRUCTURE RECONNAISSANCE REPORT",
+            "=" * 80,
+            "",
+            f"Report Generated: {now}",
+            f"Investigation Target: {scanner.current_target}",
+            f"Analysis Session: {scanner.session_id}",
+            f"Scan Depth: {scanner.max_depth} levels",
+            f"Final Status: {scanner.status.upper()}",
+            ""
+        ])
+        
+        # === EXECUTIVE SUMMARY ===
+        report.extend([
+            "EXECUTIVE SUMMARY",
+            "-" * 40,
+            "",
+            f"This report presents the findings of a comprehensive passive reconnaissance analysis "
+            f"conducted against the target '{scanner.current_target}'. The investigation employed "
+            f"multiple intelligence sources and discovered {len(nodes)} distinct digital entities "
+            f"connected through {len(edges)} verified relationships.",
+            "",
+            f"The analysis reveals a digital infrastructure comprising {infrastructure_analysis['domains']} "
+            f"domain names, {infrastructure_analysis['ips']} IP addresses, and {infrastructure_analysis['isps']} "
+            f"infrastructure service providers. Certificate transparency analysis identified "
+            f"{infrastructure_analysis['cas']} certificate authorities managing the cryptographic "
+            f"infrastructure for the investigated entities.",
+            "",
+        ])
+        
+        # === METHODOLOGY ===
+        report.extend([
+            "INVESTIGATIVE METHODOLOGY",
+            "-" * 40,
+            "",
+            "This analysis employed passive reconnaissance techniques using the following verified data sources:",
+            ""
+        ])
+        
+        provider_info = {
+            'dns': 'Standard DNS resolution and reverse DNS lookups',
+            'crtsh': 'Certificate Transparency database analysis via crt.sh',
+            'shodan': 'Internet-connected device intelligence via Shodan API'
+        }
+        
+        for provider in scanner.providers:
+            provider_name = provider.get_name()
+            stats = provider.get_statistics()
+            description = provider_info.get(provider_name, f'{provider_name} data provider')
+            
+            report.extend([
+                f"• {provider.get_display_name()}: {description}",
+                f"  - Total Requests: {stats['total_requests']}",
+                f"  - Success Rate: {stats['success_rate']:.1f}%",
+                f"  - Relationships Discovered: {stats['relationships_found']}",
+                ""
+            ])
+        
+        # === INFRASTRUCTURE ANALYSIS ===
+        report.extend([
+            "INFRASTRUCTURE ANALYSIS",
+            "-" * 40,
+            ""
+        ])
+        
+        # Domain Analysis
+        if infrastructure_analysis['domains'] > 0:
+            report.extend([
+                f"Domain Name Infrastructure ({infrastructure_analysis['domains']} entities):",
+                ""
+            ])
+            
+            domain_details = self._get_detailed_domain_analysis(nodes, edges)
+            for domain_info in domain_details[:10]:  # Top 10 domains
+                report.extend([
+                    f"• {domain_info['domain']}",
+                    f"  - Type: {domain_info['classification']}",
+                    f"  - Connected IPs: {len(domain_info['ips'])}",
+                    f"  - Certificate Status: {domain_info['cert_status']}",
+                    f"  - Relationship Confidence: {domain_info['avg_confidence']:.2f}",
+                ])
+                
+                if domain_info['security_notes']:
+                    report.extend([
+                        f"  - Security Notes: {', '.join(domain_info['security_notes'])}",
+                    ])
+                report.append("")
+        
+        # IP Address Analysis
+        if infrastructure_analysis['ips'] > 0:
+            report.extend([
+                f"IP Address Infrastructure ({infrastructure_analysis['ips']} entities):",
+                ""
+            ])
+            
+            ip_details = self._get_detailed_ip_analysis(nodes, edges)
+            for ip_info in ip_details[:8]:  # Top 8 IPs
+                report.extend([
+                    f"• {ip_info['ip']} ({ip_info['version']})",
+                    f"  - Associated Domains: {len(ip_info['domains'])}",
+                    f"  - ISP: {ip_info['isp'] or 'Unknown'}",
+                    f"  - Geographic Location: {ip_info['location'] or 'Not determined'}",
+                ])
+                
+                if ip_info['open_ports']:
+                    report.extend([
+                        f"  - Exposed Services: {', '.join(map(str, ip_info['open_ports'][:5]))}"
+                        + (f" (and {len(ip_info['open_ports']) - 5} more)" if len(ip_info['open_ports']) > 5 else ""),
+                    ])
+                report.append("")
+        
+        # === RELATIONSHIP ANALYSIS ===
+        report.extend([
+            "ENTITY RELATIONSHIP ANALYSIS",
+            "-" * 40,
+            ""
+        ])
+        
+        # Network topology insights
+        topology = self._analyze_network_topology(nodes, edges)
+        report.extend([
+            f"Network Topology Assessment:",
+            f"• Central Hubs: {len(topology['hubs'])} entities serve as primary connection points",
+            f"• Isolated Clusters: {len(topology['clusters'])} distinct groupings identified",
+            f"• Relationship Density: {topology['density']:.3f} (0=sparse, 1=fully connected)",
+            f"• Average Path Length: {topology['avg_path_length']:.2f} degrees of separation",
+            ""
+        ])
+        
+        # Key relationships
+        key_relationships = self._identify_key_relationships(edges)
+        if key_relationships:
+            report.extend([
+                "Critical Infrastructure Relationships:",
+                ""
+            ])
+            
+            for rel in key_relationships[:8]:  # Top 8 relationships
+                confidence_desc = self._describe_confidence(rel['confidence'])
+                report.extend([
+                    f"• {rel['source']} → {rel['target']}",
+                    f"  - Relationship: {self._humanize_relationship_type(rel['type'])}",
+                    f"  - Evidence Strength: {confidence_desc} ({rel['confidence']:.2f})",
+                    f"  - Discovery Method: {rel['provider']}",
+                    ""
+                ])
+        
+        # === CERTIFICATE ANALYSIS ===
+        cert_analysis = self._analyze_certificate_infrastructure(nodes)
+        if cert_analysis['total_certs'] > 0:
+            report.extend([
+                "CERTIFICATE INFRASTRUCTURE ANALYSIS",
+                "-" * 40,
+                "",
+                f"Certificate Status Overview:",
+                f"• Total Certificates Analyzed: {cert_analysis['total_certs']}",
+                f"• Valid Certificates: {cert_analysis['valid']}",
+                f"• Expired/Invalid: {cert_analysis['expired']}",
+                f"• Certificate Authorities: {len(cert_analysis['cas'])}",
+                ""
+            ])
+            
+            if cert_analysis['cas']:
+                report.extend([
+                    "Certificate Authority Distribution:",
+                    ""
+                ])
+                for ca, count in cert_analysis['cas'].most_common(5):
+                    report.extend([
+                        f"• {ca}: {count} certificate(s)",
+                    ])
+                report.append("")
+        
+        
+        # === TECHNICAL APPENDIX ===
+        report.extend([
+            "TECHNICAL APPENDIX",
+            "-" * 40,
+            "",
+            "Data Quality Assessment:",
+            f"• Total API Requests: {audit_trail.get('session_metadata', {}).get('total_requests', 0)}",
+            f"• Data Providers Used: {len(audit_trail.get('session_metadata', {}).get('providers_used', []))}",
+            f"• Relationship Confidence Distribution:",
+        ])
+        
+        # Confidence distribution
+        confidence_dist = self._calculate_confidence_distribution(edges)
+        for level, count in confidence_dist.items():
+            percentage = (count / len(edges) * 100) if edges else 0
+            report.extend([
+                f"  - {level.title()} Confidence (≥{self._get_confidence_threshold(level)}): {count} ({percentage:.1f}%)",
+            ])
+        
+        report.extend([
+            "",
+            "Correlation Analysis:",
+            f"• Entity Correlations Identified: {len(scanner.graph.correlation_index)}",
+            f"• Cross-Reference Validation: {self._count_cross_validated_relationships(edges)} relationships verified by multiple sources",
+            ""
+        ])
+        
+        # === CONCLUSION ===
+        report.extend([
+            "CONCLUSION",
+            "-" * 40,
+            "",
+            self._generate_conclusion(scanner.current_target, infrastructure_analysis, 
+                                    len(edges)),
+            "",
+            "This analysis was conducted using passive reconnaissance techniques and represents "
+            "the digital infrastructure observable through public data sources at the time of investigation. "
+            "All findings are supported by verifiable technical evidence and documented through "
+            "a complete audit trail maintained for forensic integrity.",
+            "",
+            f"Investigation completed: {now}",
+            f"Report authenticated by: DNSRecon v{self._get_version()}",
+            "",
+            "=" * 80,
+            "END OF REPORT",
+            "=" * 80
+        ])
+        
+        return "\n".join(report)
+    
+    def _analyze_infrastructure_patterns(self, nodes: List[Dict], edges: List[Dict]) -> Dict[str, Any]:
+        """Analyze infrastructure patterns and classify entities."""
+        analysis = {
+            'domains': len([n for n in nodes if n['type'] == 'domain']),
+            'ips': len([n for n in nodes if n['type'] == 'ip']),
+            'isps': len([n for n in nodes if n['type'] == 'isp']),
+            'cas': len([n for n in nodes if n['type'] == 'ca']),
+            'correlations': len([n for n in nodes if n['type'] == 'correlation_object'])
+        }
+        return analysis
+    
+    def _get_detailed_domain_analysis(self, nodes: List[Dict], edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Generate detailed analysis for each domain."""
+        domain_nodes = [n for n in nodes if n['type'] == 'domain']
+        domain_analysis = []
+        
+        for domain in domain_nodes:
+            # Find connected IPs
+            connected_ips = [e['to'] for e in edges 
+                           if e['from'] == domain['id'] and _is_valid_ip(e['to'])]
+            
+            # Determine classification
+            classification = "Primary Domain"
+            if domain['id'].startswith('www.'):
+                classification = "Web Interface"
+            elif any(subdomain in domain['id'] for subdomain in ['api.', 'mail.', 'smtp.']):
+                classification = "Service Endpoint"
+            elif domain['id'].count('.') > 1:
+                classification = "Subdomain"
+            
+            # Certificate status
+            cert_status = self._determine_certificate_status(domain)
+            
+            # Security notes
+            security_notes = []
+            if cert_status == "Expired/Invalid":
+                security_notes.append("Certificate validation issues")
+            if len(connected_ips) == 0:
+                security_notes.append("No IP resolution found")
+            if len(connected_ips) > 5:
+                security_notes.append("Multiple IP endpoints")
+            
+            # Average confidence
+            domain_edges = [e for e in edges if e['from'] == domain['id']]
+            avg_confidence = sum(e['confidence_score'] for e in domain_edges) / len(domain_edges) if domain_edges else 0
+            
+            domain_analysis.append({
+                'domain': domain['id'],
+                'classification': classification,
+                'ips': connected_ips,
+                'cert_status': cert_status,
+                'security_notes': security_notes,
+                'avg_confidence': avg_confidence
+            })
+        
+        # Sort by number of connections (most connected first)
+        return sorted(domain_analysis, key=lambda x: len(x['ips']), reverse=True)
+    
+    def _get_detailed_ip_analysis(self, nodes: List[Dict], edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Generate detailed analysis for each IP address."""
+        ip_nodes = [n for n in nodes if n['type'] == 'ip']
+        ip_analysis = []
+        
+        for ip in ip_nodes:
+            # Find connected domains
+            connected_domains = [e['from'] for e in edges 
+                               if e['to'] == ip['id'] and _is_valid_domain(e['from'])]
+            
+            # Extract metadata from attributes
+            ip_version = "IPv4"
+            location = None
+            isp = None
+            open_ports = []
+            
+            for attr in ip.get('attributes', []):
+                if attr.get('name') == 'country':
+                    location = attr.get('value')
+                elif attr.get('name') == 'org':
+                    isp = attr.get('value')
+                elif attr.get('name') == 'shodan_open_port':
+                    open_ports.append(attr.get('value'))
+                elif 'ipv6' in str(attr.get('metadata', {})).lower():
+                    ip_version = "IPv6"
+            
+            # Find ISP from relationships
+            if not isp:
+                isp_edges = [e for e in edges if e['from'] == ip['id'] and e['label'].endswith('_isp')]
+                isp = isp_edges[0]['to'] if isp_edges else None
+            
+            ip_analysis.append({
+                'ip': ip['id'],
+                'version': ip_version,
+                'domains': connected_domains,
+                'isp': isp,
+                'location': location,
+                'open_ports': open_ports
+            })
+        
+        # Sort by number of connected domains
+        return sorted(ip_analysis, key=lambda x: len(x['domains']), reverse=True)
+    
+    def _analyze_network_topology(self, nodes: List[Dict], edges: List[Dict]) -> Dict[str, Any]:
+        """Analyze network topology and identify key structural patterns."""
+        if not nodes or not edges:
+            return {'hubs': [], 'clusters': [], 'density': 0, 'avg_path_length': 0}
+        
+        # Create NetworkX graph
+        G = nx.DiGraph()
+        for node in nodes:
+            G.add_node(node['id'])
+        for edge in edges:
+            G.add_edge(edge['from'], edge['to'])
+        
+        # Convert to undirected for certain analyses
+        G_undirected = G.to_undirected()
+        
+        # Identify hubs (nodes with high degree centrality)
+        centrality = nx.degree_centrality(G_undirected)
+        hub_threshold = max(centrality.values()) * 0.7 if centrality else 0
+        hubs = [node for node, cent in centrality.items() if cent >= hub_threshold]
+        
+        # Find connected components (clusters)
+        clusters = list(nx.connected_components(G_undirected))
+        
+        # Calculate density
+        density = nx.density(G_undirected)
+        
+        # Calculate average path length (for largest component)
+        if G_undirected.number_of_nodes() > 1:
+            largest_cc = max(nx.connected_components(G_undirected), key=len)
+            subgraph = G_undirected.subgraph(largest_cc)
+            try:
+                avg_path_length = nx.average_shortest_path_length(subgraph)
+            except:
+                avg_path_length = 0
+        else:
+            avg_path_length = 0
+        
+        return {
+            'hubs': hubs,
+            'clusters': clusters,
+            'density': density,
+            'avg_path_length': avg_path_length
+        }
+    
+    def _identify_key_relationships(self, edges: List[Dict]) -> List[Dict[str, Any]]:
+        """Identify the most significant relationships in the infrastructure."""
+        # Score relationships by confidence and type importance
+        relationship_importance = {
+            'dns_a_record': 0.9,
+            'dns_aaaa_record': 0.9,
+            'crtsh_cert_issuer': 0.8,
+            'shodan_isp': 0.8,
+            'crtsh_san_certificate': 0.7,
+            'dns_mx_record': 0.7,
+            'dns_ns_record': 0.7
+        }
+        
+        scored_edges = []
+        for edge in edges:
+            base_confidence = edge.get('confidence_score', 0)
+            type_weight = relationship_importance.get(edge.get('label', ''), 0.5)
+            combined_score = (base_confidence * 0.7) + (type_weight * 0.3)
+            
+            scored_edges.append({
+                'source': edge['from'],
+                'target': edge['to'],
+                'type': edge.get('label', ''),
+                'confidence': base_confidence,
+                'provider': edge.get('source_provider', ''),
+                'score': combined_score
+            })
+        
+        # Return top relationships by score
+        return sorted(scored_edges, key=lambda x: x['score'], reverse=True)
+    
+    def _analyze_certificate_infrastructure(self, nodes: List[Dict]) -> Dict[str, Any]:
+        """Analyze certificate infrastructure across all domains."""
+        domain_nodes = [n for n in nodes if n['type'] == 'domain']
+        ca_nodes = [n for n in nodes if n['type'] == 'ca']
+        
+        valid_certs = 0
+        expired_certs = 0
+        total_certs = 0
+        cas = Counter()
+        
+        for domain in domain_nodes:
+            for attr in domain.get('attributes', []):
+                if attr.get('name') == 'cert_is_currently_valid':
+                    total_certs += 1
+                    if attr.get('value') is True:
+                        valid_certs += 1
+                    else:
+                        expired_certs += 1
+                elif attr.get('name') == 'cert_issuer_name':
+                    issuer = attr.get('value')
+                    if issuer:
+                        cas[issuer] += 1
+        
+        return {
+            'total_certs': total_certs,
+            'valid': valid_certs,
+            'expired': expired_certs,
+            'cas': cas
+        }
+    
+    def _has_expired_certificates(self, domain_node: Dict) -> bool:
+        """Check if domain has expired certificates."""
+        for attr in domain_node.get('attributes', []):
+            if (attr.get('name') == 'cert_is_currently_valid' and 
+                attr.get('value') is False):
+                return True
+        return False
+    
+    def _determine_certificate_status(self, domain_node: Dict) -> str:
+        """Determine the certificate status for a domain."""
+        has_valid = False
+        has_expired = False
+        has_any = False
+        
+        for attr in domain_node.get('attributes', []):
+            if attr.get('name') == 'cert_is_currently_valid':
+                has_any = True
+                if attr.get('value') is True:
+                    has_valid = True
+                else:
+                    has_expired = True
+        
+        if not has_any:
+            return "No Certificate Data"
+        elif has_valid and not has_expired:
+            return "Valid"
+        elif has_expired and not has_valid:
+            return "Expired/Invalid"
+        else:
+            return "Mixed Status"
+    
+    def _describe_confidence(self, confidence: float) -> str:
+        """Convert confidence score to descriptive text."""
+        if confidence >= 0.9:
+            return "Very High"
+        elif confidence >= 0.8:
+            return "High"
+        elif confidence >= 0.6:
+            return "Medium"
+        elif confidence >= 0.4:
+            return "Low"
+        else:
+            return "Very Low"
+    
+    def _humanize_relationship_type(self, rel_type: str) -> str:
+        """Convert technical relationship types to human-readable descriptions."""
+        type_map = {
+            'dns_a_record': 'DNS A Record Resolution',
+            'dns_aaaa_record': 'DNS AAAA Record (IPv6) Resolution',
+            'dns_mx_record': 'Email Server (MX) Configuration',
+            'dns_ns_record': 'Name Server Delegation',
+            'dns_cname_record': 'DNS Alias (CNAME) Resolution',
+            'crtsh_cert_issuer': 'SSL Certificate Issuer Relationship',
+            'crtsh_san_certificate': 'Shared SSL Certificate',
+            'shodan_isp': 'Internet Service Provider Assignment',
+            'shodan_a_record': 'IP-to-Domain Resolution (Shodan)',
+            'dns_ptr_record': 'Reverse DNS Resolution'
+        }
+        return type_map.get(rel_type, rel_type.replace('_', ' ').title())
+    
+    def _calculate_confidence_distribution(self, edges: List[Dict]) -> Dict[str, int]:
+        """Calculate confidence score distribution."""
+        distribution = {'high': 0, 'medium': 0, 'low': 0}
+        
+        for edge in edges:
+            confidence = edge.get('confidence_score', 0)
+            if confidence >= 0.8:
+                distribution['high'] += 1
+            elif confidence >= 0.6:
+                distribution['medium'] += 1
+            else:
+                distribution['low'] += 1
+        
+        return distribution
+    
+    def _get_confidence_threshold(self, level: str) -> str:
+        """Get confidence threshold for a level."""
+        thresholds = {'high': '0.80', 'medium': '0.60', 'low': '0.00'}
+        return thresholds.get(level, '0.00')
+    
+    def _count_cross_validated_relationships(self, edges: List[Dict]) -> int:
+        """Count relationships verified by multiple providers."""
+        # Group edges by source-target pair
+        edge_pairs = defaultdict(list)
+        for edge in edges:
+            pair_key = f"{edge['from']}->{edge['to']}"
+            edge_pairs[pair_key].append(edge.get('source_provider', ''))
+        
+        # Count pairs with multiple providers
+        cross_validated = 0
+        for pair, providers in edge_pairs.items():
+            if len(set(providers)) > 1:  # Multiple unique providers
+                cross_validated += 1
+        
+        return cross_validated
+    
+    def _generate_security_recommendations(self, infrastructure_analysis: Dict) -> List[str]:
+        """Generate actionable security recommendations."""
+        recommendations = []
+        
+        # Check for complex infrastructure
+        if infrastructure_analysis['ips'] > 10:
+            recommendations.append(
+                "Document and validate the necessity of extensive IP address infrastructure"
+            )
+        
+        if infrastructure_analysis['correlations'] > 5:
+            recommendations.append(
+                "Investigate shared infrastructure components for operational security implications"
+            )
+        
+        if not recommendations:
+            recommendations.append(
+                "Continue monitoring for changes in the identified digital infrastructure"
+            )
+        
+        return recommendations
+    
+    def _generate_conclusion(self, target: str, infrastructure_analysis: Dict, total_relationships: int) -> str:
+        """Generate a professional conclusion for the report."""
+        conclusion_parts = [
+            f"The passive reconnaissance analysis of '{target}' has successfully mapped "
+            f"a digital infrastructure ecosystem consisting of {infrastructure_analysis['domains']} "
+            f"domain names, {infrastructure_analysis['ips']} IP addresses, and "
+            f"{total_relationships} verified inter-entity relationships."
+        ]
+        
+        conclusion_parts.append(
+            "All findings in this report are based on publicly available information and "
+            "passive reconnaissance techniques. The analysis maintains full forensic integrity "
+            "with complete audit trails for all data collection activities."
+        )
+        
+        return " ".join(conclusion_parts)
+    
+    def _count_bidirectional_relationships(self, graph) -> int:
+        """Count bidirectional relationships in the graph."""
+        count = 0
+        for u, v in graph.edges():
+            if graph.has_edge(v, u):
+                count += 1
+        return count // 2  # Each pair counted twice
+    
+    def _identify_hub_nodes(self, graph, nodes: List[Dict]) -> List[str]:
+        """Identify nodes that serve as major hubs in the network."""
+        if not graph.nodes():
+            return []
+        
+        degree_centrality = nx.degree_centrality(graph.to_undirected())
+        threshold = max(degree_centrality.values()) * 0.8 if degree_centrality else 0
+        
+        return [node for node, centrality in degree_centrality.items() 
+                if centrality >= threshold]
+    
+    def _get_version(self) -> str:
+        """Get DNSRecon version for report authentication."""
+        return "1.0.0-forensic"
+    
+    def export_graph_json(self, graph_manager) -> Dict[str, Any]:
+        """
+        Export complete graph data as a JSON-serializable dictionary.
+        Moved from GraphManager to centralize export functionality.
+        
+        Args:
+            graph_manager: GraphManager instance with graph data
+            
+        Returns:
+            Complete graph data with export metadata
+        """
+        graph_data = nx.node_link_data(graph_manager.graph, edges="edges")
+        
+        return {
+            'export_metadata': {
+                'export_timestamp': datetime.now(timezone.utc).isoformat(),
+                'graph_creation_time': graph_manager.creation_time,
+                'last_modified': graph_manager.last_modified,
+                'total_nodes': graph_manager.get_node_count(),
+                'total_edges': graph_manager.get_edge_count(),
+                'graph_format': 'dnsrecon_v1_unified_model'
+            },
+            'graph': graph_data,
+            'statistics': graph_manager.get_statistics()
+        }
+    
+    def serialize_to_json(self, data: Dict[str, Any], indent: int = 2) -> str:
+        """
+        Serialize data to JSON with custom handling for non-serializable objects.
+        
+        Args:
+            data: Data to serialize
+            indent: JSON indentation level
+            
+        Returns:
+            JSON string representation
+        """
+        try:
+            return json.dumps(data, indent=indent, cls=CustomJSONEncoder, ensure_ascii=False)
+        except Exception:
+            # Fallback to aggressive cleaning
+            cleaned_data = self._clean_for_json(data)
+            return json.dumps(cleaned_data, indent=indent, ensure_ascii=False)
+    
+    def _clean_for_json(self, obj, max_depth: int = 10, current_depth: int = 0) -> Any:
+        """
+        Recursively clean an object to make it JSON serializable.
+        Handles circular references and problematic object types.
+        
+        Args:
+            obj: Object to clean
+            max_depth: Maximum recursion depth
+            current_depth: Current recursion depth
+            
+        Returns:
+            JSON-serializable object
+        """
+        if current_depth > max_depth:
+            return f"<max_depth_exceeded_{type(obj).__name__}>"
+        
+        if obj is None or isinstance(obj, (bool, int, float, str)):
+            return obj
+        elif isinstance(obj, datetime):
+            return obj.isoformat()
+        elif isinstance(obj, (set, frozenset)):
+            return list(obj)
+        elif isinstance(obj, dict):
+            cleaned = {}
+            for key, value in obj.items():
+                try:
+                    # Ensure key is string
+                    clean_key = str(key) if not isinstance(key, str) else key
+                    cleaned[clean_key] = self._clean_for_json(value, max_depth, current_depth + 1)
+                except Exception:
+                    cleaned[str(key)] = f"<serialization_error_{type(value).__name__}>"
+            return cleaned
+        elif isinstance(obj, (list, tuple)):
+            cleaned = []
+            for item in obj:
+                try:
+                    cleaned.append(self._clean_for_json(item, max_depth, current_depth + 1))
+                except Exception:
+                    cleaned.append(f"<serialization_error_{type(item).__name__}>")
+            return cleaned
+        elif hasattr(obj, '__dict__'):
+            try:
+                return self._clean_for_json(obj.__dict__, max_depth, current_depth + 1)
+            except Exception:
+                return str(obj)
+        elif hasattr(obj, 'value'):
+            # For enum-like objects
+            return obj.value
+        else:
+            return str(obj)
+    
+    def generate_filename(self, target: str, export_type: str, timestamp: Optional[datetime] = None) -> str:
+        """
+        Generate standardized filename for exports.
+        
+        Args:
+            target: Target domain/IP being scanned
+            export_type: Type of export (json, txt, summary)
+            timestamp: Optional timestamp (defaults to now)
+            
+        Returns:
+            Formatted filename with forensic naming convention
+        """
+        if timestamp is None:
+            timestamp = datetime.now(timezone.utc)
+        
+        timestamp_str = timestamp.strftime('%Y%m%d_%H%M%S')
+        safe_target = "".join(c for c in target if c.isalnum() or c in ('-', '_', '.')).rstrip()
+        
+        extension_map = {
+            'json': 'json',
+            'txt': 'txt', 
+            'summary': 'txt',
+            'targets': 'txt'
+        }
+        
+        extension = extension_map.get(export_type, 'txt')
+        return f"dnsrecon_{export_type}_{safe_target}_{timestamp_str}.{extension}"
+
+
+class CustomJSONEncoder(json.JSONEncoder):
+    """Custom JSON encoder to handle non-serializable objects."""
+    
+    def default(self, obj):
+        if isinstance(obj, datetime):
+            return obj.isoformat()
+        elif isinstance(obj, set):
+            return list(obj)
+        elif isinstance(obj, Decimal):
+            return float(obj)
+        elif hasattr(obj, '__dict__'):
+            # For custom objects, try to serialize their dict representation
+            try:
+                return obj.__dict__
+            except:
+                return str(obj)
+        elif hasattr(obj, 'value') and hasattr(obj, 'name'):
+            # For enum objects
+            return obj.value
+        else:
+            # For any other non-serializable object, convert to string
+            return str(obj)
+
+
+# Global export manager instance
+export_manager = ExportManager()