attempt bugfix

Reviewed-on: mstoeck3/dnsrecon#4

.env.example

@@ -1,5 +1,5 @@
 # ===============================================
-# DNSRecon Environment Variables
+# DNScope Environment Variables
 # ===============================================
 # Copy this file to .env and fill in your values.
 
@@ -32,3 +32,5 @@ LARGE_ENTITY_THRESHOLD=100
 MAX_RETRIES_PER_TARGET=8
 # How long cached provider responses are stored (in hours).
 CACHE_TIMEOUT_HOURS=12
+
+GRAPH_POLLING_NODE_THRESHOLD=100

README.md

@@ -1,16 +1,18 @@
-# DNSRecon - Passive Infrastructure Reconnaissance Tool
+# DNScope - Passive Infrastructure Reconnaissance Tool
 
-DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
+DNScope is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
 
-**Repo Link:** [https://git.cc24.dev/mstoeck3/dnsrecon](https://git.cc24.dev/mstoeck3/dnsrecon)
+**Repo Link:** [https://github.com/overcuriousity/DNScope](https://github.com/overcuriousity/DNScope)
 
 -----
 
 ## Concept and Philosophy
 
-The core philosophy of DNSRecon is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNSRecon will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
+The core philosophy of DNScope is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNScope will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
 
-For power users who require more in-depth information, DNSRecon can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
+For power users who require more in-depth information, DNScope can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
+
+-----
 
 -----
 
@@ -20,16 +22,18 @@ For power users who require more in-depth information, DNSRecon can be configure
   * **In-Memory Graph Analysis**: Uses NetworkX for efficient relationship mapping.
   * **Real-Time Visualization**: The graph updates dynamically as the scan progresses.
   * **Forensic Logging**: A complete audit trail of all reconnaissance activities is maintained.
-  * **Confidence Scoring**: Relationships are weighted based on the reliability of the data source.
   * **Session Management**: Supports concurrent user sessions with isolated scanner instances.
   * **Extensible Provider Architecture**: Easily add new data sources to expand the tool's capabilities.
   * **Web-Based UI**: An intuitive and interactive web interface for managing scans and visualizing results.
+  * **Export Options**: Export scan results to JSON, a list of targets to a text file, or an executive summary.
+  * **API Key Management**: Securely manage API keys for various providers through the web interface.
+  * **Provider Management**: Enable or disable providers for the current session.
 
 -----
 
 ## Technical Architecture
 
-DNSRecon is a web-based application built with a modern technology stack:
+DNScope is a web-based application built with a modern technology stack:
 
   * **Backend**: The backend is a **Flask** application that provides a REST API for the frontend and manages the scanning process.
   * **Scanning Engine**: The core scanning engine is a multi-threaded Python application that uses a provider-based architecture to query different data sources.
@@ -41,7 +45,7 @@ DNSRecon is a web-based application built with a modern technology stack:
 
 ## Data Sources
 
-DNSRecon queries the following data sources:
+DNScope queries the following data sources:
 
   * **DNS**: Standard DNS lookups (A, AAAA, CNAME, MX, NS, SOA, TXT).
   * **crt.sh**: A certificate transparency log that provides information about SSL/TLS certificates.
@@ -56,15 +60,43 @@ DNSRecon queries the following data sources:
   * Python 3.8 or higher
   * A modern web browser with JavaScript enabled
   * A Linux host for running the application
+  * Redis Server
 
-### 1\. Clone the Project
+### 1\. Install Redis
+
+It is recommended to install Redis from the official repositories.
+
+**On Debian/Ubuntu:**
 
 ```bash
-git clone https://git.cc24.dev/mstoeck3/dnsrecon
+sudo apt-get update
-cd dnsrecon
+sudo apt-get install redis-server
 ```
 
-### 2\. Install Python Dependencies
+**On CentOS/RHEL:**
+
+```bash
+sudo yum install redis
+sudo systemctl start redis
+sudo systemctl enable redis
+```
+
+You can verify that Redis is running with the following command:
+
+```bash
+redis-cli ping
+```
+
+You should see `PONG` as the response.
+
+### 2\. Clone the Project
+
+```bash
+git clone https://github.com/overcuriousity/DNScope
+cd DNScope
+```
+
+### 3\. Install Python Dependencies
 
 It is highly recommended to use a virtual environment:
 
@@ -86,10 +118,11 @@ The `requirements.txt` file contains the following dependencies:
   * gunicorn
   * redis
   * python-dotenv
+  * psycopg2-binary
 
-### 3\. Configure the Application
+### 4\. Configure the Application
 
-DNSRecon is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
+DNScope is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
 
 ```bash
 cp .env.example .env
@@ -133,30 +166,30 @@ gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
 
 ## Systemd Service
 
-To run DNSRecon as a service that starts automatically on boot, you can use `systemd`.
+To run DNScope as a service that starts automatically on boot, you can use `systemd`.
 
 ### 1\. Create a `.service` file
 
 Create a new service file in `/etc/systemd/system/`:
 
 ```bash
-sudo nano /etc/systemd/system/dnsrecon.service
+sudo nano /etc/systemd/system/DNScope.service
 ```
 
 ### 2\. Add the Service Configuration
 
-Paste the following configuration into the file. **Remember to replace `/path/to/your/dnsrecon` and `your_user` with your actual project path and username.**
+Paste the following configuration into the file. **Remember to replace `/path/to/your/DNScope` and `your_user` with your actual project path and username.**
 
 ```ini
 [Unit]
-Description=DNSRecon Application
+Description=DNScope Application
 After=network.target
 
 [Service]
 User=your_user
 Group=your_user
-WorkingDirectory=/path/to/your/dnsrecon
+WorkingDirectory=/path/to/your/DNScope
-ExecStart=/path/to/your/dnsrecon/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+ExecStart=/path/to/your/DNScope/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
 Restart=always
 Environment="SECRET_KEY=your-super-secret-and-random-key"
 Environment="FLASK_ENV=production"
@@ -173,14 +206,14 @@ Reload the `systemd` daemon, enable the service to start on boot, and then start
 
 ```bash
 sudo systemctl daemon-reload
-sudo systemctl enable dnsrecon.service
+sudo systemctl enable DNScope.service
-sudo systemctl start dnsrecon.service
+sudo systemctl start DNScope.service
 ```
 
 You can check the status of the service at any time with:
 
 ```bash
-sudo systemctl status dnsrecon.service
+sudo systemctl status DNScope.service
 ```
 
 -----
@@ -210,14 +243,14 @@ rm -rf cache/*
 ### 4\. Restart the Service
 
 ```bash
-sudo systemctl restart dnsrecon.service
+sudo systemctl restart DNScope.service
 ```
 
 -----
 
 ## Extensibility
 
-DNSRecon is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
+DNScope is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
 
   * `get_name()`: Return the name of the provider.
   * `get_display_name()`: Return a display-friendly name for the provider.

app.py

@@ -1,11 +1,12 @@
-# dnsrecon-reduced/app.py
+# DNScope-reduced/app.py
 
 """
-Flask application entry point for DNSRecon web interface.
+Flask application entry point for DNScope web interface.
 Provides REST API endpoints and serves the web interface with user session support.
-FIXED: Enhanced WebSocket integration with proper connection management.
+UPDATED: Added /api/config endpoint for graph polling optimization settings.
 """
 
+import json
 import traceback
 from flask import Flask, render_template, request, jsonify, send_file, session
 from datetime import datetime, timezone, timedelta
@@ -13,7 +14,6 @@ import io
 import os
 
 from core.session_manager import session_manager
-from flask_socketio import SocketIO
 from config import config
 from core.graph_manager import NodeType
 from utils.helpers import is_valid_target
@@ -22,38 +22,29 @@ from decimal import Decimal
 
 
 app = Flask(__name__)
-socketio = SocketIO(app, cors_allowed_origins="*")
 app.config['SECRET_KEY'] = config.flask_secret_key
 app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=config.flask_permanent_session_lifetime_hours)
 
 def get_user_scanner():
     """
-    FIXED: Retrieves the scanner for the current session with proper socketio management.
+    Retrieves the scanner for the current session, or creates a new one if none exists.
     """
-    current_flask_session_id = session.get('dnsrecon_session_id')
+    current_flask_session_id = session.get('DNScope_session_id')
     
     if current_flask_session_id:
         existing_scanner = session_manager.get_session(current_flask_session_id)
         if existing_scanner:
-            # FIXED: Ensure socketio is properly maintained
-            existing_scanner.socketio = socketio
-            print(f"✓ Retrieved existing scanner for session {current_flask_session_id[:8]}... with socketio restored")
             return current_flask_session_id, existing_scanner
     
-    # FIXED: Register socketio connection when creating new session
+    new_session_id = session_manager.create_session()
-    new_session_id = session_manager.create_session(socketio)
     new_scanner = session_manager.get_session(new_session_id)
     
     if not new_scanner:
         raise Exception("Failed to create new scanner session")
     
-    # FIXED: Ensure new scanner has socketio reference and register the connection
+    session['DNScope_session_id'] = new_session_id
-    new_scanner.socketio = socketio
-    session_manager.register_socketio_connection(new_session_id, socketio)
-    session['dnsrecon_session_id'] = new_session_id
     session.permanent = True
     
-    print(f"✓ Created new scanner for session {new_session_id[:8]}... with socketio registered")
     return new_session_id, new_scanner
 
 
@@ -63,10 +54,25 @@ def index():
     return render_template('index.html')
 
 
+@app.route('/api/config', methods=['GET'])
+def get_config():
+    """Get configuration settings for frontend."""
+    try:
+        return jsonify({
+            'success': True,
+            'config': {
+                'graph_polling_node_threshold': config.graph_polling_node_threshold
+            }
+        })
+    except Exception as e:
+        traceback.print_exc()
+        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+
+
 @app.route('/api/scan/start', methods=['POST'])
 def start_scan():
     """
-    FIXED: Starts a new reconnaissance scan with proper socketio management.
+    Starts a new reconnaissance scan.
     """
     try:
         data = request.get_json()
@@ -90,17 +96,9 @@ def start_scan():
         if not scanner:
             return jsonify({'success': False, 'error': 'Failed to get scanner instance.'}), 500
         
-        # FIXED: Ensure scanner has socketio reference and is registered
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        print(f"🚀 Starting scan for {target} with socketio enabled and registered")
-        
         success = scanner.start_scan(target, max_depth, clear_graph=clear_graph, force_rescan_target=force_rescan_target)
         
         if success:
-            # Update session with socketio-enabled scanner
-            session_manager.update_session_scanner(user_session_id, scanner)
-            
             return jsonify({
                 'success': True,
                 'message': 'Reconnaissance scan started successfully',
@@ -129,10 +127,6 @@ def stop_scan():
         if not scanner.session_id:
             scanner.session_id = user_session_id
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         scanner.stop_scan()
         session_manager.set_stop_signal(user_session_id)
         session_manager.update_scanner_status(user_session_id, 'stopped')
@@ -149,83 +143,37 @@ def stop_scan():
         return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
 
 
-@socketio.on('connect')
+@app.route('/api/scan/status', methods=['GET'])
-def handle_connect():
-    """
-    FIXED: Handle WebSocket connection with proper session management.
-    """
-    print(f'✓ WebSocket client connected: {request.sid}')
-    
-    # Try to restore existing session connection
-    current_flask_session_id = session.get('dnsrecon_session_id')
-    if current_flask_session_id:
-        # Register this socketio connection for the existing session
-        session_manager.register_socketio_connection(current_flask_session_id, socketio)
-        print(f'✓ Registered WebSocket for existing session: {current_flask_session_id[:8]}...')
-    
-    # Immediately send current status to new connection
-    get_scan_status()
-
-
-@socketio.on('disconnect')
-def handle_disconnect():
-    """
-    FIXED: Handle WebSocket disconnection gracefully.
-    """
-    print(f'✗ WebSocket client disconnected: {request.sid}')
-    
-    # Note: We don't immediately remove the socketio connection from session_manager
-    # because the user might reconnect. The cleanup will happen during session cleanup.
-
-
-@socketio.on('get_status')
 def get_scan_status():
-    """
+    """Get current scan status."""
-    FIXED: Get current scan status and emit real-time update with proper error handling.
-    """
     try:
         user_session_id, scanner = get_user_scanner()
         
         if not scanner:
-            status = {
+            return jsonify({
-                'status': 'idle', 
+                'success': True,
-                'target_domain': None, 
+                'status': {
-                'current_depth': 0,
+                    'status': 'idle', 'target_domain': None, 'current_depth': 0,
-                'max_depth': 0, 
+                    'max_depth': 0, 'progress_percentage': 0.0,
-                'progress_percentage': 0.0,
+                    'user_session_id': user_session_id
-                'user_session_id': user_session_id,
+                }
-                'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
+            })
-            }
-            print(f"📡 Emitting idle status for session {user_session_id[:8] if user_session_id else 'none'}...")
-        else:
-            if not scanner.session_id:
-                scanner.session_id = user_session_id
         
-            # FIXED: Ensure scanner has socketio reference for future updates
+        if not scanner.session_id:
-            scanner.socketio = socketio
+            scanner.session_id = user_session_id
-            session_manager.register_socketio_connection(user_session_id, socketio)
         
-            status = scanner.get_scan_status()
+        status = scanner.get_scan_status()
-            status['user_session_id'] = user_session_id
+        status['user_session_id'] = user_session_id
         
-            print(f"📡 Emitting status update: {status['status']} - "
+        return jsonify({'success': True, 'status': status})
-                  f"Nodes: {len(status.get('graph', {}).get('nodes', []))}, "
-                  f"Edges: {len(status.get('graph', {}).get('edges', []))}")
-            
-            # Update session with socketio-enabled scanner
-            session_manager.update_session_scanner(user_session_id, scanner)
-        
-        socketio.emit('scan_update', status)
     
     except Exception as e:
         traceback.print_exc()
-        error_status = {
+        return jsonify({
-            'status': 'error', 
+            'success': False, 'error': f'Internal server error: {str(e)}',
-            'message': 'Failed to get status',
+            'fallback_status': {'status': 'error', 'progress_percentage': 0.0}
-            'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
+        }), 500
-        }
+
-        print(f"⚠️ Error getting status, emitting error status")
-        socketio.emit('scan_update', error_status)
 
 
 @app.route('/api/graph', methods=['GET'])
@@ -242,10 +190,6 @@ def get_graph_data():
         if not scanner:
             return jsonify({'success': True, 'graph': empty_graph, 'user_session_id': user_session_id})
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         graph_data = scanner.get_graph_data() or empty_graph
         
         return jsonify({'success': True, 'graph': graph_data, 'user_session_id': user_session_id})
@@ -259,7 +203,9 @@ def get_graph_data():
 
 @app.route('/api/graph/large-entity/extract', methods=['POST'])
 def extract_from_large_entity():
-    """Extract a node from a large entity."""
+    """
+    FIXED: Extract a node from a large entity with proper error handling.
+    """
     try:
         data = request.get_json()
         large_entity_id = data.get('large_entity_id')
@@ -272,21 +218,66 @@ def extract_from_large_entity():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
-        # FIXED: Ensure scanner has socketio reference
+        # FIXED: Check if node exists and provide better error messages
-        scanner.socketio = socketio
+        if not scanner.graph.graph.has_node(node_id):
-        session_manager.register_socketio_connection(user_session_id, socketio)
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} not found in graph'
+            }), 404
 
+        # FIXED: Check if node is actually part of the large entity
+        node_data = scanner.graph.graph.nodes[node_id]
+        metadata = node_data.get('metadata', {})
+        current_large_entity = metadata.get('large_entity_id')
+        
+        if not current_large_entity:
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} is not part of any large entity'
+            }), 400
+            
+        if current_large_entity != large_entity_id:
+            return jsonify({
+                'success': False, 
+                'error': f'Node {node_id} belongs to large entity {current_large_entity}, not {large_entity_id}'
+            }), 400
+
+        # FIXED: Check if large entity exists
+        if not scanner.graph.graph.has_node(large_entity_id):
+            return jsonify({
+                'success': False, 
+                'error': f'Large entity {large_entity_id} not found'
+            }), 404
+
+        # Perform the extraction
         success = scanner.extract_node_from_large_entity(large_entity_id, node_id)
         
         if success:
+            # Force immediate session state update
             session_manager.update_session_scanner(user_session_id, scanner)
-            return jsonify({'success': True, 'message': f'Node {node_id} extracted successfully.'})
-        else:
-            return jsonify({'success': False, 'error': f'Failed to extract node {node_id}.'}), 500
             
+            return jsonify({
+                'success': True, 
+                'message': f'Node {node_id} extracted successfully from {large_entity_id}.',
+                'extracted_node': node_id,
+                'large_entity': large_entity_id
+            })
+        else:
+            # This should not happen with the improved checks above, but handle it gracefully
+            return jsonify({
+                'success': False, 
+                'error': f'Failed to extract node {node_id} from {large_entity_id}. Node may have already been extracted.'
+            }), 409
+
+    except json.JSONDecodeError:
+        return jsonify({'success': False, 'error': 'Invalid JSON in request body'}), 400
     except Exception as e:
         traceback.print_exc()
-        return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
+        return jsonify({
+            'success': False, 
+            'error': f'Internal server error: {str(e)}',
+            'error_type': type(e).__name__
+        }), 500
 
 @app.route('/api/graph/node/<node_id>', methods=['DELETE'])
 def delete_graph_node(node_id):
@@ -296,10 +287,6 @@ def delete_graph_node(node_id):
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
-        # FIXED: Ensure scanner has socketio reference  
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         success = scanner.graph.remove_node(node_id)
         
         if success:
@@ -325,10 +312,6 @@ def revert_graph_action():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active session found'}), 404
 
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-
         action_type = data['type']
         action_data = data['data']
         
@@ -349,7 +332,6 @@ def revert_graph_action():
                     scanner.graph.add_edge(
                         source_id=edge['from'], target_id=edge['to'],
                         relationship_type=edge['metadata']['relationship_type'],
-                        confidence_score=edge['metadata']['confidence_score'],
                         source_provider=edge['metadata']['source_provider'],
                         raw_data=edge.get('raw_data', {})
                     )
@@ -373,10 +355,6 @@ def export_results():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         # Get export data using the new export manager
         try:
             results = export_manager.export_scan_results(scanner)
@@ -428,10 +406,6 @@ def export_targets():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
 
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-
         # Use export manager for targets export
         targets_txt = export_manager.export_targets_list(scanner)
         
@@ -462,10 +436,6 @@ def export_summary():
         if not scanner:
             return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
 
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-
         # Use export manager for summary generation
         summary_txt = export_manager.generate_executive_summary(scanner)
 
@@ -498,10 +468,6 @@ def set_api_keys():
         user_session_id, scanner = get_user_scanner()
         session_config = scanner.config
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         updated_providers = []
         
         for provider_name, api_key in data.items():
@@ -534,10 +500,6 @@ def get_providers():
         user_session_id, scanner = get_user_scanner()
         base_provider_info = scanner.get_provider_info()
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         # Enhance provider info with API key source information
         enhanced_provider_info = {}
         
@@ -602,10 +564,6 @@ def configure_providers():
         user_session_id, scanner = get_user_scanner()
         session_config = scanner.config
         
-        # FIXED: Ensure scanner has socketio reference
-        scanner.socketio = socketio
-        session_manager.register_socketio_connection(user_session_id, socketio)
-        
         updated_providers = []
         
         for provider_name, settings in data.items():
@@ -634,6 +592,7 @@ def configure_providers():
         return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
 
 
+
 @app.errorhandler(404)
 def not_found(error):
     """Handle 404 errors."""
@@ -649,9 +608,9 @@ def internal_error(error):
 
 if __name__ == '__main__':
     config.load_from_env()
-    print("🚀 Starting DNSRecon with enhanced WebSocket support...")
+    app.run(
-    print(f"   Host: {config.flask_host}")
+        host=config.flask_host,
-    print(f"   Port: {config.flask_port}")
+        port=config.flask_port,
-    print(f"   Debug: {config.flask_debug}")
+        debug=config.flask_debug,
-    print("   WebSocket: Enhanced connection management enabled")
+        threaded=True
-    socketio.run(app, host=config.flask_host, port=config.flask_port, debug=config.flask_debug)
+    )

config.py

@@ -1,7 +1,7 @@
-# dnsrecon-reduced/config.py
+# DNScope-reduced/config.py
 
 """
-Configuration management for DNSRecon tool.
+Configuration management for DNScope tool.
 Handles API key storage, rate limiting, and default settings.
 """
 
@@ -13,7 +13,7 @@ from dotenv import load_dotenv
 load_dotenv()
 
 class Config:
-    """Configuration manager for DNSRecon application."""
+    """Configuration manager for DNScope application."""
     
     def __init__(self):
         """Initialize configuration with default values."""
@@ -26,6 +26,9 @@ class Config:
         self.large_entity_threshold = 100
         self.max_retries_per_target = 8
         
+        # --- Graph Polling Performance Settings ---
+        self.graph_polling_node_threshold = 100  # Stop graph auto-polling above this many nodes
+        
         # --- Provider Caching Settings ---
         self.cache_timeout_hours = 6  # Provider-specific cache timeout
         
@@ -72,6 +75,9 @@ class Config:
         self.max_retries_per_target = int(os.getenv('MAX_RETRIES_PER_TARGET', self.max_retries_per_target))
         self.cache_timeout_hours = int(os.getenv('CACHE_TIMEOUT_HOURS', self.cache_timeout_hours))
         
+        # Override graph polling threshold from environment
+        self.graph_polling_node_threshold = int(os.getenv('GRAPH_POLLING_NODE_THRESHOLD', self.graph_polling_node_threshold))
+        
         # Override Flask and session settings
         self.flask_host = os.getenv('FLASK_HOST', self.flask_host)
         self.flask_port = int(os.getenv('FLASK_PORT', self.flask_port))

core/__init__.py

@@ -1,5 +1,5 @@
 """
-Core modules for DNSRecon passive reconnaissance tool.
+Core modules for DNScope passive reconnaissance tool.
 Contains graph management, scanning orchestration, and forensic logging.
 """
 

core/graph_manager.py

@@ -1,10 +1,11 @@
-# dnsrecon-reduced/core/graph_manager.py
+# DNScope-reduced/core/graph_manager.py
 
 """
-Graph data model for DNSRecon using NetworkX.
+Graph data model for DNScope using NetworkX.
-Manages in-memory graph storage with confidence scoring and forensic metadata.
+Manages in-memory graph storage with forensic metadata.
 Now fully compatible with the unified ProviderResult data model.
-FIXED: Added proper pickle support to prevent weakref serialization errors.
+UPDATED: Fixed correlation exclusion keys to match actual attribute names.
+UPDATED: Removed export_json() method - now handled by ExportManager.
 """
 import re
 from datetime import datetime, timezone
@@ -29,10 +30,9 @@ class NodeType(Enum):
 
 class GraphManager:
     """
-    Thread-safe graph manager for DNSRecon infrastructure mapping.
+    Thread-safe graph manager for DNScope infrastructure mapping.
-    Uses NetworkX for in-memory graph storage with confidence scoring.
+    Uses NetworkX for in-memory graph storage.
     Compatible with unified ProviderResult data model.
-    FIXED: Added proper pickle support to handle NetworkX graph serialization.
     """
 
     def __init__(self):
@@ -41,57 +41,6 @@ class GraphManager:
         self.creation_time = datetime.now(timezone.utc).isoformat()
         self.last_modified = self.creation_time
         
-    def __getstate__(self):
-        """Prepare GraphManager for pickling by converting NetworkX graph to serializable format."""
-        state = self.__dict__.copy()
-        
-        # Convert NetworkX graph to a serializable format
-        if hasattr(self, 'graph') and self.graph:
-            # Extract all nodes with their data
-            nodes_data = {}
-            for node_id, attrs in self.graph.nodes(data=True):
-                nodes_data[node_id] = dict(attrs)
-            
-            # Extract all edges with their data
-            edges_data = []
-            for source, target, attrs in self.graph.edges(data=True):
-                edges_data.append({
-                    'source': source,
-                    'target': target,
-                    'attributes': dict(attrs)
-                })
-            
-            # Replace the NetworkX graph with serializable data
-            state['_graph_nodes'] = nodes_data
-            state['_graph_edges'] = edges_data
-            del state['graph']
-        
-        return state
-
-    def __setstate__(self, state):
-        """Restore GraphManager after unpickling by reconstructing NetworkX graph."""
-        # Restore basic attributes
-        self.__dict__.update(state)
-        
-        # Reconstruct NetworkX graph from serializable data
-        self.graph = nx.DiGraph()
-        
-        # Restore nodes
-        if hasattr(self, '_graph_nodes'):
-            for node_id, attrs in self._graph_nodes.items():
-                self.graph.add_node(node_id, **attrs)
-            del self._graph_nodes
-        
-        # Restore edges
-        if hasattr(self, '_graph_edges'):
-            for edge_data in self._graph_edges:
-                self.graph.add_edge(
-                    edge_data['source'], 
-                    edge_data['target'], 
-                    **edge_data['attributes']
-                )
-            del self._graph_edges
-        
     def add_node(self, node_id: str, node_type: NodeType, attributes: Optional[List[Dict[str, Any]]] = None,
                 description: str = "", metadata: Optional[Dict[str, Any]] = None) -> bool:
         """
@@ -134,7 +83,7 @@ class GraphManager:
         return is_new_node
 
     def add_edge(self, source_id: str, target_id: str, relationship_type: str,
-                confidence_score: float = 0.5, source_provider: str = "unknown",
+                source_provider: str = "unknown",
                 raw_data: Optional[Dict[str, Any]] = None) -> bool:
         """
         UPDATED: Add or update an edge between two nodes with raw relationship labels.
@@ -142,23 +91,13 @@ class GraphManager:
         if not self.graph.has_node(source_id) or not self.graph.has_node(target_id):
             return False
 
-        new_confidence = confidence_score
         
         # UPDATED: Use raw relationship type - no formatting
         edge_label = relationship_type
 
-        if self.graph.has_edge(source_id, target_id):
-            # If edge exists, update confidence if the new score is higher.
-            if new_confidence > self.graph.edges[source_id, target_id].get('confidence_score', 0):
-                self.graph.edges[source_id, target_id]['confidence_score'] = new_confidence
-                self.graph.edges[source_id, target_id]['updated_timestamp'] = datetime.now(timezone.utc).isoformat()
-                self.graph.edges[source_id, target_id]['updated_by'] = source_provider
-            return False
-
         # Add a new edge with raw attributes
         self.graph.add_edge(source_id, target_id,
                             relationship_type=edge_label,
-                            confidence_score=new_confidence,
                             source_provider=source_provider,
                             discovery_timestamp=datetime.now(timezone.utc).isoformat(),
                             raw_data=raw_data or {})
@@ -188,11 +127,6 @@ class GraphManager:
         """Get all nodes of a specific type."""
         return [n for n, d in self.graph.nodes(data=True) if d.get('type') == node_type.value]
 
-    def get_high_confidence_edges(self, min_confidence: float = 0.8) -> List[Tuple[str, str, Dict]]:
-        """Get edges with confidence score above a given threshold."""
-        return [(u, v, d) for u, v, d in self.graph.edges(data=True)
-                if d.get('confidence_score', 0) >= min_confidence]
-
     def get_graph_data(self) -> Dict[str, Any]:
         """
         Export graph data formatted for frontend visualization.
@@ -228,9 +162,9 @@ class GraphManager:
                 'from': source, 
                 'to': target,
                 'label': attrs.get('relationship_type', ''),
-                'confidence_score': attrs.get('confidence_score', 0),
                 'source_provider': attrs.get('source_provider', ''),
-                'discovery_timestamp': attrs.get('discovery_timestamp')
+                'discovery_timestamp': attrs.get('discovery_timestamp'),
+                'raw_data': attrs.get('raw_data', {})
             })
         
         return {
@@ -239,24 +173,6 @@ class GraphManager:
             'statistics': self.get_statistics()['basic_metrics']
         }
 
-    def _get_confidence_distribution(self) -> Dict[str, int]:
-        """Get distribution of edge confidence scores with empty graph handling."""
-        distribution = {'high': 0, 'medium': 0, 'low': 0}
-        
-        # FIXED: Handle empty graph case
-        if self.get_edge_count() == 0:
-            return distribution
-            
-        for _, _, data in self.graph.edges(data=True):
-            confidence = data.get('confidence_score', 0)
-            if confidence >= 0.8:
-                distribution['high'] += 1
-            elif confidence >= 0.6:
-                distribution['medium'] += 1
-            else:
-                distribution['low'] += 1
-        return distribution
-
     def get_statistics(self) -> Dict[str, Any]:
         """Get comprehensive statistics about the graph with proper empty graph handling."""
         
@@ -273,7 +189,6 @@ class GraphManager:
             },
             'node_type_distribution': {}, 
             'relationship_type_distribution': {},
-            'confidence_distribution': self._get_confidence_distribution(),
             'provider_distribution': {}
         }
         

core/logger.py

@@ -1,4 +1,4 @@
-# dnsrecon/core/logger.py
+# DNScope/core/logger.py
 
 import logging
 import threading
@@ -30,7 +30,6 @@ class RelationshipDiscovery:
     source_node: str
     target_node: str
     relationship_type: str
-    confidence_score: float
     provider: str
     raw_data: Dict[str, Any]
     discovery_method: str
@@ -38,9 +37,8 @@ class RelationshipDiscovery:
 
 class ForensicLogger:
     """
-    Thread-safe forensic logging system for DNSRecon.
+    Thread-safe forensic logging system for DNScope.
     Maintains detailed audit trail of all reconnaissance activities.
-    FIXED: Enhanced pickle support to prevent weakref issues in logging handlers.
     """
     
     def __init__(self, session_id: str = ""):
@@ -66,78 +64,49 @@ class ForensicLogger:
             'target_domains': set()
         }
         
-        # Configure standard logger with simple setup to avoid weakrefs
+        # Configure standard logger
-        self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
+        self.logger = logging.getLogger(f'DNScope.{self.session_id}')
         self.logger.setLevel(logging.INFO)
         
-        # Create minimal formatter
+        # Create formatter for structured logging
         formatter = logging.Formatter(
             '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         )
         
-        # Add console handler only if not already present (avoid duplicate handlers)
+        # Add console handler if not already present
         if not self.logger.handlers:
             console_handler = logging.StreamHandler()
             console_handler.setFormatter(formatter)
             self.logger.addHandler(console_handler)
 
     def __getstate__(self):
-        """
+        """Prepare ForensicLogger for pickling by excluding unpicklable objects."""
-        FIXED: Prepare ForensicLogger for pickling by excluding problematic objects.
-        """
         state = self.__dict__.copy()
-        
+        # Remove the unpickleable 'logger' attribute
-        # Remove potentially unpickleable attributes that may contain weakrefs
+        if 'logger' in state:
-        unpicklable_attrs = ['logger', 'lock']
+            del state['logger']
-        for attr in unpicklable_attrs:
+        if 'lock' in state:
-            if attr in state:
+            del state['lock']
-                del state[attr]
-        
-        # Convert sets to lists for JSON serialization compatibility
-        if 'session_metadata' in state:
-            metadata = state['session_metadata'].copy()
-            if 'providers_used' in metadata and isinstance(metadata['providers_used'], set):
-                metadata['providers_used'] = list(metadata['providers_used'])
-            if 'target_domains' in metadata and isinstance(metadata['target_domains'], set):
-                metadata['target_domains'] = list(metadata['target_domains'])
-            state['session_metadata'] = metadata
-        
         return state
 
     def __setstate__(self, state):
-        """
+        """Restore ForensicLogger after unpickling by reconstructing logger."""
-        FIXED: Restore ForensicLogger after unpickling by reconstructing components.
-        """
         self.__dict__.update(state)
-        
+        # Re-initialize the 'logger' attribute
-        # Re-initialize threading lock
+        self.logger = logging.getLogger(f'DNScope.{self.session_id}')
-        self.lock = threading.Lock()
-        
-        # Re-initialize logger with minimal setup
-        self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
         self.logger.setLevel(logging.INFO)
-        
         formatter = logging.Formatter(
             '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
         )
-        
-        # Only add handler if not already present
         if not self.logger.handlers:
             console_handler = logging.StreamHandler()
             console_handler.setFormatter(formatter)
             self.logger.addHandler(console_handler)
-        
+        self.lock = threading.Lock()
-        # Convert lists back to sets if needed
-        if 'session_metadata' in self.__dict__:
-            metadata = self.session_metadata
-            if 'providers_used' in metadata and isinstance(metadata['providers_used'], list):
-                metadata['providers_used'] = set(metadata['providers_used'])
-            if 'target_domains' in metadata and isinstance(metadata['target_domains'], list):
-                metadata['target_domains'] = set(metadata['target_domains'])
 
     def _generate_session_id(self) -> str:
         """Generate unique session identifier."""
-        return f"dnsrecon_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
+        return f"DNScope_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
     
     def log_api_request(self, provider: str, url: str, method: str = "GET", 
                        status_code: Optional[int] = None, 
@@ -173,26 +142,21 @@ class ForensicLogger:
             discovery_context=discovery_context
         )
         
-        with self.lock:
+        self.api_requests.append(api_request)
-            self.api_requests.append(api_request)
+        self.session_metadata['total_requests'] += 1
-            self.session_metadata['total_requests'] += 1
+        self.session_metadata['providers_used'].add(provider)
-            self.session_metadata['providers_used'].add(provider)
         
-            if target_indicator:
+        if target_indicator:
-                self.session_metadata['target_domains'].add(target_indicator)
+            self.session_metadata['target_domains'].add(target_indicator)
         
-        # Log to standard logger with error handling
+        # Log to standard logger
-        try:
+        if error:
-            if error:
+            self.logger.error(f"API Request Failed.")
-                self.logger.error(f"API Request Failed - {provider}: {url}")
+        else:
-            else:
+            self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
-                self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
-        except Exception:
-            # If logging fails, continue without breaking the application
-            pass
     
     def log_relationship_discovery(self, source_node: str, target_node: str,
-                                 relationship_type: str, confidence_score: float,
+                                 relationship_type: str,
                                  provider: str, raw_data: Dict[str, Any],
                                  discovery_method: str) -> None:
         """
@@ -202,7 +166,6 @@ class ForensicLogger:
             source_node: Source node identifier
             target_node: Target node identifier
             relationship_type: Type of relationship (e.g., 'SAN', 'A_Record')
-            confidence_score: Confidence score (0.0 to 1.0)
             provider: Provider that discovered this relationship
             raw_data: Raw data from provider response
             discovery_method: Method used to discover relationship
@@ -212,50 +175,32 @@ class ForensicLogger:
             source_node=source_node,
             target_node=target_node,
             relationship_type=relationship_type,
-            confidence_score=confidence_score,
             provider=provider,
             raw_data=raw_data,
             discovery_method=discovery_method
         )
         
-        with self.lock:
+        self.relationships.append(relationship)
-            self.relationships.append(relationship)
+        self.session_metadata['total_relationships'] += 1
-            self.session_metadata['total_relationships'] += 1
         
-        # Log to standard logger with error handling
+        self.logger.info(
-        try:
+            f"Relationship Discovered - {source_node} -> {target_node} "
-            self.logger.info(
+            f"({relationship_type}) - Provider: {provider}"
-                f"Relationship Discovered - {source_node} -> {target_node} "
+        )
-                f"({relationship_type}) - Confidence: {confidence_score:.2f} - Provider: {provider}"
-            )
-        except Exception:
-            # If logging fails, continue without breaking the application
-            pass
     
     def log_scan_start(self, target_domain: str, recursion_depth: int, 
                       enabled_providers: List[str]) -> None:
         """Log the start of a reconnaissance scan."""
-        try:
+        self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
-            self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
+        self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
-            self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
         
-            with self.lock:
+        self.session_metadata['target_domains'].add(target_domain)
-                self.session_metadata['target_domains'].add(target_domain)
-        except Exception:
-            pass
     
     def log_scan_complete(self) -> None:
         """Log the completion of a reconnaissance scan."""
-        with self.lock:
+        self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
-            self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
-            # Convert sets to lists for serialization
-            self.session_metadata['providers_used'] = list(self.session_metadata['providers_used'])
-            self.session_metadata['target_domains'] = list(self.session_metadata['target_domains'])
         
-        try:
+        self.logger.info(f"Scan Complete - Session: {self.session_id}")
-            self.logger.info(f"Scan Complete - Session: {self.session_id}")
-        except Exception:
-            pass
     
     def export_audit_trail(self) -> Dict[str, Any]:
         """
@@ -264,13 +209,16 @@ class ForensicLogger:
         Returns:
             Dictionary containing complete session audit trail
         """
-        with self.lock:
+        session_metadata_export = self.session_metadata.copy()
-            return {
+        session_metadata_export['providers_used'] = list(session_metadata_export['providers_used'])
-                'session_metadata': self.session_metadata.copy(),
+        session_metadata_export['target_domains'] = list(session_metadata_export['target_domains'])
-                'api_requests': [asdict(req) for req in self.api_requests],
+        
-                'relationships': [asdict(rel) for rel in self.relationships],
+        return {
-                'export_timestamp': datetime.now(timezone.utc).isoformat()
+            'session_metadata': session_metadata_export,
-            }
+            'api_requests': [asdict(req) for req in self.api_requests],
+            'relationships': [asdict(rel) for rel in self.relationships],
+            'export_timestamp': datetime.now(timezone.utc).isoformat()
+        }
     
     def get_forensic_summary(self) -> Dict[str, Any]:
         """
@@ -280,13 +228,7 @@ class ForensicLogger:
             Dictionary containing summary statistics
         """
         provider_stats = {}
-        
+        for provider in self.session_metadata['providers_used']:
-        # Ensure providers_used is a set for iteration
-        providers_used = self.session_metadata['providers_used']
-        if isinstance(providers_used, list):
-            providers_used = set(providers_used)
-        
-        for provider in providers_used:
             provider_requests = [req for req in self.api_requests if req.provider == provider]
             provider_relationships = [rel for rel in self.relationships if rel.provider == provider]
             
@@ -295,7 +237,6 @@ class ForensicLogger:
                 'successful_requests': len([req for req in provider_requests if req.error is None]),
                 'failed_requests': len([req for req in provider_requests if req.error is not None]),
                 'relationships_discovered': len(provider_relationships),
-                'avg_confidence': sum(rel.confidence_score for rel in provider_relationships) / len(provider_relationships) if provider_relationships else 0
             }
         
         return {

core/provider_result.py

@@ -1,7 +1,7 @@
-# dnsrecon-reduced/core/provider_result.py
+# DNScope-reduced/core/provider_result.py
 
 """
-Unified data model for DNSRecon passive reconnaissance.
+Unified data model for DNScope passive reconnaissance.
 Standardizes the data structure across all providers to ensure consistent processing.
 """
 
@@ -18,33 +18,19 @@ class StandardAttribute:
     value: Any
     type: str
     provider: str
-    confidence: float
     timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
     metadata: Optional[Dict[str, Any]] = field(default_factory=dict)
 
-    def __post_init__(self):
-        """Validate the attribute after initialization."""
-        if not isinstance(self.confidence, (int, float)) or not 0.0 <= self.confidence <= 1.0:
-            raise ValueError(f"Confidence must be between 0.0 and 1.0, got {self.confidence}")
-
-
 @dataclass
 class Relationship:
     """A unified data structure for a directional link between two nodes."""
     source_node: str
     target_node: str
     relationship_type: str
-    confidence: float
     provider: str
     timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
     raw_data: Optional[Dict[str, Any]] = field(default_factory=dict)
 
-    def __post_init__(self):
-        """Validate the relationship after initialization."""
-        if not isinstance(self.confidence, (int, float)) or not 0.0 <= self.confidence <= 1.0:
-            raise ValueError(f"Confidence must be between 0.0 and 1.0, got {self.confidence}")
-
-
 @dataclass
 class ProviderResult:
     """A container for all data returned by a provider from a single query."""
@@ -52,8 +38,7 @@ class ProviderResult:
     relationships: List[Relationship] = field(default_factory=list)
 
     def add_attribute(self, target_node: str, name: str, value: Any, attr_type: str, 
-                     provider: str, confidence: float = 0.8, 
+                     provider: str, metadata: Optional[Dict[str, Any]] = None) -> None:
-                     metadata: Optional[Dict[str, Any]] = None) -> None:
         """Helper method to add an attribute to the result."""
         self.attributes.append(StandardAttribute(
             target_node=target_node,
@@ -61,19 +46,16 @@ class ProviderResult:
             value=value,
             type=attr_type,
             provider=provider,
-            confidence=confidence,
             metadata=metadata or {}
         ))
 
     def add_relationship(self, source_node: str, target_node: str, relationship_type: str,
-                        provider: str, confidence: float = 0.8, 
+                        provider: str, raw_data: Optional[Dict[str, Any]] = None) -> None:
-                        raw_data: Optional[Dict[str, Any]] = None) -> None:
         """Helper method to add a relationship to the result."""
         self.relationships.append(Relationship(
             source_node=source_node,
             target_node=target_node,
             relationship_type=relationship_type,
-            confidence=confidence,
             provider=provider,
             raw_data=raw_data or {}
         ))

core/rate_limiter.py

@@ -1,28 +1,145 @@
-# dnsrecon-reduced/core/rate_limiter.py
+# DNScope-reduced/core/rate_limiter.py
 
 import time
+import logging
 
 class GlobalRateLimiter:
+    """
+    FIXED: Improved rate limiter with better cleanup and error handling.
+    Prevents accumulation of stale entries that cause infinite retry loops.
+    """
+    
     def __init__(self, redis_client):
         self.redis = redis_client
+        self.logger = logging.getLogger('DNScope.rate_limiter')
+        # Track last cleanup times to avoid excessive Redis operations
+        self._last_cleanup = {}
 
     def is_rate_limited(self, key, limit, period):
         """
-        Check if a key is rate-limited.
+        FIXED: Check if a key is rate-limited with improved cleanup and error handling.
+        
+        Args:
+            key: Rate limit key (e.g., provider name)
+            limit: Maximum requests allowed
+            period: Time period in seconds (60 for per-minute)
+            
+        Returns:
+            bool: True if rate limited, False otherwise
+        """
+        if limit <= 0:
+            # Rate limit of 0 or negative means no limiting
+            return False
+            
+        now = time.time()
+        rate_key = f"rate_limit:{key}"
+        
+        try:
+            # FIXED: More aggressive cleanup to prevent accumulation
+            # Only clean up if we haven't cleaned recently (every 10 seconds max)
+            should_cleanup = (
+                rate_key not in self._last_cleanup or 
+                now - self._last_cleanup.get(rate_key, 0) > 10
+            )
+            
+            if should_cleanup:
+                # Remove entries older than the period
+                removed_count = self.redis.zremrangebyscore(rate_key, 0, now - period)
+                self._last_cleanup[rate_key] = now
+                
+                if removed_count > 0:
+                    self.logger.debug(f"Rate limiter cleaned up {removed_count} old entries for {key}")
+            
+            # Get current count
+            current_count = self.redis.zcard(rate_key)
+            
+            if current_count >= limit:
+                self.logger.debug(f"Rate limited: {key} has {current_count}/{limit} requests in period")
+                return True
+            
+            # Add new timestamp with error handling
+            try:
+                # Use pipeline for atomic operations
+                pipe = self.redis.pipeline()
+                pipe.zadd(rate_key, {str(now): now})
+                pipe.expire(rate_key, int(period * 2))  # Set TTL to 2x period for safety
+                pipe.execute()
+            except Exception as e:
+                self.logger.warning(f"Failed to record rate limit entry for {key}: {e}")
+                # Don't block the request if we can't record it
+                return False
+            
+            return False
+            
+        except Exception as e:
+            self.logger.error(f"Rate limiter error for {key}: {e}")
+            # FIXED: On Redis errors, don't block requests to avoid infinite loops
+            return False
+    
+    def get_rate_limit_status(self, key, limit, period):
+        """
+        Get detailed rate limit status for debugging.
+        
+        Returns:
+            dict: Status information including current count, limit, and time to reset
         """
         now = time.time()
-        key = f"rate_limit:{key}"
+        rate_key = f"rate_limit:{key}"
         
-        # Remove old timestamps
+        try:
-        self.redis.zremrangebyscore(key, 0, now - period)
+            current_count = self.redis.zcard(rate_key)
             
-        # Check the count
+            # Get oldest entry to calculate reset time
-        count = self.redis.zcard(key)
+            oldest_entries = self.redis.zrange(rate_key, 0, 0, withscores=True)
-        if count >= limit:
+            time_to_reset = 0
+            if oldest_entries:
+                oldest_time = oldest_entries[0][1]
+                time_to_reset = max(0, period - (now - oldest_time))
+            
+            return {
+                'key': key,
+                'current_count': current_count,
+                'limit': limit,
+                'period': period,
+                'is_limited': current_count >= limit,
+                'time_to_reset': time_to_reset
+            }
+        except Exception as e:
+            self.logger.error(f"Failed to get rate limit status for {key}: {e}")
+            return {
+                'key': key,
+                'current_count': 0,
+                'limit': limit,
+                'period': period,
+                'is_limited': False,
+                'time_to_reset': 0,
+                'error': str(e)
+            }
+    
+    def reset_rate_limit(self, key):
+        """
+        ADDED: Reset rate limit for a specific key (useful for debugging).
+        """
+        rate_key = f"rate_limit:{key}"
+        try:
+            deleted = self.redis.delete(rate_key)
+            self.logger.info(f"Reset rate limit for {key} (deleted: {deleted})")
             return True
+        except Exception as e:
+            self.logger.error(f"Failed to reset rate limit for {key}: {e}")
+            return False
     
-        # Add new timestamp
+    def cleanup_all_rate_limits(self):
-        self.redis.zadd(key, {now: now})
+        """
-        self.redis.expire(key, period)
+        ADDED: Clean up all rate limit entries (useful for maintenance).
-
+        """
-        return False
+        try:
+            keys = self.redis.keys("rate_limit:*")
+            if keys:
+                deleted = self.redis.delete(*keys)
+                self.logger.info(f"Cleaned up {deleted} rate limit keys")
+                return deleted
+            return 0
+        except Exception as e:
+            self.logger.error(f"Failed to cleanup rate limits: {e}")
+            return 0

core/session_config.py

@@ -1,5 +1,5 @@
 """
-Per-session configuration management for DNSRecon.
+Per-session configuration management for DNScope.
 Provides isolated configuration instances for each user session.
 """
 

core/session_manager.py

@@ -1,4 +1,4 @@
-# dnsrecon/core/session_manager.py
+# DNScope/core/session_manager.py
 
 import threading
 import time
@@ -6,7 +6,6 @@ import uuid
 import redis
 import pickle
 from typing import Dict, Optional, Any
-import copy
 
 from core.scanner import Scanner
 from config import config
@@ -14,7 +13,7 @@ from config import config
 class SessionManager:
     """
     FIXED: Manages multiple scanner instances for concurrent user sessions using Redis.
-    Enhanced to properly maintain WebSocket connections throughout scan lifecycle.
+    Now more conservative about session creation to preserve API keys and configuration.
     """
     
     def __init__(self, session_timeout_minutes: int = 0):
@@ -31,9 +30,6 @@ class SessionManager:
         # FIXED: Add a creation lock to prevent race conditions
         self.creation_lock = threading.Lock()
         
-        # Track active socketio connections per session
-        self.active_socketio_connections = {}
-        
         # Start cleanup thread
         self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
         self.cleanup_thread.start()
@@ -44,7 +40,7 @@ class SessionManager:
         """Prepare SessionManager for pickling."""
         state = self.__dict__.copy()
         # Exclude unpickleable attributes - Redis client and threading objects
-        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock', 'active_socketio_connections']
+        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock']
         for attr in unpicklable_attrs:
             if attr in state:
                 del state[attr]
@@ -57,82 +53,33 @@ class SessionManager:
         self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
         self.lock = threading.Lock()
         self.creation_lock = threading.Lock()
-        self.active_socketio_connections = {}
         self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
         self.cleanup_thread.start()
 
     def _get_session_key(self, session_id: str) -> str:
         """Generates the Redis key for a session."""
-        return f"dnsrecon:session:{session_id}"
+        return f"DNScope:session:{session_id}"
 
     def _get_stop_signal_key(self, session_id: str) -> str:
         """Generates the Redis key for a session's stop signal."""
-        return f"dnsrecon:stop:{session_id}"
+        return f"DNScope:stop:{session_id}"
 
-    def register_socketio_connection(self, session_id: str, socketio) -> None:
+    def create_session(self) -> str:
         """
-        FIXED: Register a socketio connection for a session.
+        FIXED: Create a new user session with thread-safe creation to prevent duplicates.
-        This ensures the connection is maintained throughout the session lifecycle.
-        """
-        with self.lock:
-            self.active_socketio_connections[session_id] = socketio
-            print(f"Registered socketio connection for session {session_id}")
-
-    def get_socketio_connection(self, session_id: str):
-        """
-        FIXED: Get the active socketio connection for a session.
-        """
-        with self.lock:
-            return self.active_socketio_connections.get(session_id)
-
-    def _prepare_scanner_for_storage(self, scanner: Scanner, session_id: str) -> Scanner:
-        """
-        FIXED: Prepare scanner for storage by ensuring proper cleanup of unpicklable objects.
-        Now preserves socketio connection info for restoration.
-        """
-        # Set the session ID on the scanner for cross-process stop signal management
-        scanner.session_id = session_id
-        
-        # FIXED: Don't set socketio to None if we want to preserve real-time updates
-        # Instead, we'll restore it when loading the scanner
-        scanner.socketio = None
-        
-        # Force cleanup of any threading objects that might cause issues
-        if hasattr(scanner, 'stop_event'):
-            scanner.stop_event = None
-        if hasattr(scanner, 'scan_thread'):
-            scanner.scan_thread = None
-        if hasattr(scanner, 'executor'):
-            scanner.executor = None
-        if hasattr(scanner, 'status_logger_thread'):
-            scanner.status_logger_thread = None
-        if hasattr(scanner, 'status_logger_stop_event'):
-            scanner.status_logger_stop_event = None
-        
-        return scanner
-
-    def create_session(self, socketio=None) -> str:
-        """
-        FIXED: Create a new user session with enhanced WebSocket management.
         """
         # FIXED: Use creation lock to prevent race conditions
         with self.creation_lock:
             session_id = str(uuid.uuid4())
             print(f"=== CREATING SESSION {session_id} IN REDIS ===")
             
-            # FIXED: Register socketio connection first
-            if socketio:
-                self.register_socketio_connection(session_id, socketio)
-            
             try:
                 from core.session_config import create_session_config
                 session_config = create_session_config()
+                scanner_instance = Scanner(session_config=session_config)
                 
-                # Create scanner WITHOUT socketio to avoid weakref issues
+                # Set the session ID on the scanner for cross-process stop signal management
-                scanner_instance = Scanner(session_config=session_config, socketio=None)
+                scanner_instance.session_id = session_id
-                
-                # Prepare scanner for storage (removes problematic objects)
-                scanner_instance = self._prepare_scanner_for_storage(scanner_instance, session_id)
                 
                 session_data = {
                     'scanner': scanner_instance,
@@ -142,24 +89,12 @@ class SessionManager:
                     'status': 'active'
                 }
                 
-                # Test serialization before storing to catch issues early
+                # Serialize the entire session data dictionary using pickle
-                try:
+                serialized_data = pickle.dumps(session_data)
-                    test_serialization = pickle.dumps(session_data)
-                    print(f"Session serialization test successful ({len(test_serialization)} bytes)")
-                except Exception as pickle_error:
-                    print(f"PICKLE TEST FAILED: {pickle_error}")
-                    # Try to identify the problematic object
-                    for key, value in session_data.items():
-                        try:
-                            pickle.dumps(value)
-                            print(f"  {key}: OK")
-                        except Exception as item_error:
-                            print(f"  {key}: FAILED - {item_error}")
-                    raise pickle_error
                 
                 # Store in Redis
                 session_key = self._get_session_key(session_id)
-                self.redis_client.setex(session_key, self.session_timeout, test_serialization)
+                self.redis_client.setex(session_key, self.session_timeout, serialized_data)
                 
                 # Initialize stop signal as False
                 stop_key = self._get_stop_signal_key(session_id)
@@ -171,8 +106,6 @@ class SessionManager:
                 
             except Exception as e:
                 print(f"ERROR: Failed to create session {session_id}: {e}")
-                import traceback
-                traceback.print_exc()
                 raise
 
     def set_stop_signal(self, session_id: str) -> bool:
@@ -242,63 +175,31 @@ class SessionManager:
                 # Ensure the scanner has the correct session ID for stop signal checking
                 if 'scanner' in session_data and session_data['scanner']:
                     session_data['scanner'].session_id = session_id
-                    # FIXED: Restore socketio connection from our registry
-                    socketio_conn = self.get_socketio_connection(session_id)
-                    if socketio_conn:
-                        session_data['scanner'].socketio = socketio_conn
-                        print(f"Restored socketio connection for session {session_id}")
-                    else:
-                        print(f"No socketio connection found for session {session_id}")
-                        session_data['scanner'].socketio = None
                 return session_data
             return None
         except Exception as e:
             print(f"ERROR: Failed to get session data for {session_id}: {e}")
-            import traceback
-            traceback.print_exc()
             return None
 
     def _save_session_data(self, session_id: str, session_data: Dict[str, Any]) -> bool:
         """
         Serializes and saves session data back to Redis with updated TTL.
-        FIXED: Now preserves socketio connection during storage.
         
         Returns:
             bool: True if save was successful
         """
         try:
             session_key = self._get_session_key(session_id)
-            
+            serialized_data = pickle.dumps(session_data)
-            # Create a deep copy to avoid modifying the original scanner object
-            session_data_to_save = copy.deepcopy(session_data)
-
-            # Prepare scanner for storage if it exists
-            if 'scanner' in session_data_to_save and session_data_to_save['scanner']:
-                # FIXED: Preserve the original socketio connection before preparing for storage
-                original_socketio = session_data_to_save['scanner'].socketio
-                
-                session_data_to_save['scanner'] = self._prepare_scanner_for_storage(
-                    session_data_to_save['scanner'], 
-                    session_id
-                )
-                
-                # FIXED: If we had a socketio connection, make sure it's registered
-                if original_socketio and session_id not in self.active_socketio_connections:
-                    self.register_socketio_connection(session_id, original_socketio)
-            
-            serialized_data = pickle.dumps(session_data_to_save)
             result = self.redis_client.setex(session_key, self.session_timeout, serialized_data)
             return result
         except Exception as e:
             print(f"ERROR: Failed to save session data for {session_id}: {e}")
-            import traceback
-            traceback.print_exc()
             return False
 
     def update_session_scanner(self, session_id: str, scanner: 'Scanner') -> bool:
         """
-        FIXED: Updates just the scanner object in a session with immediate persistence.
+        Updates just the scanner object in a session with immediate persistence.
-        Now maintains socketio connection throughout the update process.
         
         Returns:
             bool: True if update was successful
@@ -306,27 +207,21 @@ class SessionManager:
         try:
             session_data = self._get_session_data(session_id)
             if session_data:
-                # FIXED: Preserve socketio connection before preparing for storage
+                # Ensure scanner has the session ID
-                original_socketio = scanner.socketio
+                scanner.session_id = session_id
-                
-                # Prepare scanner for storage
-                scanner = self._prepare_scanner_for_storage(scanner, session_id)
                 session_data['scanner'] = scanner
                 session_data['last_activity'] = time.time()
                 
-                # FIXED: Restore socketio connection after preparation
-                if original_socketio:
-                    self.register_socketio_connection(session_id, original_socketio)
-                    session_data['scanner'].socketio = original_socketio
-                
                 # Immediately save to Redis for GUI updates
                 success = self._save_session_data(session_id, session_data)
                 if success:
                     # Only log occasionally to reduce noise
                     if hasattr(self, '_last_update_log'):
                         if time.time() - self._last_update_log > 5:  # Log every 5 seconds max
+                            #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
                             self._last_update_log = time.time()
                     else:
+                        #print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
                         self._last_update_log = time.time()
                 else:
                     print(f"WARNING: Failed to save scanner state for session {session_id}")
@@ -336,8 +231,6 @@ class SessionManager:
                 return False
         except Exception as e:
             print(f"ERROR: Failed to update scanner for session {session_id}: {e}")
-            import traceback
-            traceback.print_exc()
             return False
 
     def update_scanner_status(self, session_id: str, status: str) -> bool:
@@ -370,7 +263,7 @@ class SessionManager:
 
     def get_session(self, session_id: str) -> Optional[Scanner]:
         """
-        FIXED: Get scanner instance for a session from Redis with proper socketio restoration.
+        Get scanner instance for a session from Redis with session ID management.
         """
         if not session_id:
             return None
@@ -389,15 +282,6 @@ class SessionManager:
             # Ensure the scanner can check the Redis-based stop signal
             scanner.session_id = session_id
         
-            # FIXED: Restore socketio connection from our registry
-            socketio_conn = self.get_socketio_connection(session_id)
-            if socketio_conn:
-                scanner.socketio = socketio_conn
-                print(f"✓ Restored socketio connection for session {session_id}")
-            else:
-                scanner.socketio = None
-                print(f"⚠️ No socketio connection found for session {session_id}")
-        
         return scanner
 
     def get_session_status_only(self, session_id: str) -> Optional[str]:
@@ -449,12 +333,6 @@ class SessionManager:
             # Wait a moment for graceful shutdown
             time.sleep(0.5)
             
-            # FIXED: Clean up socketio connection
-            with self.lock:
-                if session_id in self.active_socketio_connections:
-                    del self.active_socketio_connections[session_id]
-                    print(f"Cleaned up socketio connection for session {session_id}")
-            
             # Delete session data and stop signal from Redis
             session_key = self._get_session_key(session_id)
             stop_key = self._get_stop_signal_key(session_id)
@@ -466,8 +344,6 @@ class SessionManager:
             
         except Exception as e:
             print(f"ERROR: Failed to terminate session {session_id}: {e}")
-            import traceback
-            traceback.print_exc()
             return False
 
     def _cleanup_loop(self) -> None:
@@ -477,7 +353,7 @@ class SessionManager:
         while True:
             try:
                 # Clean up orphaned stop signals
-                stop_keys = self.redis_client.keys("dnsrecon:stop:*")
+                stop_keys = self.redis_client.keys("DNScope:stop:*")
                 for stop_key in stop_keys:
                     # Extract session ID from stop key
                     session_id = stop_key.decode('utf-8').split(':')[-1]
@@ -488,12 +364,6 @@ class SessionManager:
                         self.redis_client.delete(stop_key)
                         print(f"Cleaned up orphaned stop signal for session {session_id}")
                         
-                        # Also clean up socketio connection
-                        with self.lock:
-                            if session_id in self.active_socketio_connections:
-                                del self.active_socketio_connections[session_id]
-                                print(f"Cleaned up orphaned socketio for session {session_id}")
-                        
             except Exception as e:
                 print(f"Error in cleanup loop: {e}")
             
@@ -502,8 +372,8 @@ class SessionManager:
     def get_statistics(self) -> Dict[str, Any]:
         """Get session manager statistics."""
         try:
-            session_keys = self.redis_client.keys("dnsrecon:session:*")
+            session_keys = self.redis_client.keys("DNScope:session:*")
-            stop_keys = self.redis_client.keys("dnsrecon:stop:*")
+            stop_keys = self.redis_client.keys("DNScope:stop:*")
             
             active_sessions = len(session_keys)
             running_scans = 0
@@ -517,16 +387,14 @@ class SessionManager:
             return {
                 'total_active_sessions': active_sessions,
                 'running_scans': running_scans,
-                'total_stop_signals': len(stop_keys),
+                'total_stop_signals': len(stop_keys)
-                'active_socketio_connections': len(self.active_socketio_connections)
             }
         except Exception as e:
             print(f"ERROR: Failed to get statistics: {e}")
             return {
                 'total_active_sessions': 0,
                 'running_scans': 0,
-                'total_stop_signals': 0,
+                'total_stop_signals': 0
-                'active_socketio_connections': 0
             }
 
 # Global session manager instance

providers/__init__.py

@@ -1,5 +1,5 @@
 """
-Data provider modules for DNSRecon.
+Data provider modules for DNScope.
 Contains implementations for various reconnaissance data sources.
 """
 

providers/base_provider.py

@@ -1,4 +1,4 @@
-# dnsrecon/providers/base_provider.py
+# DNScope/providers/base_provider.py
 
 import time
 import requests
@@ -6,16 +6,15 @@ import threading
 from abc import ABC, abstractmethod
 from typing import Dict, Any, Optional
 
-from core.logger import get_forensic_logger
+from core.logger import get_forensic_logger  # Ensure this import is present
 from core.rate_limiter import GlobalRateLimiter
 from core.provider_result import ProviderResult
 
 
 class BaseProvider(ABC):
     """
-    Abstract base class for all DNSRecon data providers.
+    Abstract base class for all DNScope data providers.
     Now supports session-specific configuration and returns standardized ProviderResult objects.
-    FIXED: Enhanced pickle support to prevent weakref serialization errors.
     """
 
     def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30, session_config=None):
@@ -42,7 +41,6 @@ class BaseProvider(ABC):
         self.name = name
         self.timeout = actual_timeout
         self._local = threading.local()
-        self.logger = get_forensic_logger()
         self._stop_event = None
 
         # Statistics (per provider instance)
@@ -54,64 +52,34 @@ class BaseProvider(ABC):
     def __getstate__(self):
         """Prepare BaseProvider for pickling by excluding unpicklable objects."""
         state = self.__dict__.copy()
-        
+        # Exclude the unpickleable '_local' attribute and stop event
-        # Exclude unpickleable attributes that may contain weakrefs
+        unpicklable_attrs = ['_local', '_stop_event']
-        unpicklable_attrs = [
-            '_local',           # Thread-local storage (contains requests.Session)
-            '_stop_event',      # Threading event
-            'logger',           # Logger may contain weakrefs in handlers
-        ]
-        
         for attr in unpicklable_attrs:
             if attr in state:
                 del state[attr]
-        
-        # Also handle any potential weakrefs in the config object
-        if 'config' in state and hasattr(state['config'], '__getstate__'):
-            # If config has its own pickle support, let it handle itself
-            pass
-        elif 'config' in state:
-            # Otherwise, ensure config doesn't contain unpicklable objects
-            try:
-                # Test if config can be pickled
-                import pickle
-                pickle.dumps(state['config'])
-            except (TypeError, AttributeError):
-                # If config can't be pickled, we'll recreate it during unpickling
-                state['_config_class'] = type(state['config']).__name__
-                del state['config']
-        
         return state
 
     def __setstate__(self, state):
         """Restore BaseProvider after unpickling by reconstructing threading objects."""
         self.__dict__.update(state)
-        
+        # Re-initialize the '_local' attribute and stop event
-        # Re-initialize unpickleable attributes
         self._local = threading.local()
         self._stop_event = None
-        self.logger = get_forensic_logger()
-        
-        # Recreate config if it was removed during pickling
-        if not hasattr(self, 'config') and hasattr(self, '_config_class'):
-            if self._config_class == 'Config':
-                from config import config as global_config
-                self.config = global_config
-            elif self._config_class == 'SessionConfig':
-                from core.session_config import create_session_config
-                self.config = create_session_config()
-            del self._config_class
 
     @property
     def session(self):
-        """Get or create thread-local requests session."""
         if not hasattr(self._local, 'session'):
             self._local.session = requests.Session()
             self._local.session.headers.update({
-                'User-Agent': 'DNSRecon/1.0 (Passive Reconnaissance Tool)'
+                'User-Agent': 'DNScope/1.0 (Passive Reconnaissance Tool)'
             })
         return self._local.session
 
+    @property
+    def logger(self):
+        """Get the current forensic logger instance."""
+        return get_forensic_logger()
+
     @abstractmethod
     def get_name(self) -> str:
         """Return the provider name."""
@@ -265,7 +233,6 @@ class BaseProvider(ABC):
 
     def log_relationship_discovery(self, source_node: str, target_node: str,
                                  relationship_type: str,
-                                 confidence_score: float,
                                  raw_data: Dict[str, Any],
                                  discovery_method: str) -> None:
         """
@@ -275,7 +242,6 @@ class BaseProvider(ABC):
             source_node: Source node identifier
             target_node: Target node identifier
             relationship_type: Type of relationship
-            confidence_score: Confidence score
             raw_data: Raw data from provider
             discovery_method: Method used for discovery
         """
@@ -285,7 +251,6 @@ class BaseProvider(ABC):
             source_node=source_node,
             target_node=target_node,
             relationship_type=relationship_type,
-            confidence_score=confidence_score,
             provider=self.name,
             raw_data=raw_data,
             discovery_method=discovery_method

providers/correlation_provider.py

@@ -1,7 +1,8 @@
-# dnsrecon/providers/correlation_provider.py
+# dnsrecon-reduced/providers/correlation_provider.py
 
 import re
 from typing import Dict, Any, List
+from datetime import datetime, timezone
 
 from .base_provider import BaseProvider
 from core.provider_result import ProviderResult
@@ -10,7 +11,7 @@ from core.graph_manager import NodeType, GraphManager
 class CorrelationProvider(BaseProvider):
     """
     A provider that finds correlations between nodes in the graph.
-    FIXED: Enhanced pickle support to prevent weakref issues with graph references.
+    UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
     """
 
     def __init__(self, name: str = "correlation", session_config=None):
@@ -23,12 +24,16 @@ class CorrelationProvider(BaseProvider):
         self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
         self.EXCLUDED_KEYS = [
             'cert_source',
+            'a_records',
+            'mx_records',
+            'ns_records',
+            'ptr_records',
             'cert_issuer_ca_id',
             'cert_common_name',
             'cert_validity_period_days',
             'cert_issuer_name',
-            'cert_serial_number',
             'cert_entry_timestamp',
+            'cert_serial_number', # useless
             'cert_not_before',
             'cert_not_after',
             'dns_ttl',
@@ -37,40 +42,10 @@ class CorrelationProvider(BaseProvider):
             'updated_timestamp',
             'discovery_timestamp',
             'query_timestamp',
+            'shodan_ip_str',
+            'shodan_a_record',
         ]
 
-    def __getstate__(self):
-        """
-        FIXED: Prepare CorrelationProvider for pickling by excluding graph reference.
-        """
-        state = super().__getstate__()
-        
-        # Remove graph reference to prevent circular dependencies and weakrefs
-        if 'graph' in state:
-            del state['graph']
-        
-        # Also handle correlation_index which might contain complex objects
-        if 'correlation_index' in state:
-            # Clear correlation index as it will be rebuilt when needed
-            state['correlation_index'] = {}
-        
-        return state
-
-    def __setstate__(self, state):
-        """
-        FIXED: Restore CorrelationProvider after unpickling.
-        """
-        super().__setstate__(state)
-        
-        # Re-initialize graph reference (will be set by scanner)
-        self.graph = None
-        
-        # Re-initialize correlation index
-        self.correlation_index = {}
-        
-        # Re-compile regex pattern
-        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
-
     def get_name(self) -> str:
         """Return the provider name."""
         return "correlation"
@@ -94,12 +69,14 @@ class CorrelationProvider(BaseProvider):
     def query_domain(self, domain: str) -> ProviderResult:
         """
         Query the provider for information about a domain.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
         """
         return self._find_correlations(domain)
 
     def query_ip(self, ip: str) -> ProviderResult:
         """
         Query the provider for information about an IP address.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
         """
         return self._find_correlations(ip)
 
@@ -111,110 +88,196 @@ class CorrelationProvider(BaseProvider):
 
     def _find_correlations(self, node_id: str) -> ProviderResult:
         """
-        Find correlations for a given node.
+        Find correlations for a given node with enhanced filtering and error handling.
-        FIXED: Added safety checks to prevent issues when graph is None.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring and list value processing.
         """
         result = ProviderResult()
+        discovery_time = datetime.now(timezone.utc)
         
-        # FIXED: Ensure self.graph is not None before proceeding
+        # Enhanced safety checks
         if not self.graph or not self.graph.graph.has_node(node_id):
             return result
 
         try:
             node_attributes = self.graph.graph.nodes[node_id].get('attributes', [])
+            
+            # Ensure attributes is a list (handle legacy data)
+            if not isinstance(node_attributes, list):
+                return result
+                
+            correlations_found = 0
+            
+            for attr in node_attributes:
+                if not isinstance(attr, dict):
+                    continue
+                    
+                attr_name = attr.get('name', '')
+                attr_value = attr.get('value')
+                attr_provider = attr.get('provider', 'unknown')
+
+                # Prepare a list of values to iterate over
+                values_to_process = []
+                if isinstance(attr_value, list):
+                    values_to_process.extend(attr_value)
+                else:
+                    values_to_process.append(attr_value)
+
+                for value_item in values_to_process:
+                    # Enhanced filtering logic
+                    should_exclude = self._should_exclude_attribute(attr_name, value_item)
+                    
+                    if should_exclude:
+                        continue
+
+                    # Build correlation index
+                    if value_item not in self.correlation_index:
+                        self.correlation_index[value_item] = {
+                            'nodes': set(),
+                            'sources': []
+                        }
+
+                    self.correlation_index[value_item]['nodes'].add(node_id)
+
+                    source_info = {
+                        'node_id': node_id,
+                        'provider': attr_provider,
+                        'attribute': attr_name,
+                        'path': f"{attr_provider}_{attr_name}"
+                    }
+
+                    # Avoid duplicate sources
+                    existing_sources = [s for s in self.correlation_index[value_item]['sources']
+                                    if s['node_id'] == node_id and s['path'] == source_info['path']]
+                    if not existing_sources:
+                        self.correlation_index[value_item]['sources'].append(source_info)
+
+                    # Create correlation if we have multiple nodes with this value
+                    if len(self.correlation_index[value_item]['nodes']) > 1:
+                        self._create_correlation_relationships(value_item, self.correlation_index[value_item], result, discovery_time)
+                        correlations_found += 1
+                    
+            # Log correlation results
+            if correlations_found > 0:
+                self.logger.logger.info(f"Found {correlations_found} correlations for node {node_id}")
+                
         except Exception as e:
-            # If there's any issue accessing the graph, return empty result
+            self.logger.logger.error(f"Error finding correlations for {node_id}: {e}")
-            print(f"Warning: Could not access graph for correlation analysis: {e}")
-            return result
-
-        for attr in node_attributes:
-            attr_name = attr.get('name')
-            attr_value = attr.get('value')
-            attr_provider = attr.get('provider', 'unknown')
-
-            should_exclude = (
-                any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS) or
-                not isinstance(attr_value, (str, int, float, bool)) or
-                attr_value is None or
-                isinstance(attr_value, bool) or
-                (isinstance(attr_value, str) and (
-                    len(attr_value) < 4 or
-                    self.date_pattern.match(attr_value) or
-                    attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']
-                )) or
-                (isinstance(attr_value, (int, float)) and (
-                    attr_value == 0 or
-                    attr_value == 1 or
-                    abs(attr_value) > 1000000
-                ))
-            )
-
-            if should_exclude:
-                continue
-
-            if attr_value not in self.correlation_index:
-                self.correlation_index[attr_value] = {
-                    'nodes': set(),
-                    'sources': []
-                }
-
-            self.correlation_index[attr_value]['nodes'].add(node_id)
-
-            source_info = {
-                'node_id': node_id,
-                'provider': attr_provider,
-                'attribute': attr_name,
-                'path': f"{attr_provider}_{attr_name}"
-            }
-
-            existing_sources = [s for s in self.correlation_index[attr_value]['sources']
-                              if s['node_id'] == node_id and s['path'] == source_info['path']]
-            if not existing_sources:
-                self.correlation_index[attr_value]['sources'].append(source_info)
-
-            if len(self.correlation_index[attr_value]['nodes']) > 1:
-                self._create_correlation_relationships(attr_value, self.correlation_index[attr_value], result)
             
         return result
     
-    def _create_correlation_relationships(self, value: Any, correlation_data: Dict[str, Any], result: ProviderResult):
+    def _should_exclude_attribute(self, attr_name: str, attr_value: Any) -> bool:
         """
-        Create correlation relationships and add them to the provider result.
+        Enhanced logic to determine if an attribute should be excluded from correlation.
+        """
+        # Check against excluded keys (exact match or substring)
+        if any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS):
+            return True
+        
+        # Value type filtering
+        if not isinstance(attr_value, (str, int, float, bool)) or attr_value is None:
+            return True
+        
+        # Boolean values are not useful for correlation
+        if isinstance(attr_value, bool):
+            return True
+            
+        # String value filtering
+        if isinstance(attr_value, str):                
+            # Date/timestamp strings
+            if self.date_pattern.match(attr_value):
+                return True
+                
+            # Common non-useful values
+            if attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']:
+                return True
+                
+            # Very long strings that are likely unique (> 100 chars)
+            if len(attr_value) > 100:
+                return True
+        
+        # Numeric value filtering  
+        if isinstance(attr_value, (int, float)):
+            # Very common values
+            if attr_value in [0, 1]:
+                return True
+                
+            # Very large numbers (likely timestamps or unique IDs)
+            if abs(attr_value) > 1000000:
+                return True
+        
+        return False
+    
+    def _create_correlation_relationships(self, value: Any, correlation_data: Dict[str, Any], 
+                                        result: ProviderResult, discovery_time: datetime):
+        """
+        Create correlation relationships with enhanced deduplication and validation.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
         """
         correlation_node_id = f"corr_{hash(str(value)) & 0x7FFFFFFF}"
         nodes = correlation_data['nodes']
         sources = correlation_data['sources']
         
+        # Only create correlations if we have meaningful nodes (more than 1)
+        if len(nodes) < 2:
+            return
+            
+        # Limit correlation size to prevent overly large correlation objects
+        MAX_CORRELATION_SIZE = 50
+        if len(nodes) > MAX_CORRELATION_SIZE:
+            # Sample the nodes to keep correlation manageable
+            import random
+            sampled_nodes = random.sample(list(nodes), MAX_CORRELATION_SIZE)
+            nodes = set(sampled_nodes)
+            # Filter sources to match sampled nodes
+            sources = [s for s in sources if s['node_id'] in nodes]
+
         # Add the correlation node as an attribute to the result
         result.add_attribute(
             target_node=correlation_node_id,
             name="correlation_value",
             value=value,
-            attr_type=str(type(value)),
+            attr_type=str(type(value).__name__),
             provider=self.name,
-            confidence=0.9,
             metadata={
                 'correlated_nodes': list(nodes),
                 'sources': sources,
+                'correlation_size': len(nodes),
+                'value_type': type(value).__name__
             }
         )
 
+        # Create relationships with source validation and enhanced timestamps
+        created_relationships = set()
+        
         for source in sources:
             node_id = source['node_id']
             provider = source['provider']
             attribute = source['attribute']
+            
+            # Skip if we've already created this relationship
+            relationship_key = (node_id, correlation_node_id)
+            if relationship_key in created_relationships:
+                continue
+                
             relationship_label = f"corr_{provider}_{attribute}"
 
+            # Enhanced raw_data with discovery timestamp for time-based edge coloring
+            raw_data = {
+                'correlation_value': value,
+                'original_attribute': attribute,
+                'correlation_type': 'attribute_matching',
+                'correlation_size': len(nodes),
+                'discovery_timestamp': discovery_time.isoformat(),
+                'relevance_timestamp': discovery_time.isoformat()  # Correlation data is "fresh" when discovered
+            }
+
             # Add the relationship to the result
             result.add_relationship(
                 source_node=node_id,
                 target_node=correlation_node_id,
                 relationship_type=relationship_label,
                 provider=self.name,
-                confidence=0.9,
+                raw_data=raw_data
-                raw_data={
-                    'correlation_value': value,
-                    'original_attribute': attribute,
-                    'correlation_type': 'attribute_matching'
-                }
             )
+            
+            created_relationships.add(relationship_key)

providers/crtsh_provider.py

@@ -1,4 +1,4 @@
-# dnsrecon/providers/crtsh_provider.py
+# DNScope/providers/crtsh_provider.py
 
 import json
 import re
@@ -13,12 +13,12 @@ from core.provider_result import ProviderResult
 from utils.helpers import _is_valid_domain
 from core.logger import get_forensic_logger
 
-
 class CrtShProvider(BaseProvider):
     """
     Provider for querying crt.sh certificate transparency database.
-    FIXED: Now properly creates domain and CA nodes instead of large entities.
+    FIXED: Improved caching logic and error handling to prevent infinite retry loops.
     Returns standardized ProviderResult objects with caching support.
+    UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
     """
     
     def __init__(self, name=None, session_config=None):
@@ -32,7 +32,7 @@ class CrtShProvider(BaseProvider):
         self.base_url = "https://crt.sh/"
         self._stop_event = None
         
-        # Initialize cache directory (separate from BaseProvider's HTTP cache)
+        # Initialize cache directory
         self.domain_cache_dir = Path('cache') / 'crtsh'
         self.domain_cache_dir.mkdir(parents=True, exist_ok=True)
         
@@ -66,43 +66,73 @@ class CrtShProvider(BaseProvider):
     
     def _get_cache_status(self, cache_file_path: Path) -> str:
         """
-        Check cache status for a domain.
+        FIXED: More robust cache status checking with better error handling.
         Returns: 'not_found', 'fresh', or 'stale'
         """
         if not cache_file_path.exists():
             return "not_found"
         
         try:
-            with open(cache_file_path, 'r') as f:
+            # Check if file is readable and not corrupted
-                cache_data = json.load(f)
+            if cache_file_path.stat().st_size == 0:
-            
+                self.logger.logger.warning(f"Empty cache file: {cache_file_path}")
-            last_query_str = cache_data.get("last_upstream_query")
-            if not last_query_str:
                 return "stale"
                 
-            last_query = datetime.fromisoformat(last_query_str.replace('Z', '+00:00'))
+            with open(cache_file_path, 'r', encoding='utf-8') as f:
-            hours_since_query = (datetime.now(timezone.utc) - last_query).total_seconds() / 3600
+                cache_data = json.load(f)
             
+            # Validate cache structure
+            if not isinstance(cache_data, dict):
+                self.logger.logger.warning(f"Invalid cache structure: {cache_file_path}")
+                return "stale"
+            
+            last_query_str = cache_data.get("last_upstream_query")
+            if not last_query_str or not isinstance(last_query_str, str):
+                self.logger.logger.warning(f"Missing or invalid last_upstream_query: {cache_file_path}")
+                return "stale"
+            
+            try:
+                # More robust datetime parsing
+                if last_query_str.endswith('Z'):
+                    last_query = datetime.fromisoformat(last_query_str.replace('Z', '+00:00'))
+                elif '+' in last_query_str or last_query_str.endswith('UTC'):
+                    # Handle various timezone formats
+                    clean_time = last_query_str.replace('UTC', '').strip()
+                    if '+' in clean_time:
+                        clean_time = clean_time.split('+')[0]
+                    last_query = datetime.fromisoformat(clean_time).replace(tzinfo=timezone.utc)
+                else:
+                    last_query = datetime.fromisoformat(last_query_str).replace(tzinfo=timezone.utc)
+                    
+            except (ValueError, AttributeError) as e:
+                self.logger.logger.warning(f"Failed to parse timestamp in cache {cache_file_path}: {e}")
+                return "stale"
+            
+            hours_since_query = (datetime.now(timezone.utc) - last_query).total_seconds() / 3600
             cache_timeout = self.config.cache_timeout_hours
+            
             if hours_since_query < cache_timeout:
                 return "fresh"
             else:
                 return "stale"
                 
-        except (json.JSONDecodeError, ValueError, KeyError) as e:
+        except (json.JSONDecodeError, OSError, PermissionError) as e:
-            self.logger.logger.warning(f"Invalid cache file format for {cache_file_path}: {e}")
+            self.logger.logger.warning(f"Cache file error for {cache_file_path}: {e}")
+            # FIXED: Try to remove corrupted cache file
+            try:
+                cache_file_path.unlink()
+                self.logger.logger.info(f"Removed corrupted cache file: {cache_file_path}")
+            except Exception:
+                pass
+            return "not_found"
+        except Exception as e:
+            self.logger.logger.error(f"Unexpected error checking cache status for {cache_file_path}: {e}")
             return "stale"
 
     def query_domain(self, domain: str) -> ProviderResult:
         """
-        FIXED: Query crt.sh for certificates containing the domain.
+        FIXED: Simplified and more robust domain querying with better error handling.
-        Now properly creates domain and CA nodes instead of large entities.
+        UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
-        
-        Args:
-            domain: Domain to investigate
-            
-        Returns:
-            ProviderResult containing discovered relationships and attributes
         """
         if not _is_valid_domain(domain):
             return ProviderResult()
@@ -111,121 +141,158 @@ class CrtShProvider(BaseProvider):
             return ProviderResult()
 
         cache_file = self._get_cache_file_path(domain)
-        cache_status = self._get_cache_status(cache_file)
-        
         result = ProviderResult()
 
-        if cache_status == "fresh":
+        try:
-            result = self._load_from_cache(cache_file)
+            cache_status = self._get_cache_status(cache_file)
-            self.logger.logger.info(f"Using fresh cached crt.sh data for {domain}")
             
-        else:  # "stale" or "not_found"
+            if cache_status == "fresh":
-            # Query the API for the latest certificates
+                # Load from cache
+                result = self._load_from_cache(cache_file)
+                if result and (result.relationships or result.attributes):
+                    self.logger.logger.debug(f"Using fresh cached crt.sh data for {domain}")
+                    return result
+                else:
+                    # Cache exists but is empty, treat as stale
+                    cache_status = "stale"
+            
+            # Need to query API (either no cache, stale cache, or empty cache)
+            self.logger.logger.debug(f"Querying crt.sh API for {domain} (cache status: {cache_status})")
             new_raw_certs = self._query_crtsh_api(domain)
             
             if self._stop_event and self._stop_event.is_set():
                 return ProviderResult()
 
-            # Combine with old data if cache is stale
+            # FIXED: Simplified processing - just process the new data
+            # Don't try to merge with stale cache as it can cause corruption
+            raw_certificates_to_process = new_raw_certs
+            
             if cache_status == "stale":
-                old_raw_certs = self._load_raw_data_from_cache(cache_file)
+                self.logger.logger.info(f"Refreshed stale cache for {domain} with {len(raw_certificates_to_process)} certs")
-                combined_certs = old_raw_certs + new_raw_certs
+            else:
+                self.logger.logger.info(f"Created fresh cache for {domain} with {len(raw_certificates_to_process)} certs")
             
-                # Deduplicate the combined list
-                seen_ids = set()
-                unique_certs = []
-                for cert in combined_certs:
-                    cert_id = cert.get('id')
-                    if cert_id not in seen_ids:
-                        unique_certs.append(cert)
-                        seen_ids.add(cert_id)
-                
-                raw_certificates_to_process = unique_certs
-                self.logger.logger.info(f"Refreshed and merged cache for {domain}. Total unique certs: {len(raw_certificates_to_process)}")
-            else:  # "not_found"
-                raw_certificates_to_process = new_raw_certs
-            
-            # FIXED: Process certificates to create proper domain and CA nodes
             result = self._process_certificates_to_result_fixed(domain, raw_certificates_to_process)
-            self.logger.logger.info(f"Created fresh result for {domain} ({result.get_relationship_count()} relationships)")
             
-            # Save the new result and the raw data to the cache
+            # Save the result to cache
             self._save_result_to_cache(cache_file, result, raw_certificates_to_process, domain)
             
-        return result
+            return result
+
+        except requests.exceptions.RequestException as e:
+            # FIXED: Don't re-raise network errors after long idle periods
+            # Instead return empty result and log the issue
+            self.logger.logger.warning(f"Network error querying crt.sh for {domain}: {e}")
+            
+            # Try to use stale cache if available
+            if cache_status == "stale":
+                try:
+                    stale_result = self._load_from_cache(cache_file)
+                    if stale_result and (stale_result.relationships or stale_result.attributes):
+                        self.logger.logger.info(f"Using stale cache for {domain} due to network error")
+                        return stale_result
+                except Exception as cache_error:
+                    self.logger.logger.warning(f"Failed to load stale cache for {domain}: {cache_error}")
+            
+            # Return empty result instead of raising - this prevents infinite retries
+            return ProviderResult()
+            
+        except Exception as e:
+            # FIXED: Handle any other exceptions gracefully
+            self.logger.logger.error(f"Unexpected error querying crt.sh for {domain}: {e}")
+            
+            # Try stale cache as fallback
+            try:
+                if cache_file.exists():
+                    fallback_result = self._load_from_cache(cache_file)
+                    if fallback_result and (fallback_result.relationships or fallback_result.attributes):
+                        self.logger.logger.info(f"Using cached data for {domain} due to processing error")
+                        return fallback_result
+            except Exception:
+                pass
+            
+            # Return empty result to prevent retries
+            return ProviderResult()
 
     def query_ip(self, ip: str) -> ProviderResult:
         """
-        Query crt.sh for certificates containing the IP address.
+        crt.sh does not support IP-based certificate queries effectively via its API.
-        Note: crt.sh doesn't typically index by IP, so this returns empty results.
-        
-        Args:
-            ip: IP address to investigate
-            
-        Returns:
-            Empty ProviderResult (crt.sh doesn't support IP-based certificate queries effectively)
         """
         return ProviderResult()
 
     def _load_from_cache(self, cache_file_path: Path) -> ProviderResult:
-        """Load processed crt.sh data from a cache file."""
+        """FIXED: More robust cache loading with better validation."""
         try:
-            with open(cache_file_path, 'r') as f:
+            if not cache_file_path.exists() or cache_file_path.stat().st_size == 0:
+                return ProviderResult()
+                
+            with open(cache_file_path, 'r', encoding='utf-8') as f:
                 cache_content = json.load(f)
             
+            if not isinstance(cache_content, dict):
+                self.logger.logger.warning(f"Invalid cache format in {cache_file_path}")
+                return ProviderResult()
+            
             result = ProviderResult()
             
-            # Reconstruct relationships
+            # Reconstruct relationships with validation
-            for rel_data in cache_content.get("relationships", []):
+            relationships = cache_content.get("relationships", [])
-                result.add_relationship(
+            if isinstance(relationships, list):
-                    source_node=rel_data["source_node"],
+                for rel_data in relationships:
-                    target_node=rel_data["target_node"],
+                    if not isinstance(rel_data, dict):
-                    relationship_type=rel_data["relationship_type"],
+                        continue
-                    provider=rel_data["provider"],
+                    try:
-                    confidence=rel_data["confidence"],
+                        result.add_relationship(
-                    raw_data=rel_data.get("raw_data", {})
+                            source_node=rel_data.get("source_node", ""),
-                )
+                            target_node=rel_data.get("target_node", ""),
+                            relationship_type=rel_data.get("relationship_type", ""),
+                            provider=rel_data.get("provider", self.name),
+                            raw_data=rel_data.get("raw_data", {})
+                        )
+                    except (ValueError, TypeError) as e:
+                        self.logger.logger.warning(f"Skipping invalid relationship in cache: {e}")
+                        continue
             
-            # Reconstruct attributes
+            # Reconstruct attributes with validation
-            for attr_data in cache_content.get("attributes", []):
+            attributes = cache_content.get("attributes", [])
-                result.add_attribute(
+            if isinstance(attributes, list):
-                    target_node=attr_data["target_node"],
+                for attr_data in attributes:
-                    name=attr_data["name"],
+                    if not isinstance(attr_data, dict):
-                    value=attr_data["value"],
+                        continue
-                    attr_type=attr_data["type"],
+                    try:
-                    provider=attr_data["provider"],
+                        result.add_attribute(
-                    confidence=attr_data["confidence"],
+                            target_node=attr_data.get("target_node", ""),
-                    metadata=attr_data.get("metadata", {})
+                            name=attr_data.get("name", ""),
-                )
+                            value=attr_data.get("value"),
+                            attr_type=attr_data.get("type", "unknown"),
+                            provider=attr_data.get("provider", self.name),
+                            metadata=attr_data.get("metadata", {})
+                        )
+                    except (ValueError, TypeError) as e:
+                        self.logger.logger.warning(f"Skipping invalid attribute in cache: {e}")
+                        continue
             
             return result
             
-        except (json.JSONDecodeError, FileNotFoundError, KeyError) as e:
+        except (json.JSONDecodeError, OSError, PermissionError) as e:
-            self.logger.logger.error(f"Failed to load cached certificates from {cache_file_path}: {e}")
+            self.logger.logger.warning(f"Failed to load cache from {cache_file_path}: {e}")
+            return ProviderResult()
+        except Exception as e:
+            self.logger.logger.error(f"Unexpected error loading cache from {cache_file_path}: {e}")
             return ProviderResult()
 
-    def _load_raw_data_from_cache(self, cache_file_path: Path) -> List[Dict[str, Any]]:
-        """Load only the raw certificate data from a cache file."""
-        try:
-            with open(cache_file_path, 'r') as f:
-                cache_content = json.load(f)
-            return cache_content.get("raw_certificates", [])
-        except (json.JSONDecodeError, FileNotFoundError):
-            return []
-
     def _save_result_to_cache(self, cache_file_path: Path, result: ProviderResult, raw_certificates: List[Dict[str, Any]], domain: str) -> None:
-        """Save processed crt.sh result and raw data to a cache file."""
+        """FIXED: More robust cache saving with atomic writes."""
         try:
             cache_data = {
                 "domain": domain,
                 "last_upstream_query": datetime.now(timezone.utc).isoformat(),
-                "raw_certificates": raw_certificates, # Store the raw data for deduplication
+                "raw_certificates": raw_certificates,
                 "relationships": [
                     {
                         "source_node": rel.source_node,
                         "target_node": rel.target_node,
                         "relationship_type": rel.relationship_type,
-                        "confidence": rel.confidence,
                         "provider": rel.provider,
                         "raw_data": rel.raw_data
                     } for rel in result.relationships
@@ -237,40 +304,73 @@ class CrtShProvider(BaseProvider):
                         "value": attr.value,
                         "type": attr.type,
                         "provider": attr.provider,
-                        "confidence": attr.confidence,
                         "metadata": attr.metadata
                     } for attr in result.attributes
                 ]
             }
+            
             cache_file_path.parent.mkdir(parents=True, exist_ok=True)
-            with open(cache_file_path, 'w') as f:
+            
-                json.dump(cache_data, f, separators=(',', ':'), default=str)
+            # FIXED: Atomic write using temporary file
+            temp_file = cache_file_path.with_suffix('.tmp')
+            try:
+                with open(temp_file, 'w', encoding='utf-8') as f:
+                    json.dump(cache_data, f, separators=(',', ':'), default=str, ensure_ascii=False)
+                
+                # Atomic rename
+                temp_file.replace(cache_file_path)
+                self.logger.logger.debug(f"Saved cache for {domain} ({len(result.relationships)} relationships)")
+                
+            except Exception as e:
+                # Clean up temp file on error
+                if temp_file.exists():
+                    try:
+                        temp_file.unlink()
+                    except Exception:
+                        pass
+                raise e
+                
         except Exception as e:
             self.logger.logger.warning(f"Failed to save cache file for {domain}: {e}")
     
     def _query_crtsh_api(self, domain: str) -> List[Dict[str, Any]]:
-        """Query crt.sh API for raw certificate data."""
+        """FIXED: More robust API querying with better error handling."""
         url = f"{self.base_url}?q={quote(domain)}&output=json"
-        response = self.make_request(url, target_indicator=domain)
-        
-        if not response or response.status_code != 200:
-            raise requests.exceptions.RequestException(f"crt.sh API returned status {response.status_code if response else 'None'}")
         
         try:
-            certificates = response.json()
+            response = self.make_request(url, target_indicator=domain)
-        except json.JSONDecodeError:
-            self.logger.logger.error(f"crt.sh returned invalid JSON for {domain}")
-            return []
             
-        if not certificates:
+            if not response:
-            return []
+                self.logger.logger.warning(f"No response from crt.sh for {domain}")
+                return []
             
-        return certificates
+            if response.status_code != 200:
+                self.logger.logger.warning(f"crt.sh returned status {response.status_code} for {domain}")
+                return []
+            
+            # FIXED: Better JSON parsing with error handling
+            try:
+                certificates = response.json()
+            except json.JSONDecodeError as e:
+                self.logger.logger.error(f"crt.sh returned invalid JSON for {domain}: {e}")
+                return []
+
+            if not certificates or not isinstance(certificates, list):
+                self.logger.logger.debug(f"crt.sh returned no certificates for {domain}")
+                return []
+            
+            self.logger.logger.debug(f"crt.sh returned {len(certificates)} certificates for {domain}")
+            return certificates
+            
+        except Exception as e:
+            self.logger.logger.error(f"Error querying crt.sh API for {domain}: {e}")
+            raise e
 
     def _process_certificates_to_result_fixed(self, query_domain: str, certificates: List[Dict[str, Any]]) -> ProviderResult:
         """
-        FIXED: Process certificates to create proper domain and CA nodes.
+        Process certificates to create proper domain and CA nodes.
-        Now creates individual domain nodes instead of large entities.
+        FIXED: Better error handling and progress tracking.
+        UPDATED: Enhanced with certificate timestamps for time-based edge coloring.
         """
         result = ProviderResult()
 
@@ -278,6 +378,11 @@ class CrtShProvider(BaseProvider):
             self.logger.logger.info(f"CrtSh processing cancelled before processing for domain: {query_domain}")
             return result
         
+        if not certificates:
+            self.logger.logger.debug(f"No certificates to process for {query_domain}")
+            return result
+        
+        # Check for incomplete data warning
         incompleteness_warning = self._check_for_incomplete_data(query_domain, certificates)
         if incompleteness_warning:
             result.add_attribute(
@@ -285,61 +390,84 @@ class CrtShProvider(BaseProvider):
                 name="crtsh_data_warning",
                 value=incompleteness_warning,
                 attr_type='metadata',
-                provider=self.name,
+                provider=self.name
-                confidence=1.0
             )
 
         all_discovered_domains = set()
         processed_issuers = set()
+        processed_certs = 0
 
         for i, cert_data in enumerate(certificates):
-            if i % 10 == 0 and self._stop_event and self._stop_event.is_set():
+            # FIXED: More frequent stop checks and progress logging
-                self.logger.logger.info(f"CrtSh processing cancelled at certificate {i} for domain: {query_domain}")
+            if i % 5 == 0:
-                break
+                if self._stop_event and self._stop_event.is_set():
+                    self.logger.logger.info(f"CrtSh processing cancelled at certificate {i}/{len(certificates)} for domain: {query_domain}")
+                    break
                 
-            # Extract all domains from this certificate
+                if i > 0 and i % 100 == 0:
-            cert_domains = self._extract_domains_from_certificate(cert_data)
+                    self.logger.logger.debug(f"Processed {i}/{len(certificates)} certificates for {query_domain}")
-            all_discovered_domains.update(cert_domains)
 
-            # FIXED: Create CA nodes for certificate issuers (not as domain metadata)
+            try:
-            issuer_name = self._parse_issuer_organization(cert_data.get('issuer_name', ''))
+                # Extract all domains from this certificate
-            if issuer_name and issuer_name not in processed_issuers:
+                cert_domains = self._extract_domains_from_certificate(cert_data)
-                # Create relationship from query domain to CA
+                if cert_domains:
-                result.add_relationship(
+                    all_discovered_domains.update(cert_domains)
-                    source_node=query_domain,
-                    target_node=issuer_name,
-                    relationship_type='crtsh_cert_issuer',
-                    provider=self.name,
-                    confidence=0.95,
-                    raw_data={'issuer_dn': cert_data.get('issuer_name', '')}
-                )
-                processed_issuers.add(issuer_name)
 
-            # Add certificate metadata to each domain in this certificate
+                # Create CA nodes for certificate issuers with timestamp
-            cert_metadata = self._extract_certificate_metadata(cert_data)
+                issuer_name = self._parse_issuer_organization(cert_data.get('issuer_name', ''))
-            for cert_domain in cert_domains:
+                if issuer_name and issuer_name not in processed_issuers:
-                if not _is_valid_domain(cert_domain):
+                    # Enhanced raw_data with certificate timestamp for time-based edge coloring
-                    continue
+                    issuer_raw_data = {'issuer_dn': cert_data.get('issuer_name', '')}
                     
-                # Add certificate attributes to the domain
+                    # Add certificate issue date (not_before) as relevance timestamp
-                for key, value in cert_metadata.items():
+                    not_before = cert_data.get('not_before')
-                    if value is not None:
+                    if not_before:
-                        result.add_attribute(
+                        try:
-                            target_node=cert_domain,
+                            not_before_date = self._parse_certificate_date(not_before)
-                            name=f"cert_{key}",
+                            issuer_raw_data['cert_not_before'] = not_before_date.isoformat()
-                            value=value,
+                            issuer_raw_data['relevance_timestamp'] = not_before_date.isoformat()  # Standardized field
-                            attr_type='certificate_data',
+                        except Exception as e:
-                            provider=self.name,
+                            self.logger.logger.debug(f"Failed to parse not_before date for issuer: {e}")
-                            confidence=0.9,
-                            metadata={'certificate_id': cert_data.get('id')}
-                        )
                     
+                    result.add_relationship(
+                        source_node=query_domain,
+                        target_node=issuer_name,
+                        relationship_type='crtsh_cert_issuer',
+                        provider=self.name,
+                        raw_data=issuer_raw_data
+                    )
+                    processed_issuers.add(issuer_name)
+
+                # Add certificate metadata to each domain in this certificate
+                cert_metadata = self._extract_certificate_metadata(cert_data)
+                for cert_domain in cert_domains:
+                    if not _is_valid_domain(cert_domain):
+                        continue
+
+                    for key, value in cert_metadata.items():
+                        if value is not None:
+                            result.add_attribute(
+                                target_node=cert_domain,
+                                name=f"cert_{key}",
+                                value=value,
+                                attr_type='certificate_data',
+                                provider=self.name,
+                                metadata={'certificate_id': cert_data.get('id')}
+                            )
+                
+                processed_certs += 1
+                
+            except Exception as e:
+                self.logger.logger.warning(f"Error processing certificate {i} for {query_domain}: {e}")
+                continue
+        
+        # Check for stop event before creating final relationships
         if self._stop_event and self._stop_event.is_set():
             self.logger.logger.info(f"CrtSh query cancelled before relationship creation for domain: {query_domain}")
             return result
 
-        # FIXED: Create selective relationships to avoid large entities
+        # Create selective relationships to avoid large entities with enhanced timestamps
-        # Only create relationships to domains that are closely related
+        relationships_created = 0
         for discovered_domain in all_discovered_domains:
             if discovered_domain == query_domain:
                 continue
@@ -347,48 +475,80 @@ class CrtShProvider(BaseProvider):
             if not _is_valid_domain(discovered_domain):
                 continue
 
-            # FIXED: Only create relationships for domains that share a meaningful connection
-            # This prevents creating too many relationships that trigger large entity creation
             if self._should_create_relationship(query_domain, discovered_domain):
-                confidence = self._calculate_domain_relationship_confidence(
+                # Enhanced raw_data with certificate timestamp for domain relationships
-                    query_domain, discovered_domain, [], all_discovered_domains
+                domain_raw_data = {'relationship_type': 'certificate_discovery'}
+                
+                # Find the most recent certificate for this domain pair to use as timestamp
+                most_recent_cert = self._find_most_recent_cert_for_domains(
+                    certificates, query_domain, discovered_domain
                 )
+                if most_recent_cert:
+                    not_before = most_recent_cert.get('not_before')
+                    if not_before:
+                        try:
+                            not_before_date = self._parse_certificate_date(not_before)
+                            domain_raw_data['cert_not_before'] = not_before_date.isoformat()
+                            domain_raw_data['relevance_timestamp'] = not_before_date.isoformat()
+                        except Exception as e:
+                            self.logger.logger.debug(f"Failed to parse not_before date for domain relationship: {e}")
 
                 result.add_relationship(
                     source_node=query_domain,
                     target_node=discovered_domain,
                     relationship_type='crtsh_san_certificate',
                     provider=self.name,
-                    confidence=confidence,
+                    raw_data=domain_raw_data
-                    raw_data={'relationship_type': 'certificate_discovery'}
                 )
 
                 self.log_relationship_discovery(
                     source_node=query_domain,
                     target_node=discovered_domain,
                     relationship_type='crtsh_san_certificate',
-                    confidence_score=confidence,
+                    raw_data=domain_raw_data,
-                    raw_data={'relationship_type': 'certificate_discovery'},
                     discovery_method="certificate_transparency_analysis"
                 )
+                relationships_created += 1
 
-        self.logger.logger.info(f"CrtSh processing completed for {query_domain}: {len(all_discovered_domains)} domains, {result.get_relationship_count()} relationships")
+        self.logger.logger.info(f"CrtSh processing completed for {query_domain}: processed {processed_certs}/{len(certificates)} certificates, {len(all_discovered_domains)} domains, {relationships_created} relationships")
         return result
 
+    def _find_most_recent_cert_for_domains(self, certificates: List[Dict[str, Any]], 
+                                         domain1: str, domain2: str) -> Optional[Dict[str, Any]]:
+        """
+        Find the most recent certificate that contains both domains.
+        Used for determining the relevance timestamp for domain relationships.
+        """
+        most_recent_cert = None
+        most_recent_date = None
+        
+        for cert in certificates:
+            # Check if this certificate contains both domains
+            cert_domains = self._extract_domains_from_certificate(cert)
+            if domain1 in cert_domains and domain2 in cert_domains:
+                not_before = cert.get('not_before')
+                if not_before:
+                    try:
+                        cert_date = self._parse_certificate_date(not_before)
+                        if most_recent_date is None or cert_date > most_recent_date:
+                            most_recent_date = cert_date
+                            most_recent_cert = cert
+                    except Exception:
+                        continue
+        
+        return most_recent_cert
+
+    # [Rest of the methods remain the same as in the original file]
     def _should_create_relationship(self, source_domain: str, target_domain: str) -> bool:
         """
-        FIXED: Determine if a relationship should be created between two domains.
+        Determine if a relationship should be created between two domains.
-        This helps avoid creating too many relationships that trigger large entity creation.
         """
-        # Always create relationships for subdomains
         if target_domain.endswith(f'.{source_domain}') or source_domain.endswith(f'.{target_domain}'):
             return True
         
-        # Create relationships for domains that share a common parent (up to 2 levels)
         source_parts = source_domain.split('.')
         target_parts = target_domain.split('.')
         
-        # Check if they share the same root domain (last 2 parts)
         if len(source_parts) >= 2 and len(target_parts) >= 2:
             source_root = '.'.join(source_parts[-2:])
             target_root = '.'.join(target_parts[-2:])
@@ -422,7 +582,6 @@ class CrtShProvider(BaseProvider):
                 metadata['is_currently_valid'] = self._is_cert_valid(cert_data)
                 metadata['expires_soon'] = (not_after - datetime.now(timezone.utc)).days <= 30
                 
-                # Keep raw date format or convert to standard format
                 metadata['not_before'] = not_before.isoformat()
                 metadata['not_after'] = not_after.isoformat()
                 
@@ -504,14 +663,12 @@ class CrtShProvider(BaseProvider):
         """Extract all domains from certificate data."""
         domains = set()
         
-        # Extract from common name
         common_name = cert_data.get('common_name', '')
         if common_name:
             cleaned_cn = self._clean_domain_name(common_name)
             if cleaned_cn:
                 domains.update(cleaned_cn)
         
-        # Extract from name_value field (contains SANs)
         name_value = cert_data.get('name_value', '')
         if name_value:
             for line in name_value.split('\n'):
@@ -552,26 +709,6 @@ class CrtShProvider(BaseProvider):
 
         return [d for d in final_domains if _is_valid_domain(d)]
 
-    def _calculate_domain_relationship_confidence(self, domain1: str, domain2: str, 
-                                                shared_certificates: List[Dict[str, Any]],
-                                                all_discovered_domains: Set[str]) -> float:
-        """Calculate confidence score for domain relationship based on various factors."""
-        base_confidence = 0.9
-        
-        # Adjust confidence based on domain relationship context
-        relationship_context = self._determine_relationship_context(domain2, domain1)
-        
-        if relationship_context == 'exact_match':
-            context_bonus = 0.0
-        elif relationship_context == 'subdomain':
-            context_bonus = 0.1
-        elif relationship_context == 'parent_domain':
-            context_bonus = 0.05
-        else:
-            context_bonus = 0.0
-        
-        final_confidence = base_confidence + context_bonus
-        return max(0.1, min(1.0, final_confidence))
 
     def _determine_relationship_context(self, cert_domain: str, query_domain: str) -> str:
         """Determine the context of the relationship between certificate domain and query domain."""
@@ -590,12 +727,10 @@ class CrtShProvider(BaseProvider):
         """
         cert_count = len(certificates)
 
-        # Heuristic 1: Check if the number of certs hits a known hard limit.
         if cert_count >= 10000:
             return f"Result likely truncated; received {cert_count} certificates, which may be the maximum limit."
 
-        # Heuristic 2: Check if all returned certificates are old.
+        if cert_count > 1000: 
-        if cert_count > 1000: # Only apply this for a reasonable number of certs
             latest_expiry = None
             for cert in certificates:
                 try:

providers/dns_provider.py

@@ -1,7 +1,8 @@
-# dnsrecon/providers/dns_provider.py
+# DNScope/providers/dns_provider.py
 
 from dns import resolver, reversename
 from typing import Dict
+from datetime import datetime, timezone
 from .base_provider import BaseProvider
 from core.provider_result import ProviderResult
 from utils.helpers import _is_valid_ip, _is_valid_domain, get_ip_version
@@ -11,7 +12,7 @@ class DNSProvider(BaseProvider):
     """
     Provider for standard DNS resolution and reverse DNS lookups.
     Now returns standardized ProviderResult objects with IPv4 and IPv6 support.
-    FIXED: Enhanced pickle support to prevent resolver serialization issues.
+    UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
     """
 
     def __init__(self, name=None, session_config=None):
@@ -28,22 +29,6 @@ class DNSProvider(BaseProvider):
         self.resolver.timeout = 5
         self.resolver.lifetime = 10
 
-    def __getstate__(self):
-        """Prepare the object for pickling by excluding resolver."""
-        state = super().__getstate__()
-        # Remove the unpickleable 'resolver' attribute
-        if 'resolver' in state:
-            del state['resolver']
-        return state
-
-    def __setstate__(self, state):
-        """Restore the object after unpickling by reconstructing resolver."""
-        super().__setstate__(state)
-        # Re-initialize the 'resolver' attribute
-        self.resolver = resolver.Resolver()
-        self.resolver.timeout = 5
-        self.resolver.lifetime = 10
-
     def get_name(self) -> str:
         """Return the provider name."""
         return "dns"
@@ -68,6 +53,7 @@ class DNSProvider(BaseProvider):
         """
         Query DNS records for the domain to discover relationships and attributes.
         FIXED: Now creates separate attributes for each DNS record type.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
         
         Args:
             domain: Domain to investigate
@@ -79,11 +65,12 @@ class DNSProvider(BaseProvider):
             return ProviderResult()
 
         result = ProviderResult()
+        discovery_time = datetime.now(timezone.utc)
 
         # Query all record types - each gets its own attribute
         for record_type in ['A', 'AAAA', 'CNAME', 'MX', 'NS', 'SOA', 'TXT', 'SRV', 'CAA']:
             try:
-                self._query_record(domain, record_type, result)
+                self._query_record(domain, record_type, result, discovery_time)
             #except resolver.NoAnswer:
                 # This is not an error, just a confirmation that the record doesn't exist.
                 #self.logger.logger.debug(f"No {record_type} record found for {domain}")
@@ -96,6 +83,7 @@ class DNSProvider(BaseProvider):
     def query_ip(self, ip: str) -> ProviderResult:
         """
         Query reverse DNS for the IP address (supports both IPv4 and IPv6).
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
 
         Args:
             ip: IP address to investigate (IPv4 or IPv6)
@@ -108,6 +96,7 @@ class DNSProvider(BaseProvider):
 
         result = ProviderResult()
         ip_version = get_ip_version(ip)
+        discovery_time = datetime.now(timezone.utc)
 
         try:
             # Perform reverse DNS lookup (works for both IPv4 and IPv6)
@@ -123,26 +112,30 @@ class DNSProvider(BaseProvider):
                 if _is_valid_domain(hostname):
                     # Determine appropriate forward relationship type based on IP version
                     if ip_version == 6:
-                        relationship_type = 'shodan_aaaa_record'
+                        relationship_type = 'dns_aaaa_record'
                         record_prefix = 'AAAA'
                     else:
-                        relationship_type = 'shodan_a_record'
+                        relationship_type = 'dns_a_record'
                         record_prefix = 'A'
                     
+                    # Enhanced raw_data with discovery timestamp for time-based edge coloring
+                    raw_data = {
+                        'query_type': 'PTR',
+                        'ip_address': ip,
+                        'ip_version': ip_version,
+                        'hostname': hostname,
+                        'ttl': response.ttl,
+                        'discovery_timestamp': discovery_time.isoformat(),
+                        'relevance_timestamp': discovery_time.isoformat()  # DNS data is "fresh" when discovered
+                    }
+                    
                     # Add the relationship
                     result.add_relationship(
                         source_node=ip,
                         target_node=hostname,
                         relationship_type='dns_ptr_record',
                         provider=self.name,
-                        confidence=0.8,
+                        raw_data=raw_data
-                        raw_data={
-                            'query_type': 'PTR',
-                            'ip_address': ip,
-                            'ip_version': ip_version,
-                            'hostname': hostname,
-                            'ttl': response.ttl
-                        }
                     )
 
                     # Add to PTR records list
@@ -153,14 +146,7 @@ class DNSProvider(BaseProvider):
                         source_node=ip,
                         target_node=hostname,
                         relationship_type='dns_ptr_record',
-                        confidence_score=0.8,
+                        raw_data=raw_data,
-                        raw_data={
-                            'query_type': 'PTR',
-                            'ip_address': ip,
-                            'ip_version': ip_version,
-                            'hostname': hostname,
-                            'ttl': response.ttl
-                        },
                         discovery_method=f"reverse_dns_lookup_ipv{ip_version}"
                     )
 
@@ -172,7 +158,6 @@ class DNSProvider(BaseProvider):
                     value=ptr_records,
                     attr_type='dns_record',
                     provider=self.name,
-                    confidence=0.8,
                     metadata={'ttl': response.ttl, 'ip_version': ip_version}
                 )
 
@@ -187,10 +172,11 @@ class DNSProvider(BaseProvider):
 
         return result
 
-    def _query_record(self, domain: str, record_type: str, result: ProviderResult) -> None:
+    def _query_record(self, domain: str, record_type: str, result: ProviderResult, discovery_time: datetime) -> None:
         """
         FIXED: Query DNS records with unique attribute names for each record type.
         Enhanced to better handle IPv6 AAAA records.
+        UPDATED: Enhanced with discovery timestamps for time-based edge coloring.
         """
         try:
             self.total_requests += 1
@@ -234,18 +220,20 @@ class DNSProvider(BaseProvider):
                     if record_type in ['A', 'AAAA'] and _is_valid_ip(target):
                         ip_version = get_ip_version(target)
                     
+                    # Enhanced raw_data with discovery timestamp for time-based edge coloring
                     raw_data = {
                         'query_type': record_type,
                         'domain': domain,
                         'value': target,
-                        'ttl': response.ttl
+                        'ttl': response.ttl,
+                        'discovery_timestamp': discovery_time.isoformat(),
+                        'relevance_timestamp': discovery_time.isoformat()  # DNS data is "fresh" when discovered
                     }
                     
                     if ip_version:
                         raw_data['ip_version'] = ip_version
                     
                     relationship_type = f"dns_{record_type.lower()}_record"
-                    confidence = 0.8
 
                     # Add relationship
                     result.add_relationship(
@@ -253,7 +241,6 @@ class DNSProvider(BaseProvider):
                         target_node=target,
                         relationship_type=relationship_type,
                         provider=self.name,
-                        confidence=confidence,
                         raw_data=raw_data
                     )
 
@@ -269,7 +256,6 @@ class DNSProvider(BaseProvider):
                         source_node=domain,
                         target_node=target,
                         relationship_type=relationship_type,
-                        confidence_score=confidence,
                         raw_data=raw_data,
                         discovery_method=discovery_method
                     )
@@ -293,7 +279,6 @@ class DNSProvider(BaseProvider):
                     value=dns_records,
                     attr_type='dns_record_list',
                     provider=self.name,
-                    confidence=0.8,
                     metadata=metadata
                 )
 

providers/shodan_provider.py

@@ -1,4 +1,4 @@
-# dnsrecon/providers/shodan_provider.py
+# DNScope/providers/shodan_provider.py
 
 import json
 from pathlib import Path
@@ -15,6 +15,7 @@ class ShodanProvider(BaseProvider):
     """
     Provider for querying Shodan API for IP address information.
     Now returns standardized ProviderResult objects with caching support for IPv4 and IPv6.
+    UPDATED: Enhanced with last_seen timestamp for time-based edge coloring.
     """
 
     def __init__(self, name=None, session_config=None):
@@ -36,15 +37,6 @@ class ShodanProvider(BaseProvider):
         self.cache_dir = Path('cache') / 'shodan'
         self.cache_dir.mkdir(parents=True, exist_ok=True)
 
-    def __getstate__(self):
-        """Prepare the object for pickling."""
-        state = super().__getstate__()
-        return state
-
-    def __setstate__(self, state):
-        """Restore the object after unpickling."""
-        super().__setstate__(state)
-
     def _check_api_connection(self) -> bool:
         """
         FIXED: Lazy connection checking - only test when actually needed.
@@ -154,6 +146,7 @@ class ShodanProvider(BaseProvider):
         """
         Query Shodan for information about an IP address (IPv4 or IPv6), with caching of processed data.
         FIXED: Proper 404 handling to prevent unnecessary retries.
+        UPDATED: Enhanced with last_seen timestamp extraction for time-based edge coloring.
         
         Args:
             ip: IP address to investigate (IPv4 or IPv6)
@@ -313,7 +306,6 @@ class ShodanProvider(BaseProvider):
                     target_node=rel_data["target_node"],
                     relationship_type=rel_data["relationship_type"],
                     provider=rel_data["provider"],
-                    confidence=rel_data["confidence"],
                     raw_data=rel_data.get("raw_data", {})
                 )
             
@@ -325,7 +317,6 @@ class ShodanProvider(BaseProvider):
                     value=attr_data["value"],
                     attr_type=attr_data["type"],
                     provider=attr_data["provider"],
-                    confidence=attr_data["confidence"],
                     metadata=attr_data.get("metadata", {})
                 )
             
@@ -345,7 +336,6 @@ class ShodanProvider(BaseProvider):
                         "source_node": rel.source_node,
                         "target_node": rel.target_node,
                         "relationship_type": rel.relationship_type,
-                        "confidence": rel.confidence,
                         "provider": rel.provider,
                         "raw_data": rel.raw_data
                     } for rel in result.relationships
@@ -357,7 +347,6 @@ class ShodanProvider(BaseProvider):
                         "value": attr.value,
                         "type": attr.type,
                         "provider": attr.provider,
-                        "confidence": attr.confidence,
                         "metadata": attr.metadata
                     } for attr in result.attributes
                 ]
@@ -371,25 +360,40 @@ class ShodanProvider(BaseProvider):
         """
         VERIFIED: Process Shodan data creating ISP nodes with ASN attributes and proper relationships.
         Enhanced to include IP version information for IPv6 addresses.
+        UPDATED: Enhanced with last_seen timestamp for time-based edge coloring.
         """
         result = ProviderResult()
         
         # Determine IP version for metadata
         ip_version = get_ip_version(ip)
 
+        # Extract last_seen timestamp for time-based edge coloring
+        last_seen = data.get('last_seen')
+        
         # VERIFIED: Extract ISP information and create proper ISP node with ASN
         isp_name = data.get('org')
         asn_value = data.get('asn')
 
         if isp_name and asn_value:
+            # Enhanced raw_data with last_seen timestamp
+            raw_data = {
+                'asn': asn_value, 
+                'shodan_org': isp_name, 
+                'ip_version': ip_version
+            }
+            
+            # Add last_seen timestamp if available
+            if last_seen:
+                raw_data['last_seen'] = last_seen
+                raw_data['relevance_timestamp'] = last_seen  # Standardized field for time-based coloring
+            
             # Create relationship from IP to ISP
             result.add_relationship(
                 source_node=ip,
                 target_node=isp_name,
                 relationship_type='shodan_isp',
                 provider=self.name,
-                confidence=0.9,
+                raw_data=raw_data
-                raw_data={'asn': asn_value, 'shodan_org': isp_name, 'ip_version': ip_version}
             )
             
             # Add ASN as attribute to the ISP node
@@ -399,7 +403,6 @@ class ShodanProvider(BaseProvider):
                 value=asn_value,
                 attr_type='isp_info',
                 provider=self.name,
-                confidence=0.9,
                 metadata={'description': 'Autonomous System Number from Shodan', 'ip_version': ip_version}
             )
             
@@ -410,7 +413,6 @@ class ShodanProvider(BaseProvider):
                 value=isp_name,
                 attr_type='isp_info',
                 provider=self.name,
-                confidence=0.9,
                 metadata={'description': 'Organization name from Shodan', 'ip_version': ip_version}
             )
 
@@ -425,20 +427,24 @@ class ShodanProvider(BaseProvider):
                         else:
                             relationship_type = 'shodan_a_record'
                         
+                        # Enhanced raw_data with last_seen timestamp
+                        hostname_raw_data = {**data, 'ip_version': ip_version}
+                        if last_seen:
+                            hostname_raw_data['last_seen'] = last_seen
+                            hostname_raw_data['relevance_timestamp'] = last_seen
+                        
                         result.add_relationship(
                             source_node=ip,
                             target_node=hostname,
                             relationship_type=relationship_type,
                             provider=self.name,
-                            confidence=0.8,
+                            raw_data=hostname_raw_data
-                            raw_data={**data, 'ip_version': ip_version}
                         )
                         self.log_relationship_discovery(
                             source_node=ip,
                             target_node=hostname,
                             relationship_type=relationship_type,
-                            confidence_score=0.8,
+                            raw_data=hostname_raw_data,
-                            raw_data={**data, 'ip_version': ip_version},
                             discovery_method=f"shodan_host_lookup_ipv{ip_version}"
                         )
             elif key == 'ports':
@@ -450,7 +456,6 @@ class ShodanProvider(BaseProvider):
                         value=port,
                         attr_type='shodan_network_info',
                         provider=self.name,
-                        confidence=0.9,
                         metadata={'ip_version': ip_version}
                     )
             elif isinstance(value, (str, int, float, bool)) and value is not None:
@@ -461,7 +466,6 @@ class ShodanProvider(BaseProvider):
                     value=value,
                     attr_type='shodan_info',
                     provider=self.name,
-                    confidence=0.9,
                     metadata={'ip_version': ip_version}
                 )
 

requirements.txt

@@ -9,5 +9,3 @@ gunicorn
 redis
 python-dotenv
 psycopg2-binary
-Flask-SocketIO
-eventlet

static/css/main.css

@@ -1,4 +1,4 @@
-/* DNSRecon - Optimized Compact Theme */
+/* DNScope - Optimized Compact Theme */
 
 /* Reset and Base */
 * {
@@ -326,6 +326,20 @@ input[type="text"]:focus, select:focus {
     animation: progressGlow 2s ease-in-out infinite alternate;
 }
 
+.gradient-bar {
+    height: 4px;
+    background: linear-gradient(to right, #6b7280, #00bfff);
+    border-radius: 2px;
+    margin: 0.2rem 0;
+}
+
+.gradient-labels {
+    display: flex;
+    justify-content: space-between;
+    font-size: 0.6rem;
+    color: #888;
+}
+
 @keyframes progressShimmer {
     0% { transform: translateX(-100%); }
     100% { transform: translateX(100%); }
@@ -380,32 +394,59 @@ input[type="text"]:focus, select:focus {
     color: #999;
 }
 
-/* Graph Controls */
+/* Enhanced graph controls layout */
 .graph-controls {
-    position: absolute;
-    top: 8px;
-    right: 8px;
-    z-index: 10;
     display: flex;
+    flex-direction: column;
     gap: 0.3rem;
+    position: absolute;
+    top: 10px;
+    right: 10px;
+    background: rgba(26, 26, 26, 0.9);
+    padding: 0.5rem;
+    border-radius: 6px;
+    border: 1px solid #444;
+    box-shadow: 0 4px 15px rgba(0, 0, 0, 0.5);
+    z-index: 100;
+    min-width: 200px;
 }
 
-.graph-control-btn, .btn-icon-small {
+.graph-control-btn {
-    background: rgba(42, 42, 42, 0.9);
+    background: linear-gradient(135deg, #2a2a2a 0%, #1e1e1e 100%);
     border: 1px solid #555;
     color: #c7c7c7;
-    padding: 0.3rem 0.5rem;
+    padding: 0.4rem 0.8rem;
-    font-family: 'Roboto Mono', monospace;
+    border-radius: 4px;
-    font-size: 0.7rem;
     cursor: pointer;
-    transition: all 0.3s ease;
+    font-family: 'Roboto Mono', monospace;
+    font-size: 0.8rem;
+    transition: all 0.2s ease;
+    text-align: center;
 }
 
-.graph-control-btn:hover, .btn-icon-small:hover {
+.graph-control-btn:hover {
+    background: linear-gradient(135deg, #3a3a3a 0%, #2e2e2e 100%);
     border-color: #00ff41;
     color: #00ff41;
 }
 
+.graph-control-btn:disabled {
+    opacity: 0.5;
+    cursor: not-allowed;
+}
+
+.manual-refresh-btn {
+    background: linear-gradient(135deg, #4a4a2a 0%, #3e3e1e 100%);
+    border-color: #ffaa00;
+    color: #ffaa00;
+}
+
+.manual-refresh-btn:hover {
+    background: linear-gradient(135deg, #5a5a3a 0%, #4e4e2e 100%);
+    color: #ffcc33;
+    border-color: #ffcc33;
+}
+
 .graph-filter-panel {
     position: absolute;
     bottom: 8px;
@@ -474,6 +515,7 @@ input[type="text"]:focus, select:focus {
     flex-wrap: wrap;
     gap: 0.75rem;
     align-items: center;
+    max-height: 3rem;
 }
 
 .legend-item {
@@ -499,14 +541,6 @@ input[type="text"]:focus, select:focus {
     height: 2px;
 }
 
-.legend-edge.high-confidence {
-    background: #00ff41;
-}
-
-.legend-edge.medium-confidence {
-    background: #ff9900;
-}
-
 /* Provider Panel */
 .provider-panel {
     grid-area: providers;
@@ -986,11 +1020,6 @@ input[type="text"]:focus, select:focus {
     border-radius: 2px;
 }
 
-.confidence-indicator {
-    font-size: 0.6rem;
-    letter-spacing: 1px;
-}
-
 .node-link-compact {
     color: #00aaff;
     text-decoration: none;
@@ -1094,6 +1123,56 @@ input[type="text"]:focus, select:focus {
     border-left: 3px solid #00aaff;
 }
 
+.time-control-container {
+    margin-bottom: 0.5rem;
+    padding: 0.5rem;
+    background: rgba(42, 42, 42, 0.3);
+    border-radius: 4px;
+    border: 1px solid #444;
+}
+
+.time-control-label {
+    font-size: 0.8rem;
+    color: #c7c7c7;
+    margin-bottom: 0.3rem;
+    display: block;
+    font-family: 'Roboto Mono', monospace;
+}
+
+.time-control-input {
+    width: 100%;
+    padding: 0.3rem;
+    background: #1a1a1a;
+    border: 1px solid #555;
+    border-radius: 3px;
+    color: #c7c7c7;
+    font-family: 'Roboto Mono', monospace;
+    font-size: 0.75rem;
+}
+
+.time-control-input:focus {
+    outline: none;
+    border-color: #00ff41;
+    box-shadow: 0 0 5px rgba(0, 255, 65, 0.3);
+}
+
+.time-gradient-info {
+    font-size: 0.7rem;
+    color: #999;
+    margin-top: 0.3rem;
+    text-align: center;
+    font-family: 'Roboto Mono', monospace;
+}
+
+/* Edge color legend for time-based gradient */
+.time-gradient-legend {
+    margin-top: 0.5rem;
+    padding: 0.3rem;
+    background: rgba(26, 26, 26, 0.5);
+    border-radius: 3px;
+    border: 1px solid #333;
+}
+
 /* Settings Modal Specific */
 .provider-toggle {
     appearance: none !important;
@@ -1323,4 +1402,16 @@ input[type="password"]:focus {
     .provider-list {
         grid-template-columns: 1fr;
     }
+
+    .graph-controls {
+        position: relative;
+        top: auto;
+        right: auto;
+        margin-bottom: 1rem;
+        min-width: auto;
+    }
+    
+    .time-control-input {
+        font-size: 0.7rem;
+    }
 }

static/js/main.js

@@ -1,15 +1,16 @@
 /**
- * Main application logic for DNSRecon web interface
+ * Main application logic for DNScope web interface
  * Handles UI interactions, API communication, and data flow
- * FIXED: Enhanced real-time WebSocket graph updates
+ * UPDATED: Now compatible with a strictly flat, unified data model for attributes.
  */
 
-class DNSReconApp {
+class DNScopeApp {
     constructor() {
-        console.log('DNSReconApp constructor called');
+        console.log('DNScopeApp constructor called');
         this.graphManager = null;
-        this.socket = null;
         this.scanStatus = 'idle';
+        this.statusPollInterval = null;      // Separate status polling
+        this.graphPollInterval = null;       // Separate graph polling
         this.currentSessionId = null;
         
         this.elements = {};
@@ -17,13 +18,9 @@ class DNSReconApp {
         this.isScanning = false;
         this.lastGraphUpdate = null;
         
-        // FIXED: Add connection state tracking
+        // Graph polling optimization
-        this.isConnected = false;
+        this.graphPollingNodeThreshold = 500;  // Default, will be loaded from config
-        this.reconnectAttempts = 0;
+        this.graphPollingEnabled = true;
-        this.maxReconnectAttempts = 5;
-        
-        // FIXED: Track last graph data for debugging
-        this.lastGraphData = null;
         
         this.init();
     }
@@ -32,179 +29,41 @@ class DNSReconApp {
      * Initialize the application
      */
     init() {
-        console.log('DNSReconApp init called');
+        console.log('DNScopeApp init called');
         document.addEventListener('DOMContentLoaded', () => {
             console.log('DOM loaded, initializing application...');
             try {
                 this.initializeElements();
                 this.setupEventHandlers();
                 this.initializeGraph();
-                this.initializeSocket();
+                this.updateStatus();
                 this.loadProviders();
                 this.initializeEnhancedModals();
                 this.addCheckboxStyling();
+                this.loadConfig();  // Load configuration including threshold
                 
-                console.log('DNSRecon application initialized successfully');
+                this.updateGraph();
+                
+                console.log('DNScope application initialized successfully');
             } catch (error) {
-                console.error('Failed to initialize DNSRecon application:', error);
+                console.error('Failed to initialize DNScope application:', error);
                 this.showError(`Initialization failed: ${error.message}`);
             }
         });
     }
 
-    initializeSocket() {
+    /**
-        console.log('🔌 Initializing WebSocket connection...');
+     * Load configuration from backend
-        
+     */
+    async loadConfig() {
         try {
-            this.socket = io({
+            const response = await this.apiCall('/api/config');
-                transports: ['websocket', 'polling'],
+            if (response.success) {
-                timeout: 10000,
+                this.graphPollingNodeThreshold = response.config.graph_polling_node_threshold;
-                reconnection: true,
+                console.log(`Graph polling threshold set to: ${this.graphPollingNodeThreshold} nodes`);
-                reconnectionAttempts: 5,
+            }
-                reconnectionDelay: 2000
-            });
-
-            this.socket.on('connect', () => {
-                console.log('✅ WebSocket connected successfully');
-                this.isConnected = true;
-                this.reconnectAttempts = 0;
-                this.updateConnectionStatus('idle');
-                
-                console.log('📡 Requesting initial status...');
-                this.socket.emit('get_status');
-            });
-
-            this.socket.on('disconnect', (reason) => {
-                console.log('❌ WebSocket disconnected:', reason);
-                this.isConnected = false;
-                this.updateConnectionStatus('error');
-            });
-
-            this.socket.on('connect_error', (error) => {
-                console.error('❌ WebSocket connection error:', error);
-                this.reconnectAttempts++;
-                this.updateConnectionStatus('error');
-                
-                if (this.reconnectAttempts >= 5) {
-                    this.showError('WebSocket connection failed. Please refresh the page.');
-                }
-            });
-
-            this.socket.on('reconnect', (attemptNumber) => {
-                console.log('✅ WebSocket reconnected after', attemptNumber, 'attempts');
-                this.isConnected = true;
-                this.reconnectAttempts = 0;
-                this.updateConnectionStatus('idle');
-                this.socket.emit('get_status');
-            });
-
-            // FIXED: Enhanced scan_update handler with detailed graph processing and debugging
-            this.socket.on('scan_update', (data) => {
-                console.log('📨 WebSocket update received:', {
-                    status: data.status,
-                    target: data.target_domain,
-                    progress: data.progress_percentage,
-                    graphNodes: data.graph?.nodes?.length || 0,
-                    graphEdges: data.graph?.edges?.length || 0,
-                    timestamp: new Date().toISOString()
-                });
-                
-                try {
-                    // Handle status change
-                    if (data.status !== this.scanStatus) {
-                        console.log(`📄 Status change: ${this.scanStatus} → ${data.status}`);
-                        this.handleStatusChange(data.status, data.task_queue_size);
-                    }
-                    this.scanStatus = data.status;
-                    
-                    // Update status display
-                    this.updateStatusDisplay(data);
-                    
-                    // FIXED: Always update graph if data is present and graph manager exists
-                    if (data.graph && this.graphManager) {
-                        console.log('📊 Processing graph update:', {
-                            nodes: data.graph.nodes?.length || 0,
-                            edges: data.graph.edges?.length || 0,
-                            hasNodes: Array.isArray(data.graph.nodes),
-                            hasEdges: Array.isArray(data.graph.edges),
-                            isInitialized: this.graphManager.isInitialized
-                        });
-                        
-                        // FIXED: Initialize graph manager if not already done
-                        if (!this.graphManager.isInitialized) {
-                            console.log('🎯 Initializing graph manager...');
-                            this.graphManager.initialize();
-                        }
-                        
-                        // FIXED: Force graph update and verify it worked
-                        const previousNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
-                        const previousEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
-                        
-                        console.log('🔄 Before update - Nodes:', previousNodeCount, 'Edges:', previousEdgeCount);
-                        
-                        // Store the data for debugging
-                        this.lastGraphData = data.graph;
-                        
-                        // Update the graph
-                        this.graphManager.updateGraph(data.graph);
-                        this.lastGraphUpdate = Date.now();
-                        
-                        // Verify the update worked
-                        const newNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
-                        const newEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
-                        
-                        console.log('🔄 After update - Nodes:', newNodeCount, 'Edges:', newEdgeCount);
-                        
-                        if (newNodeCount !== data.graph.nodes.length || newEdgeCount !== data.graph.edges.length) {
-                            console.warn('⚠️ Graph update mismatch!', {
-                                expectedNodes: data.graph.nodes.length,
-                                actualNodes: newNodeCount,
-                                expectedEdges: data.graph.edges.length,
-                                actualEdges: newEdgeCount
-                            });
-                            
-                            // Force a complete rebuild if there's a mismatch
-                            console.log('🔧 Force rebuilding graph...');
-                            this.graphManager.clear();
-                            this.graphManager.updateGraph(data.graph);
-                        }
-                        
-                        console.log('✅ Graph updated successfully');
-                        
-                        // FIXED: Force network redraw if we're using vis.js
-                        if (this.graphManager.network) {
-                            try {
-                                this.graphManager.network.redraw();
-                                console.log('🎨 Network redrawn');
-                            } catch (redrawError) {
-                                console.warn('⚠️ Network redraw failed:', redrawError);
-                            }
-                        }
-                        
-                    } else {
-                        if (!data.graph) {
-                            console.log('⚠️ No graph data in WebSocket update');
-                        }
-                        if (!this.graphManager) {
-                            console.log('⚠️ Graph manager not available');
-                        }
-                    }
-                    
-                } catch (error) {
-                    console.error('❌ Error processing WebSocket update:', error);
-                    console.error('Update data:', data);
-                    console.error('Stack trace:', error.stack);
-                }
-            });
-
-            this.socket.on('error', (error) => {
-                console.error('❌ WebSocket error:', error);
-                this.showError('WebSocket communication error');
-            });
-
         } catch (error) {
-            console.error('❌ Failed to initialize WebSocket:', error);
+            console.warn('Failed to load config, using defaults:', error);
-            this.showError('Failed to establish real-time connection');
         }
     }
         
@@ -365,12 +224,6 @@ class DNSReconApp {
                     if (e.target === this.elements.settingsModal) this.hideSettingsModal();
                 });
             }
-            if (this.elements.saveApiKeys) {
-                this.elements.saveApiKeys.removeEventListener('click', this.saveApiKeys);
-            }
-            if (this.elements.resetApiKeys) {
-                this.elements.resetApiKeys.removeEventListener('click', this.resetApiKeys);
-            }
             
             // Setup new handlers
             const saveSettingsBtn = document.getElementById('save-settings');
@@ -425,35 +278,18 @@ class DNSReconApp {
     }
     
     /**
-     * FIXED: Initialize graph visualization with enhanced debugging
+     * Initialize graph visualization with manual refresh button
      */
     initializeGraph() {
         try {
             console.log('Initializing graph manager...');
             this.graphManager = new GraphManager('network-graph');
             
-            // FIXED: Add debugging hooks to graph manager
+            // Set up manual refresh handler
-            if (this.graphManager) {
+            this.graphManager.setManualRefreshHandler(() => {
-                // Override updateGraph to add debugging
+                console.log('Manual graph refresh requested');
-                const originalUpdateGraph = this.graphManager.updateGraph.bind(this.graphManager);
+                this.updateGraph();
-                this.graphManager.updateGraph = (graphData) => {
+            });
-                    console.log('🔧 GraphManager.updateGraph called with:', {
-                        nodes: graphData?.nodes?.length || 0,
-                        edges: graphData?.edges?.length || 0,
-                        timestamp: new Date().toISOString()
-                    });
-                    
-                    const result = originalUpdateGraph(graphData);
-                    
-                    console.log('🔧 GraphManager.updateGraph completed, network state:', {
-                        networkExists: !!this.graphManager.network,
-                        nodeDataSetLength: this.graphManager.nodes?.length || 0,
-                        edgeDataSetLength: this.graphManager.edges?.length || 0
-                    });
-                    
-                    return result;
-                };
-            }
             
             console.log('Graph manager initialized successfully');
         } catch (error) {
@@ -462,6 +298,34 @@ class DNSReconApp {
         }
     }
 
+    /**
+     * Check if graph polling should be enabled based on node count
+     */
+    shouldEnableGraphPolling() {
+        if (!this.graphManager || !this.graphManager.nodes) {
+            return true;
+        }
+        
+        const nodeCount = this.graphManager.nodes.length;
+        return nodeCount <= this.graphPollingNodeThreshold;
+    }
+    
+    /**
+     * Update manual refresh button visibility and state.
+     * The button will be visible whenever auto-polling is disabled,
+     * and enabled only when a scan is in progress.
+     */
+    updateManualRefreshButton() {
+        if (!this.graphManager || !this.graphManager.manualRefreshButton) return;
+
+        const shouldShow = !this.graphPollingEnabled;
+        this.graphManager.showManualRefreshButton(shouldShow);
+
+        if (shouldShow) {
+            this.graphManager.manualRefreshButton.disabled = false;
+        }
+    }
+    
     /**
      * Start scan with error handling
      */
@@ -474,6 +338,7 @@ class DNSReconApp {
             
             console.log(`Target: "${target}", Max depth: ${maxDepth}`);
             
+            // Validation
             if (!target) {
                 console.log('Validation failed: empty target');
                 this.showError('Please enter a target domain or IP');
@@ -488,19 +353,6 @@ class DNSReconApp {
                 return;
             }
             
-            // FIXED: Ensure WebSocket connection before starting scan
-            if (!this.isConnected) {
-                console.log('WebSocket not connected, attempting to connect...');
-                this.socket.connect();
-                
-                // Wait a moment for connection
-                await new Promise(resolve => setTimeout(resolve, 1000));
-                
-                if (!this.isConnected) {
-                    this.showWarning('WebSocket connection not established. Updates may be delayed.');
-                }
-            }
-            
             console.log('Validation passed, setting UI state to scanning...');
             this.setUIState('scanning');
             this.showInfo('Starting reconnaissance scan...');
@@ -518,28 +370,26 @@ class DNSReconApp {
                         
             if (response.success) {
                 this.currentSessionId = response.scan_id;
-                this.showSuccess('Reconnaissance scan started - watching for real-time updates');
+                this.showSuccess('Reconnaissance scan started successfully');
 
-                if (clearGraph && this.graphManager) {
+                if (clearGraph) {
-                    console.log('🧹 Clearing graph for new scan');
                     this.graphManager.clear();
+                    this.graphPollingEnabled = true; // Reset polling when starting fresh
                 }
                 
-                console.log(`✅ Scan started for ${target} with depth ${maxDepth}`);
+                console.log(`Scan started for ${target} with depth ${maxDepth}`);
                 
-                // FIXED: Immediately start listening for updates
+                // Start optimized polling
-                if (this.socket && this.isConnected) {
+                this.startOptimizedPolling();
-                    console.log('📡 Requesting initial status update...');
-                    this.socket.emit('get_status');
                 
-                    // Set up periodic status requests as backup (every 5 seconds during scan)
+                // Force an immediate status update
-                    /*this.statusRequestInterval = setInterval(() => {
+                console.log('Forcing immediate status update...');
-                        if (this.isScanning && this.socket && this.isConnected) {
+                setTimeout(() => {
-                            console.log('📡 Periodic status request...');
+                    this.updateStatus();
-                            this.socket.emit('get_status');
+                    if (this.graphPollingEnabled) {
-                        }
+                        this.updateGraph();
-                    }, 5000);*/
+                    }
-                }
+                }, 100);
                 
             } else {
                 throw new Error(response.error || 'Failed to start scan');
@@ -552,22 +402,48 @@ class DNSReconApp {
         }
     }
 
-    // FIXED: Enhanced stop scan with interval cleanup
+    /**
+     * Start optimized polling with separate status and graph intervals
+     */
+    startOptimizedPolling() {
+        console.log('=== STARTING OPTIMIZED POLLING ===');
+        
+        this.stopPolling(); // Stop any existing polling
+        
+        // Always poll status for progress bar
+        this.statusPollInterval = setInterval(() => {
+            this.updateStatus();
+            this.loadProviders();
+        }, 2000);
+        
+        // Only poll graph if enabled
+        if (this.graphPollingEnabled) {
+            this.graphPollInterval = setInterval(() => {
+                this.updateGraph();
+            }, 2000);
+            console.log('Graph polling started');
+        } else {
+            console.log('Graph polling disabled due to node count');
+        }
+        
+        this.updateManualRefreshButton();
+        console.log(`Optimized polling started - Status: enabled, Graph: ${this.graphPollingEnabled ? 'enabled' : 'disabled'}`);
+    }
+
+    /**
+     * Scan stop with immediate UI feedback
+     */
     async stopScan() {
         try {
             console.log('Stopping scan...');
             
-            // Clear status request interval
+            // Immediately disable stop button and show stopping state
-            /*if (this.statusRequestInterval) {
-                clearInterval(this.statusRequestInterval);
-                this.statusRequestInterval = null;
-            }*/
-            
             if (this.elements.stopScan) {
                 this.elements.stopScan.disabled = true;
                 this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOPPING]</span><span>Stopping...</span>';
             }
             
+            // Show immediate feedback
             this.showInfo('Stopping scan...');
             
             const response = await this.apiCall('/api/scan/stop', 'POST');
@@ -575,10 +451,14 @@ class DNSReconApp {
             if (response.success) {
                 this.showSuccess('Scan stop requested');
                 
-                // Request final status update
+                // Force immediate status update
-                if (this.socket && this.isConnected) {
+                setTimeout(() => {
-                    setTimeout(() => this.socket.emit('get_status'), 500);
+                    this.updateStatus();
-                }
+                }, 100);
+                
+                // Continue status polling for a bit to catch the status change
+                // No need to resume graph polling
+                
             } else {
                 throw new Error(response.error || 'Failed to stop scan');
             }
@@ -587,6 +467,7 @@ class DNSReconApp {
             console.error('Failed to stop scan:', error);
             this.showError(`Failed to stop scan: ${error.message}`);
             
+            // Re-enable stop button on error
             if (this.elements.stopScan) {
                 this.elements.stopScan.disabled = false;
                 this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
@@ -652,7 +533,7 @@ class DNSReconApp {
             
             // Get the filename from headers or create one
             const contentDisposition = response.headers.get('content-disposition');
-            let filename = 'dnsrecon_export.json';
+            let filename = 'DNScope_export.json';
             if (contentDisposition) {
                 const filenameMatch = contentDisposition.match(/filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/);
                 if (filenameMatch) {
@@ -743,38 +624,123 @@ class DNSReconApp {
     }
     
     /**
-     * FIXED: Update graph from server with enhanced debugging
+     * Start polling for scan updates with configurable interval
+     */
+    startPolling(interval = 2000) {
+        console.log('=== STARTING POLLING ===');
+        
+        if (this.pollInterval) {
+            console.log('Clearing existing poll interval');
+            clearInterval(this.pollInterval);
+        }
+        
+        this.pollInterval = setInterval(() => {
+            this.updateStatus();
+            this.updateGraph();
+            this.loadProviders();
+        }, interval);
+        
+        console.log(`Polling started with ${interval}ms interval`);
+    }
+    
+    /**
+     * Stop polling for updates
+     */
+    stopPolling() {
+        console.log('=== STOPPING POLLING ===');
+        
+        if (this.statusPollInterval) {
+            clearInterval(this.statusPollInterval);
+            this.statusPollInterval = null;
+        }
+        
+        if (this.graphPollInterval) {
+            clearInterval(this.graphPollInterval);
+            this.graphPollInterval = null;
+        }
+        
+        this.updateManualRefreshButton();
+    }
+
+    /**
+     * Status update with better error handling
+     */
+    async updateStatus() {
+        try {
+            const response = await this.apiCall('/api/scan/status');
+            
+            
+            if (response.success && response.status) {
+                const status = response.status;
+                
+                this.updateStatusDisplay(status);
+                
+                // Handle status changes
+                if (status.status !== this.scanStatus) {
+                    console.log(`*** STATUS CHANGED: ${this.scanStatus} -> ${status.status} ***`);
+                    this.handleStatusChange(status.status, status.task_queue_size);
+                }
+                
+                this.scanStatus = status.status;
+            } else {
+                console.error('Status update failed:', response);
+                // Don't show error for status updates to avoid spam
+            }
+            
+        } catch (error) {
+            console.error('Failed to update status:', error);
+            this.showConnectionError();
+        }
+    }
+    
+    /**
+     * Update graph from server with polling optimization
      */
     async updateGraph() {
         try {
-            console.log('Updating graph via API call...');
+            console.log('Updating graph...');
             const response = await this.apiCall('/api/graph');
             
             
             if (response.success) {
                 const graphData = response.graph;
                 
-                console.log('Graph data received from API:');
+                console.log('Graph data received:');
                 console.log('- Nodes:', graphData.nodes ? graphData.nodes.length : 0);
                 console.log('- Edges:', graphData.edges ? graphData.edges.length : 0);
                 
-                // FIXED: Always update graph, even if empty - let GraphManager handle placeholder
+                // Always update graph, even if empty - let GraphManager handle placeholder
                 if (this.graphManager) {
-                    console.log('🔧 Calling GraphManager.updateGraph from API response...');
                     this.graphManager.updateGraph(graphData);
                     this.lastGraphUpdate = Date.now();
                     
+                    // Check if we should disable graph polling after this update
+                    const nodeCount = graphData.nodes ? graphData.nodes.length : 0;
+                    const shouldEnablePolling = nodeCount <= this.graphPollingNodeThreshold;
+                    
+                    if (this.graphPollingEnabled && !shouldEnablePolling) {
+                        console.log(`Graph polling disabled: ${nodeCount} nodes exceeds threshold of ${this.graphPollingNodeThreshold}`);
+                        this.graphPollingEnabled = false;
+                        this.showWarning(`Graph auto-refresh disabled: ${nodeCount} nodes exceed threshold of ${this.graphPollingNodeThreshold}. Use manual refresh button.`);
+                        
+                        // Stop graph polling but keep status polling
+                        if (this.graphPollInterval) {
+                            clearInterval(this.graphPollInterval);
+                            this.graphPollInterval = null;
+                        }
+                        
+                        this.updateManualRefreshButton();
+                    }
+                    
                     // Update relationship count in status
                     const edgeCount = graphData.edges ? graphData.edges.length : 0;
                     if (this.elements.relationshipsDisplay) {
                         this.elements.relationshipsDisplay.textContent = edgeCount;
                     }
-                    
-                    console.log('✅ Manual graph update completed');
                 }
             } else {
                 console.error('Graph update failed:', response);
-                // FIXED: Show placeholder when graph update fails
+                // Show placeholder when graph update fails
                 if (this.graphManager && this.graphManager.container) {
                     const placeholder = this.graphManager.container.querySelector('.graph-placeholder');
                     if (placeholder) {
@@ -785,7 +751,7 @@ class DNSReconApp {
             
         } catch (error) {
             console.error('Failed to update graph:', error);
-            // FIXED: Show placeholder on error
+            // Show placeholder on error
             if (this.graphManager && this.graphManager.container) {
                 const placeholder = this.graphManager.container.querySelector('.graph-placeholder');
                 if (placeholder) {
@@ -795,7 +761,6 @@ class DNSReconApp {
         }
     }
     
-    
     /**
      * Update status display elements
      * @param {Object} status - Status object from server
@@ -866,70 +831,47 @@ class DNSReconApp {
      * @param {string} newStatus - New scan status
      */
     handleStatusChange(newStatus, task_queue_size) {
-        console.log(`📄 Status change handler: ${this.scanStatus} → ${newStatus}`);
+        console.log(`=== STATUS CHANGE: ${this.scanStatus} -> ${newStatus} ===`);
         
         switch (newStatus) {
             case 'running':
                 this.setUIState('scanning', task_queue_size);
-                this.showSuccess('Scan is running - updates in real-time');
+                this.showSuccess('Scan is running');
                 this.updateConnectionStatus('active');
                 break;
                 
             case 'completed':
                 this.setUIState('completed', task_queue_size);
+                this.stopPolling();
                 this.showSuccess('Scan completed successfully');
                 this.updateConnectionStatus('completed');
                 this.loadProviders();
-                console.log('✅ Scan completed - requesting final graph update');
-                // Request final status to ensure we have the complete graph
-                setTimeout(() => {
-                    if (this.socket && this.isConnected) {
-                        this.socket.emit('get_status');
-                    }
-                }, 1000);
                 
-                // Clear status request interval
+                // Do final graph update when scan completes
-                /*if (this.statusRequestInterval) {
+                console.log('Scan completed - performing final graph update');
-                    clearInterval(this.statusRequestInterval);
+                setTimeout(() => this.updateGraph(), 1000);
-                    this.statusRequestInterval = null;
-                }*/
                 break;
                 
             case 'failed':
                 this.setUIState('failed', task_queue_size);
+                this.stopPolling();
                 this.showError('Scan failed');
                 this.updateConnectionStatus('error');
                 this.loadProviders();
-                
-                // Clear status request interval
-                /*if (this.statusRequestInterval) {
-                    clearInterval(this.statusRequestInterval);
-                    this.statusRequestInterval = null;
-                }*/
                 break;
                 
             case 'stopped':
                 this.setUIState('stopped', task_queue_size);
+                this.stopPolling();
                 this.showSuccess('Scan stopped');
                 this.updateConnectionStatus('stopped');
                 this.loadProviders();
-                
-                // Clear status request interval
-                if (this.statusRequestInterval) {
-                    clearInterval(this.statusRequestInterval);
-                    this.statusRequestInterval = null;
-                }
                 break;
                 
             case 'idle':
                 this.setUIState('idle', task_queue_size);
+                this.stopPolling();
                 this.updateConnectionStatus('idle');
-                
-                // Clear status request interval
-                /*if (this.statusRequestInterval) {
-                    clearInterval(this.statusRequestInterval);
-                    this.statusRequestInterval = null;
-                }*/
                 break;
                 
             default:
@@ -977,11 +919,11 @@ class DNSReconApp {
 
         switch (state) {
             case 'scanning':
+            case 'running':
                 this.isScanning = true;
                 if (this.graphManager) {
                     this.graphManager.isScanning = true;
                 }
-                
                 if (this.elements.startScan) {
                     this.elements.startScan.disabled = true;
                     this.elements.startScan.classList.add('loading');
@@ -1009,14 +951,13 @@ class DNSReconApp {
                 if (this.graphManager) {
                     this.graphManager.isScanning = false;
                 }
-                
                 if (this.elements.startScan) {
-                    this.elements.startScan.disabled = !isQueueEmpty;
+                    this.elements.startScan.disabled = false;
                     this.elements.startScan.classList.remove('loading');
                     this.elements.startScan.innerHTML = '<span class="btn-icon">[RUN]</span><span>Start Reconnaissance</span>';
                 }
                 if (this.elements.addToGraph) {
-                    this.elements.addToGraph.disabled = !isQueueEmpty;
+                    this.elements.addToGraph.disabled = false;
                     this.elements.addToGraph.classList.remove('loading');
                 }
                 if (this.elements.stopScan) {
@@ -1026,6 +967,9 @@ class DNSReconApp {
                 if (this.elements.targetInput) this.elements.targetInput.disabled = false;
                 if (this.elements.maxDepth) this.elements.maxDepth.disabled = false;
                 if (this.elements.configureSettings) this.elements.configureSettings.disabled = false;
+                
+                // Update manual refresh button visibility
+                this.updateManualRefreshButton();
                 break;
         }
     }
@@ -1252,7 +1196,7 @@ class DNSReconApp {
                 } else {
                     // API key not configured - ALWAYS show input field
                     const statusClass = info.enabled ? 'enabled' : 'api-key-required';
-                    const statusText = info.enabled ? '◯ Ready for API Key' : '⚠️ API Key Required';
+                    const statusText = info.enabled ? '○ Ready for API Key' : '⚠️ API Key Required';
                     
                     inputGroup.innerHTML = `
                         <div class="provider-header">
@@ -1772,17 +1716,9 @@ class DNSReconApp {
         return groups;
     }
 
-    formatEdgeLabel(relationshipType, confidence) {
-        if (!relationshipType) return '';
-
-        const confidenceText = confidence >= 0.8 ? '●' : confidence >= 0.6 ? '◐' : '○';
-        return `${relationshipType} ${confidenceText}`;
-    }
-
     createEdgeTooltip(edge) {
         let tooltip = `<div style="font-family: 'Roboto Mono', monospace; font-size: 11px;">`;
         tooltip += `<div style="color: #00ff41; font-weight: bold; margin-bottom: 4px;">${edge.label || 'Relationship'}</div>`;
-        tooltip += `<div style="color: #999; margin-bottom: 2px;">Confidence: ${(edge.confidence_score * 100).toFixed(1)}%</div>`;
         
         // UPDATED: Use raw provider name (no formatting)
         if (edge.source_provider) {
@@ -1922,7 +1858,7 @@ class DNSReconApp {
                 html += `
                     <div class="relationship-compact-item">
                         <span class="node-link-compact" data-node-id="${innerNodeId}">${innerNodeId}</span>
-                        <button class="btn-icon-small extract-node-btn" 
+                        <button class="graph-control-btn extract-node-btn" 
                                 title="Extract to graph"
                                 data-large-entity-id="${largeEntityId}" 
                                 data-node-id="${innerNodeId}">[+]</button>
@@ -1949,8 +1885,6 @@ class DNSReconApp {
             `;
             
             node.incoming_edges.forEach(edge => {
-                const confidence = edge.data.confidence_score || 0;
-                const confidenceClass = confidence >= 0.8 ? 'high' : confidence >= 0.6 ? 'medium' : 'low';
                 
                 html += `
                     <div class="relationship-item">
@@ -1959,9 +1893,6 @@ class DNSReconApp {
                         </div>
                         <div class="relationship-type">
                             <span class="relation-label">${edge.data.relationship_type}</span>
-                            <span class="confidence-indicator confidence-${confidenceClass}" title="Confidence: ${(confidence * 100).toFixed(1)}%">
-                                ${'●'.repeat(Math.ceil(confidence * 3))}
-                            </span>
                         </div>
                     </div>
                 `;
@@ -1980,9 +1911,6 @@ class DNSReconApp {
             `;
             
             node.outgoing_edges.forEach(edge => {
-                const confidence = edge.data.confidence_score || 0;
-                const confidenceClass = confidence >= 0.8 ? 'high' : confidence >= 0.6 ? 'medium' : 'low';
-                
                 html += `
                     <div class="relationship-item">
                         <div class="relationship-target node-link" data-node-id="${edge.to}">
@@ -1990,9 +1918,6 @@ class DNSReconApp {
                         </div>
                         <div class="relationship-type">
                             <span class="relation-label">${edge.data.relationship_type}</span>
-                            <span class="confidence-indicator confidence-${confidenceClass}" title="Confidence: ${(confidence * 100).toFixed(1)}%">
-                                ${'●'.repeat(Math.ceil(confidence * 3))}
-                            </span>
                         </div>
                     </div>
                 `;
@@ -2182,6 +2107,16 @@ class DNSReconApp {
 
     async extractNode(largeEntityId, nodeId) {
         try {
+            console.log(`Extracting node ${nodeId} from large entity ${largeEntityId}`);
+            
+            // Show immediate feedback
+            const button = document.querySelector(`[data-node-id="${nodeId}"][data-large-entity-id="${largeEntityId}"]`);
+            if (button) {
+                const originalContent = button.innerHTML;
+                button.innerHTML = '[...]';
+                button.disabled = true;
+            }
+
             const response = await this.apiCall('/api/graph/large-entity/extract', 'POST', {
                 large_entity_id: largeEntityId,
                 node_id: nodeId,
@@ -2190,13 +2125,32 @@ class DNSReconApp {
             if (response.success) {
                 this.showSuccess(response.message);
 
-                // If the scanner was idle, it's now running. Start polling to see the new node appear.
+                // FIXED: Don't update local modal data - let backend be source of truth
-                if (this.scanStatus === 'idle') {
+                // Force immediate graph update to get fresh backend data
-                    this.socket.emit('get_status');
+                console.log('Extraction successful, updating graph with fresh backend data');
-                } else {
+                await this.updateGraph();
-                    // If already scanning, force a quick graph update to see the change sooner.
+                
-                    setTimeout(() => this.socket.emit('get_status'), 500);
+                // FIXED: Re-fetch graph data instead of manipulating local state
-                }
+                setTimeout(async () => {
+                    try {
+                        const graphResponse = await this.apiCall('/api/graph');
+                        if (graphResponse.success) {
+                            this.graphManager.updateGraph(graphResponse.graph);
+                            
+                            // Update modal with fresh data if still open
+                            if (this.elements.nodeModal && this.elements.nodeModal.style.display === 'block') {
+                                if (this.graphManager.nodes) {
+                                    const updatedLargeEntity = this.graphManager.nodes.get(largeEntityId);
+                                    if (updatedLargeEntity) {
+                                        this.showNodeModal(updatedLargeEntity);
+                                    }
+                                }
+                            }
+                        }
+                    } catch (error) {
+                        console.error('Error refreshing graph after extraction:', error);
+                    }
+                }, 100);
 
             } else {
                 throw new Error(response.error || 'Extraction failed on the server.');
@@ -2204,6 +2158,13 @@ class DNSReconApp {
         } catch (error) {
             console.error('Failed to extract node:', error);
             this.showError(`Extraction failed: ${error.message}`);
+            
+            // Restore button state on error
+            const button = document.querySelector(`[data-node-id="${nodeId}"][data-large-entity-id="${largeEntityId}"]`);
+            if (button) {
+                button.innerHTML = '[+]';
+                button.disabled = false;
+            }
         }
     }
 
@@ -2234,8 +2195,8 @@ class DNSReconApp {
      */
     getNodeTypeIcon(nodeType) {
         const icons = {
-            'domain': '🌐',
+            'domain': '🌍',
-            'ip': '🔢',
+            'ip': '📍',
             'asn': '🏢',
             'large_entity': '📦',
             'correlation_object': '🔗'
@@ -2376,51 +2337,6 @@ class DNSReconApp {
         }
     }
         
-    /**
-     * Save API Keys
-     */
-    async saveApiKeys() {
-        const inputs = this.elements.apiKeyInputs.querySelectorAll('input');
-        const keys = {};
-        inputs.forEach(input => {
-            const provider = input.dataset.provider;
-            const value = input.value.trim();
-            if (provider && value) {
-                keys[provider] = value;
-            }
-        });
-
-        if (Object.keys(keys).length === 0) {
-            this.showWarning('No API keys were entered.');
-            return;
-        }
-
-        try {
-            const response = await this.apiCall('/api/config/api-keys', 'POST', keys);
-            if (response.success) {
-                this.showSuccess(response.message);
-                this.hideSettingsModal();
-                this.loadProviders(); // Refresh provider status
-            } else {
-                throw new Error(response.error || 'Failed to save API keys');
-            }
-        } catch (error) {
-            this.showError(`Error saving API keys: ${error.message}`);
-        }
-    }
-
-    /**
-     * Reset API Key fields
-     */
-    resetApiKeys() {
-        const inputs = this.elements.apiKeyInputs.querySelectorAll('input');
-        inputs.forEach(input => {
-            input.value = '';
-        });
-    }
-    
-
-        
     /**
      * Make API call to server
      * @param {string} endpoint - API endpoint
@@ -2803,5 +2719,5 @@ style.textContent = `
 document.head.appendChild(style);
 
 // Initialize application when page loads
-console.log('Creating DNSReconApp instance...');
+console.log('Creating DNScopeApp instance...');
-const app = new DNSReconApp();
+const app = new DNScopeApp();

templates/index.html

@@ -4,10 +4,9 @@
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>DNSRecon - Infrastructure Reconnaissance</title>
+    <title>DNScope - Infrastructure Reconnaissance</title>
     <link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
     <script src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.7.2/socket.io.js"></script>
     <link href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" rel="stylesheet" type="text/css">
     <link
         href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@300;400;500;700&family=Special+Elite&display=swap"
@@ -19,8 +18,8 @@
         <header class="header">
             <div class="header-content">
                 <div class="logo">
-                    <span class="logo-icon">[DNS]</span>
+                    <span class="logo-icon">[DN]</span>
-                    <span class="logo-text">RECON</span>
+                    <span class="logo-text">Scope</span>
                 </div>
                 <div class="status-indicator">
                     <span id="connection-status" class="status-dot"></span>

utils/__init__.py

@@ -1,7 +1,7 @@
-# dnsrecon-reduced/utils/__init__.py
+# DNScope-reduced/utils/__init__.py
 
 """
-Utility modules for DNSRecon.
+Utility modules for DNScope.
 Contains helper functions, export management, and supporting utilities.
 """
 

utils/export_manager.py

@@ -1,7 +1,7 @@
-# dnsrecon-reduced/utils/export_manager.py
+# DNScope-reduced/utils/export_manager.py
 
 """
-Centralized export functionality for DNSRecon.
+Centralized export functionality for DNScope.
 Handles all data export operations with forensic integrity and proper formatting.
 ENHANCED: Professional forensic executive summary generation for court-ready documentation.
 """
@@ -18,7 +18,7 @@ from utils.helpers import _is_valid_domain, _is_valid_ip
 
 class ExportManager:
     """
-    Centralized manager for all DNSRecon export operations.
+    Centralized manager for all DNScope export operations.
     Maintains forensic integrity and provides consistent export formats.
     ENHANCED: Advanced forensic analysis and professional reporting capabilities.
     """
@@ -188,7 +188,6 @@ class ExportManager:
                     f"  - Type: {domain_info['classification']}",
                     f"  - Connected IPs: {len(domain_info['ips'])}",
                     f"  - Certificate Status: {domain_info['cert_status']}",
-                    f"  - Relationship Confidence: {domain_info['avg_confidence']:.2f}",
                 ])
                 
                 if domain_info['security_notes']:
@@ -247,11 +246,9 @@ class ExportManager:
             ])
             
             for rel in key_relationships[:8]:  # Top 8 relationships
-                confidence_desc = self._describe_confidence(rel['confidence'])
                 report.extend([
                     f"• {rel['source']} → {rel['target']}",
                     f"  - Relationship: {self._humanize_relationship_type(rel['type'])}",
-                    f"  - Evidence Strength: {confidence_desc} ({rel['confidence']:.2f})",
                     f"  - Discovery Method: {rel['provider']}",
                     ""
                 ])
@@ -291,21 +288,15 @@ class ExportManager:
             "Data Quality Assessment:",
             f"• Total API Requests: {audit_trail.get('session_metadata', {}).get('total_requests', 0)}",
             f"• Data Providers Used: {len(audit_trail.get('session_metadata', {}).get('providers_used', []))}",
-            f"• Relationship Confidence Distribution:",
         ])
         
-        # Confidence distribution
+        correlation_provider = next((p for p in scanner.providers if p.get_name() == 'correlation'), None)
-        confidence_dist = self._calculate_confidence_distribution(edges)
+        correlation_count = len(correlation_provider.correlation_index) if correlation_provider else 0
-        for level, count in confidence_dist.items():
-            percentage = (count / len(edges) * 100) if edges else 0
-            report.extend([
-                f"  - {level.title()} Confidence (≥{self._get_confidence_threshold(level)}): {count} ({percentage:.1f}%)",
-            ])
         
         report.extend([
             "",
             "Correlation Analysis:",
-            f"• Entity Correlations Identified: {len(scanner.graph.correlation_index)}",
+            f"• Entity Correlations Identified: {correlation_count}",
             f"• Cross-Reference Validation: {self._count_cross_validated_relationships(edges)} relationships verified by multiple sources",
             ""
         ])
@@ -324,7 +315,7 @@ class ExportManager:
             "a complete audit trail maintained for forensic integrity.",
             "",
             f"Investigation completed: {now}",
-            f"Report authenticated by: DNSRecon v{self._get_version()}",
+            f"Report authenticated by: DNScope v{self._get_version()}",
             "",
             "=" * 80,
             "END OF REPORT",
@@ -375,9 +366,7 @@ class ExportManager:
             if len(connected_ips) > 5:
                 security_notes.append("Multiple IP endpoints")
             
-            # Average confidence
             domain_edges = [e for e in edges if e['from'] == domain['id']]
-            avg_confidence = sum(e['confidence_score'] for e in domain_edges) / len(domain_edges) if domain_edges else 0
             
             domain_analysis.append({
                 'domain': domain['id'],
@@ -385,7 +374,6 @@ class ExportManager:
                 'ips': connected_ips,
                 'cert_status': cert_status,
                 'security_notes': security_notes,
-                'avg_confidence': avg_confidence
             })
         
         # Sort by number of connections (most connected first)
@@ -480,7 +468,7 @@ class ExportManager:
     
     def _identify_key_relationships(self, edges: List[Dict]) -> List[Dict[str, Any]]:
         """Identify the most significant relationships in the infrastructure."""
-        # Score relationships by confidence and type importance
+        # Score relationships by type importance
         relationship_importance = {
             'dns_a_record': 0.9,
             'dns_aaaa_record': 0.9,
@@ -491,23 +479,19 @@ class ExportManager:
             'dns_ns_record': 0.7
         }
         
-        scored_edges = []
+        edges = []
         for edge in edges:
-            base_confidence = edge.get('confidence_score', 0)
             type_weight = relationship_importance.get(edge.get('label', ''), 0.5)
-            combined_score = (base_confidence * 0.7) + (type_weight * 0.3)
             
-            scored_edges.append({
+            edges.append({
                 'source': edge['from'],
                 'target': edge['to'],
                 'type': edge.get('label', ''),
-                'confidence': base_confidence,
                 'provider': edge.get('source_provider', ''),
-                'score': combined_score
             })
         
         # Return top relationships by score
-        return sorted(scored_edges, key=lambda x: x['score'], reverse=True)
+        return sorted(edges, key=lambda x: x['score'], reverse=True)
     
     def _analyze_certificate_infrastructure(self, nodes: List[Dict]) -> Dict[str, Any]:
         """Analyze certificate infrastructure across all domains."""
@@ -570,19 +554,6 @@ class ExportManager:
         else:
             return "Mixed Status"
     
-    def _describe_confidence(self, confidence: float) -> str:
-        """Convert confidence score to descriptive text."""
-        if confidence >= 0.9:
-            return "Very High"
-        elif confidence >= 0.8:
-            return "High"
-        elif confidence >= 0.6:
-            return "Medium"
-        elif confidence >= 0.4:
-            return "Low"
-        else:
-            return "Very Low"
-    
     def _humanize_relationship_type(self, rel_type: str) -> str:
         """Convert technical relationship types to human-readable descriptions."""
         type_map = {
@@ -599,26 +570,6 @@ class ExportManager:
         }
         return type_map.get(rel_type, rel_type.replace('_', ' ').title())
     
-    def _calculate_confidence_distribution(self, edges: List[Dict]) -> Dict[str, int]:
-        """Calculate confidence score distribution."""
-        distribution = {'high': 0, 'medium': 0, 'low': 0}
-        
-        for edge in edges:
-            confidence = edge.get('confidence_score', 0)
-            if confidence >= 0.8:
-                distribution['high'] += 1
-            elif confidence >= 0.6:
-                distribution['medium'] += 1
-            else:
-                distribution['low'] += 1
-        
-        return distribution
-    
-    def _get_confidence_threshold(self, level: str) -> str:
-        """Get confidence threshold for a level."""
-        thresholds = {'high': '0.80', 'medium': '0.60', 'low': '0.00'}
-        return thresholds.get(level, '0.00')
-    
     def _count_cross_validated_relationships(self, edges: List[Dict]) -> int:
         """Count relationships verified by multiple providers."""
         # Group edges by source-target pair
@@ -694,7 +645,7 @@ class ExportManager:
                 if centrality >= threshold]
     
     def _get_version(self) -> str:
-        """Get DNSRecon version for report authentication."""
+        """Get DNScope version for report authentication."""
         return "1.0.0-forensic"
     
     def export_graph_json(self, graph_manager) -> Dict[str, Any]:
@@ -717,7 +668,7 @@ class ExportManager:
                 'last_modified': graph_manager.last_modified,
                 'total_nodes': graph_manager.get_node_count(),
                 'total_edges': graph_manager.get_edge_count(),
-                'graph_format': 'dnsrecon_v1_unified_model'
+                'graph_format': 'DNScope_v1_unified_model'
             },
             'graph': graph_data,
             'statistics': graph_manager.get_statistics()
@@ -818,7 +769,7 @@ class ExportManager:
         }
         
         extension = extension_map.get(export_type, 'txt')
-        return f"dnsrecon_{export_type}_{safe_target}_{timestamp_str}.{extension}"
+        return f"DNScope_{export_type}_{safe_target}_{timestamp_str}.{extension}"
 
 
 class CustomJSONEncoder(json.JSONEncoder):

utils/helpers.py

@@ -1,4 +1,4 @@
-# dnsrecon-reduced/utils/helpers.py
+# DNScope-reduced/utils/helpers.py
 
 import ipaddress
 from typing import Union