it

.gitignore

@@ -168,3 +168,5 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
 
+dump.rdb
+.vscode

README.md

@@ -2,272 +2,255 @@
 
 DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys.
 
-**Current Status: Phase 1 Implementation**
-- ✅ Core infrastructure and graph engine
-- ✅ Certificate transparency data provider (crt.sh)
-- ✅ Basic web interface with real-time visualization
-- ✅ Forensic logging system
-- ✅ JSON export functionality
+**Current Status: Phase 2 Implementation**
+
+  - ✅ Core infrastructure and graph engine
+  - ✅ Multi-provider support (crt.sh, DNS, Shodan)
+  - ✅ Session-based multi-user support
+  - ✅ Real-time web interface with interactive visualization
+  - ✅ Forensic logging system and JSON export
 
 ## Features
 
-### Core Capabilities
-- **Zero Contact Reconnaissance**: Passive data gathering without touching target infrastructure
-- **In-Memory Graph Analysis**: Uses NetworkX for efficient relationship mapping
-- **Real-Time Visualization**: Interactive graph updates during scanning
-- **Forensic Logging**: Complete audit trail of all reconnaissance activities
-- **Confidence Scoring**: Weighted relationships based on data source reliability
-
-### Data Sources (Phase 1)
-- **Certificate Transparency (crt.sh)**: Discovers domain relationships through SSL certificate SAN analysis
-- **Basic DNS Resolution**: A/AAAA record lookups for IP relationships
-
-### Visualization
-- **Interactive Network Graph**: Powered by vis.js with cybersecurity theme
-- **Node Types**: Domains, IP addresses, certificates, ASNs
-- **Confidence-Based Styling**: Visual indicators for relationship strength
-- **Real-Time Updates**: Graph builds dynamically as relationships are discovered
+  - **Passive Reconnaissance**: Gathers data without direct contact with target infrastructure.
+  - **In-Memory Graph Analysis**: Uses NetworkX for efficient relationship mapping.
+  - **Real-Time Visualization**: The graph updates dynamically as the scan progresses.
+  - **Forensic Logging**: A complete audit trail of all reconnaissance activities is maintained.
+  - **Confidence Scoring**: Relationships are weighted based on the reliability of the data source.
+  - **Session Management**: Supports concurrent user sessions with isolated scanner instances.
 
 ## Installation
 
 ### Prerequisites
-- Python 3.8 or higher
-- Modern web browser with JavaScript enabled
 
-### Setup
-1. **Clone or create the project directory**:
-   ```bash
-   mkdir dnsrecon
-   cd dnsrecon
-   ```
+  - Python 3.8 or higher
+  - A modern web browser with JavaScript enabled
+  - (Recommended) A Linux host for running the application and the optional DNS cache.
 
-2. **Install Python dependencies**:
-   ```bash
-   pip install -r requirements.txt
-   ```
-
-3. **Verify the directory structure**:
-   ```
-   dnsrecon/
-   ├── app.py
-   ├── config.py
-   ├── requirements.txt
-   ├── core/
-   │   ├── __init__.py
-   │   ├── graph_manager.py
-   │   ├── scanner.py
-   │   └── logger.py
-   ├── providers/
-   │   ├── __init__.py
-   │   ├── base_provider.py
-   │   └── crtsh_provider.py
-   ├── static/
-   │   ├── css/
-   │   │   └── main.css
-   │   └── js/
-   │       ├── graph.js
-   │       └── main.js
-   └── templates/
-       └── index.html
-   ```
-
-## Usage
-
-### Starting the Application
-1. **Run the Flask application**:
-   ```bash
-   python app.py
-   ```
-
-2. **Open your web browser** and navigate to:
-   ```
-   http://127.0.0.1:5000
-   ```
-
-### Basic Reconnaissance Workflow
-
-1. **Enter Target Domain**: Input the domain you want to investigate (e.g., `example.com`)
-
-2. **Select Recursion Depth**: 
-   - **Depth 1**: Direct relationships only
-   - **Depth 2**: Recommended for most investigations
-   - **Depth 3+**: Extended analysis for comprehensive mapping
-
-3. **Start Reconnaissance**: Click "Start Reconnaissance" to begin passive data gathering
-
-4. **Monitor Progress**: Watch the real-time graph build as relationships are discovered
-
-5. **Analyze Results**: Interact with the graph to explore relationships and click nodes for detailed information
-
-6. **Export Data**: Download complete results including graph data and forensic audit trail
-
-### Understanding the Visualization
-
-#### Node Types
-- 🟢 **Green Circles**: Domain names
-- 🟠 **Orange Squares**: IP addresses  
-- ⚪ **Gray Diamonds**: SSL certificates
-- 🔵 **Blue Triangles**: ASN (Autonomous System) information
-
-#### Edge Confidence
-- **Thick Green Lines**: High confidence (≥80%) - Certificate SAN relationships
-- **Medium Orange Lines**: Medium confidence (60-79%) - DNS record relationships  
-- **Thin Gray Lines**: Lower confidence (<60%) - Passive DNS or uncertain relationships
-
-### Example Investigation
-
-Let's investigate `github.com`:
-
-1. Enter `github.com` as the target domain
-2. Set recursion depth to 2
-3. Start the scan
-4. Observe relationships to other GitHub domains discovered through certificate analysis
-5. Export results for further analysis
-
-Expected discoveries might include:
-- `*.github.com` domains through certificate SANs
-- `github.io` and related domains
-- Associated IP addresses
-- Certificate authority relationships
-
-## Configuration
-
-### Environment Variables
-You can configure DNSRecon using environment variables:
+### 1\. Clone the Project
 
 ```bash
-# API keys for future providers (Phase 2)
-export VIRUSTOTAL_API_KEY="your_api_key_here"
-export SHODAN_API_KEY="your_api_key_here"
-
-# Application settings
-export DEFAULT_RECURSION_DEPTH=2
-export FLASK_DEBUG=False
+git clone https://github.com/your-repo/dnsrecon.git
+cd dnsrecon
 ```
 
-### Rate Limiting
-DNSRecon includes built-in rate limiting to be respectful to data sources:
-- **crt.sh**: 60 requests per minute
-- **DNS queries**: 100 requests per minute
+### 2\. Install Python Dependencies
 
-## Data Export Format
+It is highly recommended to use a virtual environment:
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+```
+
+### 3\. (Optional but Recommended) Set up a Local DNS Caching Resolver
+
+Running a local DNS caching resolver can significantly speed up DNS queries and reduce your network footprint. Here’s how to set up `unbound` on a Debian-based Linux distribution (like Ubuntu).
+
+**a. Install Unbound:**
+
+```bash
+sudo apt update
+sudo apt install unbound -y
+```
+
+**b. Configure Unbound:**
+Create a new configuration file for DNSRecon:
+
+```bash
+sudo nano /etc/unbound/unbound.conf.d/dnsrecon.conf
+```
+
+Add the following content to the file:
+
+```
+server:
+    # Listen on localhost for all users
+    interface: 127.0.0.1
+    access-control: 0.0.0.0/0 refuse
+    access-control: 127.0.0.0/8 allow
+
+    # Enable prefetching of popular items
+    prefetch: yes
+```
+
+**c. Restart Unbound and set it as the default resolver:**
+
+```bash
+sudo systemctl restart unbound
+sudo systemctl enable unbound
+```
+
+To use this resolver for your system, you may need to update your network settings to point to `127.0.0.1` as your DNS server.
+
+**d. Update DNSProvider to use the local resolver:**
+In `dnsrecon/providers/dns_provider.py`, you can explicitly set the resolver's nameservers in the `__init__` method:
+
+```python
+# dnsrecon/providers/dns_provider.py
+
+class DNSProvider(BaseProvider):
+    def __init__(self, session_config=None):
+        """Initialize DNS provider with session-specific configuration."""
+        super().__init__(...)
+
+        # Configure DNS resolver
+        self.resolver = dns.resolver.Resolver()
+        self.resolver.nameservers = ['127.0.0.1'] # Use local caching resolver
+        self.resolver.timeout = 5
+        self.resolver.lifetime = 10
+```
+
+## Usage (Development)
+
+### 1\. Start the Application
 
-Results are exported as JSON with the following structure:
-
-```json
-{
-  "scan_metadata": {
-    "target_domain": "example.com",
-    "max_depth": 2,
-    "final_status": "completed"
-  },
-  "graph_data": {
-    "nodes": [...],
-    "edges": [...]
-  },
-  "forensic_audit": {
-    "session_metadata": {...},
-    "api_requests": [...],
-    "relationships": [...]
-  },
-  "provider_statistics": {...}
-}
-```
-
-## Forensic Integrity
-
-DNSRecon maintains complete forensic integrity:
-
-- **API Request Logging**: Every external request is logged with timestamps, URLs, and responses
-- **Relationship Provenance**: Each discovered relationship includes source provider and discovery method
-- **Session Tracking**: Unique session IDs for investigation continuity
-- **Confidence Metadata**: Scoring rationale for all relationships
-- **Export Integrity**: Complete audit trail included in all exports
-
-## Architecture Overview
-
-### Core Components
-
-- **GraphManager**: NetworkX-based in-memory graph with confidence scoring
-- **Scanner**: Multi-provider orchestration with depth-limited BFS exploration  
-- **ForensicLogger**: Thread-safe audit trail with structured logging
-- **BaseProvider**: Abstract interface for data source plugins
-
-### Data Flow
-1. User initiates scan via web interface
-2. Scanner coordinates multiple data providers
-3. Relationships discovered and added to in-memory graph
-4. Real-time updates sent to web interface
-5. Graph visualization updates dynamically
-6. Complete audit trail maintained throughout
-
-## Troubleshooting
-
-### Common Issues
-
-**Graph not displaying**:
-- Ensure JavaScript is enabled in your browser
-- Check browser console for errors
-- Verify vis.js library is loading correctly
-
-**Scan fails to start**:
-- Check target domain is valid
-- Ensure crt.sh is accessible from your network
-- Review Flask console output for errors
-
-**No relationships discovered**:
-- Some domains may have limited certificate transparency data
-- Try a well-known domain like `google.com` to verify functionality
-- Check provider status in the interface
-
-### Debug Mode
-Enable debug mode for verbose logging:
 ```bash
-export FLASK_DEBUG=True
 python app.py
 ```
 
-## Development Roadmap
+### 2\. Open Your Browser
 
-### Phase 2 (Planned)
-- Multi-provider system with Shodan and VirusTotal integration
-- Real-time scanning with enhanced visualization
-- Provider health monitoring and failure recovery
+Navigate to `http://127.0.0.1:5000`.
 
-### Phase 3 (Planned)  
-- Advanced correlation algorithms
-- Enhanced forensic reporting
-- Performance optimization for large investigations
+### 3\. Basic Reconnaissance Workflow
+
+1.  **Enter Target Domain**: Input a domain like `example.com`.
+2.  **Select Recursion Depth**: Depth 2 is recommended for most investigations.
+3.  **Start Reconnaissance**: Click "Start Reconnaissance" to begin.
+4.  **Monitor Progress**: Watch the real-time graph build as relationships are discovered.
+5.  **Analyze and Export**: Interact with the graph and download the results when the scan is complete.
+
+## Production Deployment
+
+To deploy DNSRecon in a production environment, follow these steps:
+
+### 1\. Use a Production WSGI Server
+
+Do not use the built-in Flask development server for production. Use a WSGI server like **Gunicorn**:
+
+```bash
+pip install gunicorn
+gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+```
+
+### 2\. Configure Environment Variables
+
+Set the following environment variables for a secure and configurable deployment:
+
+```bash
+# Generate a strong, random secret key
+export SECRET_KEY='your-super-secret-and-random-key'
+
+# Set Flask to production mode
+export FLASK_ENV='production'
+export FLASK_DEBUG=False
+
+# API keys (optional, but recommended for full functionality)
+export SHODAN_API_KEY="your_shodan_key"
+```
+
+### 3\. Use a Reverse Proxy
+
+Set up a reverse proxy like **Nginx** to sit in front of the Gunicorn server. This provides several benefits, including:
+
+  - **TLS/SSL Termination**: Securely handle HTTPS traffic.
+  - **Load Balancing**: Distribute traffic across multiple application instances.
+  - **Serving Static Files**: Efficiently serve CSS and JavaScript files.
+
+**Example Nginx Configuration:**
+
+```nginx
+server {
+    listen 80;
+    server_name your_domain.com;
+
+    location / {
+        return 301 https://$host$request_uri;
+    }
+}
+
+server {
+    listen 443 ssl;
+    server_name your_domain.com;
+
+    # SSL cert configuration
+    ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;
+
+    location / {
+        proxy_pass http://127.0.0.1:5000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    }
+
+    location /static {
+        alias /path/to/your/dnsrecon/static;
+        expires 30d;
+    }
+}
+```
+
+## Autostart with systemd
+
+To run DNSRecon as a service that starts automatically on boot, you can use `systemd`.
+
+### 1\. Create a `.service` file
+
+Create a new service file in `/etc/systemd/system/`:
+
+```bash
+sudo nano /etc/systemd/system/dnsrecon.service
+```
+
+### 2\. Add the Service Configuration
+
+Paste the following configuration into the file. **Remember to replace `/path/to/your/dnsrecon` and `your_user` with your actual project path and username.**
+
+```ini
+[Unit]
+Description=DNSRecon Application
+After=network.target
+
+[Service]
+User=your_user
+Group=your_user
+WorkingDirectory=/path/to/your/dnsrecon
+ExecStart=/path/to/your/dnsrecon/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
+Restart=always
+Environment="SECRET_KEY=your-super-secret-and-random-key"
+Environment="FLASK_ENV=production"
+Environment="FLASK_DEBUG=False"
+Environment="SHODAN_API_KEY=your_shodan_key"
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### 3\. Enable and Start the Service
+
+Reload the `systemd` daemon, enable the service to start on boot, and then start it immediately:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable dnsrecon.service
+sudo systemctl start dnsrecon.service
+```
+
+You can check the status of the service at any time with:
+
+```bash
+sudo systemctl status dnsrecon.service
+```
 
 ## Security Considerations
 
-- **No Persistent Storage**: All data stored in memory only
-- **API Keys**: Stored in memory only, never written to disk
-- **Rate Limiting**: Prevents abuse of external services
-- **Local Use Only**: No authentication required (designed for local use)
-
-## Contributing
-
-DNSRecon follows a phased development approach. Currently in Phase 1 with core infrastructure completed. 
-
-### Code Quality Standards
-- Follow PEP 8 for Python code
-- Comprehensive docstrings for all functions
-- Type hints where appropriate
-- Forensic logging for all external interactions
+  - **API Keys**: API keys are stored in memory for the duration of a user session and are not written to disk.
+  - **Rate Limiting**: DNSRecon includes built-in rate limiting to be respectful to data sources.
+  - **Local Use**: The application is designed for local or trusted network use and does not have built-in authentication. **Do not expose it directly to the internet without proper security controls.**
 
 ## License
 
-This project is intended for legitimate security research and infrastructure analysis. Users are responsible for compliance with applicable laws and regulations.
-
-## Support
-
-For issues and questions:
-1. Check the troubleshooting section above
-2. Review the Flask console output for error details
-3. Ensure all dependencies are properly installed
-
----
-
-**DNSRecon v1.0 - Phase 1 Implementation**  
-*Passive Infrastructure Reconnaissance for Security Professionals*
+This project is licensed under the terms of the license agreement found in the `LICENSE` file.

app.py

@@ -1,7 +1,6 @@
 """
 Flask application entry point for DNSRecon web interface.
-Provides REST API endpoints and serves the web interface with user session support.
-Enhanced with better session debugging and isolation.
+Enhanced with user session management and task-based completion model.
 """
 
 import json
@@ -10,7 +9,7 @@ from flask import Flask, render_template, request, jsonify, send_file, session
 from datetime import datetime, timezone, timedelta
 import io
 
-from core.session_manager import session_manager
+from core.session_manager import session_manager, UserIdentifier
 from config import config
 
 
@@ -18,48 +17,72 @@ app = Flask(__name__)
 app.config['SECRET_KEY'] = 'dnsrecon-dev-key-change-in-production'
 app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=2)  # 2 hour session lifetime
 
+
 def get_user_scanner():
     """
-    Get or create scanner instance for current user session with enhanced debugging.
-    
-    Returns:
-        Tuple of (session_id, scanner_instance)
+    Enhanced user scanner retrieval with user identification and session consolidation.
+    Implements single session per user with seamless consolidation.
     """
-    # Get current Flask session info for debugging
-    current_flask_session_id = session.get('dnsrecon_session_id')
-    client_ip = request.remote_addr
-    user_agent = request.headers.get('User-Agent', '')[:100]  # Truncate for logging
+    print("=== ENHANCED GET_USER_SCANNER ===")
     
-    print("=== SESSION DEBUG ===")
-    print(f"Client IP: {client_ip}")
-    print(f"User Agent: {user_agent}")
-    print(f"Flask Session ID: {current_flask_session_id}")
-    print(f"Flask Session Keys: {list(session.keys())}")
-    
-    # Try to get existing session
-    if current_flask_session_id:
-        existing_scanner = session_manager.get_session(current_flask_session_id)
-        if existing_scanner:
-            print(f"Using existing session: {current_flask_session_id}")
-            print(f"Scanner status: {existing_scanner.status}")
-            return current_flask_session_id, existing_scanner
-        else:
-            print(f"Session {current_flask_session_id} not found in session manager")
-    
-    # Create new session
-    print("Creating new session...")
-    new_session_id = session_manager.create_session()
-    new_scanner = session_manager.get_session(new_session_id)
-    
-    # Store in Flask session
-    session['dnsrecon_session_id'] = new_session_id
-    session.permanent = True
-    
-    print(f"Created new session: {new_session_id}")
-    print(f"New scanner status: {new_scanner.status}")
-    print("=== END SESSION DEBUG ===")
-    
-    return new_session_id, new_scanner
+    try:
+        # Extract user identification from request
+        client_ip, user_agent = UserIdentifier.extract_request_info(request)
+        user_fingerprint = UserIdentifier.generate_user_fingerprint(client_ip, user_agent)
+        
+        print(f"User fingerprint: {user_fingerprint}")
+        print(f"Client IP: {client_ip}")
+        print(f"User Agent: {user_agent[:50]}...")
+        
+        # Get current Flask session info for debugging
+        current_flask_session_id = session.get('dnsrecon_session_id')
+        print(f"Flask session ID: {current_flask_session_id}")
+        
+        # Try to get existing session first
+        if current_flask_session_id:
+            existing_scanner = session_manager.get_session(current_flask_session_id)
+            if existing_scanner:
+                # Verify session belongs to current user
+                session_info = session_manager.get_session_info(current_flask_session_id)
+                if session_info.get('user_fingerprint') == user_fingerprint:
+                    print(f"Found valid existing session {current_flask_session_id} for user {user_fingerprint}")
+                    existing_scanner.session_id = current_flask_session_id
+                    return current_flask_session_id, existing_scanner
+                else:
+                    print(f"Session {current_flask_session_id} belongs to different user, will create new session")
+            else:
+                print(f"Session {current_flask_session_id} not found in Redis, will create new session")
+        
+        # Create or replace user session (this handles consolidation automatically)
+        new_session_id = session_manager.create_or_replace_user_session(client_ip, user_agent)
+        new_scanner = session_manager.get_session(new_session_id)
+        
+        if not new_scanner:
+            print(f"ERROR: Failed to retrieve newly created session {new_session_id}")
+            raise Exception("Failed to create new scanner session")
+        
+        # Store in Flask session for browser persistence
+        session['dnsrecon_session_id'] = new_session_id
+        session.permanent = True
+        
+        # Ensure session ID is set on scanner
+        new_scanner.session_id = new_session_id
+        
+        # Get session info for user feedback
+        session_info = session_manager.get_session_info(new_session_id)
+        
+        print(f"Session created/consolidated successfully")
+        print(f"  - Session ID: {new_session_id}")
+        print(f"  - User: {user_fingerprint}")
+        print(f"  - Scanner status: {new_scanner.status}")
+        print(f"  - Session age: {session_info.get('session_age_minutes', 0)} minutes")
+        
+        return new_session_id, new_scanner
+        
+    except Exception as e:
+        print(f"ERROR: Exception in get_user_scanner: {e}")
+        traceback.print_exc()
+        raise
 
 
 @app.route('/')
@@ -71,8 +94,7 @@ def index():
 @app.route('/api/scan/start', methods=['POST'])
 def start_scan():
     """
-    Start a new reconnaissance scan for the current user session.
-    Enhanced with better error handling and debugging.
+    Start a new reconnaissance scan with enhanced user session management.
     """
     print("=== API: /api/scan/start called ===")
     
@@ -90,8 +112,9 @@ def start_scan():
         
         target_domain = data['target_domain'].strip()
         max_depth = data.get('max_depth', config.default_recursion_depth)
+        clear_graph = data.get('clear_graph', True)
         
-        print(f"Parsed - target_domain: '{target_domain}', max_depth: {max_depth}")
+        print(f"Parsed - target_domain: '{target_domain}', max_depth: {max_depth}, clear_graph: {clear_graph}")
         
         # Validation
         if not target_domain:
@@ -110,40 +133,41 @@ def start_scan():
         
         print("Validation passed, getting user scanner...")
         
-        # Get user-specific scanner with enhanced debugging
+        # Get user-specific scanner with enhanced session management
         user_session_id, scanner = get_user_scanner()
+        
+        # Ensure session ID is properly set
+        if not scanner.session_id:
+            scanner.session_id = user_session_id
+        
         print(f"Using session: {user_session_id}")
         print(f"Scanner object ID: {id(scanner)}")
-        print(f"Scanner status before start: {scanner.status}")
-        
-        # Additional safety check - if scanner is somehow in running state, force reset
-        if scanner.status == 'running':
-            print(f"WARNING: Scanner in session {user_session_id} was already running - forcing reset")
-            scanner.stop_scan()
-            # Give it a moment to stop
-            import time
-            time.sleep(1)
-            
-            # If still running, force status reset
-            if scanner.status == 'running':
-                print("WARNING: Force resetting scanner status from 'running' to 'idle'")
-                scanner.status = 'idle'
         
         # Start scan
         print(f"Calling start_scan on scanner {id(scanner)}...")
-        success = scanner.start_scan(target_domain, max_depth)
+        success = scanner.start_scan(target_domain, max_depth, clear_graph=clear_graph)
         
-        print(f"scanner.start_scan returned: {success}")
-        print(f"Scanner status after start attempt: {scanner.status}")
+        # Immediately update session state regardless of success
+        session_manager.update_session_scanner(user_session_id, scanner)
         
         if success:
             scan_session_id = scanner.logger.session_id
             print(f"Scan started successfully with scan session ID: {scan_session_id}")
+            
+            # Get session info for user feedback
+            session_info = session_manager.get_session_info(user_session_id)
+            
             return jsonify({
                 'success': True,
                 'message': 'Scan started successfully',
                 'scan_id': scan_session_id,
                 'user_session_id': user_session_id,
+                'scanner_status': scanner.status,
+                'session_info': {
+                    'user_fingerprint': session_info.get('user_fingerprint', 'unknown'),
+                    'session_age_minutes': session_info.get('session_age_minutes', 0),
+                    'consolidated': session_info.get('session_age_minutes', 0) > 0
+                },
                 'debug_info': {
                     'scanner_object_id': id(scanner),
                     'scanner_status': scanner.status
@@ -174,9 +198,10 @@ def start_scan():
             'error': f'Internal server error: {str(e)}'
         }), 500
 
+
 @app.route('/api/scan/stop', methods=['POST'])
 def stop_scan():
-    """Stop the current scan for the user session."""
+    """Stop the current scan with immediate GUI feedback."""
     print("=== API: /api/scan/stop called ===")
     
     try:
@@ -184,19 +209,37 @@ def stop_scan():
         user_session_id, scanner = get_user_scanner()
         print(f"Stopping scan for session: {user_session_id}")
         
-        success = scanner.stop_scan()
-        
-        if success:
-            return jsonify({
-                'success': True,
-                'message': 'Scan stop requested',
-                'user_session_id': user_session_id
-            })
-        else:
+        if not scanner:
             return jsonify({
                 'success': False,
-                'error': 'No active scan to stop for this session'
-            }), 400
+                'error': 'No scanner found for session'
+            }), 404
+        
+        # Ensure session ID is set
+        if not scanner.session_id:
+            scanner.session_id = user_session_id
+        
+        # Use the stop mechanism
+        success = scanner.stop_scan()
+        
+        # Also set the Redis stop signal directly for extra reliability
+        session_manager.set_stop_signal(user_session_id)
+        
+        # Force immediate status update
+        session_manager.update_scanner_status(user_session_id, 'stopped')
+        
+        # Update the full scanner state
+        session_manager.update_session_scanner(user_session_id, scanner)
+        
+        print(f"Stop scan completed. Success: {success}, Scanner status: {scanner.status}")
+        
+        return jsonify({
+            'success': True,
+            'message': 'Scan stop requested - termination initiated',
+            'user_session_id': user_session_id,
+            'scanner_status': scanner.status,
+            'stop_method': 'cross_process'
+        })
         
     except Exception as e:
         print(f"ERROR: Exception in stop_scan endpoint: {e}")
@@ -209,14 +252,53 @@ def stop_scan():
 
 @app.route('/api/scan/status', methods=['GET'])
 def get_scan_status():
-    """Get current scan status and progress for the user session."""
+    """Get current scan status with enhanced session information."""
     try:
         # Get user-specific scanner
         user_session_id, scanner = get_user_scanner()
         
+        if not scanner:
+            # Return default idle status if no scanner
+            return jsonify({
+                'success': True,
+                'status': {
+                    'status': 'idle',
+                    'target_domain': None,
+                    'current_depth': 0,
+                    'max_depth': 0,
+                    'current_indicator': '',
+                    'total_indicators_found': 0,
+                    'indicators_processed': 0,
+                    'progress_percentage': 0.0,
+                    'enabled_providers': [],
+                    'graph_statistics': {},
+                    'user_session_id': user_session_id
+                }
+            })
+        
+        # Ensure session ID is set
+        if not scanner.session_id:
+            scanner.session_id = user_session_id
+        
         status = scanner.get_scan_status()
         status['user_session_id'] = user_session_id
         
+        # Add enhanced session information
+        session_info = session_manager.get_session_info(user_session_id)
+        status['session_info'] = {
+            'user_fingerprint': session_info.get('user_fingerprint', 'unknown'),
+            'session_age_minutes': session_info.get('session_age_minutes', 0),
+            'client_ip': session_info.get('client_ip', 'unknown'),
+            'last_activity': session_info.get('last_activity')
+        }
+        
+        # Additional debug info
+        status['debug_info'] = {
+            'scanner_object_id': id(scanner),
+            'session_id_set': bool(scanner.session_id),
+            'has_scan_thread': bool(scanner.scan_thread and scanner.scan_thread.is_alive())
+        }
+        
         return jsonify({
             'success': True,
             'status': status
@@ -227,17 +309,41 @@ def get_scan_status():
         traceback.print_exc()
         return jsonify({
             'success': False,
-            'error': f'Internal server error: {str(e)}'
+            'error': f'Internal server error: {str(e)}',
+            'fallback_status': {
+                'status': 'error',
+                'target_domain': None,
+                'current_depth': 0,
+                'max_depth': 0,
+                'progress_percentage': 0.0
+            }
         }), 500
 
 
 @app.route('/api/graph', methods=['GET'])
 def get_graph_data():
-    """Get current graph data for visualization for the user session."""
+    """Get current graph data with error handling."""
     try:
         # Get user-specific scanner
         user_session_id, scanner = get_user_scanner()
         
+        if not scanner:
+            # Return empty graph if no scanner
+            return jsonify({
+                'success': True,
+                'graph': {
+                    'nodes': [],
+                    'edges': [],
+                    'statistics': {
+                        'node_count': 0,
+                        'edge_count': 0,
+                        'creation_time': datetime.now(timezone.utc).isoformat(),
+                        'last_modified': datetime.now(timezone.utc).isoformat()
+                    }
+                },
+                'user_session_id': user_session_id
+            })
+        
         graph_data = scanner.get_graph_data()
         return jsonify({
             'success': True,
@@ -250,7 +356,12 @@ def get_graph_data():
         traceback.print_exc()
         return jsonify({
             'success': False,
-            'error': f'Internal server error: {str(e)}'
+            'error': f'Internal server error: {str(e)}',
+            'fallback_graph': {
+                'nodes': [],
+                'edges': [],
+                'statistics': {'node_count': 0, 'edge_count': 0}
+            }
         }), 500
 
 
@@ -264,17 +375,22 @@ def export_results():
         # Get complete results
         results = scanner.export_results()
         
-        # Add session information to export
+        # Add enhanced session information to export
+        session_info = session_manager.get_session_info(user_session_id)
         results['export_metadata'] = {
             'user_session_id': user_session_id,
+            'user_fingerprint': session_info.get('user_fingerprint', 'unknown'),
+            'client_ip': session_info.get('client_ip', 'unknown'),
+            'session_age_minutes': session_info.get('session_age_minutes', 0),
             'export_timestamp': datetime.now(timezone.utc).isoformat(),
             'export_type': 'user_session_results'
         }
         
-        # Create filename with timestamp
+        # Create filename with user fingerprint
         timestamp = datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')
         target = scanner.current_target or 'unknown'
-        filename = f"dnsrecon_{target}_{timestamp}_{user_session_id[:8]}.json"
+        user_fp = session_info.get('user_fingerprint', 'unknown')[:8]
+        filename = f"dnsrecon_{target}_{timestamp}_{user_fp}.json"
         
         # Create in-memory file
         json_data = json.dumps(results, indent=2, ensure_ascii=False)
@@ -305,19 +421,8 @@ def get_providers():
         # Get user-specific scanner
         user_session_id, scanner = get_user_scanner()
         
-        provider_stats = scanner.get_provider_statistics()
+        provider_info = scanner.get_provider_info()
         
-        # Add configuration information
-        provider_info = {}
-        for provider_name, stats in provider_stats.items():
-            provider_info[provider_name] = {
-                'statistics': stats,
-                'enabled': config.is_provider_enabled(provider_name),
-                'rate_limit': config.get_rate_limit(provider_name),
-                'requires_api_key': provider_name in ['shodan', 'virustotal']
-            }
-        
-        print(f"Returning provider info for session {user_session_id}: {list(provider_info.keys())}")
         return jsonify({
             'success': True,
             'providers': provider_info,
@@ -341,7 +446,7 @@ def set_api_keys():
     try:
         data = request.get_json()
         
-        if not data:
+        if data is None:
             return jsonify({
                 'success': False,
                 'error': 'No API keys provided'
@@ -353,16 +458,23 @@ def set_api_keys():
         
         updated_providers = []
         
-        for provider, api_key in data.items():
-            if provider in ['shodan', 'virustotal'] and api_key.strip():
-                success = session_config.set_api_key(provider, api_key.strip())
-                if success:
-                    updated_providers.append(provider)
+        # Iterate over the API keys provided in the request data
+        for provider_name, api_key in data.items():
+            # This allows us to both set and clear keys. The config
+            # handles enabling/disabling based on if the key is empty.
+            api_key_value = str(api_key or '').strip()
+            success = session_config.set_api_key(provider_name.lower(), api_key_value)
+            
+            if success:
+                updated_providers.append(provider_name)
         
         if updated_providers:
-            # Reinitialize scanner providers for this session only
+            # Reinitialize scanner providers to apply the new keys
             scanner._initialize_providers()
             
+            # Persist the updated scanner object back to the user's session
+            session_manager.update_session_scanner(user_session_id, scanner)
+            
             return jsonify({
                 'success': True,
                 'message': f'API keys updated for session {user_session_id}: {", ".join(updated_providers)}',
@@ -372,7 +484,7 @@ def set_api_keys():
         else:
             return jsonify({
                 'success': False,
-                'error': 'No valid API keys were provided'
+                'error': 'No valid API keys were provided or provider names were incorrect.'
             }), 400
     
     except Exception as e:
@@ -382,19 +494,11 @@ def set_api_keys():
             'success': False,
             'error': f'Internal server error: {str(e)}'
         }), 500
-    
-    except Exception as e:
-        print(f"ERROR: Exception in set_api_keys endpoint: {e}")
-        traceback.print_exc()
-        return jsonify({
-            'success': False,
-            'error': f'Internal server error: {str(e)}'
-        }), 500
 
 
 @app.route('/api/session/info', methods=['GET'])
 def get_session_info():
-    """Get information about the current user session."""
+    """Get enhanced information about the current user session."""
     try:
         user_session_id, scanner = get_user_scanner()
         session_info = session_manager.get_session_info(user_session_id)
@@ -445,7 +549,7 @@ def terminate_session():
 
 @app.route('/api/admin/sessions', methods=['GET'])
 def list_sessions():
-    """Admin endpoint to list all active sessions."""
+    """Admin endpoint to list all active sessions with enhanced information."""
     try:
         sessions = session_manager.list_active_sessions()
         stats = session_manager.get_statistics()
@@ -467,7 +571,7 @@ def list_sessions():
 
 @app.route('/api/health', methods=['GET'])
 def health_check():
-    """Health check endpoint with enhanced Phase 2 information."""
+    """Health check endpoint with enhanced session statistics."""
     try:
         # Get session stats
         session_stats = session_manager.get_statistics()
@@ -476,19 +580,28 @@ def health_check():
             'success': True,
             'status': 'healthy',
             'timestamp': datetime.now(timezone.utc).isoformat(),
-            'version': '1.0.0-phase2',
-            'phase': 2,
+            'version': '2.0.0-enhanced',
+            'phase': 'enhanced_architecture',
             'features': {
                 'multi_provider': True,
                 'concurrent_processing': True,
                 'real_time_updates': True,
                 'api_key_management': True,
-                'enhanced_visualization': True,
+                'visualization': True,
                 'retry_logic': True,
                 'user_sessions': True,
-                'session_isolation': True
+                'session_isolation': True,
+                'global_provider_caching': True,
+                'single_session_per_user': True,
+                'session_consolidation': True,
+                'task_completion_model': True
             },
-            'session_statistics': session_stats
+            'session_statistics': session_stats,
+            'cache_info': {
+                'global_provider_cache': True,
+                'cache_location': '.cache/<provider_name>/',
+                'cache_expiry_hours': 12
+            }
         })
     except Exception as e:
         print(f"ERROR: Exception in health_check endpoint: {e}")
@@ -519,7 +632,7 @@ def internal_error(error):
 
 
 if __name__ == '__main__':
-    print("Starting DNSRecon Flask application with user session support...")
+    print("Starting DNSRecon Flask application with enhanced user session support...")
     
     # Load configuration from environment
     config.load_from_env()

config.py

@@ -13,20 +13,18 @@ class Config:
     def __init__(self):
         """Initialize configuration with default values."""
         self.api_keys: Dict[str, Optional[str]] = {
-            'shodan': None,
-            'virustotal': None
+            'shodan': None
         }
         
         # Default settings
         self.default_recursion_depth = 2
-        self.default_timeout = 30
+        self.default_timeout = 10
         self.max_concurrent_requests = 5
         self.large_entity_threshold = 100
         
         # Rate limiting settings (requests per minute)
         self.rate_limits = {
             'crtsh': 60,          # Free service, be respectful
-            'virustotal': 4,      # Free tier limit
             'shodan': 60,         # API dependent
             'dns': 100            # Local DNS queries
         }
@@ -35,7 +33,6 @@ class Config:
         self.enabled_providers = {
             'crtsh': True,        # Always enabled (free)
             'dns': True,          # Always enabled (free)
-            'virustotal': False,  # Requires API key
             'shodan': False       # Requires API key
         }
         
@@ -53,7 +50,7 @@ class Config:
         Set API key for a provider.
         
         Args:
-            provider: Provider name (shodan, virustotal)
+            provider: Provider name (shodan, etc)
             api_key: API key string
             
         Returns:
@@ -103,9 +100,6 @@ class Config:
     
     def load_from_env(self):
         """Load configuration from environment variables."""
-        if os.getenv('VIRUSTOTAL_API_KEY'):
-            self.set_api_key('virustotal', os.getenv('VIRUSTOTAL_API_KEY'))
-        
         if os.getenv('SHODAN_API_KEY'):
             self.set_api_key('shodan', os.getenv('SHODAN_API_KEY'))
         

core/__init__.py

@@ -1,28 +1,29 @@
 """
 Core modules for DNSRecon passive reconnaissance tool.
 Contains graph management, scanning orchestration, and forensic logging.
-Phase 2: Enhanced with concurrent processing and real-time capabilities.
 """
 
-from .graph_manager import GraphManager, NodeType, RelationshipType
-from .scanner import Scanner, ScanStatus  # Remove 'scanner' global instance
+from .graph_manager import GraphManager, NodeType
+from .scanner import Scanner, ScanStatus  
 from .logger import ForensicLogger, get_forensic_logger, new_session
-from .session_manager import session_manager  # Add session manager
-from .session_config import SessionConfig, create_session_config  # Add session config
+from .session_manager import session_manager 
+from .session_config import SessionConfig, create_session_config 
+from .task_manager import TaskManager, TaskType, ReconTask
 
 __all__ = [
     'GraphManager',
     'NodeType', 
-    'RelationshipType',
     'Scanner',
     'ScanStatus',
-    # 'scanner',  # Remove this - no more global scanner
     'ForensicLogger',
     'get_forensic_logger',
     'new_session',
-    'session_manager',  # Add this
-    'SessionConfig',    # Add this
-    'create_session_config'  # Add this
+    'session_manager', 
+    'SessionConfig',
+    'create_session_config',
+    'TaskManager',
+    'TaskType',
+    'ReconTask'
 ]
 
 __version__ = "1.0.0-phase2"

core/graph_manager.py

@@ -2,11 +2,10 @@
 Graph data model for DNSRecon using NetworkX.
 Manages in-memory graph storage with confidence scoring and forensic metadata.
 """
-
-from datetime import datetime
-from typing import Dict, List, Any, Optional, Tuple
+import re
+from datetime import datetime, timezone
 from enum import Enum
-from datetime import timezone
+from typing import Dict, List, Any, Optional, Tuple
 
 import networkx as nx
 
@@ -16,38 +15,11 @@ class NodeType(Enum):
     DOMAIN = "domain"
     IP = "ip"
     ASN = "asn"
-    DNS_RECORD = "dns_record"
     LARGE_ENTITY = "large_entity"
+    CORRELATION_OBJECT = "correlation_object"
 
-
-class RelationshipType(Enum):
-    """Enumeration of supported relationship types with confidence scores."""
-    SAN_CERTIFICATE = ("san", 0.9)
-    A_RECORD = ("a_record", 0.8)
-    AAAA_RECORD = ("aaaa_record", 0.8)
-    CNAME_RECORD = ("cname", 0.8)
-    MX_RECORD = ("mx_record", 0.7)
-    NS_RECORD = ("ns_record", 0.7)
-    PTR_RECORD = ("ptr_record", 0.8)
-    SOA_RECORD = ("soa_record", 0.7)
-    TXT_RECORD = ("txt_record", 0.7)
-    SRV_RECORD = ("srv_record", 0.7)
-    CAA_RECORD = ("caa_record", 0.7)
-    DNSKEY_RECORD = ("dnskey_record", 0.7)
-    DS_RECORD = ("ds_record", 0.7)
-    RRSIG_RECORD = ("rrsig_record", 0.7)
-    SSHFP_RECORD = ("sshfp_record", 0.7)
-    TLSA_RECORD = ("tlsa_record", 0.7)
-    NAPTR_RECORD = ("naptr_record", 0.7)
-    SPF_RECORD = ("spf_record", 0.7)
-    DNS_RECORD = ("dns_record", 0.8)
-    PASSIVE_DNS = ("passive_dns", 0.6)
-    ASN_MEMBERSHIP = ("asn", 0.7)
-
-
-    def __init__(self, relationship_name: str, default_confidence: float):
-        self.relationship_name = relationship_name
-        self.default_confidence = default_confidence
+    def __repr__(self):
+        return self.value
 
 
 class GraphManager:
@@ -59,88 +31,317 @@ class GraphManager:
     def __init__(self):
         """Initialize empty directed graph."""
         self.graph = nx.DiGraph()
-        # self.lock = threading.Lock()
         self.creation_time = datetime.now(timezone.utc).isoformat()
         self.last_modified = self.creation_time
+        self.correlation_index = {}
+        # Compile regex for date filtering for efficiency
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
 
-    def add_node(self, node_id: str, node_type: NodeType,
-                 metadata: Optional[Dict[str, Any]] = None) -> bool:
-        """
-        Add a node to the graph.
+    def __getstate__(self):
+        """Prepare GraphManager for pickling, excluding compiled regex."""
+        state = self.__dict__.copy()
+        # Compiled regex patterns are not always picklable
+        if 'date_pattern' in state:
+            del state['date_pattern']
+        return state
 
-        Args:
-            node_id: Unique identifier for the node
-            node_type: Type of the node (Domain, IP, Certificate, ASN)
-            metadata: Additional metadata for the node
+    def __setstate__(self, state):
+        """Restore GraphManager state and recompile regex."""
+        self.__dict__.update(state)
+        self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
 
-        Returns:
-            bool: True if node was added, False if it already exists
-        """
-        if self.graph.has_node(node_id):
-            # Update metadata if node exists
-            existing_metadata = self.graph.nodes[node_id].get('metadata', {})
+    def _update_correlation_index(self, node_id: str, data: Any, path: List[str] = None):
+        """Recursively traverse metadata and add hashable values to the index."""
+        if path is None:
+            path = []
+
+        if isinstance(data, dict):
+            for key, value in data.items():
+                self._update_correlation_index(node_id, value, path + [key])
+        elif isinstance(data, list):
+            for i, item in enumerate(data):
+                self._update_correlation_index(node_id, item, path + [f"[{i}]"])
+        else:
+            self._add_to_correlation_index(node_id, data, ".".join(path))
+
+    def _add_to_correlation_index(self, node_id: str, value: Any, path_str: str):
+        """Add a hashable value to the correlation index, filtering out noise."""
+        if not isinstance(value, (str, int, float, bool)) or value is None:
+            return
+
+        # Ignore certain paths that contain noisy, non-unique identifiers
+        if any(keyword in path_str.lower() for keyword in ['count', 'total', 'timestamp', 'date']):
+            return
+
+        # Filter out common low-entropy values and date-like strings
+        if isinstance(value, str):
+            # FIXED: Prevent correlation on date/time strings.
+            if self.date_pattern.match(value):
+                return
+            if len(value) < 4 or value.lower() in ['true', 'false', 'unknown', 'none', 'crt.sh']:
+                return
+        elif isinstance(value, int) and abs(value) < 9999:
+            return  # Ignore small integers
+        elif isinstance(value, bool):
+            return  # Ignore boolean values
+
+        # Add the valuable correlation data to the index
+        if value not in self.correlation_index:
+            self.correlation_index[value] = {}
+        if node_id not in self.correlation_index[value]:
+            self.correlation_index[value][node_id] = []
+        if path_str not in self.correlation_index[value][node_id]:
+            self.correlation_index[value][node_id].append(path_str)
+
+    def _check_for_correlations(self, new_node_id: str, data: Any, path: List[str] = None) -> List[Dict]:
+        """Recursively traverse metadata to find correlations with existing data."""
+        if path is None:
+            path = []
+
+        all_correlations = []
+        if isinstance(data, dict):
+            for key, value in data.items():
+                if key == 'source':  # Avoid correlating on the provider name
+                    continue
+                all_correlations.extend(self._check_for_correlations(new_node_id, value, path + [key]))
+        elif isinstance(data, list):
+            for i, item in enumerate(data):
+                all_correlations.extend(self._check_for_correlations(new_node_id, item, path + [f"[{i}]"]))
+        else:
+            value = data
+            if value in self.correlation_index:
+                existing_nodes_with_paths = self.correlation_index[value]
+                unique_nodes = set(existing_nodes_with_paths.keys())
+                unique_nodes.add(new_node_id)
+
+                if len(unique_nodes) < 2:
+                    return all_correlations # Correlation must involve at least two distinct nodes
+
+                new_source = {'node_id': new_node_id, 'path': ".".join(path)}
+                all_sources = [new_source]
+                for node_id, paths in existing_nodes_with_paths.items():
+                    for p_str in paths:
+                        all_sources.append({'node_id': node_id, 'path': p_str})
+
+                all_correlations.append({
+                    'value': value,
+                    'sources': all_sources,
+                    'nodes': list(unique_nodes)
+                })
+        return all_correlations
+
+    def add_node(self, node_id: str, node_type: NodeType, attributes: Optional[Dict[str, Any]] = None,
+                description: str = "", metadata: Optional[Dict[str, Any]] = None) -> bool:
+        """Add a node to the graph, update attributes, and process correlations."""
+        is_new_node = not self.graph.has_node(node_id)
+        if is_new_node:
+            self.graph.add_node(node_id, type=node_type.value,
+                                added_timestamp=datetime.now(timezone.utc).isoformat(),
+                                attributes=attributes or {},
+                                description=description,
+                                metadata=metadata or {})
+        else:
+            # Safely merge new attributes into existing attributes
+            if attributes:
+                existing_attributes = self.graph.nodes[node_id].get('attributes', {})
+                existing_attributes.update(attributes)
+                self.graph.nodes[node_id]['attributes'] = existing_attributes
+            if description:
+                self.graph.nodes[node_id]['description'] = description
             if metadata:
+                existing_metadata = self.graph.nodes[node_id].get('metadata', {})
                 existing_metadata.update(metadata)
                 self.graph.nodes[node_id]['metadata'] = existing_metadata
+
+        if attributes and node_type != NodeType.CORRELATION_OBJECT:
+            correlations = self._check_for_correlations(node_id, attributes)
+            for corr in correlations:
+                value = corr['value']
+                
+                # STEP 1: Substring check against all existing nodes
+                if self._correlation_value_matches_existing_node(value):
+                    # Skip creating correlation node - would be redundant
+                    continue
+                
+                # STEP 2: Filter out node pairs that already have direct edges
+                eligible_nodes = self._filter_nodes_without_direct_edges(set(corr['nodes']))
+                
+                if len(eligible_nodes) < 2:
+                    # Need at least 2 nodes to create a correlation
+                    continue
+                    
+                # STEP 3: Check for existing correlation node with same connection pattern
+                correlation_nodes_with_pattern = self._find_correlation_nodes_with_same_pattern(eligible_nodes)
+                
+                if correlation_nodes_with_pattern:
+                    # STEP 4: Merge with existing correlation node
+                    target_correlation_node = correlation_nodes_with_pattern[0]
+                    self._merge_correlation_values(target_correlation_node, value, corr)
+                else:
+                    # STEP 5: Create new correlation node for eligible nodes only
+                    correlation_node_id = f"corr_{abs(hash(str(sorted(eligible_nodes))))}"
+                    self.add_node(correlation_node_id, NodeType.CORRELATION_OBJECT,
+                                metadata={'values': [value], 'sources': corr['sources'],
+                                            'correlated_nodes': list(eligible_nodes)})
+                    
+                    # Create edges from eligible nodes to this correlation node
+                    for c_node_id in eligible_nodes:
+                        if self.graph.has_node(c_node_id):
+                            attribute = corr['sources'][0]['path'].split('.')[-1]
+                            relationship_type = f"c_{attribute}"
+                            self.add_edge(c_node_id, correlation_node_id, relationship_type, confidence_score=0.9)
+
+            self._update_correlation_index(node_id, attributes)
+
+        self.last_modified = datetime.now(timezone.utc).isoformat()
+        return is_new_node
+
+    def _filter_nodes_without_direct_edges(self, node_set: set) -> set:
+        """
+        Filter out nodes that already have direct edges between them.
+        Returns set of nodes that should be included in correlation.
+        """
+        nodes_list = list(node_set)
+        eligible_nodes = set(node_set)  # Start with all nodes
+        
+        # Check all pairs of nodes
+        for i in range(len(nodes_list)):
+            for j in range(i + 1, len(nodes_list)):
+                node_a = nodes_list[i]
+                node_b = nodes_list[j]
+                
+                # Check if direct edge exists in either direction
+                if self._has_direct_edge_bidirectional(node_a, node_b):
+                    # Remove both nodes from eligible set since they're already connected
+                    eligible_nodes.discard(node_a)
+                    eligible_nodes.discard(node_b)
+        
+        return eligible_nodes
+
+    def _has_direct_edge_bidirectional(self, node_a: str, node_b: str) -> bool:
+        """
+        Check if there's a direct edge between two nodes in either direction.
+        Returns True if node_a→node_b OR node_b→node_a exists.
+        """
+        return (self.graph.has_edge(node_a, node_b) or 
+                self.graph.has_edge(node_b, node_a))
+
+    def _correlation_value_matches_existing_node(self, correlation_value: str) -> bool:
+        """
+        Check if correlation value contains any existing node ID as substring.
+        Returns True if match found (correlation node should NOT be created).
+        """
+        correlation_str = str(correlation_value).lower()
+        
+        # Check against all existing nodes
+        for existing_node_id in self.graph.nodes():
+            if existing_node_id.lower() in correlation_str:
+                return True
+        
+        return False
+
+    def _find_correlation_nodes_with_same_pattern(self, node_set: set) -> List[str]:
+        """
+        Find existing correlation nodes that have the exact same pattern of connected nodes.
+        Returns list of correlation node IDs with matching patterns.
+        """
+        correlation_nodes = self.get_nodes_by_type(NodeType.CORRELATION_OBJECT)
+        matching_nodes = []
+        
+        for corr_node_id in correlation_nodes:
+            # Get all nodes connected to this correlation node
+            connected_nodes = set()
+            
+            # Add all predecessors (nodes pointing TO the correlation node)
+            connected_nodes.update(self.graph.predecessors(corr_node_id))
+            
+            # Add all successors (nodes pointed TO by the correlation node)  
+            connected_nodes.update(self.graph.successors(corr_node_id))
+            
+            # Check if the pattern matches exactly
+            if connected_nodes == node_set:
+                matching_nodes.append(corr_node_id)
+        
+        return matching_nodes
+
+    def _merge_correlation_values(self, target_node_id: str, new_value: Any, corr_data: Dict) -> None:
+        """
+        Merge a new correlation value into an existing correlation node.
+        Uses same logic as large entity merging.
+        """
+        if not self.graph.has_node(target_node_id):
+            return
+            
+        target_metadata = self.graph.nodes[target_node_id]['metadata']
+        
+        # Get existing values (ensure it's a list)
+        existing_values = target_metadata.get('values', [])
+        if not isinstance(existing_values, list):
+            existing_values = [existing_values]
+        
+        # Add new value if not already present
+        if new_value not in existing_values:
+            existing_values.append(new_value)
+            
+        # Merge sources
+        existing_sources = target_metadata.get('sources', [])
+        new_sources = corr_data.get('sources', [])
+        
+        # Create set of unique sources based on (node_id, path) tuples
+        source_set = set()
+        for source in existing_sources + new_sources:
+            source_tuple = (source['node_id'], source['path'])
+            source_set.add(source_tuple)
+        
+        # Convert back to list of dictionaries
+        merged_sources = [{'node_id': nid, 'path': path} for nid, path in source_set]
+        
+        # Update metadata
+        target_metadata.update({
+            'values': existing_values,
+            'sources': merged_sources,
+            'correlated_nodes': list(set(target_metadata.get('correlated_nodes', []) + corr_data.get('nodes', []))),
+            'merge_count': len(existing_values),
+            'last_merge_timestamp': datetime.now(timezone.utc).isoformat()
+        })
+        
+        # Update description to reflect merged nature
+        value_count = len(existing_values)
+        node_count = len(target_metadata['correlated_nodes'])
+        self.graph.nodes[target_node_id]['description'] = (
+            f"Correlation container with {value_count} merged values "
+            f"across {node_count} nodes"
+        )
+        
+    def add_edge(self, source_id: str, target_id: str, relationship_type: str,
+                 confidence_score: float = 0.5, source_provider: str = "unknown",
+                 raw_data: Optional[Dict[str, Any]] = None) -> bool:
+        """Add or update an edge between two nodes, ensuring nodes exist."""
+        if not self.graph.has_node(source_id) or not self.graph.has_node(target_id):
             return False
 
-        node_attributes = {
-            'type': node_type.value,
-            'added_timestamp': datetime.now(timezone.utc).isoformat(),
-            'metadata': metadata or {}
-        }
-
-        self.graph.add_node(node_id, **node_attributes)
-        self.last_modified = datetime.now(timezone.utc).isoformat()
-        return True
-
-    def add_edge(self, source_id: str, target_id: str,
-                 relationship_type: RelationshipType,
-                 confidence_score: Optional[float] = None,
-                 source_provider: str = "unknown",
-                 raw_data: Optional[Dict[str, Any]] = None) -> bool:
-        """
-        Add an edge between two nodes.
-
-        Args:
-            source_id: Source node identifier
-            target_id: Target node identifier
-            relationship_type: Type of relationship
-            confidence_score: Custom confidence score (overrides default)
-            source_provider: Provider that discovered this relationship
-            raw_data: Raw data from provider response
-
-        Returns:
-            bool: True if edge was added, False if it already exists
-        """
-
-        if not self.graph.has_node(source_id) or not self.graph.has_node(target_id):
-            # If the target node is a subdomain, it should be added.
-            # The scanner will handle this logic.
-            pass
-
-        # Check if edge already exists
+        new_confidence = confidence_score
+        
+        if relationship_type.startswith("c_"):
+            edge_label = relationship_type
+        else:
+            edge_label = f"{source_provider}_{relationship_type}"
+        
         if self.graph.has_edge(source_id, target_id):
-            # Update confidence score if new score is higher
-            existing_confidence = self.graph.edges[source_id, target_id]['confidence_score']
-            new_confidence = confidence_score or relationship_type.default_confidence
-
-            if new_confidence > existing_confidence:
+            # If edge exists, update confidence if the new score is higher.
+            if new_confidence > self.graph.edges[source_id, target_id].get('confidence_score', 0):
                 self.graph.edges[source_id, target_id]['confidence_score'] = new_confidence
                 self.graph.edges[source_id, target_id]['updated_timestamp'] = datetime.now(timezone.utc).isoformat()
                 self.graph.edges[source_id, target_id]['updated_by'] = source_provider
-
             return False
 
-        edge_attributes = {
-            'relationship_type': relationship_type.relationship_name,
-            'confidence_score': confidence_score or relationship_type.default_confidence,
-            'source_provider': source_provider,
-            'discovery_timestamp': datetime.now(timezone.utc).isoformat(),
-            'raw_data': raw_data or {}
-        }
-
-        self.graph.add_edge(source_id, target_id, **edge_attributes)
+        # Add a new edge with all attributes.
+        self.graph.add_edge(source_id, target_id,
+                            relationship_type=edge_label,
+                            confidence_score=new_confidence,
+                            source_provider=source_provider,
+                            discovery_timestamp=datetime.now(timezone.utc).isoformat(),
+                            raw_data=raw_data or {})
         self.last_modified = datetime.now(timezone.utc).isoformat()
         return True
 
@@ -153,270 +354,100 @@ class GraphManager:
         return self.graph.number_of_edges()
 
     def get_nodes_by_type(self, node_type: NodeType) -> List[str]:
-        """
-        Get all nodes of a specific type.
-
-        Args:
-            node_type: Type of nodes to retrieve
-
-        Returns:
-            List of node identifiers
-        """
-        return [
-            node_id for node_id, attributes in self.graph.nodes(data=True)
-            if attributes.get('type') == node_type.value
-        ]
+        """Get all nodes of a specific type."""
+        return [n for n, d in self.graph.nodes(data=True) if d.get('type') == node_type.value]
 
     def get_neighbors(self, node_id: str) -> List[str]:
-        """
-        Get all neighboring nodes (both incoming and outgoing).
-
-        Args:
-            node_id: Node identifier
-
-        Returns:
-            List of neighboring node identifiers
-        """
+        """Get all unique neighbors (predecessors and successors) for a node."""
         if not self.graph.has_node(node_id):
             return []
-
-        predecessors = list(self.graph.predecessors(node_id))
-        successors = list(self.graph.successors(node_id))
-        return list(set(predecessors + successors))
+        return list(set(self.graph.predecessors(node_id)) | set(self.graph.successors(node_id)))
 
     def get_high_confidence_edges(self, min_confidence: float = 0.8) -> List[Tuple[str, str, Dict]]:
-        """
-        Get edges with confidence score above threshold.
-
-        Args:
-            min_confidence: Minimum confidence threshold
-
-        Returns:
-            List of tuples (source, target, attributes)
-        """
-        return [
-            (source, target, attributes)
-            for source, target, attributes in self.graph.edges(data=True)
-            if attributes.get('confidence_score', 0) >= min_confidence
-        ]
+        """Get edges with confidence score above a given threshold."""
+        return [(u, v, d) for u, v, d in self.graph.edges(data=True)
+                if d.get('confidence_score', 0) >= min_confidence]
 
     def get_graph_data(self) -> Dict[str, Any]:
-        """
-        Export graph data for visualization.
-        Uses comprehensive metadata collected during scanning.
-        """
+        """Export graph data formatted for frontend visualization."""
         nodes = []
-        edges = []
-
-        # Create nodes with the comprehensive metadata already collected
-        for node_id, attributes in self.graph.nodes(data=True):
-            node_data = {
-                'id': node_id,
-                'label': node_id,
-                'type': attributes.get('type', 'unknown'),
-                'metadata': attributes.get('metadata', {}),
-                'added_timestamp': attributes.get('added_timestamp')
-            }
-
-            # Handle certificate node labeling
-            if node_id.startswith('cert_'):
-                # For certificate nodes, create a more informative label
-                cert_metadata = node_data['metadata']
-                issuer = cert_metadata.get('issuer_name', 'Unknown')
-                valid_status = "✓" if cert_metadata.get('is_currently_valid') else "✗"
-                node_data['label'] = f"Certificate {valid_status}\n{issuer[:30]}..."
-
-            # Color coding by type
-            type_colors = {
-                'domain': {
-                    'background': '#00ff41',
-                    'border': '#00aa2e',
-                    'highlight': {'background': '#44ff75', 'border': '#00ff41'},
-                    'hover': {'background': '#22ff63', 'border': '#00cc35'}
-                },
-                'ip': {
-                    'background': '#ff9900',
-                    'border': '#cc7700',
-                    'highlight': {'background': '#ffbb44', 'border': '#ff9900'},
-                    'hover': {'background': '#ffaa22', 'border': '#dd8800'}
-                },
-                'asn': {
-                    'background': '#00aaff',
-                    'border': '#0088cc',
-                    'highlight': {'background': '#44ccff', 'border': '#00aaff'},
-                    'hover': {'background': '#22bbff', 'border': '#0099dd'}
-                },
-                'dns_record': {
-                    'background': '#9d4edd',
-                    'border': '#7b2cbf',
-                    'highlight': {'background': '#c77dff', 'border': '#9d4edd'},
-                    'hover': {'background': '#b392f0', 'border': '#8b5cf6'}
-                },
-                'large_entity': {
-                    'background': '#ff6b6b',
-                    'border': '#cc3a3a',
-                    'highlight': {'background': '#ff8c8c', 'border': '#ff6b6b'},
-                    'hover': {'background': '#ff7a7a', 'border': '#dd4a4a'}
-                }
-            }
-
-            node_color_config = type_colors.get(attributes.get('type', 'unknown'), type_colors['domain'])
-
-            node_data['color'] = node_color_config
-
-            # Add certificate validity indicator if available
-            metadata = node_data['metadata']
-            if 'certificate_data' in metadata and 'has_valid_cert' in metadata['certificate_data']:
-                node_data['has_valid_cert'] = metadata['certificate_data']['has_valid_cert']
-
+        for node_id, attrs in self.graph.nodes(data=True):
+            node_data = {'id': node_id, 'label': node_id, 'type': attrs.get('type', 'unknown'),
+                         'attributes': attrs.get('attributes', {}),
+                         'description': attrs.get('description', ''),
+                         'metadata': attrs.get('metadata', {}),
+                         'added_timestamp': attrs.get('added_timestamp')}
+            # Customize node appearance based on type and attributes
+            node_type = node_data['type']
+            attributes = node_data['attributes']
+            if node_type == 'domain' and attributes.get('certificates', {}).get('has_valid_cert') is False:
+                node_data['color'] = {'background': '#c7c7c7', 'border': '#999'} # Gray for invalid cert
+            
+            # Add incoming and outgoing edges to node data
+            if self.graph.has_node(node_id):
+                node_data['incoming_edges'] = [{'from': u, 'data': d} for u, _, d in self.graph.in_edges(node_id, data=True)]
+                node_data['outgoing_edges'] = [{'to': v, 'data': d} for _, v, d in self.graph.out_edges(node_id, data=True)]
+            
             nodes.append(node_data)
 
-        # Create edges (unchanged from original)
-        for source, target, attributes in self.graph.edges(data=True):
-            edge_data = {
-                'from': source,
-                'to': target,
-                'label': attributes.get('relationship_type', ''),
-                'confidence_score': attributes.get('confidence_score', 0),
-                'source_provider': attributes.get('source_provider', ''),
-                'discovery_timestamp': attributes.get('discovery_timestamp')
-            }
-
-            # Enhanced edge styling based on confidence
-            confidence = attributes.get('confidence_score', 0)
-            if confidence >= 0.8:
-                edge_data['color'] = {
-                    'color': '#00ff41',
-                    'highlight': '#44ff75',
-                    'hover': '#22ff63',
-                    'inherit': False
-                }
-                edge_data['width'] = 4
-            elif confidence >= 0.6:
-                edge_data['color'] = {
-                    'color': '#ff9900',
-                    'highlight': '#ffbb44',
-                    'hover': '#ffaa22',
-                    'inherit': False
-                }
-                edge_data['width'] = 3
-            else:
-                edge_data['color'] = {
-                    'color': '#666666',
-                    'highlight': '#888888',
-                    'hover': '#777777',
-                    'inherit': False
-                }
-                edge_data['width'] = 2
-
-            # Add dashed line for low confidence
-            if confidence < 0.6:
-                edge_data['dashes'] = [5, 5]
-
-            edges.append(edge_data)
-
+        edges = []
+        for source, target, attrs in self.graph.edges(data=True):
+            edges.append({'from': source, 'to': target,
+                          'label': attrs.get('relationship_type', ''),
+                          'confidence_score': attrs.get('confidence_score', 0),
+                          'source_provider': attrs.get('source_provider', ''),
+                          'discovery_timestamp': attrs.get('discovery_timestamp')})
         return {
-            'nodes': nodes,
-            'edges': edges,
-            'statistics': {
-                'node_count': len(nodes),
-                'edge_count': len(edges),
-                'creation_time': self.creation_time,
-                'last_modified': self.last_modified
-            }
+            'nodes': nodes, 'edges': edges,
+            'statistics': self.get_statistics()['basic_metrics']
         }
 
     def export_json(self) -> Dict[str, Any]:
-        """
-        Export complete graph data as JSON for download.
-
-        Returns:
-            Dictionary containing complete graph data with metadata
-        """
-        # Get basic graph data
-        graph_data = self.get_graph_data()
-
-        # Add comprehensive metadata
-        export_data = {
+        """Export complete graph data as a JSON-serializable dictionary."""
+        graph_data = nx.node_link_data(self.graph) # Use NetworkX's built-in robust serializer
+        return {
             'export_metadata': {
                 'export_timestamp': datetime.now(timezone.utc).isoformat(),
                 'graph_creation_time': self.creation_time,
                 'last_modified': self.last_modified,
-                'total_nodes': self.graph.number_of_nodes(),
-                'total_edges': self.graph.number_of_edges(),
-                'graph_format': 'dnsrecon_v1'
+                'total_nodes': self.get_node_count(),
+                'total_edges': self.get_edge_count(),
+                'graph_format': 'dnsrecon_v1_nodeling'
             },
-            'nodes': graph_data['nodes'],
-            'edges': graph_data['edges'],
-            'node_types': [node_type.value for node_type in NodeType],
-            'relationship_types': [
-                {
-                    'name': rel_type.relationship_name,
-                    'default_confidence': rel_type.default_confidence
-                }
-                for rel_type in RelationshipType
-            ],
-            'confidence_distribution': self._get_confidence_distribution()
+            'graph': graph_data,
+            'statistics': self.get_statistics()
         }
 
-        return export_data
-
     def _get_confidence_distribution(self) -> Dict[str, int]:
-        """Get distribution of confidence scores."""
+        """Get distribution of edge confidence scores."""
         distribution = {'high': 0, 'medium': 0, 'low': 0}
-
-        for _, _, attributes in self.graph.edges(data=True):
-            confidence = attributes.get('confidence_score', 0)
-            if confidence >= 0.8:
-                distribution['high'] += 1
-            elif confidence >= 0.6:
-                distribution['medium'] += 1
-            else:
-                distribution['low'] += 1
-
+        for _, _, confidence in self.graph.edges(data='confidence_score', default=0):
+            if confidence >= 0.8: distribution['high'] += 1
+            elif confidence >= 0.6: distribution['medium'] += 1
+            else: distribution['low'] += 1
         return distribution
 
     def get_statistics(self) -> Dict[str, Any]:
-        """
-        Get comprehensive graph statistics.
-
-        Returns:
-            Dictionary containing various graph metrics
-        """
-        stats = {
-            'basic_metrics': {
-                'total_nodes': self.graph.number_of_nodes(),
-                'total_edges': self.graph.number_of_edges(),
-                'creation_time': self.creation_time,
-                'last_modified': self.last_modified
-            },
-            'node_type_distribution': {},
-            'relationship_type_distribution': {},
-            'confidence_distribution': self._get_confidence_distribution(),
-            'provider_distribution': {}
-        }
-
-        # Node type distribution
+        """Get comprehensive statistics about the graph."""
+        stats = {'basic_metrics': {'total_nodes': self.get_node_count(),
+                                   'total_edges': self.get_edge_count(),
+                                   'creation_time': self.creation_time,
+                                   'last_modified': self.last_modified},
+                 'node_type_distribution': {}, 'relationship_type_distribution': {},
+                 'confidence_distribution': self._get_confidence_distribution(),
+                 'provider_distribution': {}}
+        # Calculate distributions
         for node_type in NodeType:
-            count = len(self.get_nodes_by_type(node_type))
-            stats['node_type_distribution'][node_type.value] = count
-
-        # Relationship type distribution
-        for _, _, attributes in self.graph.edges(data=True):
-            rel_type = attributes.get('relationship_type', 'unknown')
-            stats['relationship_type_distribution'][rel_type] = \
-                stats['relationship_type_distribution'].get(rel_type, 0) + 1
-
-        # Provider distribution
-        for _, _, attributes in self.graph.edges(data=True):
-            provider = attributes.get('source_provider', 'unknown')
-            stats['provider_distribution'][provider] = \
-                stats['provider_distribution'].get(provider, 0) + 1
-
+            stats['node_type_distribution'][node_type.value] = self.get_nodes_by_type(node_type).__len__()
+        for _, _, rel_type in self.graph.edges(data='relationship_type', default='unknown'):
+            stats['relationship_type_distribution'][rel_type] = stats['relationship_type_distribution'].get(rel_type, 0) + 1
+        for _, _, provider in self.graph.edges(data='source_provider', default='unknown'):
+            stats['provider_distribution'][provider] = stats['provider_distribution'].get(provider, 0) + 1
         return stats
 
     def clear(self) -> None:
-        """Clear all nodes and edges from the graph."""
+        """Clear all nodes, edges, and indices from the graph."""
         self.graph.clear()
+        self.correlation_index.clear()
         self.creation_time = datetime.now(timezone.utc).isoformat()
         self.last_modified = self.creation_time

core/logger.py

@@ -1,7 +1,4 @@
-"""
-Forensic logging system for DNSRecon tool.
-Provides structured audit trail for all reconnaissance activities.
-"""
+# dnsrecon/core/logger.py
 
 import logging
 import threading
@@ -82,7 +79,29 @@ class ForensicLogger:
             console_handler = logging.StreamHandler()
             console_handler.setFormatter(formatter)
             self.logger.addHandler(console_handler)
-    
+
+    def __getstate__(self):
+        """Prepare ForensicLogger for pickling by excluding unpicklable objects."""
+        state = self.__dict__.copy()
+        # Remove the unpickleable 'logger' attribute
+        if 'logger' in state:
+            del state['logger']
+        return state
+
+    def __setstate__(self, state):
+        """Restore ForensicLogger after unpickling by reconstructing logger."""
+        self.__dict__.update(state)
+        # Re-initialize the 'logger' attribute
+        self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
+        self.logger.setLevel(logging.INFO)
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+        )
+        if not self.logger.handlers:
+            console_handler = logging.StreamHandler()
+            console_handler.setFormatter(formatter)
+            self.logger.addHandler(console_handler)
+
     def _generate_session_id(self) -> str:
         """Generate unique session identifier."""
         return f"dnsrecon_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"

core/session_config.py

@@ -1,6 +1,6 @@
 """
-Per-session configuration management for DNSRecon.
-Provides isolated configuration instances for each user session.
+Enhanced per-session configuration management for DNSRecon.
+Provides isolated configuration instances for each user session while supporting global caching.
 """
 
 import os
@@ -9,16 +9,15 @@ from typing import Dict, Optional
 
 class SessionConfig:
     """
-    Session-specific configuration that inherits from global config
-    but maintains isolated API keys and provider settings.
+    Enhanced session-specific configuration that inherits from global config
+    but maintains isolated API keys and provider settings while supporting global caching.
     """
     
     def __init__(self):
-        """Initialize session config with global defaults."""
+        """Initialize enhanced session config with global cache support."""
         # Copy all attributes from global config
         self.api_keys: Dict[str, Optional[str]] = {
-            'shodan': None,
-            'virustotal': None
+            'shodan': None
         }
         
         # Default settings (copied from global config)
@@ -27,22 +26,39 @@ class SessionConfig:
         self.max_concurrent_requests = 5
         self.large_entity_threshold = 100
         
-        # Rate limiting settings (per session)
+        # Enhanced rate limiting settings (per session)
         self.rate_limits = {
             'crtsh': 60,
-            'virustotal': 4,
             'shodan': 60,
             'dns': 100
         }
         
-        # Provider settings (per session)
+        # Enhanced provider settings (per session)
         self.enabled_providers = {
             'crtsh': True,
             'dns': True,
-            'virustotal': False,
             'shodan': False
         }
         
+        # Task-based execution settings
+        self.task_retry_settings = {
+            'max_retries': 3,
+            'base_backoff_seconds': 1.0,
+            'max_backoff_seconds': 60.0,
+            'retry_on_rate_limit': True,
+            'retry_on_connection_error': True,
+            'retry_on_timeout': True
+        }
+        
+        # Cache settings (global across all sessions)
+        self.cache_settings = {
+            'enabled': True,
+            'expiry_hours': 12,
+            'cache_base_dir': '.cache',
+            'per_provider_directories': True,
+            'thread_safe_operations': True
+        }
+        
         # Logging configuration
         self.log_level = 'INFO'
         self.log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
@@ -51,21 +67,42 @@ class SessionConfig:
         self.flask_host = '127.0.0.1'
         self.flask_port = 5000
         self.flask_debug = True
+        
+        # Session isolation settings
+        self.session_isolation = {
+            'enforce_single_session_per_user': True,
+            'consolidate_session_data_on_replacement': True,
+            'user_fingerprinting_enabled': True,
+            'session_timeout_minutes': 60
+        }
+        
+        # Circuit breaker settings for provider reliability
+        self.circuit_breaker = {
+            'enabled': True,
+            'failure_threshold': 5,  # Failures before opening circuit
+            'recovery_timeout_seconds': 300,  # 5 minutes before trying again
+            'half_open_max_calls': 3  # Test calls when recovering
+        }
     
     def set_api_key(self, provider: str, api_key: str) -> bool:
         """
         Set API key for a provider in this session.
         
         Args:
-            provider: Provider name (shodan, virustotal)
-            api_key: API key string
+            provider: Provider name (shodan, etc)
+            api_key: API key string (empty string to clear)
             
         Returns:
             bool: True if key was set successfully
         """
         if provider in self.api_keys:
-            self.api_keys[provider] = api_key
-            self.enabled_providers[provider] = True if api_key else False
+            # Handle clearing of API keys
+            if api_key and api_key.strip():
+                self.api_keys[provider] = api_key.strip()
+                self.enabled_providers[provider] = True
+            else:
+                self.api_keys[provider] = None
+                self.enabled_providers[provider] = False
             return True
         return False
     
@@ -105,22 +142,231 @@ class SessionConfig:
         """
         return self.rate_limits.get(provider, 60)
     
-    def load_from_env(self):
-        """Load configuration from environment variables (only if not already set)."""
-        if os.getenv('VIRUSTOTAL_API_KEY') and not self.api_keys['virustotal']:
-            self.set_api_key('virustotal', os.getenv('VIRUSTOTAL_API_KEY'))
+    def get_task_retry_config(self) -> Dict[str, any]:
+        """
+        Get task retry configuration for this session.
         
+        Returns:
+            Dictionary with retry settings
+        """
+        return self.task_retry_settings.copy()
+    
+    def get_cache_config(self) -> Dict[str, any]:
+        """
+        Get cache configuration (global settings).
+        
+        Returns:
+            Dictionary with cache settings
+        """
+        return self.cache_settings.copy()
+    
+    def is_circuit_breaker_enabled(self) -> bool:
+        """Check if circuit breaker is enabled for provider reliability."""
+        return self.circuit_breaker.get('enabled', True)
+    
+    def get_circuit_breaker_config(self) -> Dict[str, any]:
+        """Get circuit breaker configuration."""
+        return self.circuit_breaker.copy()
+    
+    def update_provider_settings(self, provider_updates: Dict[str, Dict[str, any]]) -> bool:
+        """
+        Update provider-specific settings in bulk.
+        
+        Args:
+            provider_updates: Dictionary of provider -> settings updates
+            
+        Returns:
+            bool: True if updates were applied successfully
+        """
+        try:
+            for provider_name, updates in provider_updates.items():
+                # Update rate limits
+                if 'rate_limit' in updates:
+                    self.rate_limits[provider_name] = updates['rate_limit']
+                
+                # Update enabled status
+                if 'enabled' in updates:
+                    self.enabled_providers[provider_name] = updates['enabled']
+                
+                # Update API key
+                if 'api_key' in updates:
+                    self.set_api_key(provider_name, updates['api_key'])
+            
+            return True
+        except Exception as e:
+            print(f"Error updating provider settings: {e}")
+            return False
+    
+    def validate_configuration(self) -> Dict[str, any]:
+        """
+        Validate the current configuration and return validation results.
+        
+        Returns:
+            Dictionary with validation results and any issues found
+        """
+        validation_result = {
+            'valid': True,
+            'warnings': [],
+            'errors': [],
+            'provider_status': {}
+        }
+        
+        # Validate provider configurations
+        for provider_name, enabled in self.enabled_providers.items():
+            provider_status = {
+                'enabled': enabled,
+                'has_api_key': bool(self.api_keys.get(provider_name)),
+                'rate_limit': self.rate_limits.get(provider_name, 60)
+            }
+            
+            # Check for potential issues
+            if enabled and provider_name in ['shodan'] and not provider_status['has_api_key']:
+                validation_result['warnings'].append(
+                    f"Provider '{provider_name}' is enabled but missing API key"
+                )
+            
+            validation_result['provider_status'][provider_name] = provider_status
+        
+        # Validate task settings
+        if self.task_retry_settings['max_retries'] > 10:
+            validation_result['warnings'].append(
+                f"High retry count ({self.task_retry_settings['max_retries']}) may cause long delays"
+            )
+        
+        # Validate concurrent settings
+        if self.max_concurrent_requests > 10:
+            validation_result['warnings'].append(
+                f"High concurrency ({self.max_concurrent_requests}) may overwhelm providers"
+            )
+        
+        # Validate cache settings
+        if not os.path.exists(self.cache_settings['cache_base_dir']):
+            try:
+                os.makedirs(self.cache_settings['cache_base_dir'], exist_ok=True)
+            except Exception as e:
+                validation_result['errors'].append(f"Cannot create cache directory: {e}")
+                validation_result['valid'] = False
+        
+        return validation_result
+    
+    def load_from_env(self):
+        """Load configuration from environment variables with enhanced validation."""
+        # Load API keys from environment
         if os.getenv('SHODAN_API_KEY') and not self.api_keys['shodan']:
             self.set_api_key('shodan', os.getenv('SHODAN_API_KEY'))
+            print("Loaded Shodan API key from environment")
         
         # Override default settings from environment
         self.default_recursion_depth = int(os.getenv('DEFAULT_RECURSION_DEPTH', '2'))
-        self.default_timeout = 30
-        self.max_concurrent_requests = 5
+        self.default_timeout = int(os.getenv('DEFAULT_TIMEOUT', '30'))
+        self.max_concurrent_requests = int(os.getenv('MAX_CONCURRENT_REQUESTS', '5'))
+        
+        # Load task retry settings from environment
+        if os.getenv('TASK_MAX_RETRIES'):
+            self.task_retry_settings['max_retries'] = int(os.getenv('TASK_MAX_RETRIES'))
+        
+        if os.getenv('TASK_BASE_BACKOFF'):
+            self.task_retry_settings['base_backoff_seconds'] = float(os.getenv('TASK_BASE_BACKOFF'))
+        
+        # Load cache settings from environment
+        if os.getenv('CACHE_EXPIRY_HOURS'):
+            self.cache_settings['expiry_hours'] = int(os.getenv('CACHE_EXPIRY_HOURS'))
+        
+        if os.getenv('CACHE_DISABLED'):
+            self.cache_settings['enabled'] = os.getenv('CACHE_DISABLED').lower() != 'true'
+        
+        # Load circuit breaker settings
+        if os.getenv('CIRCUIT_BREAKER_DISABLED'):
+            self.circuit_breaker['enabled'] = os.getenv('CIRCUIT_BREAKER_DISABLED').lower() != 'true'
+        
+        # Flask settings
+        self.flask_debug = os.getenv('FLASK_DEBUG', 'True').lower() == 'true'
+        
+        print("Enhanced configuration loaded from environment")
+    
+    def export_config_summary(self) -> Dict[str, any]:
+        """
+        Export a summary of the current configuration for debugging/logging.
+        
+        Returns:
+            Dictionary with configuration summary (API keys redacted)
+        """
+        return {
+            'providers': {
+                provider: {
+                    'enabled': self.enabled_providers.get(provider, False),
+                    'has_api_key': bool(self.api_keys.get(provider)),
+                    'rate_limit': self.rate_limits.get(provider, 60)
+                }
+                for provider in self.enabled_providers.keys()
+            },
+            'task_settings': {
+                'max_retries': self.task_retry_settings['max_retries'],
+                'max_concurrent_requests': self.max_concurrent_requests,
+                'large_entity_threshold': self.large_entity_threshold
+            },
+            'cache_settings': {
+                'enabled': self.cache_settings['enabled'],
+                'expiry_hours': self.cache_settings['expiry_hours'],
+                'base_directory': self.cache_settings['cache_base_dir']
+            },
+            'session_settings': {
+                'isolation_enabled': self.session_isolation['enforce_single_session_per_user'],
+                'consolidation_enabled': self.session_isolation['consolidate_session_data_on_replacement'],
+                'timeout_minutes': self.session_isolation['session_timeout_minutes']
+            },
+            'circuit_breaker': {
+                'enabled': self.circuit_breaker['enabled'],
+                'failure_threshold': self.circuit_breaker['failure_threshold'],
+                'recovery_timeout': self.circuit_breaker['recovery_timeout_seconds']
+            }
+        }
 
 
 def create_session_config() -> SessionConfig:
-    """Create a new session configuration instance."""
+    """
+    Create a new enhanced session configuration instance.
+    
+    Returns:
+        Configured SessionConfig instance
+    """
     session_config = SessionConfig()
     session_config.load_from_env()
-    return session_config
+    
+    # Validate configuration and log any issues
+    validation = session_config.validate_configuration()
+    if validation['warnings']:
+        print("Configuration warnings:")
+        for warning in validation['warnings']:
+            print(f"  WARNING: {warning}")
+    
+    if validation['errors']:
+        print("Configuration errors:")
+        for error in validation['errors']:
+            print(f"  ERROR: {error}")
+    
+    if not validation['valid']:
+        raise ValueError("Configuration validation failed - see errors above")
+    
+    print(f"Enhanced session configuration created successfully")
+    return session_config
+
+
+def create_test_config() -> SessionConfig:
+    """
+    Create a test configuration with safe defaults for testing.
+    
+    Returns:
+        Test-safe SessionConfig instance
+    """
+    test_config = SessionConfig()
+    
+    # Override settings for testing
+    test_config.max_concurrent_requests = 2
+    test_config.task_retry_settings['max_retries'] = 1
+    test_config.task_retry_settings['base_backoff_seconds'] = 0.1
+    test_config.cache_settings['expiry_hours'] = 1
+    test_config.session_isolation['session_timeout_minutes'] = 10
+    
+    print("Test configuration created")
+    return test_config

core/session_manager.py

@@ -1,280 +1,575 @@
-"""
-Session manager for DNSRecon multi-user support.
-Manages individual scanner instances per user session with automatic cleanup.
-"""
+# dnsrecon/core/session_manager.py
 
 import threading
 import time
 import uuid
-from typing import Dict, Optional, Any
-from datetime import datetime, timezone
+import redis
+import pickle
+import hashlib
+from typing import Dict, Optional, Any, List, Tuple
 
 from core.scanner import Scanner
 
 
+class UserIdentifier:
+    """Handles user identification for session management."""
+    
+    @staticmethod
+    def generate_user_fingerprint(client_ip: str, user_agent: str) -> str:
+        """
+        Generate a unique fingerprint for a user based on IP and User-Agent.
+        
+        Args:
+            client_ip: Client IP address
+            user_agent: User-Agent header value
+            
+        Returns:
+            Unique user fingerprint hash
+        """
+        # Create deterministic user identifier
+        user_data = f"{client_ip}:{user_agent[:100]}"  # Limit UA to 100 chars
+        fingerprint = hashlib.sha256(user_data.encode()).hexdigest()[:16]  # 16 char fingerprint
+        return f"user_{fingerprint}"
+    
+    @staticmethod
+    def extract_request_info(request) -> Tuple[str, str]:
+        """
+        Extract client IP and User-Agent from Flask request.
+        
+        Args:
+            request: Flask request object
+            
+        Returns:
+            Tuple of (client_ip, user_agent)
+        """
+        # Handle proxy headers for real IP
+        client_ip = request.headers.get('X-Forwarded-For', '').split(',')[0].strip()
+        if not client_ip:
+            client_ip = request.headers.get('X-Real-IP', '')
+        if not client_ip:
+            client_ip = request.remote_addr or 'unknown'
+        
+        user_agent = request.headers.get('User-Agent', 'unknown')
+        
+        return client_ip, user_agent
+
+
+class SessionConsolidator:
+    """Handles consolidation of session data when replacing sessions."""
+    
+    @staticmethod
+    def consolidate_scanner_data(old_scanner: 'Scanner', new_scanner: 'Scanner') -> 'Scanner':
+        """
+        Consolidate useful data from old scanner into new scanner.
+        
+        Args:
+            old_scanner: Scanner from terminated session
+            new_scanner: New scanner instance
+            
+        Returns:
+            Enhanced new scanner with consolidated data
+        """
+        try:
+            # Consolidate graph data if old scanner has valuable data
+            if old_scanner and hasattr(old_scanner, 'graph') and old_scanner.graph:
+                old_stats = old_scanner.graph.get_statistics()
+                if old_stats['basic_metrics']['total_nodes'] > 0:
+                    print(f"Consolidating graph data: {old_stats['basic_metrics']['total_nodes']} nodes, {old_stats['basic_metrics']['total_edges']} edges")
+                    
+                    # Transfer nodes and edges to new scanner's graph
+                    for node_id, node_data in old_scanner.graph.graph.nodes(data=True):
+                        # Add node to new graph with all attributes
+                        new_scanner.graph.graph.add_node(node_id, **node_data)
+                    
+                    for source, target, edge_data in old_scanner.graph.graph.edges(data=True):
+                        # Add edge to new graph with all attributes  
+                        new_scanner.graph.graph.add_edge(source, target, **edge_data)
+                    
+                    # Update correlation index
+                    if hasattr(old_scanner.graph, 'correlation_index'):
+                        new_scanner.graph.correlation_index = old_scanner.graph.correlation_index.copy()
+                    
+                    # Update timestamps
+                    new_scanner.graph.creation_time = old_scanner.graph.creation_time
+                    new_scanner.graph.last_modified = old_scanner.graph.last_modified
+            
+            # Consolidate provider statistics
+            if old_scanner and hasattr(old_scanner, 'providers') and old_scanner.providers:
+                for old_provider in old_scanner.providers:
+                    # Find matching provider in new scanner
+                    matching_new_provider = None
+                    for new_provider in new_scanner.providers:
+                        if new_provider.get_name() == old_provider.get_name():
+                            matching_new_provider = new_provider
+                            break
+                    
+                    if matching_new_provider:
+                        # Transfer cumulative statistics
+                        matching_new_provider.total_requests += old_provider.total_requests
+                        matching_new_provider.successful_requests += old_provider.successful_requests
+                        matching_new_provider.failed_requests += old_provider.failed_requests
+                        matching_new_provider.total_relationships_found += old_provider.total_relationships_found
+                        
+                        # Transfer cache statistics if available
+                        if hasattr(old_provider, 'cache_hits'):
+                            matching_new_provider.cache_hits += getattr(old_provider, 'cache_hits', 0)
+                            matching_new_provider.cache_misses += getattr(old_provider, 'cache_misses', 0)
+                            
+                        print(f"Consolidated {old_provider.get_name()} provider stats: {old_provider.total_requests} requests")
+            
+            return new_scanner
+            
+        except Exception as e:
+            print(f"Warning: Error during session consolidation: {e}")
+            return new_scanner
+
+
 class SessionManager:
     """
-    Manages multiple scanner instances for concurrent user sessions.
-    Provides session isolation and automatic cleanup of inactive sessions.
+    Manages single scanner session per user using Redis with user identification.
+    Enforces one active session per user for consistent state management.
     """
     
     def __init__(self, session_timeout_minutes: int = 60):
         """
-        Initialize session manager.
-        
-        Args:
-            session_timeout_minutes: Minutes of inactivity before session cleanup
+        Initialize session manager with Redis backend and user tracking.
         """
-        self.sessions: Dict[str, Dict[str, Any]] = {}
+        self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
         self.session_timeout = session_timeout_minutes * 60  # Convert to seconds
         self.lock = threading.Lock()
         
+        # User identification helper
+        self.user_identifier = UserIdentifier()
+        self.consolidator = SessionConsolidator()
+        
         # Start cleanup thread
         self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
         self.cleanup_thread.start()
         
-        print(f"SessionManager initialized with {session_timeout_minutes}min timeout")
+        print(f"SessionManager initialized with Redis backend, user tracking, and {session_timeout_minutes}min timeout")
     
-    def create_session(self) -> str:
+    def __getstate__(self):
+        """Prepare SessionManager for pickling."""
+        state = self.__dict__.copy()
+        # Exclude unpickleable attributes
+        unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client']
+        for attr in unpicklable_attrs:
+            if attr in state:
+                del state[attr]
+        return state
+
+    def __setstate__(self, state):
+        """Restore SessionManager after unpickling."""
+        self.__dict__.update(state)
+        # Re-initialize unpickleable attributes
+        import redis
+        self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
+        self.lock = threading.Lock()
+        self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
+        self.cleanup_thread.start()
+
+    def _get_session_key(self, session_id: str) -> str:
+        """Generate Redis key for a session."""
+        return f"dnsrecon:session:{session_id}"
+
+    def _get_user_session_key(self, user_fingerprint: str) -> str:
+        """Generate Redis key for user -> session mapping."""
+        return f"dnsrecon:user:{user_fingerprint}"
+
+    def _get_stop_signal_key(self, session_id: str) -> str:
+        """Generate Redis key for session stop signal."""
+        return f"dnsrecon:stop:{session_id}"
+
+    def create_or_replace_user_session(self, client_ip: str, user_agent: str) -> str:
         """
-        Create a new user session with dedicated scanner instance and configuration.
-        Enhanced with better debugging and race condition protection.
+        Create new session for user, replacing any existing session.
+        Consolidates data from previous session if it exists.
         
+        Args:
+            client_ip: Client IP address
+            user_agent: User-Agent header
+            
         Returns:
-            Unique session ID
+            New session ID
         """
-        session_id = str(uuid.uuid4())
+        user_fingerprint = self.user_identifier.generate_user_fingerprint(client_ip, user_agent)
+        new_session_id = str(uuid.uuid4())
         
-        print(f"=== CREATING SESSION {session_id} ===")
+        print(f"=== CREATING/REPLACING SESSION FOR USER {user_fingerprint} ===")
         
         try:
-            # Create session-specific configuration
+            # Check for existing user session
+            existing_session_id = self._get_user_current_session(user_fingerprint)
+            old_scanner = None
+            
+            if existing_session_id:
+                print(f"Found existing session {existing_session_id} for user {user_fingerprint}")
+                # Get old scanner data for consolidation
+                old_scanner = self.get_session(existing_session_id)
+                # Terminate old session
+                self._terminate_session_internal(existing_session_id, cleanup_user_mapping=False)
+                print(f"Terminated old session {existing_session_id}")
+            
+            # Create new session config and scanner
             from core.session_config import create_session_config
             session_config = create_session_config()
+            new_scanner = Scanner(session_config=session_config)
             
-            print(f"Created session config for {session_id}")
+            # Set session ID on scanner for cross-process operations
+            new_scanner.session_id = new_session_id
             
-            # Create scanner with session config
-            from core.scanner import Scanner
-            scanner_instance = Scanner(session_config=session_config)
+            # Consolidate data from old session if available
+            if old_scanner:
+                new_scanner = self.consolidator.consolidate_scanner_data(old_scanner, new_scanner)
+                print(f"Consolidated data from previous session")
             
-            print(f"Created scanner instance {id(scanner_instance)} for session {session_id}")
-            print(f"Initial scanner status: {scanner_instance.status}")
+            # Create session data
+            session_data = {
+                'scanner': new_scanner,
+                'config': session_config,
+                'created_at': time.time(),
+                'last_activity': time.time(),
+                'status': 'active',
+                'user_fingerprint': user_fingerprint,
+                'client_ip': client_ip,
+                'user_agent': user_agent[:200]  # Truncate for storage
+            }
             
-            with self.lock:
-                self.sessions[session_id] = {
-                    'scanner': scanner_instance,
-                    'config': session_config,
-                    'created_at': time.time(),
-                    'last_activity': time.time(),
-                    'user_agent': '',
-                    'status': 'active'
-                }
+            # Store session in Redis
+            session_key = self._get_session_key(new_session_id)
+            serialized_data = pickle.dumps(session_data)
+            self.redis_client.setex(session_key, self.session_timeout, serialized_data)
             
-            print(f"Session {session_id} stored in session manager")
-            print(f"Total active sessions: {len([s for s in self.sessions.values() if s['status'] == 'active'])}")
-            print(f"=== SESSION {session_id} CREATED SUCCESSFULLY ===")
+            # Update user -> session mapping
+            user_session_key = self._get_user_session_key(user_fingerprint)
+            self.redis_client.setex(user_session_key, self.session_timeout, new_session_id.encode('utf-8'))
             
-            return session_id
+            # Initialize stop signal
+            stop_key = self._get_stop_signal_key(new_session_id)
+            self.redis_client.setex(stop_key, self.session_timeout, b'0')
+            
+            print(f"Created new session {new_session_id} for user {user_fingerprint}")
+            return new_session_id
             
         except Exception as e:
-            print(f"ERROR: Failed to create session {session_id}: {e}")
+            print(f"ERROR: Failed to create session for user {user_fingerprint}: {e}")
             raise
 
-    def get_session(self, session_id: str) -> Optional[object]:
-        """
-        Get scanner instance for a session with enhanced debugging.
-        
-        Args:
-            session_id: Session identifier
+    def _get_user_current_session(self, user_fingerprint: str) -> Optional[str]:
+        """Get current session ID for a user."""
+        try:
+            user_session_key = self._get_user_session_key(user_fingerprint)
+            session_id_bytes = self.redis_client.get(user_session_key)
+            if session_id_bytes:
+                return session_id_bytes.decode('utf-8')
+            return None
+        except Exception as e:
+            print(f"Error getting user session: {e}")
+            return None
+
+    def set_stop_signal(self, session_id: str) -> bool:
+        """Set stop signal for session (cross-process safe)."""
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            self.redis_client.setex(stop_key, self.session_timeout, b'1')
+            print(f"Stop signal set for session {session_id}")
+            return True
+        except Exception as e:
+            print(f"ERROR: Failed to set stop signal for session {session_id}: {e}")
+            return False
+
+    def is_stop_requested(self, session_id: str) -> bool:
+        """Check if stop is requested for session (cross-process safe)."""
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            value = self.redis_client.get(stop_key)
+            return value == b'1' if value is not None else False
+        except Exception as e:
+            print(f"ERROR: Failed to check stop signal for session {session_id}: {e}")
+            return False
+
+    def clear_stop_signal(self, session_id: str) -> bool:
+        """Clear stop signal for session."""
+        try:
+            stop_key = self._get_stop_signal_key(session_id)
+            self.redis_client.setex(stop_key, self.session_timeout, b'0')
+            print(f"Stop signal cleared for session {session_id}")
+            return True
+        except Exception as e:
+            print(f"ERROR: Failed to clear stop signal for session {session_id}: {e}")
+            return False
+
+    def _get_session_data(self, session_id: str) -> Optional[Dict[str, Any]]:
+        """Retrieve and deserialize session data from Redis."""
+        try:
+            session_key = self._get_session_key(session_id)
+            serialized_data = self.redis_client.get(session_key)
+            if serialized_data:
+                session_data = pickle.loads(serialized_data)
+                # Ensure scanner has correct session ID
+                if 'scanner' in session_data and session_data['scanner']:
+                    session_data['scanner'].session_id = session_id
+                return session_data
+            return None
+        except Exception as e:
+            print(f"ERROR: Failed to get session data for {session_id}: {e}")
+            return None
+
+    def _save_session_data(self, session_id: str, session_data: Dict[str, Any]) -> bool:
+        """Serialize and save session data to Redis with updated TTL."""
+        try:
+            session_key = self._get_session_key(session_id)
+            serialized_data = pickle.dumps(session_data)
+            result = self.redis_client.setex(session_key, self.session_timeout, serialized_data)
             
-        Returns:
-            Scanner instance or None if session doesn't exist
-        """
+            # Also refresh user mapping TTL if available
+            if 'user_fingerprint' in session_data:
+                user_session_key = self._get_user_session_key(session_data['user_fingerprint'])
+                self.redis_client.setex(user_session_key, self.session_timeout, session_id.encode('utf-8'))
+                
+            return result
+        except Exception as e:
+            print(f"ERROR: Failed to save session data for {session_id}: {e}")
+            return False
+
+    def update_session_scanner(self, session_id: str, scanner: 'Scanner') -> bool:
+        """Update scanner object in session with immediate persistence."""
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data:
+                # Ensure scanner has session ID
+                scanner.session_id = session_id
+                session_data['scanner'] = scanner
+                session_data['last_activity'] = time.time()
+                
+                success = self._save_session_data(session_id, session_data)
+                if success:
+                    print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
+                else:
+                    print(f"WARNING: Failed to save scanner state for session {session_id}")
+                return success
+            else:
+                print(f"WARNING: Session {session_id} not found for scanner update")
+                return False
+        except Exception as e:
+            print(f"ERROR: Failed to update scanner for session {session_id}: {e}")
+            return False
+
+    def update_scanner_status(self, session_id: str, status: str) -> bool:
+        """Quickly update scanner status for immediate GUI feedback."""
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data and 'scanner' in session_data:
+                session_data['scanner'].status = status
+                session_data['last_activity'] = time.time()
+                
+                success = self._save_session_data(session_id, session_data)
+                if success:
+                    print(f"Scanner status updated to '{status}' for session {session_id}")
+                else:
+                    print(f"WARNING: Failed to save status update for session {session_id}")
+                return success
+            return False
+        except Exception as e:
+            print(f"ERROR: Failed to update scanner status for session {session_id}: {e}")
+            return False
+
+    def get_session(self, session_id: str) -> Optional[Scanner]:
+        """Get scanner instance for session with session ID management."""
         if not session_id:
-            print("get_session called with empty session_id")
             return None
             
-        with self.lock:
-            if session_id not in self.sessions:
-                print(f"Session {session_id} not found in session manager")
-                print(f"Available sessions: {list(self.sessions.keys())}")
-                return None
-            
-            session_data = self.sessions[session_id]
-            
-            # Check if session is still active
-            if session_data['status'] != 'active':
-                print(f"Session {session_id} is not active (status: {session_data['status']})")
-                return None
-            
-            # Update last activity
-            session_data['last_activity'] = time.time()
-            scanner = session_data['scanner']
-            
-            print(f"Retrieved scanner {id(scanner)} for session {session_id}")
-            print(f"Scanner status: {scanner.status}")
-            
-            return scanner
-    
-    def get_or_create_session(self, session_id: Optional[str] = None) -> tuple[str, Scanner]:
-        """
-        Get existing session or create new one.
+        session_data = self._get_session_data(session_id)
         
-        Args:
-            session_id: Optional existing session ID
-            
-        Returns:
-            Tuple of (session_id, scanner_instance)
-        """
-        if session_id and self.get_session(session_id):
-            return session_id, self.get_session(session_id)
-        else:
-            new_session_id = self.create_session()
-            return new_session_id, self.get_session(new_session_id)
-    
+        if not session_data or session_data.get('status') != 'active':
+            return None
+        
+        # Update last activity and save back to Redis
+        session_data['last_activity'] = time.time()
+        self._save_session_data(session_id, session_data)
+        
+        scanner = session_data.get('scanner')
+        if scanner:
+            # Ensure scanner can check Redis-based stop signal
+            scanner.session_id = session_id
+        
+        return scanner
+
+    def get_session_status_only(self, session_id: str) -> Optional[str]:
+        """Get scanner status without full session retrieval (for performance)."""
+        try:
+            session_data = self._get_session_data(session_id)
+            if session_data and 'scanner' in session_data:
+                return session_data['scanner'].status
+            return None
+        except Exception as e:
+            print(f"ERROR: Failed to get session status for {session_id}: {e}")
+            return None
+
     def terminate_session(self, session_id: str) -> bool:
-        """
-        Terminate a specific session and cleanup resources.
+        """Terminate specific session with reliable stop signal and immediate status update."""
+        return self._terminate_session_internal(session_id, cleanup_user_mapping=True)
+
+    def _terminate_session_internal(self, session_id: str, cleanup_user_mapping: bool = True) -> bool:
+        """Internal session termination with configurable user mapping cleanup."""
+        print(f"=== TERMINATING SESSION {session_id} ===")
         
-        Args:
-            session_id: Session to terminate
+        try:
+            # Set stop signal first
+            self.set_stop_signal(session_id)
             
-        Returns:
-            True if session was terminated successfully
-        """
-        with self.lock:
-            if session_id not in self.sessions:
+            # Update scanner status immediately for GUI feedback
+            self.update_scanner_status(session_id, 'stopped')
+            
+            session_data = self._get_session_data(session_id)
+            if not session_data:
+                print(f"Session {session_id} not found")
                 return False
+
+            scanner = session_data.get('scanner')
+            if scanner and scanner.status == 'running':
+                print(f"Stopping scan for session: {session_id}")
+                scanner.stop_scan()
+                self.update_session_scanner(session_id, scanner)
             
-            session_data = self.sessions[session_id]
-            scanner = session_data['scanner']
+            # Wait for graceful shutdown
+            time.sleep(0.5)
             
-            # Stop any running scan
-            try:
-                if scanner.status == 'running':
-                    scanner.stop_scan()
-                    print(f"Stopped scan for session: {session_id}")
-            except Exception as e:
-                print(f"Error stopping scan for session {session_id}: {e}")
+            # Clean up user mapping if requested
+            if cleanup_user_mapping and 'user_fingerprint' in session_data:
+                user_session_key = self._get_user_session_key(session_data['user_fingerprint'])
+                self.redis_client.delete(user_session_key)
+                print(f"Cleaned up user mapping for {session_data['user_fingerprint']}")
             
-            # Mark as terminated
-            session_data['status'] = 'terminated'
-            session_data['terminated_at'] = time.time()
+            # Delete session data and stop signal
+            session_key = self._get_session_key(session_id)
+            stop_key = self._get_stop_signal_key(session_id)
+            self.redis_client.delete(session_key)
+            self.redis_client.delete(stop_key)
             
-            # Remove from active sessions after a brief delay to allow cleanup
-            threading.Timer(5.0, lambda: self._remove_session(session_id)).start()
-            
-            print(f"Terminated session: {session_id}")
+            print(f"Terminated and removed session from Redis: {session_id}")
             return True
-    
-    def _remove_session(self, session_id: str) -> None:
-        """Remove session from memory."""
-        with self.lock:
-            if session_id in self.sessions:
-                del self.sessions[session_id]
-                print(f"Removed session from memory: {session_id}")
-    
-    def get_session_info(self, session_id: str) -> Optional[Dict[str, Any]]:
-        """
-        Get session information without updating activity.
-        
-        Args:
-            session_id: Session identifier
             
-        Returns:
-            Session information dictionary or None
-        """
-        with self.lock:
-            if session_id not in self.sessions:
-                return None
+        except Exception as e:
+            print(f"ERROR: Failed to terminate session {session_id}: {e}")
+            return False
+
+    def _cleanup_loop(self) -> None:
+        """Background thread to cleanup inactive sessions and orphaned signals."""
+        while True:
+            try:
+                # Clean up orphaned stop signals
+                stop_keys = self.redis_client.keys("dnsrecon:stop:*")
+                for stop_key in stop_keys:
+                    session_id = stop_key.decode('utf-8').split(':')[-1]
+                    session_key = self._get_session_key(session_id)
+                    
+                    if not self.redis_client.exists(session_key):
+                        self.redis_client.delete(stop_key)
+                        print(f"Cleaned up orphaned stop signal for session {session_id}")
+                
+                # Clean up orphaned user mappings
+                user_keys = self.redis_client.keys("dnsrecon:user:*")
+                for user_key in user_keys:
+                    session_id_bytes = self.redis_client.get(user_key)
+                    if session_id_bytes:
+                        session_id = session_id_bytes.decode('utf-8')
+                        session_key = self._get_session_key(session_id)
+                        
+                        if not self.redis_client.exists(session_key):
+                            self.redis_client.delete(user_key)
+                            print(f"Cleaned up orphaned user mapping for session {session_id}")
+                        
+            except Exception as e:
+                print(f"Error in cleanup loop: {e}")
             
-            session_data = self.sessions[session_id]
-            scanner = session_data['scanner']
+            time.sleep(300)  # Sleep for 5 minutes
+
+    def list_active_sessions(self) -> List[Dict[str, Any]]:
+        """List all active sessions for admin purposes."""
+        try:
+            session_keys = self.redis_client.keys("dnsrecon:session:*")
+            sessions = []
+            
+            for session_key in session_keys:
+                session_id = session_key.decode('utf-8').split(':')[-1]
+                session_data = self._get_session_data(session_id)
+                
+                if session_data:
+                    scanner = session_data.get('scanner')
+                    sessions.append({
+                        'session_id': session_id,
+                        'user_fingerprint': session_data.get('user_fingerprint', 'unknown'),
+                        'client_ip': session_data.get('client_ip', 'unknown'),
+                        'created_at': session_data.get('created_at'),
+                        'last_activity': session_data.get('last_activity'),
+                        'scanner_status': scanner.status if scanner else 'unknown',
+                        'current_target': scanner.current_target if scanner else None
+                    })
+            
+            return sessions
+        except Exception as e:
+            print(f"ERROR: Failed to list active sessions: {e}")
+            return []
+
+    def get_statistics(self) -> Dict[str, Any]:
+        """Get session manager statistics."""
+        try:
+            session_keys = self.redis_client.keys("dnsrecon:session:*")
+            user_keys = self.redis_client.keys("dnsrecon:user:*")
+            stop_keys = self.redis_client.keys("dnsrecon:stop:*")
+            
+            active_sessions = len(session_keys)
+            unique_users = len(user_keys)
+            running_scans = 0
+            
+            for session_key in session_keys:
+                session_id = session_key.decode('utf-8').split(':')[-1]
+                status = self.get_session_status_only(session_id)
+                if status == 'running':
+                    running_scans += 1
+            
+            return {
+                'total_active_sessions': active_sessions,
+                'unique_users': unique_users,
+                'running_scans': running_scans,
+                'total_stop_signals': len(stop_keys),
+                'average_sessions_per_user': round(active_sessions / unique_users, 2) if unique_users > 0 else 0
+            }
+        except Exception as e:
+            print(f"ERROR: Failed to get statistics: {e}")
+            return {
+                'total_active_sessions': 0,
+                'unique_users': 0,
+                'running_scans': 0,
+                'total_stop_signals': 0,
+                'average_sessions_per_user': 0
+            }
+
+    def get_session_info(self, session_id: str) -> Dict[str, Any]:
+        """Get detailed information about a specific session."""
+        try:
+            session_data = self._get_session_data(session_id)
+            if not session_data:
+                return {'error': 'Session not found'}
+            
+            scanner = session_data.get('scanner')
             
             return {
                 'session_id': session_id,
-                'created_at': datetime.fromtimestamp(session_data['created_at'], timezone.utc).isoformat(),
-                'last_activity': datetime.fromtimestamp(session_data['last_activity'], timezone.utc).isoformat(),
-                'status': session_data['status'],
-                'scan_status': scanner.status,
-                'current_target': scanner.current_target,
-                'uptime_seconds': time.time() - session_data['created_at']
-            }
-    
-    def list_active_sessions(self) -> Dict[str, Dict[str, Any]]:
-        """
-        List all active sessions with enhanced debugging info.
-        
-        Returns:
-            Dictionary of session information
-        """
-        active_sessions = {}
-        
-        with self.lock:
-            for session_id, session_data in self.sessions.items():
-                if session_data['status'] == 'active':
-                    scanner = session_data['scanner']
-                    active_sessions[session_id] = {
-                        'session_id': session_id,
-                        'created_at': datetime.fromtimestamp(session_data['created_at'], timezone.utc).isoformat(),
-                        'last_activity': datetime.fromtimestamp(session_data['last_activity'], timezone.utc).isoformat(),
-                        'status': session_data['status'],
-                        'scan_status': scanner.status,
-                        'current_target': scanner.current_target,
-                        'uptime_seconds': time.time() - session_data['created_at'],
-                        'scanner_object_id': id(scanner)
-                    }
-        
-        return active_sessions
-    
-    def _cleanup_loop(self) -> None:
-        """Background thread to cleanup inactive sessions."""
-        while True:
-            try:
-                current_time = time.time()
-                sessions_to_cleanup = []
-                
-                with self.lock:
-                    for session_id, session_data in self.sessions.items():
-                        if session_data['status'] != 'active':
-                            continue
-                            
-                        inactive_time = current_time - session_data['last_activity']
-                        
-                        if inactive_time > self.session_timeout:
-                            sessions_to_cleanup.append(session_id)
-                
-                # Cleanup outside of lock to avoid deadlock
-                for session_id in sessions_to_cleanup:
-                    print(f"Cleaning up inactive session: {session_id}")
-                    self.terminate_session(session_id)
-                
-                # Sleep for 5 minutes between cleanup cycles
-                time.sleep(300)
-                
-            except Exception as e:
-                print(f"Error in session cleanup loop: {e}")
-                time.sleep(60)  # Sleep for 1 minute on error
-    
-    def get_statistics(self) -> Dict[str, Any]:
-        """
-        Get session manager statistics.
-        
-        Returns:
-            Statistics dictionary
-        """
-        with self.lock:
-            active_count = sum(1 for s in self.sessions.values() if s['status'] == 'active')
-            running_scans = sum(1 for s in self.sessions.values() 
-                              if s['status'] == 'active' and s['scanner'].status == 'running')
-            
-            return {
-                'total_sessions': len(self.sessions),
-                'active_sessions': active_count,
-                'running_scans': running_scans,
-                'session_timeout_minutes': self.session_timeout / 60
+                'user_fingerprint': session_data.get('user_fingerprint', 'unknown'),
+                'client_ip': session_data.get('client_ip', 'unknown'),
+                'user_agent': session_data.get('user_agent', 'unknown'),
+                'created_at': session_data.get('created_at'),
+                'last_activity': session_data.get('last_activity'),
+                'status': session_data.get('status'),
+                'scanner_status': scanner.status if scanner else 'unknown',
+                'current_target': scanner.current_target if scanner else None,
+                'session_age_minutes': round((time.time() - session_data.get('created_at', time.time())) / 60, 1)
             }
+        except Exception as e:
+            print(f"ERROR: Failed to get session info for {session_id}: {e}")
+            return {'error': f'Failed to get session info: {str(e)}'}
 
 
 # Global session manager instance

core/task_manager.py

@@ -0,0 +1,564 @@
+# dnsrecon/core/task_manager.py
+
+import threading
+import time
+import uuid
+from enum import Enum
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional, Any, Set
+from datetime import datetime, timezone, timedelta
+from collections import deque
+
+from utils.helpers import _is_valid_ip, _is_valid_domain
+
+
+class TaskStatus(Enum):
+    """Enumeration of task execution statuses."""
+    PENDING = "pending"
+    RUNNING = "running"
+    SUCCEEDED = "succeeded"
+    FAILED_RETRYING = "failed_retrying"
+    FAILED_PERMANENT = "failed_permanent"
+    CANCELLED = "cancelled"
+
+
+class TaskType(Enum):
+    """Enumeration of task types for provider queries."""
+    DOMAIN_QUERY = "domain_query"
+    IP_QUERY = "ip_query"
+    GRAPH_UPDATE = "graph_update"
+
+
+@dataclass
+class TaskResult:
+    """Result of a task execution."""
+    success: bool
+    data: Optional[Any] = None
+    error: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class ReconTask:
+    """Represents a single reconnaissance task with retry logic."""
+    task_id: str
+    task_type: TaskType
+    target: str
+    provider_name: str
+    depth: int
+    status: TaskStatus = TaskStatus.PENDING
+    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+    
+    # Retry configuration
+    max_retries: int = 3
+    current_retry: int = 0
+    base_backoff_seconds: float = 1.0
+    max_backoff_seconds: float = 60.0
+    
+    # Execution tracking
+    last_attempt_at: Optional[datetime] = None
+    next_retry_at: Optional[datetime] = None
+    execution_history: List[Dict[str, Any]] = field(default_factory=list)
+    
+    # Results
+    result: Optional[TaskResult] = None
+    
+    def __post_init__(self):
+        """Initialize additional fields after creation."""
+        if not self.task_id:
+            self.task_id = str(uuid.uuid4())[:8]
+    
+    def calculate_next_retry_time(self) -> datetime:
+        """Calculate next retry time with exponential backoff and jitter."""
+        if self.current_retry >= self.max_retries:
+            return None
+        
+        # Exponential backoff with jitter
+        backoff_time = min(
+            self.max_backoff_seconds,
+            self.base_backoff_seconds * (2 ** self.current_retry)
+        )
+        
+        # Add jitter (±25%)
+        jitter = backoff_time * 0.25 * (0.5 - hash(self.task_id) % 1000 / 1000.0)
+        final_backoff = max(self.base_backoff_seconds, backoff_time + jitter)
+        
+        return datetime.now(timezone.utc) + timedelta(seconds=final_backoff)
+    
+    def should_retry(self) -> bool:
+        """Determine if task should be retried based on status and retry count."""
+        if self.status != TaskStatus.FAILED_RETRYING:
+            return False
+        if self.current_retry >= self.max_retries:
+            return False
+        if self.next_retry_at and datetime.now(timezone.utc) < self.next_retry_at:
+            return False
+        return True
+    
+    def mark_failed(self, error: str, metadata: Dict[str, Any] = None):
+        """Mark task as failed and prepare for retry or permanent failure."""
+        self.current_retry += 1
+        self.last_attempt_at = datetime.now(timezone.utc)
+        
+        # Record execution history
+        execution_record = {
+            'attempt': self.current_retry,
+            'timestamp': self.last_attempt_at.isoformat(),
+            'error': error,
+            'metadata': metadata or {}
+        }
+        self.execution_history.append(execution_record)
+        
+        if self.current_retry >= self.max_retries:
+            self.status = TaskStatus.FAILED_PERMANENT
+            self.result = TaskResult(success=False, error=f"Permanent failure after {self.max_retries} attempts: {error}")
+        else:
+            self.status = TaskStatus.FAILED_RETRYING
+            self.next_retry_at = self.calculate_next_retry_time()
+    
+    def mark_succeeded(self, data: Any = None, metadata: Dict[str, Any] = None):
+        """Mark task as successfully completed."""
+        self.status = TaskStatus.SUCCEEDED
+        self.last_attempt_at = datetime.now(timezone.utc)
+        self.result = TaskResult(success=True, data=data, metadata=metadata or {})
+        
+        # Record successful execution
+        execution_record = {
+            'attempt': self.current_retry + 1,
+            'timestamp': self.last_attempt_at.isoformat(),
+            'success': True,
+            'metadata': metadata or {}
+        }
+        self.execution_history.append(execution_record)
+    
+    def get_summary(self) -> Dict[str, Any]:
+        """Get task summary for progress reporting."""
+        return {
+            'task_id': self.task_id,
+            'task_type': self.task_type.value,
+            'target': self.target,
+            'provider': self.provider_name,
+            'status': self.status.value,
+            'current_retry': self.current_retry,
+            'max_retries': self.max_retries,
+            'created_at': self.created_at.isoformat(),
+            'last_attempt_at': self.last_attempt_at.isoformat() if self.last_attempt_at else None,
+            'next_retry_at': self.next_retry_at.isoformat() if self.next_retry_at else None,
+            'total_attempts': len(self.execution_history),
+            'has_result': self.result is not None
+        }
+
+
+class TaskQueue:
+    """Thread-safe task queue with retry logic and priority handling."""
+    
+    def __init__(self, max_concurrent_tasks: int = 5):
+        """Initialize task queue."""
+        self.max_concurrent_tasks = max_concurrent_tasks
+        self.tasks: Dict[str, ReconTask] = {}
+        self.pending_queue = deque()
+        self.retry_queue = deque()
+        self.running_tasks: Set[str] = set()
+        
+        self._lock = threading.Lock()
+        self._stop_event = threading.Event()
+    
+    def __getstate__(self):
+        """Prepare TaskQueue for pickling by excluding unpicklable objects."""
+        state = self.__dict__.copy()
+        # Exclude the unpickleable '_lock' and '_stop_event' attributes
+        if '_lock' in state:
+            del state['_lock']
+        if '_stop_event' in state:
+            del state['_stop_event']
+        return state
+
+    def __setstate__(self, state):
+        """Restore TaskQueue after unpickling by reconstructing threading objects."""
+        self.__dict__.update(state)
+        # Re-initialize the '_lock' and '_stop_event' attributes
+        self._lock = threading.Lock()
+        self._stop_event = threading.Event()
+    
+    def add_task(self, task: ReconTask) -> str:
+        """Add task to queue."""
+        with self._lock:
+            self.tasks[task.task_id] = task
+            self.pending_queue.append(task.task_id)
+            print(f"Added task {task.task_id}: {task.provider_name} query for {task.target}")
+            return task.task_id
+    
+    def get_next_ready_task(self) -> Optional[ReconTask]:
+        """Get next task ready for execution."""
+        with self._lock:
+            # Check if we have room for more concurrent tasks
+            if len(self.running_tasks) >= self.max_concurrent_tasks:
+                return None
+            
+            # First priority: retry queue (tasks ready for retry)
+            while self.retry_queue:
+                task_id = self.retry_queue.popleft()
+                if task_id in self.tasks:
+                    task = self.tasks[task_id]
+                    if task.should_retry():
+                        task.status = TaskStatus.RUNNING
+                        self.running_tasks.add(task_id)
+                        print(f"Retrying task {task_id} (attempt {task.current_retry + 1})")
+                        return task
+            
+            # Second priority: pending queue (new tasks)
+            while self.pending_queue:
+                task_id = self.pending_queue.popleft()
+                if task_id in self.tasks:
+                    task = self.tasks[task_id]
+                    if task.status == TaskStatus.PENDING:
+                        task.status = TaskStatus.RUNNING
+                        self.running_tasks.add(task_id)
+                        print(f"Starting task {task_id}")
+                        return task
+            
+            return None
+    
+    def complete_task(self, task_id: str, success: bool, data: Any = None, 
+                     error: str = None, metadata: Dict[str, Any] = None):
+        """Mark task as completed (success or failure)."""
+        with self._lock:
+            if task_id not in self.tasks:
+                return
+            
+            task = self.tasks[task_id]
+            self.running_tasks.discard(task_id)
+            
+            if success:
+                task.mark_succeeded(data=data, metadata=metadata)
+                print(f"Task {task_id} succeeded")
+            else:
+                task.mark_failed(error or "Unknown error", metadata=metadata)
+                if task.status == TaskStatus.FAILED_RETRYING:
+                    self.retry_queue.append(task_id)
+                    print(f"Task {task_id} failed, scheduled for retry at {task.next_retry_at}")
+                else:
+                    print(f"Task {task_id} permanently failed after {task.current_retry} attempts")
+    
+    def cancel_all_tasks(self):
+        """Cancel all pending and running tasks."""
+        with self._lock:
+            self._stop_event.set()
+            for task in self.tasks.values():
+                if task.status in [TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.FAILED_RETRYING]:
+                    task.status = TaskStatus.CANCELLED
+            self.pending_queue.clear()
+            self.retry_queue.clear()
+            self.running_tasks.clear()
+            print("All tasks cancelled")
+    
+    def is_complete(self) -> bool:
+        """Check if all tasks are complete (succeeded, permanently failed, or cancelled)."""
+        with self._lock:
+            for task in self.tasks.values():
+                if task.status in [TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.FAILED_RETRYING]:
+                    return False
+            return True
+    
+    def get_statistics(self) -> Dict[str, Any]:
+        """Get queue statistics."""
+        with self._lock:
+            stats = {
+                'total_tasks': len(self.tasks),
+                'pending': len(self.pending_queue),
+                'running': len(self.running_tasks),
+                'retry_queue': len(self.retry_queue),
+                'succeeded': 0,
+                'failed_permanent': 0,
+                'cancelled': 0,
+                'failed_retrying': 0
+            }
+            
+            for task in self.tasks.values():
+                if task.status == TaskStatus.SUCCEEDED:
+                    stats['succeeded'] += 1
+                elif task.status == TaskStatus.FAILED_PERMANENT:
+                    stats['failed_permanent'] += 1
+                elif task.status == TaskStatus.CANCELLED:
+                    stats['cancelled'] += 1
+                elif task.status == TaskStatus.FAILED_RETRYING:
+                    stats['failed_retrying'] += 1
+            
+            stats['completion_rate'] = (stats['succeeded'] / stats['total_tasks'] * 100) if stats['total_tasks'] > 0 else 0
+            stats['is_complete'] = self.is_complete()
+            
+            return stats
+    
+    def get_task_summaries(self) -> List[Dict[str, Any]]:
+        """Get summaries of all tasks for detailed progress reporting."""
+        with self._lock:
+            return [task.get_summary() for task in self.tasks.values()]
+    
+    def get_failed_tasks(self) -> List[ReconTask]:
+        """Get all permanently failed tasks for analysis."""
+        with self._lock:
+            return [task for task in self.tasks.values() if task.status == TaskStatus.FAILED_PERMANENT]
+
+
+class TaskExecutor:
+    """Executes reconnaissance tasks using providers."""
+    
+    def __init__(self, providers: List, graph_manager, logger):
+        """Initialize task executor."""
+        self.providers = {provider.get_name(): provider for provider in providers}
+        self.graph = graph_manager
+        self.logger = logger
+    
+    def execute_task(self, task: ReconTask) -> TaskResult:
+        """
+        Execute a single reconnaissance task.
+        
+        Args:
+            task: Task to execute
+            
+        Returns:
+            TaskResult with success/failure information
+        """
+        try:
+            print(f"Executing task {task.task_id}: {task.provider_name} query for {task.target}")
+            
+            provider = self.providers.get(task.provider_name)
+            if not provider:
+                return TaskResult(
+                    success=False, 
+                    error=f"Provider {task.provider_name} not available"
+                )
+            
+            if not provider.is_available():
+                return TaskResult(
+                    success=False,
+                    error=f"Provider {task.provider_name} is not available (missing API key or configuration)"
+                )
+            
+            # Execute provider query based on task type
+            if task.task_type == TaskType.DOMAIN_QUERY:
+                if not _is_valid_domain(task.target):
+                    return TaskResult(success=False, error=f"Invalid domain: {task.target}")
+                    
+                relationships = provider.query_domain(task.target)
+                
+            elif task.task_type == TaskType.IP_QUERY:
+                if not _is_valid_ip(task.target):
+                    return TaskResult(success=False, error=f"Invalid IP: {task.target}")
+                    
+                relationships = provider.query_ip(task.target)
+                
+            else:
+                return TaskResult(success=False, error=f"Unsupported task type: {task.task_type}")
+            
+            # Process results and update graph
+            new_targets = set()
+            relationships_added = 0
+            
+            for source, target, rel_type, confidence, raw_data in relationships:
+                # Add nodes to graph
+                from core.graph_manager import NodeType
+                
+                if _is_valid_ip(target):
+                    self.graph.add_node(target, NodeType.IP)
+                    new_targets.add(target)
+                elif target.startswith('AS') and target[2:].isdigit():
+                    self.graph.add_node(target, NodeType.ASN)
+                elif _is_valid_domain(target):
+                    self.graph.add_node(target, NodeType.DOMAIN)
+                    new_targets.add(target)
+                
+                # Add edge to graph
+                if self.graph.add_edge(source, target, rel_type, confidence, task.provider_name, raw_data):
+                    relationships_added += 1
+            
+            # Log forensic information
+            self.logger.logger.info(
+                f"Task {task.task_id} completed: {len(relationships)} relationships found, "
+                f"{relationships_added} added to graph, {len(new_targets)} new targets"
+            )
+            
+            return TaskResult(
+                success=True,
+                data={
+                    'relationships': relationships,
+                    'new_targets': list(new_targets),
+                    'relationships_added': relationships_added
+                },
+                metadata={
+                    'provider': task.provider_name,
+                    'target': task.target,
+                    'depth': task.depth,
+                    'execution_time': datetime.now(timezone.utc).isoformat()
+                }
+            )
+            
+        except Exception as e:
+            error_msg = f"Task execution failed: {str(e)}"
+            print(f"ERROR: {error_msg} for task {task.task_id}")
+            self.logger.logger.error(error_msg)
+            
+            return TaskResult(
+                success=False,
+                error=error_msg,
+                metadata={
+                    'provider': task.provider_name,
+                    'target': task.target,
+                    'exception_type': type(e).__name__
+                }
+            )
+
+
+class TaskManager:
+    """High-level task management for reconnaissance scans."""
+    
+    def __init__(self, providers: List, graph_manager, logger, max_concurrent_tasks: int = 5):
+        """Initialize task manager."""
+        self.task_queue = TaskQueue(max_concurrent_tasks)
+        self.task_executor = TaskExecutor(providers, graph_manager, logger)
+        self.logger = logger
+        
+        # Execution control
+        self._stop_event = threading.Event()
+        self._execution_threads: List[threading.Thread] = []
+        self._is_running = False
+    
+    def create_provider_tasks(self, target: str, depth: int, providers: List) -> List[str]:
+        """
+        Create tasks for querying all eligible providers for a target.
+        
+        Args:
+            target: Domain or IP to query
+            depth: Current recursion depth
+            providers: List of available providers
+            
+        Returns:
+            List of created task IDs
+        """
+        task_ids = []
+        is_ip = _is_valid_ip(target)
+        target_key = 'ips' if is_ip else 'domains'
+        task_type = TaskType.IP_QUERY if is_ip else TaskType.DOMAIN_QUERY
+        
+        for provider in providers:
+            if provider.get_eligibility().get(target_key) and provider.is_available():
+                task = ReconTask(
+                    task_id=str(uuid.uuid4())[:8],
+                    task_type=task_type,
+                    target=target,
+                    provider_name=provider.get_name(),
+                    depth=depth,
+                    max_retries=3  # Configure retries per task type/provider
+                )
+                
+                task_id = self.task_queue.add_task(task)
+                task_ids.append(task_id)
+        
+        return task_ids
+    
+    def start_execution(self, max_workers: int = 3):
+        """Start task execution with specified number of worker threads."""
+        if self._is_running:
+            print("Task execution already running")
+            return
+        
+        self._is_running = True
+        self._stop_event.clear()
+        
+        print(f"Starting task execution with {max_workers} workers")
+        
+        for i in range(max_workers):
+            worker_thread = threading.Thread(
+                target=self._worker_loop,
+                name=f"TaskWorker-{i+1}",
+                daemon=True
+            )
+            worker_thread.start()
+            self._execution_threads.append(worker_thread)
+    
+    def stop_execution(self):
+        """Stop task execution and cancel all tasks."""
+        print("Stopping task execution")
+        self._stop_event.set()
+        self.task_queue.cancel_all_tasks()
+        self._is_running = False
+        
+        # Wait for worker threads to finish
+        for thread in self._execution_threads:
+            thread.join(timeout=5.0)
+        
+        self._execution_threads.clear()
+        print("Task execution stopped")
+    
+    def _worker_loop(self):
+        """Worker thread loop for executing tasks."""
+        thread_name = threading.current_thread().name
+        print(f"{thread_name} started")
+        
+        while not self._stop_event.is_set():
+            try:
+                # Get next task to execute
+                task = self.task_queue.get_next_ready_task()
+                
+                if task is None:
+                    # No tasks ready, check if we should exit
+                    if self.task_queue.is_complete() or self._stop_event.is_set():
+                        break
+                    time.sleep(0.1)  # Brief sleep before checking again
+                    continue
+                
+                # Execute the task
+                result = self.task_executor.execute_task(task)
+                
+                # Complete the task in queue
+                self.task_queue.complete_task(
+                    task.task_id,
+                    success=result.success,
+                    data=result.data,
+                    error=result.error,
+                    metadata=result.metadata
+                )
+                
+            except Exception as e:
+                print(f"ERROR: Worker {thread_name} encountered error: {e}")
+                # Continue running even if individual task fails
+                continue
+        
+        print(f"{thread_name} finished")
+    
+    def wait_for_completion(self, timeout_seconds: int = 300) -> bool:
+        """
+        Wait for all tasks to complete.
+        
+        Args:
+            timeout_seconds: Maximum time to wait
+            
+        Returns:
+            True if all tasks completed, False if timeout
+        """
+        start_time = time.time()
+        
+        while time.time() - start_time < timeout_seconds:
+            if self.task_queue.is_complete():
+                return True
+            
+            if self._stop_event.is_set():
+                return False
+            
+            time.sleep(1.0)  # Check every second
+        
+        print(f"Timeout waiting for task completion after {timeout_seconds} seconds")
+        return False
+    
+    def get_progress_report(self) -> Dict[str, Any]:
+        """Get detailed progress report for UI updates."""
+        stats = self.task_queue.get_statistics()
+        failed_tasks = self.task_queue.get_failed_tasks()
+        
+        return {
+            'statistics': stats,
+            'failed_tasks': [task.get_summary() for task in failed_tasks],
+            'is_running': self._is_running,
+            'worker_count': len(self._execution_threads),
+            'detailed_tasks': self.task_queue.get_task_summaries() if stats['total_tasks'] < 50 else []  # Limit detail for performance
+        }

providers/__init__.py

@@ -7,15 +7,13 @@ from .base_provider import BaseProvider, RateLimiter
 from .crtsh_provider import CrtShProvider
 from .dns_provider import DNSProvider
 from .shodan_provider import ShodanProvider
-from .virustotal_provider import VirusTotalProvider
 
 __all__ = [
     'BaseProvider',
     'RateLimiter', 
     'CrtShProvider',
     'DNSProvider',
-    'ShodanProvider',
-    'VirusTotalProvider'
+    'ShodanProvider'
 ]
 
-__version__ = "1.0.0-phase2"
+__version__ = "0.0.0-rc"

providers/base_provider.py

@@ -5,15 +5,16 @@ import requests
 import threading
 import os
 import json
+import hashlib
 from abc import ABC, abstractmethod
 from typing import List, Dict, Any, Optional, Tuple
+from datetime import datetime, timezone
 
 from core.logger import get_forensic_logger
-from core.graph_manager import RelationshipType
 
 
 class RateLimiter:
-    """Simple rate limiter for API calls."""
+    """Thread-safe rate limiter for API calls."""
 
     def __init__(self, requests_per_minute: int):
         """
@@ -25,28 +26,152 @@ class RateLimiter:
         self.requests_per_minute = requests_per_minute
         self.min_interval = 60.0 / requests_per_minute
         self.last_request_time = 0
+        self._lock = threading.Lock()
+
+    def __getstate__(self):
+        """RateLimiter is fully picklable, return full state."""
+        state = self.__dict__.copy()
+        # Exclude unpickleable lock
+        if '_lock' in state:
+            del state['_lock']
+        return state
+
+    def __setstate__(self, state):
+        """Restore RateLimiter state."""
+        self.__dict__.update(state)
+        self._lock = threading.Lock()
 
     def wait_if_needed(self) -> None:
         """Wait if necessary to respect rate limits."""
-        current_time = time.time()
-        time_since_last = current_time - self.last_request_time
+        with self._lock:
+            current_time = time.time()
+            time_since_last = current_time - self.last_request_time
 
-        if time_since_last < self.min_interval:
-            sleep_time = self.min_interval - time_since_last
-            time.sleep(sleep_time)
+            if time_since_last < self.min_interval:
+                sleep_time = self.min_interval - time_since_last
+                time.sleep(sleep_time)
 
-        self.last_request_time = time.time()
+            self.last_request_time = time.time()
+
+
+class ProviderCache:
+    """Thread-safe global cache for provider queries."""
+    
+    def __init__(self, provider_name: str, cache_expiry_hours: int = 12):
+        """
+        Initialize provider-specific cache.
+        
+        Args:
+            provider_name: Name of the provider for cache directory
+            cache_expiry_hours: Cache expiry time in hours
+        """
+        self.provider_name = provider_name
+        self.cache_expiry = cache_expiry_hours * 3600  # Convert to seconds
+        self.cache_dir = os.path.join('.cache', provider_name)
+        self._lock = threading.Lock()
+        
+        # Ensure cache directory exists with thread-safe creation
+        os.makedirs(self.cache_dir, exist_ok=True)
+        
+    def _generate_cache_key(self, method: str, url: str, params: Optional[Dict[str, Any]]) -> str:
+        """Generate unique cache key for request."""
+        cache_data = f"{method}:{url}:{json.dumps(params or {}, sort_keys=True)}"
+        return hashlib.md5(cache_data.encode()).hexdigest() + ".json"
+    
+    def get_cached_response(self, method: str, url: str, params: Optional[Dict[str, Any]]) -> Optional[requests.Response]:
+        """
+        Retrieve cached response if available and not expired.
+        
+        Returns:
+            Cached Response object or None if cache miss/expired
+        """
+        cache_key = self._generate_cache_key(method, url, params)
+        cache_path = os.path.join(self.cache_dir, cache_key)
+        
+        with self._lock:
+            if not os.path.exists(cache_path):
+                return None
+                
+            # Check if cache is expired
+            cache_age = time.time() - os.path.getmtime(cache_path)
+            if cache_age >= self.cache_expiry:
+                try:
+                    os.remove(cache_path)
+                except OSError:
+                    pass  # File might have been removed by another thread
+                return None
+            
+            try:
+                with open(cache_path, 'r', encoding='utf-8') as f:
+                    cached_data = json.load(f)
+                    
+                # Reconstruct Response object
+                response = requests.Response()
+                response.status_code = cached_data['status_code']
+                response._content = cached_data['content'].encode('utf-8')
+                response.headers.update(cached_data['headers'])
+                
+                return response
+                
+            except (json.JSONDecodeError, KeyError, IOError) as e:
+                # Cache file corrupted, remove it
+                try:
+                    os.remove(cache_path)
+                except OSError:
+                    pass
+                return None
+    
+    def cache_response(self, method: str, url: str, params: Optional[Dict[str, Any]], 
+                      response: requests.Response) -> bool:
+        """
+        Cache successful response to disk.
+        
+        Returns:
+            True if cached successfully, False otherwise
+        """
+        if response.status_code != 200:
+            return False
+            
+        cache_key = self._generate_cache_key(method, url, params)
+        cache_path = os.path.join(self.cache_dir, cache_key)
+        
+        with self._lock:
+            try:
+                cache_data = {
+                    'status_code': response.status_code,
+                    'content': response.text,
+                    'headers': dict(response.headers),
+                    'cached_at': datetime.now(timezone.utc).isoformat()
+                }
+                
+                # Write to temporary file first, then rename for atomic operation
+                temp_path = cache_path + '.tmp'
+                with open(temp_path, 'w', encoding='utf-8') as f:
+                    json.dump(cache_data, f)
+                
+                # Atomic rename to prevent partial cache files
+                os.rename(temp_path, cache_path)
+                return True
+                
+            except (IOError, OSError) as e:
+                # Clean up temp file if it exists
+                try:
+                    if os.path.exists(temp_path):
+                        os.remove(temp_path)
+                except OSError:
+                    pass
+                return False
 
 
 class BaseProvider(ABC):
     """
     Abstract base class for all DNSRecon data providers.
-    Now supports session-specific configuration.
+    Now supports global provider-specific caching and session-specific configuration.
     """
 
     def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30, session_config=None):
         """
-        Initialize base provider with session-specific configuration.
+        Initialize base provider with global caching and session-specific configuration.
 
         Args:
             name: Provider name for logging
@@ -73,26 +198,40 @@ class BaseProvider(ABC):
         self.logger = get_forensic_logger()
         self._stop_event = None
 
-        # Caching configuration (per session)
-        self.cache_dir = f'.cache/{id(self.config)}'  # Unique cache per session config
-        self.cache_expiry = 12 * 3600  # 12 hours in seconds
-        if not os.path.exists(self.cache_dir):
-            os.makedirs(self.cache_dir)
+        # GLOBAL provider-specific caching (not session-based)
+        self.cache = ProviderCache(name, cache_expiry_hours=12)
 
         # Statistics (per provider instance)
         self.total_requests = 0
         self.successful_requests = 0
         self.failed_requests = 0
         self.total_relationships_found = 0
+        self.cache_hits = 0
+        self.cache_misses = 0
 
-        print(f"Initialized {name} provider with session-specific config (rate: {actual_rate_limit}/min)")
+        print(f"Initialized {name} provider with global cache and session config (rate: {actual_rate_limit}/min)")
+
+    def __getstate__(self):
+        """Prepare BaseProvider for pickling by excluding unpicklable objects."""
+        state = self.__dict__.copy()
+        # Exclude the unpickleable '_local' attribute and stop event
+        state['_local'] = None
+        state['_stop_event'] = None
+        return state
+
+    def __setstate__(self, state):
+        """Restore BaseProvider after unpickling by reconstructing threading objects."""
+        self.__dict__.update(state)
+        # Re-initialize the '_local' attribute and stop event
+        self._local = threading.local()
+        self._stop_event = None
 
     @property
     def session(self):
         if not hasattr(self._local, 'session'):
             self._local.session = requests.Session()
             self._local.session.headers.update({
-                'User-Agent': 'DNSRecon/1.0 (Passive Reconnaissance Tool)'
+                'User-Agent': 'DNSRecon/2.0 (Passive Reconnaissance Tool)'
             })
         return self._local.session
 
@@ -101,13 +240,28 @@ class BaseProvider(ABC):
         """Return the provider name."""
         pass
 
+    @abstractmethod
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        pass
+
+    @abstractmethod
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        pass
+
+    @abstractmethod
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        pass
+
     @abstractmethod
     def is_available(self) -> bool:
         """Check if the provider is available and properly configured."""
         pass
 
     @abstractmethod
-    def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_domain(self, domain: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query the provider for information about a domain.
 
@@ -120,7 +274,7 @@ class BaseProvider(ABC):
         pass
 
     @abstractmethod
-    def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_ip(self, ip: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query the provider for information about an IP address.
 
@@ -138,46 +292,40 @@ class BaseProvider(ABC):
                     target_indicator: str = "",
                     max_retries: int = 3) -> Optional[requests.Response]:
         """
-        Make a rate-limited HTTP request with forensic logging and retry logic.
-        Now supports cancellation via stop_event from scanner.
+        Make a rate-limited HTTP request with global caching and aggressive stop signal handling.
         """
         # Check for cancellation before starting
-        if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
+        if self._is_stop_requested():
             print(f"Request cancelled before start: {url}")
             return None
 
-        # Create a unique cache key
-        cache_key = f"{self.name}_{hash(f'{method}:{url}:{json.dumps(params, sort_keys=True)}')}.json"
-        cache_path = os.path.join(self.cache_dir, cache_key)
+        # Check global cache first
+        cached_response = self.cache.get_cached_response(method, url, params)
+        if cached_response is not None:
+            print(f"Cache hit for {self.name}: {url}")
+            self.cache_hits += 1
+            return cached_response
+        
+        self.cache_misses += 1
 
-        # Check cache
-        if os.path.exists(cache_path):
-            cache_age = time.time() - os.path.getmtime(cache_path)
-            if cache_age < self.cache_expiry:
-                print(f"Returning cached response for: {url}")
-                with open(cache_path, 'r') as f:
-                    cached_data = json.load(f)
-                    response = requests.Response()
-                    response.status_code = cached_data['status_code']
-                    response._content = cached_data['content'].encode('utf-8')
-                    response.headers = cached_data['headers']
-                    return response
+        # Determine effective max_retries based on stop signal
+        effective_max_retries = 0 if self._is_stop_requested() else max_retries
+        last_exception = None
 
-        for attempt in range(max_retries + 1):
+        for attempt in range(effective_max_retries + 1):
             # Check for cancellation before each attempt
-            if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
+            if self._is_stop_requested():
                 print(f"Request cancelled during attempt {attempt + 1}: {url}")
                 return None
 
-            # Apply rate limiting (but reduce wait time if cancellation is requested)
-            if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                break
-            
-            self.rate_limiter.wait_if_needed()
+            # Apply rate limiting with cancellation awareness
+            if not self._wait_with_cancellation_check():
+                print(f"Request cancelled during rate limiting: {url}")
+                return None
 
-            # Check again after rate limiting
-            if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                print(f"Request cancelled after rate limiting: {url}")
+            # Final check before making HTTP request
+            if self._is_stop_requested():
+                print(f"Request cancelled before HTTP call: {url}")
                 return None
 
             start_time = time.time()
@@ -195,9 +343,7 @@ class BaseProvider(ABC):
                 print(f"Making {method} request to: {url} (attempt {attempt + 1})")
 
                 # Use shorter timeout if termination is requested
-                request_timeout = self.timeout
-                if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                    request_timeout = min(5, self.timeout)  # Max 5 seconds if termination requested
+                request_timeout = 2 if self._is_stop_requested() else self.timeout
 
                 # Make request
                 if method.upper() == "GET":
@@ -233,41 +379,35 @@ class BaseProvider(ABC):
                     error=None,
                     target_indicator=target_indicator
                 )
-                # Cache the successful response to disk
-                with open(cache_path, 'w') as f:
-                    json.dump({
-                        'status_code': response.status_code,
-                        'content': response.text,
-                        'headers': dict(response.headers)
-                    }, f)
+                
+                # Cache the successful response globally
+                self.cache.cache_response(method, url, params, response)
                 return response
 
             except requests.exceptions.RequestException as e:
                 error = str(e)
                 self.failed_requests += 1
                 print(f"Request failed (attempt {attempt + 1}): {error}")
+                last_exception = e
                 
-                # Check for cancellation before retrying
-                if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                    print(f"Request cancelled, not retrying: {url}")
+                # Immediately abort retries if stop requested
+                if self._is_stop_requested():
+                    print(f"Stop requested - aborting retries for: {url}")
                     break
                 
                 # Check if we should retry
-                if attempt < max_retries and self._should_retry(e):
-                    backoff_time = (2 ** attempt) * 1  # Exponential backoff: 1s, 2s, 4s
-                    print(f"Retrying in {backoff_time} seconds...")
+                if attempt < effective_max_retries and self._should_retry(e):
+                    # Exponential backoff with jitter for 429 errors
+                    if isinstance(e, requests.exceptions.HTTPError) and e.response and e.response.status_code == 429:
+                        backoff_time = min(60, 10 * (2 ** attempt))
+                        print(f"Rate limit hit. Retrying in {backoff_time} seconds...")
+                    else:
+                        backoff_time = min(2.0, (2 ** attempt) * 0.5)
+                        print(f"Retrying in {backoff_time} seconds...")
                     
-                    # Shorter backoff if termination is requested
-                    if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                        backoff_time = min(0.5, backoff_time)
-                    
-                    # Sleep with cancellation checking
-                    sleep_start = time.time()
-                    while time.time() - sleep_start < backoff_time:
-                        if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
-                            print(f"Request cancelled during backoff: {url}")
-                            return None
-                        time.sleep(0.1)  # Check every 100ms
+                    if not self._sleep_with_cancellation_check(backoff_time):
+                        print(f"Stop requested during backoff - aborting: {url}")
+                        return None
                     continue
                 else:
                     break
@@ -276,6 +416,7 @@ class BaseProvider(ABC):
                 error = f"Unexpected error: {str(e)}"
                 self.failed_requests += 1
                 print(f"Unexpected error: {error}")
+                last_exception = e
                 break
 
         # All attempts failed - log and return None
@@ -291,8 +432,56 @@ class BaseProvider(ABC):
             target_indicator=target_indicator
         )
         
+        if error and last_exception:
+            raise last_exception
+        
         return None
 
+    def _is_stop_requested(self) -> bool:
+        """
+        Enhanced stop signal checking that handles both local and Redis-based signals.
+        """
+        if hasattr(self, '_stop_event') and self._stop_event and self._stop_event.is_set():
+            return True
+        return False
+
+    def _wait_with_cancellation_check(self) -> bool:
+        """
+        Wait for rate limiting while aggressively checking for cancellation.
+        Returns False if cancelled during wait.
+        """
+        current_time = time.time()
+        time_since_last = current_time - self.rate_limiter.last_request_time
+
+        if time_since_last < self.rate_limiter.min_interval:
+            sleep_time = self.rate_limiter.min_interval - time_since_last
+            if not self._sleep_with_cancellation_check(sleep_time):
+                return False
+
+        self.rate_limiter.last_request_time = time.time()
+        return True
+
+    def _sleep_with_cancellation_check(self, sleep_time: float) -> bool:
+        """
+        Sleep for the specified time while aggressively checking for cancellation.
+        
+        Args:
+            sleep_time: Time to sleep in seconds
+            
+        Returns:
+            bool: True if sleep completed, False if cancelled
+        """
+        sleep_start = time.time()
+        check_interval = 0.05  # Check every 50ms for aggressive responsiveness
+        
+        while time.time() - sleep_start < sleep_time:
+            if self._is_stop_requested():
+                return False
+            remaining_time = sleep_time - (time.time() - sleep_start)
+            time.sleep(min(check_interval, remaining_time))
+        
+        return True
+
     def set_stop_event(self, stop_event: threading.Event) -> None:
         """
         Set the stop event for this provider to enable cancellation.
@@ -312,20 +501,20 @@ class BaseProvider(ABC):
         Returns:
             True if the request should be retried
         """
-        # Retry on connection errors, timeouts, and 5xx server errors
+        # Retry on connection errors and timeouts
         if isinstance(exception, (requests.exceptions.ConnectionError, 
                                 requests.exceptions.Timeout)):
             return True
         
         if isinstance(exception, requests.exceptions.HTTPError):
             if hasattr(exception, 'response') and exception.response:
-                # Retry on server errors (5xx) but not client errors (4xx)
-                return exception.response.status_code >= 500
+                # Retry on server errors (5xx) AND on rate-limiting errors (429)
+                return exception.response.status_code >= 500 or exception.response.status_code == 429
         
         return False
 
     def log_relationship_discovery(self, source_node: str, target_node: str,
-                                 relationship_type: RelationshipType,
+                                 relationship_type: str,
                                  confidence_score: float,
                                  raw_data: Dict[str, Any],
                                  discovery_method: str) -> None:
@@ -345,7 +534,7 @@ class BaseProvider(ABC):
         self.logger.log_relationship_discovery(
             source_node=source_node,
             target_node=target_node,
-            relationship_type=relationship_type.relationship_name,
+            relationship_type=relationship_type,
             confidence_score=confidence_score,
             provider=self.name,
             raw_data=raw_data,
@@ -354,7 +543,7 @@ class BaseProvider(ABC):
 
     def get_statistics(self) -> Dict[str, Any]:
         """
-        Get provider statistics.
+        Get provider statistics including cache performance.
 
         Returns:
             Dictionary containing provider performance metrics
@@ -366,5 +555,8 @@ class BaseProvider(ABC):
             'failed_requests': self.failed_requests,
             'success_rate': (self.successful_requests / self.total_requests * 100) if self.total_requests > 0 else 0,
             'relationships_found': self.total_relationships_found,
-            'rate_limit': self.rate_limiter.requests_per_minute
+            'rate_limit': self.rate_limiter.requests_per_minute,
+            'cache_hits': self.cache_hits,
+            'cache_misses': self.cache_misses,
+            'cache_hit_rate': (self.cache_hits / (self.cache_hits + self.cache_misses) * 100) if (self.cache_hits + self.cache_misses) > 0 else 0
         }

providers/crtsh_provider.py

@@ -9,10 +9,10 @@ import re
 from typing import List, Dict, Any, Tuple, Set
 from urllib.parse import quote
 from datetime import datetime, timezone
+import requests
 
 from .base_provider import BaseProvider
 from utils.helpers import _is_valid_domain
-from core.graph_manager import RelationshipType
 
 
 class CrtShProvider(BaseProvider):
@@ -36,6 +36,18 @@ class CrtShProvider(BaseProvider):
         """Return the provider name."""
         return "crtsh"
     
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "crt.sh"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': False}
+
     def is_available(self) -> bool:
         """
         Check if the provider is configured to be used.
@@ -133,7 +145,6 @@ class CrtShProvider(BaseProvider):
             'source': 'crt.sh'
         }
         
-        # Add computed fields
         try:
             if metadata['not_before'] and metadata['not_after']:
                 not_before = self._parse_certificate_date(metadata['not_before'])
@@ -144,8 +155,8 @@ class CrtShProvider(BaseProvider):
                 metadata['expires_soon'] = (not_after - datetime.now(timezone.utc)).days <= 30
                 
                 # Add human-readable dates
-                metadata['not_before_formatted'] = not_before.strftime('%Y-%m-%d %H:%M:%S UTC')
-                metadata['not_after_formatted'] = not_after.strftime('%Y-%m-%d %H:%M:%S UTC')
+                metadata['not_before'] = not_before.strftime('%Y-%m-%d %H:%M:%S UTC')
+                metadata['not_after'] = not_after.strftime('%Y-%m-%d %H:%M:%S UTC')
                 
         except Exception as e:
             self.logger.logger.debug(f"Error computing certificate metadata: {e}")
@@ -154,11 +165,9 @@ class CrtShProvider(BaseProvider):
         
         return metadata
 
-    def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_domain(self, domain: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query crt.sh for certificates containing the domain.
-        Creates domain-to-domain relationships and stores certificate data as metadata.
-        Now supports early termination via stop_event.
         """
         if not _is_valid_domain(domain):
             return []
@@ -173,7 +182,7 @@ class CrtShProvider(BaseProvider):
         try:
             # Query crt.sh for certificates
             url = f"{self.base_url}?q={quote(domain)}&output=json"
-            response = self.make_request(url, target_indicator=domain, max_retries=1)  # Reduce retries for faster cancellation
+            response = self.make_request(url, target_indicator=domain, max_retries=3)
             
             if not response or response.status_code != 200:
                 return []
@@ -197,10 +206,10 @@ class CrtShProvider(BaseProvider):
             domain_certificates = {}
             all_discovered_domains = set()
             
-            # Process certificates and group by domain (with cancellation checks)
+            # Process certificates with cancellation checking
             for i, cert_data in enumerate(certificates):
-                # Check for cancellation every 10 certificates
-                if i % 10 == 0 and self._stop_event and self._stop_event.is_set():
+                # Check for cancellation every 5 certificates instead of 10 for faster response
+                if i % 5 == 0 and self._stop_event and self._stop_event.is_set():
                     print(f"CrtSh processing cancelled at certificate {i} for domain: {domain}")
                     break
                     
@@ -209,6 +218,11 @@ class CrtShProvider(BaseProvider):
                 
                 # Add all domains from this certificate to our tracking
                 for cert_domain in cert_domains:
+                    # Additional stop check during domain processing
+                    if i % 20 == 0 and self._stop_event and self._stop_event.is_set():
+                        print(f"CrtSh domain processing cancelled for domain: {domain}")
+                        break
+                        
                     if not _is_valid_domain(cert_domain):
                         continue
                     
@@ -226,13 +240,13 @@ class CrtShProvider(BaseProvider):
                 print(f"CrtSh query cancelled before relationship creation for domain: {domain}")
                 return []
 
-            # Create relationships from query domain to ALL discovered domains
-            for discovered_domain in all_discovered_domains:
+            # Create relationships from query domain to ALL discovered domains with stop checking
+            for i, discovered_domain in enumerate(all_discovered_domains):
                 if discovered_domain == domain:
                     continue  # Skip self-relationships
                 
-                # Check for cancellation during relationship creation
-                if self._stop_event and self._stop_event.is_set():
+                # Check for cancellation every 10 relationships
+                if i % 10 == 0 and self._stop_event and self._stop_event.is_set():
                     print(f"CrtSh relationship creation cancelled for domain: {domain}")
                     break
 
@@ -267,7 +281,7 @@ class CrtShProvider(BaseProvider):
                 relationships.append((
                     domain,
                     discovered_domain,
-                    RelationshipType.SAN_CERTIFICATE,
+                    'san_certificate',
                     confidence,
                     relationship_raw_data
                 ))
@@ -276,7 +290,7 @@ class CrtShProvider(BaseProvider):
                 self.log_relationship_discovery(
                     source_node=domain,
                     target_node=discovered_domain,
-                    relationship_type=RelationshipType.SAN_CERTIFICATE,
+                    relationship_type='san_certificate',
                     confidence_score=confidence,
                     raw_data=relationship_raw_data,
                     discovery_method="certificate_transparency_analysis"
@@ -284,8 +298,9 @@ class CrtShProvider(BaseProvider):
 
         except json.JSONDecodeError as e:
             self.logger.logger.error(f"Failed to parse JSON response from crt.sh: {e}")
-        except Exception as e:
-            self.logger.logger.error(f"Error querying crt.sh for {domain}: {e}")
+        except requests.exceptions.RequestException as e:
+            self.logger.logger.error(f"HTTP request to crt.sh failed: {e}")
+
         
         return relationships
 
@@ -380,7 +395,7 @@ class CrtShProvider(BaseProvider):
         Returns:
             Confidence score between 0.0 and 1.0
         """
-        base_confidence = RelationshipType.SAN_CERTIFICATE.default_confidence
+        base_confidence = 0.9
         
         # Adjust confidence based on domain relationship context
         relationship_context = self._determine_relationship_context(domain2, domain1)
@@ -448,7 +463,7 @@ class CrtShProvider(BaseProvider):
         else:
             return 'related_domain'
     
-    def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_ip(self, ip: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query crt.sh for certificates containing the IP address.
         Note: crt.sh doesn't typically index by IP, so this returns empty results.
@@ -478,57 +493,56 @@ class CrtShProvider(BaseProvider):
         common_name = cert_data.get('common_name', '')
         if common_name:
             cleaned_cn = self._clean_domain_name(common_name)
-            if cleaned_cn and _is_valid_domain(cleaned_cn):
-                domains.add(cleaned_cn)
+            if cleaned_cn:
+                domains.update(cleaned_cn)
         
         # Extract from name_value field (contains SANs)
         name_value = cert_data.get('name_value', '')
         if name_value:
             # Split by newlines and clean each domain
             for line in name_value.split('\n'):
-                cleaned_domain = self._clean_domain_name(line.strip())
-                if cleaned_domain and _is_valid_domain(cleaned_domain):
-                    domains.add(cleaned_domain)
+                cleaned_domains = self._clean_domain_name(line.strip())
+                if cleaned_domains:
+                    domains.update(cleaned_domains)
         
         return domains
     
-    def _clean_domain_name(self, domain_name: str) -> str:
+    def _clean_domain_name(self, domain_name: str) -> List[str]:
         """
         Clean and normalize domain name from certificate data.
-        
-        Args:
-            domain_name: Raw domain name from certificate
-            
-        Returns:
-            Cleaned domain name or empty string if invalid
+        Now returns a list to handle wildcards correctly.
         """
         if not domain_name:
-            return ""
-        
-        # Remove common prefixes and clean up
+            return []
+
         domain = domain_name.strip().lower()
-        
+
         # Remove protocol if present
         if domain.startswith(('http://', 'https://')):
             domain = domain.split('://', 1)[1]
-        
+
         # Remove path if present
         if '/' in domain:
             domain = domain.split('/', 1)[0]
-        
+
         # Remove port if present
         if ':' in domain and not domain.count(':') > 1:  # Avoid breaking IPv6
             domain = domain.split(':', 1)[0]
-        
+
         # Handle wildcard domains
+        cleaned_domains = []
         if domain.startswith('*.'):
-            domain = domain[2:]
-        
-        # Remove any remaining invalid characters
-        domain = re.sub(r'[^\w\-\.]', '', domain)
-        
-        # Ensure it's not empty and doesn't start/end with dots or hyphens
-        if domain and not domain.startswith(('.', '-')) and not domain.endswith(('.', '-')):
-            return domain
-        
-        return ""
+            # Add both the wildcard and the base domain
+            cleaned_domains.append(domain)
+            cleaned_domains.append(domain[2:])
+        else:
+            cleaned_domains.append(domain)
+
+        # Remove any remaining invalid characters and validate
+        final_domains = []
+        for d in cleaned_domains:
+            d = re.sub(r'[^\w\-\.]', '', d)
+            if d and not d.startswith(('.', '-')) and not d.endswith(('.', '-')):
+                final_domains.append(d)
+
+        return [d for d in final_domains if _is_valid_domain(d)]

providers/dns_provider.py

@@ -5,7 +5,6 @@ import dns.reversename
 from typing import List, Dict, Any, Tuple
 from .base_provider import BaseProvider
 from utils.helpers import _is_valid_ip, _is_valid_domain
-from core.graph_manager import RelationshipType
 
 
 class DNSProvider(BaseProvider):
@@ -27,16 +26,29 @@ class DNSProvider(BaseProvider):
         self.resolver = dns.resolver.Resolver()
         self.resolver.timeout = 5
         self.resolver.lifetime = 10
+        #self.resolver.nameservers = ['127.0.0.1']
 
     def get_name(self) -> str:
         """Return the provider name."""
         return "dns"
 
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "DNS"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return False
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': True}
+
     def is_available(self) -> bool:
         """DNS is always available - no API key required."""
         return True
 
-    def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_domain(self, domain: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query DNS records for the domain to discover relationships.
 
@@ -52,12 +64,12 @@ class DNSProvider(BaseProvider):
         relationships = []
 
         # Query all record types
-        for record_type in ['A', 'AAAA', 'CNAME', 'MX', 'NS', 'SOA', 'TXT', 'SRV', 'CAA', 'DNSKEY', 'DS', 'RRSIG', 'SSHFP', 'TLSA', 'NAPTR', 'SPF']:
+        for record_type in ['A', 'AAAA', 'CNAME', 'MX', 'NS', 'SOA', 'TXT', 'SRV', 'CAA']:
             relationships.extend(self._query_record(domain, record_type))
 
         return relationships
 
-    def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def query_ip(self, ip: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query reverse DNS for the IP address.
 
@@ -93,16 +105,16 @@ class DNSProvider(BaseProvider):
                     relationships.append((
                         ip,
                         hostname,
-                        RelationshipType.PTR_RECORD,
-                        RelationshipType.PTR_RECORD.default_confidence,
+                        'ptr_record',
+                        0.8,
                         raw_data
                     ))
 
                     self.log_relationship_discovery(
                         source_node=ip,
                         target_node=hostname,
-                        relationship_type=RelationshipType.PTR_RECORD,
-                        confidence_score=RelationshipType.PTR_RECORD.default_confidence,
+                        relationship_type='ptr_record',
+                        confidence_score=0.8,
                         raw_data=raw_data,
                         discovery_method="reverse_dns_lookup"
                     )
@@ -113,7 +125,7 @@ class DNSProvider(BaseProvider):
 
         return relationships
 
-    def _query_record(self, domain: str, record_type: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def _query_record(self, domain: str, record_type: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query a specific type of DNS record for the domain.
         """
@@ -133,8 +145,9 @@ class DNSProvider(BaseProvider):
                     target = str(record.exchange).rstrip('.')
                 elif record_type == 'SOA':
                     target = str(record.mname).rstrip('.')
-                elif record_type in ['TXT', 'SPF']:
-                    target = b' '.join(record.strings).decode('utf-8', 'ignore')
+                elif record_type in ['TXT']:
+                    # TXT records are treated as metadata, not relationships.
+                    continue
                 elif record_type == 'SRV':
                     target = str(record.target).rstrip('.')
                 elif record_type == 'CAA':
@@ -142,7 +155,6 @@ class DNSProvider(BaseProvider):
                 else:
                     target = str(record)
 
-
                 if target:
                     raw_data = {
                         'query_type': record_type,
@@ -150,26 +162,25 @@ class DNSProvider(BaseProvider):
                         'value': target,
                         'ttl': response.ttl
                     }
-                    try:
-                        relationship_type_enum = getattr(RelationshipType, f"{record_type}_RECORD")
-                        relationships.append((
-                            domain,
-                            target,
-                            relationship_type_enum,
-                            relationship_type_enum.default_confidence,
-                            raw_data
-                        ))
+                    relationship_type = f"{record_type.lower()}_record"
+                    confidence = 0.8  # Default confidence for DNS records
 
-                        self.log_relationship_discovery(
-                            source_node=domain,
-                            target_node=target,
-                            relationship_type=relationship_type_enum,
-                            confidence_score=relationship_type_enum.default_confidence,
-                            raw_data=raw_data,
-                            discovery_method=f"dns_{record_type.lower()}_record"
-                        )
-                    except AttributeError:
-                        self.logger.logger.error(f"Unsupported record type '{record_type}' encountered for domain {domain}")
+                    relationships.append((
+                        domain,
+                        target,
+                        relationship_type,
+                        confidence,
+                        raw_data
+                    ))
+
+                    self.log_relationship_discovery(
+                        source_node=domain,
+                        target_node=target,
+                        relationship_type=relationship_type,
+                        confidence_score=confidence,
+                        raw_data=raw_data,
+                        discovery_method=f"dns_{record_type.lower()}_record"
+                    )
 
         except Exception as e:
             self.failed_requests += 1

providers/shodan_provider.py

@@ -7,7 +7,6 @@ import json
 from typing import List, Dict, Any, Tuple
 from .base_provider import BaseProvider
 from utils.helpers import _is_valid_ip, _is_valid_domain
-from core.graph_manager import RelationshipType
 
 
 class ShodanProvider(BaseProvider):
@@ -15,7 +14,7 @@ class ShodanProvider(BaseProvider):
     Provider for querying Shodan API for IP address and hostname information.
     Now uses session-specific API keys.
     """
-    
+
     def __init__(self, session_config=None):
         """Initialize Shodan provider with session-specific configuration."""
         super().__init__(
@@ -26,32 +25,43 @@ class ShodanProvider(BaseProvider):
         )
         self.base_url = "https://api.shodan.io"
         self.api_key = self.config.get_api_key('shodan')
-    
+
     def is_available(self) -> bool:
         """Check if Shodan provider is available (has valid API key in this session)."""
         return self.api_key is not None and len(self.api_key.strip()) > 0
-    
+
     def get_name(self) -> str:
         """Return the provider name."""
         return "shodan"
 
-    
-    def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+    def get_display_name(self) -> str:
+        """Return the provider display name for the UI."""
+        return "shodan"
+
+    def requires_api_key(self) -> bool:
+        """Return True if the provider requires an API key."""
+        return True
+
+    def get_eligibility(self) -> Dict[str, bool]:
+        """Return a dictionary indicating if the provider can query domains and/or IPs."""
+        return {'domains': True, 'ips': True}
+
+    def query_domain(self, domain: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query Shodan for information about a domain.
         Uses Shodan's hostname search to find associated IPs.
-        
+
         Args:
             domain: Domain to investigate
-            
+
         Returns:
             List of relationships discovered from Shodan data
         """
         if not _is_valid_domain(domain) or not self.is_available():
             return []
-        
+
         relationships = []
-        
+
         try:
             # Search for hostname in Shodan
             search_query = f"hostname:{domain}"
@@ -61,22 +71,22 @@ class ShodanProvider(BaseProvider):
                 'query': search_query,
                 'minify': True  # Get minimal data to reduce bandwidth
             }
-            
+
             response = self.make_request(url, method="GET", params=params, target_indicator=domain)
-            
+
             if not response or response.status_code != 200:
                 return []
-            
+
             data = response.json()
-            
+
             if 'matches' not in data:
                 return []
-            
+
             # Process search results
             for match in data['matches']:
                 ip_address = match.get('ip_str')
                 hostnames = match.get('hostnames', [])
-                
+
                 if ip_address and domain in hostnames:
                     raw_data = {
                         'ip_address': ip_address,
@@ -88,24 +98,24 @@ class ShodanProvider(BaseProvider):
                         'ports': match.get('ports', []),
                         'last_update': match.get('last_update', '')
                     }
-                    
+
                     relationships.append((
                         domain,
                         ip_address,
-                        RelationshipType.A_RECORD,  # Domain resolves to IP
-                        RelationshipType.A_RECORD.default_confidence,
+                        'a_record',  # Domain resolves to IP
+                        0.8,
                         raw_data
                     ))
-                    
+
                     self.log_relationship_discovery(
                         source_node=domain,
                         target_node=ip_address,
-                        relationship_type=RelationshipType.A_RECORD,
-                        confidence_score=RelationshipType.A_RECORD.default_confidence,
+                        relationship_type='a_record',
+                        confidence_score=0.8,
                         raw_data=raw_data,
                         discovery_method="shodan_hostname_search"
                     )
-                    
+
                     # Also create relationships to other hostnames on the same IP
                     for hostname in hostnames:
                         if hostname != domain and _is_valid_domain(hostname):
@@ -114,58 +124,56 @@ class ShodanProvider(BaseProvider):
                                 'all_hostnames': hostnames,
                                 'discovery_context': 'shared_hosting'
                             }
-                            
+
                             relationships.append((
                                 domain,
                                 hostname,
-                                RelationshipType.PASSIVE_DNS,  # Shared hosting relationship
+                                'passive_dns',  # Shared hosting relationship
                                 0.6,  # Lower confidence for shared hosting
                                 hostname_raw_data
                             ))
-                            
+
                             self.log_relationship_discovery(
                                 source_node=domain,
                                 target_node=hostname,
-                                relationship_type=RelationshipType.PASSIVE_DNS,
+                                relationship_type='passive_dns',
                                 confidence_score=0.6,
                                 raw_data=hostname_raw_data,
                                 discovery_method="shodan_shared_hosting"
                             )
-        
+
         except json.JSONDecodeError as e:
             self.logger.logger.error(f"Failed to parse JSON response from Shodan: {e}")
-        except Exception as e:
-            self.logger.logger.error(f"Error querying Shodan for domain {domain}: {e}")
-        
+
         return relationships
-    
-    def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
+
+    def query_ip(self, ip: str) -> List[Tuple[str, str, str, float, Dict[str, Any]]]:
         """
         Query Shodan for information about an IP address.
-        
+
         Args:
             ip: IP address to investigate
-            
+
         Returns:
             List of relationships discovered from Shodan IP data
         """
         if not _is_valid_ip(ip) or not self.is_available():
             return []
-        
+
         relationships = []
-        
+
         try:
             # Query Shodan host information
             url = f"{self.base_url}/shodan/host/{ip}"
             params = {'key': self.api_key}
-            
+
             response = self.make_request(url, method="GET", params=params, target_indicator=ip)
-            
+
             if not response or response.status_code != 200:
                 return []
-            
+
             data = response.json()
-            
+
             # Extract hostname relationships
             hostnames = data.get('hostnames', [])
             for hostname in hostnames:
@@ -182,73 +190,77 @@ class ShodanProvider(BaseProvider):
                         'last_update': data.get('last_update', ''),
                         'os': data.get('os', '')
                     }
-                    
+
                     relationships.append((
                         ip,
                         hostname,
-                        RelationshipType.A_RECORD,  # IP resolves to hostname
-                        RelationshipType.A_RECORD.default_confidence,
+                        'a_record',  # IP resolves to hostname
+                        0.8,
                         raw_data
                     ))
-                    
+
                     self.log_relationship_discovery(
                         source_node=ip,
                         target_node=hostname,
-                        relationship_type=RelationshipType.A_RECORD,
-                        confidence_score=RelationshipType.A_RECORD.default_confidence,
+                        relationship_type='a_record',
+                        confidence_score=0.8,
                         raw_data=raw_data,
                         discovery_method="shodan_host_lookup"
                     )
-            
+
             # Extract ASN relationship if available
             asn = data.get('asn')
             if asn:
-                asn_name = f"AS{asn}"
-                
+                # Ensure the ASN starts with "AS"
+                if isinstance(asn, str) and asn.startswith('AS'):
+                    asn_name = asn
+                    asn_number = asn[2:]
+                else:
+                    asn_name = f"AS{asn}"
+                    asn_number = str(asn)
+
                 asn_raw_data = {
                     'ip_address': ip,
-                    'asn': asn,
+                    'asn': asn_number,
                     'isp': data.get('isp', ''),
                     'org': data.get('org', '')
                 }
-                
+
                 relationships.append((
                     ip,
                     asn_name,
-                    RelationshipType.ASN_MEMBERSHIP,
-                    RelationshipType.ASN_MEMBERSHIP.default_confidence,
+                    'asn_membership',
+                    0.7,
                     asn_raw_data
                 ))
-                
+
                 self.log_relationship_discovery(
                     source_node=ip,
                     target_node=asn_name,
-                    relationship_type=RelationshipType.ASN_MEMBERSHIP,
-                    confidence_score=RelationshipType.ASN_MEMBERSHIP.default_confidence,
+                    relationship_type='asn_membership',
+                    confidence_score=0.7,
                     raw_data=asn_raw_data,
                     discovery_method="shodan_asn_lookup"
                 )
-        
+
         except json.JSONDecodeError as e:
             self.logger.logger.error(f"Failed to parse JSON response from Shodan: {e}")
-        except Exception as e:
-            self.logger.logger.error(f"Error querying Shodan for IP {ip}: {e}")
-        
+
         return relationships
-    
+
     def search_by_organization(self, org_name: str) -> List[Dict[str, Any]]:
         """
         Search Shodan for hosts belonging to a specific organization.
-        
+
         Args:
             org_name: Organization name to search for
-            
+
         Returns:
             List of host information dictionaries
         """
         if not self.is_available():
             return []
-        
+
         try:
             search_query = f"org:\"{org_name}\""
             url = f"{self.base_url}/shodan/host/search"
@@ -257,42 +269,42 @@ class ShodanProvider(BaseProvider):
                 'query': search_query,
                 'minify': True
             }
-            
+
             response = self.make_request(url, method="GET", params=params, target_indicator=org_name)
-            
+
             if response and response.status_code == 200:
                 data = response.json()
                 return data.get('matches', [])
-        
+
         except Exception as e:
             self.logger.logger.error(f"Error searching Shodan by organization {org_name}: {e}")
-        
+
         return []
-    
+
     def get_host_services(self, ip: str) -> List[Dict[str, Any]]:
         """
         Get service information for a specific IP address.
-        
+
         Args:
             ip: IP address to query
-            
+
         Returns:
             List of service information dictionaries
         """
         if not _is_valid_ip(ip) or not self.is_available():
             return []
-        
+
         try:
             url = f"{self.base_url}/shodan/host/{ip}"
             params = {'key': self.api_key}
-            
+
             response = self.make_request(url, method="GET", params=params, target_indicator=ip)
-            
+
             if response and response.status_code == 200:
                 data = response.json()
                 return data.get('data', [])  # Service banners
-        
+
         except Exception as e:
             self.logger.logger.error(f"Error getting Shodan services for IP {ip}: {e}")
-        
+
         return []

providers/virustotal_provider.py

@@ -1,333 +0,0 @@
-"""
-VirusTotal provider for DNSRecon.
-Discovers domain relationships through passive DNS and URL analysis.
-"""
-
-import json
-from typing import List, Dict, Any, Tuple
-from .base_provider import BaseProvider
-from utils.helpers import _is_valid_ip, _is_valid_domain
-from core.graph_manager import RelationshipType
-
-
-class VirusTotalProvider(BaseProvider):
-    """
-    Provider for querying VirusTotal API for passive DNS and domain reputation data.
-    Now uses session-specific API keys and rate limits.
-    """
-    
-    def __init__(self, session_config=None):
-        """Initialize VirusTotal provider with session-specific configuration."""
-        super().__init__(
-            name="virustotal",
-            rate_limit=4,  # Free tier: 4 requests per minute
-            timeout=30,
-            session_config=session_config
-        )
-        self.base_url = "https://www.virustotal.com/vtapi/v2"
-        self.api_key = self.config.get_api_key('virustotal')
-    
-    def is_available(self) -> bool:
-        """Check if VirusTotal provider is available (has valid API key in this session)."""
-        return self.api_key is not None and len(self.api_key.strip()) > 0
-    
-    def get_name(self) -> str:
-        """Return the provider name."""
-        return "virustotal"
-    
-    def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """
-        Query VirusTotal for domain information including passive DNS.
-        
-        Args:
-            domain: Domain to investigate
-            
-        Returns:
-            List of relationships discovered from VirusTotal data
-        """
-        if not _is_valid_domain(domain) or not self.is_available():
-            return []
-        
-        relationships = []
-        
-        # Query domain report
-        domain_relationships = self._query_domain_report(domain)
-        relationships.extend(domain_relationships)
-        
-        # Query passive DNS for the domain
-        passive_dns_relationships = self._query_passive_dns_domain(domain)
-        relationships.extend(passive_dns_relationships)
-        
-        return relationships
-    
-    def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """
-        Query VirusTotal for IP address information including passive DNS.
-        
-        Args:
-            ip: IP address to investigate
-            
-        Returns:
-            List of relationships discovered from VirusTotal IP data
-        """
-        if not _is_valid_ip(ip) or not self.is_available():
-            return []
-        
-        relationships = []
-        
-        # Query IP report
-        ip_relationships = self._query_ip_report(ip)
-        relationships.extend(ip_relationships)
-        
-        # Query passive DNS for the IP
-        passive_dns_relationships = self._query_passive_dns_ip(ip)
-        relationships.extend(passive_dns_relationships)
-        
-        return relationships
-    
-    def _query_domain_report(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """Query VirusTotal domain report."""
-        relationships = []
-        
-        try:
-            url = f"{self.base_url}/domain/report"
-            params = {
-                'apikey': self.api_key,
-                'domain': domain,
-                'allinfo': 1  # Get comprehensive information
-            }
-            
-            response = self.make_request(url, method="GET", params=params, target_indicator=domain)
-            
-            if not response or response.status_code != 200:
-                return []
-            
-            data = response.json()
-            
-            if data.get('response_code') != 1:
-                return []
-            
-            # Extract resolved IPs
-            resolutions = data.get('resolutions', [])
-            for resolution in resolutions:
-                ip_address = resolution.get('ip_address')
-                last_resolved = resolution.get('last_resolved')
-                
-                if ip_address and _is_valid_ip(ip_address):
-                    raw_data = {
-                        'domain': domain,
-                        'ip_address': ip_address,
-                        'last_resolved': last_resolved,
-                        'source': 'virustotal_domain_report'
-                    }
-                    
-                    relationships.append((
-                        domain,
-                        ip_address,
-                        RelationshipType.PASSIVE_DNS,
-                        RelationshipType.PASSIVE_DNS.default_confidence,
-                        raw_data
-                    ))
-                    
-                    self.log_relationship_discovery(
-                        source_node=domain,
-                        target_node=ip_address,
-                        relationship_type=RelationshipType.PASSIVE_DNS,
-                        confidence_score=RelationshipType.PASSIVE_DNS.default_confidence,
-                        raw_data=raw_data,
-                        discovery_method="virustotal_domain_resolution"
-                    )
-            
-            # Extract subdomains
-            subdomains = data.get('subdomains', [])
-            for subdomain in subdomains:
-                if subdomain != domain and _is_valid_domain(subdomain):
-                    raw_data = {
-                        'parent_domain': domain,
-                        'subdomain': subdomain,
-                        'source': 'virustotal_subdomain_discovery'
-                    }
-                    
-                    relationships.append((
-                        domain,
-                        subdomain,
-                        RelationshipType.PASSIVE_DNS,
-                        0.7,  # Medium-high confidence for subdomains
-                        raw_data
-                    ))
-                    
-                    self.log_relationship_discovery(
-                        source_node=domain,
-                        target_node=subdomain,
-                        relationship_type=RelationshipType.PASSIVE_DNS,
-                        confidence_score=0.7,
-                        raw_data=raw_data,
-                        discovery_method="virustotal_subdomain_discovery"
-                    )
-        
-        except json.JSONDecodeError as e:
-            self.logger.logger.error(f"Failed to parse JSON response from VirusTotal: {e}")
-        except Exception as e:
-            self.logger.logger.error(f"Error querying VirusTotal domain report for {domain}: {e}")
-        
-        return relationships
-    
-    def _query_ip_report(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """Query VirusTotal IP report."""
-        relationships = []
-        
-        try:
-            url = f"{self.base_url}/ip-address/report"
-            params = {
-                'apikey': self.api_key,
-                'ip': ip
-            }
-            
-            response = self.make_request(url, method="GET", params=params, target_indicator=ip)
-            
-            if not response or response.status_code != 200:
-                return []
-            
-            data = response.json()
-            
-            if data.get('response_code') != 1:
-                return []
-            
-            # Extract resolved domains
-            resolutions = data.get('resolutions', [])
-            for resolution in resolutions:
-                hostname = resolution.get('hostname')
-                last_resolved = resolution.get('last_resolved')
-                
-                if hostname and _is_valid_domain(hostname):
-                    raw_data = {
-                        'ip_address': ip,
-                        'hostname': hostname,
-                        'last_resolved': last_resolved,
-                        'source': 'virustotal_ip_report'
-                    }
-                    
-                    relationships.append((
-                        ip,
-                        hostname,
-                        RelationshipType.PASSIVE_DNS,
-                        RelationshipType.PASSIVE_DNS.default_confidence,
-                        raw_data
-                    ))
-                    
-                    self.log_relationship_discovery(
-                        source_node=ip,
-                        target_node=hostname,
-                        relationship_type=RelationshipType.PASSIVE_DNS,
-                        confidence_score=RelationshipType.PASSIVE_DNS.default_confidence,
-                        raw_data=raw_data,
-                        discovery_method="virustotal_ip_resolution"
-                    )
-        
-        except json.JSONDecodeError as e:
-            self.logger.logger.error(f"Failed to parse JSON response from VirusTotal: {e}")
-        except Exception as e:
-            self.logger.logger.error(f"Error querying VirusTotal IP report for {ip}: {e}")
-        
-        return relationships
-    
-    def _query_passive_dns_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """Query VirusTotal passive DNS for domain."""
-        # Note: VirusTotal's passive DNS API might require a premium subscription
-        # This is a placeholder for the endpoint structure
-        return []
-    
-    def _query_passive_dns_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
-        """Query VirusTotal passive DNS for IP."""
-        # Note: VirusTotal's passive DNS API might require a premium subscription
-        # This is a placeholder for the endpoint structure
-        return []
-    
-    def get_domain_reputation(self, domain: str) -> Dict[str, Any]:
-        """
-        Get domain reputation information from VirusTotal.
-        
-        Args:
-            domain: Domain to check reputation for
-            
-        Returns:
-            Dictionary containing reputation data
-        """
-        if not _is_valid_domain(domain) or not self.is_available():
-            return {}
-        
-        try:
-            url = f"{self.base_url}/domain/report"
-            params = {
-                'apikey': self.api_key,
-                'domain': domain
-            }
-            
-            response = self.make_request(url, method="GET", params=params, target_indicator=domain)
-            
-            if response and response.status_code == 200:
-                data = response.json()
-                
-                if data.get('response_code') == 1:
-                    return {
-                        'positives': data.get('positives', 0),
-                        'total': data.get('total', 0),
-                        'scan_date': data.get('scan_date', ''),
-                        'permalink': data.get('permalink', ''),
-                        'reputation_score': self._calculate_reputation_score(data)
-                    }
-        
-        except Exception as e:
-            self.logger.logger.error(f"Error getting VirusTotal reputation for domain {domain}: {e}")
-        
-        return {}
-    
-    def get_ip_reputation(self, ip: str) -> Dict[str, Any]:
-        """
-        Get IP reputation information from VirusTotal.
-        
-        Args:
-            ip: IP address to check reputation for
-            
-        Returns:
-            Dictionary containing reputation data
-        """
-        if not _is_valid_ip(ip) or not self.is_available():
-            return {}
-        
-        try:
-            url = f"{self.base_url}/ip-address/report"
-            params = {
-                'apikey': self.api_key,
-                'ip': ip
-            }
-            
-            response = self.make_request(url, method="GET", params=params, target_indicator=ip)
-            
-            if response and response.status_code == 200:
-                data = response.json()
-                
-                if data.get('response_code') == 1:
-                    return {
-                        'positives': data.get('positives', 0),
-                        'total': data.get('total', 0),
-                        'scan_date': data.get('scan_date', ''),
-                        'permalink': data.get('permalink', ''),
-                        'reputation_score': self._calculate_reputation_score(data)
-                    }
-        
-        except Exception as e:
-            self.logger.logger.error(f"Error getting VirusTotal reputation for IP {ip}: {e}")
-        
-        return {}
-    
-    def _calculate_reputation_score(self, data: Dict[str, Any]) -> float:
-        """Calculate a normalized reputation score (0.0 to 1.0)."""
-        positives = data.get('positives', 0)
-        total = data.get('total', 1)  # Avoid division by zero
-        
-        if total == 0:
-            return 1.0  # No data means neutral
-        
-        # Score is inverse of detection ratio (lower detection = higher reputation)
-        return max(0.0, 1.0 - (positives / total))

requirements.txt

@@ -4,4 +4,6 @@ requests>=2.31.0
 python-dateutil>=2.8.2
 Werkzeug>=2.3.7
 urllib3>=2.0.0
-dnspython>=2.4.2
+dnspython>=2.4.2
+gunicorn
+redis

static/css/main.css

@@ -314,9 +314,39 @@ input[type="text"]:focus, select:focus {
 
 .view-controls {
     display: flex;
+    gap: 1.5rem;
+    align-items: center;
+}
+
+.filter-group {
+    display: flex;
+    align-items: center;
     gap: 0.5rem;
 }
 
+.filter-group label {
+    font-size: 0.9rem;
+    color: #999;
+}
+
+.filter-group select,
+.filter-group input[type="range"] {
+    background-color: #1a1a1a;
+    border: 1px solid #555;
+    color: #c7c7c7;
+    padding: 0.25rem 0.5rem;
+}
+
+.filter-group select {
+    max-width: 150px;
+}
+
+#confidence-value {
+    min-width: 30px;
+    text-align: center;
+    color: #00ff41;
+}
+
 .graph-container {
     height: 800px;
     position: relative;
@@ -551,30 +581,6 @@ input[type="text"]:focus, select:focus {
     color: #555;
 }
 
-/* Modal */
-.modal {
-    display: none;
-    position: fixed;
-    z-index: 1000;
-    left: 0;
-    top: 0;
-    width: 100%;
-    height: 100%;
-    background-color: rgba(0, 0, 0, 0.8);
-    animation: fadeIn 0.3s ease-out;
-}
-
-.modal-content {
-    background-color: #2a2a2a;
-    border: 1px solid #444;
-    margin: 5% auto;
-    width: 80%;
-    max-width: 600px;
-    max-height: 80vh;
-    overflow-y: auto;
-    animation: slideInDown 0.3s ease-out;
-}
-
 @keyframes slideInDown {
     from {
         opacity: 0;
@@ -586,43 +592,6 @@ input[type="text"]:focus, select:focus {
     }
 }
 
-.modal-header {
-    background-color: #1a1a1a;
-    padding: 1rem;
-    border-bottom: 1px solid #444;
-    display: flex;
-    justify-content: space-between;
-    align-items: center;
-}
-
-.modal-header h3 {
-    color: #00ff41;
-    font-size: 1.1rem;
-}
-
-.modal-close {
-    background: transparent;
-    border: none;
-    color: #c7c7c7;
-    font-size: 1.2rem;
-    cursor: pointer;
-    font-family: 'Roboto Mono', monospace;
-}
-
-.modal-close:hover {
-    color: #ff9900;
-}
-
-.modal-body {
-    padding: 1.5rem;
-}
-
-.modal-description {
-    color: #999;
-    margin-bottom: 1.5rem;
-    line-height: 1.6;
-}
-
 .detail-row {
     display: flex;
     justify-content: space-between;
@@ -771,12 +740,6 @@ input[type="text"]:focus, select:focus {
     color: #00ff41 !important;
 }
 
-/* Animations */
-@keyframes fadeIn {
-    from { opacity: 0; transform: translateY(10px); }
-    to { opacity: 1; transform: translateY(0); }
-}
-
 .fade-in {
     animation: fadeIn 0.3s ease-out;
 }
@@ -905,4 +868,179 @@ input[type="text"]:focus, select:focus {
         transform: translateX(100%);
         opacity: 0;
     }
+}
+
+/* dnsrecon/static/css/main.css */
+
+/* ... (at the end of the file) */
+
+.large-entity-nodes-list {
+    margin-top: 1rem;
+}
+
+.large-entity-node-details {
+    margin-bottom: 0.5rem;
+    border: 1px solid #333;
+    border-radius: 3px;
+}
+
+.large-entity-node-details summary {
+    padding: 0.5rem;
+    background-color: #3a3a3a;
+    cursor: pointer;
+    outline: none;
+}
+
+.large-entity-node-details summary:hover {
+    background-color: #4a4a4a;
+}
+
+.large-entity-node-details .detail-row {
+    margin-left: 1rem;
+    margin-right: 1rem;
+}
+
+.large-entity-node-details .detail-section-header {
+    margin-left: 1rem;
+    margin-right: 1rem;
+}
+
+/* dnsrecon/static/css/main.css */
+
+/* --- Add these styles for the modal --- */
+
+.modal {
+    display: none; /* Hidden by default */
+    position: fixed; /* Stay in place */
+    z-index: 1000; /* Sit on top */
+    left: 0;
+    top: 0;
+    width: 100%;
+    height: 100%;
+    overflow: auto; /* Enable scroll if needed */
+    background-color: rgba(0,0,0,0.6); /* Black w/ opacity */
+    backdrop-filter: blur(5px);
+}
+
+.modal-content {
+    background-color: #1e1e1e;
+    margin: 10% auto;
+    padding: 20px;
+    border: 1px solid #444;
+    width: 60%;
+    max-width: 800px;
+    border-radius: 5px;
+    box-shadow: 0 5px 15px rgba(0,0,0,0.5);
+    animation: fadeIn 0.3s;
+}
+
+.modal-header {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    border-bottom: 1px solid #444;
+    padding-bottom: 10px;
+    margin-bottom: 20px;
+}
+
+.modal-header h3 {
+    margin: 0;
+    font-family: 'Special Elite', monospace;
+    color: #00ff41;
+}
+
+.modal-close {
+    background: none;
+    border: none;
+    color: #c7c7c7;
+    font-size: 24px;
+    cursor: pointer;
+    padding: 0 10px;
+}
+.modal-close:hover {
+    color: #ff6b6b;
+}
+
+.modal-body {
+    max-height: 60vh;
+    overflow-y: auto;
+}
+
+/* Styles for the new data model display */
+.modal-details-grid {
+    display: grid;
+    grid-template-columns: 1fr;
+    gap: 20px;
+}
+
+.modal-section h4 {
+    font-family: 'Special Elite', monospace;
+    color: #ff9900;
+    border-bottom: 1px dashed #555;
+    padding-bottom: 5px;
+    margin-top: 0;
+}
+
+.modal-section ul {
+    list-style-type: none;
+    padding-left: 15px;
+}
+
+.modal-section li {
+    margin-bottom: 8px;
+}
+
+.modal-section li > ul {
+    padding-left: 20px;
+    margin-top: 5px;
+}
+
+.description-text, .no-data {
+    color: #aaa;
+    font-style: italic;
+}
+
+.correlation-values-list {
+    margin-top: 1rem;
+}
+
+.correlation-value-details {
+    margin-bottom: 0.5rem;
+    border: 1px solid #333;
+    border-radius: 3px;
+}
+
+.correlation-value-details summary {
+    padding: 0.5rem;
+    background-color: #3a3a3a;
+    cursor: pointer;
+    outline: none;
+    color: #c7c7c7;
+}
+
+.correlation-value-details summary:hover {
+    background-color: #4a4a4a;
+}
+
+.correlation-value-details .detail-row {
+    margin-left: 1rem;
+    margin-right: 1rem;
+    padding: 0.5rem 0;
+}
+
+.correlation-value-details .detail-label {
+    color: #999;
+    font-weight: 500;
+}
+
+.correlation-value-details .detail-value {
+    color: #c7c7c7;
+    word-break: break-all;
+    font-family: 'Roboto Mono', monospace;
+    font-size: 0.9em;
+}
+
+@keyframes fadeIn {
+    from {opacity: 0; transform: scale(0.95);}
+    to {opacity: 1; transform: scale(1);}
 }

static/js/graph.js

@@ -1,6 +1,6 @@
 /**
  * Graph visualization module for DNSRecon
- * Handles network graph rendering using vis.js with enhanced Phase 2 features
+ * Handles network graph rendering using vis.js
  */
 
 class GraphManager {
@@ -13,7 +13,6 @@ class GraphManager {
         this.currentLayout = 'physics';
         this.nodeInfoPopup = null;
 
-        // Enhanced graph options for Phase 2
         this.options = {
             nodes: {
                 shape: 'dot',
@@ -28,13 +27,6 @@ class GraphManager {
                 },
                 borderWidth: 2,
                 borderColor: '#444',
-                shadow: {
-                    enabled: true,
-                    color: 'rgba(0, 0, 0, 0.5)',
-                    size: 5,
-                    x: 2,
-                    y: 2
-                },
                 scaling: {
                     min: 10,
                     max: 30,
@@ -48,9 +40,6 @@ class GraphManager {
                     node: (values, id, selected, hovering) => {
                         values.borderColor = '#00ff41';
                         values.borderWidth = 3;
-                        values.shadow = true;
-                        values.shadowColor = 'rgba(0, 255, 65, 0.6)';
-                        values.shadowSize = 10;
                     }
                 }
             },
@@ -82,19 +71,10 @@ class GraphManager {
                     type: 'dynamic',
                     roundness: 0.6
                 },
-                shadow: {
-                    enabled: true,
-                    color: 'rgba(0, 0, 0, 0.3)',
-                    size: 3,
-                    x: 1,
-                    y: 1
-                },
                 chosen: {
                     edge: (values, id, selected, hovering) => {
                         values.color = '#00ff41';
                         values.width = 4;
-                        values.shadow = true;
-                        values.shadowColor = 'rgba(0, 255, 65, 0.4)';
                     }
                 }
             },
@@ -150,7 +130,7 @@ class GraphManager {
     }
 
     /**
-     * Initialize the network graph with enhanced features
+     * Initialize the network graph
      */
     initialize() {
         if (this.isInitialized) {
@@ -176,7 +156,7 @@ class GraphManager {
             // Add graph controls
             this.addGraphControls();
 
-            console.log('Enhanced graph initialized successfully');
+            console.log('Graph initialized successfully');
         } catch (error) {
             console.error('Failed to initialize graph:', error);
             this.showError('Failed to initialize visualization');
@@ -191,44 +171,43 @@ class GraphManager {
         controlsContainer.className = 'graph-controls';
         controlsContainer.innerHTML = `
             <button class="graph-control-btn" id="graph-fit" title="Fit to Screen">[FIT]</button>
-            <button class="graph-control-btn" id="graph-reset" title="Reset View">[RESET]</button>
             <button class="graph-control-btn" id="graph-physics" title="Toggle Physics">[PHYSICS]</button>
             <button class="graph-control-btn" id="graph-cluster" title="Cluster Nodes">[CLUSTER]</button>
-            <button class="graph-control-btn" id="graph-clear" title="Clear Graph">[CLEAR]</button>
         `;
 
         this.container.appendChild(controlsContainer);
 
         // Add control event listeners
         document.getElementById('graph-fit').addEventListener('click', () => this.fitView());
-        document.getElementById('graph-reset').addEventListener('click', () => this.resetView());
         document.getElementById('graph-physics').addEventListener('click', () => this.togglePhysics());
         document.getElementById('graph-cluster').addEventListener('click', () => this.toggleClustering());
-        document.getElementById('graph-clear').addEventListener('click', () => this.clear());
     }
 
     /**
-     * Setup enhanced network event handlers
+     * Setup network event handlers
      */
     setupNetworkEvents() {
         if (!this.network) return;
 
-        // Node click event with enhanced details
+        // Node click event with details
         this.network.on('click', (params) => {
             if (params.nodes.length > 0) {
                 const nodeId = params.nodes[0];
                 if (this.network.isCluster(nodeId)) {
                     this.network.openCluster(nodeId);
                 } else {
-                    this.showNodeDetails(nodeId);
-                    this.highlightNodeConnections(nodeId);
+                    const node = this.nodes.get(nodeId);
+                    if (node) {
+                        this.showNodeDetails(node);
+                        this.highlightNodeConnections(nodeId);
+                    }
                 }
             } else {
                 this.clearHighlights();
             }
         });
 
-        // Enhanced hover events
+        // Hover events
         this.network.on('hoverNode', (params) => {
             const nodeId = params.node;
             const node = this.nodes.get(nodeId);
@@ -237,25 +216,12 @@ class GraphManager {
             }
         });
 
-        this.network.on('blurNode', (params) => {
-            this.hideNodeInfoPopup();
-            this.clearHoverHighlights();
-        });
-
-        // Double-click to focus on node
-        this.network.on('doubleClick', (params) => {
-            if (params.nodes.length > 0) {
-                const nodeId = params.nodes[0];
-                this.focusOnNode(nodeId);
-            }
-        });
-
-        // Context menu (right-click)
+        // FIX: Comment out the problematic context menu handler
         this.network.on('oncontext', (params) => {
             params.event.preventDefault();
-            if (params.nodes.length > 0) {
-                this.showNodeContextMenu(params.pointer.DOM, params.nodes[0]);
-            }
+            // if (params.nodes.length > 0) {
+            //     this.showNodeContextMenu(params.pointer.DOM, params.nodes[0]);
+            // }
         });
 
         // Stabilization events with progress
@@ -276,7 +242,6 @@ class GraphManager {
     }
 
     /**
-     * Update graph with new data and enhanced processing
      * @param {Object} graphData - Graph data from backend
      */
     updateGraph(graphData) {
@@ -291,9 +256,52 @@ class GraphManager {
                 this.initialize();
             }
 
-            // Process nodes with enhanced attributes
-            const processedNodes = graphData.nodes.map(node => this.processNode(node));
-            const processedEdges = graphData.edges.map(edge => this.processEdge(edge));
+            const largeEntityMap = new Map();
+            graphData.nodes.forEach(node => {
+                if (node.type === 'large_entity' && node.attributes && Array.isArray(node.attributes.nodes)) {
+                    node.attributes.nodes.forEach(nodeId => {
+                        largeEntityMap.set(nodeId, node.id);
+                    });
+                }
+            });
+
+            const processedNodes = graphData.nodes.map(node => {
+                const processed = this.processNode(node);
+                if (largeEntityMap.has(node.id)) {
+                    processed.hidden = true;
+                }
+                return processed;
+            });
+
+            const mergedEdges = {};
+            graphData.edges.forEach(edge => {
+                const fromNode = largeEntityMap.has(edge.from) ? largeEntityMap.get(edge.from) : edge.from;
+                const toNode = largeEntityMap.has(edge.to) ? largeEntityMap.get(edge.to) : edge.to;
+                const mergeKey = `${fromNode}-${toNode}-${edge.label}`;
+
+                if (!mergedEdges[mergeKey]) {
+                    mergedEdges[mergeKey] = {
+                        ...edge,
+                        from: fromNode,
+                        to: toNode,
+                        count: 0,
+                        confidence_score: 0
+                    };
+                }
+
+                mergedEdges[mergeKey].count++;
+                if (edge.confidence_score > mergedEdges[mergeKey].confidence_score) {
+                    mergedEdges[mergeKey].confidence_score = edge.confidence_score;
+                }
+            });
+
+            const processedEdges = Object.values(mergedEdges).map(edge => {
+                const processed = this.processEdge(edge);
+                if (edge.count > 1) {
+                    processed.label = `${edge.label} (${edge.count})`;
+                }
+                return processed;
+            });
 
             // Update datasets with animation
             const existingNodeIds = this.nodes.getIds();
@@ -317,15 +325,15 @@ class GraphManager {
                 setTimeout(() => this.fitView(), 800);
             }
 
-            console.log(`Enhanced graph updated: ${processedNodes.length} nodes, ${processedEdges.length} edges (${newNodes.length} new nodes, ${newEdges.length} new edges)`);
+            console.log(`Graph updated: ${processedNodes.length} nodes, ${processedEdges.length} edges (${newNodes.length} new nodes, ${newEdges.length} new edges)`);
         } catch (error) {
-            console.error('Failed to update enhanced graph:', error);
+            console.error('Failed to update graph:', error);
             this.showError('Failed to update visualization');
         }
     }
 
     /**
-     * Process node data with enhanced styling and metadata
+     * Process node data with styling and metadata
      * @param {Object} node - Raw node data
      * @returns {Object} Processed node data
      */
@@ -337,8 +345,12 @@ class GraphManager {
             size: this.getNodeSize(node.type),
             borderColor: this.getNodeBorderColor(node.type),
             shape: this.getNodeShape(node.type),
+            attributes: node.attributes || {},
+            description: node.description || '',
             metadata: node.metadata || {},
-            type: node.type
+            type: node.type,
+            incoming_edges: node.incoming_edges || [],
+            outgoing_edges: node.outgoing_edges || []
         };
 
         // Add confidence-based styling
@@ -346,25 +358,30 @@ class GraphManager {
             processedNode.borderWidth = Math.max(2, Math.floor(node.confidence * 5));
         }
 
-        // Add special styling for important nodes
-        if (this.isImportantNode(node)) {
-            processedNode.shadow = {
-                enabled: true,
-                color: 'rgba(0, 255, 65, 0.6)',
-                size: 10,
-                x: 2,
-                y: 2
-            };
-        }
-
         // Style based on certificate validity
         if (node.type === 'domain') {
-            if (node.metadata && node.metadata.certificate_data && node.metadata.certificate_data.has_valid_cert === true) {
-                processedNode.color = '#00ff41'; // Bright green for valid cert
-                processedNode.borderColor = '#00aa2e';
-            } else if (node.metadata && node.metadata.certificate_data && node.metadata.certificate_data.has_valid_cert === false) {
-                processedNode.color = '#888888'; // Muted grey color
-                processedNode.borderColor = '#666666'; // Darker grey border
+            if (node.attributes && node.attributes.certificates && node.attributes.certificates.has_valid_cert === false) {
+                processedNode.color = { background: '#888888', border: '#666666' };
+            }
+        }
+        
+        // Handle merged correlation objects (similar to large entities)
+        if (node.type === 'correlation_object') {
+            const metadata = node.metadata || {};
+            const values = metadata.values || [];
+            const mergeCount = metadata.merge_count || 1;
+            
+            if (mergeCount > 1) {
+                // Display as merged correlation container
+                processedNode.label = `Correlations (${mergeCount})`;
+                processedNode.title = `Merged correlation container with ${mergeCount} values: ${values.slice(0, 3).join(', ')}${values.length > 3 ? '...' : ''}`;
+                processedNode.borderWidth = 3; // Thicker border for merged nodes
+            } else {
+                // Single correlation value
+                const value = Array.isArray(values) && values.length > 0 ? values[0] : (metadata.value || 'Unknown');
+                const displayValue = typeof value === 'string' && value.length > 20 ? value.substring(0, 17) + '...' : value;
+                processedNode.label = `Corr: ${displayValue}`;
+                processedNode.title = `Correlation: ${value}`;
             }
         }
 
@@ -372,7 +389,7 @@ class GraphManager {
     }
 
     /**
-     * Process edge data with enhanced styling and metadata
+     * Process edge data with styling and metadata
      * @param {Object} edge - Raw edge data
      * @returns {Object} Processed edge data
      */
@@ -395,16 +412,7 @@ class GraphManager {
             }
         };
 
-        // Add animation for high-confidence edges
-        if (confidence >= 0.8) {
-            processedEdge.shadow = {
-                enabled: true,
-                color: 'rgba(0, 255, 65, 0.3)',
-                size: 5,
-                x: 1,
-                y: 1
-            };
-        }
+
 
         return processedEdge;
     }
@@ -416,7 +424,7 @@ class GraphManager {
      * @returns {string} Formatted label
      */
     formatNodeLabel(nodeId, nodeType) {
-        // Truncate long domain names
+        if (typeof nodeId !== 'string') return '';
         if (nodeId.length > 20) {
             return nodeId.substring(0, 17) + '...';
         }
@@ -447,7 +455,7 @@ class GraphManager {
             'ip': '#ff9900',         // Amber
             'asn': '#00aaff',         // Blue
             'large_entity': '#ff6b6b', // Red for large entities
-            'dns_record': '#999999'
+            'correlation_object': '#9620c0ff'
         };
         return colors[nodeType] || '#ffffff';
     }
@@ -463,7 +471,7 @@ class GraphManager {
             'domain': '#00aa2e',
             'ip': '#cc7700',
             'asn': '#0088cc',
-            'dns_record': '#999999'
+            'correlation_object': '#c235c9ff'
         };
         return borderColors[nodeType] || '#666666';
     }
@@ -478,13 +486,14 @@ class GraphManager {
             'domain': 12,
             'ip': 14,
             'asn': 16,
-            'dns_record': 8
+            'correlation_object': 8,
+            'large_entity': 5
         };
         return sizes[nodeType] || 12;
     }
     
     /**
-     * Get enhanced node shape based on type
+     * Get node shape based on type
      * @param {string} nodeType - Node type
      * @returns {string} Shape name
      */
@@ -493,7 +502,8 @@ class GraphManager {
             'domain': 'dot',
             'ip': 'square',
             'asn': 'triangle',
-            'dns_record': 'hexagon'
+            'correlation_object': 'hexagon',
+            'large_entity': 'database'
         };
         return shapes[nodeType] || 'dot';
     }
@@ -566,15 +576,12 @@ class GraphManager {
 
     /**
      * Show node details in modal
-     * @param {string} nodeId - Node identifier
+     * @param {Object} node - Node object
      */
-    showNodeDetails(nodeId) {
-        const node = this.nodes.get(nodeId);
-        if (!node) return;
-
+    showNodeDetails(node) {
         // Trigger custom event for main application to handle
         const event = new CustomEvent('nodeSelected', {
-            detail: { nodeId, node }
+            detail: { node }
         });
         document.dispatchEvent(event);
     }
@@ -720,14 +727,7 @@ class GraphManager {
         const nodeHighlights = newNodes.map(node => ({
             id: node.id,
             borderColor: '#00ff41',
-            borderWidth: 4,
-            shadow: {
-                enabled: true,
-                color: 'rgba(0, 255, 65, 0.8)',
-                size: 15,
-                x: 2,
-                y: 2
-            }
+            borderWidth: 4
         }));
         
         // Briefly highlight new edges
@@ -746,7 +746,6 @@ class GraphManager {
                 id: node.id,
                 borderColor: this.getNodeBorderColor(node.type),
                 borderWidth: 2,
-                shadow: node.shadow || { enabled: false }
             }));
             
             const edgeResets = newEdges.map(edge => ({
@@ -845,22 +844,6 @@ class GraphManager {
         }
     }
 
-    /**
-     * Reset the view to initial state
-     */
-    resetView() {
-        if (this.network) {
-            this.network.moveTo({
-                position: { x: 0, y: 0 },
-                scale: 1,
-                animation: {
-                    duration: 1000,
-                    easingFunction: 'easeInOutQuad'
-                }
-            });
-        }
-    }
-
     /**
      * Clear the graph
      */
@@ -900,17 +883,45 @@ class GraphManager {
     }
 
     /**
-     * Export graph as image (if needed for future implementation)
-     * @param {string} format - Image format ('png', 'jpeg')
-     * @returns {string} Data URL of the image
+     * Apply filters to the graph
+     * @param {string} nodeType - The type of node to show ('all' for no filter)
+     * @param {number} minConfidence - The minimum confidence score for edges to be visible
      */
-    exportAsImage(format = 'png') {
-        if (!this.network) return null;
+    applyFilters(nodeType, minConfidence) {
+        console.log(`Applying filters: nodeType=${nodeType}, minConfidence=${minConfidence}`);
 
-        // This would require additional vis.js functionality
-        // Placeholder for future implementation
-        console.log('Image export not yet implemented');
-        return null;
+        const nodeUpdates = [];
+        const edgeUpdates = [];
+
+        const allNodes = this.nodes.get({ returnType: 'Object' });
+        const allEdges = this.edges.get();
+
+        // Determine which nodes are visible based on the nodeType filter
+        for (const nodeId in allNodes) {
+            const node = allNodes[nodeId];
+            const isVisible = (nodeType === 'all' || node.type === nodeType);
+            nodeUpdates.push({ id: nodeId, hidden: !isVisible });
+        }
+
+        // Update nodes first to determine edge visibility
+        this.nodes.update(nodeUpdates);
+
+        // Determine which edges are visible based on confidence and connected nodes
+        for (const edge of allEdges) {
+            const sourceNode = this.nodes.get(edge.from);
+            const targetNode = this.nodes.get(edge.to);
+            const confidence = edge.metadata ? edge.metadata.confidence_score : 0;
+
+            const isVisible = confidence >= minConfidence &&
+                              sourceNode && !sourceNode.hidden &&
+                              targetNode && !targetNode.hidden;
+            
+            edgeUpdates.push({ id: edge.id, hidden: !isVisible });
+        }
+
+        this.edges.update(edgeUpdates);
+
+        console.log('Filters applied.');
     }
 }
 

static/js/main.js

@@ -12,10 +12,8 @@ class DNSReconApp {
         this.pollInterval = null;
         this.currentSessionId = null;
         
-        // UI Elements
         this.elements = {};
         
-        // Application state
         this.isScanning = false;
         this.lastGraphUpdate = null;
         
@@ -54,6 +52,7 @@ class DNSReconApp {
             targetDomain: document.getElementById('target-domain'),
             maxDepth: document.getElementById('max-depth'),
             startScan: document.getElementById('start-scan'),
+            addToGraph: document.getElementById('add-to-graph'),
             stopScan: document.getElementById('stop-scan'),
             exportResults: document.getElementById('export-results'),
             configureApiKeys: document.getElementById('configure-api-keys'),
@@ -79,14 +78,18 @@ class DNSReconApp {
             // API Key Modal elements
             apiKeyModal: document.getElementById('api-key-modal'),
             apiKeyModalClose: document.getElementById('api-key-modal-close'),
-            virustotalApiKey: document.getElementById('virustotal-api-key'),
-            shodanApiKey: document.getElementById('shodan-api-key'),
+            apiKeyInputs: document.getElementById('api-key-inputs'),
             saveApiKeys: document.getElementById('save-api-keys'),
             resetApiKeys: document.getElementById('reset-api-keys'),
 
             // Other elements
             sessionId: document.getElementById('session-id'),
-            connectionStatus: document.getElementById('connection-status')
+            connectionStatus: document.getElementById('connection-status'),
+            
+            // Filter elements
+            nodeTypeFilter: document.getElementById('node-type-filter'),
+            confidenceFilter: document.getElementById('confidence-filter'),
+            confidenceValue: document.getElementById('confidence-value')
         };
 
         // Verify critical elements exist
@@ -136,6 +139,11 @@ class DNSReconApp {
                 e.preventDefault();
                 this.startScan();
             });
+
+            this.elements.addToGraph.addEventListener('click', (e) => {
+                e.preventDefault();
+                this.startScan(false);
+            });
             
             this.elements.stopScan.addEventListener('click', (e) => {
                 console.log('Stop scan button clicked');
@@ -185,9 +193,9 @@ class DNSReconApp {
                 this.elements.resetApiKeys.addEventListener('click', () => this.resetApiKeys());
             }
 
-            // Custom events
+            // ** FIX: Listen for the custom event from the graph **
             document.addEventListener('nodeSelected', (e) => {
-                this.showNodeModal(e.detail.nodeId, e.detail.node);
+                this.showNodeModal(e.detail.node);
             });
             
             // Keyboard shortcuts
@@ -205,6 +213,13 @@ class DNSReconApp {
                 }
             });
             
+             // Filter events
+            this.elements.nodeTypeFilter.addEventListener('change', () => this.applyFilters());
+            this.elements.confidenceFilter.addEventListener('input', () => {
+                this.elements.confidenceValue.textContent = this.elements.confidenceFilter.value;
+                this.applyFilters();
+            });
+            
             console.log('Event handlers set up successfully');
             
         } catch (error) {
@@ -228,9 +243,9 @@ class DNSReconApp {
     }
     
     /**
-     * Start a reconnaissance scan
+     * Start scan with error handling
      */
-    async startScan() {
+    async startScan(clearGraph = true) {
         console.log('=== STARTING SCAN ===');
         
         try {
@@ -262,7 +277,8 @@ class DNSReconApp {
             
             const requestData = {
                 target_domain: targetDomain,
-                max_depth: maxDepth
+                max_depth: maxDepth,
+                clear_graph: clearGraph
             };
             
             console.log('Request data:', requestData);
@@ -273,15 +289,17 @@ class DNSReconApp {
             
             if (response.success) {
                 this.currentSessionId = response.scan_id;
-                console.log('Starting polling with session ID:', this.currentSessionId);
-                this.startPolling();
                 this.showSuccess('Reconnaissance scan started successfully');
-                
-                // Clear previous graph
-                this.graphManager.clear();
+
+                if (clearGraph) {
+                    this.graphManager.clear();
+                }
                 
                 console.log(`Scan started for ${targetDomain} with depth ${maxDepth}`);
                 
+                // Start polling immediately with faster interval for responsiveness
+                this.startPolling(1000);
+                
                 // Force an immediate status update
                 console.log('Forcing immediate status update...');
                 setTimeout(() => {
@@ -299,18 +317,43 @@ class DNSReconApp {
             this.setUIState('idle');
         }
     }
-    
     /**
-     * Stop the current scan
+     * Scan stop with immediate UI feedback
      */
     async stopScan() {
         try {
             console.log('Stopping scan...');
+            
+            // Immediately disable stop button and show stopping state
+            if (this.elements.stopScan) {
+                this.elements.stopScan.disabled = true;
+                this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOPPING]</span><span>Stopping...</span>';
+            }
+            
+            // Show immediate feedback
+            this.showInfo('Stopping scan...');
+            
             const response = await this.apiCall('/api/scan/stop', 'POST');
             
             if (response.success) {
                 this.showSuccess('Scan stop requested');
-                console.log('Scan stop requested');
+                console.log('Scan stop requested successfully');
+                
+                // Force immediate status update
+                setTimeout(() => {
+                    this.updateStatus();
+                }, 100);
+                
+                // Continue polling for a bit to catch the status change
+                this.startPolling(500); // Fast polling to catch status change
+                
+                // Stop fast polling after 10 seconds
+                setTimeout(() => {
+                    if (this.scanStatus === 'stopped' || this.scanStatus === 'idle') {
+                        this.stopPolling();
+                    }
+                }, 10000);
+                
             } else {
                 throw new Error(response.error || 'Failed to stop scan');
             }
@@ -318,6 +361,12 @@ class DNSReconApp {
         } catch (error) {
             console.error('Failed to stop scan:', error);
             this.showError(`Failed to stop scan: ${error.message}`);
+            
+            // Re-enable stop button on error
+            if (this.elements.stopScan) {
+                this.elements.stopScan.disabled = false;
+                this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
+            }
         }
     }
     
@@ -346,9 +395,9 @@ class DNSReconApp {
     }
     
     /**
-     * Start polling for scan updates
+     * Start polling for scan updates with configurable interval
      */
-    startPolling() {
+    startPolling(interval = 2000) {
         console.log('=== STARTING POLLING ===');
         
         if (this.pollInterval) {
@@ -361,9 +410,9 @@ class DNSReconApp {
             this.updateStatus();
             this.updateGraph();
             this.loadProviders();
-        }, 1000); // Poll every 1 second for debugging
+        }, interval);
         
-        console.log('Polling started with 1 second interval');
+        console.log(`Polling started with ${interval}ms interval`);
     }
     
     /**
@@ -378,7 +427,7 @@ class DNSReconApp {
     }
     
     /**
-     * Update scan status from server
+     * Status update with better error handling
      */
     async updateStatus() {
         try {
@@ -387,7 +436,7 @@ class DNSReconApp {
             
             console.log('Status response:', response);
             
-            if (response.success) {
+            if (response.success && response.status) {
                 const status = response.status;
                 console.log('Current scan status:', status.status);
                 console.log('Current progress:', status.progress_percentage + '%');
@@ -404,6 +453,7 @@ class DNSReconApp {
                 this.scanStatus = status.status;
             } else {
                 console.error('Status update failed:', response);
+                // Don't show error for status updates to avoid spam
             }
             
         } catch (error) {
@@ -532,7 +582,7 @@ class DNSReconApp {
     }
     
     /**
-     * Handle status changes
+     * Handle status changes with improved state synchronization
      * @param {string} newStatus - New scan status
      */
     handleStatusChange(newStatus) {
@@ -542,8 +592,8 @@ class DNSReconApp {
             case 'running':
                 this.setUIState('scanning');
                 this.showSuccess('Scan is running');
-                // Reset polling frequency for active scans
-                this.pollFrequency = 2000;
+                // Increase polling frequency for active scans
+                this.startPolling(1000); // Poll every 1 second for running scans
                 this.updateConnectionStatus('active');
                 break;
                 
@@ -579,9 +629,13 @@ class DNSReconApp {
                 this.stopPolling();
                 this.updateConnectionStatus('idle');
                 break;
+                
+            default:
+                console.warn(`Unknown status: ${newStatus}`);
+                break;
         }
     }
-
+    
     /**
      * Update connection status indicator
      * @param {string} status - Connection status
@@ -614,8 +668,7 @@ class DNSReconApp {
     }
     
     /**
-     * Set UI state based on scan status
-     * @param {string} state - UI state
+     * UI state management with immediate button updates
      */
     setUIState(state) {
         console.log(`Setting UI state to: ${state}`);
@@ -626,10 +679,16 @@ class DNSReconApp {
                 if (this.elements.startScan) {
                     this.elements.startScan.disabled = true;
                     this.elements.startScan.classList.add('loading');
+                    this.elements.startScan.innerHTML = '<span class="btn-icon">[SCANNING]</span><span>Scanning...</span>';
+                }
+                if (this.elements.addToGraph) {
+                    this.elements.addToGraph.disabled = true;
+                    this.elements.addToGraph.classList.add('loading');
                 }
                 if (this.elements.stopScan) {
                     this.elements.stopScan.disabled = false;
                     this.elements.stopScan.classList.remove('loading');
+                    this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
                 }
                 if (this.elements.targetDomain) this.elements.targetDomain.disabled = true;
                 if (this.elements.maxDepth) this.elements.maxDepth.disabled = true;
@@ -644,9 +703,15 @@ class DNSReconApp {
                 if (this.elements.startScan) {
                     this.elements.startScan.disabled = false;
                     this.elements.startScan.classList.remove('loading');
+                    this.elements.startScan.innerHTML = '<span class="btn-icon">[RUN]</span><span>Start Reconnaissance</span>';
+                }
+                if (this.elements.addToGraph) {
+                    this.elements.addToGraph.disabled = false;
+                    this.elements.addToGraph.classList.remove('loading');
                 }
                 if (this.elements.stopScan) {
                     this.elements.stopScan.disabled = true;
+                    this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
                 }
                 if (this.elements.targetDomain) this.elements.targetDomain.disabled = false;
                 if (this.elements.maxDepth) this.elements.maxDepth.disabled = false;
@@ -665,6 +730,7 @@ class DNSReconApp {
             
             if (response.success) {
                 this.updateProviderDisplay(response.providers);
+                this.buildApiKeyModal(response.providers);
                 console.log('Providers loaded successfully');
             }
             
@@ -699,7 +765,7 @@ class DNSReconApp {
             
             providerItem.innerHTML = `
                 <div class="provider-header">
-                    <div class="provider-name">${name.toUpperCase()}</div>
+                    <div class="provider-name">${info.display_name}</div>
                     <div class="provider-status ${statusClass}">${statusText}</div>
                 </div>
                 <div class="provider-stats">
@@ -725,110 +791,180 @@ class DNSReconApp {
             this.elements.providerList.appendChild(providerItem);
         }
     }
+
+    /**
+     * Generates the HTML for the node details view using the new data model.
+     * @param {Object} node - The node object.
+     * @returns {string} The HTML string for the node details.
+     */
+    generateNodeDetailsHtml(node) {
+        if (!node) return '<div class="detail-row"><span class="detail-value">Details not available.</span></div>';
+
+        let detailsHtml = '<div class="modal-details-grid">';
+
+        // Handle merged correlation objects similar to large entities
+        if (node.type === 'correlation_object') {
+            const metadata = node.metadata || {};
+            const values = metadata.values || [];
+            const mergeCount = metadata.merge_count || 1;
+            
+            detailsHtml += '<div class="modal-section">';
+            detailsHtml += '<h4>Correlation Details</h4>';
+            
+            if (mergeCount > 1) {
+                detailsHtml += `<p><strong>Merged Correlations:</strong> ${mergeCount} values</p>`;
+                detailsHtml += '<div class="correlation-values-list">';
+                
+                values.forEach((value, index) => {
+                    detailsHtml += `<details class="correlation-value-details">`;
+                    detailsHtml += `<summary>Value ${index + 1}: ${typeof value === 'string' && value.length > 50 ? value.substring(0, 47) + '...' : value}</summary>`;
+                    detailsHtml += `<div class="detail-row"><span class="detail-label">Full Value:</span><span class="detail-value">${value}</span></div>`;
+                    detailsHtml += `</details>`;
+                });
+                
+                detailsHtml += '</div>';
+            } else {
+                const singleValue = values.length > 0 ? values[0] : (metadata.value || 'Unknown');
+                detailsHtml += `<div class="detail-row"><span class="detail-label">Correlation Value:</span><span class="detail-value">${singleValue}</span></div>`;
+            }
+            
+            // Show correlated nodes
+            const correlatedNodes = metadata.correlated_nodes || [];
+            if (correlatedNodes.length > 0) {
+                detailsHtml += `<div class="detail-row"><span class="detail-label">Correlated Nodes:</span><span class="detail-value">${correlatedNodes.length}</span></div>`;
+                detailsHtml += '<ul>';
+                correlatedNodes.forEach(nodeId => {
+                    detailsHtml += `<li><a href="#" class="node-link" data-node-id="${nodeId}">${nodeId}</a></li>`;
+                });
+                detailsHtml += '</ul>';
+            }
+            
+            detailsHtml += '</div>';
+        }
+
+        // Continue with standard node details for all node types
+        // Section for Incoming Edges (Source Nodes)
+        if (node.incoming_edges && node.incoming_edges.length > 0) {
+            detailsHtml += '<div class="modal-section">';
+            detailsHtml += '<h4>Source Nodes (Incoming)</h4>';
+            detailsHtml += '<ul>';
+            node.incoming_edges.forEach(edge => {
+                detailsHtml += `<li><a href="#" class="node-link" data-node-id="${edge.from}">${edge.from}</a> (${edge.data.relationship_type})</li>`;
+            });
+            detailsHtml += '</ul></div>';
+        }
+
+        // Section for Outgoing Edges (Destination Nodes)
+        if (node.outgoing_edges && node.outgoing_edges.length > 0) {
+            detailsHtml += '<div class="modal-section">';
+            detailsHtml += '<h4>Destination Nodes (Outgoing)</h4>';
+            detailsHtml += '<ul>';
+            node.outgoing_edges.forEach(edge => {
+                detailsHtml += `<li><a href="#" class="node-link" data-node-id="${edge.to}">${edge.to}</a> (${edge.data.relationship_type})</li>`;
+            });
+            detailsHtml += '</ul></div>';
+        }
+
+        // Section for Attributes (skip for correlation objects - already handled above)
+        if (node.type !== 'correlation_object') {
+            detailsHtml += '<div class="modal-section">';
+            detailsHtml += '<h4>Attributes</h4>';
+            detailsHtml += this.formatObjectToHtml(node.attributes);
+            detailsHtml += '</div>';
+        }
+
+        // Section for Description
+        detailsHtml += '<div class="modal-section">';
+        detailsHtml += '<h4>Description</h4>';
+        detailsHtml += `<p class="description-text">${node.description || 'No description available.'}</p>`;
+        detailsHtml += '</div>';
+
+        // Section for Metadata (skip detailed metadata for correlation objects - already handled above)
+        if (node.type !== 'correlation_object') {
+            detailsHtml += '<div class="modal-section">';
+            detailsHtml += '<h4>Metadata</h4>';
+            detailsHtml += this.formatObjectToHtml(node.metadata);
+            detailsHtml += '</div>';
+        }
+
+        detailsHtml += '</div>';
+        return detailsHtml;
+    }
+    
+    /**
+     * Recursively formats a JavaScript object into an HTML unordered list with collapsible sections.
+     * @param {Object} obj - The object to format.
+     * @returns {string} - An HTML string representing the object.
+     */
+    formatObjectToHtml(obj) {
+        if (!obj || Object.keys(obj).length === 0) {
+            return '<p class="no-data">No data available.</p>';
+        }
+    
+        let html = '<ul>';
+        for (const key in obj) {
+            if (Object.hasOwnProperty.call(obj, key)) {
+                const value = obj[key];
+                const formattedKey = key.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase());
+    
+                if (typeof value === 'object' && value !== null) {
+                    html += `<li><details><summary><strong>${formattedKey}</strong></summary>`;
+                    html += this.formatObjectToHtml(value);
+                    html += `</details></li>`;
+                } else {
+                    html += `<li><strong>${formattedKey}:</strong> ${this.formatValue(value)}</li>`;
+                }
+            }
+        }
+        html += '</ul>';
+        return html;
+    }
+
     
     /**
      * Show node details modal
-     * @param {string} nodeId - Node identifier
      * @param {Object} node - Node data
      */
-    showNodeModal(nodeId, node) {
-        if (!this.elements.nodeModal) return;
+    showNodeModal(node) {
+        if (!this.elements.nodeModal || !node) return;
 
         if (this.elements.modalTitle) {
-            this.elements.modalTitle.textContent = `Node Details: ${nodeId}`;
+            this.elements.modalTitle.textContent = `${this.formatStatus(node.type)} Node: ${node.id}`;
         }
-
+        
         let detailsHtml = '';
-        const createDetailRow = (label, value, statusIcon = '') => {
-            const baseId = `detail-${label.replace(/[^a-zA-Z0-9]/g, '-')}`;
+        if (node.type === 'large_entity') {
+            const attributes = node.attributes || {};
+            const nodes = attributes.nodes || [];
+            const node_type = attributes.node_type || 'nodes';
+            detailsHtml += `<div class="detail-section-header">Contains ${attributes.count} ${node_type}s</div>`;
+            detailsHtml += '<div class="large-entity-nodes-list">';
 
-            if (value === null || value === undefined ||
-                (Array.isArray(value) && value.length === 0) ||
-                (typeof value === 'object' && Object.keys(value).length === 0)) {
-                return `
-                    <div class="detail-row">
-                        <span class="detail-label">${label} <span class="status-icon text-warning">✗</span></span>
-                        <span class="detail-value">N/A</span>
-                    </div>
-                `;
+            for(const innerNodeId of nodes) {
+                const innerNode = this.graphManager.nodes.get(innerNodeId);
+                detailsHtml += `<details class="large-entity-node-details">`;
+                detailsHtml += `<summary>${innerNodeId}</summary>`;
+                detailsHtml += this.generateNodeDetailsHtml(innerNode);
+                detailsHtml += `</details>`;
             }
-
-            if (Array.isArray(value)) {
-                return value.map((item, index) => {
-                    const itemId = `${baseId}-${index}`;
-                    const itemLabel = index === 0 ? `${label} <span class="status-icon text-success">✓</span>` : '';
-                    return `
-                        <div class="detail-row">
-                            <span class="detail-label">${itemLabel}</span>
-                            <span class="detail-value" id="${itemId}">${this.formatValue(item)}</span>
-                            <button class="copy-btn" onclick="copyToClipboard('${itemId}')" title="Copy">📋</button>
-                        </div>
-                    `;
-                }).join('');
-            } else {
-                const valueId = `${baseId}-0`;
-                const icon = statusIcon || '<span class="status-icon text-success">✓</span>';
-                return `
-                    <div class="detail-row">
-                        <span class="detail-label">${label} ${icon}</span>
-                        <span class="detail-value" id="${valueId}">${this.formatValue(value)}</span>
-                        <button class="copy-btn" onclick="copyToClipboard('${valueId}')" title="Copy">📋</button>
-                    </div>
-                `;
-            }
-        };
-
-        const metadata = node.metadata || {};
-        
-        // General Node Info
-        detailsHtml += createDetailRow('Node Type', node.type);
-
-        // Display data based on node type
-        switch (node.type) {
-            case 'domain':
-                detailsHtml += createDetailRow('DNS Records', metadata.dns_records);
-                detailsHtml += createDetailRow('Related Domains (SAN)', metadata.related_domains_san);
-                detailsHtml += createDetailRow('Passive DNS', metadata.passive_dns);
-                detailsHtml += createDetailRow('Shodan Data', metadata.shodan);
-                detailsHtml += createDetailRow('VirusTotal Data', metadata.virustotal);
-                break;
-            case 'ip':
-                detailsHtml += createDetailRow('Hostnames', metadata.hostnames);
-                detailsHtml += createDetailRow('Passive DNS', metadata.passive_dns);
-                detailsHtml += createDetailRow('Shodan Data', metadata.shodan);
-                detailsHtml += createDetailRow('VirusTotal Data', metadata.virustotal);
-                break;
-        }
-        
-        // Special handling for certificate data
-        if (metadata.certificate_data && Object.keys(metadata.certificate_data).length > 0) {
-            const cert = metadata.certificate_data;
-            detailsHtml += `<div class="detail-section-header">Certificate Summary</div>`;
-            detailsHtml += createDetailRow('Total Found', cert.total_certificates);
-            detailsHtml += createDetailRow('Currently Valid', cert.valid_certificates);
-            detailsHtml += createDetailRow('Expires Soon (<30d)', cert.expires_soon_count);
-            detailsHtml += createDetailRow('Unique Issuers', cert.unique_issuers ? cert.unique_issuers.join(', ') : 'N/A');
-
-            if (cert.latest_certificate) {
-                detailsHtml += `<div class="detail-section-header">Latest Certificate</div>`;
-                detailsHtml += createDetailRow('Common Name', cert.latest_certificate.common_name);
-                detailsHtml += createDetailRow('Issuer', cert.latest_certificate.issuer_name);
-                detailsHtml += createDetailRow('Valid From', new Date(cert.latest_certificate.not_before).toLocaleString());
-                detailsHtml += createDetailRow('Valid Until', new Date(cert.latest_certificate.not_after).toLocaleString());
-            }
-        }
-        
-        // Special handling for ASN data
-        if (metadata.asn_data && Object.keys(metadata.asn_data).length > 0) {
-            detailsHtml += `<div class="detail-section-header">ASN Information</div>`;
-            detailsHtml += createDetailRow('ASN', metadata.asn_data.asn);
-            detailsHtml += createDetailRow('Organization', metadata.asn_data.description);
-            detailsHtml += createDetailRow('ISP', metadata.asn_data.isp);
-            detailsHtml += createDetailRow('Country', metadata.asn_data.country);
+            detailsHtml += '</div>';
+        } else {
+             detailsHtml = this.generateNodeDetailsHtml(node);
         }
 
         if (this.elements.modalDetails) {
             this.elements.modalDetails.innerHTML = detailsHtml;
+            this.elements.modalDetails.querySelectorAll('.node-link').forEach(link => {
+                link.addEventListener('click', (e) => {
+                    e.preventDefault();
+                    const nodeId = e.target.dataset.nodeId;
+                    const nextNode = this.graphManager.nodes.get(nodeId);
+                    if (nextNode) {
+                        this.hideModal();
+                        this.showNodeModal(nextNode);
+                    }
+                });
+            });
         }
         this.elements.nodeModal.style.display = 'block';
     }
@@ -864,12 +1000,15 @@ class DNSReconApp {
      * Save API Keys
      */
     async saveApiKeys() {
-        const shodanKey = this.elements.shodanApiKey.value.trim();
-        const virustotalKey = this.elements.virustotalApiKey.value.trim();
-
+        const inputs = this.elements.apiKeyInputs.querySelectorAll('input');
         const keys = {};
-        if (shodanKey) keys.shodan = shodanKey;
-        if (virustotalKey) keys.virustotal = virustotalKey;
+        inputs.forEach(input => {
+            const provider = input.dataset.provider;
+            const value = input.value.trim();
+            if (provider && value) {
+                keys[provider] = value;
+            }
+        });
 
         if (Object.keys(keys).length === 0) {
             this.showWarning('No API keys were entered.');
@@ -894,10 +1033,24 @@ class DNSReconApp {
      * Reset API Key fields
      */
     resetApiKeys() {
-        this.elements.shodanApiKey.value = '';
-        this.elements.virustotalApiKey.value = '';
+        const inputs = this.elements.apiKeyInputs.querySelectorAll('input');
+        inputs.forEach(input => {
+            input.value = '';
+        });
     }
     
+    /**
+     * Apply graph filters
+     */
+    applyFilters() {
+        if (this.graphManager) {
+            const nodeType = this.elements.nodeTypeFilter.value;
+            const minConfidence = parseFloat(this.elements.confidenceFilter.value);
+            this.graphManager.applyFilters(nodeType, minConfidence);
+        }
+    }
+
+    
     /**
      * Check if graph data has changed
      * @param {Object} graphData - New graph data
@@ -1180,6 +1333,74 @@ class DNSReconApp {
         };
         return colors[type] || colors.info;
     }
+
+    /**
+     * Build the API key modal dynamically
+     * @param {Object} providers - Provider information
+     */
+    buildApiKeyModal(providers) {
+        if (!this.elements.apiKeyInputs) return;
+        this.elements.apiKeyInputs.innerHTML = ''; // Clear existing inputs
+        let hasApiKeyProviders = false;
+
+        for (const [name, info] of Object.entries(providers)) {
+            if (info.requires_api_key) {
+                hasApiKeyProviders = true;
+                const inputGroup = document.createElement('div');
+                inputGroup.className = 'apikey-section';
+
+                if (info.enabled) {
+                    // If the API key is set and the provider is enabled
+                    inputGroup.innerHTML = `
+                        <label for="${name}-api-key">${info.display_name} API Key</label>
+                        <div class="api-key-set-message">
+                            <span class="api-key-set-text">API Key is set</span>
+                            <button class="clear-api-key-btn" data-provider="${name}">Clear</button>
+                        </div>
+                        <p class="apikey-help">Provides infrastructure context and service information.</p>
+                    `;
+                } else {
+                    // If the API key is not set
+                    inputGroup.innerHTML = `
+                        <label for="${name}-api-key">${info.display_name} API Key</label>
+                        <input type="password" id="${name}-api-key" data-provider="${name}" placeholder="Enter ${info.display_name} API Key">
+                        <p class="apikey-help">Provides infrastructure context and service information.</p>
+                    `;
+                }
+                this.elements.apiKeyInputs.appendChild(inputGroup);
+            }
+        }
+        
+        // Add event listeners for the new clear buttons
+        this.elements.apiKeyInputs.querySelectorAll('.clear-api-key-btn').forEach(button => {
+            button.addEventListener('click', (e) => {
+                const provider = e.target.dataset.provider;
+                this.clearApiKey(provider);
+            });
+        });
+
+        if (!hasApiKeyProviders) {
+            this.elements.apiKeyInputs.innerHTML = '<p>No providers require API keys.</p>';
+        }
+    }
+
+    /**
+     * Clear an API key for a specific provider
+     * @param {string} provider The name of the provider to clear the API key for
+     */
+    async clearApiKey(provider) {
+        try {
+            const response = await this.apiCall('/api/config/api-keys', 'POST', { [provider]: '' });
+            if (response.success) {
+                this.showSuccess(`API key for ${provider} has been cleared.`);
+                this.loadProviders(); // This will rebuild the modal with the updated state
+            } else {
+                throw new Error(response.error || 'Failed to clear API key');
+            }
+        } catch (error) {
+            this.showError(`Error clearing API key: ${error.message}`);
+        }
+    }
 }
 
 // Add CSS animations for message toasts

templates/index.html

@@ -52,6 +52,10 @@
                             <span class="btn-icon">[RUN]</span>
                             <span>Start Reconnaissance</span>
                         </button>
+                        <button id="add-to-graph" class="btn btn-primary">
+                            <span class="btn-icon">[ADD]</span>
+                            <span>Add to Graph</span>
+                        </button>
                         <button id="stop-scan" class="btn btn-secondary" disabled>
                             <span class="btn-icon">[STOP]</span>
                             <span>Terminate Scan</span>
@@ -109,8 +113,22 @@
                 <div class="panel-header">
                     <h2>Infrastructure Map</h2>
                     <div class="view-controls">
-                        <button id="reset-view" class="btn-icon-small" title="Reset View">[↻]</button>
-                        <button id="fit-view" class="btn-icon-small" title="Fit to Screen">[□]</button>
+                        <div class="filter-group">
+                            <label for="node-type-filter">Node Type:</label>
+                            <select id="node-type-filter">
+                                <option value="all">All</option>
+                                <option value="domain">Domain</option>
+                                <option value="ip">IP</option>
+                                <option value="asn">ASN</option>
+                                <option value="correlation_object">Correlation Object</option>
+                                <option value="large_entity">Large Entity</option>
+                            </select>
+                        </div>
+                        <div class="filter-group">
+                            <label for="confidence-filter">Min Confidence:</label>
+                            <input type="range" id="confidence-filter" min="0" max="1" step="0.1" value="0">
+                            <span id="confidence-value">0</span>
+                        </div>
                     </div>
                 </div>
                 
@@ -135,11 +153,11 @@
                     </div>
                     <div class="legend-item">
                         <div class="legend-color" style="background-color: #c7c7c7;"></div>
-                        <span>Certificates</span>
+                        <span>Domain (invalid cert)</span>
                     </div>
                     <div class="legend-item">
                         <div class="legend-color" style="background-color: #9d4edd;"></div>
-                        <span>DNS Records</span>
+                        <span>Correlation Objects</span>
                     </div>
                     <div class="legend-item">
                         <div class="legend-edge high-confidence"></div>
@@ -168,7 +186,7 @@
 
         <footer class="footer">
             <div class="footer-content">
-                <span>DNSRecon v1.0 - Phase 1 Implementation</span>
+                <span>v0.0.0rc</span>
                 <span class="footer-separator">|</span>
                 <span>Passive Infrastructure Reconnaissance</span>
                 <span class="footer-separator">|</span>
@@ -199,16 +217,8 @@
                     <p class="modal-description">
                         Enter your API keys for enhanced data providers. Keys are stored in memory for the current session only and are never saved to disk.
                     </p>
-                    <div class="apikey-section">
-                        <label for="virustotal-api-key">VirusTotal API Key</label>
-                        <input type="password" id="virustotal-api-key" placeholder="Enter VirusTotal API Key">
-                        <p class="apikey-help">Enables passive DNS and domain reputation lookups.</p>
-                    </div>
-                    <div class="apikey-section">
-                        <label for="shodan-api-key">Shodan API Key</label>
-                        <input type="password" id="shodan-api-key" placeholder="Enter Shodan API Key">
-                        <p class="apikey-help">Provides infrastructure context and service information.</p>
-                    </div>
+                    <div id="api-key-inputs">
+                        </div>
                     <div class="button-group" style="flex-direction: row; justify-content: flex-end;">
                         <button id="reset-api-keys" class="btn btn-secondary">
                             <span>Reset</span>