mstoeck3/dnscope
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: mstoeck3:571912218e6707cfcb05beea289f136084b1a7b2
Branches Tags
mstoeck3:main mstoeck3:websockets mstoeck3:database_crtsh mstoeck3:try-fix
..
compare: mstoeck3:websockets
Branches Tags
mstoeck3:main mstoeck3:websockets mstoeck3:database_crtsh mstoeck3:try-fix

2 Commits
571912218e ... websockets

Author SHA1 Message Date
overcuriousity
c4e6a8998a iteration on ws implementation 2025-09-20 16:52:05 +02:00
overcuriousity
75a595c9cb try to implement websockets 2025-09-20 14:17:17 +02:00
26 changed files with 1722 additions and 1525 deletions
Expand all files Collapse all files

4
.env.example
View File

@@ -1,5 +1,5 @@
# ===============================================
# DNScope Environment Variables
# DNSRecon Environment Variables
# ===============================================
# Copy this file to .env and fill in your values.
@@ -32,5 +32,3 @@ LARGE_ENTITY_THRESHOLD=100
MAX_RETRIES_PER_TARGET=8
# How long cached provider responses are stored (in hours).
CACHE_TIMEOUT_HOURS=12
GRAPH_POLLING_NODE_THRESHOLD=100

82
README.md
View File

@@ -1,18 +1,16 @@
# DNScope - Passive Infrastructure Reconnaissance Tool
# DNSRecon - Passive Infrastructure Reconnaissance Tool
DNScope is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys. It is aimed at cybersecurity researchers, pentesters, and administrators who want to understand the public footprint of a target domain.
**Repo Link:** [https://github.com/overcuriousity/DNScope](https://github.com/overcuriousity/DNScope)
**Repo Link:** [https://git.cc24.dev/mstoeck3/dnsrecon](https://git.cc24.dev/mstoeck3/dnsrecon)
-----
## Concept and Philosophy
The core philosophy of DNScope is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNScope will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
The core philosophy of DNSRecon is to provide a comprehensive and accurate map of a target's infrastructure using only **passive data sources** by default. This means that, out of the box, DNSRecon will not send any traffic to the target's servers. Instead, it queries public and historical data sources to build a picture of the target's online presence. This approach is ideal for researchers and pentesters who want to gather intelligence without alerting the target, and for administrators who want to see what information about their own infrastructure is publicly available.
For power users who require more in-depth information, DNScope can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
-----
For power users who require more in-depth information, DNSRecon can be configured to use API keys for services like Shodan, which provides a wealth of information about internet-connected devices. However, this is an optional feature, and the core functionality of the tool will always remain free and passive.
-----
@@ -26,15 +24,12 @@ For power users who require more in-depth information, DNScope can be configured
* **Session Management**: Supports concurrent user sessions with isolated scanner instances.
* **Extensible Provider Architecture**: Easily add new data sources to expand the tool's capabilities.
* **Web-Based UI**: An intuitive and interactive web interface for managing scans and visualizing results.
* **Export Options**: Export scan results to JSON, a list of targets to a text file, or an executive summary.
* **API Key Management**: Securely manage API keys for various providers through the web interface.
* **Provider Management**: Enable or disable providers for the current session.
-----
## Technical Architecture
DNScope is a web-based application built with a modern technology stack:
DNSRecon is a web-based application built with a modern technology stack:
* **Backend**: The backend is a **Flask** application that provides a REST API for the frontend and manages the scanning process.
* **Scanning Engine**: The core scanning engine is a multi-threaded Python application that uses a provider-based architecture to query different data sources.
@@ -46,7 +41,7 @@ DNScope is a web-based application built with a modern technology stack:
## Data Sources
DNScope queries the following data sources:
DNSRecon queries the following data sources:
* **DNS**: Standard DNS lookups (A, AAAA, CNAME, MX, NS, SOA, TXT).
* **crt.sh**: A certificate transparency log that provides information about SSL/TLS certificates.
@@ -61,43 +56,15 @@ DNScope queries the following data sources:
* Python 3.8 or higher
* A modern web browser with JavaScript enabled
* A Linux host for running the application
* Redis Server
### 1\. Install Redis
It is recommended to install Redis from the official repositories.
**On Debian/Ubuntu:**
### 1\. Clone the Project
```bash
sudo apt-get update
sudo apt-get install redis-server
git clone https://git.cc24.dev/mstoeck3/dnsrecon
cd dnsrecon
```
**On CentOS/RHEL:**
```bash
sudo yum install redis
sudo systemctl start redis
sudo systemctl enable redis
```
You can verify that Redis is running with the following command:
```bash
redis-cli ping
```
You should see `PONG` as the response.
### 2\. Clone the Project
```bash
git clone https://github.com/overcuriousity/DNScope
cd DNScope
```
### 3\. Install Python Dependencies
### 2\. Install Python Dependencies
It is highly recommended to use a virtual environment:
@@ -119,11 +86,10 @@ The `requirements.txt` file contains the following dependencies:
* gunicorn
* redis
* python-dotenv
* psycopg2-binary
### 4\. Configure the Application
### 3\. Configure the Application
DNScope is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
DNSRecon is configured using a `.env` file. You can copy the provided example file and edit it to suit your needs:
```bash
cp .env.example .env
@@ -167,30 +133,30 @@ gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
## Systemd Service
To run DNScope as a service that starts automatically on boot, you can use `systemd`.
To run DNSRecon as a service that starts automatically on boot, you can use `systemd`.
### 1\. Create a `.service` file
Create a new service file in `/etc/systemd/system/`:
```bash
sudo nano /etc/systemd/system/DNScope.service
sudo nano /etc/systemd/system/dnsrecon.service
```
### 2\. Add the Service Configuration
Paste the following configuration into the file. **Remember to replace `/path/to/your/DNScope` and `your_user` with your actual project path and username.**
Paste the following configuration into the file. **Remember to replace `/path/to/your/dnsrecon` and `your_user` with your actual project path and username.**
```ini
[Unit]
Description=DNScope Application
Description=DNSRecon Application
After=network.target
[Service]
User=your_user
Group=your_user
WorkingDirectory=/path/to/your/DNScope
ExecStart=/path/to/your/DNScope/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
WorkingDirectory=/path/to/your/dnsrecon
ExecStart=/path/to/your/dnsrecon/venv/bin/gunicorn --workers 4 --bind 0.0.0.0:5000 app:app
Restart=always
Environment="SECRET_KEY=your-super-secret-and-random-key"
Environment="FLASK_ENV=production"
@@ -207,14 +173,14 @@ Reload the `systemd` daemon, enable the service to start on boot, and then start
```bash
sudo systemctl daemon-reload
sudo systemctl enable DNScope.service
sudo systemctl start DNScope.service
sudo systemctl enable dnsrecon.service
sudo systemctl start dnsrecon.service
```
You can check the status of the service at any time with:
```bash
sudo systemctl status DNScope.service
sudo systemctl status dnsrecon.service
```
-----
@@ -244,14 +210,14 @@ rm -rf cache/*
### 4\. Restart the Service
```bash
sudo systemctl restart DNScope.service
sudo systemctl restart dnsrecon.service
```
-----
## Extensibility
DNScope is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
DNSRecon is designed to be extensible, and adding new providers is a straightforward process. To add a new provider, you will need to create a new Python file in the `providers` directory that inherits from the `BaseProvider` class. The new provider will need to implement the following methods:
* `get_name()`: Return the name of the provider.
* `get_display_name()`: Return a display-friendly name for the provider.

242
app.py
View File

@@ -1,12 +1,11 @@
# DNScope-reduced/app.py
# dnsrecon-reduced/app.py
"""
Flask application entry point for DNScope web interface.
Flask application entry point for DNSRecon web interface.
Provides REST API endpoints and serves the web interface with user session support.
UPDATED: Added /api/config endpoint for graph polling optimization settings.
FIXED: Enhanced WebSocket integration with proper connection management.
"""
import json
import traceback
from flask import Flask, render_template, request, jsonify, send_file, session
from datetime import datetime, timezone, timedelta
@@ -14,6 +13,7 @@ import io
import os
from core.session_manager import session_manager
from flask_socketio import SocketIO
from config import config
from core.graph_manager import NodeType
from utils.helpers import is_valid_target
@@ -22,29 +22,38 @@ from decimal import Decimal
app = Flask(__name__)
socketio = SocketIO(app, cors_allowed_origins="*")
app.config['SECRET_KEY'] = config.flask_secret_key
app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(hours=config.flask_permanent_session_lifetime_hours)
def get_user_scanner():
"""
Retrieves the scanner for the current session, or creates a new one if none exists.
FIXED: Retrieves the scanner for the current session with proper socketio management.
"""
current_flask_session_id = session.get('DNScope_session_id')
current_flask_session_id = session.get('dnsrecon_session_id')
if current_flask_session_id:
existing_scanner = session_manager.get_session(current_flask_session_id)
if existing_scanner:
# FIXED: Ensure socketio is properly maintained
existing_scanner.socketio = socketio
print(f"✓ Retrieved existing scanner for session {current_flask_session_id[:8]}... with socketio restored")
return current_flask_session_id, existing_scanner
new_session_id = session_manager.create_session()
# FIXED: Register socketio connection when creating new session
new_session_id = session_manager.create_session(socketio)
new_scanner = session_manager.get_session(new_session_id)
if not new_scanner:
raise Exception("Failed to create new scanner session")
session['DNScope_session_id'] = new_session_id
# FIXED: Ensure new scanner has socketio reference and register the connection
new_scanner.socketio = socketio
session_manager.register_socketio_connection(new_session_id, socketio)
session['dnsrecon_session_id'] = new_session_id
session.permanent = True
print(f"✓ Created new scanner for session {new_session_id[:8]}... with socketio registered")
return new_session_id, new_scanner
@@ -54,25 +63,10 @@ def index():
return render_template('index.html')
@app.route('/api/config', methods=['GET'])
def get_config():
"""Get configuration settings for frontend."""
try:
return jsonify({
'success': True,
'config': {
'graph_polling_node_threshold': config.graph_polling_node_threshold
}
})
except Exception as e:
traceback.print_exc()
return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
@app.route('/api/scan/start', methods=['POST'])
def start_scan():
"""
Starts a new reconnaissance scan.
FIXED: Starts a new reconnaissance scan with proper socketio management.
"""
try:
data = request.get_json()
@@ -96,9 +90,17 @@ def start_scan():
if not scanner:
return jsonify({'success': False, 'error': 'Failed to get scanner instance.'}), 500
# FIXED: Ensure scanner has socketio reference and is registered
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
print(f"🚀 Starting scan for {target} with socketio enabled and registered")
success = scanner.start_scan(target, max_depth, clear_graph=clear_graph, force_rescan_target=force_rescan_target)
if success:
# Update session with socketio-enabled scanner
session_manager.update_session_scanner(user_session_id, scanner)
return jsonify({
'success': True,
'message': 'Reconnaissance scan started successfully',
@@ -127,6 +129,10 @@ def stop_scan():
if not scanner.session_id:
scanner.session_id = user_session_id
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
scanner.stop_scan()
session_manager.set_stop_signal(user_session_id)
session_manager.update_scanner_status(user_session_id, 'stopped')
@@ -143,37 +149,83 @@ def stop_scan():
return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
@app.route('/api/scan/status', methods=['GET'])
@socketio.on('connect')
def handle_connect():
"""
FIXED: Handle WebSocket connection with proper session management.
"""
print(f'✓ WebSocket client connected: {request.sid}')
# Try to restore existing session connection
current_flask_session_id = session.get('dnsrecon_session_id')
if current_flask_session_id:
# Register this socketio connection for the existing session
session_manager.register_socketio_connection(current_flask_session_id, socketio)
print(f'✓ Registered WebSocket for existing session: {current_flask_session_id[:8]}...')
# Immediately send current status to new connection
get_scan_status()
@socketio.on('disconnect')
def handle_disconnect():
"""
FIXED: Handle WebSocket disconnection gracefully.
"""
print(f'✗ WebSocket client disconnected: {request.sid}')
# Note: We don't immediately remove the socketio connection from session_manager
# because the user might reconnect. The cleanup will happen during session cleanup.
@socketio.on('get_status')
def get_scan_status():
"""Get current scan status."""
"""
FIXED: Get current scan status and emit real-time update with proper error handling.
"""
try:
user_session_id, scanner = get_user_scanner()
if not scanner:
return jsonify({
'success': True,
'status': {
'status': 'idle', 'target_domain': None, 'current_depth': 0,
'max_depth': 0, 'progress_percentage': 0.0,
'user_session_id': user_session_id
status = {
'status': 'idle',
'target_domain': None,
'current_depth': 0,
'max_depth': 0,
'progress_percentage': 0.0,
'user_session_id': user_session_id,
'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
}
})
print(f"📡 Emitting idle status for session {user_session_id[:8] if user_session_id else 'none'}...")
else:
if not scanner.session_id:
scanner.session_id = user_session_id
# FIXED: Ensure scanner has socketio reference for future updates
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
status = scanner.get_scan_status()
status['user_session_id'] = user_session_id
return jsonify({'success': True, 'status': status})
print(f"📡 Emitting status update: {status['status']} - "
f"Nodes: {len(status.get('graph', {}).get('nodes', []))}, "
f"Edges: {len(status.get('graph', {}).get('edges', []))}")
# Update session with socketio-enabled scanner
session_manager.update_session_scanner(user_session_id, scanner)
socketio.emit('scan_update', status)
except Exception as e:
traceback.print_exc()
return jsonify({
'success': False, 'error': f'Internal server error: {str(e)}',
'fallback_status': {'status': 'error', 'progress_percentage': 0.0}
}), 500
error_status = {
'status': 'error',
'message': 'Failed to get status',
'graph': {'nodes': [], 'edges': [], 'statistics': {'node_count': 0, 'edge_count': 0}}
}
print(f"⚠️ Error getting status, emitting error status")
socketio.emit('scan_update', error_status)
@app.route('/api/graph', methods=['GET'])
@@ -190,6 +242,10 @@ def get_graph_data():
if not scanner:
return jsonify({'success': True, 'graph': empty_graph, 'user_session_id': user_session_id})
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
graph_data = scanner.get_graph_data() or empty_graph
return jsonify({'success': True, 'graph': graph_data, 'user_session_id': user_session_id})
@@ -203,9 +259,7 @@ def get_graph_data():
@app.route('/api/graph/large-entity/extract', methods=['POST'])
def extract_from_large_entity():
"""
FIXED: Extract a node from a large entity with proper error handling.
"""
"""Extract a node from a large entity."""
try:
data = request.get_json()
large_entity_id = data.get('large_entity_id')
@@ -218,66 +272,21 @@ def extract_from_large_entity():
if not scanner:
return jsonify({'success': False, 'error': 'No active session found'}), 404
# FIXED: Check if node exists and provide better error messages
if not scanner.graph.graph.has_node(node_id):
return jsonify({
'success': False,
'error': f'Node {node_id} not found in graph'
}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
# FIXED: Check if node is actually part of the large entity
node_data = scanner.graph.graph.nodes[node_id]
metadata = node_data.get('metadata', {})
current_large_entity = metadata.get('large_entity_id')
if not current_large_entity:
return jsonify({
'success': False,
'error': f'Node {node_id} is not part of any large entity'
}), 400
if current_large_entity != large_entity_id:
return jsonify({
'success': False,
'error': f'Node {node_id} belongs to large entity {current_large_entity}, not {large_entity_id}'
}), 400
# FIXED: Check if large entity exists
if not scanner.graph.graph.has_node(large_entity_id):
return jsonify({
'success': False,
'error': f'Large entity {large_entity_id} not found'
}), 404
# Perform the extraction
success = scanner.extract_node_from_large_entity(large_entity_id, node_id)
if success:
# Force immediate session state update
session_manager.update_session_scanner(user_session_id, scanner)
return jsonify({
'success': True,
'message': f'Node {node_id} extracted successfully from {large_entity_id}.',
'extracted_node': node_id,
'large_entity': large_entity_id
})
return jsonify({'success': True, 'message': f'Node {node_id} extracted successfully.'})
else:
# This should not happen with the improved checks above, but handle it gracefully
return jsonify({
'success': False,
'error': f'Failed to extract node {node_id} from {large_entity_id}. Node may have already been extracted.'
}), 409
return jsonify({'success': False, 'error': f'Failed to extract node {node_id}.'}), 500
except json.JSONDecodeError:
return jsonify({'success': False, 'error': 'Invalid JSON in request body'}), 400
except Exception as e:
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}',
'error_type': type(e).__name__
}), 500
return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
@app.route('/api/graph/node/<node_id>', methods=['DELETE'])
def delete_graph_node(node_id):
@@ -287,6 +296,10 @@ def delete_graph_node(node_id):
if not scanner:
return jsonify({'success': False, 'error': 'No active session found'}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
success = scanner.graph.remove_node(node_id)
if success:
@@ -312,6 +325,10 @@ def revert_graph_action():
if not scanner:
return jsonify({'success': False, 'error': 'No active session found'}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
action_type = data['type']
action_data = data['data']
@@ -356,6 +373,10 @@ def export_results():
if not scanner:
return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
# Get export data using the new export manager
try:
results = export_manager.export_scan_results(scanner)
@@ -407,6 +428,10 @@ def export_targets():
if not scanner:
return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
# Use export manager for targets export
targets_txt = export_manager.export_targets_list(scanner)
@@ -437,6 +462,10 @@ def export_summary():
if not scanner:
return jsonify({'success': False, 'error': 'No active scanner session found'}), 404
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
# Use export manager for summary generation
summary_txt = export_manager.generate_executive_summary(scanner)
@@ -469,6 +498,10 @@ def set_api_keys():
user_session_id, scanner = get_user_scanner()
session_config = scanner.config
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
updated_providers = []
for provider_name, api_key in data.items():
@@ -501,6 +534,10 @@ def get_providers():
user_session_id, scanner = get_user_scanner()
base_provider_info = scanner.get_provider_info()
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
# Enhance provider info with API key source information
enhanced_provider_info = {}
@@ -565,6 +602,10 @@ def configure_providers():
user_session_id, scanner = get_user_scanner()
session_config = scanner.config
# FIXED: Ensure scanner has socketio reference
scanner.socketio = socketio
session_manager.register_socketio_connection(user_session_id, socketio)
updated_providers = []
for provider_name, settings in data.items():
@@ -593,7 +634,6 @@ def configure_providers():
return jsonify({'success': False, 'error': f'Internal server error: {str(e)}'}), 500
@app.errorhandler(404)
def not_found(error):
"""Handle 404 errors."""
@@ -609,9 +649,9 @@ def internal_error(error):
if __name__ == '__main__':
config.load_from_env()
app.run(
host=config.flask_host,
port=config.flask_port,
debug=config.flask_debug,
threaded=True
)
print("🚀 Starting DNSRecon with enhanced WebSocket support...")
print(f" Host: {config.flask_host}")
print(f" Port: {config.flask_port}")
print(f" Debug: {config.flask_debug}")
print(" WebSocket: Enhanced connection management enabled")
socketio.run(app, host=config.flask_host, port=config.flask_port, debug=config.flask_debug)

12
config.py
View File

@@ -1,7 +1,7 @@
# DNScope-reduced/config.py
# dnsrecon-reduced/config.py
"""
Configuration management for DNScope tool.
Configuration management for DNSRecon tool.
Handles API key storage, rate limiting, and default settings.
"""
@@ -13,7 +13,7 @@ from dotenv import load_dotenv
load_dotenv()
class Config:
"""Configuration manager for DNScope application."""
"""Configuration manager for DNSRecon application."""
def __init__(self):
"""Initialize configuration with default values."""
@@ -26,9 +26,6 @@ class Config:
self.large_entity_threshold = 100
self.max_retries_per_target = 8
# --- Graph Polling Performance Settings ---
self.graph_polling_node_threshold = 100 # Stop graph auto-polling above this many nodes
# --- Provider Caching Settings ---
self.cache_timeout_hours = 6 # Provider-specific cache timeout
@@ -75,9 +72,6 @@ class Config:
self.max_retries_per_target = int(os.getenv('MAX_RETRIES_PER_TARGET', self.max_retries_per_target))
self.cache_timeout_hours = int(os.getenv('CACHE_TIMEOUT_HOURS', self.cache_timeout_hours))
# Override graph polling threshold from environment
self.graph_polling_node_threshold = int(os.getenv('GRAPH_POLLING_NODE_THRESHOLD', self.graph_polling_node_threshold))
# Override Flask and session settings
self.flask_host = os.getenv('FLASK_HOST', self.flask_host)
self.flask_port = int(os.getenv('FLASK_PORT', self.flask_port))

2
core/__init__.py
View File

@@ -1,5 +1,5 @@
"""
Core modules for DNScope passive reconnaissance tool.
Core modules for DNSRecon passive reconnaissance tool.
Contains graph management, scanning orchestration, and forensic logging.
"""

61
core/graph_manager.py
View File

@@ -1,11 +1,10 @@
# DNScope-reduced/core/graph_manager.py
# dnsrecon-reduced/core/graph_manager.py
"""
Graph data model for DNScope using NetworkX.
Graph data model for DNSRecon using NetworkX.
Manages in-memory graph storage with confidence scoring and forensic metadata.
Now fully compatible with the unified ProviderResult data model.
UPDATED: Fixed correlation exclusion keys to match actual attribute names.
UPDATED: Removed export_json() method - now handled by ExportManager.
FIXED: Added proper pickle support to prevent weakref serialization errors.
"""
import re
from datetime import datetime, timezone
@@ -30,9 +29,10 @@ class NodeType(Enum):
class GraphManager:
"""
Thread-safe graph manager for DNScope infrastructure mapping.
Thread-safe graph manager for DNSRecon infrastructure mapping.
Uses NetworkX for in-memory graph storage with confidence scoring.
Compatible with unified ProviderResult data model.
FIXED: Added proper pickle support to handle NetworkX graph serialization.
"""
def __init__(self):
@@ -41,6 +41,57 @@ class GraphManager:
self.creation_time = datetime.now(timezone.utc).isoformat()
self.last_modified = self.creation_time
def __getstate__(self):
"""Prepare GraphManager for pickling by converting NetworkX graph to serializable format."""
state = self.__dict__.copy()
# Convert NetworkX graph to a serializable format
if hasattr(self, 'graph') and self.graph:
# Extract all nodes with their data
nodes_data = {}
for node_id, attrs in self.graph.nodes(data=True):
nodes_data[node_id] = dict(attrs)
# Extract all edges with their data
edges_data = []
for source, target, attrs in self.graph.edges(data=True):
edges_data.append({
'source': source,
'target': target,
'attributes': dict(attrs)
})
# Replace the NetworkX graph with serializable data
state['_graph_nodes'] = nodes_data
state['_graph_edges'] = edges_data
del state['graph']
return state
def __setstate__(self, state):
"""Restore GraphManager after unpickling by reconstructing NetworkX graph."""
# Restore basic attributes
self.__dict__.update(state)
# Reconstruct NetworkX graph from serializable data
self.graph = nx.DiGraph()
# Restore nodes
if hasattr(self, '_graph_nodes'):
for node_id, attrs in self._graph_nodes.items():
self.graph.add_node(node_id, **attrs)
del self._graph_nodes
# Restore edges
if hasattr(self, '_graph_edges'):
for edge_data in self._graph_edges:
self.graph.add_edge(
edge_data['source'],
edge_data['target'],
**edge_data['attributes']
)
del self._graph_edges
def add_node(self, node_id: str, node_type: NodeType, attributes: Optional[List[Dict[str, Any]]] = None,
description: str = "", metadata: Optional[Dict[str, Any]] = None) -> bool:
"""

99
core/logger.py
View File

@@ -1,4 +1,4 @@
# DNScope/core/logger.py
# dnsrecon/core/logger.py
import logging
import threading
@@ -38,8 +38,9 @@ class RelationshipDiscovery:
class ForensicLogger:
"""
Thread-safe forensic logging system for DNScope.
Thread-safe forensic logging system for DNSRecon.
Maintains detailed audit trail of all reconnaissance activities.
FIXED: Enhanced pickle support to prevent weakref issues in logging handlers.
"""
def __init__(self, session_id: str = ""):
@@ -65,49 +66,78 @@ class ForensicLogger:
'target_domains': set()
}
# Configure standard logger
self.logger = logging.getLogger(f'DNScope.{self.session_id}')
# Configure standard logger with simple setup to avoid weakrefs
self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
self.logger.setLevel(logging.INFO)
# Create formatter for structured logging
# Create minimal formatter
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Add console handler if not already present
# Add console handler only if not already present (avoid duplicate handlers)
if not self.logger.handlers:
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
self.logger.addHandler(console_handler)
def __getstate__(self):
"""Prepare ForensicLogger for pickling by excluding unpicklable objects."""
"""
FIXED: Prepare ForensicLogger for pickling by excluding problematic objects.
"""
state = self.__dict__.copy()
# Remove the unpickleable 'logger' attribute
if 'logger' in state:
del state['logger']
if 'lock' in state:
del state['lock']
# Remove potentially unpickleable attributes that may contain weakrefs
unpicklable_attrs = ['logger', 'lock']
for attr in unpicklable_attrs:
if attr in state:
del state[attr]
# Convert sets to lists for JSON serialization compatibility
if 'session_metadata' in state:
metadata = state['session_metadata'].copy()
if 'providers_used' in metadata and isinstance(metadata['providers_used'], set):
metadata['providers_used'] = list(metadata['providers_used'])
if 'target_domains' in metadata and isinstance(metadata['target_domains'], set):
metadata['target_domains'] = list(metadata['target_domains'])
state['session_metadata'] = metadata
return state
def __setstate__(self, state):
"""Restore ForensicLogger after unpickling by reconstructing logger."""
"""
FIXED: Restore ForensicLogger after unpickling by reconstructing components.
"""
self.__dict__.update(state)
# Re-initialize the 'logger' attribute
self.logger = logging.getLogger(f'DNScope.{self.session_id}')
# Re-initialize threading lock
self.lock = threading.Lock()
# Re-initialize logger with minimal setup
self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
self.logger.setLevel(logging.INFO)
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Only add handler if not already present
if not self.logger.handlers:
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
self.logger.addHandler(console_handler)
self.lock = threading.Lock()
# Convert lists back to sets if needed
if 'session_metadata' in self.__dict__:
metadata = self.session_metadata
if 'providers_used' in metadata and isinstance(metadata['providers_used'], list):
metadata['providers_used'] = set(metadata['providers_used'])
if 'target_domains' in metadata and isinstance(metadata['target_domains'], list):
metadata['target_domains'] = set(metadata['target_domains'])
def _generate_session_id(self) -> str:
"""Generate unique session identifier."""
return f"DNScope_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
return f"dnsrecon_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
def log_api_request(self, provider: str, url: str, method: str = "GET",
status_code: Optional[int] = None,
@@ -143,6 +173,7 @@ class ForensicLogger:
discovery_context=discovery_context
)
with self.lock:
self.api_requests.append(api_request)
self.session_metadata['total_requests'] += 1
self.session_metadata['providers_used'].add(provider)
@@ -150,11 +181,15 @@ class ForensicLogger:
if target_indicator:
self.session_metadata['target_domains'].add(target_indicator)
# Log to standard logger
# Log to standard logger with error handling
try:
if error:
self.logger.error(f"API Request Failed.")
self.logger.error(f"API Request Failed - {provider}: {url}")
else:
self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
except Exception:
# If logging fails, continue without breaking the application
pass
def log_relationship_discovery(self, source_node: str, target_node: str,
relationship_type: str, confidence_score: float,
@@ -183,29 +218,44 @@ class ForensicLogger:
discovery_method=discovery_method
)
with self.lock:
self.relationships.append(relationship)
self.session_metadata['total_relationships'] += 1
# Log to standard logger with error handling
try:
self.logger.info(
f"Relationship Discovered - {source_node} -> {target_node} "
f"({relationship_type}) - Confidence: {confidence_score:.2f} - Provider: {provider}"
)
except Exception:
# If logging fails, continue without breaking the application
pass
def log_scan_start(self, target_domain: str, recursion_depth: int,
enabled_providers: List[str]) -> None:
"""Log the start of a reconnaissance scan."""
try:
self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
self.session_metadata['target_domains'].update(target_domain)
with self.lock:
self.session_metadata['target_domains'].add(target_domain)
except Exception:
pass
def log_scan_complete(self) -> None:
"""Log the completion of a reconnaissance scan."""
with self.lock:
self.session_metadata['end_time'] = datetime.now(timezone.utc).isoformat()
# Convert sets to lists for serialization
self.session_metadata['providers_used'] = list(self.session_metadata['providers_used'])
self.session_metadata['target_domains'] = list(self.session_metadata['target_domains'])
try:
self.logger.info(f"Scan Complete - Session: {self.session_id}")
except Exception:
pass
def export_audit_trail(self) -> Dict[str, Any]:
"""
@@ -214,6 +264,7 @@ class ForensicLogger:
Returns:
Dictionary containing complete session audit trail
"""
with self.lock:
return {
'session_metadata': self.session_metadata.copy(),
'api_requests': [asdict(req) for req in self.api_requests],
@@ -229,7 +280,13 @@ class ForensicLogger:
Dictionary containing summary statistics
"""
provider_stats = {}
for provider in self.session_metadata['providers_used']:
# Ensure providers_used is a set for iteration
providers_used = self.session_metadata['providers_used']
if isinstance(providers_used, list):
providers_used = set(providers_used)
for provider in providers_used:
provider_requests = [req for req in self.api_requests if req.provider == provider]
provider_relationships = [rel for rel in self.relationships if rel.provider == provider]

4
core/provider_result.py
View File

@@ -1,7 +1,7 @@
# DNScope-reduced/core/provider_result.py
# dnsrecon-reduced/core/provider_result.py
"""
Unified data model for DNScope passive reconnaissance.
Unified data model for DNSRecon passive reconnaissance.
Standardizes the data structure across all providers to ensure consistent processing.
"""

143
core/rate_limiter.py
View File

@@ -1,145 +1,28 @@
# DNScope-reduced/core/rate_limiter.py
# dnsrecon-reduced/core/rate_limiter.py
import time
import logging
class GlobalRateLimiter:
"""
FIXED: Improved rate limiter with better cleanup and error handling.
Prevents accumulation of stale entries that cause infinite retry loops.
"""
def __init__(self, redis_client):
self.redis = redis_client
self.logger = logging.getLogger('DNScope.rate_limiter')
# Track last cleanup times to avoid excessive Redis operations
self._last_cleanup = {}
def is_rate_limited(self, key, limit, period):
"""
FIXED: Check if a key is rate-limited with improved cleanup and error handling.
Args:
key: Rate limit key (e.g., provider name)
limit: Maximum requests allowed
period: Time period in seconds (60 for per-minute)
Returns:
bool: True if rate limited, False otherwise
"""
if limit <= 0:
# Rate limit of 0 or negative means no limiting
return False
now = time.time()
rate_key = f"rate_limit:{key}"
try:
# FIXED: More aggressive cleanup to prevent accumulation
# Only clean up if we haven't cleaned recently (every 10 seconds max)
should_cleanup = (
rate_key not in self._last_cleanup or
now - self._last_cleanup.get(rate_key, 0) > 10
)
if should_cleanup:
# Remove entries older than the period
removed_count = self.redis.zremrangebyscore(rate_key, 0, now - period)
self._last_cleanup[rate_key] = now
if removed_count > 0:
self.logger.debug(f"Rate limiter cleaned up {removed_count} old entries for {key}")
# Get current count
current_count = self.redis.zcard(rate_key)
if current_count >= limit:
self.logger.debug(f"Rate limited: {key} has {current_count}/{limit} requests in period")
return True
# Add new timestamp with error handling
try:
# Use pipeline for atomic operations
pipe = self.redis.pipeline()
pipe.zadd(rate_key, {str(now): now})
pipe.expire(rate_key, int(period * 2)) # Set TTL to 2x period for safety
pipe.execute()
except Exception as e:
self.logger.warning(f"Failed to record rate limit entry for {key}: {e}")
# Don't block the request if we can't record it
return False
return False
except Exception as e:
self.logger.error(f"Rate limiter error for {key}: {e}")
# FIXED: On Redis errors, don't block requests to avoid infinite loops
return False
def get_rate_limit_status(self, key, limit, period):
"""
Get detailed rate limit status for debugging.
Returns:
dict: Status information including current count, limit, and time to reset
Check if a key is rate-limited.
"""
now = time.time()
rate_key = f"rate_limit:{key}"
key = f"rate_limit:{key}"
try:
current_count = self.redis.zcard(rate_key)
# Remove old timestamps
self.redis.zremrangebyscore(key, 0, now - period)
# Get oldest entry to calculate reset time
oldest_entries = self.redis.zrange(rate_key, 0, 0, withscores=True)
time_to_reset = 0
if oldest_entries:
oldest_time = oldest_entries[0][1]
time_to_reset = max(0, period - (now - oldest_time))
return {
'key': key,
'current_count': current_count,
'limit': limit,
'period': period,
'is_limited': current_count >= limit,
'time_to_reset': time_to_reset
}
except Exception as e:
self.logger.error(f"Failed to get rate limit status for {key}: {e}")
return {
'key': key,
'current_count': 0,
'limit': limit,
'period': period,
'is_limited': False,
'time_to_reset': 0,
'error': str(e)
}
def reset_rate_limit(self, key):
"""
ADDED: Reset rate limit for a specific key (useful for debugging).
"""
rate_key = f"rate_limit:{key}"
try:
deleted = self.redis.delete(rate_key)
self.logger.info(f"Reset rate limit for {key} (deleted: {deleted})")
# Check the count
count = self.redis.zcard(key)
if count >= limit:
return True
except Exception as e:
self.logger.error(f"Failed to reset rate limit for {key}: {e}")
return False
def cleanup_all_rate_limits(self):
"""
ADDED: Clean up all rate limit entries (useful for maintenance).
"""
try:
keys = self.redis.keys("rate_limit:*")
if keys:
deleted = self.redis.delete(*keys)
self.logger.info(f"Cleaned up {deleted} rate limit keys")
return deleted
return 0
except Exception as e:
self.logger.error(f"Failed to cleanup rate limits: {e}")
return 0
# Add new timestamp
self.redis.zadd(key, {now: now})
self.redis.expire(key, period)
return False

785
core/scanner.py
View File

File diff suppressed because it is too large Load Diff

2
core/session_config.py
View File

@@ -1,5 +1,5 @@
"""
Per-session configuration management for DNScope.
Per-session configuration management for DNSRecon.
Provides isolated configuration instances for each user session.
"""

182
core/session_manager.py
View File

@@ -1,4 +1,4 @@
# DNScope/core/session_manager.py
# dnsrecon/core/session_manager.py
import threading
import time
@@ -6,6 +6,7 @@ import uuid
import redis
import pickle
from typing import Dict, Optional, Any
import copy
from core.scanner import Scanner
from config import config
@@ -13,7 +14,7 @@ from config import config
class SessionManager:
"""
FIXED: Manages multiple scanner instances for concurrent user sessions using Redis.
Now more conservative about session creation to preserve API keys and configuration.
Enhanced to properly maintain WebSocket connections throughout scan lifecycle.
"""
def __init__(self, session_timeout_minutes: int = 0):
@@ -30,6 +31,9 @@ class SessionManager:
# FIXED: Add a creation lock to prevent race conditions
self.creation_lock = threading.Lock()
# Track active socketio connections per session
self.active_socketio_connections = {}
# Start cleanup thread
self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
self.cleanup_thread.start()
@@ -40,7 +44,7 @@ class SessionManager:
"""Prepare SessionManager for pickling."""
state = self.__dict__.copy()
# Exclude unpickleable attributes - Redis client and threading objects
unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock']
unpicklable_attrs = ['lock', 'cleanup_thread', 'redis_client', 'creation_lock', 'active_socketio_connections']
for attr in unpicklable_attrs:
if attr in state:
del state[attr]
@@ -53,33 +57,82 @@ class SessionManager:
self.redis_client = redis.StrictRedis(db=0, decode_responses=False)
self.lock = threading.Lock()
self.creation_lock = threading.Lock()
self.active_socketio_connections = {}
self.cleanup_thread = threading.Thread(target=self._cleanup_loop, daemon=True)
self.cleanup_thread.start()
def _get_session_key(self, session_id: str) -> str:
"""Generates the Redis key for a session."""
return f"DNScope:session:{session_id}"
return f"dnsrecon:session:{session_id}"
def _get_stop_signal_key(self, session_id: str) -> str:
"""Generates the Redis key for a session's stop signal."""
return f"DNScope:stop:{session_id}"
return f"dnsrecon:stop:{session_id}"
def create_session(self) -> str:
def register_socketio_connection(self, session_id: str, socketio) -> None:
"""
FIXED: Create a new user session with thread-safe creation to prevent duplicates.
FIXED: Register a socketio connection for a session.
This ensures the connection is maintained throughout the session lifecycle.
"""
with self.lock:
self.active_socketio_connections[session_id] = socketio
print(f"Registered socketio connection for session {session_id}")
def get_socketio_connection(self, session_id: str):
"""
FIXED: Get the active socketio connection for a session.
"""
with self.lock:
return self.active_socketio_connections.get(session_id)
def _prepare_scanner_for_storage(self, scanner: Scanner, session_id: str) -> Scanner:
"""
FIXED: Prepare scanner for storage by ensuring proper cleanup of unpicklable objects.
Now preserves socketio connection info for restoration.
"""
# Set the session ID on the scanner for cross-process stop signal management
scanner.session_id = session_id
# FIXED: Don't set socketio to None if we want to preserve real-time updates
# Instead, we'll restore it when loading the scanner
scanner.socketio = None
# Force cleanup of any threading objects that might cause issues
if hasattr(scanner, 'stop_event'):
scanner.stop_event = None
if hasattr(scanner, 'scan_thread'):
scanner.scan_thread = None
if hasattr(scanner, 'executor'):
scanner.executor = None
if hasattr(scanner, 'status_logger_thread'):
scanner.status_logger_thread = None
if hasattr(scanner, 'status_logger_stop_event'):
scanner.status_logger_stop_event = None
return scanner
def create_session(self, socketio=None) -> str:
"""
FIXED: Create a new user session with enhanced WebSocket management.
"""
# FIXED: Use creation lock to prevent race conditions
with self.creation_lock:
session_id = str(uuid.uuid4())
print(f"=== CREATING SESSION {session_id} IN REDIS ===")
# FIXED: Register socketio connection first
if socketio:
self.register_socketio_connection(session_id, socketio)
try:
from core.session_config import create_session_config
session_config = create_session_config()
scanner_instance = Scanner(session_config=session_config)
# Set the session ID on the scanner for cross-process stop signal management
scanner_instance.session_id = session_id
# Create scanner WITHOUT socketio to avoid weakref issues
scanner_instance = Scanner(session_config=session_config, socketio=None)
# Prepare scanner for storage (removes problematic objects)
scanner_instance = self._prepare_scanner_for_storage(scanner_instance, session_id)
session_data = {
'scanner': scanner_instance,
@@ -89,12 +142,24 @@ class SessionManager:
'status': 'active'
}
# Serialize the entire session data dictionary using pickle
serialized_data = pickle.dumps(session_data)
# Test serialization before storing to catch issues early
try:
test_serialization = pickle.dumps(session_data)
print(f"Session serialization test successful ({len(test_serialization)} bytes)")
except Exception as pickle_error:
print(f"PICKLE TEST FAILED: {pickle_error}")
# Try to identify the problematic object
for key, value in session_data.items():
try:
pickle.dumps(value)
print(f" {key}: OK")
except Exception as item_error:
print(f" {key}: FAILED - {item_error}")
raise pickle_error
# Store in Redis
session_key = self._get_session_key(session_id)
self.redis_client.setex(session_key, self.session_timeout, serialized_data)
self.redis_client.setex(session_key, self.session_timeout, test_serialization)
# Initialize stop signal as False
stop_key = self._get_stop_signal_key(session_id)
@@ -106,6 +171,8 @@ class SessionManager:
except Exception as e:
print(f"ERROR: Failed to create session {session_id}: {e}")
import traceback
traceback.print_exc()
raise
def set_stop_signal(self, session_id: str) -> bool:
@@ -175,31 +242,63 @@ class SessionManager:
# Ensure the scanner has the correct session ID for stop signal checking
if 'scanner' in session_data and session_data['scanner']:
session_data['scanner'].session_id = session_id
# FIXED: Restore socketio connection from our registry
socketio_conn = self.get_socketio_connection(session_id)
if socketio_conn:
session_data['scanner'].socketio = socketio_conn
print(f"Restored socketio connection for session {session_id}")
else:
print(f"No socketio connection found for session {session_id}")
session_data['scanner'].socketio = None
return session_data
return None
except Exception as e:
print(f"ERROR: Failed to get session data for {session_id}: {e}")
import traceback
traceback.print_exc()
return None
def _save_session_data(self, session_id: str, session_data: Dict[str, Any]) -> bool:
"""
Serializes and saves session data back to Redis with updated TTL.
FIXED: Now preserves socketio connection during storage.
Returns:
bool: True if save was successful
"""
try:
session_key = self._get_session_key(session_id)
serialized_data = pickle.dumps(session_data)
# Create a deep copy to avoid modifying the original scanner object
session_data_to_save = copy.deepcopy(session_data)
# Prepare scanner for storage if it exists
if 'scanner' in session_data_to_save and session_data_to_save['scanner']:
# FIXED: Preserve the original socketio connection before preparing for storage
original_socketio = session_data_to_save['scanner'].socketio
session_data_to_save['scanner'] = self._prepare_scanner_for_storage(
session_data_to_save['scanner'],
session_id
)
# FIXED: If we had a socketio connection, make sure it's registered
if original_socketio and session_id not in self.active_socketio_connections:
self.register_socketio_connection(session_id, original_socketio)
serialized_data = pickle.dumps(session_data_to_save)
result = self.redis_client.setex(session_key, self.session_timeout, serialized_data)
return result
except Exception as e:
print(f"ERROR: Failed to save session data for {session_id}: {e}")
import traceback
traceback.print_exc()
return False
def update_session_scanner(self, session_id: str, scanner: 'Scanner') -> bool:
"""
Updates just the scanner object in a session with immediate persistence.
FIXED: Updates just the scanner object in a session with immediate persistence.
Now maintains socketio connection throughout the update process.
Returns:
bool: True if update was successful
@@ -207,21 +306,27 @@ class SessionManager:
try:
session_data = self._get_session_data(session_id)
if session_data:
# Ensure scanner has the session ID
scanner.session_id = session_id
# FIXED: Preserve socketio connection before preparing for storage
original_socketio = scanner.socketio
# Prepare scanner for storage
scanner = self._prepare_scanner_for_storage(scanner, session_id)
session_data['scanner'] = scanner
session_data['last_activity'] = time.time()
# FIXED: Restore socketio connection after preparation
if original_socketio:
self.register_socketio_connection(session_id, original_socketio)
session_data['scanner'].socketio = original_socketio
# Immediately save to Redis for GUI updates
success = self._save_session_data(session_id, session_data)
if success:
# Only log occasionally to reduce noise
if hasattr(self, '_last_update_log'):
if time.time() - self._last_update_log > 5: # Log every 5 seconds max
#print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
self._last_update_log = time.time()
else:
#print(f"Scanner state updated for session {session_id} (status: {scanner.status})")
self._last_update_log = time.time()
else:
print(f"WARNING: Failed to save scanner state for session {session_id}")
@@ -231,6 +336,8 @@ class SessionManager:
return False
except Exception as e:
print(f"ERROR: Failed to update scanner for session {session_id}: {e}")
import traceback
traceback.print_exc()
return False
def update_scanner_status(self, session_id: str, status: str) -> bool:
@@ -263,7 +370,7 @@ class SessionManager:
def get_session(self, session_id: str) -> Optional[Scanner]:
"""
Get scanner instance for a session from Redis with session ID management.
FIXED: Get scanner instance for a session from Redis with proper socketio restoration.
"""
if not session_id:
return None
@@ -282,6 +389,15 @@ class SessionManager:
# Ensure the scanner can check the Redis-based stop signal
scanner.session_id = session_id
# FIXED: Restore socketio connection from our registry
socketio_conn = self.get_socketio_connection(session_id)
if socketio_conn:
scanner.socketio = socketio_conn
print(f"✓ Restored socketio connection for session {session_id}")
else:
scanner.socketio = None
print(f"⚠️ No socketio connection found for session {session_id}")
return scanner
def get_session_status_only(self, session_id: str) -> Optional[str]:
@@ -333,6 +449,12 @@ class SessionManager:
# Wait a moment for graceful shutdown
time.sleep(0.5)
# FIXED: Clean up socketio connection
with self.lock:
if session_id in self.active_socketio_connections:
del self.active_socketio_connections[session_id]
print(f"Cleaned up socketio connection for session {session_id}")
# Delete session data and stop signal from Redis
session_key = self._get_session_key(session_id)
stop_key = self._get_stop_signal_key(session_id)
@@ -344,6 +466,8 @@ class SessionManager:
except Exception as e:
print(f"ERROR: Failed to terminate session {session_id}: {e}")
import traceback
traceback.print_exc()
return False
def _cleanup_loop(self) -> None:
@@ -353,7 +477,7 @@ class SessionManager:
while True:
try:
# Clean up orphaned stop signals
stop_keys = self.redis_client.keys("DNScope:stop:*")
stop_keys = self.redis_client.keys("dnsrecon:stop:*")
for stop_key in stop_keys:
# Extract session ID from stop key
session_id = stop_key.decode('utf-8').split(':')[-1]
@@ -364,6 +488,12 @@ class SessionManager:
self.redis_client.delete(stop_key)
print(f"Cleaned up orphaned stop signal for session {session_id}")
# Also clean up socketio connection
with self.lock:
if session_id in self.active_socketio_connections:
del self.active_socketio_connections[session_id]
print(f"Cleaned up orphaned socketio for session {session_id}")
except Exception as e:
print(f"Error in cleanup loop: {e}")
@@ -372,8 +502,8 @@ class SessionManager:
def get_statistics(self) -> Dict[str, Any]:
"""Get session manager statistics."""
try:
session_keys = self.redis_client.keys("DNScope:session:*")
stop_keys = self.redis_client.keys("DNScope:stop:*")
session_keys = self.redis_client.keys("dnsrecon:session:*")
stop_keys = self.redis_client.keys("dnsrecon:stop:*")
active_sessions = len(session_keys)
running_scans = 0
@@ -387,14 +517,16 @@ class SessionManager:
return {
'total_active_sessions': active_sessions,
'running_scans': running_scans,
'total_stop_signals': len(stop_keys)
'total_stop_signals': len(stop_keys),
'active_socketio_connections': len(self.active_socketio_connections)
}
except Exception as e:
print(f"ERROR: Failed to get statistics: {e}")
return {
'total_active_sessions': 0,
'running_scans': 0,
'total_stop_signals': 0
'total_stop_signals': 0,
'active_socketio_connections': 0
}
# Global session manager instance

2
providers/__init__.py
View File

@@ -1,5 +1,5 @@
"""
Data provider modules for DNScope.
Data provider modules for DNSRecon.
Contains implementations for various reconnaissance data sources.
"""

48
providers/base_provider.py
View File

@@ -1,4 +1,4 @@
# DNScope/providers/base_provider.py
# dnsrecon/providers/base_provider.py
import time
import requests
@@ -13,8 +13,9 @@ from core.provider_result import ProviderResult
class BaseProvider(ABC):
"""
Abstract base class for all DNScope data providers.
Abstract base class for all DNSRecon data providers.
Now supports session-specific configuration and returns standardized ProviderResult objects.
FIXED: Enhanced pickle support to prevent weakref serialization errors.
"""
def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30, session_config=None):
@@ -53,26 +54,61 @@ class BaseProvider(ABC):
def __getstate__(self):
"""Prepare BaseProvider for pickling by excluding unpicklable objects."""
state = self.__dict__.copy()
# Exclude the unpickleable '_local' attribute and stop event
unpicklable_attrs = ['_local', '_stop_event']
# Exclude unpickleable attributes that may contain weakrefs
unpicklable_attrs = [
'_local', # Thread-local storage (contains requests.Session)
'_stop_event', # Threading event
'logger', # Logger may contain weakrefs in handlers
]
for attr in unpicklable_attrs:
if attr in state:
del state[attr]
# Also handle any potential weakrefs in the config object
if 'config' in state and hasattr(state['config'], '__getstate__'):
# If config has its own pickle support, let it handle itself
pass
elif 'config' in state:
# Otherwise, ensure config doesn't contain unpicklable objects
try:
# Test if config can be pickled
import pickle
pickle.dumps(state['config'])
except (TypeError, AttributeError):
# If config can't be pickled, we'll recreate it during unpickling
state['_config_class'] = type(state['config']).__name__
del state['config']
return state
def __setstate__(self, state):
"""Restore BaseProvider after unpickling by reconstructing threading objects."""
self.__dict__.update(state)
# Re-initialize the '_local' attribute and stop event
# Re-initialize unpickleable attributes
self._local = threading.local()
self._stop_event = None
self.logger = get_forensic_logger()
# Recreate config if it was removed during pickling
if not hasattr(self, 'config') and hasattr(self, '_config_class'):
if self._config_class == 'Config':
from config import config as global_config
self.config = global_config
elif self._config_class == 'SessionConfig':
from core.session_config import create_session_config
self.config = create_session_config()
del self._config_class
@property
def session(self):
"""Get or create thread-local requests session."""
if not hasattr(self._local, 'session'):
self._local.session = requests.Session()
self._local.session.headers.update({
'User-Agent': 'DNScope/1.0 (Passive Reconnaissance Tool)'
'User-Agent': 'DNSRecon/1.0 (Passive Reconnaissance Tool)'
})
return self._local.session

160
providers/correlation_provider.py
View File

@@ -1,4 +1,4 @@
# DNScope/providers/correlation_provider.py
# dnsrecon/providers/correlation_provider.py
import re
from typing import Dict, Any, List
@@ -10,6 +10,7 @@ from core.graph_manager import NodeType, GraphManager
class CorrelationProvider(BaseProvider):
"""
A provider that finds correlations between nodes in the graph.
FIXED: Enhanced pickle support to prevent weakref issues with graph references.
"""
def __init__(self, name: str = "correlation", session_config=None):
@@ -26,8 +27,8 @@ class CorrelationProvider(BaseProvider):
'cert_common_name',
'cert_validity_period_days',
'cert_issuer_name',
'cert_serial_number',
'cert_entry_timestamp',
'cert_serial_number', # useless
'cert_not_before',
'cert_not_after',
'dns_ttl',
@@ -38,6 +39,38 @@ class CorrelationProvider(BaseProvider):
'query_timestamp',
]
def __getstate__(self):
"""
FIXED: Prepare CorrelationProvider for pickling by excluding graph reference.
"""
state = super().__getstate__()
# Remove graph reference to prevent circular dependencies and weakrefs
if 'graph' in state:
del state['graph']
# Also handle correlation_index which might contain complex objects
if 'correlation_index' in state:
# Clear correlation index as it will be rebuilt when needed
state['correlation_index'] = {}
return state
def __setstate__(self, state):
"""
FIXED: Restore CorrelationProvider after unpickling.
"""
super().__setstate__(state)
# Re-initialize graph reference (will be set by scanner)
self.graph = None
# Re-initialize correlation index
self.correlation_index = {}
# Re-compile regex pattern
self.date_pattern = re.compile(r'^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}')
def get_name(self) -> str:
"""Return the provider name."""
return "correlation"
@@ -78,38 +111,47 @@ class CorrelationProvider(BaseProvider):
def _find_correlations(self, node_id: str) -> ProviderResult:
"""
Find correlations for a given node with enhanced filtering and error handling.
Find correlations for a given node.
FIXED: Added safety checks to prevent issues when graph is None.
"""
result = ProviderResult()
# Enhanced safety checks
# FIXED: Ensure self.graph is not None before proceeding
if not self.graph or not self.graph.graph.has_node(node_id):
return result
try:
node_attributes = self.graph.graph.nodes[node_id].get('attributes', [])
# Ensure attributes is a list (handle legacy data)
if not isinstance(node_attributes, list):
except Exception as e:
# If there's any issue accessing the graph, return empty result
print(f"Warning: Could not access graph for correlation analysis: {e}")
return result
correlations_found = 0
for attr in node_attributes:
if not isinstance(attr, dict):
continue
attr_name = attr.get('name', '')
attr_name = attr.get('name')
attr_value = attr.get('value')
attr_provider = attr.get('provider', 'unknown')
# Enhanced filtering logic
should_exclude = self._should_exclude_attribute(attr_name, attr_value)
should_exclude = (
any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS) or
not isinstance(attr_value, (str, int, float, bool)) or
attr_value is None or
isinstance(attr_value, bool) or
(isinstance(attr_value, str) and (
len(attr_value) < 4 or
self.date_pattern.match(attr_value) or
attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']
)) or
(isinstance(attr_value, (int, float)) and (
attr_value == 0 or
attr_value == 1 or
abs(attr_value) > 1000000
))
)
if should_exclude:
continue
# Build correlation index
if attr_value not in self.correlation_index:
self.correlation_index[attr_value] = {
'nodes': set(),
@@ -125,119 +167,42 @@ class CorrelationProvider(BaseProvider):
'path': f"{attr_provider}_{attr_name}"
}
# Avoid duplicate sources
existing_sources = [s for s in self.correlation_index[attr_value]['sources']
if s['node_id'] == node_id and s['path'] == source_info['path']]
if not existing_sources:
self.correlation_index[attr_value]['sources'].append(source_info)
# Create correlation if we have multiple nodes with this value
if len(self.correlation_index[attr_value]['nodes']) > 1:
self._create_correlation_relationships(attr_value, self.correlation_index[attr_value], result)
correlations_found += 1
# Log correlation results
if correlations_found > 0:
self.logger.logger.info(f"Found {correlations_found} correlations for node {node_id}")
except Exception as e:
self.logger.logger.error(f"Error finding correlations for {node_id}: {e}")
return result
def _should_exclude_attribute(self, attr_name: str, attr_value: Any) -> bool:
"""
Enhanced logic to determine if an attribute should be excluded from correlation.
"""
# Check against excluded keys (exact match or substring)
if any(excluded_key in attr_name or attr_name == excluded_key for excluded_key in self.EXCLUDED_KEYS):
return True
# Value type filtering
if not isinstance(attr_value, (str, int, float, bool)) or attr_value is None:
return True
# Boolean values are not useful for correlation
if isinstance(attr_value, bool):
return True
# String value filtering
if isinstance(attr_value, str):
# Date/timestamp strings
if self.date_pattern.match(attr_value):
return True
# Common non-useful values
if attr_value.lower() in ['unknown', 'none', 'null', 'n/a', 'true', 'false', '0', '1']:
return True
# Very long strings that are likely unique (> 100 chars)
if len(attr_value) > 100:
return True
# Numeric value filtering
if isinstance(attr_value, (int, float)):
# Very common values
if attr_value in [0, 1]:
return True
# Very large numbers (likely timestamps or unique IDs)
if abs(attr_value) > 1000000:
return True
return False
def _create_correlation_relationships(self, value: Any, correlation_data: Dict[str, Any], result: ProviderResult):
"""
Create correlation relationships with enhanced deduplication and validation.
Create correlation relationships and add them to the provider result.
"""
correlation_node_id = f"corr_{hash(str(value)) & 0x7FFFFFFF}"
nodes = correlation_data['nodes']
sources = correlation_data['sources']
# Only create correlations if we have meaningful nodes (more than 1)
if len(nodes) < 2:
return
# Limit correlation size to prevent overly large correlation objects
MAX_CORRELATION_SIZE = 50
if len(nodes) > MAX_CORRELATION_SIZE:
# Sample the nodes to keep correlation manageable
import random
sampled_nodes = random.sample(list(nodes), MAX_CORRELATION_SIZE)
nodes = set(sampled_nodes)
# Filter sources to match sampled nodes
sources = [s for s in sources if s['node_id'] in nodes]
# Add the correlation node as an attribute to the result
result.add_attribute(
target_node=correlation_node_id,
name="correlation_value",
value=value,
attr_type=str(type(value).__name__),
attr_type=str(type(value)),
provider=self.name,
confidence=0.9,
metadata={
'correlated_nodes': list(nodes),
'sources': sources,
'correlation_size': len(nodes),
'value_type': type(value).__name__
}
)
# Create relationships with source validation
created_relationships = set()
for source in sources:
node_id = source['node_id']
provider = source['provider']
attribute = source['attribute']
# Skip if we've already created this relationship
relationship_key = (node_id, correlation_node_id)
if relationship_key in created_relationships:
continue
relationship_label = f"corr_{provider}_{attribute}"
# Add the relationship to the result
@@ -250,9 +215,6 @@ class CorrelationProvider(BaseProvider):
raw_data={
'correlation_value': value,
'original_attribute': attribute,
'correlation_type': 'attribute_matching',
'correlation_size': len(nodes)
'correlation_type': 'attribute_matching'
}
)
created_relationships.add(relationship_key)

339
providers/crtsh_provider.py
View File

@@ -1,4 +1,4 @@
# DNScope/providers/crtsh_provider.py
# dnsrecon/providers/crtsh_provider.py
import json
import re
@@ -13,10 +13,11 @@ from core.provider_result import ProviderResult
from utils.helpers import _is_valid_domain
from core.logger import get_forensic_logger
class CrtShProvider(BaseProvider):
"""
Provider for querying crt.sh certificate transparency database.
FIXED: Improved caching logic and error handling to prevent infinite retry loops.
FIXED: Now properly creates domain and CA nodes instead of large entities.
Returns standardized ProviderResult objects with caching support.
"""
@@ -31,7 +32,7 @@ class CrtShProvider(BaseProvider):
self.base_url = "https://crt.sh/"
self._stop_event = None
# Initialize cache directory
# Initialize cache directory (separate from BaseProvider's HTTP cache)
self.domain_cache_dir = Path('cache') / 'crtsh'
self.domain_cache_dir.mkdir(parents=True, exist_ok=True)
@@ -65,72 +66,43 @@ class CrtShProvider(BaseProvider):
def _get_cache_status(self, cache_file_path: Path) -> str:
"""
FIXED: More robust cache status checking with better error handling.
Check cache status for a domain.
Returns: 'not_found', 'fresh', or 'stale'
"""
if not cache_file_path.exists():
return "not_found"
try:
# Check if file is readable and not corrupted
if cache_file_path.stat().st_size == 0:
self.logger.logger.warning(f"Empty cache file: {cache_file_path}")
return "stale"
with open(cache_file_path, 'r', encoding='utf-8') as f:
with open(cache_file_path, 'r') as f:
cache_data = json.load(f)
# Validate cache structure
if not isinstance(cache_data, dict):
self.logger.logger.warning(f"Invalid cache structure: {cache_file_path}")
return "stale"
last_query_str = cache_data.get("last_upstream_query")
if not last_query_str or not isinstance(last_query_str, str):
self.logger.logger.warning(f"Missing or invalid last_upstream_query: {cache_file_path}")
if not last_query_str:
return "stale"
try:
# More robust datetime parsing
if last_query_str.endswith('Z'):
last_query = datetime.fromisoformat(last_query_str.replace('Z', '+00:00'))
elif '+' in last_query_str or last_query_str.endswith('UTC'):
# Handle various timezone formats
clean_time = last_query_str.replace('UTC', '').strip()
if '+' in clean_time:
clean_time = clean_time.split('+')[0]
last_query = datetime.fromisoformat(clean_time).replace(tzinfo=timezone.utc)
else:
last_query = datetime.fromisoformat(last_query_str).replace(tzinfo=timezone.utc)
except (ValueError, AttributeError) as e:
self.logger.logger.warning(f"Failed to parse timestamp in cache {cache_file_path}: {e}")
return "stale"
hours_since_query = (datetime.now(timezone.utc) - last_query).total_seconds() / 3600
cache_timeout = self.config.cache_timeout_hours
cache_timeout = self.config.cache_timeout_hours
if hours_since_query < cache_timeout:
return "fresh"
else:
return "stale"
except (json.JSONDecodeError, OSError, PermissionError) as e:
self.logger.logger.warning(f"Cache file error for {cache_file_path}: {e}")
# FIXED: Try to remove corrupted cache file
try:
cache_file_path.unlink()
self.logger.logger.info(f"Removed corrupted cache file: {cache_file_path}")
except Exception:
pass
return "not_found"
except Exception as e:
self.logger.logger.error(f"Unexpected error checking cache status for {cache_file_path}: {e}")
except (json.JSONDecodeError, ValueError, KeyError) as e:
self.logger.logger.warning(f"Invalid cache file format for {cache_file_path}: {e}")
return "stale"
def query_domain(self, domain: str) -> ProviderResult:
"""
FIXED: Simplified and more robust domain querying with better error handling.
FIXED: Query crt.sh for certificates containing the domain.
Now properly creates domain and CA nodes instead of large entities.
Args:
domain: Domain to investigate
Returns:
ProviderResult containing discovered relationships and attributes
"""
if not _is_valid_domain(domain):
return ProviderResult()
@@ -139,155 +111,115 @@ class CrtShProvider(BaseProvider):
return ProviderResult()
cache_file = self._get_cache_file_path(domain)
result = ProviderResult()
try:
cache_status = self._get_cache_status(cache_file)
if cache_status == "fresh":
# Load from cache
result = self._load_from_cache(cache_file)
if result and (result.relationships or result.attributes):
self.logger.logger.debug(f"Using fresh cached crt.sh data for {domain}")
return result
else:
# Cache exists but is empty, treat as stale
cache_status = "stale"
result = ProviderResult()
# Need to query API (either no cache, stale cache, or empty cache)
self.logger.logger.debug(f"Querying crt.sh API for {domain} (cache status: {cache_status})")
if cache_status == "fresh":
result = self._load_from_cache(cache_file)
self.logger.logger.info(f"Using fresh cached crt.sh data for {domain}")
else: # "stale" or "not_found"
# Query the API for the latest certificates
new_raw_certs = self._query_crtsh_api(domain)
if self._stop_event and self._stop_event.is_set():
return ProviderResult()
# FIXED: Simplified processing - just process the new data
# Don't try to merge with stale cache as it can cause corruption
# Combine with old data if cache is stale
if cache_status == "stale":
old_raw_certs = self._load_raw_data_from_cache(cache_file)
combined_certs = old_raw_certs + new_raw_certs
# Deduplicate the combined list
seen_ids = set()
unique_certs = []
for cert in combined_certs:
cert_id = cert.get('id')
if cert_id not in seen_ids:
unique_certs.append(cert)
seen_ids.add(cert_id)
raw_certificates_to_process = unique_certs
self.logger.logger.info(f"Refreshed and merged cache for {domain}. Total unique certs: {len(raw_certificates_to_process)}")
else: # "not_found"
raw_certificates_to_process = new_raw_certs
if cache_status == "stale":
self.logger.logger.info(f"Refreshed stale cache for {domain} with {len(raw_certificates_to_process)} certs")
else:
self.logger.logger.info(f"Created fresh cache for {domain} with {len(raw_certificates_to_process)} certs")
# FIXED: Process certificates to create proper domain and CA nodes
result = self._process_certificates_to_result_fixed(domain, raw_certificates_to_process)
self.logger.logger.info(f"Created fresh result for {domain} ({result.get_relationship_count()} relationships)")
# Save the result to cache
# Save the new result and the raw data to the cache
self._save_result_to_cache(cache_file, result, raw_certificates_to_process, domain)
return result
except requests.exceptions.RequestException as e:
# FIXED: Don't re-raise network errors after long idle periods
# Instead return empty result and log the issue
self.logger.logger.warning(f"Network error querying crt.sh for {domain}: {e}")
# Try to use stale cache if available
if cache_status == "stale":
try:
stale_result = self._load_from_cache(cache_file)
if stale_result and (stale_result.relationships or stale_result.attributes):
self.logger.logger.info(f"Using stale cache for {domain} due to network error")
return stale_result
except Exception as cache_error:
self.logger.logger.warning(f"Failed to load stale cache for {domain}: {cache_error}")
# Return empty result instead of raising - this prevents infinite retries
return ProviderResult()
except Exception as e:
# FIXED: Handle any other exceptions gracefully
self.logger.logger.error(f"Unexpected error querying crt.sh for {domain}: {e}")
# Try stale cache as fallback
try:
if cache_file.exists():
fallback_result = self._load_from_cache(cache_file)
if fallback_result and (fallback_result.relationships or fallback_result.attributes):
self.logger.logger.info(f"Using cached data for {domain} due to processing error")
return fallback_result
except Exception:
pass
# Return empty result to prevent retries
return ProviderResult()
def query_ip(self, ip: str) -> ProviderResult:
"""
crt.sh does not support IP-based certificate queries effectively via its API.
Query crt.sh for certificates containing the IP address.
Note: crt.sh doesn't typically index by IP, so this returns empty results.
Args:
ip: IP address to investigate
Returns:
Empty ProviderResult (crt.sh doesn't support IP-based certificate queries effectively)
"""
return ProviderResult()
def _load_from_cache(self, cache_file_path: Path) -> ProviderResult:
"""FIXED: More robust cache loading with better validation."""
"""Load processed crt.sh data from a cache file."""
try:
if not cache_file_path.exists() or cache_file_path.stat().st_size == 0:
return ProviderResult()
with open(cache_file_path, 'r', encoding='utf-8') as f:
with open(cache_file_path, 'r') as f:
cache_content = json.load(f)
if not isinstance(cache_content, dict):
self.logger.logger.warning(f"Invalid cache format in {cache_file_path}")
return ProviderResult()
result = ProviderResult()
# Reconstruct relationships with validation
relationships = cache_content.get("relationships", [])
if isinstance(relationships, list):
for rel_data in relationships:
if not isinstance(rel_data, dict):
continue
try:
# Reconstruct relationships
for rel_data in cache_content.get("relationships", []):
result.add_relationship(
source_node=rel_data.get("source_node", ""),
target_node=rel_data.get("target_node", ""),
relationship_type=rel_data.get("relationship_type", ""),
provider=rel_data.get("provider", self.name),
confidence=float(rel_data.get("confidence", 0.8)),
source_node=rel_data["source_node"],
target_node=rel_data["target_node"],
relationship_type=rel_data["relationship_type"],
provider=rel_data["provider"],
confidence=rel_data["confidence"],
raw_data=rel_data.get("raw_data", {})
)
except (ValueError, TypeError) as e:
self.logger.logger.warning(f"Skipping invalid relationship in cache: {e}")
continue
# Reconstruct attributes with validation
attributes = cache_content.get("attributes", [])
if isinstance(attributes, list):
for attr_data in attributes:
if not isinstance(attr_data, dict):
continue
try:
# Reconstruct attributes
for attr_data in cache_content.get("attributes", []):
result.add_attribute(
target_node=attr_data.get("target_node", ""),
name=attr_data.get("name", ""),
value=attr_data.get("value"),
attr_type=attr_data.get("type", "unknown"),
provider=attr_data.get("provider", self.name),
confidence=float(attr_data.get("confidence", 0.9)),
target_node=attr_data["target_node"],
name=attr_data["name"],
value=attr_data["value"],
attr_type=attr_data["type"],
provider=attr_data["provider"],
confidence=attr_data["confidence"],
metadata=attr_data.get("metadata", {})
)
except (ValueError, TypeError) as e:
self.logger.logger.warning(f"Skipping invalid attribute in cache: {e}")
continue
return result
except (json.JSONDecodeError, OSError, PermissionError) as e:
self.logger.logger.warning(f"Failed to load cache from {cache_file_path}: {e}")
return ProviderResult()
except Exception as e:
self.logger.logger.error(f"Unexpected error loading cache from {cache_file_path}: {e}")
except (json.JSONDecodeError, FileNotFoundError, KeyError) as e:
self.logger.logger.error(f"Failed to load cached certificates from {cache_file_path}: {e}")
return ProviderResult()
def _load_raw_data_from_cache(self, cache_file_path: Path) -> List[Dict[str, Any]]:
"""Load only the raw certificate data from a cache file."""
try:
with open(cache_file_path, 'r') as f:
cache_content = json.load(f)
return cache_content.get("raw_certificates", [])
except (json.JSONDecodeError, FileNotFoundError):
return []
def _save_result_to_cache(self, cache_file_path: Path, result: ProviderResult, raw_certificates: List[Dict[str, Any]], domain: str) -> None:
"""FIXED: More robust cache saving with atomic writes."""
"""Save processed crt.sh result and raw data to a cache file."""
try:
cache_data = {
"domain": domain,
"last_upstream_query": datetime.now(timezone.utc).isoformat(),
"raw_certificates": raw_certificates,
"raw_certificates": raw_certificates, # Store the raw data for deduplication
"relationships": [
{
"source_node": rel.source_node,
@@ -310,68 +242,35 @@ class CrtShProvider(BaseProvider):
} for attr in result.attributes
]
}
cache_file_path.parent.mkdir(parents=True, exist_ok=True)
# FIXED: Atomic write using temporary file
temp_file = cache_file_path.with_suffix('.tmp')
try:
with open(temp_file, 'w', encoding='utf-8') as f:
json.dump(cache_data, f, separators=(',', ':'), default=str, ensure_ascii=False)
# Atomic rename
temp_file.replace(cache_file_path)
self.logger.logger.debug(f"Saved cache for {domain} ({len(result.relationships)} relationships)")
except Exception as e:
# Clean up temp file on error
if temp_file.exists():
try:
temp_file.unlink()
except Exception:
pass
raise e
with open(cache_file_path, 'w') as f:
json.dump(cache_data, f, separators=(',', ':'), default=str)
except Exception as e:
self.logger.logger.warning(f"Failed to save cache file for {domain}: {e}")
def _query_crtsh_api(self, domain: str) -> List[Dict[str, Any]]:
"""FIXED: More robust API querying with better error handling."""
"""Query crt.sh API for raw certificate data."""
url = f"{self.base_url}?q={quote(domain)}&output=json"
try:
response = self.make_request(url, target_indicator=domain)
if not response:
self.logger.logger.warning(f"No response from crt.sh for {domain}")
return []
if not response or response.status_code != 200:
raise requests.exceptions.RequestException(f"crt.sh API returned status {response.status_code if response else 'None'}")
if response.status_code != 200:
self.logger.logger.warning(f"crt.sh returned status {response.status_code} for {domain}")
return []
# FIXED: Better JSON parsing with error handling
try:
certificates = response.json()
except json.JSONDecodeError as e:
self.logger.logger.error(f"crt.sh returned invalid JSON for {domain}: {e}")
except json.JSONDecodeError:
self.logger.logger.error(f"crt.sh returned invalid JSON for {domain}")
return []
if not certificates or not isinstance(certificates, list):
self.logger.logger.debug(f"crt.sh returned no certificates for {domain}")
if not certificates:
return []
self.logger.logger.debug(f"crt.sh returned {len(certificates)} certificates for {domain}")
return certificates
except Exception as e:
self.logger.logger.error(f"Error querying crt.sh API for {domain}: {e}")
raise e
def _process_certificates_to_result_fixed(self, query_domain: str, certificates: List[Dict[str, Any]]) -> ProviderResult:
"""
Process certificates to create proper domain and CA nodes.
FIXED: Better error handling and progress tracking.
FIXED: Process certificates to create proper domain and CA nodes.
Now creates individual domain nodes instead of large entities.
"""
result = ProviderResult()
@@ -379,11 +278,6 @@ class CrtShProvider(BaseProvider):
self.logger.logger.info(f"CrtSh processing cancelled before processing for domain: {query_domain}")
return result
if not certificates:
self.logger.logger.debug(f"No certificates to process for {query_domain}")
return result
# Check for incomplete data warning
incompleteness_warning = self._check_for_incomplete_data(query_domain, certificates)
if incompleteness_warning:
result.add_attribute(
@@ -397,27 +291,20 @@ class CrtShProvider(BaseProvider):
all_discovered_domains = set()
processed_issuers = set()
processed_certs = 0
for i, cert_data in enumerate(certificates):
# FIXED: More frequent stop checks and progress logging
if i % 5 == 0:
if self._stop_event and self._stop_event.is_set():
self.logger.logger.info(f"CrtSh processing cancelled at certificate {i}/{len(certificates)} for domain: {query_domain}")
if i % 10 == 0 and self._stop_event and self._stop_event.is_set():
self.logger.logger.info(f"CrtSh processing cancelled at certificate {i} for domain: {query_domain}")
break
if i > 0 and i % 100 == 0:
self.logger.logger.debug(f"Processed {i}/{len(certificates)} certificates for {query_domain}")
try:
# Extract all domains from this certificate
cert_domains = self._extract_domains_from_certificate(cert_data)
if cert_domains:
all_discovered_domains.update(cert_domains)
# Create CA nodes for certificate issuers
# FIXED: Create CA nodes for certificate issuers (not as domain metadata)
issuer_name = self._parse_issuer_organization(cert_data.get('issuer_name', ''))
if issuer_name and issuer_name not in processed_issuers:
# Create relationship from query domain to CA
result.add_relationship(
source_node=query_domain,
target_node=issuer_name,
@@ -434,6 +321,7 @@ class CrtShProvider(BaseProvider):
if not _is_valid_domain(cert_domain):
continue
# Add certificate attributes to the domain
for key, value in cert_metadata.items():
if value is not None:
result.add_attribute(
@@ -446,19 +334,12 @@ class CrtShProvider(BaseProvider):
metadata={'certificate_id': cert_data.get('id')}
)
processed_certs += 1
except Exception as e:
self.logger.logger.warning(f"Error processing certificate {i} for {query_domain}: {e}")
continue
# Check for stop event before creating final relationships
if self._stop_event and self._stop_event.is_set():
self.logger.logger.info(f"CrtSh query cancelled before relationship creation for domain: {query_domain}")
return result
# Create selective relationships to avoid large entities
relationships_created = 0
# FIXED: Create selective relationships to avoid large entities
# Only create relationships to domains that are closely related
for discovered_domain in all_discovered_domains:
if discovered_domain == query_domain:
continue
@@ -466,6 +347,8 @@ class CrtShProvider(BaseProvider):
if not _is_valid_domain(discovered_domain):
continue
# FIXED: Only create relationships for domains that share a meaningful connection
# This prevents creating too many relationships that trigger large entity creation
if self._should_create_relationship(query_domain, discovered_domain):
confidence = self._calculate_domain_relationship_confidence(
query_domain, discovered_domain, [], all_discovered_domains
@@ -488,22 +371,24 @@ class CrtShProvider(BaseProvider):
raw_data={'relationship_type': 'certificate_discovery'},
discovery_method="certificate_transparency_analysis"
)
relationships_created += 1
self.logger.logger.info(f"CrtSh processing completed for {query_domain}: processed {processed_certs}/{len(certificates)} certificates, {len(all_discovered_domains)} domains, {relationships_created} relationships")
self.logger.logger.info(f"CrtSh processing completed for {query_domain}: {len(all_discovered_domains)} domains, {result.get_relationship_count()} relationships")
return result
# [Rest of the methods remain the same as in the original file]
def _should_create_relationship(self, source_domain: str, target_domain: str) -> bool:
"""
Determine if a relationship should be created between two domains.
FIXED: Determine if a relationship should be created between two domains.
This helps avoid creating too many relationships that trigger large entity creation.
"""
# Always create relationships for subdomains
if target_domain.endswith(f'.{source_domain}') or source_domain.endswith(f'.{target_domain}'):
return True
# Create relationships for domains that share a common parent (up to 2 levels)
source_parts = source_domain.split('.')
target_parts = target_domain.split('.')
# Check if they share the same root domain (last 2 parts)
if len(source_parts) >= 2 and len(target_parts) >= 2:
source_root = '.'.join(source_parts[-2:])
target_root = '.'.join(target_parts[-2:])
@@ -537,6 +422,7 @@ class CrtShProvider(BaseProvider):
metadata['is_currently_valid'] = self._is_cert_valid(cert_data)
metadata['expires_soon'] = (not_after - datetime.now(timezone.utc)).days <= 30
# Keep raw date format or convert to standard format
metadata['not_before'] = not_before.isoformat()
metadata['not_after'] = not_after.isoformat()
@@ -618,12 +504,14 @@ class CrtShProvider(BaseProvider):
"""Extract all domains from certificate data."""
domains = set()
# Extract from common name
common_name = cert_data.get('common_name', '')
if common_name:
cleaned_cn = self._clean_domain_name(common_name)
if cleaned_cn:
domains.update(cleaned_cn)
# Extract from name_value field (contains SANs)
name_value = cert_data.get('name_value', '')
if name_value:
for line in name_value.split('\n'):
@@ -670,6 +558,7 @@ class CrtShProvider(BaseProvider):
"""Calculate confidence score for domain relationship based on various factors."""
base_confidence = 0.9
# Adjust confidence based on domain relationship context
relationship_context = self._determine_relationship_context(domain2, domain1)
if relationship_context == 'exact_match':
@@ -701,10 +590,12 @@ class CrtShProvider(BaseProvider):
"""
cert_count = len(certificates)
# Heuristic 1: Check if the number of certs hits a known hard limit.
if cert_count >= 10000:
return f"Result likely truncated; received {cert_count} certificates, which may be the maximum limit."
if cert_count > 1000:
# Heuristic 2: Check if all returned certificates are old.
if cert_count > 1000: # Only apply this for a reasonable number of certs
latest_expiry = None
for cert in certificates:
try:

23
providers/dns_provider.py
View File

@@ -1,4 +1,4 @@
# DNScope/providers/dns_provider.py
# dnsrecon/providers/dns_provider.py
from dns import resolver, reversename
from typing import Dict
@@ -11,6 +11,7 @@ class DNSProvider(BaseProvider):
"""
Provider for standard DNS resolution and reverse DNS lookups.
Now returns standardized ProviderResult objects with IPv4 and IPv6 support.
FIXED: Enhanced pickle support to prevent resolver serialization issues.
"""
def __init__(self, name=None, session_config=None):
@@ -27,6 +28,22 @@ class DNSProvider(BaseProvider):
self.resolver.timeout = 5
self.resolver.lifetime = 10
def __getstate__(self):
"""Prepare the object for pickling by excluding resolver."""
state = super().__getstate__()
# Remove the unpickleable 'resolver' attribute
if 'resolver' in state:
del state['resolver']
return state
def __setstate__(self, state):
"""Restore the object after unpickling by reconstructing resolver."""
super().__setstate__(state)
# Re-initialize the 'resolver' attribute
self.resolver = resolver.Resolver()
self.resolver.timeout = 5
self.resolver.lifetime = 10
def get_name(self) -> str:
"""Return the provider name."""
return "dns"
@@ -106,10 +123,10 @@ class DNSProvider(BaseProvider):
if _is_valid_domain(hostname):
# Determine appropriate forward relationship type based on IP version
if ip_version == 6:
relationship_type = 'dns_aaaa_record'
relationship_type = 'shodan_aaaa_record'
record_prefix = 'AAAA'
else:
relationship_type = 'dns_a_record'
relationship_type = 'shodan_a_record'
record_prefix = 'A'
# Add the relationship

11
providers/shodan_provider.py
View File

@@ -1,4 +1,4 @@
# DNScope/providers/shodan_provider.py
# dnsrecon/providers/shodan_provider.py
import json
from pathlib import Path
@@ -36,6 +36,15 @@ class ShodanProvider(BaseProvider):
self.cache_dir = Path('cache') / 'shodan'
self.cache_dir.mkdir(parents=True, exist_ok=True)
def __getstate__(self):
"""Prepare the object for pickling."""
state = super().__getstate__()
return state
def __setstate__(self, state):
"""Restore the object after unpickling."""
super().__setstate__(state)
def _check_api_connection(self) -> bool:
"""
FIXED: Lazy connection checking - only test when actually needed.

2
requirements.txt
View File

@@ -9,3 +9,5 @@ gunicorn
redis
python-dotenv
psycopg2-binary
Flask-SocketIO
eventlet

15
static/css/main.css
View File

@@ -1,4 +1,4 @@
/* DNScope - Optimized Compact Theme */
/* DNSRecon - Optimized Compact Theme */
/* Reset and Base */
* {
@@ -474,7 +474,6 @@ input[type="text"]:focus, select:focus {
flex-wrap: wrap;
gap: 0.75rem;
align-items: center;
max-height: 3rem;
}
.legend-item {
@@ -1325,15 +1324,3 @@ input[type="password"]:focus {
grid-template-columns: 1fr;
}
}
.manual-refresh-btn {
background: rgba(92, 76, 44, 0.9) !important; /* Orange/amber background */
border: 1px solid #7a6a3a !important;
color: #ffcc00 !important; /* Bright yellow text */
}
.manual-refresh-btn:hover {
border-color: #ffcc00 !important;
color: #fff !important;
background: rgba(112, 96, 54, 0.9) !important;
}

64
static/js/graph.js
View File

@@ -1,8 +1,8 @@
// DNScope-reduced/static/js/graph.js
// dnsrecon-reduced/static/js/graph.js
/**
* Graph visualization module for DNScope
* Graph visualization module for DNSRecon
* Handles network graph rendering using vis.js with proper large entity node hiding
* UPDATED: Added manual refresh button for polling optimization when graph becomes large
* UPDATED: Now compatible with a strictly flat, unified data model for attributes.
*/
const contextMenuCSS = `
.graph-context-menu {
@@ -72,10 +72,6 @@ class GraphManager {
this.largeEntityMembers = new Set();
this.isScanning = false;
// Manual refresh button for polling optimization
this.manualRefreshButton = null;
this.manualRefreshHandler = null; // Store the handler
this.options = {
nodes: {
shape: 'dot',
@@ -258,7 +254,6 @@ class GraphManager {
/**
* Add interactive graph controls
* UPDATED: Added manual refresh button for polling optimization
*/
addGraphControls() {
const controlsContainer = document.createElement('div');
@@ -269,9 +264,6 @@ class GraphManager {
<button class="graph-control-btn" id="graph-cluster" title="Cluster Nodes">[CLUSTER]</button>
<button class="graph-control-btn" id="graph-unhide" title="Unhide All">[UNHIDE]</button>
<button class="graph-control-btn" id="graph-revert" title="Revert Last Action">[REVERT]</button>
<button class="graph-control-btn manual-refresh-btn" id="graph-manual-refresh"
title="Manual Refresh - Auto-refresh disabled due to large graph"
style="display: none;">[REFRESH]</button>
`;
this.container.appendChild(controlsContainer);
@@ -282,35 +274,6 @@ class GraphManager {
document.getElementById('graph-cluster').addEventListener('click', () => this.toggleClustering());
document.getElementById('graph-unhide').addEventListener('click', () => this.unhideAll());
document.getElementById('graph-revert').addEventListener('click', () => this.revertLastAction());
// Manual refresh button - handler will be set by main app
this.manualRefreshButton = document.getElementById('graph-manual-refresh');
// If a handler was set before the button existed, attach it now
if (this.manualRefreshButton && this.manualRefreshHandler) {
this.manualRefreshButton.addEventListener('click', this.manualRefreshHandler);
}
}
/**
* Set the manual refresh button click handler
* @param {Function} handler - Function to call when manual refresh is clicked
*/
setManualRefreshHandler(handler) {
this.manualRefreshHandler = handler;
// If the button already exists, attach the handler
if (this.manualRefreshButton && typeof handler === 'function') {
this.manualRefreshButton.addEventListener('click', handler);
}
}
/**
* Show or hide the manual refresh button
* @param {boolean} show - Whether to show the button
*/
showManualRefreshButton(show) {
if (this.manualRefreshButton) {
this.manualRefreshButton.style.display = show ? 'inline-block' : 'none';
}
}
addFilterPanel() {
@@ -390,6 +353,9 @@ class GraphManager {
});
}
/**
* @param {Object} graphData - Graph data from backend
*/
updateGraph(graphData) {
if (!graphData || !graphData.nodes || !graphData.edges) {
console.warn('Invalid graph data received');
@@ -416,18 +382,16 @@ class GraphManager {
const nodeMap = new Map(graphData.nodes.map(node => [node.id, node]));
// FIXED: Process all nodes first, then apply hiding logic correctly
// Filter out hidden nodes before processing for rendering
const filteredNodes = graphData.nodes.filter(node =>
!(node.metadata && node.metadata.large_entity_id)
);
const processedNodes = graphData.nodes.map(node => {
const processed = this.processNode(node);
// FIXED: Only hide if node is still a large entity member
if (node.metadata && node.metadata.large_entity_id) {
processed.hidden = true;
} else {
// FIXED: Ensure extracted nodes are visible
processed.hidden = false;
}
return processed;
});
@@ -437,7 +401,6 @@ class GraphManager {
let fromId = edge.from;
let toId = edge.to;
// FIXED: Only re-route if nodes are STILL in large entities
if (fromNode && fromNode.metadata && fromNode.metadata.large_entity_id) {
fromId = fromNode.metadata.large_entity_id;
}
@@ -460,7 +423,6 @@ class GraphManager {
const newNodes = processedNodes.filter(node => !existingNodeIds.includes(node.id));
const newEdges = processedEdges.filter(edge => !existingEdgeIds.includes(edge.id));
// FIXED: Update all nodes to ensure extracted nodes become visible
this.nodes.update(processedNodes);
this.edges.update(processedEdges);
@@ -644,7 +606,7 @@ class GraphManager {
formatEdgeLabel(relationshipType, confidence) {
if (!relationshipType) return '';
const confidenceText = confidence >= 0.8 ? '●' : confidence >= 0.6 ? '●' : '○';
const confidenceText = confidence >= 0.8 ? '●' : confidence >= 0.6 ? '◐' : '○';
return `${relationshipType} ${confidenceText}`;
}
@@ -1454,7 +1416,7 @@ class GraphManager {
menuItems += `
<li data-action="hide" data-node-id="${nodeId}">
<span class="menu-icon">👻</span>
<span class="menu-icon">👁️‍🗨️</span>
<span>Hide Node</span>
</li>
<li data-action="delete" data-node-id="${nodeId}">

556
static/js/main.js
View File

@@ -1,16 +1,15 @@
/**
* Main application logic for DNScope web interface
* Main application logic for DNSRecon web interface
* Handles UI interactions, API communication, and data flow
* UPDATED: Now compatible with a strictly flat, unified data model for attributes.
* FIXED: Enhanced real-time WebSocket graph updates
*/
class DNScopeApp {
class DNSReconApp {
constructor() {
console.log('DNScopeApp constructor called');
console.log('DNSReconApp constructor called');
this.graphManager = null;
this.socket = null;
this.scanStatus = 'idle';
this.statusPollInterval = null; // Separate status polling
this.graphPollInterval = null; // Separate graph polling
this.currentSessionId = null;
this.elements = {};
@@ -18,9 +17,13 @@ class DNScopeApp {
this.isScanning = false;
this.lastGraphUpdate = null;
// Graph polling optimization
this.graphPollingNodeThreshold = 500; // Default, will be loaded from config
this.graphPollingEnabled = true;
// FIXED: Add connection state tracking
this.isConnected = false;
this.reconnectAttempts = 0;
this.maxReconnectAttempts = 5;
// FIXED: Track last graph data for debugging
this.lastGraphData = null;
this.init();
}
@@ -29,41 +32,179 @@ class DNScopeApp {
* Initialize the application
*/
init() {
console.log('DNScopeApp init called');
console.log('DNSReconApp init called');
document.addEventListener('DOMContentLoaded', () => {
console.log('DOM loaded, initializing application...');
try {
this.initializeElements();
this.setupEventHandlers();
this.initializeGraph();
this.updateStatus();
this.initializeSocket();
this.loadProviders();
this.initializeEnhancedModals();
this.addCheckboxStyling();
this.loadConfig(); // Load configuration including threshold
this.updateGraph();
console.log('DNScope application initialized successfully');
console.log('DNSRecon application initialized successfully');
} catch (error) {
console.error('Failed to initialize DNScope application:', error);
console.error('Failed to initialize DNSRecon application:', error);
this.showError(`Initialization failed: ${error.message}`);
}
});
}
/**
* Load configuration from backend
*/
async loadConfig() {
initializeSocket() {
console.log('🔌 Initializing WebSocket connection...');
try {
const response = await this.apiCall('/api/config');
if (response.success) {
this.graphPollingNodeThreshold = response.config.graph_polling_node_threshold;
console.log(`Graph polling threshold set to: ${this.graphPollingNodeThreshold} nodes`);
this.socket = io({
transports: ['websocket', 'polling'],
timeout: 10000,
reconnection: true,
reconnectionAttempts: 5,
reconnectionDelay: 2000
});
this.socket.on('connect', () => {
console.log('✅ WebSocket connected successfully');
this.isConnected = true;
this.reconnectAttempts = 0;
this.updateConnectionStatus('idle');
console.log('📡 Requesting initial status...');
this.socket.emit('get_status');
});
this.socket.on('disconnect', (reason) => {
console.log('❌ WebSocket disconnected:', reason);
this.isConnected = false;
this.updateConnectionStatus('error');
});
this.socket.on('connect_error', (error) => {
console.error('❌ WebSocket connection error:', error);
this.reconnectAttempts++;
this.updateConnectionStatus('error');
if (this.reconnectAttempts >= 5) {
this.showError('WebSocket connection failed. Please refresh the page.');
}
});
this.socket.on('reconnect', (attemptNumber) => {
console.log('✅ WebSocket reconnected after', attemptNumber, 'attempts');
this.isConnected = true;
this.reconnectAttempts = 0;
this.updateConnectionStatus('idle');
this.socket.emit('get_status');
});
// FIXED: Enhanced scan_update handler with detailed graph processing and debugging
this.socket.on('scan_update', (data) => {
console.log('📨 WebSocket update received:', {
status: data.status,
target: data.target_domain,
progress: data.progress_percentage,
graphNodes: data.graph?.nodes?.length || 0,
graphEdges: data.graph?.edges?.length || 0,
timestamp: new Date().toISOString()
});
try {
// Handle status change
if (data.status !== this.scanStatus) {
console.log(`📄 Status change: ${this.scanStatus} → ${data.status}`);
this.handleStatusChange(data.status, data.task_queue_size);
}
this.scanStatus = data.status;
// Update status display
this.updateStatusDisplay(data);
// FIXED: Always update graph if data is present and graph manager exists
if (data.graph && this.graphManager) {
console.log('📊 Processing graph update:', {
nodes: data.graph.nodes?.length || 0,
edges: data.graph.edges?.length || 0,
hasNodes: Array.isArray(data.graph.nodes),
hasEdges: Array.isArray(data.graph.edges),
isInitialized: this.graphManager.isInitialized
});
// FIXED: Initialize graph manager if not already done
if (!this.graphManager.isInitialized) {
console.log('🎯 Initializing graph manager...');
this.graphManager.initialize();
}
// FIXED: Force graph update and verify it worked
const previousNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
const previousEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
console.log('🔄 Before update - Nodes:', previousNodeCount, 'Edges:', previousEdgeCount);
// Store the data for debugging
this.lastGraphData = data.graph;
// Update the graph
this.graphManager.updateGraph(data.graph);
this.lastGraphUpdate = Date.now();
// Verify the update worked
const newNodeCount = this.graphManager.nodes ? this.graphManager.nodes.length : 0;
const newEdgeCount = this.graphManager.edges ? this.graphManager.edges.length : 0;
console.log('🔄 After update - Nodes:', newNodeCount, 'Edges:', newEdgeCount);
if (newNodeCount !== data.graph.nodes.length || newEdgeCount !== data.graph.edges.length) {
console.warn('⚠️ Graph update mismatch!', {
expectedNodes: data.graph.nodes.length,
actualNodes: newNodeCount,
expectedEdges: data.graph.edges.length,
actualEdges: newEdgeCount
});
// Force a complete rebuild if there's a mismatch
console.log('🔧 Force rebuilding graph...');
this.graphManager.clear();
this.graphManager.updateGraph(data.graph);
}
console.log('✅ Graph updated successfully');
// FIXED: Force network redraw if we're using vis.js
if (this.graphManager.network) {
try {
this.graphManager.network.redraw();
console.log('🎨 Network redrawn');
} catch (redrawError) {
console.warn('⚠️ Network redraw failed:', redrawError);
}
}
} else {
if (!data.graph) {
console.log('⚠️ No graph data in WebSocket update');
}
if (!this.graphManager) {
console.log('⚠️ Graph manager not available');
}
}
} catch (error) {
console.warn('Failed to load config, using defaults:', error);
console.error('❌ Error processing WebSocket update:', error);
console.error('Update data:', data);
console.error('Stack trace:', error.stack);
}
});
this.socket.on('error', (error) => {
console.error('❌ WebSocket error:', error);
this.showError('WebSocket communication error');
});
} catch (error) {
console.error('❌ Failed to initialize WebSocket:', error);
this.showError('Failed to establish real-time connection');
}
}
@@ -284,19 +425,36 @@ class DNScopeApp {
}
/**
* Initialize graph visualization with manual refresh button
* FIXED: Initialize graph visualization with enhanced debugging
*/
initializeGraph() {
try {
console.log('Initializing graph manager...');
this.graphManager = new GraphManager('network-graph');
// Set up manual refresh handler
this.graphManager.setManualRefreshHandler(() => {
console.log('Manual graph refresh requested');
this.updateGraph();
// FIXED: Add debugging hooks to graph manager
if (this.graphManager) {
// Override updateGraph to add debugging
const originalUpdateGraph = this.graphManager.updateGraph.bind(this.graphManager);
this.graphManager.updateGraph = (graphData) => {
console.log('🔧 GraphManager.updateGraph called with:', {
nodes: graphData?.nodes?.length || 0,
edges: graphData?.edges?.length || 0,
timestamp: new Date().toISOString()
});
const result = originalUpdateGraph(graphData);
console.log('🔧 GraphManager.updateGraph completed, network state:', {
networkExists: !!this.graphManager.network,
nodeDataSetLength: this.graphManager.nodes?.length || 0,
edgeDataSetLength: this.graphManager.edges?.length || 0
});
return result;
};
}
console.log('Graph manager initialized successfully');
} catch (error) {
console.error('Failed to initialize graph manager:', error);
@@ -304,34 +462,6 @@ class DNScopeApp {
}
}
/**
* Check if graph polling should be enabled based on node count
*/
shouldEnableGraphPolling() {
if (!this.graphManager || !this.graphManager.nodes) {
return true;
}
const nodeCount = this.graphManager.nodes.length;
return nodeCount <= this.graphPollingNodeThreshold;
}
/**
* Update manual refresh button visibility and state.
* The button will be visible whenever auto-polling is disabled,
* and enabled only when a scan is in progress.
*/
updateManualRefreshButton() {
if (!this.graphManager || !this.graphManager.manualRefreshButton) return;
const shouldShow = !this.graphPollingEnabled;
this.graphManager.showManualRefreshButton(shouldShow);
if (shouldShow) {
this.graphManager.manualRefreshButton.disabled = false;
}
}
/**
* Start scan with error handling
*/
@@ -344,7 +474,6 @@ class DNScopeApp {
console.log(`Target: "${target}", Max depth: ${maxDepth}`);
// Validation
if (!target) {
console.log('Validation failed: empty target');
this.showError('Please enter a target domain or IP');
@@ -359,6 +488,19 @@ class DNScopeApp {
return;
}
// FIXED: Ensure WebSocket connection before starting scan
if (!this.isConnected) {
console.log('WebSocket not connected, attempting to connect...');
this.socket.connect();
// Wait a moment for connection
await new Promise(resolve => setTimeout(resolve, 1000));
if (!this.isConnected) {
this.showWarning('WebSocket connection not established. Updates may be delayed.');
}
}
console.log('Validation passed, setting UI state to scanning...');
this.setUIState('scanning');
this.showInfo('Starting reconnaissance scan...');
@@ -376,26 +518,28 @@ class DNScopeApp {
if (response.success) {
this.currentSessionId = response.scan_id;
this.showSuccess('Reconnaissance scan started successfully');
this.showSuccess('Reconnaissance scan started - watching for real-time updates');
if (clearGraph) {
if (clearGraph && this.graphManager) {
console.log('🧹 Clearing graph for new scan');
this.graphManager.clear();
this.graphPollingEnabled = true; // Reset polling when starting fresh
}
console.log(`Scan started for ${target} with depth ${maxDepth}`);
console.log(`✅ Scan started for ${target} with depth ${maxDepth}`);
// Start optimized polling
this.startOptimizedPolling();
// FIXED: Immediately start listening for updates
if (this.socket && this.isConnected) {
console.log('📡 Requesting initial status update...');
this.socket.emit('get_status');
// Force an immediate status update
console.log('Forcing immediate status update...');
setTimeout(() => {
this.updateStatus();
if (this.graphPollingEnabled) {
this.updateGraph();
// Set up periodic status requests as backup (every 5 seconds during scan)
/*this.statusRequestInterval = setInterval(() => {
if (this.isScanning && this.socket && this.isConnected) {
console.log('📡 Periodic status request...');
this.socket.emit('get_status');
}
}, 5000);*/
}
}, 100);
} else {
throw new Error(response.error || 'Failed to start scan');
@@ -408,48 +552,22 @@ class DNScopeApp {
}
}
/**
* Start optimized polling with separate status and graph intervals
*/
startOptimizedPolling() {
console.log('=== STARTING OPTIMIZED POLLING ===');
this.stopPolling(); // Stop any existing polling
// Always poll status for progress bar
this.statusPollInterval = setInterval(() => {
this.updateStatus();
this.loadProviders();
}, 2000);
// Only poll graph if enabled
if (this.graphPollingEnabled) {
this.graphPollInterval = setInterval(() => {
this.updateGraph();
}, 2000);
console.log('Graph polling started');
} else {
console.log('Graph polling disabled due to node count');
}
this.updateManualRefreshButton();
console.log(`Optimized polling started - Status: enabled, Graph: ${this.graphPollingEnabled ? 'enabled' : 'disabled'}`);
}
/**
* Scan stop with immediate UI feedback
*/
// FIXED: Enhanced stop scan with interval cleanup
async stopScan() {
try {
console.log('Stopping scan...');
// Immediately disable stop button and show stopping state
// Clear status request interval
/*if (this.statusRequestInterval) {
clearInterval(this.statusRequestInterval);
this.statusRequestInterval = null;
}*/
if (this.elements.stopScan) {
this.elements.stopScan.disabled = true;
this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOPPING]</span><span>Stopping...</span>';
}
// Show immediate feedback
this.showInfo('Stopping scan...');
const response = await this.apiCall('/api/scan/stop', 'POST');
@@ -457,14 +575,10 @@ class DNScopeApp {
if (response.success) {
this.showSuccess('Scan stop requested');
// Force immediate status update
setTimeout(() => {
this.updateStatus();
}, 100);
// Continue status polling for a bit to catch the status change
// No need to resume graph polling
// Request final status update
if (this.socket && this.isConnected) {
setTimeout(() => this.socket.emit('get_status'), 500);
}
} else {
throw new Error(response.error || 'Failed to stop scan');
}
@@ -473,7 +587,6 @@ class DNScopeApp {
console.error('Failed to stop scan:', error);
this.showError(`Failed to stop scan: ${error.message}`);
// Re-enable stop button on error
if (this.elements.stopScan) {
this.elements.stopScan.disabled = false;
this.elements.stopScan.innerHTML = '<span class="btn-icon">[STOP]</span><span>Terminate Scan</span>';
@@ -539,7 +652,7 @@ class DNScopeApp {
// Get the filename from headers or create one
const contentDisposition = response.headers.get('content-disposition');
let filename = 'DNScope_export.json';
let filename = 'dnsrecon_export.json';
if (contentDisposition) {
const filenameMatch = contentDisposition.match(/filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/);
if (filenameMatch) {
@@ -630,123 +743,38 @@ class DNScopeApp {
}
/**
* Start polling for scan updates with configurable interval
*/
startPolling(interval = 2000) {
console.log('=== STARTING POLLING ===');
if (this.pollInterval) {
console.log('Clearing existing poll interval');
clearInterval(this.pollInterval);
}
this.pollInterval = setInterval(() => {
this.updateStatus();
this.updateGraph();
this.loadProviders();
}, interval);
console.log(`Polling started with ${interval}ms interval`);
}
/**
* Stop polling for updates
*/
stopPolling() {
console.log('=== STOPPING POLLING ===');
if (this.statusPollInterval) {
clearInterval(this.statusPollInterval);
this.statusPollInterval = null;
}
if (this.graphPollInterval) {
clearInterval(this.graphPollInterval);
this.graphPollInterval = null;
}
this.updateManualRefreshButton();
}
/**
* Status update with better error handling
*/
async updateStatus() {
try {
const response = await this.apiCall('/api/scan/status');
if (response.success && response.status) {
const status = response.status;
this.updateStatusDisplay(status);
// Handle status changes
if (status.status !== this.scanStatus) {
console.log(`*** STATUS CHANGED: ${this.scanStatus} -> ${status.status} ***`);
this.handleStatusChange(status.status, status.task_queue_size);
}
this.scanStatus = status.status;
} else {
console.error('Status update failed:', response);
// Don't show error for status updates to avoid spam
}
} catch (error) {
console.error('Failed to update status:', error);
this.showConnectionError();
}
}
/**
* Update graph from server with polling optimization
* FIXED: Update graph from server with enhanced debugging
*/
async updateGraph() {
try {
console.log('Updating graph...');
console.log('Updating graph via API call...');
const response = await this.apiCall('/api/graph');
if (response.success) {
const graphData = response.graph;
console.log('Graph data received:');
console.log('Graph data received from API:');
console.log('- Nodes:', graphData.nodes ? graphData.nodes.length : 0);
console.log('- Edges:', graphData.edges ? graphData.edges.length : 0);
// Always update graph, even if empty - let GraphManager handle placeholder
// FIXED: Always update graph, even if empty - let GraphManager handle placeholder
if (this.graphManager) {
console.log('🔧 Calling GraphManager.updateGraph from API response...');
this.graphManager.updateGraph(graphData);
this.lastGraphUpdate = Date.now();
// Check if we should disable graph polling after this update
const nodeCount = graphData.nodes ? graphData.nodes.length : 0;
const shouldEnablePolling = nodeCount <= this.graphPollingNodeThreshold;
if (this.graphPollingEnabled && !shouldEnablePolling) {
console.log(`Graph polling disabled: ${nodeCount} nodes exceeds threshold of ${this.graphPollingNodeThreshold}`);
this.graphPollingEnabled = false;
this.showWarning(`Graph auto-refresh disabled: ${nodeCount} nodes exceed threshold of ${this.graphPollingNodeThreshold}. Use manual refresh button.`);
// Stop graph polling but keep status polling
if (this.graphPollInterval) {
clearInterval(this.graphPollInterval);
this.graphPollInterval = null;
}
this.updateManualRefreshButton();
}
// Update relationship count in status
const edgeCount = graphData.edges ? graphData.edges.length : 0;
if (this.elements.relationshipsDisplay) {
this.elements.relationshipsDisplay.textContent = edgeCount;
}
console.log('✅ Manual graph update completed');
}
} else {
console.error('Graph update failed:', response);
// Show placeholder when graph update fails
// FIXED: Show placeholder when graph update fails
if (this.graphManager && this.graphManager.container) {
const placeholder = this.graphManager.container.querySelector('.graph-placeholder');
if (placeholder) {
@@ -757,7 +785,7 @@ class DNScopeApp {
} catch (error) {
console.error('Failed to update graph:', error);
// Show placeholder on error
// FIXED: Show placeholder on error
if (this.graphManager && this.graphManager.container) {
const placeholder = this.graphManager.container.querySelector('.graph-placeholder');
if (placeholder) {
@@ -767,6 +795,7 @@ class DNScopeApp {
}
}
/**
* Update status display elements
* @param {Object} status - Status object from server
@@ -837,47 +866,70 @@ class DNScopeApp {
* @param {string} newStatus - New scan status
*/
handleStatusChange(newStatus, task_queue_size) {
console.log(`=== STATUS CHANGE: ${this.scanStatus} -> ${newStatus} ===`);
console.log(`📄 Status change handler: ${this.scanStatus} → ${newStatus}`);
switch (newStatus) {
case 'running':
this.setUIState('scanning', task_queue_size);
this.showSuccess('Scan is running');
this.showSuccess('Scan is running - updates in real-time');
this.updateConnectionStatus('active');
break;
case 'completed':
this.setUIState('completed', task_queue_size);
this.stopPolling();
this.showSuccess('Scan completed successfully');
this.updateConnectionStatus('completed');
this.loadProviders();
console.log('✅ Scan completed - requesting final graph update');
// Request final status to ensure we have the complete graph
setTimeout(() => {
if (this.socket && this.isConnected) {
this.socket.emit('get_status');
}
}, 1000);
// Do final graph update when scan completes
console.log('Scan completed - performing final graph update');
setTimeout(() => this.updateGraph(), 100);
// Clear status request interval
/*if (this.statusRequestInterval) {
clearInterval(this.statusRequestInterval);
this.statusRequestInterval = null;
}*/
break;
case 'failed':
this.setUIState('failed', task_queue_size);
this.stopPolling();
this.showError('Scan failed');
this.updateConnectionStatus('error');
this.loadProviders();
// Clear status request interval
/*if (this.statusRequestInterval) {
clearInterval(this.statusRequestInterval);
this.statusRequestInterval = null;
}*/
break;
case 'stopped':
this.setUIState('stopped', task_queue_size);
this.stopPolling();
this.showSuccess('Scan stopped');
this.updateConnectionStatus('stopped');
this.loadProviders();
// Clear status request interval
if (this.statusRequestInterval) {
clearInterval(this.statusRequestInterval);
this.statusRequestInterval = null;
}
break;
case 'idle':
this.setUIState('idle', task_queue_size);
this.stopPolling();
this.updateConnectionStatus('idle');
// Clear status request interval
/*if (this.statusRequestInterval) {
clearInterval(this.statusRequestInterval);
this.statusRequestInterval = null;
}*/
break;
default:
@@ -925,11 +977,11 @@ class DNScopeApp {
switch (state) {
case 'scanning':
case 'running':
this.isScanning = true;
if (this.graphManager) {
this.graphManager.isScanning = true;
}
if (this.elements.startScan) {
this.elements.startScan.disabled = true;
this.elements.startScan.classList.add('loading');
@@ -957,13 +1009,14 @@ class DNScopeApp {
if (this.graphManager) {
this.graphManager.isScanning = false;
}
if (this.elements.startScan) {
this.elements.startScan.disabled = false;
this.elements.startScan.disabled = !isQueueEmpty;
this.elements.startScan.classList.remove('loading');
this.elements.startScan.innerHTML = '<span class="btn-icon">[RUN]</span><span>Start Reconnaissance</span>';
}
if (this.elements.addToGraph) {
this.elements.addToGraph.disabled = false;
this.elements.addToGraph.disabled = !isQueueEmpty;
this.elements.addToGraph.classList.remove('loading');
}
if (this.elements.stopScan) {
@@ -973,9 +1026,6 @@ class DNScopeApp {
if (this.elements.targetInput) this.elements.targetInput.disabled = false;
if (this.elements.maxDepth) this.elements.maxDepth.disabled = false;
if (this.elements.configureSettings) this.elements.configureSettings.disabled = false;
// Update manual refresh button visibility
this.updateManualRefreshButton();
break;
}
}
@@ -1202,7 +1252,7 @@ class DNScopeApp {
} else {
// API key not configured - ALWAYS show input field
const statusClass = info.enabled ? 'enabled' : 'api-key-required';
const statusText = info.enabled ? '○ Ready for API Key' : '⚠️ API Key Required';
const statusText = info.enabled ? '◯ Ready for API Key' : '⚠️ API Key Required';
inputGroup.innerHTML = `
<div class="provider-header">
@@ -2132,16 +2182,6 @@ class DNScopeApp {
async extractNode(largeEntityId, nodeId) {
try {
console.log(`Extracting node ${nodeId} from large entity ${largeEntityId}`);
// Show immediate feedback
const button = document.querySelector(`[data-node-id="${nodeId}"][data-large-entity-id="${largeEntityId}"]`);
if (button) {
const originalContent = button.innerHTML;
button.innerHTML = '[...]';
button.disabled = true;
}
const response = await this.apiCall('/api/graph/large-entity/extract', 'POST', {
large_entity_id: largeEntityId,
node_id: nodeId,
@@ -2150,32 +2190,13 @@ class DNScopeApp {
if (response.success) {
this.showSuccess(response.message);
// FIXED: Don't update local modal data - let backend be source of truth
// Force immediate graph update to get fresh backend data
console.log('Extraction successful, updating graph with fresh backend data');
await this.updateGraph();
// FIXED: Re-fetch graph data instead of manipulating local state
setTimeout(async () => {
try {
const graphResponse = await this.apiCall('/api/graph');
if (graphResponse.success) {
this.graphManager.updateGraph(graphResponse.graph);
// Update modal with fresh data if still open
if (this.elements.nodeModal && this.elements.nodeModal.style.display === 'block') {
if (this.graphManager.nodes) {
const updatedLargeEntity = this.graphManager.nodes.get(largeEntityId);
if (updatedLargeEntity) {
this.showNodeModal(updatedLargeEntity);
// If the scanner was idle, it's now running. Start polling to see the new node appear.
if (this.scanStatus === 'idle') {
this.socket.emit('get_status');
} else {
// If already scanning, force a quick graph update to see the change sooner.
setTimeout(() => this.socket.emit('get_status'), 500);
}
}
}
}
} catch (error) {
console.error('Error refreshing graph after extraction:', error);
}
}, 100);
} else {
throw new Error(response.error || 'Extraction failed on the server.');
@@ -2183,13 +2204,6 @@ class DNScopeApp {
} catch (error) {
console.error('Failed to extract node:', error);
this.showError(`Extraction failed: ${error.message}`);
// Restore button state on error
const button = document.querySelector(`[data-node-id="${nodeId}"][data-large-entity-id="${largeEntityId}"]`);
if (button) {
button.innerHTML = '[+]';
button.disabled = false;
}
}
}
@@ -2220,8 +2234,8 @@ class DNScopeApp {
*/
getNodeTypeIcon(nodeType) {
const icons = {
'domain': '🌍',
'ip': '📍',
'domain': '🌐',
'ip': '🔢',
'asn': '🏢',
'large_entity': '📦',
'correlation_object': '🔗'
@@ -2789,5 +2803,5 @@ style.textContent = `
document.head.appendChild(style);
// Initialize application when page loads
console.log('Creating DNScopeApp instance...');
const app = new DNScopeApp();
console.log('Creating DNSReconApp instance...');
const app = new DNSReconApp();

7
templates/index.html
View File

@@ -4,9 +4,10 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DNScope - Infrastructure Reconnaissance</title>
<title>DNSRecon - Infrastructure Reconnaissance</title>
<link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
<script src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.7.2/socket.io.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" rel="stylesheet" type="text/css">
<link
href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@300;400;500;700&family=Special+Elite&display=swap"
@@ -18,8 +19,8 @@
<header class="header">
<div class="header-content">
<div class="logo">
<span class="logo-icon">[DN]</span>
<span class="logo-text">Scope</span>
<span class="logo-icon">[DNS]</span>
<span class="logo-text">RECON</span>
</div>
<div class="status-indicator">
<span id="connection-status" class="status-dot"></span>

4
utils/__init__.py
View File

@@ -1,7 +1,7 @@
# DNScope-reduced/utils/__init__.py
# dnsrecon-reduced/utils/__init__.py
"""
Utility modules for DNScope.
Utility modules for DNSRecon.
Contains helper functions, export management, and supporting utilities.
"""

14
utils/export_manager.py
View File

@@ -1,7 +1,7 @@
# DNScope-reduced/utils/export_manager.py
# dnsrecon-reduced/utils/export_manager.py
"""
Centralized export functionality for DNScope.
Centralized export functionality for DNSRecon.
Handles all data export operations with forensic integrity and proper formatting.
ENHANCED: Professional forensic executive summary generation for court-ready documentation.
"""
@@ -18,7 +18,7 @@ from utils.helpers import _is_valid_domain, _is_valid_ip
class ExportManager:
"""
Centralized manager for all DNScope export operations.
Centralized manager for all DNSRecon export operations.
Maintains forensic integrity and provides consistent export formats.
ENHANCED: Advanced forensic analysis and professional reporting capabilities.
"""
@@ -324,7 +324,7 @@ class ExportManager:
"a complete audit trail maintained for forensic integrity.",
"",
f"Investigation completed: {now}",
f"Report authenticated by: DNScope v{self._get_version()}",
f"Report authenticated by: DNSRecon v{self._get_version()}",
"",
"=" * 80,
"END OF REPORT",
@@ -694,7 +694,7 @@ class ExportManager:
if centrality >= threshold]
def _get_version(self) -> str:
"""Get DNScope version for report authentication."""
"""Get DNSRecon version for report authentication."""
return "1.0.0-forensic"
def export_graph_json(self, graph_manager) -> Dict[str, Any]:
@@ -717,7 +717,7 @@ class ExportManager:
'last_modified': graph_manager.last_modified,
'total_nodes': graph_manager.get_node_count(),
'total_edges': graph_manager.get_edge_count(),
'graph_format': 'DNScope_v1_unified_model'
'graph_format': 'dnsrecon_v1_unified_model'
},
'graph': graph_data,
'statistics': graph_manager.get_statistics()
@@ -818,7 +818,7 @@ class ExportManager:
}
extension = extension_map.get(export_type, 'txt')
return f"DNScope_{export_type}_{safe_target}_{timestamp_str}.{extension}"
return f"dnsrecon_{export_type}_{safe_target}_{timestamp_str}.{extension}"
class CustomJSONEncoder(json.JSONEncoder):

2
utils/helpers.py
View File

@@ -1,4 +1,4 @@
# DNScope-reduced/utils/helpers.py
# dnsrecon-reduced/utils/helpers.py
import ipaddress
from typing import Union