mstoeck3/dnsrecon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

full re implementation

Browse Source
This commit is contained in:
overcuriousity 2025-09-10 13:53:32 +02:00
parent 29e36e34be
commit 696cec0723
32 changed files with 4731 additions and 7955 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

330
README.md
View File

@ -1,107 +1,273 @@
# DNS Reconnaissance Tool
# DNSRecon - Passive Infrastructure Reconnaissance Tool
A comprehensive DNS reconnaissance tool designed for investigators to gather intelligence on hostnames and IP addresses through multiple data sources.
DNSRecon is an interactive, passive reconnaissance tool designed to map adversary infrastructure. It operates on a "free-by-default" model, ensuring core functionality without subscriptions, while allowing power users to enhance its capabilities with paid API keys.
**Current Status: Phase 1 Implementation**
- ✅ Core infrastructure and graph engine
- ✅ Certificate transparency data provider (crt.sh)
- ✅ Basic web interface with real-time visualization
- ✅ Forensic logging system
- ✅ JSON export functionality
## Features
- **DNS Resolution**: Query multiple DNS servers (1.1.1.1, 8.8.8.8, 9.9.9.9)
- **TLD Expansion**: Automatically try all IANA TLDs for hostname-only inputs
- **Certificate Transparency**: Query crt.sh for SSL certificate information
- **Recursive Discovery**: Automatically discover and analyze subdomains
- **External Intelligence**: Optional Shodan and VirusTotal integration
- **Multiple Interfaces**: Both CLI and web interface available
- **Comprehensive Reports**: JSON and text output formats
### Core Capabilities
- **Zero Contact Reconnaissance**: Passive data gathering without touching target infrastructure
- **In-Memory Graph Analysis**: Uses NetworkX for efficient relationship mapping
- **Real-Time Visualization**: Interactive graph updates during scanning
- **Forensic Logging**: Complete audit trail of all reconnaissance activities
- **Confidence Scoring**: Weighted relationships based on data source reliability
### Data Sources (Phase 1)
- **Certificate Transparency (crt.sh)**: Discovers domain relationships through SSL certificate SAN analysis
- **Basic DNS Resolution**: A/AAAA record lookups for IP relationships
### Visualization
- **Interactive Network Graph**: Powered by vis.js with cybersecurity theme
- **Node Types**: Domains, IP addresses, certificates, ASNs
- **Confidence-Based Styling**: Visual indicators for relationship strength
- **Real-Time Updates**: Graph builds dynamically as relationships are discovered
## Installation
```bash
# Clone or create the project structure
mkdir dns-recon-tool && cd dns-recon-tool
### Prerequisites
- Python 3.8 or higher
- Modern web browser with JavaScript enabled
# Install dependencies
### Setup
1. **Clone or create the project directory**:
```bash
mkdir dnsrecon
cd dnsrecon
```
2. **Install Python dependencies**:
```bash
pip install -r requirements.txt
```
3. **Verify the directory structure**:
```
dnsrecon/
├── app.py
├── config.py
├── requirements.txt
├── core/
│ ├── __init__.py
│ ├── graph_manager.py
│ ├── scanner.py
│ └── logger.py
├── providers/
│ ├── __init__.py
│ ├── base_provider.py
│ └── crtsh_provider.py
├── static/
│ ├── css/
│ │ └── main.css
│ └── js/
│ ├── graph.js
│ └── main.js
└── templates/
└── index.html
```
## Usage
### Command Line Interface
### Starting the Application
1. **Run the Flask application**:
```bash
# Basic domain scan
python -m src.main example.com
# Try all TLDs for hostname
python -m src.main example
# With API keys and custom depth
python -m src.main example.com --shodan-key YOUR_KEY --virustotal-key YOUR_KEY --max-depth 3
# Save reports
python -m src.main example.com --output results
# JSON only output
python -m src.main example.com --json-only
python app.py
```
### Web Interface
```bash
# Start web server
python -m src.main --web
# Custom port
python -m src.main --web --port 8080
2. **Open your web browser** and navigate to:
```
http://127.0.0.1:5000
```
Then open http://localhost:5000 in your browser.
### Basic Reconnaissance Workflow
1. **Enter Target Domain**: Input the domain you want to investigate (e.g., `example.com`)
2. **Select Recursion Depth**:
- **Depth 1**: Direct relationships only
- **Depth 2**: Recommended for most investigations
- **Depth 3+**: Extended analysis for comprehensive mapping
3. **Start Reconnaissance**: Click "Start Reconnaissance" to begin passive data gathering
4. **Monitor Progress**: Watch the real-time graph build as relationships are discovered
5. **Analyze Results**: Interact with the graph to explore relationships and click nodes for detailed information
6. **Export Data**: Download complete results including graph data and forensic audit trail
### Understanding the Visualization
#### Node Types
- 🟢 **Green Circles**: Domain names
- 🟠 **Orange Squares**: IP addresses
- ⚪ **Gray Diamonds**: SSL certificates
- 🔵 **Blue Triangles**: ASN (Autonomous System) information
#### Edge Confidence
- **Thick Green Lines**: High confidence (≥80%) - Certificate SAN relationships
- **Medium Orange Lines**: Medium confidence (60-79%) - DNS record relationships
- **Thin Gray Lines**: Lower confidence (<60%) - Passive DNS or uncertain relationships
### Example Investigation
Let's investigate `github.com`:
1. Enter `github.com` as the target domain
2. Set recursion depth to 2
3. Start the scan
4. Observe relationships to other GitHub domains discovered through certificate analysis
5. Export results for further analysis
Expected discoveries might include:
- `*.github.com` domains through certificate SANs
- `github.io` and related domains
- Associated IP addresses
- Certificate authority relationships
## Configuration
The tool uses the following default settings:
- DNS Servers: 1.1.1.1, 8.8.8.8, 9.9.9.9
- Max Recursion Depth: 2
- Rate Limits: DNS (10/s), crt.sh (2/s), Shodan (0.5/s), VirusTotal (0.25/s)
### Environment Variables
You can configure DNSRecon using environment variables:
## API Keys
For enhanced reconnaissance, obtain API keys from:
- [Shodan](https://shodan.io) - Port scanning and service detection
- [VirusTotal](https://virustotal.com) - Security analysis and reputation
## Output
The tool generates two types of reports:
### JSON Report
Complete machine-readable data including:
- All discovered hostnames and IPs
- DNS records by type
- Certificate information
- External service results
- Metadata and timing
### Text Report
Human-readable summary with:
- Executive summary
- Hostnames by discovery depth
- IP address analysis
- DNS record details
- Certificate analysis
- Security findings
## Architecture
```bash
# API keys for future providers (Phase 2)
export VIRUSTOTAL_API_KEY="your_api_key_here"
export SHODAN_API_KEY="your_api_key_here"
# Application settings
export DEFAULT_RECURSION_DEPTH=2
export FLASK_DEBUG=False
```
src/
├── main.py # CLI entry point
├── web_app.py # Flask web interface
├── config.py # Configuration management
├── data_structures.py # Data models
├── dns_resolver.py # DNS functionality
├── certificate_checker.py # crt.sh integration
├── shodan_client.py # Shodan API
├── virustotal_client.py # VirusTotal API
├── tld_fetcher.py # IANA TLD handling
├── reconnaissance.py # Main logic
└── report_generator.py # Report generation
### Rate Limiting
DNSRecon includes built-in rate limiting to be respectful to data sources:
- **crt.sh**: 60 requests per minute
- **DNS queries**: 100 requests per minute
## Data Export Format
Results are exported as JSON with the following structure:
```json
{
"scan_metadata": {
"target_domain": "example.com",
"max_depth": 2,
"final_status": "completed"
},
"graph_data": {
"nodes": [...],
"edges": [...]
},
"forensic_audit": {
"session_metadata": {...},
"api_requests": [...],
"relationships": [...]
},
"provider_statistics": {...}
}
```
## Forensic Integrity
DNSRecon maintains complete forensic integrity:
- **API Request Logging**: Every external request is logged with timestamps, URLs, and responses
- **Relationship Provenance**: Each discovered relationship includes source provider and discovery method
- **Session Tracking**: Unique session IDs for investigation continuity
- **Confidence Metadata**: Scoring rationale for all relationships
- **Export Integrity**: Complete audit trail included in all exports
## Architecture Overview
### Core Components
- **GraphManager**: NetworkX-based in-memory graph with confidence scoring
- **Scanner**: Multi-provider orchestration with depth-limited BFS exploration
- **ForensicLogger**: Thread-safe audit trail with structured logging
- **BaseProvider**: Abstract interface for data source plugins
### Data Flow
1. User initiates scan via web interface
2. Scanner coordinates multiple data providers
3. Relationships discovered and added to in-memory graph
4. Real-time updates sent to web interface
5. Graph visualization updates dynamically
6. Complete audit trail maintained throughout
## Troubleshooting
### Common Issues
**Graph not displaying**:
- Ensure JavaScript is enabled in your browser
- Check browser console for errors
- Verify vis.js library is loading correctly
**Scan fails to start**:
- Check target domain is valid
- Ensure crt.sh is accessible from your network
- Review Flask console output for errors
**No relationships discovered**:
- Some domains may have limited certificate transparency data
- Try a well-known domain like `google.com` to verify functionality
- Check provider status in the interface
### Debug Mode
Enable debug mode for verbose logging:
```bash
export FLASK_DEBUG=True
python app.py
```
## Development Roadmap
### Phase 2 (Planned)
- Multi-provider system with Shodan and VirusTotal integration
- Real-time scanning with enhanced visualization
- Provider health monitoring and failure recovery
### Phase 3 (Planned)
- Advanced correlation algorithms
- Enhanced forensic reporting
- Performance optimization for large investigations
## Security Considerations
- **No Persistent Storage**: All data stored in memory only
- **API Keys**: Stored in memory only, never written to disk
- **Rate Limiting**: Prevents abuse of external services
- **Local Use Only**: No authentication required (designed for local use)
## Contributing
DNSRecon follows a phased development approach. Currently in Phase 1 with core infrastructure completed.
### Code Quality Standards
- Follow PEP 8 for Python code
- Comprehensive docstrings for all functions
- Type hints where appropriate
- Forensic logging for all external interactions
## License
This project is intended for legitimate security research and infrastructure analysis. Users are responsible for compliance with applicable laws and regulations.
## Support
For issues and questions:
1. Check the troubleshooting section above
2. Review the Flask console output for error details
3. Ensure all dependencies are properly installed
---
**DNSRecon v1.0 - Phase 1 Implementation**
*Passive Infrastructure Reconnaissance for Security Professionals*

329
app.py Normal file
View File

@ -0,0 +1,329 @@
"""
Flask application entry point for DNSRecon web interface.
Provides REST API endpoints and serves the web interface.
"""
import json
import traceback
from flask import Flask, render_template, request, jsonify, send_file
from datetime import datetime
import io
from core.scanner import scanner
from config import config
app = Flask(__name__)
app.config['SECRET_KEY'] = 'dnsrecon-dev-key-change-in-production'
@app.route('/')
def index():
"""Serve the main web interface."""
return render_template('index.html')
@app.route('/api/scan/start', methods=['POST'])
def start_scan():
"""
Start a new reconnaissance scan.
Expects JSON payload:
{
"target_domain": "example.com",
"max_depth": 2
}
"""
print("=== API: /api/scan/start called ===")
try:
print("Getting JSON data from request...")
data = request.get_json()
print(f"Request data: {data}")
if not data or 'target_domain' not in data:
print("ERROR: Missing target_domain in request")
return jsonify({
'success': False,
'error': 'Missing target_domain in request'
}), 400
target_domain = data['target_domain'].strip()
max_depth = data.get('max_depth', config.default_recursion_depth)
print(f"Parsed - target_domain: '{target_domain}', max_depth: {max_depth}")
# Validation
if not target_domain:
print("ERROR: Target domain cannot be empty")
return jsonify({
'success': False,
'error': 'Target domain cannot be empty'
}), 400
if not isinstance(max_depth, int) or max_depth < 1 or max_depth > 5:
print(f"ERROR: Invalid max_depth: {max_depth}")
return jsonify({
'success': False,
'error': 'Max depth must be an integer between 1 and 5'
}), 400
print("Validation passed, calling scanner.start_scan...")
# Start scan
success = scanner.start_scan(target_domain, max_depth)
print(f"scanner.start_scan returned: {success}")
if success:
session_id = scanner.logger.session_id
print(f"Scan started successfully with session ID: {session_id}")
return jsonify({
'success': True,
'message': 'Scan started successfully',
'scan_id': session_id
})
else:
print("ERROR: Scanner returned False")
return jsonify({
'success': False,
'error': 'Failed to start scan (scan may already be running)'
}), 409
except Exception as e:
print(f"ERROR: Exception in start_scan endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/scan/stop', methods=['POST'])
def stop_scan():
"""Stop the current scan."""
print("=== API: /api/scan/stop called ===")
try:
success = scanner.stop_scan()
if success:
return jsonify({
'success': True,
'message': 'Scan stop requested'
})
else:
return jsonify({
'success': False,
'error': 'No active scan to stop'
}), 400
except Exception as e:
print(f"ERROR: Exception in stop_scan endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/scan/status', methods=['GET'])
def get_scan_status():
"""Get current scan status and progress."""
try:
status = scanner.get_scan_status()
return jsonify({
'success': True,
'status': status
})
except Exception as e:
print(f"ERROR: Exception in get_scan_status endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/graph', methods=['GET'])
def get_graph_data():
"""Get current graph data for visualization."""
try:
graph_data = scanner.get_graph_data()
return jsonify({
'success': True,
'graph': graph_data
})
except Exception as e:
print(f"ERROR: Exception in get_graph_data endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/export', methods=['GET'])
def export_results():
"""Export complete scan results as downloadable JSON."""
try:
# Get complete results
results = scanner.export_results()
# Create filename with timestamp
timestamp = datetime.now(datetime.UTC).strftime('%Y%m%d_%H%M%S')
target = scanner.current_target or 'unknown'
filename = f"dnsrecon_{target}_{timestamp}.json"
# Create in-memory file
json_data = json.dumps(results, indent=2, ensure_ascii=False)
file_obj = io.BytesIO(json_data.encode('utf-8'))
return send_file(
file_obj,
as_attachment=True,
download_name=filename,
mimetype='application/json'
)
except Exception as e:
print(f"ERROR: Exception in export_results endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Export failed: {str(e)}'
}), 500
@app.route('/api/providers', methods=['GET'])
def get_providers():
"""Get information about available providers."""
print("=== API: /api/providers called ===")
try:
provider_stats = scanner.get_provider_statistics()
# Add configuration information
provider_info = {}
for provider_name, stats in provider_stats.items():
provider_info[provider_name] = {
'statistics': stats,
'enabled': config.is_provider_enabled(provider_name),
'rate_limit': config.get_rate_limit(provider_name),
'requires_api_key': provider_name in ['shodan', 'virustotal']
}
print(f"Returning provider info: {list(provider_info.keys())}")
return jsonify({
'success': True,
'providers': provider_info
})
except Exception as e:
print(f"ERROR: Exception in get_providers endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/config/api-keys', methods=['POST'])
def set_api_keys():
"""
Set API keys for providers (stored in memory only).
Expects JSON payload:
{
"shodan": "api_key_here",
"virustotal": "api_key_here"
}
"""
try:
data = request.get_json()
if not data:
return jsonify({
'success': False,
'error': 'No API keys provided'
}), 400
updated_providers = []
for provider, api_key in data.items():
if provider in ['shodan', 'virustotal'] and api_key.strip():
success = config.set_api_key(provider, api_key.strip())
if success:
updated_providers.append(provider)
if updated_providers:
# Reinitialize scanner providers
scanner._initialize_providers()
return jsonify({
'success': True,
'message': f'API keys updated for: {", ".join(updated_providers)}',
'updated_providers': updated_providers
})
else:
return jsonify({
'success': False,
'error': 'No valid API keys were provided'
}), 400
except Exception as e:
print(f"ERROR: Exception in set_api_keys endpoint: {e}")
traceback.print_exc()
return jsonify({
'success': False,
'error': f'Internal server error: {str(e)}'
}), 500
@app.route('/api/health', methods=['GET'])
def health_check():
"""Health check endpoint."""
return jsonify({
'success': True,
'status': 'healthy',
'timestamp': datetime.now(datetime.UTC).isoformat(),
'version': '1.0.0-phase1'
})
@app.errorhandler(404)
def not_found(error):
"""Handle 404 errors."""
return jsonify({
'success': False,
'error': 'Endpoint not found'
}), 404
@app.errorhandler(500)
def internal_error(error):
"""Handle 500 errors."""
print(f"ERROR: 500 Internal Server Error: {error}")
traceback.print_exc()
return jsonify({
'success': False,
'error': 'Internal server error'
}), 500
if __name__ == '__main__':
print("Starting DNSRecon Flask application...")
# Load configuration from environment
config.load_from_env()
# Start Flask application
print(f"Starting server on {config.flask_host}:{config.flask_port}")
app.run(
host=config.flask_host,
port=config.flask_port,
debug=config.flask_debug,
threaded=True
)

117
config.py Normal file
View File

@ -0,0 +1,117 @@
"""
Configuration management for DNSRecon tool.
Handles API key storage, rate limiting, and default settings.
"""
import os
from typing import Dict, Optional
class Config:
"""Configuration manager for DNSRecon application."""
def __init__(self):
"""Initialize configuration with default values."""
self.api_keys: Dict[str, Optional[str]] = {
'shodan': None,
'virustotal': None
}
# Default settings
self.default_recursion_depth = 2
self.default_timeout = 30
self.max_concurrent_requests = 5
# Rate limiting settings (requests per minute)
self.rate_limits = {
'crtsh': 60, # Free service, be respectful
'virustotal': 4, # Free tier limit
'shodan': 60, # API dependent
'dns': 100 # Local DNS queries
}
# Provider settings
self.enabled_providers = {
'crtsh': True, # Always enabled (free)
'dns': True, # Always enabled (free)
'virustotal': False, # Requires API key
'shodan': False # Requires API key
}
# Logging configuration
self.log_level = 'INFO'
self.log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
# Flask configuration
self.flask_host = '127.0.0.1'
self.flask_port = 5000
self.flask_debug = True
def set_api_key(self, provider: str, api_key: str) -> bool:
"""
Set API key for a provider.
Args:
provider: Provider name (shodan, virustotal)
api_key: API key string
Returns:
bool: True if key was set successfully
"""
if provider in self.api_keys:
self.api_keys[provider] = api_key
self.enabled_providers[provider] = True if api_key else False
return True
return False
def get_api_key(self, provider: str) -> Optional[str]:
"""
Get API key for a provider.
Args:
provider: Provider name
Returns:
API key or None if not set
"""
return self.api_keys.get(provider)
def is_provider_enabled(self, provider: str) -> bool:
"""
Check if a provider is enabled.
Args:
provider: Provider name
Returns:
bool: True if provider is enabled
"""
return self.enabled_providers.get(provider, False)
def get_rate_limit(self, provider: str) -> int:
"""
Get rate limit for a provider.
Args:
provider: Provider name
Returns:
Rate limit in requests per minute
"""
return self.rate_limits.get(provider, 60)
def load_from_env(self):
"""Load configuration from environment variables."""
if os.getenv('VIRUSTOTAL_API_KEY'):
self.set_api_key('virustotal', os.getenv('VIRUSTOTAL_API_KEY'))
if os.getenv('SHODAN_API_KEY'):
self.set_api_key('shodan', os.getenv('SHODAN_API_KEY'))
# Override default settings from environment
self.default_recursion_depth = int(os.getenv('DEFAULT_RECURSION_DEPTH', '2'))
self.flask_debug = os.getenv('FLASK_DEBUG', 'True').lower() == 'true'
# Global configuration instance
config = Config()

22
core/__init__.py Normal file
View File

@ -0,0 +1,22 @@
"""
Core modules for DNSRecon passive reconnaissance tool.
Contains graph management, scanning orchestration, and forensic logging.
"""
from .graph_manager import GraphManager, NodeType, RelationshipType
from .scanner import Scanner, ScanStatus, scanner
from .logger import ForensicLogger, get_forensic_logger, new_session
__all__ = [
'GraphManager',
'NodeType',
'RelationshipType',
'Scanner',
'ScanStatus',
'scanner',
'ForensicLogger',
'get_forensic_logger',
'new_session'
]
__version__ = "1.0.0-phase1"

355
core/graph_manager.py Normal file
View File

@ -0,0 +1,355 @@
"""
Graph data model for DNSRecon using NetworkX.
Manages in-memory graph storage with confidence scoring and forensic metadata.
"""
import json
import threading
from datetime import datetime
from typing import Dict, List, Any, Optional, Tuple, Set
from enum import Enum
import networkx as nx
class NodeType(Enum):
"""Enumeration of supported node types."""
DOMAIN = "domain"
IP = "ip"
CERTIFICATE = "certificate"
ASN = "asn"
class RelationshipType(Enum):
"""Enumeration of supported relationship types with confidence scores."""
SAN_CERTIFICATE = ("san", 0.9) # Certificate SAN relationships
A_RECORD = ("a_record", 0.8) # A/AAAA record relationships
CNAME_RECORD = ("cname", 0.8) # CNAME relationships
PASSIVE_DNS = ("passive_dns", 0.6) # Passive DNS relationships
ASN_MEMBERSHIP = ("asn", 0.7) # ASN relationships
MX_RECORD = ("mx_record", 0.7) # MX record relationships
NS_RECORD = ("ns_record", 0.7) # NS record relationships
def __init__(self, relationship_name: str, default_confidence: float):
self.relationship_name = relationship_name
self.default_confidence = default_confidence
class GraphManager:
"""
Thread-safe graph manager for DNSRecon infrastructure mapping.
Uses NetworkX for in-memory graph storage with confidence scoring.
"""
def __init__(self):
"""Initialize empty directed graph."""
self.graph = nx.DiGraph()
#self.lock = threading.Lock()
self.creation_time = datetime.now(datetime.UTC).isoformat()
self.last_modified = self.creation_time
def add_node(self, node_id: str, node_type: NodeType,
metadata: Optional[Dict[str, Any]] = None) -> bool:
"""
Add a node to the graph.
Args:
node_id: Unique identifier for the node
node_type: Type of the node (Domain, IP, Certificate, ASN)
metadata: Additional metadata for the node
Returns:
bool: True if node was added, False if it already exists
"""
if self.graph.has_node(node_id):
# Update metadata if node exists
existing_metadata = self.graph.nodes[node_id].get('metadata', {})
if metadata:
existing_metadata.update(metadata)
self.graph.nodes[node_id]['metadata'] = existing_metadata
return False
node_attributes = {
'type': node_type.value,
'added_timestamp': datetime.now(datetime.UTC).isoformat(),
'metadata': metadata or {}
}
self.graph.add_node(node_id, **node_attributes)
self.last_modified = datetime.now(datetime.UTC).isoformat()
return True
def add_edge(self, source_id: str, target_id: str,
relationship_type: RelationshipType,
confidence_score: Optional[float] = None,
source_provider: str = "unknown",
raw_data: Optional[Dict[str, Any]] = None) -> bool:
"""
Add an edge between two nodes.
Args:
source_id: Source node identifier
target_id: Target node identifier
relationship_type: Type of relationship
confidence_score: Custom confidence score (overrides default)
source_provider: Provider that discovered this relationship
raw_data: Raw data from provider response
Returns:
bool: True if edge was added, False if it already exists
"""
#with self.lock:
# Ensure both nodes exist
if not self.graph.has_node(source_id) or not self.graph.has_node(target_id):
return False
# Check if edge already exists
if self.graph.has_edge(source_id, target_id):
# Update confidence score if new score is higher
existing_confidence = self.graph.edges[source_id, target_id]['confidence_score']
new_confidence = confidence_score or relationship_type.default_confidence
if new_confidence > existing_confidence:
self.graph.edges[source_id, target_id]['confidence_score'] = new_confidence
self.graph.edges[source_id, target_id]['updated_timestamp'] = datetime.now(datetime.UTC).isoformat()
self.graph.edges[source_id, target_id]['updated_by'] = source_provider
return False
edge_attributes = {
'relationship_type': relationship_type.relationship_name,
'confidence_score': confidence_score or relationship_type.default_confidence,
'source_provider': source_provider,
'discovery_timestamp': datetime.now(datetime.UTC).isoformat(),
'raw_data': raw_data or {}
}
self.graph.add_edge(source_id, target_id, **edge_attributes)
self.last_modified = datetime.now(datetime.UTC).isoformat()
return True
def get_node_count(self) -> int:
"""Get total number of nodes in the graph."""
#with self.lock:
return self.graph.number_of_nodes()
def get_edge_count(self) -> int:
"""Get total number of edges in the graph."""
#with self.lock:
return self.graph.number_of_edges()
def get_nodes_by_type(self, node_type: NodeType) -> List[str]:
"""
Get all nodes of a specific type.
Args:
node_type: Type of nodes to retrieve
Returns:
List of node identifiers
"""
#with self.lock:
return [
node_id for node_id, attributes in self.graph.nodes(data=True)
if attributes.get('type') == node_type.value
]
def get_neighbors(self, node_id: str) -> List[str]:
"""
Get all neighboring nodes (both incoming and outgoing).
Args:
node_id: Node identifier
Returns:
List of neighboring node identifiers
"""
#with self.lock:
if not self.graph.has_node(node_id):
return []
predecessors = list(self.graph.predecessors(node_id))
successors = list(self.graph.successors(node_id))
return list(set(predecessors + successors))
def get_high_confidence_edges(self, min_confidence: float = 0.8) -> List[Tuple[str, str, Dict]]:
"""
Get edges with confidence score above threshold.
Args:
min_confidence: Minimum confidence threshold
Returns:
List of tuples (source, target, attributes)
"""
#with self.lock:
return [
(source, target, attributes)
for source, target, attributes in self.graph.edges(data=True)
if attributes.get('confidence_score', 0) >= min_confidence
]
def get_graph_data(self) -> Dict[str, Any]:
"""
Export graph data for visualization.
Returns:
Dictionary containing nodes and edges for frontend visualization
"""
#with self.lock:
nodes = []
edges = []
# Format nodes for visualization
for node_id, attributes in self.graph.nodes(data=True):
node_data = {
'id': node_id,
'label': node_id,
'type': attributes.get('type', 'unknown'),
'metadata': attributes.get('metadata', {}),
'added_timestamp': attributes.get('added_timestamp')
}
# Color coding by type
type_colors = {
'domain': '#00ff41', # Green for domains
'ip': '#ff9900', # Amber for IPs
'certificate': '#c7c7c7', # Gray for certificates
'asn': '#00aaff' # Blue for ASNs
}
node_data['color'] = type_colors.get(attributes.get('type'), '#ffffff')
nodes.append(node_data)
# Format edges for visualization
for source, target, attributes in self.graph.edges(data=True):
edge_data = {
'from': source,
'to': target,
'label': attributes.get('relationship_type', ''),
'confidence_score': attributes.get('confidence_score', 0),
'source_provider': attributes.get('source_provider', ''),
'discovery_timestamp': attributes.get('discovery_timestamp')
}
# Edge styling based on confidence
confidence = attributes.get('confidence_score', 0)
if confidence >= 0.8:
edge_data['color'] = '#00ff41' # Green for high confidence
edge_data['width'] = 3
elif confidence >= 0.6:
edge_data['color'] = '#ff9900' # Amber for medium confidence
edge_data['width'] = 2
else:
edge_data['color'] = '#444444' # Dark gray for low confidence
edge_data['width'] = 1
edges.append(edge_data)
return {
'nodes': nodes,
'edges': edges,
'statistics': {
'node_count': len(nodes),
'edge_count': len(edges),
'creation_time': self.creation_time,
'last_modified': self.last_modified
}
}
def export_json(self) -> Dict[str, Any]:
"""
Export complete graph data as JSON for download.
Returns:
Dictionary containing complete graph data with metadata
"""
#with self.lock:
# Get basic graph data
graph_data = self.get_graph_data()
# Add comprehensive metadata
export_data = {
'export_metadata': {
'export_timestamp': datetime.now(datetime.UTC).isoformat(),
'graph_creation_time': self.creation_time,
'last_modified': self.last_modified,
'total_nodes': self.graph.number_of_nodes(),
'total_edges': self.graph.number_of_edges(),
'graph_format': 'dnsrecon_v1'
},
'nodes': graph_data['nodes'],
'edges': graph_data['edges'],
'node_types': [node_type.value for node_type in NodeType],
'relationship_types': [
{
'name': rel_type.relationship_name,
'default_confidence': rel_type.default_confidence
}
for rel_type in RelationshipType
],
'confidence_distribution': self._get_confidence_distribution()
}
return export_data
def _get_confidence_distribution(self) -> Dict[str, int]:
"""Get distribution of confidence scores."""
distribution = {'high': 0, 'medium': 0, 'low': 0}
for _, _, attributes in self.graph.edges(data=True):
confidence = attributes.get('confidence_score', 0)
if confidence >= 0.8:
distribution['high'] += 1
elif confidence >= 0.6:
distribution['medium'] += 1
else:
distribution['low'] += 1
return distribution
def get_statistics(self) -> Dict[str, Any]:
"""
Get comprehensive graph statistics.
Returns:
Dictionary containing various graph metrics
"""
#with self.lock:
stats = {
'basic_metrics': {
'total_nodes': self.graph.number_of_nodes(),
'total_edges': self.graph.number_of_edges(),
'creation_time': self.creation_time,
'last_modified': self.last_modified
},
'node_type_distribution': {},
'relationship_type_distribution': {},
'confidence_distribution': self._get_confidence_distribution(),
'provider_distribution': {}
}
# Node type distribution
for node_type in NodeType:
count = len(self.get_nodes_by_type(node_type))
stats['node_type_distribution'][node_type.value] = count
# Relationship type distribution
for _, _, attributes in self.graph.edges(data=True):
rel_type = attributes.get('relationship_type', 'unknown')
stats['relationship_type_distribution'][rel_type] = \
stats['relationship_type_distribution'].get(rel_type, 0) + 1
# Provider distribution
for _, _, attributes in self.graph.edges(data=True):
provider = attributes.get('source_provider', 'unknown')
stats['provider_distribution'][provider] = \
stats['provider_distribution'].get(provider, 0) + 1
return stats
def clear(self) -> None:
"""Clear all nodes and edges from the graph."""
#with self.lock:
self.graph.clear()
self.creation_time = datetime.now(datetime.UTC).isoformat()
self.last_modified = self.creation_time

270
core/logger.py Normal file
View File

@ -0,0 +1,270 @@
"""
Forensic logging system for DNSRecon tool.
Provides structured audit trail for all reconnaissance activities.
"""
import json
import logging
import threading
from datetime import datetime
from typing import Dict, Any, Optional, List
from dataclasses import dataclass, asdict
@dataclass
class APIRequest:
"""Structured representation of an API request for forensic logging."""
timestamp: str
provider: str
url: str
method: str
status_code: Optional[int]
response_size: Optional[int]
duration_ms: Optional[float]
error: Optional[str]
target_indicator: str
discovery_context: Optional[str]
@dataclass
class RelationshipDiscovery:
"""Structured representation of a discovered relationship."""
timestamp: str
source_node: str
target_node: str
relationship_type: str
confidence_score: float
provider: str
raw_data: Dict[str, Any]
discovery_method: str
class ForensicLogger:
"""
Thread-safe forensic logging system for DNSRecon.
Maintains detailed audit trail of all reconnaissance activities.
"""
def __init__(self, session_id: str = None):
"""
Initialize forensic logger.
Args:
session_id: Unique identifier for this reconnaissance session
"""
self.session_id = session_id or self._generate_session_id()
#self.lock = threading.Lock()
# Initialize audit trail storage
self.api_requests: List[APIRequest] = []
self.relationships: List[RelationshipDiscovery] = []
self.session_metadata = {
'session_id': self.session_id,
'start_time': datetime.now(datetime.UTC).isoformat(),
'end_time': None,
'total_requests': 0,
'total_relationships': 0,
'providers_used': set(),
'target_domains': set()
}
# Configure standard logger
self.logger = logging.getLogger(f'dnsrecon.{self.session_id}')
self.logger.setLevel(logging.INFO)
# Create formatter for structured logging
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Add console handler if not already present
if not self.logger.handlers:
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
self.logger.addHandler(console_handler)
def _generate_session_id(self) -> str:
"""Generate unique session identifier."""
return f"dnsrecon_{datetime.now(datetime.UTC).strftime('%Y%m%d_%H%M%S')}"
def log_api_request(self, provider: str, url: str, method: str = "GET",
status_code: Optional[int] = None,
response_size: Optional[int] = None,
duration_ms: Optional[float] = None,
error: Optional[str] = None,
target_indicator: str = "",
discovery_context: Optional[str] = None) -> None:
"""
Log an API request for forensic audit trail.
Args:
provider: Name of the data provider
url: Request URL
method: HTTP method
status_code: HTTP response status code
response_size: Size of response in bytes
duration_ms: Request duration in milliseconds
error: Error message if request failed
target_indicator: The indicator being investigated
discovery_context: Context of how this indicator was discovered
"""
#with self.lock:
api_request = APIRequest(
timestamp=datetime.now(datetime.UTC).isoformat(),
provider=provider,
url=url,
method=method,
status_code=status_code,
response_size=response_size,
duration_ms=duration_ms,
error=error,
target_indicator=target_indicator,
discovery_context=discovery_context
)
self.api_requests.append(api_request)
self.session_metadata['total_requests'] += 1
self.session_metadata['providers_used'].add(provider)
if target_indicator:
self.session_metadata['target_domains'].add(target_indicator)
# Log to standard logger
if error:
self.logger.error(f"API Request Failed - {provider}: {url} - {error}")
else:
self.logger.info(f"API Request - {provider}: {url} - Status: {status_code}")
def log_relationship_discovery(self, source_node: str, target_node: str,
relationship_type: str, confidence_score: float,
provider: str, raw_data: Dict[str, Any],
discovery_method: str) -> None:
"""
Log discovery of a new relationship between indicators.
Args:
source_node: Source node identifier
target_node: Target node identifier
relationship_type: Type of relationship (e.g., 'SAN', 'A_Record')
confidence_score: Confidence score (0.0 to 1.0)
provider: Provider that discovered this relationship
raw_data: Raw data from provider response
discovery_method: Method used to discover relationship
"""
#with self.lock:
relationship = RelationshipDiscovery(
timestamp=datetime.now(datetime.UTC).isoformat(),
source_node=source_node,
target_node=target_node,
relationship_type=relationship_type,
confidence_score=confidence_score,
provider=provider,
raw_data=raw_data,
discovery_method=discovery_method
)
self.relationships.append(relationship)
self.session_metadata['total_relationships'] += 1
self.logger.info(
f"Relationship Discovered - {source_node} -> {target_node} "
f"({relationship_type}) - Confidence: {confidence_score:.2f} - Provider: {provider}"
)
def log_scan_start(self, target_domain: str, recursion_depth: int,
enabled_providers: List[str]) -> None:
"""Log the start of a reconnaissance scan."""
self.logger.info(f"Scan Started - Target: {target_domain}, Depth: {recursion_depth}")
self.logger.info(f"Enabled Providers: {', '.join(enabled_providers)}")
#with self.lock:
self.session_metadata['target_domains'].add(target_domain)
def log_scan_complete(self) -> None:
"""Log the completion of a reconnaissance scan."""
#with self.lock:
self.session_metadata['end_time'] = datetime.now(datetime.UTC).isoformat()
self.session_metadata['providers_used'] = list(self.session_metadata['providers_used'])
self.session_metadata['target_domains'] = list(self.session_metadata['target_domains'])
self.logger.info(f"Scan Complete - Session: {self.session_id}")
self.logger.info(f"Total API Requests: {self.session_metadata['total_requests']}")
self.logger.info(f"Total Relationships: {self.session_metadata['total_relationships']}")
def export_audit_trail(self) -> Dict[str, Any]:
"""
Export complete audit trail for forensic analysis.
Returns:
Dictionary containing complete session audit trail
"""
#with self.lock:
return {
'session_metadata': self.session_metadata.copy(),
'api_requests': [asdict(req) for req in self.api_requests],
'relationships': [asdict(rel) for rel in self.relationships],
'export_timestamp': datetime.now(datetime.UTC).isoformat()
}
def get_forensic_summary(self) -> Dict[str, Any]:
"""
Get summary statistics for forensic reporting.
Returns:
Dictionary containing summary statistics
"""
#with self.lock:
provider_stats = {}
for provider in self.session_metadata['providers_used']:
provider_requests = [req for req in self.api_requests if req.provider == provider]
provider_relationships = [rel for rel in self.relationships if rel.provider == provider]
provider_stats[provider] = {
'total_requests': len(provider_requests),
'successful_requests': len([req for req in provider_requests if req.error is None]),
'failed_requests': len([req for req in provider_requests if req.error is not None]),
'relationships_discovered': len(provider_relationships),
'avg_confidence': sum(rel.confidence_score for rel in provider_relationships) / len(provider_relationships) if provider_relationships else 0
}
return {
'session_id': self.session_id,
'duration_minutes': self._calculate_session_duration(),
'total_requests': self.session_metadata['total_requests'],
'total_relationships': self.session_metadata['total_relationships'],
'unique_indicators': len(set([rel.source_node for rel in self.relationships] + [rel.target_node for rel in self.relationships])),
'provider_statistics': provider_stats
}
def _calculate_session_duration(self) -> float:
"""Calculate session duration in minutes."""
if not self.session_metadata['end_time']:
end_time = datetime.now(datetime.UTC)
else:
end_time = datetime.fromisoformat(self.session_metadata['end_time'])
start_time = datetime.fromisoformat(self.session_metadata['start_time'])
duration = (end_time - start_time).total_seconds() / 60
return round(duration, 2)
# Global logger instance for the current session
_current_logger: Optional[ForensicLogger] = None
_logger_lock = threading.Lock()
def get_forensic_logger() -> ForensicLogger:
"""Get or create the current forensic logger instance."""
global _current_logger
with _logger_lock:
if _current_logger is None:
_current_logger = ForensicLogger()
return _current_logger
def new_session() -> ForensicLogger:
"""Start a new forensic logging session."""
global _current_logger
with _logger_lock:
_current_logger = ForensicLogger()
return _current_logger

461
core/scanner.py Normal file
View File

@ -0,0 +1,461 @@
"""
Main scanning orchestrator for DNSRecon.
Coordinates data gathering from multiple providers and builds the infrastructure graph.
"""
import threading
import time
import traceback
from typing import List, Set, Dict, Any, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
from core.graph_manager import GraphManager, NodeType, RelationshipType
from core.logger import get_forensic_logger, new_session
from providers.crtsh_provider import CrtShProvider
from config import config
class ScanStatus:
"""Enumeration of scan statuses."""
IDLE = "idle"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
STOPPED = "stopped"
class Scanner:
"""
Main scanning orchestrator for DNSRecon passive reconnaissance.
Manages multi-provider data gathering and graph construction.
"""
def __init__(self):
"""Initialize scanner with default providers and empty graph."""
print("Initializing Scanner instance...")
try:
from providers.base_provider import BaseProvider
self.graph = GraphManager()
self.providers: List[BaseProvider] = []
self.status = ScanStatus.IDLE
self.current_target = None
self.current_depth = 0
self.max_depth = 2
self.stop_requested = False
self.scan_thread = None
# Scanning progress tracking
self.total_indicators_found = 0
self.indicators_processed = 0
self.current_indicator = ""
# Initialize providers
print("Calling _initialize_providers...")
self._initialize_providers()
# Initialize logger
print("Initializing forensic logger...")
self.logger = get_forensic_logger()
print("Scanner initialization complete")
except Exception as e:
print(f"ERROR: Scanner initialization failed: {e}")
traceback.print_exc()
raise
def _initialize_providers(self) -> None:
"""Initialize available providers based on configuration."""
self.providers = []
print("Initializing providers...")
# Always add free providers
if config.is_provider_enabled('crtsh'):
try:
crtsh_provider = CrtShProvider()
if crtsh_provider.is_available():
self.providers.append(crtsh_provider)
print("✓ CrtSh provider initialized successfully")
else:
print("✗ CrtSh provider is not available")
except Exception as e:
print(f"✗ Failed to initialize CrtSh provider: {e}")
traceback.print_exc()
print(f"Initialized {len(self.providers)} providers")
def _debug_threads(self):
"""Debug function to show current threads."""
print("=== THREAD DEBUG INFO ===")
for t in threading.enumerate():
print(f"Thread: {t.name} | Alive: {t.is_alive()} | Daemon: {t.daemon}")
print("=== END THREAD DEBUG ===")
def start_scan(self, target_domain: str, max_depth: int = 2) -> bool:
"""
Start a new reconnaissance scan.
Args:
target_domain: Initial domain to investigate
max_depth: Maximum recursion depth
Returns:
bool: True if scan started successfully
"""
print(f"Scanner.start_scan called with target='{target_domain}', depth={max_depth}")
try:
print("Checking current status...")
self._debug_threads()
if self.status == ScanStatus.RUNNING:
print("Scan already running, rejecting new scan")
return False
# Check if we have any providers
if not self.providers:
print("No providers available, cannot start scan")
return False
print(f"Current status: {self.status}, Providers: {len(self.providers)}")
# Stop any existing scan thread
if self.scan_thread and self.scan_thread.is_alive():
print("Stopping existing scan thread...")
self.stop_requested = True
self.scan_thread.join(timeout=2.0)
if self.scan_thread.is_alive():
print("WARNING: Could not stop existing thread")
return False
# Reset state
print("Resetting scanner state...")
#print("Running graph.clear()")
#self.graph.clear()
print("running self.current_target = target_domain.lower().strip()")
self.current_target = target_domain.lower().strip()
self.max_depth = max_depth
self.current_depth = 0
self.stop_requested = False
self.total_indicators_found = 0
self.indicators_processed = 0
self.current_indicator = self.current_target
# Start new forensic session
print("Starting new forensic session...")
self.logger = new_session()
# FOR DEBUGGING: Run scan synchronously instead of in thread
print("Running scan synchronously for debugging...")
self._execute_scan_sync(self.current_target, max_depth)
return True
except Exception as e:
print(f"ERROR: Exception in start_scan: {e}")
traceback.print_exc()
return False
def stop_scan(self) -> bool:
"""
Request scan termination.
Returns:
bool: True if stop request was accepted
"""
try:
if self.status == ScanStatus.RUNNING:
self.stop_requested = True
print("Scan stop requested")
return True
print("No active scan to stop")
return False
except Exception as e:
print(f"ERROR: Exception in stop_scan: {e}")
traceback.print_exc()
return False
def get_scan_status(self) -> Dict[str, Any]:
"""
Get current scan status and progress.
Returns:
Dictionary containing scan status information
"""
try:
return {
'status': self.status,
'target_domain': self.current_target,
'current_depth': self.current_depth,
'max_depth': self.max_depth,
'current_indicator': self.current_indicator,
'total_indicators_found': self.total_indicators_found,
'indicators_processed': self.indicators_processed,
'progress_percentage': self._calculate_progress(),
'enabled_providers': [provider.get_name() for provider in self.providers],
'graph_statistics': self.graph.get_statistics()
}
except Exception as e:
print(f"ERROR: Exception in get_scan_status: {e}")
traceback.print_exc()
return {
'status': 'error',
'target_domain': None,
'current_depth': 0,
'max_depth': 0,
'current_indicator': '',
'total_indicators_found': 0,
'indicators_processed': 0,
'progress_percentage': 0.0,
'enabled_providers': [],
'graph_statistics': {}
}
def _calculate_progress(self) -> float:
"""Calculate scan progress percentage."""
if self.total_indicators_found == 0:
return 0.0
return min(100.0, (self.indicators_processed / self.total_indicators_found) * 100)
def _execute_scan_sync(self, target_domain: str, max_depth: int) -> None:
"""
Execute the reconnaissance scan synchronously (for debugging).
Args:
target_domain: Target domain to investigate
max_depth: Maximum recursion depth
"""
print(f"_execute_scan_sync started for {target_domain} with depth {max_depth}")
try:
print("Setting status to RUNNING")
self.status = ScanStatus.RUNNING
# Log scan start
enabled_providers = [provider.get_name() for provider in self.providers]
self.logger.log_scan_start(target_domain, max_depth, enabled_providers)
print(f"Logged scan start with providers: {enabled_providers}")
# Initialize with target domain
print(f"Adding target domain '{target_domain}' as initial node")
self.graph.add_node(target_domain, NodeType.DOMAIN)
# BFS-style exploration with depth limiting
current_level_domains = {target_domain}
processed_domains = set()
print(f"Starting BFS exploration...")
for depth in range(max_depth + 1):
if self.stop_requested:
print(f"Stop requested at depth {depth}")
break
self.current_depth = depth
print(f"Processing depth level {depth} with {len(current_level_domains)} domains")
if not current_level_domains:
print("No domains to process at this level")
break
# Update progress tracking
self.total_indicators_found += len(current_level_domains)
next_level_domains = set()
# Process domains at current depth level
for domain in current_level_domains:
if self.stop_requested:
print(f"Stop requested while processing domain {domain}")
break
if domain in processed_domains:
print(f"Domain {domain} already processed, skipping")
continue
print(f"Processing domain: {domain}")
self.current_indicator = domain
self.indicators_processed += 1
# Query all providers for this domain
discovered_domains = self._query_providers_for_domain(domain)
print(f"Discovered {len(discovered_domains)} new domains from {domain}")
# Add discovered domains to next level if not at max depth
if depth < max_depth:
for discovered_domain in discovered_domains:
if discovered_domain not in processed_domains:
next_level_domains.add(discovered_domain)
print(f"Adding {discovered_domain} to next level")
processed_domains.add(domain)
current_level_domains = next_level_domains
print(f"Completed depth {depth}, {len(next_level_domains)} domains for next level")
# Finalize scan
if self.stop_requested:
self.status = ScanStatus.STOPPED
print("Scan completed with STOPPED status")
else:
self.status = ScanStatus.COMPLETED
print("Scan completed with COMPLETED status")
self.logger.log_scan_complete()
# Print final statistics
stats = self.graph.get_statistics()
print(f"Final scan statistics:")
print(f" - Total nodes: {stats['basic_metrics']['total_nodes']}")
print(f" - Total edges: {stats['basic_metrics']['total_edges']}")
print(f" - Domains processed: {len(processed_domains)}")
except Exception as e:
print(f"ERROR: Scan execution failed with error: {e}")
traceback.print_exc()
self.status = ScanStatus.FAILED
self.logger.logger.error(f"Scan failed: {e}")
def _query_providers_for_domain(self, domain: str) -> Set[str]:
"""
Query all enabled providers for information about a domain.
Args:
domain: Domain to investigate
Returns:
Set of newly discovered domains
"""
print(f"Querying {len(self.providers)} providers for domain: {domain}")
discovered_domains = set()
if not self.providers:
print("No providers available")
return discovered_domains
# Query providers sequentially for debugging
for provider in self.providers:
if self.stop_requested:
print("Stop requested, cancelling provider queries")
break
try:
print(f"Querying provider: {provider.get_name()}")
relationships = provider.query_domain(domain)
print(f"Provider {provider.get_name()} returned {len(relationships)} relationships")
for source, target, rel_type, confidence, raw_data in relationships:
print(f"Processing relationship: {source} -> {target} ({rel_type.relationship_name})")
# Add target node to graph if it doesn't exist
self.graph.add_node(target, NodeType.DOMAIN)
# Add relationship
success = self.graph.add_edge(
source, target, rel_type, confidence,
provider.get_name(), raw_data
)
if success:
print(f"Added new relationship: {source} -> {target}")
else:
print(f"Relationship already exists or failed to add: {source} -> {target}")
discovered_domains.add(target)
except Exception as e:
print(f"Provider {provider.get_name()} failed for {domain}: {e}")
traceback.print_exc()
self.logger.logger.error(f"Provider {provider.get_name()} failed for {domain}: {e}")
print(f"Total unique domains discovered: {len(discovered_domains)}")
return discovered_domains
def get_graph_data(self) -> Dict[str, Any]:
"""
Get current graph data for visualization.
Returns:
Graph data formatted for frontend
"""
return self.graph.get_graph_data()
def export_results(self) -> Dict[str, Any]:
"""
Export complete scan results including graph and audit trail.
Returns:
Dictionary containing complete scan results
"""
# Get graph data
graph_data = self.graph.export_json()
# Get forensic audit trail
audit_trail = self.logger.export_audit_trail()
# Get provider statistics
provider_stats = {}
for provider in self.providers:
provider_stats[provider.get_name()] = provider.get_statistics()
# Combine all results
export_data = {
'scan_metadata': {
'target_domain': self.current_target,
'max_depth': self.max_depth,
'final_status': self.status,
'total_indicators_processed': self.indicators_processed,
'enabled_providers': list(provider_stats.keys())
},
'graph_data': graph_data,
'forensic_audit': audit_trail,
'provider_statistics': provider_stats,
'scan_summary': self.logger.get_forensic_summary()
}
return export_data
def remove_provider(self, provider_name: str) -> bool:
"""
Remove a provider from the scanner.
Args:
provider_name: Name of provider to remove
Returns:
bool: True if provider was removed
"""
for i, provider in enumerate(self.providers):
if provider.get_name() == provider_name:
self.providers.pop(i)
return True
return False
def get_provider_statistics(self) -> Dict[str, Dict[str, Any]]:
"""
Get statistics for all providers.
Returns:
Dictionary mapping provider names to their statistics
"""
stats = {}
for provider in self.providers:
stats[provider.get_name()] = provider.get_statistics()
return stats
class ScannerProxy:
def __init__(self):
self._scanner = None
print("ScannerProxy initialized")
def __getattr__(self, name):
if self._scanner is None:
print("Creating new Scanner instance...")
self._scanner = Scanner()
print("Scanner instance created")
return getattr(self._scanner, name)
# Global scanner instance
scanner = ScannerProxy()

976
dnsrecon.py
View File

@ -1,976 +0,0 @@
#!/usr/bin/env python3
"""
Enhanced DNS Reconnaissance Tool with Recursive Analysis
Copyright (c) 2025 mstoeck3.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
import subprocess
import json
import requests
import argparse
import sys
import time
import os
import re
import ipaddress
from datetime import datetime
from typing import Dict, List, Optional, Any, Set
from urllib.parse import urlparse
import threading
from queue import Queue, Empty
class EnhancedDNSReconTool:
def __init__(self, shodan_api_key: Optional[str] = None, virustotal_api_key: Optional[str] = None):
self.shodan_api_key = shodan_api_key
self.virustotal_api_key = virustotal_api_key
self.output_dir = "dns_recon_results"
self.session = requests.Session()
self.session.headers.update({
'User-Agent': 'EnhancedDNSReconTool/2.0 (Educational/Research Purpose)'
})
# Track processed items to avoid infinite recursion
self.processed_domains: Set[str] = set()
self.processed_ips: Set[str] = set()
# Results storage for recursive analysis
self.all_results: Dict[str, Any] = {}
# Rate limiting
self.last_vt_request = 0
self.last_shodan_request = 0
self.vt_rate_limit = 4 # 4 requests per minute for free tier
self.shodan_rate_limit = 1 # 1 request per second for free tier
def check_dependencies(self) -> bool:
"""Check if required system tools are available."""
required_tools = ['dig', 'whois']
missing_tools = []
for tool in required_tools:
try:
subprocess.run([tool, '--help'],
capture_output=True, check=False, timeout=5)
except (subprocess.TimeoutExpired, FileNotFoundError):
missing_tools.append(tool)
if missing_tools:
print(f"❌ Missing required tools: {', '.join(missing_tools)}")
print("Install with: apt install dnsutils whois (Ubuntu/Debian)")
return False
return True
def run_command(self, cmd: str, timeout: int = 30) -> str:
"""Run shell command with timeout and error handling."""
try:
result = subprocess.run(
cmd, shell=True, capture_output=True,
text=True, timeout=timeout
)
return result.stdout.strip() if result.stdout else result.stderr.strip()
except subprocess.TimeoutExpired:
return "Error: Command timed out"
except Exception as e:
return f"Error: {str(e)}"
def rate_limit_virustotal(self):
"""Implement rate limiting for VirusTotal API."""
current_time = time.time()
time_since_last = current_time - self.last_vt_request
min_interval = 60 / self.vt_rate_limit # seconds between requests
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
print(f" Rate limiting: waiting {sleep_time:.1f}s for VirusTotal...")
time.sleep(sleep_time)
self.last_vt_request = time.time()
def rate_limit_shodan(self):
"""Implement rate limiting for Shodan API."""
current_time = time.time()
time_since_last = current_time - self.last_shodan_request
min_interval = 1 / self.shodan_rate_limit # seconds between requests
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
time.sleep(sleep_time)
self.last_shodan_request = time.time()
def query_virustotal_domain(self, domain: str) -> Dict[str, Any]:
"""Query VirusTotal API for domain information."""
if not self.virustotal_api_key:
return {
'success': False,
'message': 'No VirusTotal API key provided'
}
print(f"🔍 Querying VirusTotal for domain: {domain}")
try:
self.rate_limit_virustotal()
url = f"https://www.virustotal.com/vtapi/v2/domain/report"
params = {
'apikey': self.virustotal_api_key,
'domain': domain
}
response = self.session.get(url, params=params, timeout=30)
if response.status_code == 200:
data = response.json()
# Extract key information
result = {
'success': True,
'domain': domain,
'response_code': data.get('response_code', 0),
'verbose_msg': data.get('verbose_msg', ''),
'detection_ratio': f"{data.get('positives', 0)}/{data.get('total', 0)}"
}
# Add scan results if available
if 'scans' in data:
result['scan_engines'] = len(data['scans'])
result['malicious_engines'] = sum(1 for scan in data['scans'].values() if scan.get('detected', False))
result['scan_summary'] = {}
# Categorize detections
for engine, scan_result in data['scans'].items():
if scan_result.get('detected', False):
category = scan_result.get('result', 'malicious')
if category not in result['scan_summary']:
result['scan_summary'][category] = []
result['scan_summary'][category].append(engine)
# Add additional data if available
for key in ['subdomains', 'detected_urls', 'undetected_urls', 'resolutions']:
if key in data:
result[key] = data[key]
return result
else:
return {
'success': False,
'error': f"HTTP {response.status_code}",
'message': response.text[:200]
}
except Exception as e:
return {
'success': False,
'error': str(e),
'message': 'VirusTotal domain query failed'
}
def query_virustotal_ip(self, ip: str) -> Dict[str, Any]:
"""Query VirusTotal API for IP information."""
if not self.virustotal_api_key:
return {
'success': False,
'message': 'No VirusTotal API key provided'
}
print(f"🔍 Querying VirusTotal for IP: {ip}")
try:
self.rate_limit_virustotal()
url = f"https://www.virustotal.com/vtapi/v2/ip-address/report"
params = {
'apikey': self.virustotal_api_key,
'ip': ip
}
response = self.session.get(url, params=params, timeout=30)
if response.status_code == 200:
data = response.json()
result = {
'success': True,
'ip': ip,
'response_code': data.get('response_code', 0),
'verbose_msg': data.get('verbose_msg', ''),
'detection_ratio': f"{data.get('positives', 0)}/{data.get('total', 0)}"
}
# Add scan results if available
if 'scans' in data:
result['scan_engines'] = len(data['scans'])
result['malicious_engines'] = sum(1 for scan in data['scans'].values() if scan.get('detected', False))
# Add additional data
for key in ['detected_urls', 'undetected_urls', 'resolutions', 'asn', 'country']:
if key in data:
result[key] = data[key]
return result
else:
return {
'success': False,
'error': f"HTTP {response.status_code}",
'message': response.text[:200]
}
except Exception as e:
return {
'success': False,
'error': str(e),
'message': 'VirusTotal IP query failed'
}
def get_dns_records(self, domain: str, record_type: str,
server: Optional[str] = None) -> Dict[str, Any]:
"""Fetch DNS records with comprehensive error handling and proper parsing."""
server_flag = f"@{server}" if server else ""
cmd = f"dig {domain} {record_type} {server_flag} +noall +answer"
output = self.run_command(cmd)
# Parse the output into structured data
records = []
if output and not output.startswith("Error:"):
for line in output.split('\n'):
line = line.strip()
if line and not line.startswith(';') and not line.startswith('>>'):
# Split on any whitespace (handles both tabs and spaces)
parts = line.split()
if len(parts) >= 4:
name = parts[0].rstrip('.')
# Check if second field is numeric (TTL)
if len(parts) >= 5 and parts[1].isdigit():
# Format: name TTL class type data
ttl = parts[1]
dns_class = parts[2]
dns_type = parts[3]
data = ' '.join(parts[4:])
else:
# Format: name class type data (no TTL shown)
ttl = ''
dns_class = parts[1]
dns_type = parts[2]
data = ' '.join(parts[3:]) if len(parts) > 3 else ''
# Validate that we have the expected record type
if dns_type.upper() == record_type.upper():
records.append({
'name': name,
'ttl': ttl,
'class': dns_class,
'type': dns_type,
'data': data
})
return {
'query': f"{domain} {record_type}",
'server': server or 'system',
'raw_output': output,
'records': records,
'record_count': len(records)
}
def get_comprehensive_dns(self, domain: str) -> Dict[str, Any]:
"""Get comprehensive DNS information."""
print(f"🔍 Gathering DNS records for {domain}...")
# Standard record types
record_types = ['A', 'AAAA', 'MX', 'NS', 'SOA', 'TXT', 'CNAME',
'CAA', 'SRV', 'PTR']
# DNS servers to query
dns_servers = [
None, # System default
'1.1.1.1', # Cloudflare
'8.8.8.8', # Google
'9.9.9.9', # Quad9
]
dns_results = {}
for record_type in record_types:
dns_results[record_type] = {}
for server in dns_servers:
server_name = server or 'system'
result = self.get_dns_records(domain, record_type, server)
dns_results[record_type][server_name] = result
time.sleep(0.1) # Rate limiting
# Try DNSSEC validation
dnssec_cmd = f"dig {domain} +dnssec +noall +answer"
dns_results['DNSSEC'] = {
'system': {
'query': f"{domain} +dnssec",
'raw_output': self.run_command(dnssec_cmd),
'records': [],
'record_count': 0
}
}
return dns_results
def perform_reverse_dns(self, ip: str) -> Dict[str, Any]:
"""Perform reverse DNS lookup on IP address."""
print(f"🔄 Reverse DNS lookup for {ip}")
try:
# Validate IP address
ipaddress.ip_address(ip)
# Perform reverse DNS lookup
cmd = f"dig -x {ip} +short"
output = self.run_command(cmd)
hostnames = []
if output and not output.startswith("Error:"):
hostnames = [line.strip().rstrip('.') for line in output.split('\n') if line.strip()]
return {
'success': True,
'ip': ip,
'hostnames': hostnames,
'hostname_count': len(hostnames),
'raw_output': output
}
except Exception as e:
return {
'success': False,
'ip': ip,
'error': str(e),
'hostnames': [],
'hostname_count': 0
}
def extract_subdomains_from_certificates(self, domain: str) -> Set[str]:
"""Extract subdomains from certificate transparency logs."""
print(f"📋 Extracting subdomains from certificates for {domain}")
try:
url = f"https://crt.sh/?q=%.{domain}&output=json"
response = self.session.get(url, timeout=30)
subdomains = set()
if response.status_code == 200:
cert_data = response.json()
for cert in cert_data:
name_value = cert.get('name_value', '')
if name_value:
# Handle multiple domains in one certificate
domains_in_cert = [d.strip() for d in name_value.split('\n')]
for subdomain in domains_in_cert:
# Clean up the subdomain
subdomain = subdomain.lower().strip()
if subdomain and '.' in subdomain:
# Only include subdomains of the target domain
if subdomain.endswith(f".{domain}") or subdomain == domain:
subdomains.add(subdomain)
elif subdomain.startswith("*."):
# Handle wildcard certificates
clean_subdomain = subdomain[2:]
if clean_subdomain.endswith(f".{domain}") or clean_subdomain == domain:
subdomains.add(clean_subdomain)
return subdomains
except Exception as e:
print(f" Error extracting subdomains: {e}")
return set()
def extract_ips_from_dns(self, dns_data: Dict[str, Any]) -> Set[str]:
"""Extract IP addresses from DNS records."""
ips = set()
# Extract from A records
for server_data in dns_data.get('A', {}).values():
for record in server_data.get('records', []):
ip = record.get('data', '')
if ip and self.is_valid_ip(ip):
ips.add(ip)
# Extract from AAAA records
for server_data in dns_data.get('AAAA', {}).values():
for record in server_data.get('records', []):
ipv6 = record.get('data', '')
if ipv6 and self.is_valid_ip(ipv6):
ips.add(ipv6)
return ips
def is_valid_ip(self, ip: str) -> bool:
"""Check if string is a valid IP address."""
try:
ipaddress.ip_address(ip)
return True
except ValueError:
return False
def get_whois_data(self, domain: str) -> Dict[str, Any]:
"""Fetch and parse WHOIS data with improved parsing."""
print(f"📋 Fetching WHOIS data for {domain}...")
raw_whois = self.run_command(f"whois {domain}")
# Basic parsing of common WHOIS fields
whois_data = {
'raw': raw_whois,
'parsed': {}
}
if not raw_whois.startswith("Error:"):
lines = raw_whois.split('\n')
for line in lines:
line = line.strip()
if ':' in line and not line.startswith('%') and not line.startswith('#') and not line.startswith('>>>'):
# Handle different WHOIS formats
if line.count(':') == 1:
key, value = line.split(':', 1)
else:
# Multiple colons - take first as key, rest as value
parts = line.split(':', 2)
key, value = parts[0], ':'.join(parts[1:])
key = key.strip().lower().replace(' ', '_').replace('-', '_')
value = value.strip()
if value and key:
# Handle multiple values for same key (like name servers)
if key in whois_data['parsed']:
# Convert to list if not already
if not isinstance(whois_data['parsed'][key], list):
whois_data['parsed'][key] = [whois_data['parsed'][key]]
whois_data['parsed'][key].append(value)
else:
whois_data['parsed'][key] = value
return whois_data
def get_certificate_transparency(self, domain: str) -> Dict[str, Any]:
"""Query certificate transparency logs via crt.sh."""
print(f"🔐 Querying certificate transparency logs for {domain}...")
try:
# Query crt.sh API
url = f"https://crt.sh/?q=%.{domain}&output=json"
response = self.session.get(url, timeout=30)
if response.status_code == 200:
cert_data = response.json()
# Extract unique subdomains
subdomains = set()
cert_details = []
for cert in cert_data:
# Extract subdomains from name_value
name_value = cert.get('name_value', '')
if name_value:
# Handle multiple domains in one certificate
domains_in_cert = [d.strip() for d in name_value.split('\n')]
subdomains.update(domains_in_cert)
cert_details.append({
'id': cert.get('id'),
'issuer': cert.get('issuer_name'),
'common_name': cert.get('common_name'),
'name_value': cert.get('name_value'),
'not_before': cert.get('not_before'),
'not_after': cert.get('not_after'),
'serial_number': cert.get('serial_number')
})
return {
'success': True,
'total_certificates': len(cert_data),
'unique_subdomains': sorted(list(subdomains)),
'subdomain_count': len(subdomains),
'certificates': cert_details[:50] # Limit for output size
}
else:
return {
'success': False,
'error': f"HTTP {response.status_code}",
'message': 'Failed to fetch certificate data'
}
except Exception as e:
return {
'success': False,
'error': str(e),
'message': 'Request to crt.sh failed'
}
def query_shodan(self, domain: str) -> Dict[str, Any]:
"""Query Shodan API for domain information."""
if not self.shodan_api_key:
return {
'success': False,
'message': 'No Shodan API key provided'
}
print(f"🔎 Querying Shodan for {domain}...")
try:
self.rate_limit_shodan()
# Search for the domain
url = f"https://api.shodan.io/shodan/host/search"
params = {
'key': self.shodan_api_key,
'query': f'hostname:{domain}'
}
response = self.session.get(url, params=params, timeout=30)
if response.status_code == 200:
data = response.json()
return {
'success': True,
'total_results': data.get('total', 0),
'matches': data.get('matches', [])[:10], # Limit results
'facets': data.get('facets', {})
}
else:
return {
'success': False,
'error': f"HTTP {response.status_code}",
'message': response.text[:200]
}
except Exception as e:
return {
'success': False,
'error': str(e),
'message': 'Shodan query failed'
}
def query_shodan_ip(self, ip: str) -> Dict[str, Any]:
"""Query Shodan API for IP information."""
if not self.shodan_api_key:
return {
'success': False,
'message': 'No Shodan API key provided'
}
print(f"🔎 Querying Shodan for IP {ip}...")
try:
self.rate_limit_shodan()
url = f"https://api.shodan.io/shodan/host/{ip}"
params = {'key': self.shodan_api_key}
response = self.session.get(url, params=params, timeout=30)
if response.status_code == 200:
data = response.json()
return {
'success': True,
'ip': ip,
'data': data
}
else:
return {
'success': False,
'error': f"HTTP {response.status_code}",
'message': response.text[:200]
}
except Exception as e:
return {
'success': False,
'error': str(e),
'message': 'Shodan IP query failed'
}
def analyze_domain_recursively(self, domain: str, depth: int = 0, max_depth: int = 2) -> Dict[str, Any]:
"""Perform comprehensive analysis on a domain with recursive subdomain discovery."""
if domain in self.processed_domains or depth > max_depth:
return {}
self.processed_domains.add(domain)
print(f"\n{' ' * depth}🎯 Analyzing domain: {domain} (depth {depth})")
results = {
'domain': domain,
'timestamp': datetime.now().isoformat(),
'depth': depth,
'dns_records': {},
'whois': {},
'certificate_transparency': {},
'virustotal_domain': {},
'shodan': {},
'discovered_ips': {},
'discovered_subdomains': {}
}
# DNS Records
results['dns_records'] = self.get_comprehensive_dns(domain)
# Extract IP addresses from DNS records
discovered_ips = self.extract_ips_from_dns(results['dns_records'])
# WHOIS (only for primary domain to avoid rate limiting)
if depth == 0:
results['whois'] = self.get_whois_data(domain)
# Certificate Transparency
results['certificate_transparency'] = self.get_certificate_transparency(domain)
# VirusTotal Domain Analysis
results['virustotal_domain'] = self.query_virustotal_domain(domain)
# Shodan Domain Analysis
results['shodan'] = self.query_shodan(domain)
# Extract subdomains from certificate transparency
if depth < max_depth:
subdomains = self.extract_subdomains_from_certificates(domain)
# Filter out already processed subdomains
new_subdomains = subdomains - self.processed_domains
new_subdomains.discard(domain) # Remove the current domain itself
print(f"{' ' * depth}📋 Found {len(new_subdomains)} new subdomains to analyze")
# Recursively analyze subdomains (limit to prevent excessive recursion)
for subdomain in list(new_subdomains)[:20]: # Limit to 20 subdomains per domain
if subdomain not in self.processed_domains:
subdomain_results = self.analyze_domain_recursively(subdomain, depth + 1, max_depth)
if subdomain_results:
results['discovered_subdomains'][subdomain] = subdomain_results
# Analyze discovered IP addresses
for ip in discovered_ips:
if ip not in self.processed_ips:
ip_results = self.analyze_ip_recursively(ip, depth)
if ip_results:
results['discovered_ips'][ip] = ip_results
# Store in global results
self.all_results[domain] = results
return results
def analyze_ip_recursively(self, ip: str, depth: int = 0) -> Dict[str, Any]:
"""Perform comprehensive analysis on an IP address."""
if ip in self.processed_ips:
return {}
self.processed_ips.add(ip)
print(f"{' ' * depth}🌐 Analyzing IP: {ip}")
results = {
'ip': ip,
'timestamp': datetime.now().isoformat(),
'reverse_dns': {},
'virustotal_ip': {},
'shodan_ip': {},
'discovered_domains': {}
}
# Reverse DNS lookup
results['reverse_dns'] = self.perform_reverse_dns(ip)
# VirusTotal IP Analysis
results['virustotal_ip'] = self.query_virustotal_ip(ip)
# Shodan IP Analysis
results['shodan_ip'] = self.query_shodan_ip(ip)
# Analyze discovered domains from reverse DNS
reverse_dns = results['reverse_dns']
if reverse_dns.get('success') and reverse_dns.get('hostnames'):
for hostname in reverse_dns['hostnames'][:5]: # Limit to 5 hostnames
if hostname not in self.processed_domains and hostname.count('.') >= 1:
# Only analyze if it's a reasonable hostname and not already processed
domain_results = self.analyze_domain_recursively(hostname, depth + 1, max_depth=1)
if domain_results:
results['discovered_domains'][hostname] = domain_results
return results
def create_comprehensive_summary(self, filename: str) -> None:
"""Create comprehensive summary report with recursive analysis results."""
with open(filename, 'w', encoding='utf-8') as f:
f.write("Enhanced DNS Reconnaissance Report with Recursive Analysis\n")
f.write("=" * 65 + "\n")
f.write(f"Analysis completed at: {datetime.now().isoformat()}\n")
f.write(f"Total domains analyzed: {len(self.processed_domains)}\n")
f.write(f"Total IP addresses analyzed: {len(self.processed_ips)}\n\n")
# Executive Summary
f.write("EXECUTIVE SUMMARY\n")
f.write("-" * 17 + "\n")
total_threats = 0
domains_with_issues = []
ips_with_issues = []
# Count threats across all analyzed domains and IPs
for domain, domain_data in self.all_results.items():
# Check VirusTotal results for domain
vt_domain = domain_data.get('virustotal_domain', {})
if vt_domain.get('success') and vt_domain.get('malicious_engines', 0) > 0:
total_threats += 1
domains_with_issues.append(domain)
# Check discovered IPs
for ip, ip_data in domain_data.get('discovered_ips', {}).items():
vt_ip = ip_data.get('virustotal_ip', {})
if vt_ip.get('success') and vt_ip.get('malicious_engines', 0) > 0:
total_threats += 1
ips_with_issues.append(ip)
f.write(f"Security Status: {'⚠️ THREATS DETECTED' if total_threats > 0 else '✅ NO THREATS DETECTED'}\n")
f.write(f"Total Security Issues: {total_threats}\n")
if domains_with_issues:
f.write(f"Domains with issues: {', '.join(domains_with_issues[:5])}\n")
if ips_with_issues:
f.write(f"IPs with issues: {', '.join(ips_with_issues[:5])}\n")
f.write("\n")
# Process each domain in detail
for domain, domain_data in self.all_results.items():
if domain_data.get('depth', 0) == 0: # Only show primary domains in detail
self._write_domain_analysis(f, domain, domain_data)
# Summary of all discovered assets
f.write("\nASSET DISCOVERY SUMMARY\n")
f.write("-" * 23 + "\n")
f.write(f"All Discovered Domains ({len(self.processed_domains)}):\n")
for domain in sorted(self.processed_domains):
f.write(f" {domain}\n")
f.write(f"\nAll Discovered IP Addresses ({len(self.processed_ips)}):\n")
for ip in sorted(self.processed_ips, key=ipaddress.IPv4Address):
f.write(f" {ip}\n")
f.write(f"\n{'=' * 65}\n")
f.write("Report Generation Complete\n")
def _write_domain_analysis(self, f, domain: str, domain_data: Dict[str, Any]) -> None:
"""Write detailed domain analysis to file."""
f.write(f"\nDETAILED ANALYSIS: {domain.upper()}\n")
f.write("=" * (20 + len(domain)) + "\n")
# DNS Records Summary
dns_data = domain_data.get('dns_records', {})
f.write("DNS Records Summary:\n")
for record_type in ['A', 'AAAA', 'MX', 'NS', 'TXT']:
system_records = dns_data.get(record_type, {}).get('system', {}).get('records', [])
f.write(f" {record_type}: {len(system_records)} records\n")
# Security Analysis
f.write(f"\nSecurity Analysis:\n")
# VirusTotal Domain Results
vt_domain = domain_data.get('virustotal_domain', {})
if vt_domain.get('success'):
detection_ratio = vt_domain.get('detection_ratio', '0/0')
malicious_engines = vt_domain.get('malicious_engines', 0)
f.write(f" VirusTotal Domain: {detection_ratio} ({malicious_engines} flagged as malicious)\n")
if malicious_engines > 0:
f.write(f" ⚠️ SECURITY ALERT: Domain flagged by {malicious_engines} security engines\n")
scan_summary = vt_domain.get('scan_summary', {})
for category, engines in scan_summary.items():
f.write(f" {category}: {', '.join(engines[:3])}\n")
else:
f.write(f" VirusTotal Domain: {vt_domain.get('message', 'Not available')}\n")
# Certificate Information
cert_data = domain_data.get('certificate_transparency', {})
if cert_data.get('success'):
f.write(f" SSL Certificates: {cert_data.get('total_certificates', 0)} found\n")
f.write(f" Subdomains from Certificates: {cert_data.get('subdomain_count', 0)}\n")
# Discovered Assets
discovered_ips = domain_data.get('discovered_ips', {})
discovered_subdomains = domain_data.get('discovered_subdomains', {})
if discovered_ips:
f.write(f"\nDiscovered IP Addresses ({len(discovered_ips)}):\n")
for ip, ip_data in discovered_ips.items():
vt_ip = ip_data.get('virustotal_ip', {})
reverse_dns = ip_data.get('reverse_dns', {})
f.write(f" {ip}:\n")
# Reverse DNS
if reverse_dns.get('success') and reverse_dns.get('hostnames'):
f.write(f" Reverse DNS: {', '.join(reverse_dns['hostnames'][:3])}\n")
# VirusTotal IP results
if vt_ip.get('success'):
detection_ratio = vt_ip.get('detection_ratio', '0/0')
malicious_engines = vt_ip.get('malicious_engines', 0)
f.write(f" VirusTotal: {detection_ratio}")
if malicious_engines > 0:
f.write(f" ⚠️ FLAGGED BY {malicious_engines} ENGINES")
f.write("\n")
# Shodan IP results
shodan_ip = ip_data.get('shodan_ip', {})
if shodan_ip.get('success'):
shodan_data = shodan_ip.get('data', {})
ports = shodan_data.get('ports', [])
if ports:
f.write(f" Shodan Ports: {', '.join(map(str, ports[:10]))}\n")
f.write("\n")
if discovered_subdomains:
f.write(f"Discovered Subdomains ({len(discovered_subdomains)}):\n")
for subdomain, subdomain_data in discovered_subdomains.items():
f.write(f" {subdomain}\n")
# Quick security check for subdomain
vt_subdomain = subdomain_data.get('virustotal_domain', {})
if vt_subdomain.get('success') and vt_subdomain.get('malicious_engines', 0) > 0:
f.write(f" ⚠️ Security Issue: Flagged by VirusTotal\n")
subdomain_ips = subdomain_data.get('discovered_ips', {})
if subdomain_ips:
f.write(f" IPs: {', '.join(list(subdomain_ips.keys())[:3])}\n")
f.write("\n")
def save_results(self, domain: str) -> None:
"""Save results in multiple formats."""
if not os.path.exists(self.output_dir):
os.makedirs(self.output_dir)
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
base_filename = f"{self.output_dir}/{domain}_{timestamp}"
# Save complete JSON (all recursive data)
json_file = f"{base_filename}_complete.json"
with open(json_file, 'w', encoding='utf-8') as f:
json.dump(self.all_results, f, indent=2, ensure_ascii=False, default=str)
# Save comprehensive summary
summary_file = f"{base_filename}_analysis.txt"
self.create_comprehensive_summary(summary_file)
# Save asset list (domains and IPs)
assets_file = f"{base_filename}_assets.txt"
with open(assets_file, 'w', encoding='utf-8') as f:
f.write("Discovered Assets Summary\n")
f.write("=" * 25 + "\n\n")
f.write(f"Domains ({len(self.processed_domains)}):\n")
for domain in sorted(self.processed_domains):
f.write(f"{domain}\n")
f.write(f"\nIP Addresses ({len(self.processed_ips)}):\n")
for ip in sorted(self.processed_ips, key=lambda x: ipaddress.IPv4Address(x)):
f.write(f"{ip}\n")
print(f"\n📄 Results saved:")
print(f" Complete JSON: {json_file}")
print(f" Analysis Report: {summary_file}")
print(f" Asset List: {assets_file}")
def run_enhanced_reconnaissance(self, domain: str, max_depth: int = 2) -> Dict[str, Any]:
"""Run enhanced recursive DNS reconnaissance."""
print(f"\n🚀 Starting enhanced DNS reconnaissance for: {domain}")
print(f" Max recursion depth: {max_depth}")
print(f" APIs enabled: VirusTotal={bool(self.virustotal_api_key)}, Shodan={bool(self.shodan_api_key)}")
start_time = time.time()
# Clear previous results
self.processed_domains.clear()
self.processed_ips.clear()
self.all_results.clear()
# Start recursive analysis
results = self.analyze_domain_recursively(domain, depth=0, max_depth=max_depth)
end_time = time.time()
duration = end_time - start_time
print(f"\n✅ Enhanced reconnaissance completed in {duration:.1f} seconds")
print(f" Domains analyzed: {len(self.processed_domains)}")
print(f" IP addresses analyzed: {len(self.processed_ips)}")
return results
def main():
parser = argparse.ArgumentParser(
description="Enhanced DNS Reconnaissance Tool with Recursive Analysis - Use only on domains you own or have permission to test",
epilog="LEGAL NOTICE: Unauthorized reconnaissance may violate applicable laws. Use responsibly."
)
parser.add_argument('domain', help='Target domain (e.g., example.com)')
parser.add_argument('--shodan-key', help='Shodan API key for additional reconnaissance')
parser.add_argument('--virustotal-key', help='VirusTotal API key for threat intelligence')
parser.add_argument('--max-depth', type=int, default=2,
help='Maximum recursion depth for subdomain analysis (default: 2)')
parser.add_argument('--output-dir', default='dns_recon_results',
help='Output directory for results')
args = parser.parse_args()
# Validate domain format
if not re.match(r'^[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', args.domain):
print("❌ Invalid domain format. Please provide a valid domain (e.g., example.com)")
sys.exit(1)
# Initialize tool
tool = EnhancedDNSReconTool(
shodan_api_key=args.shodan_key,
virustotal_api_key=args.virustotal_key
)
tool.output_dir = args.output_dir
# Check dependencies
if not tool.check_dependencies():
sys.exit(1)
# Warn about API keys
if not args.virustotal_key:
print("⚠️ No VirusTotal API key provided. Threat intelligence will be limited.")
if not args.shodan_key:
print("⚠️ No Shodan API key provided. Host intelligence will be limited.")
try:
# Run enhanced reconnaissance
results = tool.run_enhanced_reconnaissance(args.domain, args.max_depth)
# Save results
tool.save_results(args.domain)
print(f"\n🎯 Enhanced reconnaissance completed for {args.domain}")
except KeyboardInterrupt:
print("\n⏹️ Reconnaissance interrupted by user")
sys.exit(0)
except Exception as e:
print(f"❌ Error during reconnaissance: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

15
providers/__init__.py Normal file
View File

@ -0,0 +1,15 @@
"""
Data provider modules for DNSRecon.
Contains implementations for various reconnaissance data sources.
"""
from .base_provider import BaseProvider, RateLimiter
from .crtsh_provider import CrtShProvider
__all__ = [
'BaseProvider',
'RateLimiter',
'CrtShProvider'
]
__version__ = "1.0.0-phase1"

326
providers/base_provider.py Normal file
View File

@ -0,0 +1,326 @@
"""
Abstract base provider class for DNSRecon data sources.
Defines the interface and common functionality for all providers.
"""
import time
import requests
import threading
from abc import ABC, abstractmethod
from typing import List, Dict, Any, Optional, Tuple
from datetime import datetime
from core.logger import get_forensic_logger
from core.graph_manager import NodeType, RelationshipType
class RateLimiter:
"""Simple rate limiter for API calls."""
def __init__(self, requests_per_minute: int):
"""
Initialize rate limiter.
Args:
requests_per_minute: Maximum requests allowed per minute
"""
self.requests_per_minute = requests_per_minute
self.min_interval = 60.0 / requests_per_minute
self.last_request_time = 0
def wait_if_needed(self) -> None:
"""Wait if necessary to respect rate limits."""
current_time = time.time()
time_since_last = current_time - self.last_request_time
if time_since_last < self.min_interval:
sleep_time = self.min_interval - time_since_last
time.sleep(sleep_time)
self.last_request_time = time.time()
class BaseProvider(ABC):
"""
Abstract base class for all DNSRecon data providers.
Provides common functionality and defines the provider interface.
"""
def __init__(self, name: str, rate_limit: int = 60, timeout: int = 30):
"""
Initialize base provider.
Args:
name: Provider name for logging
rate_limit: Requests per minute limit
timeout: Request timeout in seconds
"""
self.name = name
self.rate_limiter = RateLimiter(rate_limit)
self.timeout = timeout
self._local = threading.local()
self.logger = get_forensic_logger()
# Statistics
self.total_requests = 0
self.successful_requests = 0
self.failed_requests = 0
self.total_relationships_found = 0
@property
def session(self):
if not hasattr(self._local, 'session'):
self._local.session = requests.Session()
self._local.session.headers.update({
'User-Agent': 'DNSRecon/1.0 (Passive Reconnaissance Tool)'
})
return self._local.session
@abstractmethod
def get_name(self) -> str:
"""Return the provider name."""
pass
@abstractmethod
def is_available(self) -> bool:
"""Check if the provider is available and properly configured."""
pass
@abstractmethod
def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
"""
Query the provider for information about a domain.
Args:
domain: Domain to investigate
Returns:
List of tuples: (source_node, target_node, relationship_type, confidence, raw_data)
"""
pass
@abstractmethod
def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
"""
Query the provider for information about an IP address.
Args:
ip: IP address to investigate
Returns:
List of tuples: (source_node, target_node, relationship_type, confidence, raw_data)
"""
pass
def make_request(self, url: str, method: str = "GET",
params: Optional[Dict[str, Any]] = None,
headers: Optional[Dict[str, str]] = None,
target_indicator: str = "") -> Optional[requests.Response]:
"""
Make a rate-limited HTTP request with forensic logging.
Args:
url: Request URL
method: HTTP method
params: Query parameters
headers: Additional headers
target_indicator: The indicator being investigated
Returns:
Response object or None if request failed
"""
# Apply rate limiting
self.rate_limiter.wait_if_needed()
start_time = time.time()
response = None
error = None
try:
self.total_requests += 1
# Prepare request
request_headers = self.session.headers.copy()
if headers:
request_headers.update(headers)
print(f"Making {method} request to: {url}")
# Make request
if method.upper() == "GET":
response = self.session.get(
url,
params=params,
headers=request_headers,
timeout=self.timeout
)
elif method.upper() == "POST":
response = self.session.post(
url,
json=params,
headers=request_headers,
timeout=self.timeout
)
else:
raise ValueError(f"Unsupported HTTP method: {method}")
print(f"Response status: {response.status_code}")
response.raise_for_status()
self.successful_requests += 1
except requests.exceptions.RequestException as e:
error = str(e)
self.failed_requests += 1
print(f"Request failed: {error}")
except Exception as e:
error = f"Unexpected error: {str(e)}"
self.failed_requests += 1
print(f"Unexpected error: {error}")
# Calculate duration and log request
duration_ms = (time.time() - start_time) * 1000
self.logger.log_api_request(
provider=self.name,
url=url,
method=method.upper(),
status_code=response.status_code if response else None,
response_size=len(response.content) if response else None,
duration_ms=duration_ms,
error=error,
target_indicator=target_indicator
)
return response if error is None else None
def log_relationship_discovery(self, source_node: str, target_node: str,
relationship_type: RelationshipType,
confidence_score: float,
raw_data: Dict[str, Any],
discovery_method: str) -> None:
"""
Log discovery of a new relationship.
Args:
source_node: Source node identifier
target_node: Target node identifier
relationship_type: Type of relationship
confidence_score: Confidence score
raw_data: Raw data from provider
discovery_method: Method used for discovery
"""
self.total_relationships_found += 1
self.logger.log_relationship_discovery(
source_node=source_node,
target_node=target_node,
relationship_type=relationship_type.relationship_name,
confidence_score=confidence_score,
provider=self.name,
raw_data=raw_data,
discovery_method=discovery_method
)
def get_statistics(self) -> Dict[str, Any]:
"""
Get provider statistics.
Returns:
Dictionary containing provider performance metrics
"""
return {
'name': self.name,
'total_requests': self.total_requests,
'successful_requests': self.successful_requests,
'failed_requests': self.failed_requests,
'success_rate': (self.successful_requests / self.total_requests * 100) if self.total_requests > 0 else 0,
'relationships_found': self.total_relationships_found,
'rate_limit': self.rate_limiter.requests_per_minute
}
def reset_statistics(self) -> None:
"""Reset provider statistics."""
self.total_requests = 0
self.successful_requests = 0
self.failed_requests = 0
self.total_relationships_found = 0
def _extract_domain_from_url(self, url: str) -> Optional[str]:
"""
Extract domain from URL.
Args:
url: URL string
Returns:
Domain name or None if extraction fails
"""
try:
# Remove protocol
if '://' in url:
url = url.split('://', 1)[1]
# Remove path
if '/' in url:
url = url.split('/', 1)[0]
# Remove port
if ':' in url:
url = url.split(':', 1)[0]
return url.lower()
except Exception:
return None
def _is_valid_domain(self, domain: str) -> bool:
"""
Basic domain validation.
Args:
domain: Domain string to validate
Returns:
True if domain appears valid
"""
if not domain or len(domain) > 253:
return False
# Check for valid characters and structure
parts = domain.split('.')
if len(parts) < 2:
return False
for part in parts:
if not part or len(part) > 63:
return False
if not part.replace('-', '').replace('_', '').isalnum():
return False
return True
def _is_valid_ip(self, ip: str) -> bool:
"""
Basic IP address validation.
Args:
ip: IP address string to validate
Returns:
True if IP appears valid
"""
try:
parts = ip.split('.')
if len(parts) != 4:
return False
for part in parts:
num = int(part)
if not 0 <= num <= 255:
return False
return True
except (ValueError, AttributeError):
return False

272
providers/crtsh_provider.py Normal file
View File

@ -0,0 +1,272 @@
"""
Certificate Transparency provider using crt.sh.
Discovers domain relationships through certificate SAN analysis.
"""
import json
import re
from typing import List, Dict, Any, Tuple, Set
from urllib.parse import quote
from .base_provider import BaseProvider
from core.graph_manager import RelationshipType
class CrtShProvider(BaseProvider):
"""
Provider for querying crt.sh certificate transparency database.
Discovers domain relationships through certificate Subject Alternative Names (SANs).
"""
def __init__(self):
"""Initialize CrtSh provider with appropriate rate limiting."""
super().__init__(
name="crtsh",
rate_limit=60, # Be respectful to the free service
timeout=30
)
self.base_url = "https://crt.sh/"
def get_name(self) -> str:
"""Return the provider name."""
return "crtsh"
def is_available(self) -> bool:
"""
Check if the provider is configured to be used.
This method is intentionally simple and does not perform a network request
to avoid blocking application startup.
"""
return True
def query_domain(self, domain: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
"""
Query crt.sh for certificates containing the domain.
Args:
domain: Domain to investigate
Returns:
List of relationships discovered from certificate analysis
"""
if not self._is_valid_domain(domain):
return []
relationships = []
try:
# Query crt.sh for certificates
url = f"{self.base_url}?q={quote(domain)}&output=json"
response = self.make_request(url, target_indicator=domain)
if not response or response.status_code != 200:
return []
certificates = response.json()
if not certificates:
return []
# Process certificates to extract relationships
seen_certificates = set()
for cert_data in certificates:
cert_id = cert_data.get('id')
if not cert_id or cert_id in seen_certificates:
continue
seen_certificates.add(cert_id)
# Extract domains from certificate
cert_domains = self._extract_domains_from_certificate(cert_data)
if domain in cert_domains and len(cert_domains) > 1:
# Create relationships between domains found in the same certificate
for related_domain in cert_domains:
if related_domain != domain and self._is_valid_domain(related_domain):
# Create SAN relationship
raw_data = {
'certificate_id': cert_id,
'issuer': cert_data.get('issuer_name', ''),
'not_before': cert_data.get('not_before', ''),
'not_after': cert_data.get('not_after', ''),
'serial_number': cert_data.get('serial_number', ''),
'all_domains': list(cert_domains)
}
relationships.append((
domain,
related_domain,
RelationshipType.SAN_CERTIFICATE,
RelationshipType.SAN_CERTIFICATE.default_confidence,
raw_data
))
# Log the discovery
self.log_relationship_discovery(
source_node=domain,
target_node=related_domain,
relationship_type=RelationshipType.SAN_CERTIFICATE,
confidence_score=RelationshipType.SAN_CERTIFICATE.default_confidence,
raw_data=raw_data,
discovery_method="certificate_san_analysis"
)
except json.JSONDecodeError as e:
self.logger.logger.error(f"Failed to parse JSON response from crt.sh: {e}")
except Exception as e:
self.logger.logger.error(f"Error querying crt.sh for {domain}: {e}")
return relationships
def query_ip(self, ip: str) -> List[Tuple[str, str, RelationshipType, float, Dict[str, Any]]]:
"""
Query crt.sh for certificates containing the IP address.
Note: crt.sh doesn't typically index by IP, so this returns empty results.
Args:
ip: IP address to investigate
Returns:
Empty list (crt.sh doesn't support IP-based certificate queries effectively)
"""
# crt.sh doesn't effectively support IP-based certificate queries
# This would require parsing certificate details for IP SANs, which is complex
return []
def _extract_domains_from_certificate(self, cert_data: Dict[str, Any]) -> Set[str]:
"""
Extract all domains from certificate data.
Args:
cert_data: Certificate data from crt.sh API
Returns:
Set of unique domain names found in the certificate
"""
domains = set()
# Extract from common name
common_name = cert_data.get('common_name', '')
if common_name:
cleaned_cn = self._clean_domain_name(common_name)
if cleaned_cn and self._is_valid_domain(cleaned_cn):
domains.add(cleaned_cn)
# Extract from name_value field (contains SANs)
name_value = cert_data.get('name_value', '')
if name_value:
# Split by newlines and clean each domain
for line in name_value.split('\n'):
cleaned_domain = self._clean_domain_name(line.strip())
if cleaned_domain and self._is_valid_domain(cleaned_domain):
domains.add(cleaned_domain)
return domains
def _clean_domain_name(self, domain_name: str) -> str:
"""
Clean and normalize domain name from certificate data.
Args:
domain_name: Raw domain name from certificate
Returns:
Cleaned domain name or empty string if invalid
"""
if not domain_name:
return ""
# Remove common prefixes and clean up
domain = domain_name.strip().lower()
# Remove protocol if present
if domain.startswith(('http://', 'https://')):
domain = domain.split('://', 1)[1]
# Remove path if present
if '/' in domain:
domain = domain.split('/', 1)[0]
# Remove port if present
if ':' in domain and not domain.count(':') > 1: # Avoid breaking IPv6
domain = domain.split(':', 1)[0]
# Handle wildcard domains
if domain.startswith('*.'):
domain = domain[2:]
# Remove any remaining invalid characters
domain = re.sub(r'[^\w\-\.]', '', domain)
# Ensure it's not empty and doesn't start/end with dots or hyphens
if domain and not domain.startswith(('.', '-')) and not domain.endswith(('.', '-')):
return domain
return ""
def get_certificate_details(self, certificate_id: str) -> Dict[str, Any]:
"""
Get detailed information about a specific certificate.
Args:
certificate_id: Certificate ID from crt.sh
Returns:
Dictionary containing certificate details
"""
try:
url = f"{self.base_url}?id={certificate_id}&output=json"
response = self.make_request(url, target_indicator=f"cert_{certificate_id}")
if response and response.status_code == 200:
return response.json()
except Exception as e:
self.logger.logger.error(f"Error fetching certificate details for {certificate_id}: {e}")
return {}
def search_certificates_by_serial(self, serial_number: str) -> List[Dict[str, Any]]:
"""
Search for certificates by serial number.
Args:
serial_number: Certificate serial number
Returns:
List of matching certificates
"""
try:
url = f"{self.base_url}?serial={quote(serial_number)}&output=json"
response = self.make_request(url, target_indicator=f"serial_{serial_number}")
if response and response.status_code == 200:
return response.json()
except Exception as e:
self.logger.logger.error(f"Error searching certificates by serial {serial_number}: {e}")
return []
def get_issuer_certificates(self, issuer_name: str) -> List[Dict[str, Any]]:
"""
Get certificates issued by a specific CA.
Args:
issuer_name: Certificate Authority name
Returns:
List of certificates from the specified issuer
"""
try:
url = f"{self.base_url}?issuer={quote(issuer_name)}&output=json"
response = self.make_request(url, target_indicator=f"issuer_{issuer_name}")
if response and response.status_code == 200:
return response.json()
except Exception as e:
self.logger.logger.error(f"Error fetching certificates for issuer {issuer_name}: {e}")
return []

0
providers/dns_provider.py Normal file
View File

8
requirements.txt
View File

@ -1,4 +1,6 @@
Flask>=2.3.3
networkx>=3.1
requests>=2.31.0
flask>=2.3.0
dnspython>=2.4.0
click>=8.1.0
python-dateutil>=2.8.2
Werkzeug>=2.3.7
urllib3>=2.0.0

20
src/__init__.py
View File

@ -1,20 +0,0 @@
# File: src/__init__.py
"""DNS Reconnaissance Tool Package."""
__version__ = "1.0.0"
__author__ = "DNS Recon Tool"
__email__ = ""
__description__ = "A comprehensive DNS reconnaissance tool for investigators"
from .main import main
from .config import Config
from .reconnaissance import ReconnaissanceEngine
from .data_structures import ReconData
__all__ = [
'main',
'Config',
'ReconnaissanceEngine',
'ReconData'
]

356
src/certificate_checker.py
View File

@ -1,356 +0,0 @@
# File: src/certificate_checker.py
"""Certificate transparency log checker using crt.sh with forensic operation tracking."""
import requests
import json
import time
import logging
import socket
import uuid
from datetime import datetime
from typing import List, Optional, Set
from .data_structures import Certificate
from .config import Config
# Module logger
logger = logging.getLogger(__name__)
class CertificateChecker:
"""Check certificates using crt.sh with simple query caching and forensic tracking."""
CRT_SH_URL = "https://crt.sh/"
def __init__(self, config: Config):
self.config = config
self.last_request = 0
self.query_count = 0
self.connection_failures = 0
self.max_connection_failures = 3
# Simple HTTP request cache to avoid duplicate queries
self._http_cache = {} # query_string -> List[Certificate]
logger.info("🔐 Certificate checker initialized with HTTP request caching")
self._test_connectivity()
def _test_connectivity(self):
"""Test if we can reach crt.sh."""
try:
logger.info("Testing connectivity to crt.sh...")
try:
socket.gethostbyname('crt.sh')
logger.debug("DNS resolution for crt.sh successful")
except socket.gaierror as e:
logger.warning(f"DNS resolution failed for crt.sh: {e}")
return False
response = requests.get(
self.CRT_SH_URL,
params={'q': 'example.com', 'output': 'json'},
timeout=10,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
if response.status_code in [200, 404]:
logger.info("crt.sh connectivity test successful")
return True
else:
logger.warning(f"crt.sh returned status {response.status_code}")
return False
except requests.exceptions.ConnectionError as e:
logger.warning(f"Cannot reach crt.sh: {e}")
return False
except requests.exceptions.Timeout:
logger.warning("crt.sh connectivity test timed out")
return False
except Exception as e:
logger.warning(f"Unexpected error testing crt.sh connectivity: {e}")
return False
def _rate_limit(self):
"""Apply rate limiting for crt.sh."""
now = time.time()
time_since_last = now - self.last_request
min_interval = 1.0 / self.config.CRT_SH_RATE_LIMIT
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
logger.debug(f"crt.sh rate limiting: sleeping for {sleep_time:.2f}s")
time.sleep(sleep_time)
self.last_request = time.time()
self.query_count += 1
def get_certificates(self, domain: str, operation_id: Optional[str] = None) -> List[Certificate]:
"""Get certificates for a domain with forensic tracking."""
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🔐 Getting certificates for domain: {domain} (operation: {operation_id})")
if self.connection_failures >= self.max_connection_failures:
logger.warning(f"Skipping certificate lookup for {domain} due to repeated connection failures")
return []
certificates = []
# Query for the domain itself
domain_certs = self._query_crt_sh(domain, operation_id)
certificates.extend(domain_certs)
# Query for wildcard certificates
wildcard_certs = self._query_crt_sh(f"%.{domain}", operation_id)
certificates.extend(wildcard_certs)
# Remove duplicates based on certificate ID
unique_certs = {cert.id: cert for cert in certificates}
final_certs = list(unique_certs.values())
if final_certs:
logger.info(f"✅ Found {len(final_certs)} unique certificates for {domain}")
else:
logger.debug(f"ℹ️ No certificates found for {domain}")
return final_certs
def _query_crt_sh(self, query: str, operation_id: str) -> List[Certificate]:
"""Query crt.sh API with HTTP caching and forensic tracking."""
# Check HTTP cache first
cache_key = f"{query}:{operation_id}" # Include operation_id in cache key
if query in self._http_cache:
logger.debug(f"Using cached HTTP result for crt.sh query: {query}")
# Need to create new Certificate objects with current operation_id
cached_certs = self._http_cache[query]
return [
Certificate(
id=cert.id,
issuer=cert.issuer,
subject=cert.subject,
not_before=cert.not_before,
not_after=cert.not_after,
is_wildcard=cert.is_wildcard,
operation_id=operation_id # Use current operation_id
) for cert in cached_certs
]
# Not cached, make the HTTP request
certificates = self._make_http_request(query, operation_id)
# Cache the HTTP result (without operation_id)
if certificates:
self._http_cache[query] = [
Certificate(
id=cert.id,
issuer=cert.issuer,
subject=cert.subject,
not_before=cert.not_before,
not_after=cert.not_after,
is_wildcard=cert.is_wildcard,
operation_id="" # Cache without operation_id
) for cert in certificates
]
return certificates
def _make_http_request(self, query: str, operation_id: str) -> List[Certificate]:
"""Make actual HTTP request to crt.sh API with retry logic and forensic tracking."""
certificates = []
self._rate_limit()
logger.debug(f"🌐 Making HTTP request to crt.sh for: {query} (operation: {operation_id})")
max_retries = 2
backoff_delays = [1, 3]
for attempt in range(max_retries):
try:
params = {
'q': query,
'output': 'json'
}
response = requests.get(
self.CRT_SH_URL,
params=params,
timeout=self.config.HTTP_TIMEOUT,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
logger.debug(f"📡 crt.sh API response for {query}: {response.status_code}")
if response.status_code == 200:
try:
data = response.json()
logger.debug(f"crt.sh returned {len(data)} certificate entries for {query}")
for cert_data in data:
try:
not_before = self._parse_date(cert_data.get('not_before'))
not_after = self._parse_date(cert_data.get('not_after'))
if not_before and not_after:
# Create Certificate with forensic metadata
certificate = Certificate(
id=cert_data.get('id'),
issuer=cert_data.get('issuer_name', ''),
subject=cert_data.get('name_value', ''),
not_before=not_before,
not_after=not_after,
is_wildcard='*.' in cert_data.get('name_value', ''),
operation_id=operation_id # Forensic tracking
)
certificates.append(certificate)
logger.debug(f"📋 Parsed certificate ID {certificate.id} for {query}")
else:
logger.debug(f"Skipped certificate with invalid dates: {cert_data.get('id')}")
except (ValueError, TypeError, KeyError) as e:
logger.debug(f"Error parsing certificate data: {e}")
continue
self.connection_failures = 0
logger.info(f"✅ Successfully processed {len(certificates)} certificates from crt.sh for {query}")
return certificates
except json.JSONDecodeError as e:
logger.warning(f"Invalid JSON response from crt.sh for {query}: {e}")
if attempt < max_retries - 1:
time.sleep(backoff_delays[attempt])
continue
return certificates
elif response.status_code == 404:
logger.debug(f"ℹ️ No certificates found for {query} (404)")
self.connection_failures = 0
return certificates
elif response.status_code == 429:
logger.warning(f"⚠️ crt.sh rate limit exceeded for {query}")
if attempt < max_retries - 1:
time.sleep(5)
continue
return certificates
else:
logger.warning(f"⚠️ crt.sh HTTP error for {query}: {response.status_code}")
if attempt < max_retries - 1:
time.sleep(backoff_delays[attempt])
continue
return certificates
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:
error_type = "Connection Error" if isinstance(e, requests.exceptions.ConnectionError) else "Timeout"
logger.warning(f"⚠️ crt.sh {error_type} for {query} (attempt {attempt+1}/{max_retries}): {e}")
if isinstance(e, requests.exceptions.ConnectionError):
self.connection_failures += 1
if attempt < max_retries - 1:
time.sleep(backoff_delays[attempt])
continue
except requests.exceptions.RequestException as e:
logger.warning(f"🌐 crt.sh network error for {query} (attempt {attempt+1}/{max_retries}): {e}")
if attempt < max_retries - 1:
time.sleep(backoff_delays[attempt])
continue
except Exception as e:
logger.error(f"❌ Unexpected error querying crt.sh for {query}: {e}")
if attempt < max_retries - 1:
time.sleep(backoff_delays[attempt])
continue
logger.warning(f"⚠️ All {max_retries} attempts failed for crt.sh query: {query}")
return certificates
def _parse_date(self, date_str: str) -> Optional[datetime]:
"""Parse date string with multiple format support."""
if not date_str:
return None
date_formats = [
'%Y-%m-%dT%H:%M:%S',
'%Y-%m-%dT%H:%M:%SZ',
'%Y-%m-%d %H:%M:%S',
'%Y-%m-%dT%H:%M:%S.%f',
'%Y-%m-%dT%H:%M:%S.%fZ',
]
for fmt in date_formats:
try:
return datetime.strptime(date_str, fmt)
except ValueError:
continue
try:
return datetime.fromisoformat(date_str.replace('Z', '+00:00'))
except ValueError:
pass
logger.debug(f"Could not parse date: {date_str}")
return None
def extract_subdomains_from_certificates(self, certificates: List[Certificate]) -> Set[str]:
"""Extract subdomains from certificate subjects."""
subdomains = set()
logger.debug(f"🌿 Extracting subdomains from {len(certificates)} certificates")
for cert in certificates:
# Parse subject field for domain names
subject_lines = cert.subject.split('\n')
for line in subject_lines:
line = line.strip()
# Skip wildcard domains for recursion (they don't resolve directly)
if line.startswith('*.'):
logger.debug(f"⭐ Skipping wildcard domain: {line}")
continue
if self._is_valid_domain(line):
subdomains.add(line.lower())
logger.debug(f"🌿 Found subdomain from certificate: {line}")
if subdomains:
logger.info(f"🌿 Extracted {len(subdomains)} subdomains from certificates")
else:
logger.debug("ℹ️ No subdomains extracted from certificates")
return subdomains
def _is_valid_domain(self, domain: str) -> bool:
"""Basic domain validation."""
if not domain or '.' not in domain:
return False
domain = domain.lower().strip()
if domain.startswith('www.'):
domain = domain[4:]
if len(domain) < 3 or len(domain) > 255:
return False
# Must not be an IP address
try:
socket.inet_aton(domain)
return False
except socket.error:
pass
# Check for reasonable domain structure
parts = domain.split('.')
if len(parts) < 2:
return False
for part in parts:
if len(part) < 1 or len(part) > 63:
return False
if not part.replace('-', '').replace('_', '').isalnum():
return False
return True

80
src/config.py
View File

@ -1,80 +0,0 @@
# File: src/config.py
"""Configuration settings for the reconnaissance tool."""
import os
import logging
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class Config:
"""Configuration class for the reconnaissance tool."""
# DNS servers to query
DNS_SERVERS: List[str] = None
# API keys
shodan_key: Optional[str] = None
virustotal_key: Optional[str] = None
# Rate limiting (requests per second)
# DNS servers are generally quite robust, increased from 10 to 50/s
DNS_RATE_LIMIT: float = 50.0
CRT_SH_RATE_LIMIT: float = 2.0
SHODAN_RATE_LIMIT: float = 0.5 # Shodan is more restrictive
VIRUSTOTAL_RATE_LIMIT: float = 0.25 # VirusTotal is very restrictive
# Recursive depth
max_depth: int = 2
# Timeouts
DNS_TIMEOUT: int = 5
HTTP_TIMEOUT: int = 20
# Logging level
log_level: str = "INFO"
def __post_init__(self):
if self.DNS_SERVERS is None:
# Use multiple reliable DNS servers
self.DNS_SERVERS = [
'1.1.1.1', # Cloudflare
'8.8.8.8', # Google
'9.9.9.9' # Quad9
]
@classmethod
def from_args(cls, shodan_key: Optional[str] = None,
virustotal_key: Optional[str] = None,
max_depth: int = 2,
log_level: str = "INFO") -> 'Config':
"""Create config from command line arguments."""
return cls(
shodan_key=shodan_key,
virustotal_key=virustotal_key,
max_depth=max_depth,
log_level=log_level.upper()
)
def setup_logging(self, cli_mode: bool = True):
"""Set up logging configuration."""
log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
if cli_mode:
# For CLI, use a more readable format
log_format = '%(asctime)s [%(levelname)s] %(message)s'
logging.basicConfig(
level=getattr(logging, self.log_level, logging.INFO),
format=log_format,
datefmt='%H:%M:%S'
)
# Set specific loggers
logging.getLogger('urllib3').setLevel(logging.WARNING) # Reduce HTTP noise
logging.getLogger('requests').setLevel(logging.WARNING) # Reduce HTTP noise
if self.log_level == "DEBUG":
logging.getLogger(__name__.split('.')[0]).setLevel(logging.DEBUG)
return logging.getLogger(__name__)

475
src/data_structures.py
View File

@ -1,475 +0,0 @@
# File: src/data_structures.py
"""Enhanced data structures for forensic DNS reconnaissance with full provenance tracking."""
from dataclasses import dataclass, field
from typing import Dict, List, Set, Optional, Any, Tuple
from datetime import datetime
from enum import Enum
import json
import logging
import uuid
# Set up logging for this module
logger = logging.getLogger(__name__)
class OperationType(Enum):
"""Types of discovery operations."""
INITIAL_TARGET = "initial_target"
TLD_EXPANSION = "tld_expansion"
DNS_A = "dns_a"
DNS_AAAA = "dns_aaaa"
DNS_MX = "dns_mx"
DNS_NS = "dns_ns"
DNS_TXT = "dns_txt"
DNS_CNAME = "dns_cname"
DNS_SOA = "dns_soa"
DNS_PTR = "dns_ptr"
DNS_SRV = "dns_srv"
DNS_CAA = "dns_caa"
DNS_DNSKEY = "dns_dnskey"
DNS_DS = "dns_ds"
DNS_RRSIG = "dns_rrsig"
DNS_NSEC = "dns_nsec"
DNS_NSEC3 = "dns_nsec3"
DNS_REVERSE = "dns_reverse"
CERTIFICATE_CHECK = "certificate_check"
SHODAN_LOOKUP = "shodan_lookup"
VIRUSTOTAL_IP = "virustotal_ip"
VIRUSTOTAL_DOMAIN = "virustotal_domain"
class DiscoveryMethod(Enum):
"""How a hostname was discovered."""
INITIAL_TARGET = "initial_target"
TLD_EXPANSION = "tld_expansion"
DNS_RECORD_VALUE = "dns_record_value"
CERTIFICATE_SUBJECT = "certificate_subject"
DNS_SUBDOMAIN_EXTRACTION = "dns_subdomain_extraction"
@dataclass
class DNSRecord:
"""DNS record information with enhanced metadata."""
record_type: str
value: str
ttl: Optional[int] = None
discovered_at: datetime = field(default_factory=datetime.now)
operation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
def to_dict(self) -> dict:
return {
'record_type': self.record_type,
'value': self.value,
'ttl': self.ttl,
'discovered_at': self.discovered_at.isoformat() if self.discovered_at else None,
'operation_id': self.operation_id
}
@dataclass
class Certificate:
"""Certificate information with enhanced metadata."""
id: int
issuer: str
subject: str
not_before: datetime
not_after: datetime
is_wildcard: bool = False
discovered_at: datetime = field(default_factory=datetime.now)
operation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
is_valid_now: bool = field(default=True) # Based on current timestamp vs cert validity
def __post_init__(self):
"""Calculate if certificate is currently valid."""
now = datetime.now()
self.is_valid_now = self.not_before <= now <= self.not_after
def to_dict(self) -> dict:
return {
'id': self.id,
'issuer': self.issuer,
'subject': self.subject,
'not_before': self.not_before.isoformat() if self.not_before else None,
'not_after': self.not_after.isoformat() if self.not_after else None,
'is_wildcard': self.is_wildcard,
'discovered_at': self.discovered_at.isoformat() if self.discovered_at else None,
'operation_id': self.operation_id,
'is_valid_now': self.is_valid_now
}
@dataclass
class ShodanResult:
"""Shodan scan result with metadata."""
ip: str
ports: List[int]
services: Dict[str, Any]
organization: Optional[str] = None
country: Optional[str] = None
discovered_at: datetime = field(default_factory=datetime.now)
operation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
def to_dict(self) -> dict:
return {
'ip': self.ip,
'ports': self.ports,
'services': self.services,
'organization': self.organization,
'country': self.country,
'discovered_at': self.discovered_at.isoformat() if self.discovered_at else None,
'operation_id': self.operation_id
}
@dataclass
class VirusTotalResult:
"""VirusTotal scan result with metadata."""
resource: str # IP or domain
positives: int
total: int
scan_date: datetime
permalink: str
discovered_at: datetime = field(default_factory=datetime.now)
operation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
def to_dict(self) -> dict:
return {
'resource': self.resource,
'positives': self.positives,
'total': self.total,
'scan_date': self.scan_date.isoformat() if self.scan_date else None,
'permalink': self.permalink,
'discovered_at': self.discovered_at.isoformat() if self.discovered_at else None,
'operation_id': self.operation_id
}
@dataclass
class DiscoveryOperation:
"""A single discovery operation with complete metadata."""
operation_id: str = field(default_factory=lambda: str(uuid.uuid4()))
operation_type: OperationType = OperationType.INITIAL_TARGET
target: str = ""
timestamp: datetime = field(default_factory=datetime.now)
success: bool = True
error_message: Optional[str] = None
# Results of the operation
dns_records: List[DNSRecord] = field(default_factory=list)
certificates: List[Certificate] = field(default_factory=list)
shodan_results: List[ShodanResult] = field(default_factory=list)
virustotal_results: List[VirusTotalResult] = field(default_factory=list)
discovered_hostnames: List[str] = field(default_factory=list) # New hostnames found
discovered_ips: List[str] = field(default_factory=list) # New IPs found
# Operation-specific metadata
metadata: Dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict:
return {
'operation_id': self.operation_id,
'operation_type': self.operation_type.value,
'target': self.target,
'timestamp': self.timestamp.isoformat() if self.timestamp else None,
'success': self.success,
'error_message': self.error_message,
'dns_records': [record.to_dict() for record in self.dns_records],
'certificates': [cert.to_dict() for cert in self.certificates],
'shodan_results': [result.to_dict() for result in self.shodan_results],
'virustotal_results': [result.to_dict() for result in self.virustotal_results],
'discovered_hostnames': self.discovered_hostnames,
'discovered_ips': self.discovered_ips,
'metadata': self.metadata
}
@dataclass
class DiscoveryEdge:
"""Represents a discovery relationship between two hostnames."""
source_hostname: str # The hostname that led to the discovery
target_hostname: str # The hostname that was discovered
discovery_method: DiscoveryMethod
operation_id: str
timestamp: datetime
metadata: Dict[str, Any] = field(default_factory=dict) # Additional context
def to_dict(self) -> dict:
return {
'source_hostname': self.source_hostname,
'target_hostname': self.target_hostname,
'discovery_method': self.discovery_method.value,
'operation_id': self.operation_id,
'timestamp': self.timestamp.isoformat() if self.timestamp else None,
'metadata': self.metadata
}
@dataclass
class DiscoveryNode:
"""A discovered hostname with complete provenance and current state."""
hostname: str
depth: int = 0
first_seen: datetime = field(default_factory=datetime.now)
last_updated: datetime = field(default_factory=datetime.now)
# Current validity state
dns_exists: Optional[bool] = None # Does this hostname resolve?
last_dns_check: Optional[datetime] = None
resolved_ips: Set[str] = field(default_factory=set)
# Discovery provenance - can have multiple discovery paths
discovery_methods: Set[DiscoveryMethod] = field(default_factory=set)
discovered_by_operations: Set[str] = field(default_factory=set) # operation_ids
# Associated data (current state)
dns_records_by_type: Dict[str, List[DNSRecord]] = field(default_factory=dict)
certificates: List[Certificate] = field(default_factory=list)
shodan_results: List[ShodanResult] = field(default_factory=list)
virustotal_results: List[VirusTotalResult] = field(default_factory=list)
reverse_dns: Optional[str] = None
def add_dns_record(self, record: DNSRecord) -> None:
"""Add a DNS record to this node."""
record_type = record.record_type
if record_type not in self.dns_records_by_type:
self.dns_records_by_type[record_type] = []
self.dns_records_by_type[record_type].append(record)
self.last_updated = datetime.now()
# Update resolved IPs if this is an A or AAAA record
if record_type in ['A', 'AAAA']:
self.resolved_ips.add(record.value)
self.dns_exists = True
self.last_dns_check = record.discovered_at
def add_certificate(self, certificate: Certificate) -> None:
"""Add a certificate to this node."""
self.certificates.append(certificate)
self.last_updated = datetime.now()
def get_all_dns_records(self) -> List[DNSRecord]:
"""Get all DNS records for this node."""
all_records = []
for records in self.dns_records_by_type.values():
all_records.extend(records)
return all_records
def get_current_certificates(self) -> List[Certificate]:
"""Get currently valid certificates."""
return [cert for cert in self.certificates if cert.is_valid_now]
def get_expired_certificates(self) -> List[Certificate]:
"""Get expired certificates."""
return [cert for cert in self.certificates if not cert.is_valid_now]
def to_dict(self) -> dict:
return {
'hostname': self.hostname,
'depth': self.depth,
'first_seen': self.first_seen.isoformat() if self.first_seen else None,
'last_updated': self.last_updated.isoformat() if self.last_updated else None,
'dns_exists': self.dns_exists,
'last_dns_check': self.last_dns_check.isoformat() if self.last_dns_check else None,
'resolved_ips': sorted(list(self.resolved_ips)),
'discovery_methods': [method.value for method in self.discovery_methods],
'discovered_by_operations': sorted(list(self.discovered_by_operations)),
'dns_records_by_type': {
record_type: [record.to_dict() for record in records]
for record_type, records in self.dns_records_by_type.items()
},
'certificates': [cert.to_dict() for cert in self.certificates],
'current_certificates': [cert.to_dict() for cert in self.get_current_certificates()],
'expired_certificates': [cert.to_dict() for cert in self.get_expired_certificates()],
'shodan_results': [result.to_dict() for result in self.shodan_results],
'virustotal_results': [result.to_dict() for result in self.virustotal_results],
'reverse_dns': self.reverse_dns
}
@dataclass
class ForensicReconData:
"""Enhanced reconnaissance data with full forensic tracking and graph structure."""
# Core graph structure
nodes: Dict[str, DiscoveryNode] = field(default_factory=dict) # hostname -> node
edges: List[DiscoveryEdge] = field(default_factory=list)
operations: Dict[str, DiscoveryOperation] = field(default_factory=dict) # operation_id -> operation
# Quick lookup indexes
ip_addresses: Set[str] = field(default_factory=set)
operation_timeline: List[str] = field(default_factory=list) # ordered operation_ids
# Metadata
start_time: datetime = field(default_factory=datetime.now)
end_time: Optional[datetime] = None
scan_config: Dict[str, Any] = field(default_factory=dict)
def add_node(self, hostname: str, depth: int = 0,
discovery_method: DiscoveryMethod = DiscoveryMethod.INITIAL_TARGET,
operation_id: Optional[str] = None) -> DiscoveryNode:
"""Add or get a discovery node."""
hostname = hostname.lower()
if hostname not in self.nodes:
node = DiscoveryNode(hostname=hostname, depth=depth)
self.nodes[hostname] = node
logger.debug(f"Created new node: {hostname} at depth {depth}")
else:
node = self.nodes[hostname]
# Update depth if this is a shallower discovery
if depth < node.depth:
node.depth = depth
logger.debug(f"Updated node {hostname} depth: {node.depth} -> {depth}")
# Track discovery method and operation
node.discovery_methods.add(discovery_method)
if operation_id:
node.discovered_by_operations.add(operation_id)
return node
def add_edge(self, source: str, target: str, discovery_method: DiscoveryMethod,
operation_id: str, metadata: Dict[str, Any] = None) -> None:
"""Add a discovery edge."""
edge = DiscoveryEdge(
source_hostname=source.lower(),
target_hostname=target.lower(),
discovery_method=discovery_method,
operation_id=operation_id,
timestamp=datetime.now(),
metadata=metadata or {}
)
self.edges.append(edge)
logger.debug(f"Added edge: {source} -> {target} via {discovery_method.value}")
def add_operation(self, operation: DiscoveryOperation) -> None:
"""Add an operation to the timeline."""
self.operations[operation.operation_id] = operation
self.operation_timeline.append(operation.operation_id)
logger.debug(f"Added operation: {operation.operation_type.value} on {operation.target}")
def get_node(self, hostname: str) -> Optional[DiscoveryNode]:
"""Get a node by hostname."""
return self.nodes.get(hostname.lower())
def get_children(self, hostname: str) -> List[str]:
"""Get all hostnames discovered from this hostname."""
hostname = hostname.lower()
children = []
for edge in self.edges:
if edge.source_hostname == hostname:
children.append(edge.target_hostname)
return children
def get_parents(self, hostname: str) -> List[str]:
"""Get all hostnames that led to discovering this hostname."""
hostname = hostname.lower()
parents = []
for edge in self.edges:
if edge.target_hostname == hostname:
parents.append(edge.source_hostname)
return parents
def get_discovery_path(self, hostname: str) -> List[Tuple[str, str, DiscoveryMethod]]:
"""Get the discovery path(s) to a hostname."""
hostname = hostname.lower()
paths = []
def trace_path(current: str, visited: Set[str], current_path: List):
if current in visited:
return # Avoid cycles
visited.add(current)
parents = self.get_parents(current)
if not parents:
# This is a root node, we have a complete path
paths.append(current_path.copy())
else:
for parent in parents:
# Find the edge that connects parent to current
for edge in self.edges:
if edge.source_hostname == parent and edge.target_hostname == current:
new_path = [(parent, current, edge.discovery_method)] + current_path
trace_path(parent, visited.copy(), new_path)
trace_path(hostname, set(), [])
return paths
def get_stats(self) -> Dict[str, int]:
"""Get current statistics."""
total_dns_records = sum(len(node.get_all_dns_records()) for node in self.nodes.values())
total_certificates = sum(len(node.certificates) for node in self.nodes.values())
current_certificates = sum(len(node.get_current_certificates()) for node in self.nodes.values())
expired_certificates = sum(len(node.get_expired_certificates()) for node in self.nodes.values())
total_shodan = sum(len(node.shodan_results) for node in self.nodes.values())
total_virustotal = sum(len(node.virustotal_results) for node in self.nodes.values())
return {
'hostnames': len(self.nodes),
'ip_addresses': len(self.ip_addresses),
'discovery_edges': len(self.edges),
'operations_performed': len(self.operations),
'dns_records': total_dns_records,
'certificates_total': total_certificates,
'certificates_current': current_certificates,
'certificates_expired': expired_certificates,
'shodan_results': total_shodan,
'virustotal_results': total_virustotal
}
def to_dict(self) -> dict:
"""Export data as a serializable dictionary."""
logger.info(f"Serializing ForensicReconData with {len(self.nodes)} nodes, {len(self.edges)} edges, {len(self.operations)} operations")
return {
'nodes': {hostname: node.to_dict() for hostname, node in self.nodes.items()},
'edges': [edge.to_dict() for edge in self.edges],
'operations': {op_id: op.to_dict() for op_id, op in self.operations.items()},
'operation_timeline': self.operation_timeline,
'ip_addresses': sorted(list(self.ip_addresses)),
'metadata': {
'start_time': self.start_time.isoformat() if self.start_time else None,
'end_time': self.end_time.isoformat() if self.end_time else None,
'scan_config': self.scan_config,
'stats': self.get_stats()
},
'graph_analysis': self._generate_graph_analysis()
}
def _generate_graph_analysis(self) -> Dict[str, Any]:
"""Generate graph analysis metadata."""
# Depth distribution
depth_distribution = {}
for node in self.nodes.values():
depth = node.depth
depth_distribution[depth] = depth_distribution.get(depth, 0) + 1
# Root nodes (no parents)
root_nodes = [hostname for hostname in self.nodes.keys()
if not self.get_parents(hostname)]
# Leaf nodes (no children)
leaf_nodes = [hostname for hostname in self.nodes.keys()
if not self.get_children(hostname)]
# Discovery method distribution
method_distribution = {}
for edge in self.edges:
method = edge.discovery_method.value
method_distribution[method] = method_distribution.get(method, 0) + 1
return {
'depth_distribution': depth_distribution,
'max_depth': max(depth_distribution.keys()) if depth_distribution else 0,
'root_nodes': root_nodes,
'leaf_nodes': leaf_nodes,
'discovery_method_distribution': method_distribution,
'total_discovery_paths': len(self.edges)
}
def to_json(self) -> str:
"""Export data as JSON."""
try:
return json.dumps(self.to_dict(), indent=2, ensure_ascii=False)
except Exception as e:
logger.error(f"Failed to serialize to JSON: {e}")
return json.dumps({
'error': str(e),
'stats': self.get_stats(),
'timestamp': datetime.now().isoformat()
}, indent=2)
# Backward compatibility aliases (for gradual migration)
ReconData = ForensicReconData

368
src/dns_resolver.py
View File

@ -1,368 +0,0 @@
# File: src/dns_resolver.py
"""DNS resolution functionality with enhanced TLD testing and forensic operation tracking."""
import dns.resolver
import dns.reversename
import dns.query
import dns.zone
from typing import List, Dict, Optional, Set
import socket
import time
import logging
import uuid
from .data_structures import DNSRecord, ReconData
from .config import Config
# Module logger
logger = logging.getLogger(__name__)
class DNSResolver:
"""DNS resolution and record lookup with optimized TLD testing and forensic tracking."""
# All DNS record types to query
RECORD_TYPES = [
'A', 'AAAA', 'MX', 'NS', 'TXT', 'CNAME', 'SOA', 'PTR',
'SRV', 'CAA', 'DNSKEY', 'DS', 'RRSIG', 'NSEC', 'NSEC3'
]
def __init__(self, config: Config):
self.config = config
self.last_request = 0
self.query_count = 0
logger.info(f"🌐 DNS resolver initialized with {len(config.DNS_SERVERS)} servers: {config.DNS_SERVERS}")
logger.info(f"⚡ DNS rate limit: {config.DNS_RATE_LIMIT}/s, timeout: {config.DNS_TIMEOUT}s")
def _rate_limit(self):
"""Apply rate limiting - more graceful for DNS servers."""
now = time.time()
time_since_last = now - self.last_request
min_interval = 1.0 / self.config.DNS_RATE_LIMIT
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
# Only log if sleep is significant to reduce spam
if sleep_time > 0.1:
logger.debug(f"⏸️ DNS rate limiting: sleeping for {sleep_time:.2f}s")
time.sleep(sleep_time)
self.last_request = time.time()
self.query_count += 1
def resolve_hostname_fast(self, hostname: str) -> List[str]:
"""Fast hostname resolution optimized for TLD testing."""
ips = []
logger.debug(f"🚀 Fast resolving hostname: {hostname}")
# Use only the first DNS server and shorter timeout for TLD testing
resolver = dns.resolver.Resolver()
resolver.nameservers = [self.config.DNS_SERVERS[0]] # Use primary DNS only
resolver.timeout = 2 # Shorter timeout for TLD testing
resolver.lifetime = 2 # Total query time limit
try:
# Try A records only for speed (most common)
answers = resolver.resolve(hostname, 'A')
for answer in answers:
ips.append(str(answer))
logger.debug(f"⚡ Fast A record for {hostname}: {answer}")
except dns.resolver.NXDOMAIN:
logger.debug(f"❌ NXDOMAIN for {hostname}")
except dns.resolver.NoAnswer:
logger.debug(f"⚠️ No A record for {hostname}")
except dns.resolver.Timeout:
logger.debug(f"⏱️ Timeout for {hostname}")
except Exception as e:
logger.debug(f"⚠️ Error fast resolving {hostname}: {e}")
if ips:
logger.debug(f"⚡ Fast resolved {hostname} to {len(ips)} IPs: {ips}")
return ips
def resolve_hostname(self, hostname: str, operation_id: Optional[str] = None) -> List[str]:
"""Resolve hostname to IP addresses (full resolution with retries)."""
ips = []
logger.debug(f"🔍 Resolving hostname: {hostname}")
for dns_server in self.config.DNS_SERVERS:
self._rate_limit()
resolver = dns.resolver.Resolver()
resolver.nameservers = [dns_server]
resolver.timeout = self.config.DNS_TIMEOUT
try:
# Try A records
answers = resolver.resolve(hostname, 'A')
for answer in answers:
ips.append(str(answer))
logger.debug(f"✅ A record for {hostname}: {answer}")
except dns.resolver.NXDOMAIN:
logger.debug(f"❌ NXDOMAIN for {hostname} A record on {dns_server}")
except dns.resolver.NoAnswer:
logger.debug(f"⚠️ No A record for {hostname} on {dns_server}")
except Exception as e:
logger.debug(f"⚠️ Error resolving A record for {hostname} on {dns_server}: {e}")
try:
# Try AAAA records (IPv6)
answers = resolver.resolve(hostname, 'AAAA')
for answer in answers:
ips.append(str(answer))
logger.debug(f"✅ AAAA record for {hostname}: {answer}")
except dns.resolver.NXDOMAIN:
logger.debug(f"❌ NXDOMAIN for {hostname} AAAA record on {dns_server}")
except dns.resolver.NoAnswer:
logger.debug(f"⚠️ No AAAA record for {hostname} on {dns_server}")
except Exception as e:
logger.debug(f"⚠️ Error resolving AAAA record for {hostname} on {dns_server}: {e}")
unique_ips = list(set(ips))
if unique_ips:
logger.info(f"✅ Resolved {hostname} to {len(unique_ips)} unique IPs: {unique_ips}")
else:
logger.debug(f"❌ No IPs found for {hostname}")
return unique_ips
def get_all_dns_records(self, hostname: str, operation_id: Optional[str] = None) -> List[DNSRecord]:
"""Get all DNS records for a hostname with forensic tracking."""
records = []
successful_queries = 0
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"📋 Getting all DNS records for: {hostname} (operation: {operation_id})")
for record_type in self.RECORD_TYPES:
type_found = False
for dns_server in self.config.DNS_SERVERS:
self._rate_limit()
resolver = dns.resolver.Resolver()
resolver.nameservers = [dns_server]
resolver.timeout = self.config.DNS_TIMEOUT
try:
answers = resolver.resolve(hostname, record_type)
for answer in answers:
# Create DNSRecord with forensic metadata
record = DNSRecord(
record_type=record_type,
value=str(answer),
ttl=answers.ttl,
operation_id=operation_id # Forensic tracking
)
records.append(record)
if not type_found:
logger.debug(f"✅ Found {record_type} record for {hostname}: {answer}")
type_found = True
if not type_found:
successful_queries += 1
break # Found records, no need to query other DNS servers for this type
except dns.resolver.NXDOMAIN:
logger.debug(f"❌ NXDOMAIN for {hostname} {record_type} on {dns_server}")
break # Domain doesn't exist, no point checking other servers
except dns.resolver.NoAnswer:
logger.debug(f"⚠️ No {record_type} record for {hostname} on {dns_server}")
continue # Try next DNS server
except dns.resolver.Timeout:
logger.debug(f"⏱️ Timeout for {hostname} {record_type} on {dns_server}")
continue # Try next DNS server
except Exception as e:
logger.debug(f"⚠️ Error querying {record_type} for {hostname} on {dns_server}: {e}")
continue # Try next DNS server
logger.info(f"📋 Found {len(records)} DNS records for {hostname} across {len(set(r.record_type for r in records))} record types")
# Log query statistics every 100 queries
if self.query_count % 100 == 0:
logger.info(f"📊 DNS query statistics: {self.query_count} total queries performed")
return records
def query_specific_record_type(self, hostname: str, record_type: str, operation_id: Optional[str] = None) -> List[DNSRecord]:
"""Query a specific DNS record type with forensic tracking."""
records = []
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🎯 Querying {record_type} records for {hostname} (operation: {operation_id})")
for dns_server in self.config.DNS_SERVERS:
self._rate_limit()
resolver = dns.resolver.Resolver()
resolver.nameservers = [dns_server]
resolver.timeout = self.config.DNS_TIMEOUT
try:
answers = resolver.resolve(hostname, record_type)
for answer in answers:
# Create DNSRecord with forensic metadata
record = DNSRecord(
record_type=record_type,
value=str(answer),
ttl=answers.ttl,
operation_id=operation_id # Forensic tracking
)
records.append(record)
logger.debug(f"✅ {record_type} record for {hostname}: {answer}")
break # Found records, no need to query other DNS servers
except dns.resolver.NXDOMAIN:
logger.debug(f"❌ NXDOMAIN for {hostname} {record_type} on {dns_server}")
break # Domain doesn't exist, no point checking other servers
except dns.resolver.NoAnswer:
logger.debug(f"⚠️ No {record_type} record for {hostname} on {dns_server}")
continue # Try next DNS server
except dns.resolver.Timeout:
logger.debug(f"⏱️ Timeout for {hostname} {record_type} on {dns_server}")
continue # Try next DNS server
except Exception as e:
logger.debug(f"⚠️ Error querying {record_type} for {hostname} on {dns_server}: {e}")
continue # Try next DNS server
logger.debug(f"🎯 Found {len(records)} {record_type} records for {hostname}")
return records
def reverse_dns_lookup(self, ip: str, operation_id: Optional[str] = None) -> Optional[str]:
"""Perform reverse DNS lookup with forensic tracking."""
logger.debug(f"🔍 Reverse DNS lookup for: {ip} (operation: {operation_id or 'auto'})")
try:
self._rate_limit()
hostname = socket.gethostbyaddr(ip)[0]
logger.info(f"✅ Reverse DNS for {ip}: {hostname}")
return hostname
except socket.herror:
logger.debug(f"❌ No reverse DNS for {ip}")
return None
except Exception as e:
logger.debug(f"⚠️ Error in reverse DNS for {ip}: {e}")
return None
def extract_subdomains_from_dns(self, records: List[DNSRecord]) -> Set[str]:
"""Extract potential subdomains from DNS records."""
subdomains = set()
logger.debug(f"🌿 Extracting subdomains from {len(records)} DNS records")
for record in records:
value = record.value.lower()
# Extract from different record types
try:
if record.record_type == 'MX':
# MX record format: "priority hostname"
parts = value.split()
if len(parts) >= 2:
hostname = parts[-1].rstrip('.') # Take the last part (hostname)
if self._is_valid_hostname(hostname):
subdomains.add(hostname)
logger.debug(f"🌿 Found subdomain from MX: {hostname}")
elif record.record_type in ['CNAME', 'NS']:
# Direct hostname records
hostname = value.rstrip('.')
if self._is_valid_hostname(hostname):
subdomains.add(hostname)
logger.debug(f"🌿 Found subdomain from {record.record_type}: {hostname}")
elif record.record_type == 'TXT':
# Search for domain-like strings in TXT records
# Common patterns: include:example.com, v=spf1 include:_spf.google.com
words = value.replace(',', ' ').replace(';', ' ').split()
for word in words:
# Look for include: patterns
if word.startswith('include:'):
hostname = word[8:].rstrip('.')
if self._is_valid_hostname(hostname):
subdomains.add(hostname)
logger.debug(f"🌿 Found subdomain from TXT include: {hostname}")
# Look for other domain patterns
elif '.' in word and not word.startswith('http'):
clean_word = word.strip('",\'()[]{}').rstrip('.')
if self._is_valid_hostname(clean_word):
subdomains.add(clean_word)
logger.debug(f"🌿 Found subdomain from TXT: {clean_word}")
elif record.record_type == 'SRV':
# SRV record format: "priority weight port target"
parts = value.split()
if len(parts) >= 4:
hostname = parts[-1].rstrip('.') # Target hostname
if self._is_valid_hostname(hostname):
subdomains.add(hostname)
logger.debug(f"🌿 Found subdomain from SRV: {hostname}")
except Exception as e:
logger.debug(f"⚠️ Error extracting subdomain from {record.record_type} record '{value}': {e}")
continue
if subdomains:
logger.info(f"🌿 Extracted {len(subdomains)} potential subdomains")
else:
logger.debug("❌ No subdomains extracted from DNS records")
return subdomains
def _is_valid_hostname(self, hostname: str) -> bool:
"""Basic hostname validation."""
if not hostname or len(hostname) > 255:
return False
# Must contain at least one dot
if '.' not in hostname:
return False
# Must not be an IP address
if self._looks_like_ip(hostname):
return False
# Basic character check - allow international domains
# Remove overly restrictive character filtering
if not hostname.replace('-', '').replace('.', '').replace('_', '').isalnum():
# Allow some special cases for internationalized domains
try:
hostname.encode('ascii')
except UnicodeEncodeError:
return False # Skip non-ASCII for now
# Must have reasonable length parts
parts = hostname.split('.')
if len(parts) < 2:
return False
# Each part should be reasonable length
for part in parts:
if len(part) < 1 or len(part) > 63:
return False
return True
def _looks_like_ip(self, text: str) -> bool:
"""Check if text looks like an IP address."""
try:
socket.inet_aton(text)
return True
except socket.error:
pass
try:
socket.inet_pton(socket.AF_INET6, text)
return True
except socket.error:
pass
return False

220
src/main.py
View File

@ -1,220 +0,0 @@
# File: src/main.py
"""Main CLI interface for the forensic reconnaissance tool with two-mode operation."""
import click
import json
import sys
import logging
from pathlib import Path
from .config import Config
from .reconnaissance import ForensicReconnaissanceEngine
from .report_generator import ForensicReportGenerator
from .web_app import create_app
# Module logger
logger = logging.getLogger(__name__)
@click.command()
@click.argument('target', required=False)
@click.option('--web', is_flag=True, help='Start web interface instead of CLI')
@click.option('--shodan-key', help='Shodan API key')
@click.option('--virustotal-key', help='VirusTotal API key')
@click.option('--max-depth', default=2, help='Maximum recursion depth for full domain mode (default: 2)')
@click.option('--output', '-o', help='Output file prefix (will create .json and .txt files)')
@click.option('--json-only', is_flag=True, help='Only output JSON')
@click.option('--text-only', is_flag=True, help='Only output text report')
@click.option('--port', default=5000, help='Port for web interface (default: 5000)')
@click.option('--verbose', '-v', is_flag=True, help='Enable verbose logging (DEBUG level)')
@click.option('--quiet', '-q', is_flag=True, help='Quiet mode (WARNING level only)')
def main(target, web, shodan_key, virustotal_key, max_depth, output, json_only, text_only, port, verbose, quiet):
"""Forensic DNS Reconnaissance Tool - Two-Mode Operation
MODE 1 - Hostname-only (e.g., 'cc24'):
Expands hostname to all TLDs (cc24.com, cc24.net, etc.)
No recursive enumeration to avoid third-party infrastructure noise
Perfect for discovering domains using a specific hostname
MODE 2 - Full domain (e.g., 'cc24.com'):
Full recursive reconnaissance with subdomain discovery
Maps complete infrastructure of the specified domain
Uses max-depth for recursive enumeration
Creates forensic-grade evidence chain with operation tracking
Examples:
recon cc24 # Mode 1: Find all cc24.* domains (no recursion)
recon cc24.com # Mode 2: Map cc24.com infrastructure (with forensic tracking)
recon cc24.com --max-depth 3 # Mode 2: Deeper recursive enumeration
recon cc24 -v # Mode 1: Verbose TLD expansion
recon --web # Start web interface with forensic visualization
"""
# Determine log level
if verbose:
log_level = "DEBUG"
elif quiet:
log_level = "WARNING"
else:
log_level = "INFO"
# Create configuration and setup logging
config = Config.from_args(shodan_key, virustotal_key, max_depth, log_level)
config.setup_logging(cli_mode=True)
if web:
# Start web interface
logger.info("🌐 Starting forensic web interface...")
app = create_app(config)
logger.info(f"🚀 Forensic web interface starting on http://0.0.0.0:{port}")
app.run(host='0.0.0.0', port=port, debug=False)
return
if not target:
click.echo("Error: TARGET is required for CLI mode. Use --web for web interface.")
sys.exit(1)
# Initialize forensic reconnaissance engine
logger.info("🔬 Initializing forensic reconnaissance engine...")
engine = ForensicReconnaissanceEngine(config)
# Set up progress callback for CLI
def progress_callback(message, percentage=None):
if percentage is not None:
click.echo(f"[{percentage:3d}%] {message}")
else:
click.echo(f" {message}")
engine.set_progress_callback(progress_callback)
# Display startup information
click.echo("=" * 80)
click.echo("FORENSIC DNS RECONNAISSANCE TOOL")
click.echo("=" * 80)
click.echo(f"Target: {target}")
# Show operation mode
if '.' in target:
click.echo(f"Mode: Full domain reconnaissance (recursive depth: {max_depth})")
click.echo(" → Will map complete infrastructure with forensic tracking")
click.echo(" → Creates evidence chain for each discovery operation")
else:
click.echo(f"Mode: Hostname-only reconnaissance (TLD expansion)")
click.echo(" → Will find all domains using this hostname (no recursion)")
click.echo(" → Limited forensic tracking for efficiency")
click.echo(f"DNS servers: {', '.join(config.DNS_SERVERS[:3])}{'...' if len(config.DNS_SERVERS) > 3 else ''}")
click.echo(f"DNS rate limit: {config.DNS_RATE_LIMIT}/s")
if shodan_key:
click.echo("🕵️ Shodan integration enabled")
else:
click.echo("Shodan integration disabled (no API key)")
if virustotal_key:
click.echo("🛡️ VirusTotal integration enabled")
else:
click.echo("VirusTotal integration disabled (no API key)")
click.echo("")
# Run forensic reconnaissance
try:
logger.info(f"🎯 Starting forensic reconnaissance for target: {target}")
data = engine.run_reconnaissance(target)
# Display final statistics
stats = data.get_stats()
graph_analysis = data._generate_graph_analysis()
click.echo("")
click.echo("=" * 80)
click.echo("FORENSIC RECONNAISSANCE COMPLETE")
click.echo("=" * 80)
click.echo(f"Hostnames discovered: {stats['hostnames']}")
click.echo(f"IP addresses found: {stats['ip_addresses']}")
click.echo(f"Discovery relationships: {stats['discovery_edges']}")
click.echo(f"Operations performed: {stats['operations_performed']}")
click.echo(f"DNS records collected: {stats['dns_records']}")
click.echo(f"Certificates found: {stats['certificates_total']} ({stats['certificates_current']} valid, {stats['certificates_expired']} expired)")
click.echo(f"Shodan results: {stats['shodan_results']}")
click.echo(f"VirusTotal results: {stats['virustotal_results']}")
click.echo(f"Maximum discovery depth: {graph_analysis['max_depth']}")
# Calculate and display timing
if data.end_time and data.start_time:
duration = data.end_time - data.start_time
click.echo(f"Total time: {duration}")
click.echo("")
# Generate forensic reports
logger.info("📄 Generating forensic reports...")
report_gen = ForensicReportGenerator(data)
if output:
# Save to files
saved_files = []
if not text_only:
json_file = f"{output}.json"
try:
json_content = data.to_json()
with open(json_file, 'w', encoding='utf-8') as f:
f.write(json_content)
saved_files.append(json_file)
logger.info(f"📄 Forensic JSON report saved: {json_file}")
except Exception as e:
logger.error(f"❌ Failed to save JSON report: {e}")
if not json_only:
text_file = f"{output}.txt"
try:
with open(text_file, 'w', encoding='utf-8') as f:
f.write(report_gen.generate_text_report())
saved_files.append(text_file)
logger.info(f"📄 Forensic text report saved: {text_file}")
except Exception as e:
logger.error(f"❌ Failed to save text report: {e}")
if saved_files:
click.echo(f"📁 Forensic reports saved:")
for file in saved_files:
click.echo(f" {file}")
else:
# Output to stdout
if json_only:
try:
click.echo(data.to_json())
except Exception as e:
logger.error(f"❌ Failed to generate JSON output: {e}")
click.echo(f"Error generating JSON: {e}")
elif text_only:
try:
click.echo(report_gen.generate_text_report())
except Exception as e:
logger.error(f"❌ Failed to generate text report: {e}")
click.echo(f"Error generating text report: {e}")
else:
# Default: show text report
try:
click.echo(report_gen.generate_text_report())
click.echo(f"\n📊 To get JSON output with full forensic data, use: --json-only")
click.echo(f"💾 To save reports, use: --output filename")
click.echo(f"🌐 For interactive visualization, use: --web")
except Exception as e:
logger.error(f"❌ Failed to generate report: {e}")
click.echo(f"Error generating report: {e}")
except KeyboardInterrupt:
logger.warning("⚠️ Forensic reconnaissance interrupted by user")
click.echo("\n🛑 Reconnaissance interrupted by user.")
sys.exit(1)
except Exception as e:
logger.error(f"❌ Error during forensic reconnaissance: {e}", exc_info=True)
click.echo(f"❌ Error during reconnaissance: {e}")
if verbose:
raise # Re-raise in verbose mode to show full traceback
sys.exit(1)
if __name__ == '__main__':
main()

670
src/reconnaissance.py
View File

@ -1,670 +0,0 @@
# File: src/reconnaissance.py
"""Enhanced reconnaissance logic with forensic-grade operation tracking and provenance."""
import threading
import concurrent.futures
import logging
from datetime import datetime
from typing import Set, List, Optional, Tuple, Dict, Any
from .data_structures import (
ForensicReconData, DiscoveryNode, DiscoveryOperation, DiscoveryEdge,
OperationType, DiscoveryMethod, DNSRecord, Certificate
)
from .config import Config
from .dns_resolver import DNSResolver
from .certificate_checker import CertificateChecker
from .shodan_client import ShodanClient
from .virustotal_client import VirusTotalClient
from .tld_fetcher import TLDFetcher
# Set up logging for this module
logger = logging.getLogger(__name__)
class ForensicReconnaissanceEngine:
"""Enhanced reconnaissance engine with complete forensic tracking and provenance."""
def __init__(self, config: Config):
self.config = config
# Initialize clients
self.dns_resolver = DNSResolver(config)
self.cert_checker = CertificateChecker(config)
self.tld_fetcher = TLDFetcher()
# Optional clients
self.shodan_client = None
if config.shodan_key:
self.shodan_client = ShodanClient(config.shodan_key, config)
logger.info("🕵️ Shodan client initialized")
self.virustotal_client = None
if config.virustotal_key:
self.virustotal_client = VirusTotalClient(config.virustotal_key, config)
logger.info("🛡️ VirusTotal client initialized")
# Progress tracking
self.progress_callback = None
self._lock = threading.Lock()
# Track operation mode
self.is_hostname_only_mode = False
# Operation tracking
self.pending_operations = []
self.completed_operations = []
def set_progress_callback(self, callback):
"""Set callback for progress updates."""
self.progress_callback = callback
def set_shared_data(self, shared_data: ForensicReconData):
"""Set shared data object for live updates."""
self.data = shared_data
logger.info("📊 Using shared forensic data object for live updates")
def _update_progress(self, message: str, percentage: int = None):
"""Update progress if callback is set."""
logger.info(f"Progress: {message} ({percentage}%)" if percentage else f"Progress: {message}")
if self.progress_callback:
self.progress_callback(message, percentage)
def run_reconnaissance(self, target: str) -> ForensicReconData:
"""Run forensic reconnaissance with complete operation tracking."""
# Initialize data structure
if not hasattr(self, 'data') or self.data is None:
self.data = ForensicReconData()
logger.info("🔬 Created new forensic data structure")
self.data.start_time = datetime.now()
self.data.scan_config = {
'target': target,
'max_depth': self.config.max_depth,
'dns_servers': self.config.DNS_SERVERS,
'shodan_enabled': self.shodan_client is not None,
'virustotal_enabled': self.virustotal_client is not None
}
logger.info(f"🎯 Starting forensic reconnaissance for target: {target}")
try:
# Determine operation mode and create initial operation
if '.' in target:
self.is_hostname_only_mode = False
logger.info(f"🔍 Full domain mode: {target}")
self._create_initial_operation(target, OperationType.INITIAL_TARGET)
else:
self.is_hostname_only_mode = True
logger.info(f"🔍 Hostname expansion mode: {target}")
self._perform_tld_expansion(target)
# Process all discovery operations
self._process_discovery_queue()
# Perform external lookups for full domain mode
if not self.is_hostname_only_mode:
self._perform_external_lookups()
# Finalize
self.data.end_time = datetime.now()
duration = self.data.end_time - self.data.start_time
stats = self.data.get_stats()
logger.info(f"🏁 Forensic reconnaissance complete in {duration}")
logger.info(f"📊 Final stats: {stats}")
self._update_progress("Forensic reconnaissance complete", 100)
except Exception as e:
logger.error(f"❌ Error during forensic reconnaissance: {e}", exc_info=True)
raise
return self.data
def _create_initial_operation(self, target: str, operation_type: OperationType):
"""Create initial operation for the target."""
operation = DiscoveryOperation(
operation_type=operation_type,
target=target,
discovered_hostnames=[target]
)
self.data.add_operation(operation)
# Create initial node
node = self.data.add_node(
hostname=target,
depth=0,
discovery_method=DiscoveryMethod.INITIAL_TARGET,
operation_id=operation.operation_id
)
# Queue initial DNS operations
self._queue_dns_operations_for_hostname(target, 0)
logger.info(f"🔍 Created initial operation for {target}")
def _perform_tld_expansion(self, hostname: str):
"""Perform TLD expansion with forensic tracking."""
logger.info(f"🌐 Starting TLD expansion for: {hostname}")
# Create TLD expansion operation
operation = DiscoveryOperation(
operation_type=OperationType.TLD_EXPANSION,
target=hostname,
metadata={'expansion_type': 'smart_prioritized'}
)
self._update_progress(f"Expanding {hostname} to all TLDs", 5)
# Get prioritized TLD lists
priority_tlds, normal_tlds, deprioritized_tlds = self.tld_fetcher.get_prioritized_tlds()
valid_domains = set()
# Phase 1: Priority TLDs
logger.info("🎯 Phase 1: Checking priority TLDs...")
priority_results = self._check_tlds_parallel(hostname, priority_tlds, operation.operation_id)
valid_domains.update(priority_results)
# Phase 2: Normal TLDs (if needed)
if len(valid_domains) < 5:
logger.info("🔍 Phase 2: Checking normal TLDs...")
normal_results = self._check_tlds_parallel(hostname, normal_tlds, operation.operation_id)
valid_domains.update(normal_results)
# Phase 3: Deprioritized TLDs (if really needed)
if len(valid_domains) < 2:
logger.info("🔎 Phase 3: Checking deprioritized TLDs...")
depri_results = self._check_tlds_parallel(hostname, deprioritized_tlds, operation.operation_id)
valid_domains.update(depri_results)
# Update operation with results
operation.discovered_hostnames = list(valid_domains)
operation.success = len(valid_domains) > 0
self.data.add_operation(operation)
# Create nodes for all discovered domains
for domain in valid_domains:
node = self.data.add_node(
hostname=domain,
depth=0,
discovery_method=DiscoveryMethod.TLD_EXPANSION,
operation_id=operation.operation_id
)
# Add discovery edge (synthetic source for TLD expansion)
self.data.add_edge(
source=f"tld_expansion:{hostname}",
target=domain,
discovery_method=DiscoveryMethod.TLD_EXPANSION,
operation_id=operation.operation_id,
metadata={'original_hostname': hostname}
)
logger.info(f"✅ TLD expansion complete: {len(valid_domains)} domains found")
# Queue lightweight operations for hostname-only mode
for domain in valid_domains:
self._queue_lightweight_operations(domain)
def _check_tlds_parallel(self, hostname: str, tlds: List[str], operation_id: str) -> Set[str]:
"""Check TLDs in parallel with operation tracking."""
valid_domains = set()
max_workers = min(20, len(tlds))
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_tld = {
executor.submit(self._check_single_tld_forensic, hostname, tld, operation_id): tld
for tld in tlds
}
for future in concurrent.futures.as_completed(future_to_tld):
tld = future_to_tld[future]
try:
result = future.result(timeout=10)
if result:
domain, ips = result
valid_domains.add(domain)
# Track discovered IPs
for ip in ips:
self.data.ip_addresses.add(ip)
except Exception as e:
logger.debug(f"Error checking {hostname}.{tld}: {e}")
return valid_domains
def _check_single_tld_forensic(self, hostname: str, tld: str, operation_id: str) -> Optional[Tuple[str, List[str]]]:
"""Check single TLD with forensic tracking."""
full_hostname = f"{hostname}.{tld}"
# Quick DNS resolution
ips = self.dns_resolver.resolve_hostname_fast(full_hostname)
if ips:
# Create DNS operation record
dns_operation = DiscoveryOperation(
operation_type=OperationType.DNS_A,
target=full_hostname,
discovered_ips=ips,
metadata={'tld_expansion': True, 'parent_operation': operation_id}
)
# Add DNS records
for ip in ips:
record = DNSRecord(
record_type='A',
value=ip,
operation_id=dns_operation.operation_id
)
dns_operation.dns_records.append(record)
self.data.add_operation(dns_operation)
return (full_hostname, ips)
return None
def _queue_dns_operations_for_hostname(self, hostname: str, depth: int):
"""Queue comprehensive DNS operations for a hostname."""
# Map DNS record types to operation types
dns_operations = [
(OperationType.DNS_A, 'A'),
(OperationType.DNS_AAAA, 'AAAA'),
(OperationType.DNS_MX, 'MX'),
(OperationType.DNS_NS, 'NS'),
(OperationType.DNS_TXT, 'TXT'),
(OperationType.DNS_CNAME, 'CNAME'),
(OperationType.DNS_SOA, 'SOA'),
(OperationType.DNS_SRV, 'SRV'),
(OperationType.DNS_CAA, 'CAA'),
]
for operation_type, record_type in dns_operations:
self.pending_operations.append({
'type': 'dns_query',
'operation_type': operation_type,
'record_type': record_type,
'hostname': hostname,
'depth': depth
})
# Queue certificate operation
self.pending_operations.append({
'type': 'certificate_check',
'operation_type': OperationType.CERTIFICATE_CHECK,
'hostname': hostname,
'depth': depth
})
logger.debug(f"📋 Queued {len(dns_operations) + 1} operations for {hostname}")
def _queue_lightweight_operations(self, hostname: str):
"""Queue lightweight operations for hostname-only mode."""
# Only essential DNS operations
essential_operations = [
(OperationType.DNS_A, 'A'),
(OperationType.DNS_AAAA, 'AAAA'),
(OperationType.DNS_MX, 'MX'),
(OperationType.DNS_TXT, 'TXT'),
]
for operation_type, record_type in essential_operations:
self.pending_operations.append({
'type': 'dns_query',
'operation_type': operation_type,
'record_type': record_type,
'hostname': hostname,
'depth': 0
})
logger.debug(f"📋 Queued {len(essential_operations)} lightweight operations for {hostname}")
def _process_discovery_queue(self):
"""Process all queued discovery operations."""
operation_count = 0
total_operations = len(self.pending_operations)
logger.info(f"⚙️ Processing {total_operations} operations")
while self.pending_operations:
operation_spec = self.pending_operations.pop(0)
operation_count += 1
# Calculate progress
progress = 20 + (operation_count * 60 // max(total_operations, 1))
try:
if operation_spec['type'] == 'dns_query':
self._execute_dns_operation(operation_spec)
elif operation_spec['type'] == 'certificate_check':
self._execute_certificate_operation(operation_spec)
# Update progress periodically
if operation_count % 10 == 0:
self._update_progress(f"Processed {operation_count}/{total_operations} operations", progress)
except Exception as e:
logger.error(f"❌ Error processing operation {operation_spec}: {e}")
continue
logger.info(f"✅ Completed processing {operation_count} operations")
def _execute_dns_operation(self, operation_spec: Dict[str, Any]):
"""Execute a single DNS operation with full tracking."""
hostname = operation_spec['hostname']
record_type = operation_spec['record_type']
operation_type = operation_spec['operation_type']
depth = operation_spec['depth']
logger.debug(f"🔍 DNS {record_type} query for {hostname}")
# Create operation
operation = DiscoveryOperation(
operation_type=operation_type,
target=hostname,
metadata={'record_type': record_type, 'depth': depth}
)
try:
# Perform DNS query using the updated method with operation_id
records = self.dns_resolver.query_specific_record_type(hostname, record_type, operation.operation_id)
operation.dns_records = records
operation.success = len(records) > 0
# Process results
node = self.data.get_node(hostname)
if not node:
node = self.data.add_node(hostname, depth)
new_hostnames = set()
new_ips = set()
for record in records:
node.add_dns_record(record)
# Extract IPs and hostnames
if record.record_type in ['A', 'AAAA']:
new_ips.add(record.value)
self.data.ip_addresses.add(record.value)
elif record.record_type in ['MX', 'NS', 'CNAME']:
# Extract hostname from MX (format: "priority hostname")
if record.record_type == 'MX':
parts = record.value.split()
if len(parts) >= 2:
extracted_hostname = parts[-1].rstrip('.')
else:
continue
else:
extracted_hostname = record.value.rstrip('.')
if self._is_valid_hostname(extracted_hostname) and extracted_hostname != hostname:
new_hostnames.add(extracted_hostname)
# Update operation results
operation.discovered_hostnames = list(new_hostnames)
operation.discovered_ips = list(new_ips)
# Add discovery edges and queue new operations
for new_hostname in new_hostnames:
# Add node
new_node = self.data.add_node(
hostname=new_hostname,
depth=depth + 1,
discovery_method=DiscoveryMethod.DNS_RECORD_VALUE,
operation_id=operation.operation_id
)
# Add edge
self.data.add_edge(
source=hostname,
target=new_hostname,
discovery_method=DiscoveryMethod.DNS_RECORD_VALUE,
operation_id=operation.operation_id,
metadata={'record_type': record_type}
)
# Queue operations for new hostname (if not in hostname-only mode and within depth)
if not self.is_hostname_only_mode and depth + 1 <= self.config.max_depth:
self._queue_dns_operations_for_hostname(new_hostname, depth + 1)
logger.debug(f"✅ DNS {record_type} for {hostname}: {len(records)} records, {len(new_hostnames)} new hostnames")
except Exception as e:
operation.success = False
operation.error_message = str(e)
logger.debug(f"❌ DNS {record_type} query failed for {hostname}: {e}")
self.data.add_operation(operation)
def _execute_certificate_operation(self, operation_spec: Dict[str, Any]):
"""Execute certificate check operation."""
hostname = operation_spec['hostname']
depth = operation_spec['depth']
# Skip certificates in hostname-only mode
if self.is_hostname_only_mode:
logger.debug(f"⭐ Skipping certificate check for {hostname} (hostname-only mode)")
return
logger.debug(f"🔍 Certificate check for {hostname}")
operation = DiscoveryOperation(
operation_type=OperationType.CERTIFICATE_CHECK,
target=hostname,
metadata={'depth': depth}
)
try:
# Use updated method with operation_id
certificates = self.cert_checker.get_certificates(hostname, operation.operation_id)
operation.certificates = certificates
operation.success = len(certificates) > 0
# Process certificates
node = self.data.get_node(hostname)
if node:
new_hostnames = set()
for cert in certificates:
node.add_certificate(cert)
# Extract hostnames from certificate subjects
subject_hostnames = self.cert_checker.extract_subdomains_from_certificates([cert])
for subject_hostname in subject_hostnames:
if subject_hostname != hostname and self._is_valid_hostname(subject_hostname):
new_hostnames.add(subject_hostname)
# Update operation and create edges
operation.discovered_hostnames = list(new_hostnames)
for new_hostname in new_hostnames:
# Add node
new_node = self.data.add_node(
hostname=new_hostname,
depth=depth + 1,
discovery_method=DiscoveryMethod.CERTIFICATE_SUBJECT,
operation_id=operation.operation_id
)
# Add edge
self.data.add_edge(
source=hostname,
target=new_hostname,
discovery_method=DiscoveryMethod.CERTIFICATE_SUBJECT,
operation_id=operation.operation_id
)
# Queue operations for new hostname (within depth limit)
if depth + 1 <= self.config.max_depth:
self._queue_dns_operations_for_hostname(new_hostname, depth + 1)
logger.debug(f"✅ Certificates for {hostname}: {len(certificates)} certs, {len(new_hostnames)} new hostnames")
except Exception as e:
operation.success = False
operation.error_message = str(e)
logger.debug(f"❌ Certificate check failed for {hostname}: {e}")
self.data.add_operation(operation)
def _perform_external_lookups(self):
"""Perform external service lookups with operation tracking."""
if self.is_hostname_only_mode:
logger.info("⭐ Skipping external lookups (hostname-only mode)")
return
self._update_progress("Performing external service lookups", 85)
# Reverse DNS
self._perform_reverse_dns_lookups()
# Shodan lookups
if self.shodan_client:
self._perform_shodan_lookups()
# VirusTotal lookups
if self.virustotal_client:
self._perform_virustotal_lookups()
def _perform_reverse_dns_lookups(self):
"""Perform reverse DNS lookups with operation tracking."""
logger.info(f"🔄 Performing reverse DNS for {len(self.data.ip_addresses)} IPs")
for ip in self.data.ip_addresses:
operation = DiscoveryOperation(
operation_type=OperationType.DNS_REVERSE,
target=ip
)
try:
# Use updated method with operation_id
reverse_hostname = self.dns_resolver.reverse_dns_lookup(ip, operation.operation_id)
if reverse_hostname:
operation.discovered_hostnames = [reverse_hostname]
operation.success = True
# Update nodes that have this IP
for node in self.data.nodes.values():
if ip in node.resolved_ips:
node.reverse_dns = reverse_hostname
logger.debug(f"🔄 Reverse DNS {ip} -> {reverse_hostname}")
else:
operation.success = False
except Exception as e:
operation.success = False
operation.error_message = str(e)
self.data.add_operation(operation)
def _perform_shodan_lookups(self):
"""Perform Shodan lookups with operation tracking."""
logger.info(f"🕵️ Performing Shodan lookups for {len(self.data.ip_addresses)} IPs")
for ip in self.data.ip_addresses:
operation = DiscoveryOperation(
operation_type=OperationType.SHODAN_LOOKUP,
target=ip
)
try:
# Use updated method with operation_id
result = self.shodan_client.lookup_ip(ip, operation.operation_id)
if result:
operation.shodan_results = [result]
operation.success = True
# Add to relevant nodes
for node in self.data.nodes.values():
if ip in node.resolved_ips:
node.shodan_results.append(result)
logger.debug(f"🕵️ Shodan {ip}: {len(result.ports)} ports")
else:
operation.success = False
except Exception as e:
operation.success = False
operation.error_message = str(e)
self.data.add_operation(operation)
def _perform_virustotal_lookups(self):
"""Perform VirusTotal lookups with operation tracking."""
total_resources = len(self.data.ip_addresses) + len(self.data.nodes)
logger.info(f"🛡️ Performing VirusTotal lookups for {total_resources} resources")
# Check IPs
for ip in self.data.ip_addresses:
operation = DiscoveryOperation(
operation_type=OperationType.VIRUSTOTAL_IP,
target=ip
)
try:
# Use updated method with operation_id
result = self.virustotal_client.lookup_ip(ip, operation.operation_id)
if result:
operation.virustotal_results = [result]
operation.success = True
# Add to relevant nodes
for node in self.data.nodes.values():
if ip in node.resolved_ips:
node.virustotal_results.append(result)
logger.debug(f"🛡️ VirusTotal {ip}: {result.positives}/{result.total}")
else:
operation.success = False
except Exception as e:
operation.success = False
operation.error_message = str(e)
self.data.add_operation(operation)
# Check domains
for hostname, node in self.data.nodes.items():
operation = DiscoveryOperation(
operation_type=OperationType.VIRUSTOTAL_DOMAIN,
target=hostname
)
try:
# Use updated method with operation_id
result = self.virustotal_client.lookup_domain(hostname, operation.operation_id)
if result:
operation.virustotal_results = [result]
operation.success = True
node.virustotal_results.append(result)
logger.debug(f"🛡️ VirusTotal {hostname}: {result.positives}/{result.total}")
else:
operation.success = False
except Exception as e:
operation.success = False
operation.error_message = str(e)
self.data.add_operation(operation)
def _is_valid_hostname(self, hostname: str) -> bool:
"""Validate hostname format."""
if not hostname or '.' not in hostname or len(hostname) > 255:
return False
# Basic validation
parts = hostname.split('.')
if len(parts) < 2:
return False
for part in parts:
if not part or len(part) > 63:
return False
return True
# Backward compatibility
ReconnaissanceEngine = ForensicReconnaissanceEngine

451
src/report_generator.py
View File

@ -1,451 +0,0 @@
# File: src/report_generator.py
"""Enhanced report generation with forensic details and discovery graph visualization."""
from datetime import datetime
from typing import Dict, Any, List, Set
from .data_structures import ForensicReconData, DiscoveryMethod, OperationType
import logging
logger = logging.getLogger(__name__)
class ForensicReportGenerator:
"""Generate comprehensive forensic reports with discovery provenance."""
def __init__(self, data: ForensicReconData):
self.data = data
def generate_text_report(self) -> str:
"""Generate comprehensive forensic text report."""
report = []
# Header
report.append("=" * 80)
report.append("FORENSIC DNS RECONNAISSANCE REPORT")
report.append("=" * 80)
report.append(f"Scan Start: {self.data.start_time}")
if self.data.end_time:
report.append(f"Scan End: {self.data.end_time}")
duration = self.data.end_time - self.data.start_time
report.append(f"Duration: {duration}")
report.append(f"Target: {self.data.scan_config.get('target', 'Unknown')}")
report.append(f"Max Depth: {self.data.scan_config.get('max_depth', 'Unknown')}")
report.append("")
# Executive Summary
report.append("EXECUTIVE SUMMARY")
report.append("-" * 40)
stats = self.data.get_stats()
report.append(f"Discovered Hostnames: {stats['hostnames']}")
report.append(f"IP Addresses Found: {stats['ip_addresses']}")
report.append(f"Operations Performed: {stats['operations_performed']}")
report.append(f"Discovery Relationships: {stats['discovery_edges']}")
report.append(f"DNS Records Collected: {stats['dns_records']}")
report.append(f"Total Certificates: {stats['certificates_total']}")
report.append(f" └─ Currently Valid: {stats['certificates_current']}")
report.append(f" └─ Expired: {stats['certificates_expired']}")
report.append(f"Shodan Results: {stats['shodan_results']}")
report.append(f"VirusTotal Results: {stats['virustotal_results']}")
report.append("")
# Discovery Graph Analysis
graph_analysis = self.data._generate_graph_analysis()
report.append("DISCOVERY GRAPH ANALYSIS")
report.append("-" * 40)
report.append(f"Maximum Discovery Depth: {graph_analysis['max_depth']}")
report.append(f"Root Nodes (Initial Targets): {len(graph_analysis['root_nodes'])}")
report.append(f"Leaf Nodes (No Further Discoveries): {len(graph_analysis['leaf_nodes'])}")
report.append("")
# Depth Distribution
report.append("Discovery Depth Distribution:")
for depth, count in sorted(graph_analysis['depth_distribution'].items()):
report.append(f" Depth {depth}: {count} hostnames")
report.append("")
# Discovery Methods Distribution
report.append("Discovery Methods Used:")
for method, count in sorted(graph_analysis['discovery_method_distribution'].items()):
report.append(f" {method}: {count} discoveries")
report.append("")
# Discovery Tree
report.append("DISCOVERY TREE")
report.append("-" * 40)
report.extend(self._generate_discovery_tree())
report.append("")
# Detailed Node Analysis
report.append("DETAILED NODE ANALYSIS")
report.append("-" * 40)
report.extend(self._generate_node_details())
report.append("")
# Operations Timeline
report.append("OPERATIONS TIMELINE")
report.append("-" * 40)
report.extend(self._generate_operations_timeline())
report.append("")
# Security Analysis
security_findings = self._analyze_security_findings()
if security_findings:
report.append("SECURITY ANALYSIS")
report.append("-" * 40)
report.extend(security_findings)
report.append("")
# Certificate Analysis
cert_analysis = self._analyze_certificates()
if cert_analysis:
report.append("CERTIFICATE ANALYSIS")
report.append("-" * 40)
report.extend(cert_analysis)
report.append("")
# DNS Record Analysis
report.append("DNS RECORD ANALYSIS")
report.append("-" * 40)
report.extend(self._analyze_dns_records())
report.append("")
return "\n".join(report)
def _generate_discovery_tree(self) -> List[str]:
"""Generate a tree view of hostname discoveries."""
tree_lines = []
# Find root nodes
graph_analysis = self.data._generate_graph_analysis()
root_nodes = graph_analysis['root_nodes']
if not root_nodes:
tree_lines.append("No root nodes found")
return tree_lines
# Generate tree for each root
for root in sorted(root_nodes):
tree_lines.extend(self._build_tree_branch(root, "", set()))
return tree_lines
def _build_tree_branch(self, hostname: str, prefix: str, visited: Set[str]) -> List[str]:
"""Build a tree branch for a hostname."""
lines = []
# Avoid cycles
if hostname in visited:
lines.append(f"{prefix}{hostname} [CYCLE]")
return lines
visited.add(hostname)
# Get node info
node = self.data.get_node(hostname)
if not node:
lines.append(f"{prefix}{hostname} [NO NODE DATA]")
return lines
# Node info
node_info = f"{hostname} (depth:{node.depth}"
if node.resolved_ips:
node_info += f", IPs:{len(node.resolved_ips)}"
if node.certificates:
valid_certs = len(node.get_current_certificates())
expired_certs = len(node.get_expired_certificates())
node_info += f", certs:{valid_certs}+{expired_certs}"
node_info += ")"
lines.append(f"{prefix}{node_info}")
# Get children
children = self.data.get_children(hostname)
children.sort()
for i, child in enumerate(children):
is_last = (i == len(children) - 1)
child_prefix = prefix + ("└── " if is_last else "├── ")
next_prefix = prefix + (" " if is_last else "│ ")
# Find discovery method for this child
discovery_method = "unknown"
for edge in self.data.edges:
if edge.source_hostname == hostname and edge.target_hostname == child:
discovery_method = edge.discovery_method.value
break
lines.append(f"{child_prefix}[{discovery_method}]")
lines.extend(self._build_tree_branch(child, next_prefix, visited.copy()))
return lines
def _generate_node_details(self) -> List[str]:
"""Generate detailed analysis of each node."""
details = []
# Sort nodes by depth, then alphabetically
sorted_nodes = sorted(self.data.nodes.items(),
key=lambda x: (x[1].depth, x[0]))
for hostname, node in sorted_nodes:
details.append(f"\n{hostname} (Depth {node.depth})")
details.append("-" * (len(hostname) + 20))
# Discovery provenance
details.append(f"First Seen: {node.first_seen}")
details.append(f"Last Updated: {node.last_updated}")
details.append(f"Discovery Methods: {', '.join(m.value for m in node.discovery_methods)}")
# Discovery paths
paths = self.data.get_discovery_path(hostname)
if paths:
details.append("Discovery Paths:")
for i, path in enumerate(paths[:3]): # Show max 3 paths
path_str = " -> ".join([f"{src}[{method.value}]{tgt}" for src, tgt, method in path])
details.append(f" Path {i+1}: {path_str}")
if len(paths) > 3:
details.append(f" ... and {len(paths) - 3} more paths")
# DNS status
if node.dns_exists is not None:
status = "EXISTS" if node.dns_exists else "NOT FOUND"
details.append(f"DNS Status: {status} (checked: {node.last_dns_check})")
# IP addresses
if node.resolved_ips:
details.append(f"Resolved IPs: {', '.join(sorted(node.resolved_ips))}")
# Reverse DNS
if node.reverse_dns:
details.append(f"Reverse DNS: {node.reverse_dns}")
# DNS records summary
total_records = len(node.get_all_dns_records())
if total_records > 0:
record_types = list(node.dns_records_by_type.keys())
details.append(f"DNS Records: {total_records} records ({', '.join(sorted(record_types))})")
# Certificates summary
current_certs = len(node.get_current_certificates())
expired_certs = len(node.get_expired_certificates())
if current_certs > 0 or expired_certs > 0:
details.append(f"Certificates: {current_certs} valid, {expired_certs} expired")
# External results
if node.shodan_results:
details.append(f"Shodan: {len(node.shodan_results)} results")
if node.virustotal_results:
vt_detections = sum(r.positives for r in node.virustotal_results)
details.append(f"VirusTotal: {len(node.virustotal_results)} scans, {vt_detections} total detections")
return details
def _generate_operations_timeline(self) -> List[str]:
"""Generate operations timeline."""
timeline = []
# Sort operations by timestamp
sorted_ops = []
for op_id in self.data.operation_timeline:
if op_id in self.data.operations:
sorted_ops.append(self.data.operations[op_id])
# Group operations by type for summary
op_summary = {}
for op in sorted_ops:
op_type = op.operation_type.value
if op_type not in op_summary:
op_summary[op_type] = {'total': 0, 'successful': 0, 'failed': 0}
op_summary[op_type]['total'] += 1
if op.success:
op_summary[op_type]['successful'] += 1
else:
op_summary[op_type]['failed'] += 1
# Operations summary
timeline.append("Operations Summary:")
for op_type, counts in sorted(op_summary.items()):
timeline.append(f" {op_type}: {counts['successful']}/{counts['total']} successful")
timeline.append("")
# Recent operations (last 20)
timeline.append("Recent Operations (last 20):")
recent_ops = sorted_ops[-20:] if len(sorted_ops) > 20 else sorted_ops
for op in recent_ops:
timestamp = op.timestamp.strftime("%H:%M:%S.%f")[:-3]
status = "✓" if op.success else "✗"
target_short = op.target[:30] + "..." if len(op.target) > 30 else op.target
timeline.append(f" {timestamp} {status} {op.operation_type.value:15} {target_short}")
# Show key results
if op.discovered_hostnames:
hostname_list = ", ".join(op.discovered_hostnames[:3])
if len(op.discovered_hostnames) > 3:
hostname_list += f" (+{len(op.discovered_hostnames) - 3} more)"
timeline.append(f" └─ Discovered: {hostname_list}")
if op.error_message:
timeline.append(f" └─ Error: {op.error_message[:50]}...")
return timeline
def _analyze_security_findings(self) -> List[str]:
"""Analyze security-related findings."""
findings = []
# VirusTotal detections
high_risk_resources = []
medium_risk_resources = []
for node in self.data.nodes.values():
for vt_result in node.virustotal_results:
if vt_result.positives > 5:
high_risk_resources.append((node.hostname, vt_result))
elif vt_result.positives > 0:
medium_risk_resources.append((node.hostname, vt_result))
if high_risk_resources:
findings.append("🚨 HIGH RISK FINDINGS:")
for hostname, vt_result in high_risk_resources:
findings.append(f" {hostname}: {vt_result.positives}/{vt_result.total} detections")
findings.append(f" Report: {vt_result.permalink}")
if medium_risk_resources:
findings.append("⚠️ MEDIUM RISK FINDINGS:")
for hostname, vt_result in medium_risk_resources[:5]: # Show max 5
findings.append(f" {hostname}: {vt_result.positives}/{vt_result.total} detections")
if len(medium_risk_resources) > 5:
findings.append(f" ... and {len(medium_risk_resources) - 5} more resources with detections")
# Expired certificates still in use
nodes_with_expired_certs = []
for hostname, node in self.data.nodes.items():
expired = node.get_expired_certificates()
current = node.get_current_certificates()
if expired and not current: # Only expired certs, no valid ones
nodes_with_expired_certs.append((hostname, len(expired)))
if nodes_with_expired_certs:
findings.append("📜 CERTIFICATE ISSUES:")
for hostname, count in nodes_with_expired_certs:
findings.append(f" {hostname}: {count} expired certificates, no valid ones")
return findings
def _analyze_certificates(self) -> List[str]:
"""Analyze certificate findings."""
cert_analysis = []
# Certificate statistics
total_certs = 0
valid_certs = 0
expired_certs = 0
wildcard_certs = 0
cert_authorities = {}
for node in self.data.nodes.values():
for cert in node.certificates:
total_certs += 1
if cert.is_valid_now:
valid_certs += 1
else:
expired_certs += 1
if cert.is_wildcard:
wildcard_certs += 1
# Count certificate authorities
issuer_short = cert.issuer.split(',')[0] if ',' in cert.issuer else cert.issuer
cert_authorities[issuer_short] = cert_authorities.get(issuer_short, 0) + 1
if total_certs == 0:
cert_analysis.append("No certificates found.")
return cert_analysis
cert_analysis.append(f"Total Certificates: {total_certs}")
cert_analysis.append(f" Currently Valid: {valid_certs}")
cert_analysis.append(f" Expired: {expired_certs}")
cert_analysis.append(f" Wildcard Certificates: {wildcard_certs}")
cert_analysis.append("")
# Top certificate authorities
cert_analysis.append("Certificate Authorities:")
sorted_cas = sorted(cert_authorities.items(), key=lambda x: x[1], reverse=True)
for ca, count in sorted_cas[:5]:
cert_analysis.append(f" {ca}: {count} certificates")
cert_analysis.append("")
# Expiring soon (within 30 days)
from datetime import timedelta
soon = datetime.now() + timedelta(days=30)
expiring_soon = []
for hostname, node in self.data.nodes.items():
for cert in node.get_current_certificates():
if cert.not_after <= soon:
expiring_soon.append((hostname, cert.not_after, cert.id))
if expiring_soon:
cert_analysis.append("Certificates Expiring Soon (within 30 days):")
for hostname, expiry, cert_id in sorted(expiring_soon, key=lambda x: x[1]):
cert_analysis.append(f" {hostname}: expires {expiry.strftime('%Y-%m-%d')} (cert ID: {cert_id})")
return cert_analysis
def _analyze_dns_records(self) -> List[str]:
"""Analyze DNS record patterns."""
dns_analysis = []
# Record type distribution
record_type_counts = {}
total_records = 0
for node in self.data.nodes.values():
for record_type, records in node.dns_records_by_type.items():
record_type_counts[record_type] = record_type_counts.get(record_type, 0) + len(records)
total_records += len(records)
dns_analysis.append(f"Total DNS Records: {total_records}")
dns_analysis.append("Record Type Distribution:")
for record_type, count in sorted(record_type_counts.items()):
percentage = (count / total_records * 100) if total_records > 0 else 0
dns_analysis.append(f" {record_type}: {count} ({percentage:.1f}%)")
dns_analysis.append("")
# Interesting findings
interesting = []
# Multiple MX records
multi_mx_nodes = []
for hostname, node in self.data.nodes.items():
mx_records = node.dns_records_by_type.get('MX', [])
if len(mx_records) > 1:
multi_mx_nodes.append((hostname, len(mx_records)))
if multi_mx_nodes:
interesting.append("Multiple MX Records:")
for hostname, count in multi_mx_nodes:
interesting.append(f" {hostname}: {count} MX records")
# CAA records (security-relevant)
caa_nodes = []
for hostname, node in self.data.nodes.items():
if 'CAA' in node.dns_records_by_type:
caa_nodes.append(hostname)
if caa_nodes:
interesting.append(f"Domains with CAA Records: {len(caa_nodes)}")
for hostname in caa_nodes[:5]: # Show first 5
interesting.append(f" {hostname}")
if interesting:
dns_analysis.append("Interesting DNS Findings:")
dns_analysis.extend(interesting)
return dns_analysis
# Backward compatibility
ReportGenerator = ForensicReportGenerator

177
src/shodan_client.py
View File

@ -1,177 +0,0 @@
# File: src/shodan_client.py
"""Shodan API integration with forensic operation tracking."""
import requests
import time
import logging
import uuid
from typing import Optional, Dict, Any, List
from .data_structures import ShodanResult
from .config import Config
# Module logger
logger = logging.getLogger(__name__)
class ShodanClient:
"""Shodan API client with forensic tracking."""
BASE_URL = "https://api.shodan.io"
def __init__(self, api_key: str, config: Config):
self.api_key = api_key
self.config = config
self.last_request = 0
logger.info(f"🕵️ Shodan client initialized with API key ending in: ...{api_key[-4:] if len(api_key) > 4 else api_key}")
def _rate_limit(self):
"""Apply rate limiting for Shodan."""
now = time.time()
time_since_last = now - self.last_request
min_interval = 1.0 / self.config.SHODAN_RATE_LIMIT
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
logger.debug(f"⏸️ Shodan rate limiting: sleeping for {sleep_time:.2f}s")
time.sleep(sleep_time)
self.last_request = time.time()
def lookup_ip(self, ip: str, operation_id: Optional[str] = None) -> Optional[ShodanResult]:
"""Lookup IP address information with forensic tracking."""
self._rate_limit()
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🔍 Querying Shodan for IP: {ip} (operation: {operation_id})")
try:
url = f"{self.BASE_URL}/shodan/host/{ip}"
params = {'key': self.api_key}
response = requests.get(
url,
params=params,
timeout=self.config.HTTP_TIMEOUT,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
logger.debug(f"📡 Shodan API response for {ip}: {response.status_code}")
if response.status_code == 200:
data = response.json()
ports = []
services = {}
for service in data.get('data', []):
port = service.get('port')
if port:
ports.append(port)
services[str(port)] = {
'product': service.get('product', ''),
'version': service.get('version', ''),
'banner': service.get('data', '').strip()[:200] if service.get('data') else ''
}
# Create ShodanResult with forensic metadata
result = ShodanResult(
ip=ip,
ports=sorted(list(set(ports))),
services=services,
organization=data.get('org'),
country=data.get('country_name'),
operation_id=operation_id # Forensic tracking
)
logger.info(f"✅ Shodan result for {ip}: {len(result.ports)} ports, org: {result.organization}")
return result
elif response.status_code == 404:
logger.debug(f"ℹ️ IP {ip} not found in Shodan database")
return None
elif response.status_code == 401:
logger.error("❌ Shodan API key is invalid or expired")
return None
elif response.status_code == 429:
logger.warning("⚠️ Shodan API rate limit exceeded")
return None
else:
logger.warning(f"⚠️ Shodan API error for {ip}: HTTP {response.status_code}")
try:
error_data = response.json()
logger.debug(f"Shodan error details: {error_data}")
except:
pass
return None
except requests.exceptions.Timeout:
logger.warning(f"⏱️ Shodan query timeout for {ip}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"🌐 Shodan network error for {ip}: {e}")
return None
except Exception as e:
logger.error(f"❌ Unexpected error querying Shodan for {ip}: {e}")
return None
def search_domain(self, domain: str, operation_id: Optional[str] = None) -> List[str]:
"""Search for IPs associated with a domain with forensic tracking."""
self._rate_limit()
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🔍 Searching Shodan for domain: {domain} (operation: {operation_id})")
try:
url = f"{self.BASE_URL}/shodan/host/search"
params = {
'key': self.api_key,
'query': f'hostname:{domain}',
'limit': 100
}
response = requests.get(
url,
params=params,
timeout=self.config.HTTP_TIMEOUT,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
logger.debug(f"📡 Shodan search response for {domain}: {response.status_code}")
if response.status_code == 200:
data = response.json()
ips = []
for match in data.get('matches', []):
ip = match.get('ip_str')
if ip:
ips.append(ip)
unique_ips = list(set(ips))
logger.info(f"🔍 Shodan search for {domain} found {len(unique_ips)} unique IPs")
return unique_ips
elif response.status_code == 401:
logger.error("❌ Shodan API key is invalid for search")
return []
elif response.status_code == 429:
logger.warning("⚠️ Shodan search rate limit exceeded")
return []
else:
logger.warning(f"⚠️ Shodan search error for {domain}: HTTP {response.status_code}")
return []
except requests.exceptions.Timeout:
logger.warning(f"⏱️ Shodan search timeout for {domain}")
return []
except requests.exceptions.RequestException as e:
logger.error(f"🌐 Shodan search network error for {domain}: {e}")
return []
except Exception as e:
logger.error(f"❌ Unexpected error searching Shodan for {domain}: {e}")
return []

213
src/tld_fetcher.py
View File

@ -1,213 +0,0 @@
# File: src/tld_fetcher.py
"""Fetch and cache IANA TLD list with smart prioritization."""
import requests
import logging
from typing import List, Set, Optional, Tuple
import os
import time
# Module logger
logger = logging.getLogger(__name__)
class TLDFetcher:
"""Fetches and caches IANA TLD list with smart prioritization."""
IANA_TLD_URL = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
CACHE_FILE = "tlds_cache.txt"
CACHE_DURATION = 86400 # 24 hours in seconds
# Common TLDs that should be checked first (high success rate)
PRIORITY_TLDS = {
# Generic top-level domains (most common)
'com', 'org', 'net', 'edu', 'gov', 'mil', 'int', 'info', 'biz', 'name',
'io', 'co', 'me', 'tv', 'cc', 'ly', 'to', 'us', 'uk', 'ca',
# Major country codes (high usage)
'de', 'fr', 'it', 'es', 'nl', 'be', 'ch', 'at', 'se', 'no', 'dk', 'fi',
'au', 'nz', 'jp', 'kr', 'cn', 'hk', 'sg', 'my', 'th', 'in', 'br', 'mx',
'ru', 'pl', 'cz', 'hu', 'ro', 'bg', 'hr', 'si', 'sk', 'lt', 'lv', 'ee',
'ie', 'pt', 'gr', 'cy', 'mt', 'lu', 'is', 'tr', 'il', 'za', 'ng', 'eg',
# Popular new gTLDs (established, not spam-prone)
'app', 'dev', 'tech', 'blog', 'news', 'shop', 'store', 'cloud', 'digital',
'website', 'site', 'online', 'world', 'global', 'international'
}
# TLDs to deprioritize (often have wildcard DNS or low-quality domains)
DEPRIORITIZED_PATTERNS = [
'xn--', # Internationalized domain names (often less common)
# These TLDs are known for high wildcard/parking rates
'tk', 'ml', 'ga', 'cf', # Free TLDs often misused
'top', 'win', 'download', 'stream', 'science', 'click', 'link',
'loan', 'men', 'racing', 'review', 'party', 'trade', 'date',
'cricket', 'accountant', 'faith', 'gdn', 'realtor'
]
def __init__(self):
self._tlds: Optional[Set[str]] = None
self._prioritized_tlds: Optional[Tuple[List[str], List[str], List[str]]] = None
logger.info("🌐 TLD fetcher initialized with smart prioritization")
def get_tlds(self) -> Set[str]:
"""Get list of TLDs, using cache if available."""
if self._tlds is None:
logger.debug("🔍 Loading TLD list...")
self._tlds = self._load_tlds()
logger.info(f"✅ Loaded {len(self._tlds)} TLDs")
return self._tlds
def get_prioritized_tlds(self) -> Tuple[List[str], List[str], List[str]]:
"""Get TLDs sorted by priority: (priority, normal, deprioritized)."""
if self._prioritized_tlds is None:
all_tlds = self.get_tlds()
logger.debug("📊 Categorizing TLDs by priority...")
priority_list = []
normal_list = []
deprioritized_list = []
for tld in all_tlds:
tld_lower = tld.lower()
if tld_lower in self.PRIORITY_TLDS:
priority_list.append(tld_lower)
elif any(pattern in tld_lower for pattern in self.DEPRIORITIZED_PATTERNS):
deprioritized_list.append(tld_lower)
else:
normal_list.append(tld_lower)
# Sort each category alphabetically for consistency
priority_list.sort()
normal_list.sort()
deprioritized_list.sort()
self._prioritized_tlds = (priority_list, normal_list, deprioritized_list)
logger.info(f"📊 TLD prioritization complete: "
f"{len(priority_list)} priority, "
f"{len(normal_list)} normal, "
f"{len(deprioritized_list)} deprioritized")
return self._prioritized_tlds
def _load_tlds(self) -> Set[str]:
"""Load TLDs from cache or fetch from IANA."""
if self._is_cache_valid():
logger.debug("📂 Loading TLDs from cache")
return self._load_from_cache()
else:
logger.info("🌐 Fetching fresh TLD list from IANA")
return self._fetch_and_cache()
def _is_cache_valid(self) -> bool:
"""Check if cache file exists and is recent."""
if not os.path.exists(self.CACHE_FILE):
logger.debug("❌ TLD cache file does not exist")
return False
cache_age = time.time() - os.path.getmtime(self.CACHE_FILE)
is_valid = cache_age < self.CACHE_DURATION
if is_valid:
logger.debug(f"✅ TLD cache is valid (age: {cache_age/3600:.1f} hours)")
else:
logger.debug(f"❌ TLD cache is expired (age: {cache_age/3600:.1f} hours)")
return is_valid
def _load_from_cache(self) -> Set[str]:
"""Load TLDs from cache file."""
try:
with open(self.CACHE_FILE, 'r', encoding='utf-8') as f:
tlds = set()
for line in f:
line = line.strip().lower()
if line and not line.startswith('#'):
tlds.add(line)
logger.info(f"📂 Loaded {len(tlds)} TLDs from cache")
return tlds
except Exception as e:
logger.error(f"❌ Error loading TLD cache: {e}")
# Fall back to fetching fresh data
return self._fetch_and_cache()
def _fetch_and_cache(self) -> Set[str]:
"""Fetch TLDs from IANA and cache them."""
try:
logger.info(f"📡 Fetching TLD list from: {self.IANA_TLD_URL}")
response = requests.get(
self.IANA_TLD_URL,
timeout=30,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
response.raise_for_status()
tlds = set()
lines_processed = 0
for line in response.text.split('\n'):
line = line.strip().lower()
if line and not line.startswith('#'):
tlds.add(line)
lines_processed += 1
logger.info(f"✅ Fetched {len(tlds)} TLDs from IANA (processed {lines_processed} lines)")
# Cache the results
try:
with open(self.CACHE_FILE, 'w', encoding='utf-8') as f:
f.write(response.text)
logger.info(f"💾 TLD list cached to {self.CACHE_FILE}")
except Exception as cache_error:
logger.warning(f"⚠️ Could not cache TLD list: {cache_error}")
return tlds
except requests.exceptions.Timeout:
logger.error("⏱️ Timeout fetching TLD list from IANA")
return self._get_fallback_tlds()
except requests.exceptions.RequestException as e:
logger.error(f"🌐 Network error fetching TLD list: {e}")
return self._get_fallback_tlds()
except Exception as e:
logger.error(f"❌ Unexpected error fetching TLD list: {e}")
return self._get_fallback_tlds()
def _get_fallback_tlds(self) -> Set[str]:
"""Return a minimal set of short TLDs if fetch fails."""
logger.warning("⚠️ Using fallback TLD list")
# Use only short, well-established TLDs as fallback
fallback_tlds = {
# 2-character TLDs (country codes - most established)
'ad', 'ae', 'af', 'ag', 'ai', 'al', 'am', 'ao', 'aq', 'ar', 'as', 'at',
'au', 'aw', 'ax', 'az', 'ba', 'bb', 'bd', 'be', 'bf', 'bg', 'bh', 'bi',
'bj', 'bl', 'bm', 'bn', 'bo', 'bq', 'br', 'bs', 'bt', 'bv', 'bw', 'by',
'bz', 'ca', 'cc', 'cd', 'cf', 'cg', 'ch', 'ci', 'ck', 'cl', 'cm', 'cn',
'co', 'cr', 'cu', 'cv', 'cw', 'cx', 'cy', 'cz', 'de', 'dj', 'dk', 'dm',
'do', 'dz', 'ec', 'ee', 'eg', 'eh', 'er', 'es', 'et', 'eu', 'fi', 'fj',
'fk', 'fm', 'fo', 'fr', 'ga', 'gb', 'gd', 'ge', 'gf', 'gg', 'gh', 'gi',
'gl', 'gm', 'gn', 'gp', 'gq', 'gr', 'gs', 'gt', 'gu', 'gw', 'gy', 'hk',
'hm', 'hn', 'hr', 'ht', 'hu', 'id', 'ie', 'il', 'im', 'in', 'io', 'iq',
'ir', 'is', 'it', 'je', 'jm', 'jo', 'jp', 'ke', 'kg', 'kh', 'ki', 'km',
'kn', 'kp', 'kr', 'kw', 'ky', 'kz', 'la', 'lb', 'lc', 'li', 'lk', 'lr',
'ls', 'lt', 'lu', 'lv', 'ly', 'ma', 'mc', 'md', 'me', 'mf', 'mg', 'mh',
'mk', 'ml', 'mm', 'mn', 'mo', 'mp', 'mq', 'mr', 'ms', 'mt', 'mu', 'mv',
'mw', 'mx', 'my', 'mz', 'na', 'nc', 'ne', 'nf', 'ng', 'ni', 'nl', 'no',
'np', 'nr', 'nu', 'nz', 'om', 'pa', 'pe', 'pf', 'pg', 'ph', 'pk', 'pl',
'pm', 'pn', 'pr', 'ps', 'pt', 'pw', 'py', 'qa', 're', 'ro', 'rs', 'ru',
'rw', 'sa', 'sb', 'sc', 'sd', 'se', 'sg', 'sh', 'si', 'sj', 'sk', 'sl',
'sm', 'sn', 'so', 'sr', 'ss', 'st', 'sv', 'sx', 'sy', 'sz', 'tc', 'td',
'tf', 'tg', 'th', 'tj', 'tk', 'tl', 'tm', 'tn', 'to', 'tr', 'tt', 'tv',
'tw', 'tz', 'ua', 'ug', 'uk', 'um', 'us', 'uy', 'uz', 'va', 'vc', 've',
'vg', 'vi', 'vn', 'vu', 'wf', 'ws', 'ye', 'yt', 'za', 'zm', 'zw',
# 3-character TLDs (generic - most common)
'com', 'org', 'net', 'edu', 'gov', 'mil', 'int'
}
logger.info(f"📋 Using {len(fallback_tlds)} fallback TLDs (≤3 characters)")
return fallback_tlds

227
src/virustotal_client.py
View File

@ -1,227 +0,0 @@
# File: src/virustotal_client.py
"""VirusTotal API integration with forensic operation tracking."""
import requests
import time
import logging
import uuid
from datetime import datetime
from typing import Optional
from .data_structures import VirusTotalResult
from .config import Config
# Module logger
logger = logging.getLogger(__name__)
class VirusTotalClient:
"""VirusTotal API client with forensic tracking."""
BASE_URL = "https://www.virustotal.com/vtapi/v2"
def __init__(self, api_key: str, config: Config):
self.api_key = api_key
self.config = config
self.last_request = 0
logger.info(f"🛡️ VirusTotal client initialized with API key ending in: ...{api_key[-4:] if len(api_key) > 4 else api_key}")
def _rate_limit(self):
"""Apply rate limiting for VirusTotal."""
now = time.time()
time_since_last = now - self.last_request
min_interval = 1.0 / self.config.VIRUSTOTAL_RATE_LIMIT
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
logger.debug(f"⏸️ VirusTotal rate limiting: sleeping for {sleep_time:.2f}s")
time.sleep(sleep_time)
self.last_request = time.time()
def lookup_ip(self, ip: str, operation_id: Optional[str] = None) -> Optional[VirusTotalResult]:
"""Lookup IP address reputation with forensic tracking."""
self._rate_limit()
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🔍 Querying VirusTotal for IP: {ip} (operation: {operation_id})")
try:
url = f"{self.BASE_URL}/ip-address/report"
params = {
'apikey': self.api_key,
'ip': ip
}
response = requests.get(
url,
params=params,
timeout=self.config.HTTP_TIMEOUT,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
logger.debug(f"📡 VirusTotal API response for IP {ip}: {response.status_code}")
if response.status_code == 200:
data = response.json()
logger.debug(f"VirusTotal IP response data keys: {data.keys()}")
if data.get('response_code') == 1:
# Count detected URLs
detected_urls = data.get('detected_urls', [])
positives = sum(1 for url in detected_urls if url.get('positives', 0) > 0)
total = len(detected_urls)
# Parse scan date
scan_date = datetime.now()
if data.get('scan_date'):
try:
scan_date = datetime.fromisoformat(data['scan_date'].replace('Z', '+00:00'))
except ValueError:
try:
scan_date = datetime.strptime(data['scan_date'], '%Y-%m-%d %H:%M:%S')
except ValueError:
logger.debug(f"Could not parse scan_date: {data.get('scan_date')}")
# Create VirusTotalResult with forensic metadata
result = VirusTotalResult(
resource=ip,
positives=positives,
total=total,
scan_date=scan_date,
permalink=data.get('permalink', f'https://www.virustotal.com/gui/ip-address/{ip}'),
operation_id=operation_id # Forensic tracking
)
logger.info(f"✅ VirusTotal result for IP {ip}: {result.positives}/{result.total} detections")
return result
elif data.get('response_code') == 0:
logger.debug(f"ℹ️ IP {ip} not found in VirusTotal database")
return None
else:
logger.debug(f"VirusTotal returned response_code: {data.get('response_code')}")
return None
elif response.status_code == 204:
logger.warning("⚠️ VirusTotal API rate limit exceeded")
return None
elif response.status_code == 403:
logger.error("❌ VirusTotal API key is invalid or lacks permissions")
return None
else:
logger.warning(f"⚠️ VirusTotal API error for IP {ip}: HTTP {response.status_code}")
try:
error_data = response.json()
logger.debug(f"VirusTotal error details: {error_data}")
except:
pass
return None
except requests.exceptions.Timeout:
logger.warning(f"⏱️ VirusTotal query timeout for IP {ip}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"🌐 VirusTotal network error for IP {ip}: {e}")
return None
except Exception as e:
logger.error(f"❌ Unexpected error querying VirusTotal for IP {ip}: {e}")
return None
def lookup_domain(self, domain: str, operation_id: Optional[str] = None) -> Optional[VirusTotalResult]:
"""Lookup domain reputation with forensic tracking."""
self._rate_limit()
# Generate operation ID if not provided
if operation_id is None:
operation_id = str(uuid.uuid4())
logger.debug(f"🔍 Querying VirusTotal for domain: {domain} (operation: {operation_id})")
try:
url = f"{self.BASE_URL}/domain/report"
params = {
'apikey': self.api_key,
'domain': domain
}
response = requests.get(
url,
params=params,
timeout=self.config.HTTP_TIMEOUT,
headers={'User-Agent': 'DNS-Recon-Tool/1.0'}
)
logger.debug(f"📡 VirusTotal API response for domain {domain}: {response.status_code}")
if response.status_code == 200:
data = response.json()
logger.debug(f"VirusTotal domain response data keys: {data.keys()}")
if data.get('response_code') == 1:
# Count detected URLs
detected_urls = data.get('detected_urls', [])
positives = sum(1 for url in detected_urls if url.get('positives', 0) > 0)
total = len(detected_urls)
# Also check for malicious/suspicious categories
categories = data.get('categories', [])
if any(cat in ['malicious', 'suspicious', 'phishing', 'malware']
for cat in categories):
positives += 1
# Parse scan date
scan_date = datetime.now()
if data.get('scan_date'):
try:
scan_date = datetime.fromisoformat(data['scan_date'].replace('Z', '+00:00'))
except ValueError:
try:
scan_date = datetime.strptime(data['scan_date'], '%Y-%m-%d %H:%M:%S')
except ValueError:
logger.debug(f"Could not parse scan_date: {data.get('scan_date')}")
# Create VirusTotalResult with forensic metadata
result = VirusTotalResult(
resource=domain,
positives=positives,
total=max(total, 1), # Ensure total is at least 1
scan_date=scan_date,
permalink=data.get('permalink', f'https://www.virustotal.com/gui/domain/{domain}'),
operation_id=operation_id # Forensic tracking
)
logger.info(f"✅ VirusTotal result for domain {domain}: {result.positives}/{result.total} detections")
return result
elif data.get('response_code') == 0:
logger.debug(f"ℹ️ Domain {domain} not found in VirusTotal database")
return None
else:
logger.debug(f"VirusTotal returned response_code: {data.get('response_code')}")
return None
elif response.status_code == 204:
logger.warning("⚠️ VirusTotal API rate limit exceeded")
return None
elif response.status_code == 403:
logger.error("❌ VirusTotal API key is invalid or lacks permissions")
return None
else:
logger.warning(f"⚠️ VirusTotal API error for domain {domain}: HTTP {response.status_code}")
try:
error_data = response.json()
logger.debug(f"VirusTotal error details: {error_data}")
except:
pass
return None
except requests.exceptions.Timeout:
logger.warning(f"⏱️ VirusTotal query timeout for domain {domain}")
return None
except requests.exceptions.RequestException as e:
logger.error(f"🌐 VirusTotal network error for domain {domain}: {e}")
return None
except Exception as e:
logger.error(f"❌ Unexpected error querying VirusTotal for domain {domain}: {e}")
return None

393
src/web_app.py
View File

@ -1,393 +0,0 @@
# File: src/web_app.py
"""Flask web application for forensic reconnaissance tool."""
from flask import Flask, render_template, request, jsonify, send_from_directory
import threading
import time
import logging
from .config import Config
from .reconnaissance import ForensicReconnaissanceEngine
from .report_generator import ForensicReportGenerator
from .data_structures import ForensicReconData
# Set up logging for this module
logger = logging.getLogger(__name__)
# Global variables for tracking ongoing scans
active_scans = {}
scan_lock = threading.Lock()
def create_app(config: Config):
"""Create Flask application."""
app = Flask(__name__,
template_folder='../templates',
static_folder='../static')
app.config['SECRET_KEY'] = 'forensic-recon-tool-secret-key'
# Set up logging for web app
config.setup_logging(cli_mode=False)
logger.info("🌐 Forensic web application initialized")
@app.route('/')
def index():
"""Main page."""
return render_template('index.html')
@app.route('/api/scan', methods=['POST'])
def start_scan():
"""Start a new forensic reconnaissance scan."""
try:
data = request.get_json()
target = data.get('target')
scan_config = Config.from_args(
shodan_key=data.get('shodan_key'),
virustotal_key=data.get('virustotal_key'),
max_depth=data.get('max_depth', 2)
)
if not target:
logger.warning("⚠️ Scan request missing target")
return jsonify({'error': 'Target is required'}), 400
# Generate scan ID
scan_id = f"{target}_{int(time.time())}"
logger.info(f"🚀 Starting new forensic scan: {scan_id} for target: {target}")
# Create shared ForensicReconData object for live updates
shared_data = ForensicReconData()
shared_data.scan_config = {
'target': target,
'max_depth': scan_config.max_depth,
'shodan_enabled': scan_config.shodan_key is not None,
'virustotal_enabled': scan_config.virustotal_key is not None
}
# Initialize scan data with the shared forensic data object
with scan_lock:
active_scans[scan_id] = {
'status': 'starting',
'progress': 0,
'message': 'Initializing forensic scan...',
'data': shared_data, # Share the forensic data object from the start
'error': None,
'live_stats': shared_data.get_stats(), # Use forensic stats
'latest_discoveries': []
}
# Start forensic reconnaissance in background thread
thread = threading.Thread(
target=run_forensic_reconnaissance_background,
args=(scan_id, target, scan_config, shared_data)
)
thread.daemon = True
thread.start()
return jsonify({'scan_id': scan_id})
except Exception as e:
logger.error(f"❌ Error starting scan: {e}", exc_info=True)
return jsonify({'error': str(e)}), 500
@app.route('/api/scan/<scan_id>/status')
def get_scan_status(scan_id):
"""Get scan status and progress with live forensic discoveries."""
with scan_lock:
if scan_id not in active_scans:
return jsonify({'error': 'Scan not found'}), 404
scan_data = active_scans[scan_id].copy()
# Update live stats from forensic data if available
if scan_data.get('data') and hasattr(scan_data['data'], 'get_stats'):
scan_data['live_stats'] = scan_data['data'].get_stats()
# Don't include the full forensic data object in status (too large)
if 'data' in scan_data:
del scan_data['data']
return jsonify(scan_data)
@app.route('/api/scan/<scan_id>/report')
def get_scan_report(scan_id):
"""Get forensic scan report."""
with scan_lock:
if scan_id not in active_scans:
return jsonify({'error': 'Scan not found'}), 404
scan_data = active_scans[scan_id]
if scan_data['status'] != 'completed' or not scan_data['data']:
return jsonify({'error': 'Scan not completed'}), 400
try:
# Generate forensic report
report_gen = ForensicReportGenerator(scan_data['data'])
return jsonify({
'json_report': scan_data['data'].to_json(),
'text_report': report_gen.generate_text_report()
})
except Exception as e:
logger.error(f"❌ Error generating report for {scan_id}: {e}", exc_info=True)
return jsonify({'error': f'Failed to generate report: {str(e)}'}), 500
@app.route('/api/scan/<scan_id>/graph')
def get_scan_graph(scan_id):
"""Get graph data for visualization."""
with scan_lock:
if scan_id not in active_scans:
return jsonify({'error': 'Scan not found'}), 404
scan_data = active_scans[scan_id]
if scan_data['status'] != 'completed' or not scan_data['data']:
return jsonify({'error': 'Scan not completed'}), 400
try:
forensic_data = scan_data['data']
# Extract nodes for graph
nodes = []
for hostname, node in forensic_data.nodes.items():
# Determine node color based on depth
color_map = {
0: '#00ff41', # Green for root
1: '#ff9900', # Orange for depth 1
2: '#ff6b6b', # Red for depth 2
3: '#4ecdc4', # Teal for depth 3
4: '#45b7d1', # Blue for depth 4+
}
color = color_map.get(node.depth, '#666666')
# Calculate node size based on number of connections and data
connections = len(forensic_data.get_children(hostname)) + len(forensic_data.get_parents(hostname))
dns_records = len(node.get_all_dns_records())
certificates = len(node.certificates)
# Size based on importance (connections + data)
size = max(8, min(20, 8 + connections * 2 + dns_records // 3 + certificates))
nodes.append({
'id': hostname,
'label': hostname,
'depth': node.depth,
'color': color,
'size': size,
'dns_records': dns_records,
'certificates': certificates,
'ip_addresses': list(node.resolved_ips),
'discovery_methods': [method.value for method in node.discovery_methods],
'first_seen': node.first_seen.isoformat() if node.first_seen else None
})
# Extract edges for graph
edges = []
for edge in forensic_data.edges:
# Skip synthetic TLD expansion edges for cleaner visualization
if edge.source_hostname.startswith('tld_expansion:'):
continue
# Color edges by discovery method
method_colors = {
'initial_target': '#00ff41',
'tld_expansion': '#ff9900',
'dns_record_value': '#4ecdc4',
'certificate_subject': '#ff6b6b',
'dns_subdomain_extraction': '#45b7d1'
}
edges.append({
'source': edge.source_hostname,
'target': edge.target_hostname,
'method': edge.discovery_method.value,
'color': method_colors.get(edge.discovery_method.value, '#666666'),
'operation_id': edge.operation_id,
'timestamp': edge.timestamp.isoformat() if edge.timestamp else None
})
# Graph statistics
stats = {
'node_count': len(nodes),
'edge_count': len(edges),
'max_depth': max([node['depth'] for node in nodes]) if nodes else 0,
'discovery_methods': list(set([edge['method'] for edge in edges])),
'root_nodes': [node['id'] for node in nodes if node['depth'] == 0]
}
return jsonify({
'nodes': nodes,
'edges': edges,
'stats': stats
})
except Exception as e:
logger.error(f"⚠️ Error generating graph data for {scan_id}: {e}", exc_info=True)
return jsonify({'error': f'Failed to generate graph: {str(e)}'}), 500
@app.route('/api/scan/<scan_id>/live-data')
def get_live_scan_data(scan_id):
"""Get current forensic reconnaissance data (for real-time updates)."""
with scan_lock:
if scan_id not in active_scans:
return jsonify({'error': 'Scan not found'}), 404
scan_data = active_scans[scan_id]
forensic_data = scan_data['data']
if not forensic_data:
return jsonify({
'hostnames': [],
'ip_addresses': [],
'stats': {
'hostnames': 0,
'ip_addresses': 0,
'discovery_edges': 0,
'operations_performed': 0,
'dns_records': 0,
'certificates_total': 0,
'certificates_current': 0,
'certificates_expired': 0,
'shodan_results': 0,
'virustotal_results': 0
},
'latest_discoveries': []
})
# Extract data from forensic structure for frontend
try:
hostnames = sorted(list(forensic_data.nodes.keys()))
ip_addresses = sorted(list(forensic_data.ip_addresses))
stats = forensic_data.get_stats()
# Generate activity log from recent operations
latest_discoveries = []
recent_operations = forensic_data.operation_timeline[-10:] # Last 10 operations
for op_id in recent_operations:
if op_id in forensic_data.operations:
operation = forensic_data.operations[op_id]
activity_entry = {
'timestamp': operation.timestamp.timestamp(),
'message': f"{operation.operation_type.value}: {operation.target}"
}
# Add result summary
if operation.discovered_hostnames:
activity_entry['message'] += f" → {len(operation.discovered_hostnames)} hostnames"
if operation.discovered_ips:
activity_entry['message'] += f" → {len(operation.discovered_ips)} IPs"
latest_discoveries.append(activity_entry)
# Update scan data with latest discoveries
scan_data['latest_discoveries'] = latest_discoveries
return jsonify({
'hostnames': hostnames,
'ip_addresses': ip_addresses,
'stats': stats,
'latest_discoveries': latest_discoveries
})
except Exception as e:
logger.error(f"❌ Error extracting live data for {scan_id}: {e}", exc_info=True)
# Return minimal data structure
return jsonify({
'hostnames': [],
'ip_addresses': [],
'stats': {
'hostnames': len(forensic_data.nodes) if forensic_data.nodes else 0,
'ip_addresses': len(forensic_data.ip_addresses) if forensic_data.ip_addresses else 0,
'discovery_edges': 0,
'operations_performed': 0,
'dns_records': 0,
'certificates_total': 0,
'certificates_current': 0,
'certificates_expired': 0,
'shodan_results': 0,
'virustotal_results': 0
},
'latest_discoveries': []
})
return app
def run_forensic_reconnaissance_background(scan_id: str, target: str, config: Config, shared_data: ForensicReconData):
"""Run forensic reconnaissance in background thread with shared forensic data object."""
def update_progress(message: str, percentage: int = None):
"""Update scan progress and live forensic statistics."""
with scan_lock:
if scan_id in active_scans:
active_scans[scan_id]['message'] = message
if percentage is not None:
active_scans[scan_id]['progress'] = percentage
# Update live stats from the shared forensic data object
if shared_data:
active_scans[scan_id]['live_stats'] = shared_data.get_stats()
# Add to latest discoveries (keep last 10)
if 'latest_discoveries' not in active_scans[scan_id]:
active_scans[scan_id]['latest_discoveries'] = []
# Create activity entry
activity_entry = {
'timestamp': time.time(),
'message': message
}
active_scans[scan_id]['latest_discoveries'].append(activity_entry)
# Keep only last 10 discoveries
active_scans[scan_id]['latest_discoveries'] = \
active_scans[scan_id]['latest_discoveries'][-10:]
logger.info(f"[{scan_id}] {message} ({percentage}%)" if percentage else f"[{scan_id}] {message}")
try:
logger.info(f"🔧 Initializing forensic reconnaissance engine for scan: {scan_id}")
# Initialize forensic engine
engine = ForensicReconnaissanceEngine(config)
engine.set_progress_callback(update_progress)
# IMPORTANT: Pass the shared forensic data object to the engine
engine.set_shared_data(shared_data)
# Update status
with scan_lock:
active_scans[scan_id]['status'] = 'running'
logger.info(f"🚀 Starting forensic reconnaissance for: {target}")
# Run forensic reconnaissance - this will populate the shared_data object incrementally
final_data = engine.run_reconnaissance(target)
logger.info(f"✅ Forensic reconnaissance completed for scan: {scan_id}")
# Update with final results (the shared_data should already be populated)
with scan_lock:
active_scans[scan_id]['status'] = 'completed'
active_scans[scan_id]['progress'] = 100
active_scans[scan_id]['message'] = 'Forensic reconnaissance completed'
active_scans[scan_id]['data'] = final_data # This should be the same as shared_data
active_scans[scan_id]['live_stats'] = final_data.get_stats()
# Log final forensic statistics
final_stats = final_data.get_stats()
logger.info(f"📊 Final forensic stats for {scan_id}: {final_stats}")
# Log discovery graph analysis
graph_analysis = final_data._generate_graph_analysis()
logger.info(f"🌐 Discovery graph: {len(final_data.nodes)} nodes, {len(final_data.edges)} edges, max depth: {graph_analysis['max_depth']}")
except Exception as e:
logger.error(f"❌ Error in forensic reconnaissance for {scan_id}: {e}", exc_info=True)
# Handle errors
with scan_lock:
active_scans[scan_id]['status'] = 'error'
active_scans[scan_id]['error'] = str(e)
active_scans[scan_id]['message'] = f'Error: {str(e)}'

601
static/css/main.css Normal file
View File

@ -0,0 +1,601 @@
/* DNSRecon - Tactical/Cybersecurity Theme */
/* Reset and Base Styles */
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Roboto Mono', 'Courier New', monospace;
background-color: #1a1a1a;
color: #c7c7c7;
line-height: 1.6;
min-height: 100vh;
overflow-x: hidden;
}
/* Container and Layout */
.container {
min-height: 100vh;
display: flex;
flex-direction: column;
}
/* Header */
.header {
background-color: #0a0a0a;
border-bottom: 2px solid #444;
padding: 1rem 2rem;
box-shadow: inset 0 0 15px rgba(0,0,0,0.5);
}
.header-content {
display: flex;
justify-content: space-between;
align-items: center;
max-width: 1400px;
margin: 0 auto;
}
.logo {
display: flex;
align-items: center;
font-family: 'Special Elite', 'Courier New', monospace;
font-size: 1.5rem;
font-weight: 700;
}
.logo-icon {
color: #00ff41;
margin-right: 0.5rem;
text-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
}
.logo-text {
color: #c7c7c7;
text-shadow: 0 0 5px rgba(199, 199, 199, 0.3);
}
.status-indicator {
display: flex;
align-items: center;
gap: 0.5rem;
}
.status-dot {
width: 8px;
height: 8px;
border-radius: 50%;
background-color: #00ff41;
box-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
animation: pulse 2s infinite;
}
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.5; }
}
.status-text {
font-size: 0.9rem;
color: #00ff41;
}
/* Main Content */
.main-content {
flex: 1;
padding: 2rem;
max-width: 1400px;
margin: 0 auto;
width: 100%;
display: grid;
grid-template-columns: 1fr 1fr;
grid-template-rows: auto auto 1fr auto;
gap: 1.5rem;
grid-template-areas:
"control status"
"visualization visualization"
"visualization visualization"
"providers providers";
}
/* Panel Base Styles */
.panel-header {
background-color: #2a2a2a;
padding: 1rem;
border-bottom: 1px solid #444;
display: flex;
justify-content: space-between;
align-items: center;
}
.panel-header h2 {
font-size: 1.1rem;
color: #00ff41;
text-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
font-weight: 500;
}
section {
background-color: #2a2a2a;
border: 1px solid #444;
box-shadow: inset 0 0 15px rgba(0,0,0,0.5);
}
/* Control Panel */
.control-panel {
grid-area: control;
}
.form-container {
padding: 1.5rem;
}
.input-group {
margin-bottom: 1.5rem;
}
.input-group label {
display: block;
margin-bottom: 0.5rem;
color: #c7c7c7;
font-size: 0.9rem;
font-weight: 500;
}
input[type="text"], select {
width: 100%;
padding: 0.75rem;
background-color: #1a1a1a;
border: 1px solid #555;
color: #c7c7c7;
font-family: 'Roboto Mono', monospace;
font-size: 0.9rem;
transition: border-color 0.3s ease, box-shadow 0.3s ease;
}
input[type="text"]:focus, select:focus {
outline: none;
border-color: #00ff41;
box-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
}
.button-group {
display: flex;
flex-direction: column;
gap: 0.75rem;
}
/* Buttons */
.btn {
display: flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
padding: 0.75rem 1rem;
font-family: 'Roboto Mono', monospace;
font-size: 0.9rem;
border: none;
cursor: pointer;
transition: all 0.3s ease;
text-transform: uppercase;
}
.btn-primary {
background-color: #2c5c34;
color: #e0e0e0;
}
.btn-primary:hover:not(:disabled) {
background-color: #3b7b46;
box-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
}
.btn-secondary {
background-color: #4a4a4a;
color: #c7c7c7;
}
.btn-secondary:hover:not(:disabled) {
background-color: #5a5a5a;
}
.btn-secondary:active {
background-color: #6a4f2a;
}
.btn:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.btn-icon {
color: #00ff41;
font-weight: 700;
}
.btn-icon-small {
background: transparent;
border: 1px solid #555;
color: #c7c7c7;
padding: 0.25rem 0.5rem;
font-family: 'Roboto Mono', monospace;
font-size: 0.8rem;
cursor: pointer;
transition: all 0.3s ease;
}
.btn-icon-small:hover {
border-color: #00ff41;
color: #00ff41;
}
/* Status Panel */
.status-panel {
grid-area: status;
}
.status-content {
padding: 1.5rem;
}
.status-row {
display: flex;
justify-content: space-between;
margin-bottom: 0.75rem;
padding-bottom: 0.25rem;
border-bottom: 1px solid #333;
}
.status-label {
color: #999;
font-size: 0.9rem;
}
.status-value {
color: #00ff41;
font-weight: 500;
text-shadow: 0 0 3px rgba(0, 255, 65, 0.3);
}
.progress-bar {
margin: 1rem 1.5rem;
height: 8px;
background-color: #1a1a1a;
border: 1px solid #444;
overflow: hidden;
}
.progress-fill {
height: 100%;
background: linear-gradient(90deg, #00ff41, #00aa2e);
width: 0%;
transition: width 0.3s ease;
box-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
}
/* Visualization Panel */
.visualization-panel {
grid-area: visualization;
min-height: 500px;
}
.view-controls {
display: flex;
gap: 0.5rem;
}
.graph-container {
height: 500px;
position: relative;
background-color: #1a1a1a;
border-top: 1px solid #444;
}
.graph-placeholder {
display: flex;
align-items: center;
justify-content: center;
height: 100%;
color: #666;
text-align: center;
}
.placeholder-content {
max-width: 300px;
}
.placeholder-icon {
font-size: 3rem;
margin-bottom: 1rem;
color: #444;
}
.placeholder-text {
font-size: 1.1rem;
margin-bottom: 0.5rem;
color: #999;
}
.placeholder-subtext {
font-size: 0.9rem;
color: #666;
}
.legend {
display: flex;
flex-wrap: wrap;
gap: 1rem;
padding: 1rem 1.5rem;
background-color: #222;
border-top: 1px solid #444;
}
.legend-item {
display: flex;
align-items: center;
gap: 0.5rem;
font-size: 0.8rem;
}
.legend-color {
width: 12px;
height: 12px;
border-radius: 50%;
}
.legend-edge {
width: 20px;
height: 2px;
}
.legend-edge.high-confidence {
background-color: #00ff41;
}
.legend-edge.medium-confidence {
background-color: #ff9900;
}
/* Provider Panel */
.provider-panel {
grid-area: providers;
}
.provider-list {
padding: 1.5rem;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1rem;
}
.provider-item {
background-color: #1a1a1a;
border: 1px solid #444;
padding: 1rem;
display: flex;
justify-content: space-between;
align-items: center;
}
.provider-name {
font-weight: 500;
color: #c7c7c7;
}
.provider-status {
font-size: 0.8rem;
padding: 0.25rem 0.5rem;
border-radius: 3px;
}
.provider-status.enabled {
background-color: #2c5c34;
color: #e0e0e0;
}
.provider-status.disabled {
background-color: #5c2c2c;
color: #e0e0e0;
}
.provider-stats {
font-size: 0.8rem;
color: #999;
margin-top: 0.5rem;
}
/* Footer */
.footer {
background-color: #0a0a0a;
border-top: 1px solid #444;
padding: 1rem 2rem;
text-align: center;
font-size: 0.8rem;
color: #999;
}
.footer-content {
max-width: 1400px;
margin: 0 auto;
}
.footer-separator {
margin: 0 1rem;
color: #555;
}
/* Modal */
.modal {
display: none;
position: fixed;
z-index: 1000;
left: 0;
top: 0;
width: 100%;
height: 100%;
background-color: rgba(0, 0, 0, 0.8);
}
.modal-content {
background-color: #2a2a2a;
border: 1px solid #444;
margin: 5% auto;
width: 80%;
max-width: 600px;
max-height: 80vh;
overflow-y: auto;
}
.modal-header {
background-color: #1a1a1a;
padding: 1rem;
border-bottom: 1px solid #444;
display: flex;
justify-content: space-between;
align-items: center;
}
.modal-header h3 {
color: #00ff41;
font-size: 1.1rem;
}
.modal-close {
background: transparent;
border: none;
color: #c7c7c7;
font-size: 1.2rem;
cursor: pointer;
font-family: 'Roboto Mono', monospace;
}
.modal-close:hover {
color: #ff9900;
}
.modal-body {
padding: 1.5rem;
}
.detail-row {
display: flex;
justify-content: space-between;
margin-bottom: 0.75rem;
padding-bottom: 0.25rem;
border-bottom: 1px solid #333;
}
.detail-label {
color: #999;
font-weight: 500;
}
.detail-value {
color: #c7c7c7;
}
/* Responsive Design */
@media (max-width: 768px) {
.main-content {
grid-template-columns: 1fr;
grid-template-areas:
"control"
"status"
"visualization"
"providers";
padding: 1rem;
}
.header {
padding: 1rem;
}
.header-content {
flex-direction: column;
gap: 1rem;
text-align: center;
}
.button-group {
flex-direction: column;
}
.legend {
flex-direction: column;
align-items: flex-start;
}
.provider-list {
grid-template-columns: 1fr;
}
}
/* Graph specific styles */
.vis-network {
background-color: #1a1a1a !important;
}
.vis-tooltip {
background-color: #2a2a2a !important;
border: 1px solid #444 !important;
color: #c7c7c7 !important;
font-family: 'Roboto Mono', monospace !important;
font-size: 0.8rem !important;
}
/* Loading and Error States */
.loading {
opacity: 0.6;
pointer-events: none;
}
.error {
color: #ff6b6b !important;
border-color: #ff6b6b !important;
}
.success {
color: #00ff41 !important;
}
/* Animations */
@keyframes fadeIn {
from { opacity: 0; transform: translateY(10px); }
to { opacity: 1; transform: translateY(0); }
}
.fade-in {
animation: fadeIn 0.3s ease-out;
}
/* Utility Classes */
.hidden {
display: none !important;
}
.text-center {
text-align: center;
}
.text-success {
color: #00ff41;
}
.text-warning {
color: #ff9900;
}
.text-error {
color: #ff6b6b;
}
.glow {
text-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
}
.amber {
color: #ff9900;
}

565
static/js/graph.js Normal file
View File

@ -0,0 +1,565 @@
/**
* Graph visualization module for DNSRecon
* Handles network graph rendering using vis.js
*/
class GraphManager {
constructor(containerId) {
this.container = document.getElementById(containerId);
this.network = null;
this.nodes = new vis.DataSet();
this.edges = new vis.DataSet();
this.isInitialized = false;
// Graph options for cybersecurity theme
this.options = {
nodes: {
shape: 'dot',
size: 12,
font: {
size: 11,
color: '#c7c7c7',
face: 'Roboto Mono, monospace',
background: 'rgba(26, 26, 26, 0.8)',
strokeWidth: 1,
strokeColor: '#000000'
},
borderWidth: 2,
borderColor: '#444',
shadow: {
enabled: true,
color: 'rgba(0, 0, 0, 0.3)',
size: 3,
x: 1,
y: 1
},
scaling: {
min: 8,
max: 20
}
},
edges: {
width: 2,
color: {
color: '#444',
highlight: '#00ff41',
hover: '#ff9900'
},
font: {
size: 10,
color: '#999',
face: 'Roboto Mono, monospace',
background: 'rgba(26, 26, 26, 0.8)',
strokeWidth: 1,
strokeColor: '#000000'
},
arrows: {
to: {
enabled: true,
scaleFactor: 0.8,
type: 'arrow'
}
},
smooth: {
enabled: true,
type: 'dynamic',
roundness: 0.5
},
shadow: {
enabled: true,
color: 'rgba(0, 0, 0, 0.2)',
size: 2,
x: 1,
y: 1
}
},
physics: {
enabled: true,
stabilization: {
enabled: true,
iterations: 100,
updateInterval: 25
},
barnesHut: {
gravitationalConstant: -2000,
centralGravity: 0.3,
springLength: 95,
springConstant: 0.04,
damping: 0.09,
avoidOverlap: 0.1
},
maxVelocity: 50,
minVelocity: 0.1,
solver: 'barnesHut',
timestep: 0.35,
adaptiveTimestep: true
},
interaction: {
hover: true,
hoverConnectedEdges: true,
selectConnectedEdges: true,
tooltipDelay: 200,
hideEdgesOnDrag: false,
hideNodesOnDrag: false
},
layout: {
improvedLayout: true
}
};
this.setupEventHandlers();
}
/**
* Initialize the network graph
*/
initialize() {
if (this.isInitialized) {
return;
}
try {
const data = {
nodes: this.nodes,
edges: this.edges
};
this.network = new vis.Network(this.container, data, this.options);
this.setupNetworkEvents();
this.isInitialized = true;
// Hide placeholder
const placeholder = this.container.querySelector('.graph-placeholder');
if (placeholder) {
placeholder.style.display = 'none';
}
console.log('Graph initialized successfully');
} catch (error) {
console.error('Failed to initialize graph:', error);
this.showError('Failed to initialize visualization');
}
}
/**
* Setup network event handlers
*/
setupNetworkEvents() {
if (!this.network) return;
// Node click event
this.network.on('click', (params) => {
if (params.nodes.length > 0) {
const nodeId = params.nodes[0];
this.showNodeDetails(nodeId);
}
});
// Hover events for tooltips
this.network.on('hoverNode', (params) => {
const nodeId = params.node;
const node = this.nodes.get(nodeId);
if (node) {
this.showTooltip(params.pointer.DOM, node);
}
});
this.network.on('blurNode', () => {
this.hideTooltip();
});
// Stabilization events
this.network.on('stabilizationProgress', (params) => {
const progress = params.iterations / params.total;
this.updateStabilizationProgress(progress);
});
this.network.on('stabilizationIterationsDone', () => {
this.onStabilizationComplete();
});
}
/**
* Update graph with new data
* @param {Object} graphData - Graph data from backend
*/
updateGraph(graphData) {
if (!graphData || !graphData.nodes || !graphData.edges) {
console.warn('Invalid graph data received');
return;
}
try {
// Initialize if not already done
if (!this.isInitialized) {
this.initialize();
}
// Process nodes
const processedNodes = graphData.nodes.map(node => this.processNode(node));
const processedEdges = graphData.edges.map(edge => this.processEdge(edge));
// Update datasets
this.nodes.clear();
this.edges.clear();
this.nodes.add(processedNodes);
this.edges.add(processedEdges);
// Fit the view if this is the first update or graph is small
if (processedNodes.length <= 10) {
setTimeout(() => this.fitView(), 500);
}
console.log(`Graph updated: ${processedNodes.length} nodes, ${processedEdges.length} edges`);
} catch (error) {
console.error('Failed to update graph:', error);
this.showError('Failed to update visualization');
}
}
/**
* Process node data for visualization
* @param {Object} node - Raw node data
* @returns {Object} Processed node data
*/
processNode(node) {
const processedNode = {
id: node.id,
label: this.formatNodeLabel(node.id, node.type),
title: this.createNodeTooltip(node),
color: this.getNodeColor(node.type),
size: this.getNodeSize(node.type),
borderColor: this.getNodeBorderColor(node.type),
metadata: node.metadata || {}
};
// Add type-specific styling
if (node.type === 'domain') {
processedNode.shape = 'dot';
} else if (node.type === 'ip') {
processedNode.shape = 'square';
} else if (node.type === 'certificate') {
processedNode.shape = 'diamond';
} else if (node.type === 'asn') {
processedNode.shape = 'triangle';
}
return processedNode;
}
/**
* Process edge data for visualization
* @param {Object} edge - Raw edge data
* @returns {Object} Processed edge data
*/
processEdge(edge) {
const confidence = edge.confidence_score || 0;
const processedEdge = {
id: `${edge.from}-${edge.to}`,
from: edge.from,
to: edge.to,
label: this.formatEdgeLabel(edge.label, confidence),
title: this.createEdgeTooltip(edge),
width: this.getEdgeWidth(confidence),
color: this.getEdgeColor(confidence),
dashes: confidence < 0.6 ? [5, 5] : false
};
return processedEdge;
}
/**
* Format node label for display
* @param {string} nodeId - Node identifier
* @param {string} nodeType - Node type
* @returns {string} Formatted label
*/
formatNodeLabel(nodeId, nodeType) {
// Truncate long domain names
if (nodeId.length > 20) {
return nodeId.substring(0, 17) + '...';
}
return nodeId;
}
/**
* Format edge label for display
* @param {string} relationshipType - Type of relationship
* @param {number} confidence - Confidence score
* @returns {string} Formatted label
*/
formatEdgeLabel(relationshipType, confidence) {
if (!relationshipType) return '';
const confidenceText = confidence >= 0.8 ? '●' : confidence >= 0.6 ? '◐' : '○';
return `${relationshipType} ${confidenceText}`;
}
/**
* Get node color based on type
* @param {string} nodeType - Node type
* @returns {string} Color value
*/
getNodeColor(nodeType) {
const colors = {
'domain': '#00ff41', // Green
'ip': '#ff9900', // Amber
'certificate': '#c7c7c7', // Gray
'asn': '#00aaff' // Blue
};
return colors[nodeType] || '#ffffff';
}
/**
* Get node border color based on type
* @param {string} nodeType - Node type
* @returns {string} Border color value
*/
getNodeBorderColor(nodeType) {
const borderColors = {
'domain': '#00aa2e',
'ip': '#cc7700',
'certificate': '#999999',
'asn': '#0088cc'
};
return borderColors[nodeType] || '#666666';
}
/**
* Get node size based on type
* @param {string} nodeType - Node type
* @returns {number} Node size
*/
getNodeSize(nodeType) {
const sizes = {
'domain': 12,
'ip': 14,
'certificate': 10,
'asn': 16
};
return sizes[nodeType] || 12;
}
/**
* Get edge color based on confidence
* @param {number} confidence - Confidence score
* @returns {string} Edge color
*/
getEdgeColor(confidence) {
if (confidence >= 0.8) {
return '#00ff41'; // High confidence - green
} else if (confidence >= 0.6) {
return '#ff9900'; // Medium confidence - amber
} else {
return '#666666'; // Low confidence - gray
}
}
/**
* Get edge width based on confidence
* @param {number} confidence - Confidence score
* @returns {number} Edge width
*/
getEdgeWidth(confidence) {
if (confidence >= 0.8) {
return 3;
} else if (confidence >= 0.6) {
return 2;
} else {
return 1;
}
}
/**
* Create node tooltip
* @param {Object} node - Node data
* @returns {string} HTML tooltip content
*/
createNodeTooltip(node) {
let tooltip = `<div style="font-family: 'Roboto Mono', monospace; font-size: 11px;">`;
tooltip += `<div style="color: #00ff41; font-weight: bold; margin-bottom: 4px;">${node.id}</div>`;
tooltip += `<div style="color: #999; margin-bottom: 2px;">Type: ${node.type}</div>`;
if (node.metadata && Object.keys(node.metadata).length > 0) {
tooltip += `<div style="color: #999; margin-top: 4px; border-top: 1px solid #444; padding-top: 4px;">`;
tooltip += `Click for details</div>`;
}
tooltip += `</div>`;
return tooltip;
}
/**
* Create edge tooltip
* @param {Object} edge - Edge data
* @returns {string} HTML tooltip content
*/
createEdgeTooltip(edge) {
let tooltip = `<div style="font-family: 'Roboto Mono', monospace; font-size: 11px;">`;
tooltip += `<div style="color: #00ff41; font-weight: bold; margin-bottom: 4px;">${edge.label || 'Relationship'}</div>`;
tooltip += `<div style="color: #999; margin-bottom: 2px;">Confidence: ${(edge.confidence_score * 100).toFixed(1)}%</div>`;
if (edge.source_provider) {
tooltip += `<div style="color: #999; margin-bottom: 2px;">Source: ${edge.source_provider}</div>`;
}
if (edge.discovery_timestamp) {
const date = new Date(edge.discovery_timestamp);
tooltip += `<div style="color: #666; font-size: 10px;">Discovered: ${date.toLocaleString()}</div>`;
}
tooltip += `</div>`;
return tooltip;
}
/**
* Show node details in modal
* @param {string} nodeId - Node identifier
*/
showNodeDetails(nodeId) {
const node = this.nodes.get(nodeId);
if (!node) return;
// Trigger custom event for main application to handle
const event = new CustomEvent('nodeSelected', {
detail: { nodeId, node }
});
document.dispatchEvent(event);
}
/**
* Show tooltip
* @param {Object} position - Mouse position
* @param {Object} node - Node data
*/
showTooltip(position, node) {
// Tooltip is handled by vis.js automatically
// This method is for custom tooltip implementation if needed
}
/**
* Hide tooltip
*/
hideTooltip() {
// Tooltip hiding is handled by vis.js automatically
}
/**
* Update stabilization progress
* @param {number} progress - Progress value (0-1)
*/
updateStabilizationProgress(progress) {
// Could show a progress indicator if needed
console.log(`Graph stabilization: ${(progress * 100).toFixed(1)}%`);
}
/**
* Handle stabilization completion
*/
onStabilizationComplete() {
console.log('Graph stabilization complete');
}
/**
* Fit the view to show all nodes
*/
fitView() {
if (this.network) {
this.network.fit({
animation: {
duration: 1000,
easingFunction: 'easeInOutQuad'
}
});
}
}
/**
* Reset the view to initial state
*/
resetView() {
if (this.network) {
this.network.moveTo({
position: { x: 0, y: 0 },
scale: 1,
animation: {
duration: 1000,
easingFunction: 'easeInOutQuad'
}
});
}
}
/**
* Clear the graph
*/
clear() {
this.nodes.clear();
this.edges.clear();
// Show placeholder
const placeholder = this.container.querySelector('.graph-placeholder');
if (placeholder) {
placeholder.style.display = 'flex';
}
}
/**
* Show error message
* @param {string} message - Error message
*/
showError(message) {
const placeholder = this.container.querySelector('.graph-placeholder .placeholder-text');
if (placeholder) {
placeholder.textContent = `Error: ${message}`;
placeholder.style.color = '#ff6b6b';
}
}
/**
* Setup control event handlers
*/
setupEventHandlers() {
// Reset view button
document.addEventListener('DOMContentLoaded', () => {
const resetBtn = document.getElementById('reset-view');
if (resetBtn) {
resetBtn.addEventListener('click', () => this.resetView());
}
const fitBtn = document.getElementById('fit-view');
if (fitBtn) {
fitBtn.addEventListener('click', () => this.fitView());
}
});
}
/**
* Get network statistics
* @returns {Object} Statistics object
*/
getStatistics() {
return {
nodeCount: this.nodes.length,
edgeCount: this.edges.length,
//isStabilized: this.network ? this.network.isStabilized() : false
};
}
/**
* Export graph as image (if needed for future implementation)
* @param {string} format - Image format ('png', 'jpeg')
* @returns {string} Data URL of the image
*/
exportAsImage(format = 'png') {
if (!this.network) return null;
// This would require additional vis.js functionality
// Placeholder for future implementation
console.log('Image export not yet implemented');
return null;
}
}
// Export for use in main.js
window.GraphManager = GraphManager;

969
static/js/main.js Normal file
View File

@ -0,0 +1,969 @@
/**
* Main application logic for DNSRecon web interface
* Handles UI interactions, API communication, and data flow
* DEBUG VERSION WITH EXTRA LOGGING
*/
class DNSReconApp {
constructor() {
console.log('DNSReconApp constructor called');
this.graphManager = null;
this.scanStatus = 'idle';
this.pollInterval = null;
this.currentSessionId = null;
// UI Elements
this.elements = {};
// Application state
this.isScanning = false;
this.lastGraphUpdate = null;
this.init();
}
/**
* Initialize the application
*/
init() {
console.log('DNSReconApp init called');
document.addEventListener('DOMContentLoaded', () => {
console.log('DOM loaded, initializing application...');
try {
this.initializeElements();
this.setupEventHandlers();
this.initializeGraph();
this.updateStatus();
this.loadProviders();
console.log('DNSRecon application initialized successfully');
} catch (error) {
console.error('Failed to initialize DNSRecon application:', error);
this.showError(`Initialization failed: ${error.message}`);
}
});
}
/**
* Initialize DOM element references
*/
initializeElements() {
console.log('Initializing DOM elements...');
this.elements = {
// Form elements
targetDomain: document.getElementById('target-domain'),
maxDepth: document.getElementById('max-depth'),
startScan: document.getElementById('start-scan'),
stopScan: document.getElementById('stop-scan'),
exportResults: document.getElementById('export-results'),
// Status elements
scanStatus: document.getElementById('scan-status'),
targetDisplay: document.getElementById('target-display'),
depthDisplay: document.getElementById('depth-display'),
progressDisplay: document.getElementById('progress-display'),
indicatorsDisplay: document.getElementById('indicators-display'),
relationshipsDisplay: document.getElementById('relationships-display'),
progressFill: document.getElementById('progress-fill'),
// Provider elements
providerList: document.getElementById('provider-list'),
// Modal elements
nodeModal: document.getElementById('node-modal'),
modalTitle: document.getElementById('modal-title'),
modalDetails: document.getElementById('modal-details'),
modalClose: document.getElementById('modal-close'),
// Other elements
sessionId: document.getElementById('session-id'),
connectionStatus: document.getElementById('connection-status')
};
// Verify critical elements exist
const requiredElements = ['targetDomain', 'startScan', 'scanStatus'];
for (const elementName of requiredElements) {
if (!this.elements[elementName]) {
throw new Error(`Required element '${elementName}' not found in DOM`);
}
}
console.log('DOM elements initialized successfully');
this.createMessageContainer();
}
/**
* Create a message container for showing user feedback
*/
createMessageContainer() {
// Check if message container already exists
let messageContainer = document.getElementById('message-container');
if (!messageContainer) {
messageContainer = document.createElement('div');
messageContainer.id = 'message-container';
messageContainer.className = 'message-container';
messageContainer.style.cssText = `
position: fixed;
top: 20px;
right: 20px;
z-index: 1000;
max-width: 400px;
`;
document.body.appendChild(messageContainer);
console.log('Message container created');
}
}
/**
* Setup event handlers
*/
setupEventHandlers() {
console.log('Setting up event handlers...');
try {
// Form interactions
this.elements.startScan.addEventListener('click', (e) => {
console.log('Start scan button clicked');
e.preventDefault();
this.startScan();
});
this.elements.stopScan.addEventListener('click', (e) => {
console.log('Stop scan button clicked');
e.preventDefault();
this.stopScan();
});
this.elements.exportResults.addEventListener('click', (e) => {
console.log('Export results button clicked');
e.preventDefault();
this.exportResults();
});
// Enter key support for target domain input
this.elements.targetDomain.addEventListener('keypress', (e) => {
if (e.key === 'Enter' && !this.isScanning) {
console.log('Enter key pressed in domain input');
this.startScan();
}
});
// Modal interactions
if (this.elements.modalClose) {
this.elements.modalClose.addEventListener('click', () => this.hideModal());
}
if (this.elements.nodeModal) {
this.elements.nodeModal.addEventListener('click', (e) => {
if (e.target === this.elements.nodeModal) {
this.hideModal();
}
});
}
// Custom events
document.addEventListener('nodeSelected', (e) => {
this.showNodeModal(e.detail.nodeId, e.detail.node);
});
// Keyboard shortcuts
document.addEventListener('keydown', (e) => {
if (e.key === 'Escape') {
this.hideModal();
}
});
// Window events
window.addEventListener('beforeunload', () => {
if (this.isScanning) {
return 'A scan is currently in progress. Are you sure you want to leave?';
}
});
console.log('Event handlers set up successfully');
} catch (error) {
console.error('Failed to setup event handlers:', error);
throw error;
}
}
/**
* Initialize graph visualization
*/
initializeGraph() {
try {
console.log('Initializing graph manager...');
this.graphManager = new GraphManager('network-graph');
console.log('Graph manager initialized successfully');
} catch (error) {
console.error('Failed to initialize graph manager:', error);
this.showError('Failed to initialize graph visualization');
}
}
/**
* Start a reconnaissance scan
*/
async startScan() {
console.log('=== STARTING SCAN ===');
try {
const targetDomain = this.elements.targetDomain.value.trim();
const maxDepth = parseInt(this.elements.maxDepth.value);
console.log(`Target domain: "${targetDomain}", Max depth: ${maxDepth}`);
// Validation
if (!targetDomain) {
console.log('Validation failed: empty domain');
this.showError('Please enter a target domain');
this.elements.targetDomain.focus();
return;
}
if (!this.isValidDomain(targetDomain)) {
console.log(`Validation failed: invalid domain format for "${targetDomain}"`);
this.showError('Please enter a valid domain name (e.g., example.com)');
this.elements.targetDomain.focus();
return;
}
console.log('Validation passed, setting UI state to scanning...');
this.setUIState('scanning');
this.showInfo('Starting reconnaissance scan...');
console.log('Making API call to start scan...');
const requestData = {
target_domain: targetDomain,
max_depth: maxDepth
};
console.log('Request data:', requestData);
const response = await this.apiCall('/api/scan/start', 'POST', requestData);
console.log('API response received:', response);
if (response.success) {
this.currentSessionId = response.scan_id;
console.log('Starting polling with session ID:', this.currentSessionId);
this.startPolling();
this.showSuccess('Reconnaissance scan started successfully');
// Clear previous graph
this.graphManager.clear();
console.log(`Scan started for ${targetDomain} with depth ${maxDepth}`);
// Force an immediate status update
console.log('Forcing immediate status update...');
setTimeout(() => {
this.updateStatus();
this.updateGraph();
}, 100);
} else {
throw new Error(response.error || 'Failed to start scan');
}
} catch (error) {
console.error('Failed to start scan:', error);
this.showError(`Failed to start scan: ${error.message}`);
this.setUIState('idle');
}
}
/**
* Stop the current scan
*/
async stopScan() {
try {
console.log('Stopping scan...');
const response = await this.apiCall('/api/scan/stop', 'POST');
if (response.success) {
this.showSuccess('Scan stop requested');
console.log('Scan stop requested');
} else {
throw new Error(response.error || 'Failed to stop scan');
}
} catch (error) {
console.error('Failed to stop scan:', error);
this.showError(`Failed to stop scan: ${error.message}`);
}
}
/**
* Export scan results
*/
async exportResults() {
try {
console.log('Exporting results...');
// Create a temporary link to trigger download
const link = document.createElement('a');
link.href = '/api/export';
link.download = ''; // Let server determine filename
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
this.showSuccess('Results export initiated');
console.log('Results export initiated');
} catch (error) {
console.error('Failed to export results:', error);
this.showError(`Failed to export results: ${error.message}`);
}
}
/**
* Start polling for scan updates
*/
startPolling() {
console.log('=== STARTING POLLING ===');
if (this.pollInterval) {
console.log('Clearing existing poll interval');
clearInterval(this.pollInterval);
}
this.pollInterval = setInterval(() => {
console.log('--- Polling tick ---');
this.updateStatus();
this.updateGraph();
}, 1000); // Poll every 1 second for debugging
console.log('Polling started with 1 second interval');
}
/**
* Stop polling for updates
*/
stopPolling() {
console.log('=== STOPPING POLLING ===');
if (this.pollInterval) {
clearInterval(this.pollInterval);
this.pollInterval = null;
}
}
/**
* Update scan status from server
*/
async updateStatus() {
try {
console.log('Updating status...');
const response = await this.apiCall('/api/scan/status');
console.log('Status response:', response);
if (response.success) {
const status = response.status;
console.log('Current scan status:', status.status);
console.log('Current progress:', status.progress_percentage + '%');
console.log('Graph stats:', status.graph_statistics);
this.updateStatusDisplay(status);
// Handle status changes
if (status.status !== this.scanStatus) {
console.log(`*** STATUS CHANGED: ${this.scanStatus} -> ${status.status} ***`);
this.handleStatusChange(status.status);
}
this.scanStatus = status.status;
} else {
console.error('Status update failed:', response);
}
} catch (error) {
console.error('Failed to update status:', error);
this.showConnectionError();
}
}
/**
* Update graph from server
*/
async updateGraph() {
try {
console.log('Updating graph...');
const response = await this.apiCall('/api/graph');
console.log('Graph response:', response);
if (response.success) {
const graphData = response.graph;
console.log('Graph data received:');
console.log('- Nodes:', graphData.nodes ? graphData.nodes.length : 0);
console.log('- Edges:', graphData.edges ? graphData.edges.length : 0);
if (graphData.nodes) {
graphData.nodes.forEach(node => {
console.log(` Node: ${node.id} (${node.type})`);
});
}
if (graphData.edges) {
graphData.edges.forEach(edge => {
console.log(` Edge: ${edge.from} -> ${edge.to} (${edge.label})`);
});
}
// Only update if data has changed
if (this.hasGraphChanged(graphData)) {
console.log('*** GRAPH DATA CHANGED - UPDATING VISUALIZATION ***');
this.graphManager.updateGraph(graphData);
this.lastGraphUpdate = Date.now();
// Update relationship count in status
const edgeCount = graphData.edges ? graphData.edges.length : 0;
if (this.elements.relationshipsDisplay) {
this.elements.relationshipsDisplay.textContent = edgeCount;
}
} else {
console.log('Graph data unchanged, skipping update');
}
} else {
console.error('Graph update failed:', response);
}
} catch (error) {
console.error('Failed to update graph:', error);
// Don't show error for graph updates to avoid spam
}
}
/**
* Update status display elements
* @param {Object} status - Status object from server
*/
updateStatusDisplay(status) {
try {
console.log('Updating status display...');
// Update status text
if (this.elements.scanStatus) {
this.elements.scanStatus.textContent = this.formatStatus(status.status);
console.log('Updated status display:', status.status);
}
if (this.elements.targetDisplay) {
this.elements.targetDisplay.textContent = status.target_domain || 'None';
}
if (this.elements.depthDisplay) {
this.elements.depthDisplay.textContent = `${status.current_depth}/${status.max_depth}`;
}
if (this.elements.progressDisplay) {
this.elements.progressDisplay.textContent = `${status.progress_percentage.toFixed(1)}%`;
}
if (this.elements.indicatorsDisplay) {
this.elements.indicatorsDisplay.textContent = status.indicators_processed || 0;
}
// Update progress bar
if (this.elements.progressFill) {
this.elements.progressFill.style.width = `${status.progress_percentage}%`;
}
// Update session ID
if (this.currentSessionId && this.elements.sessionId) {
this.elements.sessionId.textContent = `Session: ${this.currentSessionId}`;
}
console.log('Status display updated successfully');
} catch (error) {
console.error('Error updating status display:', error);
}
}
/**
* Handle status changes
* @param {string} newStatus - New scan status
*/
handleStatusChange(newStatus) {
console.log(`=== STATUS CHANGE: ${this.scanStatus} -> ${newStatus} ===`);
switch (newStatus) {
case 'running':
this.setUIState('scanning');
this.showSuccess('Scan is running');
break;
case 'completed':
this.setUIState('completed');
this.stopPolling();
this.showSuccess('Scan completed successfully');
// Force a final graph update
console.log('Scan completed - forcing final graph update');
setTimeout(() => this.updateGraph(), 100);
break;
case 'failed':
this.setUIState('failed');
this.stopPolling();
this.showError('Scan failed');
break;
case 'stopped':
this.setUIState('stopped');
this.stopPolling();
this.showSuccess('Scan stopped');
break;
case 'idle':
this.setUIState('idle');
this.stopPolling();
break;
}
}
/**
* Set UI state based on scan status
* @param {string} state - UI state
*/
setUIState(state) {
console.log(`Setting UI state to: ${state}`);
switch (state) {
case 'scanning':
this.isScanning = true;
if (this.elements.startScan) this.elements.startScan.disabled = true;
if (this.elements.stopScan) this.elements.stopScan.disabled = false;
if (this.elements.targetDomain) this.elements.targetDomain.disabled = true;
if (this.elements.maxDepth) this.elements.maxDepth.disabled = true;
break;
case 'idle':
case 'completed':
case 'failed':
case 'stopped':
this.isScanning = false;
if (this.elements.startScan) this.elements.startScan.disabled = false;
if (this.elements.stopScan) this.elements.stopScan.disabled = true;
if (this.elements.targetDomain) this.elements.targetDomain.disabled = false;
if (this.elements.maxDepth) this.elements.maxDepth.disabled = false;
break;
}
}
/**
* Load provider information
*/
async loadProviders() {
try {
console.log('Loading providers...');
const response = await this.apiCall('/api/providers');
if (response.success) {
this.updateProviderDisplay(response.providers);
console.log('Providers loaded successfully');
}
} catch (error) {
console.error('Failed to load providers:', error);
}
}
/**
* Update provider display
* @param {Object} providers - Provider information
*/
updateProviderDisplay(providers) {
if (!this.elements.providerList) return;
this.elements.providerList.innerHTML = '';
for (const [name, info] of Object.entries(providers)) {
const providerItem = document.createElement('div');
providerItem.className = 'provider-item';
const status = info.enabled ? 'enabled' : 'disabled';
const statusClass = info.enabled ? 'enabled' : 'disabled';
providerItem.innerHTML = `
<div>
<div class="provider-name">${name.toUpperCase()}</div>
<div class="provider-stats">
Requests: ${info.statistics.total_requests || 0} |
Success Rate: ${(info.statistics.success_rate || 0).toFixed(1)}%
</div>
</div>
<div class="provider-status ${statusClass}">${status}</div>
`;
this.elements.providerList.appendChild(providerItem);
}
}
/**
* Show node details modal
* @param {string} nodeId - Node identifier
* @param {Object} node - Node data
*/
showNodeModal(nodeId, node) {
if (!this.elements.nodeModal) return;
if (this.elements.modalTitle) {
this.elements.modalTitle.textContent = `Node Details: ${nodeId}`;
}
let detailsHtml = '';
detailsHtml += `<div class="detail-row"><span class="detail-label">Identifier:</span><span class="detail-value">${nodeId}</span></div>`;
detailsHtml += `<div class="detail-row"><span class="detail-label">Type:</span><span class="detail-value">${node.metadata.type || 'Unknown'}</span></div>`;
if (node.metadata) {
for (const [key, value] of Object.entries(node.metadata)) {
if (key !== 'type') {
detailsHtml += `<div class="detail-row"><span class="detail-label">${this.formatLabel(key)}:</span><span class="detail-value">${this.formatValue(value)}</span></div>`;
}
}
}
if (this.elements.modalDetails) {
this.elements.modalDetails.innerHTML = detailsHtml;
}
this.elements.nodeModal.style.display = 'block';
}
/**
* Hide the modal
*/
hideModal() {
if (this.elements.nodeModal) {
this.elements.nodeModal.style.display = 'none';
}
}
/**
* Check if graph data has changed
* @param {Object} graphData - New graph data
* @returns {boolean} True if data has changed
*/
hasGraphChanged(graphData) {
// Simple check based on node and edge counts
const currentStats = this.graphManager.getStatistics();
const newNodeCount = graphData.nodes ? graphData.nodes.length : 0;
const newEdgeCount = graphData.edges ? graphData.edges.length : 0;
const changed = currentStats.nodeCount !== newNodeCount || currentStats.edgeCount !== newEdgeCount;
console.log(`Graph change check: Current(${currentStats.nodeCount}n, ${currentStats.edgeCount}e) vs New(${newNodeCount}n, ${newEdgeCount}e) = ${changed}`);
return changed;
}
/**
* Make API call to server
* @param {string} endpoint - API endpoint
* @param {string} method - HTTP method
* @param {Object} data - Request data
* @returns {Promise<Object>} Response data
*/
async apiCall(endpoint, method = 'GET', data = null) {
console.log(`Making API call: ${method} ${endpoint}`, data ? data : '(no data)');
try {
const options = {
method: method,
headers: {
'Content-Type': 'application/json'
}
};
if (data && method !== 'GET') {
options.body = JSON.stringify(data);
console.log('Request body:', options.body);
}
console.log('Fetch options:', options);
const response = await fetch(endpoint, options);
console.log('Response status:', response.status, response.statusText);
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const result = await response.json();
console.log('Response data:', result);
return result;
} catch (error) {
console.error(`API call failed for ${method} ${endpoint}:`, error);
throw error;
}
}
/**
* Validate domain name - improved validation
* @param {string} domain - Domain to validate
* @returns {boolean} True if valid
*/
isValidDomain(domain) {
console.log(`Validating domain: "${domain}"`);
// Basic checks
if (!domain || typeof domain !== 'string') {
console.log('Validation failed: empty or non-string domain');
return false;
}
if (domain.length > 253) {
console.log('Validation failed: domain too long');
return false;
}
if (domain.startsWith('.') || domain.endsWith('.')) {
console.log('Validation failed: domain starts or ends with dot');
return false;
}
if (domain.includes('..')) {
console.log('Validation failed: domain contains double dots');
return false;
}
// Split into parts and validate each
const parts = domain.split('.');
if (parts.length < 2) {
console.log('Validation failed: domain has less than 2 parts');
return false;
}
// Check each part
for (const part of parts) {
if (!part || part.length > 63) {
console.log(`Validation failed: invalid part "${part}"`);
return false;
}
if (part.startsWith('-') || part.endsWith('-')) {
console.log(`Validation failed: part "${part}" starts or ends with hyphen`);
return false;
}
if (!/^[a-zA-Z0-9-]+$/.test(part)) {
console.log(`Validation failed: part "${part}" contains invalid characters`);
return false;
}
}
// Check TLD (last part) is alphabetic
const tld = parts[parts.length - 1];
if (!/^[a-zA-Z]{2,}$/.test(tld)) {
console.log(`Validation failed: invalid TLD "${tld}"`);
return false;
}
console.log('Domain validation passed');
return true;
}
/**
* Format status text for display
* @param {string} status - Raw status
* @returns {string} Formatted status
*/
formatStatus(status) {
const statusMap = {
'idle': 'Idle',
'running': 'Running',
'completed': 'Completed',
'failed': 'Failed',
'stopped': 'Stopped'
};
return statusMap[status] || status;
}
/**
* Format label for display
* @param {string} label - Raw label
* @returns {string} Formatted label
*/
formatLabel(label) {
return label.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase());
}
/**
* Format value for display
* @param {*} value - Raw value
* @returns {string} Formatted value
*/
formatValue(value) {
if (Array.isArray(value)) {
return value.join(', ');
} else if (typeof value === 'object') {
return JSON.stringify(value, null, 2);
} else {
return String(value);
}
}
/**
* Show success message
* @param {string} message - Success message
*/
showSuccess(message) {
this.showMessage(message, 'success');
}
/**
* Show info message
* @param {string} message - Info message
*/
showInfo(message) {
this.showMessage(message, 'info');
}
/**
* Show error message
* @param {string} message - Error message
*/
showError(message) {
this.showMessage(message, 'error');
}
/**
* Show connection error
*/
showConnectionError() {
if (this.elements.connectionStatus) {
this.elements.connectionStatus.style.backgroundColor = '#ff6b6b';
}
const statusText = this.elements.connectionStatus?.parentElement?.querySelector('.status-text');
if (statusText) {
statusText.textContent = 'Connection Error';
}
}
/**
* Show message with visual feedback
* @param {string} message - Message text
* @param {string} type - Message type (success, error, warning, info)
*/
showMessage(message, type = 'info') {
console.log(`${type.toUpperCase()}: ${message}`);
// Create message element
const messageElement = document.createElement('div');
messageElement.className = `message-toast message-${type}`;
messageElement.style.cssText = `
background: ${this.getMessageColor(type)};
color: #fff;
padding: 12px 20px;
margin-bottom: 10px;
border-radius: 4px;
font-family: 'Roboto Mono', monospace;
font-size: 0.9rem;
box-shadow: 0 4px 6px rgba(0,0,0,0.3);
border-left: 4px solid ${this.getMessageBorderColor(type)};
animation: slideInRight 0.3s ease-out;
`;
messageElement.innerHTML = `
<div style="display: flex; justify-content: space-between; align-items: center;">
<span>${message}</span>
<button onclick="this.parentElement.parentElement.remove()"
style="background: none; border: none; color: #fff; cursor: pointer; font-size: 16px; margin-left: 10px;">×</button>
</div>
`;
// Add to container
const container = document.getElementById('message-container');
if (container) {
container.appendChild(messageElement);
// Auto-remove after delay
setTimeout(() => {
if (messageElement.parentNode) {
messageElement.style.animation = 'slideOutRight 0.3s ease-out';
setTimeout(() => {
if (messageElement.parentNode) {
messageElement.remove();
}
}, 300);
}
}, type === 'error' ? 8000 : 5000); // Errors stay longer
}
// Update connection status to show activity
if (type === 'success' && this.elements.connectionStatus) {
this.elements.connectionStatus.style.backgroundColor = '#00ff41';
setTimeout(() => {
if (this.elements.connectionStatus) {
this.elements.connectionStatus.style.backgroundColor = '#00ff41';
}
}, 2000);
}
}
/**
* Get message background color based on type
* @param {string} type - Message type
* @returns {string} CSS color
*/
getMessageColor(type) {
const colors = {
'success': '#2c5c34',
'error': '#5c2c2c',
'warning': '#5c4c2c',
'info': '#2c3e5c'
};
return colors[type] || colors.info;
}
/**
* Get message border color based on type
* @param {string} type - Message type
* @returns {string} CSS color
*/
getMessageBorderColor(type) {
const colors = {
'success': '#00ff41',
'error': '#ff6b6b',
'warning': '#ff9900',
'info': '#00aaff'
};
return colors[type] || colors.info;
}
}
// Add CSS animations for message toasts
const style = document.createElement('style');
style.textContent = `
@keyframes slideInRight {
from {
transform: translateX(100%);
opacity: 0;
}
to {
transform: translateX(0);
opacity: 1;
}
}
@keyframes slideOutRight {
from {
transform: translateX(0);
opacity: 1;
}
to {
transform: translateX(100%);
opacity: 0;
}
}
.message-container {
pointer-events: auto;
}
.message-toast {
pointer-events: auto;
}
`;
document.head.appendChild(style);
// Initialize application when page loads
console.log('Creating DNSReconApp instance...');
const app = new DNSReconApp();

1016
static/script.js
View File

File diff suppressed because it is too large Load Diff

688
static/style.css
View File

@ -1,688 +0,0 @@
/*
███████╗██████╗ ███████╗ ██████╗████████╗ ██████╗ ██████╗ ██╗ ██╗███████╗
██╔════╝██╔══██╗██╔════╝██╔═══██╗╚══██╔══╝ ██╔═══██╗██╔═══██╗╚██╗██╔╝██╔════╝
███████╗██████╔╝█████╗ ██║ ██║ ██║ ██║ ██║██║ ██║ ╚███╔╝ ███████╗
╚════██║██╔══██╗██╔══╝ ██║ ██║ ██║ ██║ ██║██║ ██║ ██╔██╗ ╚════██║
███████║██║ ██║███████╗╚██████╔╝ ██║ ╚██████╔╝╚██████╔╝██╔╝ ██╗███████║
╚══════╝╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝
TACTICAL THEME - DNS RECONNAISSANCE INTERFACE
STYLE OVERRIDE
*/
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Roboto Mono', 'Lucida Console', Monaco, monospace;
line-height: 1.6;
color: #c7c7c7; /* Light grey for readability */
/* Dark, textured background for a gritty feel */
background-color: #1a1a1a;
background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='4' height='4' viewBox='0 0 4 4'%3E%3Cpath fill='%23333333' fill-opacity='0.4' d='M1 3h1v1H1V3zm2-2h1v1H3V1z'%3E%3C/path%3E%3C/svg%3E");
min-height: 100vh;
}
.container {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
header {
text-align: center;
color: #e0e0e0;
margin-bottom: 40px;
border-bottom: 1px solid #444;
padding-bottom: 20px;
}
header h1 {
font-family: 'Special Elite', 'Courier New', monospace; /* Stencil / Typewriter font */
font-size: 2.8rem;
color: #00ff41; /* Night-vision green */
text-shadow: 0 0 5px rgba(0, 255, 65, 0.5);
margin-bottom: 10px;
letter-spacing: 2px;
}
header p {
font-size: 1.1rem;
color: #a0a0a0;
}
.scan-form, .progress-section, .results-section {
background: #2a2a2a; /* Dark charcoal */
border-radius: 4px; /* Sharper edges */
border: 1px solid #444;
box-shadow: inset 0 0 15px rgba(0,0,0,0.5);
padding: 30px;
margin-bottom: 25px;
}
.scan-form h2, .progress-section h2, .results-section h2 {
margin-bottom: 20px;
color: #e0e0e0;
border-bottom: 1px solid #555;
padding-bottom: 10px;
text-transform: uppercase; /* Military style */
letter-spacing: 1px;
}
.form-group {
margin-bottom: 20px;
}
.form-group label {
display: block;
margin-bottom: 8px;
font-weight: 600;
color: #b0b0b0;
text-transform: uppercase;
font-size: 0.9rem;
}
.form-group input, .form-group select {
width: 100%;
padding: 12px;
background: #1a1a1a;
border: 1px solid #555;
border-radius: 2px;
font-size: 16px;
color: #00ff41; /* Green text for input fields */
font-family: 'Roboto Mono', monospace;
transition: all 0.2s ease-in-out;
}
.form-group input:focus, .form-group select:focus {
outline: none;
border-color: #ff9900; /* Amber focus color */
box-shadow: 0 0 5px rgba(255, 153, 0, 0.5);
}
.api-keys {
background: rgba(0,0,0,0.3);
padding: 20px;
border-radius: 4px;
border: 1px solid #444;
margin: 20px 0;
}
.api-keys h3 {
margin-bottom: 15px;
color: #c7c7c7;
}
.btn-primary, .btn-secondary {
padding: 12px 24px;
border: 1px solid #666;
border-radius: 2px;
font-size: 16px;
font-weight: 600;
cursor: pointer;
transition: all 0.2s ease-in-out;
margin-right: 10px;
margin-bottom: 10px;
text-transform: uppercase;
letter-spacing: 1px;
}
.btn-primary {
background: #2c5c34; /* Dark military green */
color: #e0e0e0;
border-color: #3b7b46;
}
.btn-primary:hover {
background: #3b7b46; /* Lighter green on hover */
color: #fff;
border-color: #4cae5c;
}
.btn-secondary {
background: #4a4a4a; /* Dark grey */
color: #c7c7c7;
border-color: #666;
}
.btn-secondary:hover {
background: #5a5a5a;
}
.btn-secondary.active {
background: #6a4f2a; /* Amber/Brown for active state */
color: #fff;
border-color: #ff9900;
}
.progress-bar {
width: 100%;
height: 20px;
background: #1a1a1a;
border: 1px solid #555;
border-radius: 2px;
overflow: hidden;
margin-bottom: 15px;
padding: 2px;
}
.progress-fill {
height: 100%;
background: #ff9900; /* Solid amber progress fill */
width: 0%;
transition: width 0.3s ease;
border-radius: 0;
}
#progressMessage {
font-weight: 500;
color: #a0a0a0;
margin-bottom: 20px;
}
.scan-controls {
text-align: center;
}
.results-controls {
margin-bottom: 20px;
text-align: center;
}
.report-container {
background: #0a0a0a; /* Near-black terminal background */
border-radius: 4px;
border: 1px solid #333;
padding: 20px;
max-height: 600px;
overflow-y: auto;
box-shadow: inset 0 0 10px #000;
}
#reportContent {
color: #00ff41; /* Classic terminal green */
font-family: 'Courier New', monospace;
font-size: 14px;
line-height: 1.4;
white-space: pre-wrap;
word-wrap: break-word;
}
.hostname-list, .ip-list {
display: flex;
flex-wrap: wrap;
gap: 5px;
}
.discovery-item {
background: #2a2a2a;
color: #00ff41;
padding: 2px 6px;
border-radius: 2px;
font-family: 'Courier New', monospace;
font-size: 0.8rem;
border: 1px solid #444;
}
.activity-list {
max-height: 150px;
overflow-y: auto;
}
.activity-item {
color: #a0a0a0;
font-family: 'Courier New', monospace;
font-size: 0.8rem;
padding: 2px 0;
border-bottom: 1px solid #333;
}
.activity-item:last-child {
border-bottom: none;
}
/* Live Discoveries Base Styling */
.live-discoveries {
background: rgba(0, 20, 0, 0.6);
border: 1px solid #00ff41;
border-radius: 4px;
padding: 20px;
margin-top: 20px;
}
.live-discoveries h3 {
color: #00ff41;
margin-bottom: 15px;
text-transform: uppercase;
letter-spacing: 1px;
}
/* Enhanced styling for live discoveries when shown in results view */
.results-section .live-discoveries {
background: rgba(0, 40, 0, 0.8);
border: 2px solid #00ff41;
border-radius: 4px;
padding: 20px;
margin-bottom: 25px;
box-shadow: 0 0 10px rgba(0, 255, 65, 0.3);
}
.results-section .live-discoveries h3 {
color: #00ff41;
text-shadow: 0 0 3px rgba(0, 255, 65, 0.5);
}
/* Ensure the progress section flows nicely when showing both progress and results */
.progress-section.with-results {
margin-bottom: 0;
border-bottom: none;
}
.results-section.with-live-data {
border-top: 1px solid #444;
padding-top: 20px;
}
/* Better spacing for the combined view */
.progress-section + .results-section {
margin-top: 0;
}
/* Hide specific progress elements while keeping the section visible */
.progress-section .progress-bar.hidden,
.progress-section #progressMessage.hidden,
.progress-section .scan-controls.hidden {
display: none !important;
}
.stats-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
gap: 15px;
margin-bottom: 20px;
}
.stat-item {
display: flex;
justify-content: space-between;
align-items: center;
padding: 8px 12px;
background: rgba(0, 0, 0, 0.5);
border: 1px solid #333;
border-radius: 2px;
}
.stat-label {
color: #a0a0a0;
font-size: 0.9rem;
}
.stat-value {
color: #00ff41;
font-weight: bold;
font-family: 'Courier New', monospace;
transition: background-color 0.3s ease;
}
/* Animation for final stats highlight */
@keyframes finalHighlight {
0% { background-color: #ff9900; }
100% { background-color: transparent; }
}
.stat-value.final {
animation: finalHighlight 2s ease-in-out;
}
.discoveries-list {
margin-top: 20px;
}
.discoveries-list h4 {
color: #ff9900;
margin-bottom: 15px;
border-bottom: 1px solid #444;
padding-bottom: 5px;
}
.discovery-section {
margin-bottom: 15px;
padding: 10px;
background: rgba(0, 0, 0, 0.3);
border: 1px solid #333;
border-radius: 2px;
}
.discovery-section strong {
color: #c7c7c7;
display: block;
margin-bottom: 8px;
font-size: 0.9rem;
}
/* Tactical loading spinner */
.loading {
display: inline-block;
width: 20px;
height: 20px;
border: 3px solid rgba(199, 199, 199, 0.3);
border-radius: 50%;
border-top-color: #00ff41; /* Night-vision green spinner */
animation: spin 1s linear infinite;
}
@keyframes spin {
to { transform: rotate(360deg); }
}
/* Responsive design adjustments */
@media (max-width: 768px) {
.container {
padding: 10px;
}
header h1 {
font-size: 2.2rem;
}
.scan-form, .progress-section, .results-section {
padding: 20px;
}
.btn-primary, .btn-secondary {
width: 100%;
margin-right: 0;
}
.results-controls {
display: flex;
flex-wrap: wrap;
justify-content: center;
}
.results-controls button {
flex: 1;
min-width: 120px;
}
.stats-grid {
grid-template-columns: repeat(2, 1fr);
gap: 10px;
}
.stat-item {
padding: 6px 8px;
}
.stat-label, .stat-value {
font-size: 0.8rem;
}
.hostname-list, .ip-list {
flex-direction: column;
align-items: flex-start;
}
/* Responsive adjustments for the combined view */
.results-section .live-discoveries {
padding: 15px;
margin-bottom: 15px;
}
.results-section .live-discoveries .stats-grid {
grid-template-columns: repeat(2, 1fr);
gap: 10px;
}
}
/* Add this CSS to the existing style.css file */
/* Graph Section */
.graph-section {
background: #2a2a2a;
border-radius: 4px;
border: 1px solid #444;
box-shadow: inset 0 0 15px rgba(0,0,0,0.5);
padding: 30px;
margin-bottom: 25px;
}
.graph-controls {
margin-bottom: 20px;
text-align: center;
}
.graph-controls button {
margin: 0 5px;
}
.graph-legend {
display: flex;
justify-content: space-between;
margin-bottom: 20px;
padding: 15px;
background: rgba(0,0,0,0.3);
border-radius: 4px;
border: 1px solid #444;
flex-wrap: wrap;
}
.graph-legend h4, .graph-legend h5 {
color: #e0e0e0;
margin-bottom: 10px;
text-transform: uppercase;
letter-spacing: 1px;
}
.legend-items, .legend-methods {
display: flex;
flex-direction: column;
gap: 8px;
}
.legend-item {
display: flex;
align-items: center;
gap: 10px;
font-size: 0.9rem;
color: #c7c7c7;
}
.legend-color {
width: 16px;
height: 16px;
border-radius: 50%;
border: 1px solid #666;
}
.method-item {
font-size: 0.9rem;
color: #a0a0a0;
margin-bottom: 4px;
}
.graph-container {
position: relative;
width: 100%;
height: 600px;
background: #0a0a0a;
border-radius: 4px;
border: 1px solid #333;
margin-bottom: 20px;
overflow: hidden;
}
#discoveryGraph {
width: 100%;
height: 100%;
cursor: grab;
}
#discoveryGraph:active {
cursor: grabbing;
}
.graph-tooltip {
position: absolute;
background: rgba(0, 0, 0, 0.9);
color: #00ff41;
padding: 10px;
border-radius: 4px;
border: 1px solid #00ff41;
font-family: 'Courier New', monospace;
font-size: 0.8rem;
pointer-events: none;
opacity: 0;
transition: opacity 0.2s ease;
z-index: 1000;
max-width: 300px;
line-height: 1.4;
}
.graph-tooltip.visible {
opacity: 1;
}
.graph-info {
display: flex;
justify-content: space-between;
align-items: flex-start;
flex-wrap: wrap;
gap: 20px;
}
.graph-stats {
display: flex;
gap: 20px;
flex-wrap: wrap;
}
.selected-node-info {
background: rgba(0, 40, 0, 0.8);
border: 1px solid #00ff41;
border-radius: 4px;
padding: 15px;
min-width: 250px;
}
.selected-node-info h4 {
color: #00ff41;
margin-bottom: 10px;
text-transform: uppercase;
letter-spacing: 1px;
}
#nodeDetails {
font-family: 'Courier New', monospace;
font-size: 0.8rem;
line-height: 1.4;
color: #c7c7c7;
}
#nodeDetails .detail-item {
margin-bottom: 8px;
border-bottom: 1px solid #333;
padding-bottom: 4px;
}
#nodeDetails .detail-item:last-child {
border-bottom: none;
}
#nodeDetails .detail-label {
color: #00ff41;
font-weight: bold;
}
#nodeDetails .detail-value {
color: #c7c7c7;
margin-left: 10px;
}
/* Graph Nodes and Links Styling (applied via D3) */
.graph-node {
stroke: #333;
stroke-width: 2px;
cursor: pointer;
transition: all 0.3s ease;
}
.graph-node:hover {
stroke: #fff;
stroke-width: 3px;
}
.graph-node.selected {
stroke: #ff9900;
stroke-width: 4px;
}
.graph-link {
stroke-opacity: 0.6;
stroke-width: 2px;
transition: all 0.3s ease;
}
.graph-link:hover {
stroke-opacity: 1;
stroke-width: 3px;
}
.graph-link.highlighted {
stroke-opacity: 1;
stroke-width: 3px;
}
.graph-label {
font-family: 'Courier New', monospace;
font-size: 10px;
fill: #c7c7c7;
text-anchor: middle;
pointer-events: none;
opacity: 0.8;
}
.graph-label.visible {
opacity: 1;
}
/* Responsive adjustments for graph */
@media (max-width: 768px) {
.graph-container {
height: 400px;
}
.graph-legend {
flex-direction: column;
gap: 15px;
}
.legend-items, .legend-methods {
flex-direction: row;
flex-wrap: wrap;
gap: 15px;
}
.graph-info {
flex-direction: column;
align-items: stretch;
}
.graph-stats {
justify-content: center;
}
.selected-node-info {
min-width: auto;
width: 100%;
}
.graph-tooltip {
font-size: 0.7rem;
max-width: 250px;
}
}

250
templates/index.html
View File

@ -3,117 +3,193 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DNS Reconnaissance Tool</title>
<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
<title>DNSRecon - Infrastructure Reconnaissance</title>
<link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
<script src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" rel="stylesheet" type="text/css">
<link href="https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@300;400;500;700&family=Special+Elite&display=swap" rel="stylesheet">
</head>
<body>
<div class="container">
<header>
<h1>🔍 DNS Reconnaissance Tool</h1>
<p>Comprehensive domain and IP intelligence gathering</p>
<!-- Header -->
<header class="header">
<div class="header-content">
<div class="logo">
<span class="logo-icon">[DNS]</span>
<span class="logo-text">RECON</span>
</div>
<div class="status-indicator">
<span id="connection-status" class="status-dot"></span>
<span class="status-text">System Online</span>
</div>
</div>
</header>
<div class="scan-form" id="scanForm">
<h2>Start New Scan</h2>
<div class="form-group">
<label for="target">Target (domain.com or hostname):</label>
<input type="text" id="target" placeholder="example.com or example" required>
<!-- Main Content -->
<main class="main-content">
<!-- Control Panel -->
<section class="control-panel">
<div class="panel-header">
<h2>Target Configuration</h2>
</div>
<div class="form-group">
<label for="maxDepth">Max Recursion Depth:</label>
<select id="maxDepth">
<option value="1">1</option>
<option value="2" selected>2</option>
<option value="3">3</option>
<option value="4">4</option>
<option value="5">5</option>
<div class="form-container">
<div class="input-group">
<label for="target-domain">Target Domain</label>
<input type="text" id="target-domain" placeholder="example.com" autocomplete="off">
</div>
<div class="input-group">
<label for="max-depth">Recursion Depth</label>
<select id="max-depth">
<option value="1">Depth 1 - Direct relationships</option>
<option value="2" selected>Depth 2 - Recommended</option>
<option value="3">Depth 3 - Extended analysis</option>
<option value="4">Depth 4 - Deep reconnaissance</option>
<option value="5">Depth 5 - Maximum depth</option>
</select>
</div>
<div class="api-keys">
<h3>Optional API Keys</h3>
<div class="form-group">
<label for="shodanKey">Shodan API Key:</label>
<input type="password" id="shodanKey" placeholder="Optional - for port scanning data">
<div class="button-group">
<button id="start-scan" class="btn btn-primary">
<span class="btn-icon">[RUN]</span>
<span>Start Reconnaissance</span>
</button>
<button id="stop-scan" class="btn btn-secondary" disabled>
<span class="btn-icon">[STOP]</span>
<span>Terminate Scan</span>
</button>
<button id="export-results" class="btn btn-secondary">
<span class="btn-icon">[EXPORT]</span>
<span>Download Results</span>
</button>
</div>
</div>
</section>
<!-- Status Panel -->
<section class="status-panel">
<div class="panel-header">
<h2>Reconnaissance Status</h2>
</div>
<div class="form-group">
<label for="virustotalKey">VirusTotal API Key:</label>
<input type="password" id="virustotalKey" placeholder="Optional - for security analysis">
<div class="status-content">
<div class="status-row">
<span class="status-label">Current Status:</span>
<span id="scan-status" class="status-value">Idle</span>
</div>
<div class="status-row">
<span class="status-label">Target:</span>
<span id="target-display" class="status-value">None</span>
</div>
<div class="status-row">
<span class="status-label">Depth:</span>
<span id="depth-display" class="status-value">0/0</span>
</div>
<div class="status-row">
<span class="status-label">Progress:</span>
<span id="progress-display" class="status-value">0%</span>
</div>
<div class="status-row">
<span class="status-label">Indicators:</span>
<span id="indicators-display" class="status-value">0</span>
</div>
<div class="status-row">
<span class="status-label">Relationships:</span>
<span id="relationships-display" class="status-value">0</span>
</div>
</div>
<button id="startScan" class="btn-primary">Start Reconnaissance</button>
</div>
<div class="progress-section" id="progressSection" style="display: none;">
<h2>Scan Progress</h2>
<div class="progress-bar">
<div class="progress-fill" id="progressFill"></div>
<div id="progress-fill" class="progress-fill"></div>
</div>
<p id="progressMessage">Initializing...</p>
<div class="scan-controls">
<button id="newScan" class="btn-secondary">New Scan</button>
</section>
<!-- Visualization Panel -->
<section class="visualization-panel">
<div class="panel-header">
<h2>Infrastructure Map</h2>
<div class="view-controls">
<button id="reset-view" class="btn-icon-small" title="Reset View">[↻]</button>
<button id="fit-view" class="btn-icon-small" title="Fit to Screen">[□]</button>
</div>
</div>
<div class="results-section" id="resultsSection" style="display: none;">
<h2>Reconnaissance Results</h2>
<div class="results-controls">
<button id="showJson" class="btn-secondary">Show JSON</button>
<button id="showText" class="btn-secondary active">Show Text Report</button>
<button id="showGraphView" class="btn-secondary">Show Graph</button>
<button id="downloadJson" class="btn-secondary">Download JSON</button>
<button id="downloadText" class="btn-secondary">Download Text</button>
</div>
<div class="report-container">
<pre id="reportContent"></pre>
</div>
</div>
<div class="graph-section" id="graphSection" style="display: none;">
<h2>Discovery Graph</h2>
<div class="graph-stats">
<div class="stat-item">
<span class="stat-label">Nodes:</span>
<span id="graphNodes" class="stat-value">0</span>
</div>
<div class="stat-item">
<span class="stat-label">Edges:</span>
<span id="graphEdges" class="stat-value">0</span>
</div>
<div class="stat-item">
<span class="stat-label">Max Depth:</span>
<span id="graphDepth" class="stat-value">0</span>
</div>
</div>
<div class="graph-controls">
<button id="showGraph" class="btn-secondary active">Show Graph</button>
<button id="hideGraph" class="btn-secondary">Hide Graph</button>
<button id="resetZoom" class="btn-secondary">Reset Zoom</button>
<button id="toggleLabels" class="btn-secondary">Show Labels</button>
</div>
<div class="graph-display-area">
<svg id="discoveryGraph"></svg>
<div id="graphTooltip" class="graph-tooltip"></div>
</div>
<div class="selected-node-details" id="selectedNodeInfo" style="display: none;">
<h3>Selected Node Details</h3>
<div id="nodeDetails"></div>
<div id="network-graph" class="graph-container">
<div class="graph-placeholder">
<div class="placeholder-content">
<div class="placeholder-icon">[○]</div>
<div class="placeholder-text">Infrastructure map will appear here</div>
<div class="placeholder-subtext">Start a reconnaissance scan to visualize relationships</div>
</div>
</div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/7.8.5/d3.min.js"></script>
<script src="{{ url_for('static', filename='script.js') }}"></script>
<div class="legend">
<div class="legend-item">
<div class="legend-color" style="background-color: #00ff41;"></div>
<span>Domains</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background-color: #ff9900;"></div>
<span>IP Addresses</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background-color: #c7c7c7;"></div>
<span>Certificates</span>
</div>
<div class="legend-item">
<div class="legend-edge high-confidence"></div>
<span>High Confidence</span>
</div>
<div class="legend-item">
<div class="legend-edge medium-confidence"></div>
<span>Medium Confidence</span>
</div>
</div>
</section>
<!-- Provider Panel -->
<section class="provider-panel">
<div class="panel-header">
<h2>Data Providers</h2>
</div>
<div id="provider-list" class="provider-list">
<!-- Provider status will be populated here -->
</div>
</section>
</main>
<!-- Footer -->
<footer class="footer">
<div class="footer-content">
<span>DNSRecon v1.0 - Phase 1 Implementation</span>
<span class="footer-separator">|</span>
<span>Passive Infrastructure Reconnaissance</span>
<span class="footer-separator">|</span>
<span id="session-id">Session: Loading...</span>
</div>
</footer>
<!-- Node Details Modal -->
<div id="node-modal" class="modal">
<div class="modal-content">
<div class="modal-header">
<h3 id="modal-title">Node Details</h3>
<button id="modal-close" class="modal-close">[×]</button>
</div>
<div class="modal-body">
<div id="modal-details">
<!-- Node details will be populated here -->
</div>
</div>
</div>
</div>
</div>
<!-- Scripts -->
<script src="{{ url_for('static', filename='js/graph.js') }}"></script>
<script src="{{ url_for('static', filename='js/main.js') }}"></script>
</body>
</html>

1440
tlds_cache.txt
View File

File diff suppressed because it is too large Load Diff