RBuster: Building a Modern Directory Discovery Tool in Rust

Web directory enumeration remains a crucial aspect of security assessments and penetration testing. While tools like DirBuster have served the community well, the evolving landscape of web applications and security measures demands more sophisticated approaches. This article details my journey in creating RBuster, a modern alternative written in Rust, designed to address current challenges and limitations.

The Current State of Directory Enumeration

Directory enumeration tools have been essential in security assessments for years. DirBuster, one of the most well-known tools, was revolutionary when it was released but now shows its age in several ways:

Performance Limitations

The Java-based architecture of DirBuster, while reliable, struggles with modern performance requirements. Single-threaded operations and inefficient resource utilization lead to slower scanning speeds, especially when dealing with large wordlists or multiple targets.

False Positive Handling

Many malicious websites return HTTP 200 status codes for non-existent pages, implementing custom error pages or redirect mechanisms. Traditional tools struggle to differentiate between legitimate content and sophisticated error pages, leading to numerous false positives.

The RBuster Approach

When designing RBuster, I focused on addressing these limitations while leveraging modern technologies and approaches:

Why Rust?

Choosing Rust was deliberate, offering several advantages:

• Memory safety without garbage collection or manual memory allocation
• High-performance concurrent programming
• Rich ecosystem of modern libraries
• Strong type system preventing common bugs

Go would also be an excellent choice for this project.

Asynchronous Architecture with Tokio

The heart of RBuster’s performance lies in its asynchronous architecture built on Tokio. Here’s how it works:

let semaphore = Arc::new(Semaphore::new(opt.threads as usize));
let mut tasks = FuturesUnordered::new();

for path in batch {
    let _permit = semaphore.acquire().await.unwrap();
    tasks.push(tokio::spawn(async move {
        // Request handling logic
    }));
}

In this code snippet, we use Tokio’s concurrency model to efficiently manage a large number of asynchronous tasks.

1. Semaphore for Concurrency Control: We use a Semaphore to limit the number of concurrent tasks. This ensures that only a fixed number of tasks can run simultaneously, preventing resource exhaustion.

2. Task Management with FuturesUnordered: FuturesUnordered is a collection that allows us to manage multiple asynchronous tasks concurrently. It efficiently schedules and executes tasks as resources become available.

3. Permit Acquisition: Before spawning a new task, we acquire a permit from the semaphore (let _permit = semaphore.acquire().await.unwrap();). This operation is asynchronous and will wait if no permits are available.

4. Automatic Permit Release: The permit is automatically released when the task completes. This is due to the fact that the permit is limited to the scope of the task.

By combining these elements, the code efficiently processes tasks in parallel while maintaining control over system resource usage, leading to high performance and stability.

This approach allows for:

• Efficient resource utilization
• Non-blocking I/O operations
• Controlled concurrency with semaphores
• Dynamic task scheduling

Smart Rate Limiting Detection

RBuster implements a rate limiting detection and handling system, specifically designed for web servers that don’t provide standard rate limiting headers (like Retry-After):

let throttle = Arc::new(Mutex::new(0));
// On error (like connection refused or timeout)
{
    let mut throttle = throttle.lock().unwrap();
    *throttle = (*throttle + 50).min(1000);
}
// On lot of success we try to reduce the delay
{
    let mut throttle = throttle.lock().unwrap();
    if *throttle > 0 {
        *throttle = (*throttle - 10).max(0);
    }
}

This adaptive system:

• Starts with no delay
• Incrementally increases delay when errors occur (up to 1000ms)
• Gradually decreases delay on successful requests
• Finds the optimal request rate automatically

High-Performance Architecture with Tokio

At the heart of RBuster’s performance resides the sophisticated use of Tokio to manage asynchronous operations. Let’s explore how this process works.

let semaphore = Arc::new(Semaphore::new(opt.threads as usize));
let mut tasks = FuturesUnordered::new();

for path in batch {
    let _permit = semaphore.acquire().await.unwrap();
    tasks.push(tokio::spawn(async move {
        // Request handling
        // Permit is automatically released when the task completes
    }));
}

This architecture provides several benefits:

1. Efficient Resource Management

   • The Semaphore ensures we never exceed the specified number of concurrent requests
   • Each permit represents one available slot for concurrent execution
   • When a task completes, its permit is automatically released for reuse

2. Non-blocking Operations

   • Tokio’s runtime handles task scheduling efficiently
   • I/O operations don’t block the entire program
   • CPU resources are utilized effectively across all tasks

3. Controlled Concurrency

   • The FuturesUnordered collection manages multiple concurrent tasks
   • Tasks are automatically scheduled and executed as resources become available
   • Natural back-pressure is maintained through the semaphore system

4. Memory Efficiency

   • Batch processing prevents memory exhaustion with large wordlists
   • Resources are released promptly after task completion
   • The system scales well with available system resources

This simply gives the code:

// Batch processing for memory efficiency
let batch_size = 100; // Example batchsize 
for batch in paths.chunks(batch_size) {
    let mut tasks = FuturesUnordered::new();
    
    for path in batch {
        // Clone required values for the async task
        let client = client.clone();
        let semaphore = Arc::clone(&semaphore);
        
        tasks.push(tokio::spawn(async move {
            let _permit = semaphore.acquire().await.unwrap();
            // The actual HTTP request happens here
            // When this scope ends, the permit is automatically released
        }));
    }
    
    // Wait for all tasks in the batch to complete
    while let Some(_) = tasks.next().await {}
}

Duplicate Detection System

To handle sophisticated error pages and false positives, RBuster implements content fingerprinting:

let mut hasher = Sha256::new();
hasher.update(&content);
let checksum = hex::encode(hasher.finalize());

let mut checksums = checksums.lock().unwrap();
if !checksums.contains(&checksum) {
    checksums.insert(checksum);
    // Process unique content
}

This approach effectively identifies:

• Custom error pages
• Redirect loops
• WAF challenge pages
• Dynamic content patterns

Phishing Kit Detection

An innovative feature of RBuster is its ability to detect and analyze potential phishing kits. The detection algorithm works by understanding common patterns in how phishing kits are accidentally left on servers:

async fn detect_phishing_kit(
    client: &Client,
    url: &str,
    download_path: Option<&str>,
) -> Result<(), Box<dyn std::error::Error>> {
    let extensions = vec!["zip", "7z", "tar", "gz", "rar"];
    let parsed_url = Url::parse(url)?;
    let host = parsed_url.host_str().unwrap_or("");
    let path = parsed_url.path();

The algorithm works through several strategies:

1. Path-Based Detection:

   • Analyzes each directory in the URL path
   • Attempts to find archives named after the directory
   • Example: For /login/netflix/, checks for login.zip, netflix.zip, etc.

2. Domain-Based Detection:

   • Uses the domain name as a potential archive name
   • Common pattern where phishers name their archives after the target domain
   • Example: For bank.com, checks for bank.com.zip

3. Content Validation:

   fn is_valid_archive(content_type: &str, content_length: u64) -> bool {
       let archive_types = [
           "application/zip",
           "application/x-7z-compressed",
           "application/x-tar",
           "application/gzip",
           "application/x-rar-compressed",
       ];
       // Validates size and content type
   }
   

   • Verifies proper archive MIME types
   • Checks file size (between 1 byte and 100MB)
   • Prevents false positives from non-archive files

This approach allows you to retrieve a few kits that you can analyze manually.

Technical Challenges and Solutions

WAF Evasion Techniques

To handle modern WAFs, RBuster implements several evasion techniques:

• Random delays between requests
• Custom user agent rotation
• Proxy support for distributed scanning

Of course, this is not a real browser that is emulated, so there are many places where RBUSTER will not be able to go, like sites protected by Cloudflare’s “Under Attack” mode (based on a cryptographic JavaScript challenge). A future approach would be to connect RBUSTER to the FLARE RESOLVER proxy which handles this automatically.

Future Improvements

Some improvements are possible:

1. Smarter Extension Handling

   • Context-aware extension selection
   • Custom extension patterns
   • Framework-specific extensions

2. Extension Handling Improvements

   • Implement smart extension detection based on server technology
   • Add support for custom extension patterns
   • Develop context-aware extension selection
   • Support for framework-specific extension patterns

3. Reporting and Integration

   • Detailed HTML reports
   • API integration capabilities

This project showcases the power of Rust in security tooling and the importance of adapting our tools to modern web application architectures. As web technologies evolve, so must our security testing approaches and tools.