A lot has changed since then. RAG has gone from being a novel technique that most people hadn’t heard of to something that is now embedded (pun intended) in production systems across industries. Companies have built RAG-powered chatbots, internal knowledge assistants, customer support systems, and document retrieval tools. The problem is that many of these systems were stood up quickly, and the teams running them don’t have a reliable way to know when things are going wrong, or more importantly, why things are going wrong.

That’s where the RAG Triad comes in.

Before we get into the framework itself, I want to make sure the terminology is clear since a few of these terms are going to come up repeatedly throughout the article. When we talk about retrieval in the context of RAG, we’re referring to the step where the system searches through your documents, knowledge base, or other data sources to find the most relevant pieces of information based on a user’s question. Think of it as the system going to look something up before it tries to answer. Generation is the step where a large language model (often referred to as an LLM, which is the AI model doing the heavy lifting, like GPT or Claude) takes whatever was retrieved and uses it to construct a response in natural language. The retrieval step finds the information, and the generation step turns that information into an answer. Everything we’re going to talk about below evaluates how well those two steps are working together, and where things tend to fall apart.

What is the RAG Triad?

The RAG Triad is an evaluation framework that breaks down the quality of a RAG system into three distinct metrics: Context Relevance, Faithfulness (sometimes called Groundedness), and Answer Relevance. Each metric evaluates a different stage of the pipeline, and each one can fail independently of the others. This is important because when your RAG system gives a bad answer, the root cause could be in your retrieval step, your generation step, or both, and the fix is completely different depending on which one is actually broken.

I’ve said this before in the context of security and I’ll say it again here: you can’t fix what you can’t measure. If you’re tweaking prompts and hoping the output gets better without actually measuring which part of the system is failing, you’re essentially guessing. And guessing at scale, especially when these systems are customer-facing, is a great way to erode trust fast.

The Three Legs (and How Each One Breaks)

Context Relevance

Context relevance asks a simple question: did your retrieval step actually pull back the right documents for the user’s query? This is the foundation of everything, because if the wrong context gets fed into the LLM, it doesn’t matter how good your model is or how tight your prompt engineering is. The answer will be wrong from the start.

Here’s an example that illustrates the problem well. A user asks “What’s our refund policy for enterprise customers?” and the system retrieves your general FAQ page instead of the enterprise contract terms. The model generates a response that sounds confident and well-structured, but it’s based on the wrong source material. The user might not even realize the answer is incorrect, and that’s the dangerous part. The system didn’t hallucinate in the traditional sense; it faithfully summarized the wrong documents.

When context relevance scores are low, the problem lives in your retrieval pipeline, and the people who need to be looking at this are typically your data engineers or ML engineers, whoever built and maintains the system that decides which documents get pulled for a given query.

The things they’d look at start with the chunking strategy. Chunking is the process of breaking your documents into smaller pieces before they get stored in the system. When a user asks a question, the system doesn’t search through entire documents; it searches through these smaller chunks and returns the ones that seem most relevant. If your chunks are too large, the system might return a big block of text where only one sentence is actually relevant, which dilutes the quality of what the LLM has to work with. If your chunks are too small, you might lose important context because a key piece of information got split across two separate chunks that the system doesn’t know belong together.

They’d also look at the embedding model, which is the component that converts both your document chunks and the user’s query into numerical representations (called vectors) so the system can measure how similar they are. If your embedding model wasn’t trained on content that resembles your domain-specific data, it might not understand the relationships between terms that matter in your industry, and the similarity scores it produces won’t reflect actual relevance.

Another common fix is adding a re-ranker between your initial search results and the LLM. The way this works is that your vector search might return, say, the top 20 chunks that seem relevant based on similarity scores, but similarity doesn’t always equal relevance. A re-ranker (tools like Cohere Rerank are popular for this) takes those initial results and rescores them using a more sophisticated model that’s specifically designed to evaluate whether a piece of text actually answers a given question. The result is that the chunks that make it to the LLM are much more likely to be the right ones.

You might also look at query rewriting, which is where you transform the user’s natural language question into something that maps better to how your documents are structured. A user might ask a casual question in a way that doesn’t match the terminology used in your knowledge base, and rewriting the query before it hits the search step can dramatically improve what comes back.

Faithfulness (Groundedness)

Faithfulness measures whether the LLM stuck to what the retrieved context actually says, or whether it started making things up. This is the hallucination metric, and it’s probably the one that keeps most teams up at night because the outputs can look incredibly convincing even when they’re fabricated.

The retrieved documents might be exactly right, but the model adds details, invents statistics, or draws conclusions that aren’t supported by anything in the context. For instance, the context might say “revenue grew in Q3” and the model outputs “revenue grew 15% in Q3” when no percentage was mentioned anywhere in the source material. That 15% sounds specific and credible, which makes it more dangerous than an obviously wrong answer would be.

When faithfulness scores are low, the problem is in your generation step, not your retrieval. This is where the person managing the LLM configuration and the system prompt (the set of instructions that tell the model how to behave, what to prioritize, and what constraints to follow) needs to step in. In many teams this is a developer or an AI/ML engineer, but increasingly it’s also someone in a prompt engineering or AI product role.

The fixes here look different from the retrieval side. You’d start by tightening the system prompt to explicitly instruct the model to only use the provided context when generating its response, and to say “I don’t know” or “I don’t have enough information to answer that” when the context doesn’t contain what’s needed. You’d also want to actually test whether the model follows through on that instruction, because telling it to do something and having it consistently do it are two different things.

Another lever is temperature, which is a setting that controls how much creative freedom the LLM has when generating text. A higher temperature means the model is more likely to introduce variety and take liberties with its phrasing, which is great for creative writing but not what you want when the goal is to faithfully represent source material. Lowering the temperature makes the model more conservative and more likely to stick closely to what it was given.

You can also improve faithfulness by filtering out low-relevance chunks before they even reach the LLM. If the model receives five chunks but only two of them are actually relevant, the other three become noise that the model might draw from when constructing its answer, increasing the chance it says something that isn’t grounded in the right information.

Answer Relevance

Answer relevance is the one that tends to sneak past teams because the system can retrieve decent context, not hallucinate at all, and still completely miss the point of what the user was asking. Someone asks “How do I cancel my subscription?” and gets a technically accurate paragraph about subscription pricing tiers. The information is correct, the model didn’t make anything up, but it didn’t answer the actual question.

This is often a query understanding problem, and figuring out who owns the fix can be tricky because it could be a retrieval issue, a generation issue, or both. If the retrieval step returned documents about pricing instead of cancellation, that’s a retrieval problem and falls back to the data/ML engineering team. But if the retrieval step returned the right documents and the model just chose to summarize the wrong part, that’s a generation and prompt engineering problem.

The system misinterprets what the user is actually asking for, or the prompt doesn’t sufficiently guide the model to focus on answering the specific question rather than just summarizing whatever relevant-looking context it received. Query classification and intent detection (techniques that try to understand what the user is really asking before the search even happens) can help on the retrieval side, while better prompt engineering can address the generation side. Sometimes the fix is as straightforward as restructuring your system prompt to tell the model “answer the user’s specific question, don’t just summarize the context.”

How You Actually Evaluate This

The most common approach today is using an LLM as a judge. You take a separate LLM call (not the same one generating your answers) and use it to evaluate the output of your RAG pipeline. For each query, you capture three things: the original question, the retrieved context chunks, and the generated answer. Then you send those to the evaluator model with specific prompts designed for each metric.

For context relevance, you’d ask something like “Given this user question, rate how relevant each retrieved chunk is on a scale of 0 to 1.” For faithfulness, you’d ask “Can every claim in this answer be directly traced back to the retrieved context?” For answer relevance, you’d ask “Does this answer actually address what the user was asking?”

Tools like RAGAS, DeepEval, and TruLens have these evaluator prompts built in so you don’t have to write them from scratch. You run them against a dataset of real or synthetic queries and get scores for each dimension. This is not a one-time exercise either; you want to be running these evaluations regularly, especially as your document corpus changes, as your chunking strategy evolves, or as you swap out models.

You can also layer in human evaluation where actual people review a sample of outputs. This is more expensive and slower, but it catches things that LLM judges miss. Most mature teams that I’ve talked to do both: automated eval for coverage and speed, human eval for calibration and edge cases.

The Key Insight

The thing that I keep coming back to is that the RAG Triad gives you a diagnostic framework, not just a quality score. When something goes wrong (and it will), you can pinpoint whether the problem is in retrieval, generation, or query understanding, and then target your fix accordingly. This is fundamentally different from the approach I see a lot of teams taking, which is essentially just tweaking prompts randomly and hoping the output looks better on the next batch of test queries.

It reminds me a lot of what I’ve seen in security over the years. You can’t just throw tools at a problem and hope it goes away. You need to understand the system as a whole, identify where the actual vulnerability is, and then apply the right remediation. The RAG Triad is that diagnostic layer for your AI pipeline.

If you’re building or maintaining RAG systems and haven’t started thinking about structured evaluation yet, I’d strongly recommend looking into RAGAS or DeepEval as a starting point. The setup time is minimal compared to the visibility you get, and once you can see which metric is failing, the path forward becomes a lot clearer.

As always, let me know what you think. If you’re evaluating RAG systems in production, I’d love to hear what’s working and what’s not. You can find me on LinkedIn.

What if every WiFi handshake my pwnagotchi captured could automatically upload to the cloud? What if I could get real-time notifications on my phone and decide which networks to crack with a simple Telegram command? What if the cracking happened on AWS spot instances instead of my subpar laptop? This would save me time and effort of going back and forth to the device, copying files, and running hashcat on a device that is not nearly as powerful as some cloud instances.

What is a Pwnagotchi?

If you’re not familiar, a pwnagotchi is a “pet” that learns from WiFi networks around it. It’s built on a Raspberry Pi Zero and uses AI to optimize its handshake capture strategies. Think of it as a Tamagotchi for hackers—except instead of feeding it virtual food, you’re feeding it WiFi handshakes.

There are already great tutorials out there for building a basic pwnagotchi, so I won’t rehash the standard setup. I used jayofelony’s fork which includes updated plugins, better hardware compatibility, and active maintenance, this made the initial setup much smoother than the original.

Instead, this post focuses on what I added on top: a complete cloud pipeline for automated cracking.

The Hardware Stack

Let’s start with what I’m running:

• Raspberry Pi Zero WH – The brains of the operation

• Waveshare 2.13” e-Paper Display (V4) – For that classic pwnagotchi face

• PiSugar Battery Module – Older V2 model, with 5V output.

• 3D Printed Case with Button – Protection and style

• Bluetooth Tethering – Connected to my phone for internet access

The Waveshare display is perfect for this project. It’s low power, highly visible even in sunlight, and gives the pwnagotchi that retro digital pet aesthetic, I love the faces this thing makes.

Assembly Tips

When it comes to the assembly, there are a few things that I think help. First, knowing when the pwnagotchi is booting is difficult with a case, but if you use the transparent nut/bolt that comes with the piSugar, it will magnify the green LED, which makes it much easier to see what is going on.

Second, I recommend using a 3D printed case. There are so many variations on the build that it’s best to find one that you like and print it out, I ended up going with PETG for the sake of durability, but PLA is fine for prototyping, in the images below you will see the PLA case I poorly printed during prototyping.

Third, ensure that you have a USB A to Micro USB with data transfer. Setting this up is infinitely easier when you can plug the Pi directly into your computer and see the console output, ssh into it, and SCP files back and forth.

The Architecture

Most pwnagotchi setups require you to manually SSH in, grab the PCAP files, and run hashcat yourself. There is a service that can be used to crack the pcap files, Distributed WPA PSK auditor, but I wanted a fully automated pipeline and a quicker turnaround.

Here’s how it works:

1. Pwnagotchi captures a WiFi handshake and saves the PCAP file
2. Custom S3 upload plugin automatically uploads the file to an S3 bucket’s staging/ folder
3. Lambda function triggers when a new file arrives it then sends a Telegram notification
4. I receive a message on my phone with the network SSID, BSSID, and job ID
5. I reply via Telegram with /approve [job-id] or /reject [job-id] (clicking on the approve or rejected link copies it to your clipboard)
6. Another Lambda function processes my command and moves the file to approved/ or rejected/
7. A job launcher Lambda detects approved files and spins up an EC2 spot instance
8. The EC2 instance runs GPU-accelerated hashcat with the rockyou wordlist
9. Results are uploaded back to S3 and I get a Telegram notification with the cracked password

The entire system is serverless except for the short-lived EC2 instances that actually do the cracking. This keeps costs low, I only pay for compute when I’m actively cracking a handshake.

Code Deep Dive

Let me show you some of the key pieces. All the sensitive info has been replaced with placeholders, but you’ll get the idea.

Terraform: S3 Bucket Structure

The foundation of this system is a well-organized S3 bucket. I used Terraform to define the entire infrastructure as code:

# Main S3 bucket for storing pcap files
resource "aws_s3_bucket" "pcap_bucket" {
  bucket = local.bucket_name

  tags = merge(var.tags, {
    Name = "${local.name_prefix}-bucket"
  })
}

# Enable server-side encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "pcap_bucket" {
  bucket = aws_s3_bucket.pcap_bucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# Block public access (security best practice)
resource "aws_s3_bucket_public_access_block" "pcap_bucket" {
  bucket = aws_s3_bucket.pcap_bucket.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Lifecycle policy to auto-delete old files
resource "aws_s3_bucket_lifecycle_configuration" "pcap_bucket" {
  bucket = aws_s3_bucket.pcap_bucket.id

  # Auto-delete completed pcaps after 30 days
  rule {
    id     = "delete-completed"
    status = "Enabled"

    filter {
      prefix = "completed/"
    }

    expiration {
      days = var.s3_lifecycle_expiration_days
    }
  }

  # Auto-delete rejected pcaps after 30 days
  rule {
    id     = "delete-rejected"
    status = "Enabled"

    filter {
      prefix = "rejected/"
    }

    expiration {
      days = var.s3_lifecycle_expiration_days
    }
  }
}

The folder structure is simple but effective:

• staging/ – New uploads go here
• approved/ – Files I’ve approved for cracking
• processing/ – Currently being cracked
• completed/ – Finished jobs
• rejected/ – Networks I chose not to crack
• results/ – Cracked passwords and job metadata

The lifecycle policies ensure I don’t pay for storage forever. Old files automatically delete after 30 days.

Lambda: Upload Handler

When a new PCAP file hits the staging/ folder, this Lambda function begins to work:

"""
Pwnagotchi PCAP Cracker - Upload Handler Lambda
Triggered when a new pcap is uploaded to S3 staging/ folder
Sends Telegram notification with approval/reject options
"""

import json
import os
import boto3
import urllib3
from datetime import datetime
import uuid

# Initialize AWS clients
s3 = boto3.client('s3')
http = urllib3.PoolManager()

# Environment variables (set in Lambda configuration)
TELEGRAM_BOT_TOKEN = os.environ['TELEGRAM_BOT_TOKEN']  # <YOUR_BOT_TOKEN>
TELEGRAM_CHAT_ID = os.environ['TELEGRAM_CHAT_ID']      # <YOUR_CHAT_ID>
S3_BUCKET = os.environ['S3_BUCKET']                    # <YOUR_BUCKET_NAME>


def extract_pcap_metadata(bucket, key):
    """
    Extract SSID and BSSID from pcap file
    For MVP: parse filename (format: SSID_BSSID.pcap)
    """
    filename = key.split('/')[-1]
    
    if '_' in filename and filename.endswith('.pcap'):
        parts = filename.replace('.pcap', '').split('_')
        if len(parts) >= 2:
            ssid = parts[0]
            bssid = '_'.join(parts[1:])
            return ssid, bssid
    
    # Fallback: use filename as SSID
    ssid = filename.replace('.pcap', '')
    bssid = 'unknown'
    return ssid, bssid


def send_telegram_message(message):
    """Send message to Telegram using Bot API"""
    url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
    
    payload = {
        'chat_id': TELEGRAM_CHAT_ID,
        'text': message,
        'parse_mode': 'Markdown'
    }
    
    response = http.request(
        'POST',
        url,
        body=json.dumps(payload).encode('utf-8'),
        headers={'Content-Type': 'application/json'}
    )
    
    if response.status != 200:
        raise Exception(f"Failed to send Telegram message: {response.status}")
    
    return json.loads(response.data.decode('utf-8'))


def lambda_handler(event, context):
    """Main Lambda handler"""
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        
        # Only process .pcap files in staging/
        if not key.startswith('staging/') or not key.endswith('.pcap'):
            continue
        
        # Generate unique job ID
        job_id = str(uuid.uuid4())[:8]
        
        # Extract metadata
        ssid, bssid = extract_pcap_metadata(bucket, key)
        
        # Format Telegram message
        message = f"""
🆕 *New Network Captured*

📡 *SSID:* `{ssid}`
🔗 *BSSID:* `{bssid}`
📦 *Job ID:* `{job_id}`
🕐 *Time:* {datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')} UTC

*Reply with:*
✅ `/approve {job_id}` - Start cracking
❌ `/reject {job_id}` - Ignore this network
"""
        
        # Send notification
        send_telegram_message(message)
        print(f"Telegram notification sent for job {job_id}")
    
    return {'statusCode': 200, 'body': 'Notifications sent'}

The beauty of this setup is that I get an instant notification every time my pwnagotchi catches a new network and updates while it goes through the steps. With a lot of companies sharing spaces or having a few floors of a building there is a huge chance that you will get multiple handshakes from neighboring networks, which you are not authorized to attack. By having the ability to approve or reject a network I can ensure that I am only cracking handshakes that I am authorized to crack.

Here’s what the notification looks like when a new network is captured:

🆕 New Network Captured

📡 SSID: CoffeeShop_Guest
🔗 BSSID: a4:12:34:56:78:9a
📦 Job ID: f3a8b2c1
🕐 Time: 2026-01-28 14:32:15 UTC
📁 File: CoffeeShop_Guest_a4_12_34_56_78_9a.pcap

Reply with:
✅ /approve f3a8b2c1 - Start cracking
❌ /reject f3a8b2c1 - Ignore this network

I tap /approve f3a8b2c1 and get an immediate confirmation:

✅ Approved!

📦 Job ID: f3a8b2c1
📡 SSID: CoffeeShop_Guest
📁 Status: Moved to approved/

⚡ The cracking job will start automatically.
You'll receive a notification when complete.

A minute later, the EC2 instance spins up and I get a status update:

⚙️ Cracking Started

📦 Job ID: f3a8b2c1
📡 SSID: CoffeeShop_Guest
🖥️ Instance: i-0a1b2c3d4e5f6g7h8
⏱️ Status: Processing...

Running hashcat on g4dn.xlarge
Estimated time: 5-15 minutes

And finally, when the password is cracked:

🎉 Password Cracked!

📦 Job ID: f3a8b2c1
📡 SSID: CoffeeShop_Guest
🔓 Password: coffee2026

⏱️ Time taken: 8 minutes 32 seconds
💰 Cost: $0.42
📁 Full results saved to S3

Or, if it fails:

❌ Cracking Failed

📦 Job ID: f3a8b2c1
📡 SSID: CoffeeShop_Guest

Password not found in rockyou.txt wordlist.
⏱️ Time taken: 12 minutes
💰 Cost: $0.62

Try a different wordlist or move on to the next target.

This real-time feedback is very helpful because it lets me know exactly what is going on with the pipeline, and if there are any issues I can issue a /kill [job-id] command to terminate the EC2 instance and investigate what went wrong, and the pcap file is still in S3 so I can try again later if I want to.

Lambda: Telegram Webhook

This Lambda function acts as a webhook endpoint for the Telegram bot. When I send a command like /approve abc123, it processes the request and moves files around in S3:

def handle_approve(job_id, chat_id):
    """Handle /approve command"""
    print(f"Processing /approve for job {job_id}")
    
    # Find the job files
    pcap_key, json_key = find_job_by_id(job_id)
    
    if not pcap_key:
        send_telegram_message(
            f"❌ Job `{job_id}` not found. It may have already been processed.",
            chat_id
        )
        return
    
    try:
        # Move files from staging/ to approved/
        dest_pcap, dest_json = move_files('staging', 'approved', pcap_key, json_key)
        
        # Update metadata status
        metadata = update_metadata_status(dest_json, 'approved')
        
        # Send confirmation
        ssid = metadata.get('ssid', 'Unknown')
        message = f"""
✅ *Approved!*

📦 *Job ID:* `{job_id}`
📡 *SSID:* `{ssid}`
📁 *Status:* Moved to approved/

⚡ The cracking job will start automatically.
You'll receive a notification when complete.
"""
        send_telegram_message(message, chat_id)
        
    except Exception as e:
        send_telegram_message(f"❌ Error: {str(e)}", chat_id)

I also added /status and /kill commands. /status [job-id] shows where a job is in the pipeline, and /kill [job-id] terminates a running EC2 instance if I change my mind mid-crack or if it is taking too long.

Pwnagotchi Configuration

On the pwnagotchi side, the config is straightforward. Here’s a sanitized version of my config.toml:

# Basic device configuration
main.name = "[yourpwnagotchi]"
main.whitelist = [
    "[YourHomeNetwork]",
    "[YourNeighborsNetwork]"
]

# PiSugar battery support
main.plugins.pisugar2.enabled = true

# Bluetooth tethering to phone
main.plugins.bt-tether.enabled = true
main.plugins.bt-tether.share_internet = true
main.plugins.bt-tether.phone-name = "[YourPhone]"
main.plugins.bt-tether.phone = "android"
main.plugins.bt-tether.ip = "[YOUR:Phone:IP:Address]"
main.plugins.bt-tether.mac = "[YOUR:MAC:ADDRESS]"

# Waveshare display configuration
ui.display.enabled = true
ui.display.type = "waveshare_4"
ui.invert = true

# Web interface access (if you want to use the web interface)
ui.web.address = "0.0.0.0"
ui.web.username = "[admin]"
ui.web.password = "[your_secure_password]"

The S3 upload plugin I wrote isn’t included here, but it’s a simple Python script that watches the handshakes directory and uses boto3 to upload new PCAP files whenever they’re created.

The Cracking Pipeline

I didn’t include the actual cracking script code here (it’s basically a bash wrapper around hashcat), but here’s what happens:

1. EC2 Spot Instance Launches – I use g4dn.xlarge instances with NVIDIA GPUs
2. User Data Script Runs – Installs hashcat, aircrack-ng, and downloads the rockyou wordlist
3. PCAP Conversion – Converts the PCAP to a hashcat-compatible format
4. GPU-Accelerated Cracking – Runs hashcat with optimized settings
5. Results Upload – If a password is found, it’s uploaded to S3 results/ folder
6. Telegram Notification – I get a message with the cracked password (or a failure notice)
7. Instance Terminates – No manual cleanup needed

The whole thing is cost-effective because spot instances are cheap and I’m only running them for minutes at a time. A typical cracking job costs less than dollar.

Community and Resources

If you’re interested in building your own pwnagotchi, here are some great resources:

• Official Site: pwnagotchi.ai – Start here for the basics
• jayofelony’s Fork: github.com/jayofelony/pwnagotchi
• Discord: The pwnagotchi community is active and helpful (Pwnagotchi Unofficial)
• Reddit: Several subreddits dedicated to WiFi security and pwnagotchi builds

Final Thoughts

This project taught me a lot about integrating hardware with serverless architecture. The pwnagotchi itself is a fun hardware hack, but connecting it to AWS Lambda, S3, and EC2 made it something that you can use regularly.

The key insight for me was this: you don’t need everything running 24/7. The pwnagotchi runs on battery. The S3 bucket just sits there. Lambda functions only charge for executions. EC2 instances spin up when needed and die when done.

It’s a model that works for a lot of side projects. Build the always-on parts cheap (or free), and use on-demand compute for the heavy lifting.

If you build something similar, I’d love to hear about it. And if you’re part of the pwnagotchi community, feel free to reach out, I’m always looking to learn new tricks.

Happy hacking! 🤖🔓

I remember sitting in a conference room years ago, surrounded by product managers, developers, and designers. We were three sprints into a project and nobody could agree on what “done” actually meant. The requirements document was 47 pages long. The developers were building features that nobody asked for. The designers were frustrated. And the client? They just wanted something that solved their problem.

That project taught me something that I carry with me to this day: the quality of your input determines the quality of your output. Back then, we were talking about requirements documents and user stories. Today, we might be talking about prompts and AI-generated code. But the fundamental truth remains the same whether you are writing every line by hand or letting AI handle the typing.

Here is the thing: tools change, but the hard problem stays the same. You can mass produce code faster than ever, but that does not help if you are building the wrong thing. You cannot interview your users with a keyboard shortcut. You cannot synthesize conflicting feedback into a coherent product vision with a CLI command. You cannot decide which problem is worth solving first by asking an LLM.

That is product management. And in a world where writing code is getting easier every year, the ability to define what gets built clearly, precisely, testably, is the skill that matters most.

The Problem Is Not New, But It Is Getting Louder

Y Combinator reported that 25% of startups in their Winter 2025 batch had codebases that were 95% AI-generated. That statistic makes a lot of people nervous about the future of software development. But here is what it actually reveals: the bottleneck was never typing speed.

The developers getting the best results, whether they are using AI tools or not, are the ones who actually understand what they are trying to build before they start.

In May 2025, a Swedish vibe coding app called Lovable was found to have security vulnerabilities in 170 out of 1,645 web applications it helped create. The code looked fine, it ran fine, but it was fundamentally broken because nobody stopped to ask the right questions first.

That is not an AI problem. That is a requirements problem. And requirements problems have been shipping broken software since long before anyone heard of large language models.

Enter Design Thinking (Yes, the Sticky Note People)

I know what you are thinking. “Design thinking? That is for UX designers and people who like whiteboards.”

The LUMA Institute has been teaching human-centered design for years. Their framework breaks down into three core skills: Looking, Understanding, and Making (A for aligning or adapting depending on who you ask). It is not complicated, but it is powerful, and it is exactly what most development processes are missing.

Looking is about observing human experience. Who are your users? What are they actually trying to accomplish? What frustrates them?

Understanding is about analyzing challenges and opportunities. What patterns emerge? What assumptions are you making?

Making is about envisioning future possibilities. What does success look like? How will you know when you get there?

Sound familiar? It should. These are the exact questions you need to answer before you write a single line of code, regardless of how that code gets written.

From Observations to User Stories: The Methods That Actually Work

Here is where design thinking gets practical. LUMA does not just give you philosophy. It gives you a framework, specific methods for moving from “I have a bunch of observations” to “I know exactly what to build.” Here are the modules I use most often, with some examples.

Rose, Thorn, Bud: Sorting the Signal from the Noise

When you are in the Looking phase, you collect a lot of information. User interviews, support tickets, analytics data, competitor analysis. It is overwhelming. Rose, Thorn, Bud helps you make sense of it.

The method is simple. Take all your observations and sort them into three categories:

Roses are things that are working well. These are the features users love, the workflows that feel smooth, the moments of delight. Do not skip this category. Understanding what works is just as important as understanding what does not.

Thorns are pain points. These are the frustrations, the workarounds, the things that make users curse under their breath. Every thorn is a potential user story waiting to happen.

Buds are opportunities. These are the “what if” moments. The features users wish existed. The adjacent problems you could solve. Buds often become your most innovative features.

Here is what this looks like in practice. Say you are building an expense tracking app for small business owners. After interviewing ten users, you might end up with:

Roses:

• Users love being able to photograph receipts on their phone
• The monthly summary report saves them hours at tax time
• Integration with their bank account means less manual entry

Thorns:

• Categorizing expenses is tedious and error-prone
• No way to split a single receipt across multiple expense categories
• Forgot to log expenses until a week later, then could not remember the details

Buds:

• Several users mentioned wishing they could see spending trends over time
• One user manually tracks which expenses are tax-deductible versus not
• A few users share expense responsibilities with a business partner

Now you have raw material for user stories. But which ones should you build first?

Importance/Difficulty Matrix: Picking Your Battles

Not all user stories are created equal. Some will take months to build and barely move the needle. Others can be shipped in a week and transform the user experience. The Importance/Difficulty Matrix helps you see the difference.

Draw a 2x2 grid. The vertical axis is Difficulty (how hard is this to implement?). The horizontal axis is Importance (how much does this matter to users?). Start with the importance and do your best to ensure that each item is in its own spot, no overlapping (if possible). Once you have the importance you can start to move each item up towards the proper difficulty level. This exercise helps you identify easy wins, important considerations for difficult initiatives, etc.

Plot each potential feature or user story on the grid.

High Importance, Low Difficulty (bottom right): These are your quick wins. Do these first. They deliver value fast and build momentum.

High Importance, High Difficulty (top right): These are your major projects. Important but need significant investment. Plan carefully.

Low Importance, Low Difficulty (bottom left): Fill-in work. Nice to have but not urgent. Good for junior developers or slow weeks.

Low Importance, High Difficulty (top left): Time sinks. Avoid these. They burn resources without delivering proportional value.

Taking our expense app thorns and buds:

Look at that. Four quick wins jumped out immediately. Those become your first sprint. The multi-user sharing might seem cool, but the effort-to-value ratio is terrible right now. Maybe later, once you have nailed the core experience.

Affinity Clustering: Finding the Patterns You Missed

Sometimes your observations do not fit neatly into categories. You have sticky notes everywhere and no clear picture. Affinity Clustering helps you find the hidden structure.

The process works like this:

1. Write each observation, quote, or insight on its own sticky note (or digital equivalent)
2. Start grouping notes that seem related. Do not think too hard. Go with your gut.
3. As clusters form, give them names. What theme connects these observations?
4. Look at the clusters. Which ones are biggest? Which ones surprise you?

I did this recently with support tickets for a SaaS product. The obvious clusters were what you would expect: billing issues, feature requests, bug reports. But then an unexpected cluster emerged: users trying to do things the product was never designed for. They were using a project management tool to run their entire small business.

That cluster became a product pivot discussion. We never would have seen it if we had just triaged tickets the normal way. This does not mean you always need to pivot, in the end we realized that a proper integration would be more beneficial, but that integration became a priority.

For our expense app, Affinity Clustering might reveal that three seemingly unrelated thorns are actually the same underlying problem:

• “Categorizing expenses is tedious”
• “Forgot to log expenses until a week later”
• “Can’t remember what this $47.32 charge was for”

Take the time to really dig into the root cause and why it matters when you try to title the cluster. Users lose the context needed to accurately log expenses as time passes. Cluster name: “Context Decay: delays in reporting expenses lead to poor reporting, gaps in actual finances, and frustration from those filing/processing the expenses” vs “Context Decay”.

Now instead of three separate features, you have one user story that addresses the root cause:

As a small business owner, I want to log expenses immediately after purchase with minimal friction so that I capture accurate details before I forget them.

That is a much better foundation than “make categorizing easier,” whether you are writing the code yourself or handing it off to an AI tool.

The User Story: Clarity That Scales

User stories have been around since the early days of Agile. The format is simple:

As a [type of user], I want [some goal] so that [some reason].

For example: “As a busy parent, I want to save my grocery list so that I do not have to remember everything at the store.”

But the magic is not in the format. It is in what the format forces you to do. You have to think about who the user is. You have to articulate what they actually want. And most importantly, you have to explain why it matters.

This is where the INVEST criteria comes in. Good user stories should be:

• Independent: Not tangled up with other stories
• Negotiable: Open to conversation, not set in stone
• Valuable: Actually useful to someone
• Estimable: Clear enough to estimate the work
• Small: Completable in a reasonable timeframe
• Testable: You can verify when it is done

These criteria matter whether you are handing the story to a junior developer, a senior architect, or an AI coding assistant. Vague input produces vague output, while clear input produces clear output. The tool does not change the equation.

Acceptance Criteria: The Bridge Between “What” and “Done”

This is where things get really interesting. In Agile, every user story comes with acceptance criteria. These are the specific conditions that must be met for the story to be considered complete. The most common format uses Given/When/Then:

Given [some context] When [some action is taken] Then [some outcome is expected]

For example:

• Given a user is logged in
• When they click “Save List”
• Then their grocery items should persist across sessions

This is not just documentation, this is a testing specification. And whether you are writing tests by hand, using TDD, or letting AI generate your test suite, acceptance criteria tell you what “correct” actually means.

Putting It All Together: From Sticky Notes to Shipping Code

Let me walk you through the full workflow using our expense app example.

Step 1: Looking (Rose, Thorn, Bud)

You interview users and observe their behavior. You collect observations and sort them:

• Rose: “I love that I can just snap a photo of the receipt”
• Thorn: “I always forget to log my expenses until it’s too late”
• Thorn: “The categories never match how I actually think about spending”
• Bud: “Wish it could just tell me what category something should be”

Step 2: Understanding (Affinity Clustering)

You notice the thorns cluster around a theme: users lose context over time. The bud suggests users want intelligence, not just data entry.

Cluster: Context Decay and Cognitive Load: delays in reporting expenses lead to poor reporting, gaps in actual finances, and frustration from those filing/processing the expenses

The real problem is not the interface. It is that expense tracking requires too much mental effort at the wrong time.

Step 3: Prioritizing (Importance/Difficulty Matrix)

You brainstorm potential solutions and plot them:

• AI-powered auto-categorization: High importance, Medium difficulty → Build it
• Push notification reminders: High importance, Low difficulty → Build it now
• Voice memo for expense context: Medium importance, Low difficulty → Quick win
• Full accounting system integration: Medium importance, High difficulty → Later

Step 4: Writing User Stories

Based on your analysis, you write:

As a small business owner who makes frequent purchases, I want expenses to be automatically categorized based on the merchant and amount so that I spend less mental energy on bookkeeping.

Step 5: Defining Acceptance Criteria

Given a user photographs a receipt from a merchant they have used before When the app processes the image Then it should auto-suggest the same category used previously with 90%+ confidence

Given a user photographs a receipt from a new merchant When the app processes the image Then it should suggest a category based on merchant type and purchase amount And allow the user to confirm or change the category with one tap

Given a user overrides an auto-suggested category When they encounter the same merchant again Then the system should learn from the correction and suggest the new category

Step 6: Building

At this point, you have everything you need to build the feature correctly. If you are coding by hand, you have clear requirements and testable criteria that you can put into a ticketing system (e.g. Jira, Trello, etc). If you are using AI tools, your prompt practically writes itself (but you should still use a ticketing system and create a feature branch based on that ticket, but I digress):

You are building an intelligent expense categorization feature. Here is the context:

User Story: As a small business owner who makes frequent purchases, I want 
expenses to be automatically categorized based on the merchant and amount 
so that I spend less mental energy on bookkeeping.

The core insight from user research: Users lose context over time and find 
manual categorization tedious. They want the system to learn their preferences.

Acceptance Criteria:
1. Given a user photographs a receipt from a merchant they have used before
   When the app processes the image
   Then it should auto-suggest the same category used previously with 90%+ confidence

2. Given a user photographs a receipt from a new merchant
   When the app processes the image
   Then it should suggest a category based on merchant type and purchase amount
   And allow the user to confirm or change the category with one tap

3. Given a user overrides an auto-suggested category
   When they encounter the same merchant again
   Then the system should learn from the correction and suggest the new category

Tech stack: React Native, TypeScript, Supabase, OpenAI API for OCR
Design constraints: Must work offline with sync, categorization UI must be 
completable in under 3 seconds

Please implement this feature including:
- Data model for storing merchant-category associations and user corrections
- Service layer for category prediction logic
- React Native components for the categorization UI
- Unit tests that verify each acceptance criterion

Compare that to “build me an expense categorization feature.” The difference is not subtle, and it does not matter whether a human or an AI is on the receiving end.

QA: Testing What Actually Matters

Here is something that keeps me up at night: AI can generate tests for the code it writes. Sounds great, right? The problem is that AI-generated tests often only cover the happy path. They test what the code does, not what it should do.

This is why acceptance criteria matter so much. They are not just documentation. They are your test specification. When you write:

Given a user with no internet connection When they try to save a task Then the task should be queued locally and synced when connection is restored

You have just written a test case! A test case that reflects real user needs that you discovered through the Looking and Understanding phases of design thinking.

The teams that are shipping quality software are the ones who:

1. Start with design thinking to understand the actual problem
2. Use methods like Rose/Thorn/Bud, Affinity Clustering, and Importance/Difficulty to synthesize insights
3. Write user stories with clear acceptance criteria
4. Use those criteria to verify the implementation, however it was built
5. Review the output against the original user story, not just “does it run”

Metrics That Actually Mean Something

When you have clear user stories and acceptance criteria, measuring success becomes almost trivial. Did we meet the acceptance criteria? Yes or no. Does the feature solve the problem described in the user story? Yes or no.

Compare this to the alternative: “We shipped 47 features this quarter.” Great. Did any of them matter?

Some metrics that actually work:

• Acceptance Criteria Pass Rate: What percentage of your acceptance criteria are met on first deployment?
• User Story Completion: Not “did we build it” but “did we solve the problem”
• Iteration Count: How many times did you have to go back and refine? (Fewer is better and cheaper, and it usually means your original specification was clearer)
• Post-Deploy Defects by Criteria Type: Where are the bugs? In the functionality? The edge cases? The performance? This tells you where your acceptance criteria need more rigor.

A Word of Caution

I am not saying design thinking and user stories will solve all your development problems. Complex systems still have complex bugs. Edge cases still slip through. Requirements still change mid-project.

But here is the thing: those problems are way easier to catch when you know what you were trying to build in the first place. When you have clear acceptance criteria, you can actually test whether the implementation is correct. When you have done the design thinking work, you can recognize when you are solving the wrong problem.

The developers I know who are thriving are not the ones who type the fastest or use the newest tools. They are the ones who got better at thinking clearly about what they are building.

Getting Started

If you want to try this approach, here is what I would suggest:

1. Start with Rose/Thorn/Bud: Before you decide what to build, understand the landscape. Talk to users, review support tickets, observe behavior. Sort everything into roses (working well), thorns (pain points), and buds (opportunities). Do not skip this step. You cannot write good user stories about problems you do not understand.

2. Cluster your observations: Spread out your roses, thorns, and buds. Group the ones that seem related. Name the clusters. This is where you discover that five seemingly unrelated complaints are actually one underlying problem. These clusters become the foundation for your user stories.

3. Prioritize with Importance/Difficulty: Now you have clusters of insights. Plot them on the matrix. Which problems are high importance and low difficulty? Those are your starting points. The matrix does not tell you what features to build. It tells you which problems to solve first.

4. Let user stories emerge: For each prioritized problem, write a user story. The story should flow naturally from your research. “As a [user you interviewed], I want [solution to the thorn you identified] so that [benefit that addresses the underlying cluster].” If you did the design thinking work, the user stories should almost write themselves.

5. Define acceptance criteria: For each user story, write Given/When/Then statements that describe what success looks like. These come directly from your observations. What did users say they needed? What would make the thorn disappear?

6. Store the context in your codebase: Create a directory (something like /docs/stories or /.context) and add it to your .gitignore. Save your user stories and acceptance criteria as markdown or text files. When you are working with AI coding tools like Claude Code or Cursor, they can reference these files directly. This means you do not have to paste the same context into every prompt. The AI has access to the full picture: the user story, the acceptance criteria, the reasoning behind your decisions. Update these files as your understanding evolves. Even if you are not using AI tools, having this documentation in your repo keeps everyone aligned.

7. Build with confidence: With your context in place, you know exactly what you are building and how to verify it. Whether you are writing code by hand, pair programming, or using AI assistance, the foundation is solid.

Notice what is missing from this list: “Pick a feature.” You do not start by deciding what to build, you start by understanding what problems exist. The features reveal themselves through the process.

The Bottom Line

The way we write code is changing. AI tools are getting better every month. But the teams that will thrive are not the ones who learn to use the newest IDE or write the fanciest prompts. They are the ones who learn to think clearly about what they are building before they start.

This is product management. Whether your title says “PM” or not, the moment you start defining problems, prioritizing solutions, and writing acceptance criteria, you are doing product work. Design thinking gives you the methods. User stories give you the format. Acceptance criteria give you the tests. Together, they give you the ability to build software that actually solves problems.

The irony is that as building gets easier, the human skills of understanding users, synthesizing insights, and defining success become more valuable, not less. The sticky notes on the whiteboard are not obsolete. They are now the most important part of the process.

The revolution is not in the tools, it is in what you feed them, and feeding them well is a product management problem.

Boris Cherny’s Post on X

After months of testing various AI coding platforms (as I detailed in my vibe coding review), I’ve come to appreciate that the implementation details matter as much as the underlying AI model. Boris’s approach represents a fundamentally different philosophy from the “conversational app builder” platforms I tested. His focus is on production workflows rather than rapid prototyping.

The Parallel Processing Paradigm

Running 5-15 Claudes Simultaneously

Boris runs 5 Claude agents in his terminal (numbered tabs 1-5) plus 5-10 more on claude.ai/code, all working simultaneously. He uses iTerm2 system notifications to know when any agent needs input, allowing him to work on multiple features across different branches without waiting for any single agent to complete.

This isn’t just about speed. It’s about matching how developers actually think. While Claude works on implementing feature A, you can be planning feature B with another agent, reviewing feature C’s output, and testing feature D. The cognitive overhead of managing multiple agents is lower than you’d expect because each one maintains its own context and conversation history.

Key setup elements:

• iTerm2 configured for system notifications when agents need input
• Numbered terminal tabs (1-5) for quick identification
• Browser sessions on claude.ai/code for additional parallelism
• Mobile app sessions kicked off throughout the day for background work
• --teleport command for moving sessions between terminal and web

The mobile workflow is particularly clever: start a few complex tasks from your phone in the morning, let them run while you commute or grab coffee, then check results when you’re at your desk.

The CLAUDE.md Strategy: Team Knowledge Management

One of the most underappreciated features Boris highlights is the shared CLAUDE.md file, which is a repository-level instruction set that the entire team maintains and Claude reads before every interaction.

How It Works

Every time Claude makes a mistake or the team establishes a new pattern, it gets added to CLAUDE.md. Here’s a real example from the Claude Code repository:

Note: Bun is a fast JavaScript runtime and package manager, similar to Node.js and npm but significantly faster. The Claude Code team has standardized on it for their workflow.

# Development Workflow

**Always use `bun`, not `npm`.**

# 1. Make changes

# 2. Typecheck (fast)
bun run typecheck

# 3. Run tests
bun run test -- -t "test name"      # Single suite
bun run test:file -- "glob"         # Specific files

# 4. Lint before committing
bun run lint:file -- "file1.ts"     # Specific files
bun run lint                         # All files

# 5. Before creating PR
bun run lint:claude && bun run test

This creates a compounding knowledge base. Instead of repeatedly telling Claude “don’t use enums, use string literal unions,” you add it once to CLAUDE.md and it applies to all future interactions for all team members.

GitHub Integration: Living Documentation

The Claude Code team uses the Claude Code GitHub Action to update CLAUDE.md directly from code reviews. This means improvements to the knowledge base happen as a natural part of the development workflow.

How it works:

During code review, you can tag the Claude bot:

@.claude add to CLAUDE.md to never use enums, always prefer literal unions

The bot responds with a plan, makes the change to CLAUDE.md, and commits it. This is what Dan Shipper calls “Compounding Engineering.” Each code review makes the system slightly smarter for everyone on the team.

To set this up:

1. Install the Claude Code GitHub Action from the GitHub Marketplace
2. Configure it with your repository
3. Grant it write access to your repo
4. Start tagging @.claude in your code reviews

Within a few weeks, your CLAUDE.md will evolve from a basic template into a comprehensive guide that captures your team’s collective knowledge.

Plan Mode: The Secret to One-Shot Success

Boris’s sessions typically start in Plan mode (triggered with shift+tab twice). This is crucial: instead of letting Claude jump straight to implementation, he iterates on the approach first:

1. Describe the goal in Plan mode
2. Refine the plan through back-and-forth until it’s solid
3. Switch to auto-accept edits mode
4. Let Claude implement (usually succeeds in one shot)

This mirrors my recommendation from the vibe coding article about establishing approach before implementation, but Boris takes it further by using a dedicated mode rather than just prompting for it.

The time investment in planning pays off exponentially. A good plan lets Claude work autonomously without breaking existing functionality—the exact failure mode I encountered repeatedly with vibe coding platforms.

Slash Commands: Automating Inner Loops

Slash commands save you from repeated prompting and make it possible for Claude to use your workflows too. Boris creates custom slash commands for every “inner loop” workflow that he does many times a day. What Boris is referring to as “slash commands” are actually what Claude Code calls “skills.”

If you don’t know how to create skills, check out my guide.

What are inner loop workflows?

These are the small, repetitive tasks you do constantly during development:

• Running tests
• Committing and pushing code
• Checking type errors
• Running the linter
• Deploying to staging

Commands are stored in .claude/commands/ and checked into git so the whole team can use them.

Example: The /commit-push-pr command

This is Boris’s most-used command, running dozens of times per day:

Create .claude/commands/commit-push-pr.sh:

#!/bin/bash
# Commits changes, pushes to remote, and creates a PR

set -e  # Exit on any error

echo "📝 Preparing to commit and push..."

# Pre-compute git status to avoid back-and-forth with Claude
MODIFIED_FILES=$(git diff --name-only)
BRANCH=$(git branch --show-current)
UPSTREAM_BRANCH=$(git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "")

if [ -z "$MODIFIED_FILES" ]; then
    echo "❌ No changes to commit"
    exit 1
fi

echo "Modified files:"
echo "$MODIFIED_FILES"
echo ""

# Get commit message from Claude or user
read -p "Commit message: " COMMIT_MSG

if [ -z "$COMMIT_MSG" ]; then
    echo "❌ Commit message required"
    exit 1
fi

# Commit changes
git add -A
git commit -m "$COMMIT_MSG"

echo "✅ Committed changes"

# Push to remote
if [ -z "$UPSTREAM_BRANCH" ]; then
    git push -u origin "$BRANCH"
else
    git push
fi

echo "✅ Pushed to remote"

# Create PR if gh CLI is available
if command -v gh &> /dev/null; then
    read -p "Create PR? (y/n): " CREATE_PR
    if [ "$CREATE_PR" = "y" ]; then
        gh pr create --fill
        echo "✅ PR created"
    fi
fi

Make it executable:

chmod +x .claude/commands/commit-push-pr.sh

Why inline bash matters:

Notice the command pre-computes git status at the start:

MODIFIED_FILES=$(git diff --name-only)
BRANCH=$(git branch --show-current)

This information is gathered once and available throughout the script. Without this, Claude would need to:

1. Run git status
2. Wait for response
3. Ask what files to commit
4. Wait for response
5. Run git commit
6. And so on…

By pre-computing everything, the command runs quickly without back-and-forth.

More useful commands to create:

.claude/commands/test-focused.sh:

#!/bin/bash
# Run tests for files that changed

CHANGED_FILES=$(git diff --name-only | grep -E '\.(ts|js|tsx|jsx)$')

for file in $CHANGED_FILES; do
    # Convert source file to test file path
    TEST_FILE=$(echo $file | sed 's/src/src/__tests__/; s/\.tsx\?/.test.ts/')
    
    if [ -f "$TEST_FILE" ]; then
        echo "Testing $TEST_FILE..."
        npm test -- "$TEST_FILE"
    fi
done

.claude/commands/quick-check.sh:

#!/bin/bash
# Fast checks before committing

echo "Running quick checks..."

# Type check
echo "1/3 Type checking..."
npm run typecheck

# Lint
echo "2/3 Linting..."  
npm run lint

# Quick tests (not full suite)
echo "3/3 Running changed file tests..."
.claude/commands/test-focused.sh

echo "✅ All checks passed!"

Now Claude can run /quick-check before committing, or you can run it manually. The key is identifying your most common workflows and automating them.

Subagents: Specialized AI Workers

Subagents are like specialized team members with specific expertise. Rather than asking the generalist Claude to both implement code AND simplify it, you hand off to a specialist agent once implementation is done.

Boris maintains several subagents in .claude/agents/ for common post-processing tasks:

• code-simplifier: Refactors Claude’s output after implementation
• verify-app: Runs comprehensive end-to-end tests on Claude Code itself
• build-validator: Checks build integrity
• code-architect: Reviews large changes for architectural consistency

How to create your first subagent:

Create a file .claude/agents/code-simplifier.md:

# Code Simplifier Agent

You are a specialist in code simplification and refactoring.
Your job is to take working code and make it more maintainable 
without changing behavior.

## Your responsibilities:
1. Remove duplicate code
2. Extract complex logic into well-named functions
3. Simplify conditional statements
4. Improve variable and function names
5. Add helpful comments for complex logic

## What you should NOT do:
- Do not change functionality
- Do not add new features
- Do not remove tests
- Do not modify public APIs

## Process:
1. Read the files that were just modified
2. Identify simplification opportunities
3. Make changes one file at a time
4. Run tests after each change
5. Report what was simplified

Using subagents:

After Claude implements a feature, you can hand off to a subagent:

You: "Great! Now invoke the code-simplifier agent to clean this up"
Claude: [Calls code-simplifier subagent]
Code Simplifier: [Reviews and refactors the code]

Start with one or two subagents for your most common post-processing tasks, then add more as you identify patterns.

The Verification Loop: 2-3x Quality Improvement

Boris emphasizes that the single most important factor for quality is giving Claude a way to verify its work. Without this feedback loop, Claude can’t iterate to fix problems.

For Claude Code itself, Boris uses the Claude Chrome extension to:

1. Open a browser
2. Test the UI
3. Iterate until code works and UX feels good

This automated testing happens for every change landed to claude.ai/code. The extension can click through interfaces, verify visual elements, and report issues back to Claude for fixes.

Verification looks different per domain:

• CLI tools: Run the tool and verify output
• Web apps: Browser automation testing
• Mobile apps: Simulator testing
• APIs: Automated integration tests
• Data pipelines: Sample data validation

The principle is universal: create a fast, reliable way for Claude to check its own work, and quality will dramatically improve.

Hooks: Automated Code Quality

Hooks in Claude Code let you run commands automatically at specific points in the workflow. Boris uses PostToolUse hooks to automatically format code after Claude makes changes.

What are hooks?

Think of hooks as automatic quality checks that run without you having to remember them. They’re triggered by specific events in Claude’s workflow:

• PostToolUse: Runs after Claude uses a tool (like editing a file)
• PreToolUse: Runs before Claude uses a tool
• Stop: Runs when a long task completes
• Error: Runs when something goes wrong

Setting up your first hook:

Create or edit .claude/settings.json in your project:

{
  "PostToolUse": [
    {
      "matcher": "Write|Edit",
      "hooks": [
        {
          "type": "command",
          "command": "npm run format || true"
        }
      ]
    }
  ]
}

This configuration does several things:

• matcher: Triggers only when Claude writes or edits files (not when reading)
• command: Runs your code formatter
• || true: Ensures the hook doesn’t fail if formatting has warnings

If you’re using Bun (like the Claude Code team), it would look like:

{
  "PostToolUse": [
    {
      "matcher": "Write|Edit", 
      "hooks": [
        {
          "type": "command",
          "command": "bun run format || true"
        }
      ]
    }
  ]
}

Why this matters:

Without this hook, you’d need to:

1. Let Claude make changes
2. Remember to run the formatter
3. Commit the formatting changes separately
4. Or worse, have CI fail because of formatting issues

With the hook, formatting happens automatically after every file change. Claude’s code is already formatted by the time you review it.

Other useful hooks:

{
  "PostToolUse": [
    {
      "matcher": "Write|Edit",
      "hooks": [
        {
          "type": "command",
          "command": "npm run format || true"
        },
        {
          "type": "command", 
          "command": "npm run lint:fix || true"
        }
      ]
    }
  ],
  "Stop": [
    {
      "matcher": "*",
      "hooks": [
        {
          "type": "command",
          "command": ".claude/commands/verify-app.sh"
        }
      ]
    }
  ]
}

This setup:

1. Formats and lints code after every edit
2. Runs comprehensive verification when tasks complete

Start with just the formatting hook, then add more as you identify patterns.

Permission Management: Security Without Friction

Rather than using --dangerously-skip-permissions, Boris pre-allows safe commands through /permissions:

Bash(bq query:*)
Bash(bun run build:*)
Bash(bun run lint:file:*)
Bash(bun run test:*)
Bash(bun run test:file:*)
Bash(bun run typecheck:*)
Bash(test:*)
Bash(cc:*)
Bash(comm:*)
Bash(find:*)

These permissions are stored in .claude/settings.json and shared with the team. It’s a middle ground: Claude can work autonomously on common operations while still requiring confirmation for potentially dangerous commands.

For sandbox environments or very long-running tasks, Boris will occasionally use --permission-mode=dontAsk to let Claude “cook without being blocked,” but this is reserved for isolated contexts.

MCP Integration: External Tool Access

Claude Code uses MCP (Model Context Protocol) to interact with Boris’s entire tool ecosystem:

• Slack: Search conversations, post updates
• BigQuery: Run analytics queries via bq CLI
• Sentry: Pull error logs automatically
• Custom internal tools: Anything with a CLI interface

The Slack MCP configuration is checked into .mcp.json and shared:

{
  "mcpServers": {
    "slack": {
      "type": "http",
      "url": "https://slack.mcp.anthropic.com/mcp"
    }
  }
}

This means Claude can autonomously:

• Search for relevant Slack discussions when debugging
• Post status updates to team channels
• Query production analytics to verify changes
• Pull error logs to understand user issues

The friction of context-switching between tools disappears when Claude can access them directly.

The Ralph Wiggum Plugin: Long-Running Task Safety

For very long-running tasks (deployments, migrations, extensive refactors), Boris uses the “ralph-wiggum” plugin. This plugin was originally created by Geoffrey Huntley and implements a verification step when tasks complete.

The plugin is named after a character from The Simpsons who is famous for enthusiastically declaring “I’m helping!” while unknowingly making situations worse. The name is perfect because it captures a real risk with AI: agents that work unsupervised for hours might produce code that seems complete but actually breaks things.

The ralph-wiggum plugin ensures that when Claude finishes a multi-hour task unsupervised, the results are actually correct before merging. It runs a comprehensive verification suite and can even alert you if something seems off.

Combined with Stop hooks, this creates a safety net: long tasks can run overnight or while you’re in meetings, with automatic verification before results are committed.

The Model Choice: Opus 4.5 with Extended Thinking

Boris runs Opus 4.5 with extended thinking enabled for everything, even though it’s slower than Sonnet. His reasoning might surprise you:

“You have to steer it less and it’s better at tool use, so it is almost always faster than using a smaller model in the end.”

This contradicts conventional wisdom about using faster models for simple tasks. But Boris’s experience shows that the end-to-end time from prompt to working code is actually lower with Opus because:

1. Fewer correction cycles: Opus gets it right more often on the first try
2. Better tool use: Less back-and-forth when running commands
3. Deeper understanding: Handles complex refactors that would require multiple iterations with smaller models

The “extended thinking” mode (previously called “chain of thought”) lets the model work through problems more thoroughly before responding. You’ll see Claude’s reasoning process in real-time, which helps you understand its approach and catch potential issues early.

Think of it this way: a slower, more capable model that succeeds in one attempt is faster than a quick model that requires three rounds of corrections.

Multi-Branch Parallelism: Avoiding Conflicts

A critical detail that Boris mentions: when running multiple agents on the same codebase, each agent works on its own feature branch in its own repository clone.

Why this approach matters:

When you run 3-5 agents simultaneously all making changes to the main branch, you’re going to hit merge conflicts constantly. It becomes a mess of competing edits where Agent 1’s changes conflict with Agent 2’s changes, and you spend more time resolving conflicts than actually developing.

Instead, here’s the workflow Boris uses:

Setting up isolated workspaces:

# Create a directory for your parallel work
mkdir -p ~/code/myproject-parallel

# Clone your repo multiple times
cd ~/code/myproject-parallel
git clone git@github.com:yourorg/myproject.git agent1
git clone git@github.com:yourorg/myproject.git agent2  
git clone git@github.com:yourorg/myproject.git agent3

# In each clone, create a feature branch
cd agent1 && git checkout -b feature/authentication
cd ../agent2 && git checkout -b feature/database-migration
cd ../agent3 && git checkout -b docs/api-updates

Now each agent has:

• Its own working directory
• Its own feature branch
• Complete isolation from other agents
• Full context for its specific task

Benefits of this approach:

• PRs remain clean and focused
• Merge conflicts are rare (each branch diverges from main separately)
• Each agent has complete context for its branch
• You can abandon failed experiments without affecting other work
• Code reviews are clearer because each PR has a single purpose

Managing the workflow:

Open a terminal tab for each agent workspace:

# Tab 1: Authentication feature
cd ~/code/myproject-parallel/agent1
claude-cli

# Tab 2: Database migration
cd ~/code/myproject-parallel/agent2
claude-cli

# Tab 3: Documentation updates
cd ~/code/myproject-parallel/agent3
claude-cli

This maps to the cognitive model of working on multiple features. Each lives in its own mental and physical space until it’s ready to merge back to main.

Comparing Approaches: Vibe Coding vs. Claude Code

Having tested both paradigms extensively, the difference is clear:

Vibe Coding Platforms (Replit, Tempo, Lovable, etc.):

• Optimized for rapid prototyping and demos
• Single-agent, conversational interface
• Struggle with iterative refinement
• Break down when adding features to existing code
• Best for initial generation

Claude Code (Boris’s Workflow):

• Optimized for production development
• Multi-agent parallel processing
• Designed for iterative improvement
• Handles complex refactors through planning and verification
• Best for real applications

The vibe coding platforms excel at the first 20% of development, which is getting a working prototype fast. Claude Code, used properly, excels at the remaining 80%, which includes refining, extending, and maintaining real applications over time.

Lessons for Your Workflow

Even if you’re not ready to run 15 parallel Claude agents, several principles apply immediately:

1. Document Patterns in CLAUDE.md

Start with a simple file:

# Project Guidelines

## Tech Stack
- Use TypeScript strict mode
- Prefer functional components in React
- Use Tailwind for styling

## Testing
- Write tests before implementation
- Use Jest for unit tests
- Use Playwright for E2E tests

## Common Mistakes to Avoid
- Don't use `any` type
- Don't commit console.logs
- Don't skip error handling

Update it whenever Claude makes a mistake or you establish a new pattern.

2. Use Plan Mode for Complex Tasks

Before implementing anything non-trivial:

1. Switch to Plan mode
2. Describe your goal
3. Iterate on the approach
4. Only then switch to implementation

This single change will dramatically improve success rate.

3. Create Slash Commands for Repetitive Workflows

Identify the 3-5 workflows you do most often:

• Running tests
• Committing and pushing
• Building and deploying
• Generating documentation
• Running type checks

Create slash commands for these and share them with your team.

4. Build Verification Loops

For every project, invest in making verification fast and automated:

• Set up hot-reload for web apps
• Create test scripts that run in <10 seconds
• Build sample data generators for testing
• Set up automated E2E tests for critical paths

Then give Claude access to run these verifications.

5. Start with 2-3 Parallel Agents

You don’t need 15 agents on day one. Start with 2-3:

• Agent 1: Feature implementation
• Agent 2: Test writing
• Agent 3: Documentation updates

Even this modest parallelism will change how you work.

The Future of AI-Assisted Development

Boris’s workflow represents a mature approach to AI-assisted development. It’s not about replacing developers or eliminating coding—it’s about orchestrating AI agents as force multipliers.

The developers who will thrive aren’t those who can prompt the hardest, but those who can:

• Architect systems that AI agents can navigate
• Create verification loops that enable autonomy
• Build knowledge bases (like CLAUDE.md) that compound over time
• Manage parallel workstreams effectively
• Integrate AI into existing tool ecosystems

This echoes my conclusion from Coding Beyond AI: the future belongs to those who can architect, direct, and orchestrate AI tools, not those who simply use them as fancy autocomplete.

Getting Started: Your Path to Multi-Agent Development

If you’re using Claude Code (or any AI coding assistant), here’s how to progressively build up to Boris’s workflow. Each step builds on the previous one, and you should only move to the next step once you’re comfortable with the current one.

Step 1: Create Your CLAUDE.md File

Start by creating a file called CLAUDE.md in your project’s root directory. This will be Claude’s instruction manual for your codebase.

Initial template to get started:

# Project Guidelines

## Tech Stack
- Primary language: [Your language here]
- Framework: [Your framework]
- Package manager: [npm, yarn, bun, etc.]

## Testing
- Test framework: [Jest, Vitest, etc.]
- Command to run tests: [your command]

## Common Commands
- Start dev server: [your command]
- Build for production: [your command]
- Run linter: [your command]

## Common Mistakes to Avoid
(Start empty - you'll add to this as you go)

As you work with Claude, anytime it makes a mistake or you establish a new pattern, add it to the “Common Mistakes to Avoid” section. For example:

## Common Mistakes to Avoid
- Don't use `any` type in TypeScript
- Don't commit console.logs to production code
- Always include error handling in async functions
- Prefer functional components over class components in React

This file compounds in value over time. After a month of updates, Claude will know your codebase’s quirks better than most new team members.

Step 2: Learn Plan Mode for Complex Tasks

Plan mode is accessed by pressing Shift+Tab twice in Claude Code. It changes how Claude approaches your request.

Without Plan mode:

You: "Add authentication to the app"
Claude: [Immediately starts writing code]

With Plan mode:

You: "Add authentication to the app"
Claude: [Provides a detailed plan]
  1. Set up authentication provider (e.g., Auth0, Supabase)
  2. Create login/signup components
  3. Add protected route wrapper
  4. Implement token storage
  5. Add logout functionality
  
You: "Looks good, but let's use session storage instead of local storage"
Claude: [Updates plan]
You: "Perfect, proceed with implementation"
Claude: [Implements the updated plan]

When to use Plan mode:

• Implementing new features
• Large refactors
• Anything that touches multiple files
• When you’re not 100% sure of the approach

When to skip Plan mode:

• Fixing typos or simple bugs
• Updating documentation
• Making obvious, small changes

Start using Plan mode for any task that would take more than 5 minutes to implement manually.

Step 3: Set Up Your First Slash Command

Slash commands automate repetitive workflows. Let’s create a simple one for running your test suite.

Create a directory structure:

your-project/
  .claude/
    commands/
      test.sh

Example test.sh:

#!/bin/bash
# Runs the test suite with common options

echo "Running test suite..."

# Run tests with coverage
npm test -- --coverage --watchAll=false

# Check if tests passed
if [ $? -eq 0 ]; then
    echo "✅ All tests passed!"
else
    echo "❌ Tests failed. Review output above."
    exit 1
fi

Make it executable:

chmod +x .claude/commands/test.sh

Now when you type /test in Claude Code, it will run this script. You can create commands for:

• /commit-push - Commits and pushes changes
• /deploy - Deploys to staging
• /format - Runs code formatter
• /typecheck - Runs TypeScript type checking

The key is identifying the workflows you do most often and automating them. Start with just one or two commands, then add more as you identify patterns.

Step 4: Build Your First Verification Loop

Verification loops let Claude check its own work, which Boris identified as the single most important factor for quality.

For a web application:

Create a simple test script that Claude can run:

#!/bin/bash
# .claude/commands/verify-app.sh

echo "Starting app verification..."

# Start the dev server in the background
npm run dev &
DEV_PID=$!

# Wait for server to start
sleep 5

# Check if app is responding
if curl -s http://localhost:3000 > /dev/null; then
    echo "✅ App started successfully"
    
    # Run basic smoke tests
    npm run test:e2e
    
    RESULT=$?
else
    echo "❌ App failed to start"
    RESULT=1
fi

# Clean up
kill $DEV_PID

exit $RESULT

For an API:

#!/bin/bash
# .claude/commands/verify-api.sh

echo "Verifying API endpoints..."

# Start API server
npm run start:test &
API_PID=$!

sleep 3

# Test key endpoints
echo "Testing /health endpoint..."
curl -f http://localhost:8080/health || exit 1

echo "Testing /api/users endpoint..."
curl -f http://localhost:8080/api/users || exit 1

echo "✅ All endpoints responding"

kill $API_PID

Now when Claude makes changes, you can ask it to run /verify-app or /verify-api to check its work. Even better, set up a hook to run verification automatically.

Step 5: Start Running Multiple Agents

Once you’re comfortable with the above, you’re ready to run multiple Claude agents in parallel.

Terminal setup (iTerm2 on Mac):

1. Open iTerm2 Preferences (Cmd+,)
2. Go to Profiles > Your Profile > Terminal
3. Enable “Notifications when idle for” - set to 5 seconds
4. Check “Send notification when current session’s…”

This will give you a notification when Claude needs your input.

Running 2-3 agents to start:

Open 3 terminal tabs and number them:

• Tab 1: Feature implementation
• Tab 2: Test writing
• Tab 3: Documentation updates

In each tab, start a Claude session:

# Tab 1
claude-cli

# In Claude Code
> Implement user authentication feature

# Tab 2  
claude-cli

# In Claude Code
> Write comprehensive tests for the authentication feature

# Tab 3
claude-cli

# In Claude Code
> Update README and API documentation for authentication

Now you can work on all three in parallel. When Tab 1 finishes implementation, you’ll get a notification. Review it, and while Claude in Tab 2 is still writing tests, you can start Tab 1 on the next feature.

Managing multiple branches:

Each agent should work on its own branch to avoid conflicts:

# Tab 1
git checkout -b feature/auth
claude-cli --branch feature/auth

# Tab 2
git checkout -b feature/auth-tests  
claude-cli --branch feature/auth-tests

# Tab 3
git checkout -b docs/auth
claude-cli --branch docs/auth

This keeps work isolated until you’re ready to merge.

Step 6: Integrate with Your Tools via MCP

MCP (Model Context Protocol) lets Claude interact with your external tools. Start with one or two integrations that would save you the most time.

Example: Slack integration

Create .mcp.json in your project root:

{
  "mcpServers": {
    "slack": {
      "type": "http",
      "url": "https://slack.mcp.anthropic.com/mcp",
      "token": "your-slack-token"
    }
  }
}

Now Claude can search Slack for context:

You: "Check our Slack discussions about the authentication implementation"
Claude: [Searches Slack, finds relevant threads]

Common MCP integrations:

• Slack for team communications
• Linear/Jira for issue tracking
• Sentry for error monitoring
• DataDog for metrics
• Any tool with a CLI or API

Start with whichever tool you find yourself manually checking most often during development.

Conclusion

Boris Cherny’s workflow isn’t just about using Claude Code effectively. It’s a blueprint for the future of software development. The multi-agent approach, combined with team knowledge bases, automated verification, and thoughtful integration of AI into existing tools, creates a development environment that’s both more powerful and more maintainable than traditional workflows.

The irony is that this “advanced” setup isn’t actually that complicated. Most of it is configuration files checked into git, commands that run automatically, and patterns that your team already follows (now just documented for AI to understand).

The barrier isn’t technical complexity. It’s the shift in thinking from “AI as tool” to “AI as colleague.” Once you make that leap, the possibilities expand dramatically.

Start small. Pick one or two practices from this article and implement them this week. Add your CLAUDE.md file. Try Plan mode. Create a verification script. The compounding returns will surprise you.

Have you experimented with multi-agent workflows? Share your experiences in the comments below, or reach out to me on LinkedIn to discuss your AI development strategies.

My Claude Code Skills Repo to get you started:

• https://github.com/angakh/claude-code-skills

Related Reading:

• Creating Custom Skills in Claude Code
• Coding Beyond AI

The solution? Custom skills, which Claude Code calls “slash commands.” These are reusable scripts that Claude (and you) can invoke with a simple command like /test or /deploy. They’re essentially bash scripts with superpowers, and they’re one of the most underutilized features of Claude Code.

What Are Skills?

Skills in Claude Code are executable scripts stored in your project’s .claude/commands/ directory. They can be written in bash, Python, Node.js, or any language that can run on your system. Once created, both you and Claude can invoke them by name.

The key difference from regular scripts:

• Skills are git-tracked so your whole team shares them
• Skills are discoverable by Claude without additional prompting
• Skills can pre-compute context to avoid back-and-forth with the AI
• Skills appear in Claude’s autocomplete when you type /

Your First Skill: Running Tests

Let’s start with something simple but useful. A skill that runs your test suite with the right options.

Create the file .claude/commands/test.sh:

#!/bin/bash
# Runs the test suite with coverage

echo "Running test suite..."

# Detect which test runner you're using
if [ -f "package.json" ]; then
    if grep -q "vitest" package.json; then
        npm run test -- --coverage
    elif grep -q "jest" package.json; then
        npm test -- --coverage --watchAll=false
    else
        npm test
    fi
elif [ -f "pytest.ini" ] || [ -f "pyproject.toml" ]; then
    pytest --cov
elif [ -f "go.mod" ]; then
    go test ./... -cover
else
    echo "❌ Could not detect test framework"
    exit 1
fi

# Report results
if [ $? -eq 0 ]; then
    echo "✅ All tests passed!"
else
    echo "❌ Tests failed. Review output above."
    exit 1
fi

Make it executable:

chmod +x .claude/commands/test.sh

Now you (or Claude) can simply type /test and it will run with the appropriate test runner and options.

The Anatomy of a Good Skill

Looking at that test skill, notice a few important patterns:

1. Clear Feedback

echo "Running test suite..."
echo "✅ All tests passed!"

Skills should tell you what they’re doing and what happened. Claude uses this output to understand if the skill succeeded.

2. Exit Codes Matter

if [ $? -eq 0 ]; then
    echo "✅ All tests passed!"
else
    echo "❌ Tests failed."
    exit 1
fi

Return 0 for success, non-zero for failure. Claude uses this to know if it should continue or investigate the error.

3. Context Detection

if grep -q "vitest" package.json; then
    npm run test -- --coverage
elif grep -q "jest" package.json; then
    npm test -- --coverage --watchAll=false

Good skills adapt to your project automatically. This test skill works with multiple test frameworks without you needing to remember which one you’re using.

A More Complex Example: Smart Commit

Let’s create a skill that makes committing code intelligent. It will:

• Check for uncommitted changes
• Run tests before committing
• Generate a meaningful commit message suggestion
• Handle the git workflow

Create .claude/commands/commit.sh:

#!/bin/bash
# Smart commit with tests and meaningful messages

set -e  # Exit on any error

# Check for changes
MODIFIED_FILES=$(git diff --name-only)
STAGED_FILES=$(git diff --cached --name-only)

if [ -z "$MODIFIED_FILES" ] && [ -z "$STAGED_FILES" ]; then
    echo "❌ No changes to commit"
    exit 1
fi

echo "📝 Changes detected in:"
echo "$MODIFIED_FILES"
echo ""

# Run tests first
echo "🧪 Running tests before commit..."
./.claude/commands/test.sh

if [ $? -ne 0 ]; then
    echo "❌ Tests failed. Fix them before committing."
    exit 1
fi

# Generate commit message suggestion
echo ""
echo "Analyzing changes for commit message..."

# Get the types of files changed
HAS_TESTS=$(echo "$MODIFIED_FILES" | grep -c "test\|spec" || true)
HAS_DOCS=$(echo "$MODIFIED_FILES" | grep -c "README\|\.md" || true)
HAS_CONFIG=$(echo "$MODIFIED_FILES" | grep -c "config\|\.json\|\.yaml" || true)

# Suggest a commit message prefix
if [ $HAS_TESTS -gt 0 ]; then
    echo "💡 Suggested prefix: test: "
elif [ $HAS_DOCS -gt 0 ]; then
    echo "💡 Suggested prefix: docs: "
elif [ $HAS_CONFIG -gt 0 ]; then
    echo "💡 Suggested prefix: config: "
else
    echo "💡 Suggested prefix: feat: or fix: "
fi

echo ""
read -p "Commit message: " COMMIT_MSG

if [ -z "$COMMIT_MSG" ]; then
    echo "❌ Commit message required"
    exit 1
fi

# Stage and commit
git add -A
git commit -m "$COMMIT_MSG"

echo "✅ Committed: $COMMIT_MSG"

# Ask about pushing
read -p "Push to remote? (y/n): " SHOULD_PUSH
if [ "$SHOULD_PUSH" = "y" ]; then
    BRANCH=$(git branch --show-current)
    git push -u origin "$BRANCH"
    echo "✅ Pushed to origin/$BRANCH"
fi

Now /commit becomes a smart workflow that ensures quality before committing.

Skills That Pre-Compute Context

One of the most powerful patterns is pre-computing information that Claude would otherwise need to ask about. This eliminates back-and-forth and makes skills faster.

Example: Status skill

Create .claude/commands/status.sh:

#!/bin/bash
# Shows comprehensive project status

echo "📊 Project Status Report"
echo "========================"
echo ""

# Git status
echo "🔀 Git Status:"
BRANCH=$(git branch --show-current)
MODIFIED=$(git diff --name-only | wc -l)
STAGED=$(git diff --cached --name-only | wc -l)
echo "  Branch: $BRANCH"
echo "  Modified files: $MODIFIED"
echo "  Staged files: $STAGED"
echo ""

# Dependency status
echo "📦 Dependencies:"
if [ -f "package.json" ]; then
    OUTDATED=$(npm outdated | tail -n +2 | wc -l || echo "0")
    echo "  Outdated packages: $OUTDATED"
fi
echo ""

# Test status
echo "🧪 Last Test Run:"
if [ -f "coverage/coverage-summary.json" ]; then
    COVERAGE=$(cat coverage/coverage-summary.json | grep -o '"lines":{"pct":[0-9.]*' | grep -o '[0-9.]*$')
    echo "  Coverage: $COVERAGE%"
else
    echo "  No coverage data available"
fi
echo ""

# Build status
echo "🏗️  Build Status:"
if [ -d "dist" ] || [ -d "build" ]; then
    echo "  ✅ Build artifacts present"
else
    echo "  ⚠️  No build artifacts found"
fi

Now when Claude needs to understand your project state, it can run /status once and get everything, rather than running multiple separate commands.

Skills for Different Workflows

Here are templates for common development workflows:

Code Quality Checks

.claude/commands/quality.sh:

#!/bin/bash
echo "Running code quality checks..."

# Linting
echo "1/3 Linting..."
npm run lint

# Type checking
echo "2/3 Type checking..."
npm run typecheck

# Tests
echo "3/3 Testing..."
./.claude/commands/test.sh

echo "✅ All quality checks passed!"

Quick Fix Workflow

.claude/commands/quick-fix.sh:

#!/bin/bash
# Auto-fix common issues

echo "Applying automatic fixes..."

# Fix linting issues
npm run lint -- --fix

# Format code
npm run format

# Update imports
npm run organize-imports

echo "✅ Auto-fixes applied. Review changes before committing."

Deployment

.claude/commands/deploy-staging.sh:

#!/bin/bash
# Deploy to staging environment

set -e

echo "🚀 Deploying to staging..."

# Run quality checks first
./.claude/commands/quality.sh

# Build production bundle
echo "Building production bundle..."
npm run build

# Deploy (adjust for your platform)
echo "Deploying to staging server..."
# Examples:
# vercel deploy --prod
# aws s3 sync ./dist s3://staging-bucket
# ssh staging "cd /app && git pull && pm2 restart app"

echo "✅ Deployed to staging!"

Making Skills Discoverable

Claude automatically discovers skills in .claude/commands/, but you can help both Claude and your team by documenting them.

Create .claude/commands/README.md:

# Project Skills

Quick reference for available skills:

## Development
- `/test` - Run test suite with coverage
- `/quality` - Run all quality checks (lint, typecheck, test)
- `/quick-fix` - Auto-fix linting and formatting issues

## Git Workflow
- `/commit` - Smart commit with tests and message suggestions
- `/status` - Show comprehensive project status

## Deployment
- `/deploy-staging` - Deploy to staging environment
- `/deploy-prod` - Deploy to production (requires confirmation)

## Usage
Type `/` in Claude Code to see available skills with autocomplete.

Advanced: Skills with Parameters

Skills can accept parameters, though the syntax is bash-standard rather than special:

.claude/commands/test-file.sh:

#!/bin/bash
# Run tests for a specific file
# Usage: /test-file path/to/test

if [ -z "$1" ]; then
    echo "Usage: /test-file <path-to-test-file>"
    exit 1
fi

TEST_FILE="$1"

if [ ! -f "$TEST_FILE" ]; then
    echo "❌ File not found: $TEST_FILE"
    exit 1
fi

echo "Running tests in $TEST_FILE..."
npm test -- "$TEST_FILE"

Claude can invoke this with: /test-file src/utils/helper.test.ts

Best Practices

After creating dozens of skills, here are the patterns that work best:

1. Start Small

Don’t try to automate everything at once. Start with your most repetitive task and build from there.

2. Make Them Idempotent

Skills should be safe to run multiple times. If something is already done, they should recognize that and skip it.

3. Fail Fast with Clear Messages

If something is wrong, exit immediately with a clear explanation of what and why.

4. Chain Skills

Skills can call other skills. Your /commit skill calls /test, for example.

5. Version Control Them

Commit your .claude/commands/ directory. When teammates clone the repo, they get all your skills immediately.

When to Create a Skill vs. Using a Script

Create a skill when:

• You or Claude do it more than twice a week
• It has project-specific context
• The team should use the same approach
• You want it discoverable via / autocomplete

Use a regular script when:

• It’s one-off or rarely used
• It’s system-level, not project-specific
• It needs to be called from other scripts programmatically

Real-World Impact

After implementing a suite of skills, you’ll notice:

1. Faster Development - No more typing out test commands or remembering flags
2. Consistent Workflows - Everyone on the team runs tests the same way
3. Better AI Collaboration - Claude can execute complex workflows autonomously
4. Documentation in Code - Your skills become living documentation of how things should be done

The Boris Cherny workflow I detailed in my multi-agent article relies heavily on well-designed skills. His team uses them dozens of times per day, and they’re a key reason multiple Claude agents can work effectively in parallel.

Getting Started Today

Here’s your action plan:

1. Create the directory: mkdir -p .claude/commands
2. Create your first skill: Start with /test using the example above
3. Make it executable: chmod +x .claude/commands/test.sh
4. Try it: Type /test in Claude Code
5. Commit it: git add .claude/ && git commit -m "feat: add test skill"

Within a week of using your first skill, you’ll identify three more to create. Within a month, you’ll have a suite of skills that fundamentally change how you develop.

The time investment is minimal (10-15 minutes per skill), but the compounding returns are substantial. Each skill you create makes both you and Claude more effective at your specific workflow.

My Claude Code Skills Repo to get you started: If you want to give this a shot, I created a repo with a few skills I use daily. You can fork it and add it to your project. As always let me know what you think.

• https://github.com/angakh/claude-code-skills

Related Reading:

• The Multi-Agent Approach: How Claude Code’s Creator Uses the Tool

When you’re reading something (yes, even this post), you should always question the sources. Who is saying what? What is their agenda? For example, if an oil company is releasing their research on climate impact, you should be skeptical, right?

“I completely trust BP Oil’s research on the impact of their 87-day crude oil spill in the Gulf of Mexico.”
— No One Ever

Who or What is NANDA?

NANDA is a research project at MIT focused on building decentralized AI infrastructure. NANDA stands for Networked Agents and Decentralized AI, which I believe should have the acronym NADA (I’m just being a punk). This is actually important technology that’s going to be essential for the future of agentic AI. Think of it as DNS for AI agents—NANDA is working on what they call the “NANDA Index Quilt.”

What is the NANDA Index Quilt?

“agents, resources, and tools across platforms, organizations and protocols. Through such an approach, we allow for global interoperability, discoverability, and flexible governance of agents”
— p.2, Beyond DNS: Unlocking the Internet of AI Agents via the NANDA Index and Verified AgentFacts

The Methodology Raises Questions

When NANDA’s actual report surfaced, some issues became apparent:

Small Sample Size: Despite claims of analyzing “300 public implementations,” the real methodology reveals just 52 interviews and 153 survey responses. That’s not exactly a comprehensive industry survey.

Strict Success Definition: They define “failure” as lacking “rapid revenue acceleration” or “measurable P&L impact.” This excludes efficiency gains, process improvements, cost savings, and capability building—basically any outcome that isn’t immediate revenue growth. For example, I helped a Fortune 100 company with an audit issue. What normally took four people 2-3 months can now be done in under ten minutes with a customized AI agent. Those four people are now free to work on more important things, but apparently that doesn’t count as “success.”

Selective Focus: Their own data shows a 67% success rate for purchased AI solutions and documents companies achieving “$2-10M annually” in savings. Yet somehow this becomes a “95% failure” narrative in the headlines.

What Other Research Is Showing

While NANDA’s small sample painted a mixed picture, larger, more comprehensive studies tell a different story:

Deloitte’s State of GenAI Report:

• 74% of organizations’ most advanced AI initiatives meet or exceed ROI expectations
• 20% report ROI exceeding 30%
• State of Generative AI in the Enterprise 2024

McKinsey’s Global AI Survey:

• 71% of organizations regularly use GenAI across 1,491 participants in 101 countries
• Majority report cost reductions within business units
• Companies with systematic approaches show significantly higher success rates
• The state of AI: How organizations are rewiring to capture value

Boston Consulting Group’s Study:

• Companies investing over $50 million in GenAI show significantly higher success rates
• Success correlates with systematic upskilling and strategic implementation
• BCG AI Radar: From Potential to Profit with GenAI

These studies, with much larger sample sizes and longer observation periods, suggest the reality is far more positive than these click-bait headlines implied.

What the Data Actually Shows

Here’s what NANDA’s research actually reveals:

• 90% of employees regularly use AI tools for work (the “shadow AI economy”)
• External partnerships achieve 67% success rates vs 33% for internal builds
• Multiple documented cases of companies achieving millions in measurable savings
• Main barriers are organizational, not technological

This paints a picture of widespread AI adoption with clear best practices emerging, not the catastrophic failure narrative that made headlines. Please note, the catastrophe narrative came from media outlets that grabbed one statistic (95% don’t show rapid revenue acceleration) and turned it into “AI is failing everywhere.” That’s not what NANDA said, but it’s what gets clicks.

The Real Issue: Media Sensationalism

To be honest, most of what I initially thought was bias was actually something simpler: lazy journalism. NANDA’s research, while flawed, isn’t portraying a catastrophe. They’re documenting what they call the “GenAI Divide”…some organizations succeed, most struggle with implementation.

The catastrophe narrative came from media outlets that grabbed one statistic (95% don’t show rapid revenue acceleration) and turned it into “AI is failing everywhere.” That’s not what NANDA said, but it’s what gets clicks.

The Bigger Picture

This bothers me for two reasons:

1. How quickly media turned nuanced research into clickbait headlines
2. How a small, obviously limited sample got extrapolated to industry-wide conclusions

Do you know how many companies are using AI right now? All of them. A study of 52 companies doesn’t represent that reality.

As someone who’s spent years advocating for democratized technology, sloppy research and sensationalist reporting undermine trust in both academia and the technology industry.

The Bottom Line

Before you make any strategic decisions based on headlines about “95% failure,” consider reading the actual source. Just because Fortune regurgitated an article doesn’t mean their clickbait headline reflects reality.

The real story is more nuanced: AI adoption is massive, external partnerships work better than internal builds, and organizations are achieving meaningful value when they approach implementation strategically. That’s not as dramatic as “95% failure,” but it’s the truth and a lot more useful for making actual business decisions. If someone tells you “it’s just plug ‘n play”, run.

Do your own research. Look at the methodology. And remember that clickbait headlines are often just that: headlines, not truth.

This analysis is based on NANDA’s own published research and publicly available information about their institutional partnerships and commercial interests.

Here are some of the other articles I read to write this post (which is why it took me a week to post this!):

• MIT report: 95% of generative AI pilots at companies are failing
• An MIT report that 95% of AI pilots fail spooked investors. But it’s the reason why those pilots failed that should make the C-suite anxious
• MIT’s NANDA’s Website
• MIT’s NANDA’s Whitepaper
• NANDA’s paper on decentralized AI tech
• NANDA’s paper on the NANDA Index Quilt

Google Chrome (worst)

Chrome provides a hidden full-page screenshot feature within its Developer Tools:

1. Open Developer Tools: Press Ctrl + Shift + I (Windows) or Cmd + Option + I (Mac)
2. Open Command Menu: Press Ctrl + Shift + P (Windows) or Cmd + Shift + P (Mac)
3. Run Screenshot Command: Type “screenshot” and select “Capture full size screenshot”

The screenshot will be automatically saved to your default Downloads folder.

Mozilla Firefox (method two is super easy)

Firefox offers multiple methods for taking full-page screenshots:

Method 1: Developer Tools

1. Open Developer Tools: Right-click on the page and select “Inspect Element” or press Ctrl + Shift + I (Windows) or Cmd + Option + I (Mac)
2. Access Settings: Click the “…” (three-dot) menu in the Developer Tools toolbar
3. Enable Screenshot Tool: Select “Settings” and check “Take a screenshot of the entire page” under “Available Toolbox Buttons”
4. Take Screenshot: Click the camera icon that appears in the toolbar

Method 2: Built-in Screenshot Tool

1. Right-click anywhere on the webpage
2. Select “Take Screenshot”
3. Click “Save full page” in the screenshot interface
4. Click “Download” to save the image

Microsoft Edge (easiest)

Edge includes a dedicated “Web Capture” feature for screenshots:

1. Open Web Capture: Click the menu (…) button and select “Web capture” or press Ctrl + Shift + S
2. Capture Full Page: Select “Capture full page” from the options
3. Save or Edit: Choose to save, copy, or annotate the screenshot before saving

Safari

Safari on Mac provides screenshot functionality through its developer tools:

1. Enable Developer Menu: Go to Safari > Settings > Advanced and check “Show features for web developers”
2. Open Web Inspector: From the Develop menu, select “Show Web Inspector”
3. Capture Screenshot: In the Elements tab, right-click on the <html> element and select “Capture Screenshot”

The tools that I have tested are as follows:

• VS Code with Augment
• VS Code with CoPilot
• Cursor
• Kiro
• Claude Code (I know, its not an IDE)

VS Code with Augment or CoPilot

If you are used to working in VS Code, this is one way you can stay within your comfort zone and try the different AI tools that are at your disposal. Personally I felt that in this match up Augment surpassed CoPilot purely based on its context. The interface with Augment is fairly easy it provides you with explanations using natural language for prompting and inline code changes that look a lot like cherry picking commits. “Do you want to accept this change?” vs it just writing everything for you. As it goes through your code and starts to make “changes” (you have to accept the changes for them to be applied) you get to see what the changes are. At the end of the vibe coding or as Dharmesh calls it “Pair coding” session, it will provide you with an update as to what it did, why it did it, and how you can verify or test the changes. In addition to this, it gives you the option to apply all the changes it has made, or you can open up each file and accept the changes one by one.

CoPilot does very similar things but it had a lot of issues for the problems that I was trying to solve, the context was not there. It was making changes in what seemed like a vacuum.

The Extension Philosophy

This represents the “evolution” approach to AI integration—taking what works and gradually adding AI capabilities. VS Code’s strength lies in its massive ecosystem with over 30,000 extensions and millions of users, offering proven stability and extensive customization options. However, these AI features can feel bolted on rather than native, and you’re limited by legacy architecture decisions that weren’t designed with AI first workflows in mind. They are still pretty cool though, so if you are on the fence, grab your favorite drink, load up your calming spotify play list and explore.

NOTE:

Both Augment and CoPilot require network bandwidth for cloud based AI processing, which can introduce latency that affects typing responsiveness. For enterprise development, consider that your code is being sent to external servers, which may raise privacy and security concerns depending on your organization’s requirements.

Cursor

The sad thing about VS Code’s implementation is that Cursor feels like what VS Code should have been. You can import your settings from VS Code and now you have what behaves like VS Code but the AI is already built in. Cursor’s context capability and coding is extremely well rounded but like many of the LLMs that try to code, it sometimes falls short. Things like dependencies, even errors in syntax can be common. I am sure as time goes on all of these LLMs will get better but depending on what you are working on, these can be a real hinderance and counter productive. Remember the whole point of using AI in coding is to help you move quickly, but if you are spending more time troubleshooting the code that you didn’t write, that you don’t understand, what are you really doing?

Example of a Cursor Issue

I had a json template that I was using to display company data for a demo. I wanted to create a few different companies and use that same JSON template. I asked Cursor to take the data I had and generate a JSON file in the same manner as the one that already existed. This should not be a difficult task, if anything this is the type of stuff you would ask AI to do. Where did it go wrong? Everywhere! The JSON was complete garbage! Not the content, but the structure and syntax were totally broken. This is something that it should have easily picked up on. But it didn’t, it totally missed it.

The AI First Philosophy

Cursor represents the “revolution” approach, designed from the ground up with AI collaboration in mind. It’s not trying to retrofit AI into an existing editor; it’s built specifically for the AI era. This means:

• Cohesive AI integration throughout the entire development experience
• Modern architecture optimized for AI features from day one
• Multi-model support letting you choose between different AI providers
• Seamless AI chat integration without leaving your coding context

The Modern Developer Experience

Cursor feels like what an IDE should be in 2024, with a clean, modern interface that doesn’t feel cluttered with legacy features. The visual diff interface shows exactly what the AI wants to change before you accept it, and the intelligent code editing can modify existing code based on natural language instructions. However, as noted, the smaller ecosystem and potential for AI generated errors means you need to stay vigilant about code quality. New files/code have to be reviewed thoroughly before committing to a repo, you don’t want to be that guy.

Kiro

Kiro is very similar to Cursor in the sense that you install, you import from VS Code (if you want) and it is ready to go. It has a cool splash page when you launch the IDE where you can Vibe or Spec. Vibe is for when you have a particular task you want to accomplish or you want to prototype something. Spec is when you use natural language to plan out what you are trying to do, ideate and then build. This is similar to how replit and lovable approach coding. It is a robust IDE but feels lightweight. But its ability to understand your codebase is not 100% there. For example I asked it to review the code for ieps.ai and provide me with some insight into what my application does. It broke everything down to the best of its ability but it totally missed an entire layer of complexity. I am using S3 buckets for storage and it assumed that everything was stored locally. It also missed lambda function calls and conversational agents.

This really surprised me because I am a huge fan of Claude Code and when it comes to the LLM behind Kiro and Claude Code, they are using the same underlying LLM!

But Kiro has something that I have not really seen in the other IDEs. Agent Hooks, Agent Steering and MCP Servers are available out of the box. You need to connect, but the capability is right there. These additional features can allow for customization on a level that I have not seen yet.

Amazon’s Entry into AI Powered Development

As Amazon’s entry into the AI IDE space, Kiro brings some interesting innovations but also reveals the challenges of building truly context aware development environments. The Vibe and Spec modes represent a thoughtful approach to different development workflows. For example if you are in Vibe mode you can quickly prototyping or work on a specific task. In Spec mode you can plan and setup some guidelines prior to starting development. In previous posts regarding vibe coding I have covered how important this is.

Context Limitations and Learning Curve

This really highlights a common challenge across AI powered IDEs. The fact that it missed critical infrastructure components like S3 buckets and Lambda functions means it has a hard time decyphering modern distributed architectures. This emphasizes the importance of maintaining your understanding of what the AI is doing rather than blindly trusting its analysis. If a junior developer asked Kiro what this app was doing, they might try to implement something that is already there, screwing up more of the code and then trying to figure out what went wrong.

Claude Code (not an IDE)

In my opinion Claude Code is the closest to having someone you trust working next to you. The interface is not ideal for many, its CLI (command line interface), but for someone who grew up on vim, it feels like home. Claudes reasoning, context, and ability to code surpass all of the above options. When it starts to tackle a problem it provides you with an approach and asks if you agree with it before it starts. It shows you how many tokens it is using and provides you with a summary of what it just did. It is awesome! BUT! There are two drawbacks that I hope get resolved soon. For those that are working on Windows OS, they will need to have virtualization enabled, install ubuntu (via microsoft store, not that bad but still its an extra step) and it doesn’t have a chat history. As you reach the limit of its contextual window you will get a context percentage at the bottom of the prompt window and as it starts to countdown you will begin to sweat.

Over a decade ago I committed this code to github: screenrc. I was working for a presidential campaign in Boston and the internet we had at the headquarters of this campaign was line of sight to the prudential building. This meant that crappy weather would disrupt your connection. If you were ssh’d into a remote box running something and got disconnected your work was lost. So I used “screen” and it was helpful. I am sharing this because if you are one of the unfortunate ones that only has access to a windows machine you may want to run screen in ubuntu and have a few “tabs” open as you work. You can have one for claude code, one for git commits, etc.

If you are on a mac? It is flawless. You can drag and drop images to help guide the front-end development of an app. I really hope for the sake of the Windows users that they offer a better solution to integrating with ClaudeCode, because as of writing this it still feels like the best of the bunch.

The Conversational Approach

Claude Code represents a fundamentally different philosophy—conversational interface that feels like pair programming with an expert rather than just autocomplete with a dash of ADHD and RedBull. The deep reasoning capabilities mean it can explain not just what code does, but why, and the safety focused design helps avoid common security pitfalls.

Terminal Native Benefits and Challenges

The terminal native approach makes it lightweight and fast, easily integrated into existing workflows, and perfect for remote development. However, the lack of chat history and context window limitations can be frustrating during longer coding sessions. The Windows setup requirements (virtualization and Ubuntu) add friction for some developers, but for those comfortable with command-line environments, it offers an unparalleled collaborative coding experience.

The Trust Factor

What sets Claude Code apart is its transparency—showing token usage, asking for approval before major changes, and providing clear summaries of actions taken. For me this built a level of trust that I just don’t have with the other IDE’s.

The Real Battle: Philosophy vs. Features

This isn’t just about which tool has the most features…it’s about fundamentally different philosophies of how AI should integrate into development workflows, and from what I’ve seen from hands on experience, each approach has real world implications.

The Extension Approach (VS Code)

VS Code represents the “evolution” philosophy: take what works and gradually add AI capabilities. This approach offers:

Pros:

• Familiar interface for existing users
• Massive ecosystem and community
• Proven stability and reliability
• Freedom to mix and match AI providers

Cons:

• AI features can feel bolted-on (as experienced with CoPilot’s context issues)
• Potential for feature conflicts
• Limited by legacy architecture decisions
• Inconsistent AI integration across different extensions

The AI-Native Approach (Claude Code, Cursor, Kiro)

These tools represent the “revolution” philosophy: design for AI first workflows from day one.

Pros:

• Cohesive AI integration (Cursor’s seamless experience)
• Modern architecture optimized for AI features
• Consistent user experience
• Purpose built for AI collaboration (Claude Code’s conversational approach)

Cons:

• Smaller ecosystems
• Less proven in production environments
• Potential vendor lock-in
• Learning curve for existing developers
• Context understanding limitations (as seen with Kiro missing infrastructure components)

My Take: What Actually Matters

After testing all these tools extensively, here’s what I’ve learned:

Context is King: The tools that understand your entire project (like Augment) consistently outperform those that work in isolation. When CoPilot was making changes “in a vacuum,” it became clear that AI without proper context is just fancy autocomplete.

Trust Through Transparency: Claude Code’s approach of explaining its reasoning and asking for approval creates a collaborative relationship rather than a “just trust me” dynamic. You learn while you code.

The Debugging Tax: Remember, if you’re spending more time troubleshooting AI generated code that you don’t understand, you’re not actually moving faster. This is especially important for junior developers who might be tempted to accept everything the AI suggests.

Interface Matters Less Than You Think: While Cursor’s modern UI is appealing, Claude Code’s CLI approach proves that good AI collaboration transcends interface preferences. What matters is the quality of the AI’s reasoning and its ability to work with your specific codebase.

For Individual Developers

Just an idea - All of the IDE’s mentioned in this post have free tiers. Trying it out has never been easier and I would strongly suggest you create a branch of an existing repo and try to tackle the same problem in each IDE on a branch named after the IDE. Use the same prompt and see what you get.

If you’re comfortable with VS Code and want to start gradually, Augment provides the best context aware experience within a familiar environment.

For those ready to embrace AI first development, Cursor offers the most polished experience, but be prepared to verify and understand the code it generates.

If you’re comfortable with command line interfaces and prioritize AI reasoning quality over flashy UIs, Claude Code provides the most trustworthy collaborative experience.

For Teams and Learning

Claude Code is excellent for learning because it explains its reasoning, making it ideal for junior developers who want to understand not just what code does, but why.

Cursor works well for teams that need a modern, collaborative environment with good visual feedback on changes.

Avoid tools with poor context understanding (like Kiro missing key infrastructure) for complex projects—the time spent correcting misunderstandings negates the productivity benefits.

The Bottom Line

The “winner” depends entirely on your workflow, experience level, and what you value most. For me it is hands down Claude Code and it’s not even close. But regardless of which tool you choose, remember to stay curious, ask the AI to explain its decisions, and never stop understanding what your code actually does.

The real victory isn’t finding the perfect AI assistant…it’s learning to collaborate effectively with AI while maintaining your skills and understanding as a developer.

I’ve spent months testing these platforms, investing real money to access full feature sets across multiple services. From invite-only beta platforms to established players, I tested Envato, Build.ai, Builder.ai, Replit, Lovable, Tempo, Emergent, and several others. Here’s what I discovered about the current state of conversational app development.

The Testing Ground: A Real-World Project

For consistency, I asked each platform to build the same application: an event management tool for my wife’s special events role at a local private school. The requirements included OAuth authentication, user profiles, event creation and management, team member invitations, and budget tracking—a reasonably complex application that would test each platform’s capabilities.

What They All Get Right: The Magic of First Impressions

The initial results were genuinely impressive. Every platform I tested could take my description and generate a mostly functional application from a single prompt. Within minutes, I had working prototypes with:

• OAuth authentication systems
• User profiles and management
• Event creation and editing interfaces
• Team invitation functionality
• Budget tracking components
• Basic responsive design

This first iteration capability is transformative. For rapid prototyping or proof-of-concept development, these tools are unmatched. The speed from idea to working demo is remarkable and represents a genuine breakthrough in application development accessibility.

The Pricing Puzzle: Different Models, Different Pain Points

The platforms take notably different approaches to monetization:

Credit-Based Systems (Emergent)

• Pay-per-use model with real-time credit consumption
• Deployment costs ~50 credits ($20 USD)
• Costs escalate quickly with iterations
• Transparent but expensive for extensive development

Subscription + Microtransactions (Replit)

• $25/month base subscription
• Agent checkpoints: $0.25 each
• Assistant checkpoints: $0.05 each
• Free deployments
• Occasionally waives fees for AI-caused errors

The Hidden Truth: Builder.ai’s Revelation

One particularly eye-opening discovery was Builder.ai, which marketed itself as an AI coding platform but actually employed human developers working behind the scenes. This “smoke and mirrors” approach highlights the importance of understanding what’s actually powering these platforms.

The Critical Flaw: Where AI Development Breaks Down

Here’s where every platform I tested failed: iteration and feature addition. The moment you try to modify or extend the initial application, the AI systems struggle with code organization and context management. I encountered numerous examples of this breakdown:

Case Study 1: The SVG Disaster

When requesting an update to an SVG code snippet, one platform generated malformed code with a closing tag </svg>vg>, causing compilation errors that required additional credits to resolve.

Case Study 2: The Button Color Catastrophe

I requested simple form validation that would change a button’s color to green when all fields were completed. The AI successfully implemented this feature, but somehow broke:

• Login functionality
• File upload capabilities
• User account creation
• API endpoint connections
• Nearly every other system component

The button turned green perfectly, but the application became unusable.

The Solution: Hybrid Development Approach

The most effective strategy I discovered combines these platforms’ strengths with traditional development tools:

1. Use vibe coding for rapid prototyping
2. Export to GitHub (most platforms offer this)
3. Continue development locally with traditional AI coding assistants
4. Leverage better context windows in tools like Claude, Copilot, or Cursor

This approach gives you the speed of initial AI generation with the control and reliability of established development workflows.

Platform Comparison

Replit: The Developer’s Choice

Among all platforms tested, Replit emerged as the most developer-friendly option. It provides:

• Integrated development environment with full IDE capabilities
• Built-in deployment pipeline with autoscaling
• Complete GitHub integration and management
• Object storage and database creation tools
• Comprehensive logging and console access
• Cost bypass mechanism through GitHub sync (changes pulled from GitHub don’t count as checkpoints)

However, Replit has limitations with complex operations like PDF processing, where external services (Lambda functions with S3 storage) become necessary.

Pro Tips for AI-Assisted Coding

Through extensive testing, I’ve identified several strategies that dramatically improve results:

1. Code Organization Strategy

AI models tend to cram everything into single files. Use this prompt to improve maintainability:

Please review [filename].tsx and break up the functions into separate files. 
I'd like to organize the code into these categories: services, handlers, 
endpoints, and middleware. Each category should be in its own file.

2. Planning Before Coding

Always establish approach before implementation:

Do not write any code yet. First, provide me with your approach to 
[describe your goal] and ask me if I agree with it. We should ensure 
we're in agreement before writing any code.

3. Controlled Implementation

Once you’ve agreed on the approach:

I agree with this approach. Please update only one file at a time and 
ask if I have questions about the changes. If I don't have questions, 
you can write [Option A: the complete file] or [Option B: only the 
changed code with clear start/end markers].

Option A works better for non-integrated tools Option B enables longer conversations by conserving context window

IDE Integration: Augment Leads the Pack

For traditional IDE-based development, I recommend Augment. It excels at:

• Rapid code indexing
• Contextual code suggestions
• Natural language code queries
• Seamless VS Code integration

The Verdict: Promise Partially Delivered

Vibe coding platforms represent a genuine breakthrough in application development accessibility, but they’re not the complete solution they promise to be. They excel at:

• Rapid prototyping
• Initial application generation
• Lowering barriers to entry
• Proof-of-concept development

However, they struggle with:

• Iterative development
• Complex feature additions
• Code maintainability
• Context management

Looking Forward

The future likely belongs to hybrid approaches that combine the rapid generation capabilities of vibe coding platforms with the precision and control of traditional development tools. As these platforms mature and improve their iteration capabilities, they may eventually deliver on their full promise.

For now, treat them as powerful prototyping tools that can jumpstart your development process, but be prepared to transition to traditional development methods for serious application building.

The democratization of app development is happening, just not quite as seamlessly as the marketing suggests. The key is understanding these tools’ strengths and limitations, then using them strategically within a broader development workflow. Like I mentioned in my previous post about Are Coding Skills Following the Typists Path, the future belongs to those who can effectively prompt, architect, direct, and integrate with AI tools.

Have you experimented with vibe coding platforms? Share your experiences and insights in the comments below. ```

In this post we will be getting technical, so if that is not your thing, don’t feel bad about hitting the back button.

1. Docker and Dev Containers

What are Dev Containers and how do they work?

Dev Containers are development environments containerized using Docker that allow developers to use a consistent, pre-configured environment. They encapsulate dependencies, runtimes, and tools needed for development.

Dev Containers work by leveraging Docker’s containerization technology but with a focus on development rather than deployment. When a developer opens a project with Dev Container support (in VS Code or other compatible IDEs), the IDE builds and runs the container, then connects to it for development tasks like editing, debugging, and running code.

How do “.devcontainer/Dockerfile” and “.devcontainer/devcontainer.json” work together?

These two files form the foundation of a Dev Container:

“.devcontainer/Dockerfile”: Defines the base container image and steps to install required tools and dependencies.

FROM python:3.11
RUN apt-get update && apt-get install -y \
    git \
    curl \
    && rm -rf /var/lib/apt/lists/*
RUN pip install poetry

“.devcontainer/devcontainer.json”: Configures how the Dev Container integrates with the IDE and environment.

{
  "name": "Python Project",
  "build": {
    "dockerfile": "Dockerfile",
    "context": ".."
  },
  "customizations": {
    "vscode": {
      "extensions": ["ms-python.python", "ms-python.vscode-pylance"]
    }
  },
  "forwardPorts": [8000],
  "postCreateCommand": "poetry install"
}

The Dockerfile builds the container, while the devcontainer.json file configures how the IDE interacts with it, including IDE extensions to install, ports to forward, and commands to run after container creation.

Benefits for team collaboration

Consistency: Every team member works in the exact same environment, eliminating “works on my machine” problems

Onboarding: New developers can be productive within minutes by simply opening the project in their IDE

Isolation: Projects with different dependencies don’t conflict with each other

Version control: The development environment itself is versioned alongside the code

How they help achieve parity with production

Dev Containers can use the same base images as production containers, shared dependencies ensure development behaviors match production, environment variables can be configured similarly to production, and service dependencies (databases, message queues) can be included via Docker Compose.

2. Cloud Development Environments

What are cloud IDEs like Google’s Project IDX or GitHub Codespaces?

Cloud Development Environments provide fully functional development environments hosted in the cloud and accessible through web browsers or local IDEs. They eliminate the need to set up local development environments completely.

GitHub Codespaces: Pre-configured cloud environments integrated with GitHub repositories

Google’s Project IDX: Google’s cloud development platform designed for web and mobile app development

GitPod: Open source cloud development environments that can integrate with GitHub, GitLab, and Bitbucket

How do they differ from local Dev Containers?

Resource allocation: Cloud environments use cloud resources instead of local computer power

Access: Accessible from any device with a web browser

Setup time: Instant access without local Docker installation or configuration

Cost model: Usually involves usage-based pricing rather than local hardware costs

Performance: Network latency can affect the development experience

Configuration files they use

GitHub Codespaces: Uses the same “.devcontainer” configuration as local Dev Containers

Project IDX: Uses “.idx/dev.nix” configuration files based on the Nix package manager

Example “.idx/dev.nix” for Project IDX:

{ pkgs, ... }: {
  channel = "stable";
  
  packages = [
    pkgs.nodejs_20
    pkgs.yarn
    pkgs.python311
  ];
  
  idx.extensions = [
    "dbaeumer.vscode-eslint"
    "esbenp.prettier-vscode"
  ];
  
  idx.previews = {
    enable = true;
    previews = [
      {
        command = ["npm" "run" "dev"];
        manager = "web";
        id = "web";
      }
    ];
  };
}

Advantages and limitations

Advantages:

• Work from anywhere with internet access
• No local setup required
• Consistent environment for all team members
• Easily scalable resources for intensive tasks
• Collaboration features like real-time pair programming

Limitations:

• Requires internet connectivity
• Potential latency issues
• Monthly costs for team usage
• Less control over the underlying infrastructure
• Privacy/security concerns with proprietary code in cloud environments

3. Other Approaches

How do tools like Docker Compose fit into development workflows?

Docker Compose allows developers to define and run multi-container Docker applications. It’s often used alongside Dev Containers to set up supporting services needed for development (databases, caches, message queues), create a network of interconnected services that mirror production, and manage environment variables and volumes across multiple containers.

Example “docker-compose.yml”:

version: '3'
services:
  app:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    depends_on:
      - db
      - redis
  
  db:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: devpassword
      POSTGRES_USER: devuser
    volumes:
      - pgdata:/var/lib/postgresql/data
  
  redis:
    image: redis:7
    ports:
      - "6379:6379"
volumes:
  pgdata:

Differences between Dev Environments and Docker Compose

Dev Containers focus on the development environment itself (IDE integration, extensions, tools)

Docker Compose orchestrates multiple services that work together

Dev Containers can integrate with Docker Compose to provide both aspects

Role of package managers like “uv” and task runners like “just”

Modern package managers like “uv” (for Python, written in Rust) improve dependency management speed and reliability. I highly recommend “uv” it is so much faster.

Task runners like “just” provide a consistent interface for common development tasks

Example “justfile”:

default:
    @just --list

# Run project unit tests
test:
    uv run -- pytest

# Run MLflow server
mlflow:
    uv run -- mlflow server --host 127.0.0.1 --port 5000

# Serve latest registered model locally
serve:
    uv run -- mlflow models serve -m models:/mymodel/latest -h 0.0.0.0 -p 8080

These tools help standardize common development tasks across the team, regardless of the environment they’re working in.

4. Best Practices

When to choose each approach

Dev Containers: For teams with complex development environments who want IDE integration

Cloud Development: For distributed teams, or when onboarding needs to be extremely fast

Docker Compose: For applications with multiple interconnected services

Package managers/task runners: As complementary tools in any environment

Ensuring development matches production

Use the same base images and version tags when possible, document all dependencies explicitly, use infrastructure-as-code to define both environments, test in a staging environment that mirrors production before deployment, and include all critical services in the development environment.

Trade-offs between simplicity and completeness

Simple environments are faster to set up but may miss edge cases

Complete environments catch more issues but require more resources and maintenance

Start with the minimal viable environment and incrementally add complexity as needed

Focus on matching the aspects of production that affect development most directly

Managing environment variables

Use “.env” files for development-specific variables, never commit production secrets to version control, consider tools like “direnv” to manage environment switching, use secret management services for production environments, and define default values in the codebase with clear documentation.

Example approach with “.env.example” and “.gitignore”:

# .env.example (committed to version control)
DATABASE_URL=postgresql://devuser:devpassword@db:5432/devdb
REDIS_URL=redis://redis:6379/0
API_KEY=example_key_for_development

# .gitignore
.env

Real-world scenario: Full-stack web application

For a typical full-stack web application with a React frontend, Node.js API, and PostgreSQL database:

Dev Container approach:

• “.devcontainer/Dockerfile” with Node.js, PostgreSQL client tools
• “.devcontainer/devcontainer.json” with VS Code extensions for React, Node
• “docker-compose.yml” for PostgreSQL service

Cloud IDE approach:

• GitHub Codespaces configuration with the same Dev Container setup
• Environment variables set through the Codespaces secrets

Local only approach:

• “docker-compose.yml” with services for frontend, backend, and database
• Volume mounts for live code reloading

Hybrid approach:

• Dev Container for the development environment
• Docker Compose for service dependencies
• Task runner (“just” or npm scripts) for common commands
• Environment managed through “.env” files with “.env.example” templates

The best solution depends on your or your team’s specific needs, but containerized environments (either local or cloud-based) have been leveraged to ensure consistency and reduce onboarding friction for a while now.

The evolution from “works on my machine” to “works for everyone” represents more than just a technical advancement—it’s a fundamental shift in how we think about development environments. We’ve moved from treating environment setup as a necessary evil to embracing it as a core part of our development workflow.

Whether you choose local dev containers, cloud development environments, or a hybrid approach, the key is consistency and reproducibility. The days of spending hours debugging environment-specific issues are largely behind us, replaced by systems that ensure every developer on your team can be productive from day one.

The infrastructure choices you make today will determine how smoothly your team scales tomorrow. Choose tools that grow with your team and make the complex simple, not the simple complex.

I hope you found this post helpful, thanks for reading.