Superpowers 6

blog.fsck.com192 pointsby seahorseemoji77 comments

Discussion summary

Discussions about Superpowers 6 highlight concerns over lack of benchmarking and subjective evaluation methods. Some users compare it to other tools like obra's Superpowers and GSD, with mixed opinions on its effectiveness.

What the discussion says

  • Critics argue the product lacks proper benchmarks and relies on subjective assessments.
  • Some users suggest it may perform better with open-source models rather than proprietary ones.
  • Others believe the concept of convincing LLMs to improve is often ineffective.
I can't take this product seriously when they don't run benchmarks.
johnfn
This is about that, I believe.
probablycorey

Comments

Hacker News

All these prompt and skill based git repos are sus... nothing is benchmarked -its all so subjective and unproven and breaks with model updates -everyone and his uncle has a 'secret sauce skill' -that just proves to me the subjectivity of this endeavor.

by AIorNot

This is about that.

by probablycorey

To be blunt I can't take this product seriously when they don't even run benchmarks. Your prompts make Claude better? Cool: prove it. Methods to evaluate LLM performance exist, they're called evals/benchmarks, and every company that is serious about AI runs them when they release a new version. (Of course benchmarks have their own issues, but squabbling over which benchmark is best and what issues there are is step 2 in being a Serious AI Company and step 1 is running them at all!) The fact that the only proof they have that 6 is better than five is a hacky table in a screenshot from Fable is, honestly, concerning.

by johnfn

i just dont find skills work flow all that generic enough.

by cyanydeez

seems like this would perform better with cheap open-source models on OpenCode compared to proprietary models like Claude Code or Codex.

by byzantinegene

Superpowers is pretty much convincing LLMs they can do better. It almost never works that way.

by RomanPushkin

Anyone have an opinion comparing this to GSD?

by mortsnort

I've loved Superpowers right along. I think a lot of what it does has been ingested into Claude Code proper now so I'll be interested to see if this release actually changes things up.

by SoMomentary

So far SDD (Spec Driven Development) with openspec hit the right balance for me, the Workflow is not too heavy while execution still churning good result given the spec is done well.

by wejick

I've been using Compound Engineering for the past few months too, but now that I'm seeing mention of how Superpowers consumes a lot of tokens and context, I wonder if Compound Engineering does the same.

I often end my Claude Opus 1M sessions at around 60-80% context, and that's with doing `/context` once in a while, and forcing the agent to wrap up and write handover notes, so that I can start on a new unit with fresh context.

by ValentineC

This is great in concept but what prevents me from using it is TDD. I don't want to waste tokens on producing code that doesn't ship to the end user. Design by Contract is a far superior approach. If you've never heard of Design by Contract I don't blame you, our culture really failed to bring it mainstream. But I swear by it and it gives me real superpowers. Maybe I should fork this and gut the TDD part and replace it.

by mempko

What programming language are you using for Design by Contract?

by simonw

Nothing about Superpowers forces you to use TDD, brainstorm first, etc. It’s not rigid about the workflow.

by CharlesW

I think I’ve mostly found superpowers helpful, especially for TDD. It’s cool from this blog (and their GitHub) that they’ve verified it in various ways too. One issue is that I’ll sometimes see it wanting to write a whole doc for a follow on feature with a similar structure and have to tell it to just use the last one again.

The most annoying thing is that it always pauses before implementing to ask if I want it to use Subagents (which it always recommends) or not.

by mcintyre1994

This is the point where I usually start a new session with a fresh context window and invoke the sub-agent development skill and just point to the spec+plan.

I’d be curious to see if there was a way to automatically do this for me in Claude Code.

by lucrum

I feel like all these fat skills on top of agents will become stale very quickly. Unless it is for a very specific workflow with a need for deterministic outputs, I just don't see them having high value.

If I do need such workflows I just use plan mode, and it is 90% sufficient. I created a skill that hooks on top of plan mode because of its shortcomings, but I'm pretty sure even this will become obsolete soon as models improve.

https://github.com/oliver-im/jidoka

by imgyuri

My use case is I have them installed and let Claude decide when to use them. Looks like for my recent sessions it has been using superpowers:test-driven-development 5%, and superpowers:subagent-driven... 1%. I haven't really been working on new projects this past week though which seems to be where they fire off the most, in particular the "writing a plan" one.

by linsomniac

The cool thing about superpowers is it's built using evals rather than just vibes.

by ra

I can't believe that a bunch of Markdown files now comes with a "Commercial Services" section. It feels like an elaborate GitHub Karma farm. Everything has to be commercialized and advertised.

by YuukiRey

I gave Superpowers 5.x a whirl for a week, and aside from consuming a stupid amount of tokens, it did materially worse across all my personal benchmarks and general day-to-day development compared to plain Codex/Claude. I'm convinced it's either some 4D ploy by the AI cartels to set tokens ablaze, or it only provides Superpowers to those without any power to begin with. Rating: 1/5 Pinocchios. Would not recommend.

by artisin

Same here. I tried something similar to Superpowers and it went completely overboard for a small bug fix - writing a TDD, generating artifacts, etc.

by mattm

I only use superpowers when I want to stick with composer2.5(fast) for everything. If I use it with other models it's terrible. With composer2.5 it is slightly better, though not much.

by jorl17

  The model 
    |
  The harness
    |
  The harness of the harness?!!
This structure doesn't make any sense to me. It's like adding a half leather half shag wrap to your steering wheel. Not to mention the harness itself is updated almost multiple times daily. I'm sure these framework authors are keeping tabs on the harness of the harness performance for every release of the harness.

by smrtinsert

And the fact that this article’s story is basically “I prompted Fable with a goal and went to sleep and the model got it done” is telling me that the latest models have gotten past the need for Superpowers… even the creators of superpowers is just using a simple /goal!

by jannyfer

I found it slowed me down significantly at first, and produced more verbose code. After a few weeks of using it, I think I've gotten used to it (sometimes I explicitly bypass it, but it's good enough to know which skill to use).

Yeah on the token consumption, I'll be doing something small at work, and it'll consume a lot of tokens.

by nevi-me

It sounded like it might not hurt and seemed endorsed by Claude and codex because they both had plugins for it by default BUT I ripped it out after I kept seeing Claude/codex TDD things like when I asked them to make a pydantic model immutable. I’d end up with unit tests testing that my immutably configured pydantic model was immutable or tests that setting foo=bar was actually foo=bar in app config.

by steve-atx-7600

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

  • Hacker News
  • All these prompt and skill based git repos are sus... nothing is benchmarked -its all so subjective and unproven and breaks with model updates -everyone and his uncle has a 'secret sauce skill' -that just proves to me the subjectivity of this endeavor.
    by AIorNot
  • by devnonymous
  • I thought this would be about https://github.com/obra/superpowers
    by rahimnathwani
  • This is about that.
    by probablycorey
  • To be blunt I can't take this product seriously when they don't even run benchmarks. Your prompts make Claude better? Cool: prove it. Methods to evaluate LLM performance exist, they're called evals/benchmarks, and every company that is serious about AI runs them when they release a new version. (Of course benchmarks have their own issues, but squabbling over which benchmark is best and what issues there are is step 2 in being a Serious AI Company and step 1 is running them at all!) The fact that the only proof they have that 6 is better than five is a hacky table in a screenshot from Fable is, honestly, concerning.
    by johnfn
  • To be blunt you should perhaps read the README before being condescending and dismissive. https://github.com/prime-radiant-inc/superpowers-evals
    by devnonymous
  • i just dont find skills work flow all that generic enough.
    by cyanydeez
  • seems like this would perform better with cheap open-source models on OpenCode compared to proprietary models like Claude Code or Codex.
    by byzantinegene
  • Superpowers is pretty much convincing LLMs they can do better. It almost never works that way.
    by RomanPushkin
  • Anyone have an opinion comparing this to GSD?
    by mortsnort
  • I've loved Superpowers right along. I think a lot of what it does has been ingested into Claude Code proper now so I'll be interested to see if this release actually changes things up.
    by SoMomentary
  • So far SDD (Spec Driven Development) with openspec hit the right balance for me, the Workflow is not too heavy while execution still churning good result given the spec is done well.
    by wejick
  • I’m a big fan of the Compound Engineering plugin from Every. As an amateur developer it helps me brainstorm, plan and implement apps very well.

    https://github.com/everyinc/compound-engineering-plugin

    by michelb
  • I've been using Compound Engineering for the past few months too, but now that I'm seeing mention of how Superpowers consumes a lot of tokens and context, I wonder if Compound Engineering does the same.

    I often end my Claude Opus 1M sessions at around 60-80% context, and that's with doing `/context` once in a while, and forcing the agent to wrap up and write handover notes, so that I can start on a new unit with fresh context.

    by ValentineC
  • This is great in concept but what prevents me from using it is TDD. I don't want to waste tokens on producing code that doesn't ship to the end user. Design by Contract is a far superior approach. If you've never heard of Design by Contract I don't blame you, our culture really failed to bring it mainstream. But I swear by it and it gives me real superpowers. Maybe I should fork this and gut the TDD part and replace it.
    by mempko
  • What programming language are you using for Design by Contract?
    by simonw
  • Nothing about Superpowers forces you to use TDD, brainstorm first, etc. It’s not rigid about the workflow.
    by CharlesW
  • I think I’ve mostly found superpowers helpful, especially for TDD. It’s cool from this blog (and their GitHub) that they’ve verified it in various ways too. One issue is that I’ll sometimes see it wanting to write a whole doc for a follow on feature with a similar structure and have to tell it to just use the last one again.

    The most annoying thing is that it always pauses before implementing to ask if I want it to use Subagents (which it always recommends) or not.

    by mcintyre1994
  • This is the point where I usually start a new session with a fresh context window and invoke the sub-agent development skill and just point to the spec+plan.

    I’d be curious to see if there was a way to automatically do this for me in Claude Code.

    by lucrum
  • I feel like all these fat skills on top of agents will become stale very quickly. Unless it is for a very specific workflow with a need for deterministic outputs, I just don't see them having high value.

    If I do need such workflows I just use plan mode, and it is 90% sufficient. I created a skill that hooks on top of plan mode because of its shortcomings, but I'm pretty sure even this will become obsolete soon as models improve.

    https://github.com/oliver-im/jidoka

    by imgyuri
  • My use case is I have them installed and let Claude decide when to use them. Looks like for my recent sessions it has been using superpowers:test-driven-development 5%, and superpowers:subagent-driven... 1%. I haven't really been working on new projects this past week though which seems to be where they fire off the most, in particular the "writing a plan" one.
    by linsomniac
  • The cool thing about superpowers is it's built using evals rather than just vibes.
    by ra
  • I can't believe that a bunch of Markdown files now comes with a "Commercial Services" section. It feels like an elaborate GitHub Karma farm. Everything has to be commercialized and advertised.
    by YuukiRey
  • I gave Superpowers 5.x a whirl for a week, and aside from consuming a stupid amount of tokens, it did materially worse across all my personal benchmarks and general day-to-day development compared to plain Codex/Claude. I'm convinced it's either some 4D ploy by the AI cartels to set tokens ablaze, or it only provides Superpowers to those without any power to begin with. Rating: 1/5 Pinocchios. Would not recommend.
    by artisin
  • Same here. I tried something similar to Superpowers and it went completely overboard for a small bug fix - writing a TDD, generating artifacts, etc.
    by mattm
  • I only use superpowers when I want to stick with composer2.5(fast) for everything. If I use it with other models it's terrible. With composer2.5 it is slightly better, though not much.
    by jorl17
  •   The model 
        |
      The harness
        |
      The harness of the harness?!!
    
    This structure doesn't make any sense to me. It's like adding a half leather half shag wrap to your steering wheel. Not to mention the harness itself is updated almost multiple times daily. I'm sure these framework authors are keeping tabs on the harness of the harness performance for every release of the harness.
    by smrtinsert
  • And the fact that this article’s story is basically “I prompted Fable with a goal and went to sleep and the model got it done” is telling me that the latest models have gotten past the need for Superpowers… even the creators of superpowers is just using a simple /goal!
    by jannyfer
  • I found it slowed me down significantly at first, and produced more verbose code. After a few weeks of using it, I think I've gotten used to it (sometimes I explicitly bypass it, but it's good enough to know which skill to use).

    Yeah on the token consumption, I'll be doing something small at work, and it'll consume a lot of tokens.

    by nevi-me
  • It sounded like it might not hurt and seemed endorsed by Claude and codex because they both had plugins for it by default BUT I ripped it out after I kept seeing Claude/codex TDD things like when I asked them to make a pydantic model immutable. I’d end up with unit tests testing that my immutably configured pydantic model was immutable or tests that setting foo=bar was actually foo=bar in app config.
    by steve-atx-7600

Related stories