How concerned should we be about Astra's recurrent architecture?

Join the discussion

• Hacker News
• Looped transformers are an old idea, has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
  by BoredomIsFun
• there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
  by teravor
• I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.

  I must not have the right idea of what is happening here.

  by DarkByte
• I'm literally zero concerned.

  Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.

  You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.

  One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.

  That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?

  In my opinion, no more so than the residual stream of existing transformers. It changes zero.

  by kelseyfrog
• If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.
  by imtringued
• You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
  by dist-epoch
• A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
  by bulder
• >That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI? In my opinion, no more so than the residual stream of existing transformers. It changes zero.

  9.2.1 CoT Controllability

  We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]

  9.3 External Evaluation for Monitorability - UK AISI

  To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:

  No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]

  [1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...

  [2] https://deploymentsafety.openai.com/gpt-6-astra/external-eva...

  by famouswaffles
• Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.

  Somewhat unclear how particularly novel this is vs a way to save compute.

  by nighthawk454
• So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
  by mentalgear
• I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
  by thinking_cactus
• Seems conceptually connected to the "repeat yourself" hack that improves models by duplicating layers: https://dnhkng.github.io/posts/rys/
  by pu_pe
• Yep, an old idea, that has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
  by BoredomIsFun
• Related ongoing thread:

  OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395

  by dang
• I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
  by khalic
• Link to this study?
  by gr_norm
• How can you possibly believe this? If this were true, why do open weight models think in real English tokens and not arbitrary characters?
  by simianwords
• How can they do that? Does state propagate between each token output?

  That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?

  by thinking_cactus
• Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT
  by markasoftware
• The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
  by anon291
• CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
  by buppermint
• >In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory

  I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.

  You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.

  RNN xhat[t+1]=f(x[t],h[t])

  Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])

  by kjshsh123
• On further thought, I think the author's intent was to say that classic RNNs have "unbounded temporal accumulation in the hidden state".
  by kjshsh123
• Sebastian Raschka posted about this architecture:

  > A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".

  > It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.

  > About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."

  > Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.

  > In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.

  > In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.

  > Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)

  > While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.

  > In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.

  https://x.com/rasbt/status/2095141254958858496

  by anon373839
• I don't understand why he's citing a NeurIPS 2025 paper when cross-layer parameter sharing was introduced in ALBERT already in 2019.
  by KKKKkkkk1
• Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
  by zormino
• I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
  by samrus
• Why? I talk in my head and then enunciate only that which is relevant. My speech rate inside is incredibly fast.
  by anon291
• What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.
  by scotty79
• Chain of thought seems very specifically like an instantiation of the System 2 thinking of the System 1 and 2 thinking Daniel Kahneman popularized.

  Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?

  by big_toast