Open-sourcing circuit tracing tools
150 points by jlaneve 1 day ago | 20 comments- rob-olmos 1 day agoAnthropic employees Sholto Douglas & Trenton Bricken did an interview recently with Dwarkesh Patel, pieces here and there was about the circuit tracing insights.
https://www.dwarkesh.com/p/sholto-trenton-2 -- search the transcript for "circuit" for the quick bits.
Eg, "If you look at the circuit, you can see that it's not actually doing any of the math, it's paying attention to that you think the answer's four and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four."
- sanex 1 day agoThe conversation about this on Dwarkesh was interesting and I'm glad we're getting access to the tool.
https://open.spotify.com/episode/3H46XEWBlUeTY1c1mHolqh?si=L...
- jexp 1 day agoImported the graph json into Neo4j
Have fun
https://gist.github.com/jexp/8d991d1e543c5a576a3f1ee70132ce7...
- ofou 1 day agoIs this Garcon [1], or a new tool?
[1]: https://transformer-circuits.pub/2021/garcon/index.html
- e_ameisen 1 day agoHi, paper author here.
This is a new tool which relies on existing introspection libraries like TransformerLens (which is similar in spirit to Garcon) to build an attribution graph. This graph displays intermediate computational steps the model took to sample a token.
For more details on the method, see this paper: https://transformer-circuits.pub/2025/attribution-graphs/met....
For examples of using it to study Gemma 2, check out the linked notebooks: https://github.com/safety-research/circuit-tracer/blob/main/...)
We also document some findings on Claude 3.5 Haiku here: https://transformer-circuits.pub/2025/attribution-graphs/bio...)
- e_ameisen 1 day ago
- Eduard 1 day agothought this was about PCB tracing and was disappointed.
- dvh 1 day agoIf you only want to trace veroboards (stripboards) and not full blown PCBs I made a browser tool for that: https://github.com/dvhx/stripboard2schematic
- Workaccount2 1 day agoBy total coincidence I have a project at the prototype stage that I will be building (hopefully starting tonight) on a strip board. Thanks!
- Workaccount2 1 day ago
- 1wheel 1 day agoIt can be! Here's a circuit showing how the model processes "PCB tracing stands for" to output "printed":
https://www.neuronpedia.org/gemma-2-2b/graph?slug=pcb-tracin...
- Henchman21 1 day agoSame here! Then I immediately thought: I wish people would stop misusing words followed by I guess I think I’m in charge of words now. Then, idly: I’m starting to resemble that “Old Man Yells at Cloud” meme
Funny things, thoughts.
- AdamH12113 1 day agoYeah, I actually have a decades-old two-layer board that I need to reproduce and I would love to be able to feed images of it into some sort of tool and have it generate a schematic (or at least a netlist) automatically.
- duskwuff 1 day agoIt's not automatic, but one way I've seen people reverse-engineer PCBs (and ICs!) is to import scans of the subject in Kicad, then start tracing out the connections on screen.
- duskwuff 1 day ago
- Archit3ch 20 hours agoI was excited for a moment.
- buescher 1 day agoYou and me both. The reverse engineering tools are out there even if most of the search results are AI slop that recommends common layout tools. If I really needed the work done though I'd just pay one of the overseas services and clean up from there.
- tacker2000 1 day agohaha same here
- asadm 1 day agough same here.
- mrheosuper 1 day agoSame. Those AI bros keep stealing our terminology.
- forgotpwagain 1 day agothought this was about tracing neural circuits in the brain and was disappointed.
- dvh 1 day ago
- Tostino 1 day agoThis type of stuff is really important in my opinion. Getting this type of stuff open sourced allows academics and other researchers to try and do this type of interpretability research on a more level playing field.
I think the more people looking at this the better. I have a feeling there will be some breakthroughs in identifying important circuits and being able to make more efficient model architectures that are bootstrapped from some identified primitives.
- qtwhat 1 day agoCurious if we say "thank you", the model will be more activated and result in better answer. ^^