Training recipe
Imitation first. Then its own mistakes.
The brain never sees a reward. It copies an expert that can see everything, then drives on its own while the expert marks what it should have done. Below: where the data comes from, the six steps, every setting, and what came out.
Data recipe
- 01Expert demonstrationsA pure-pursuit controller that reads the true state drives the training streets with noise injected into its executed steering, so the recordings include recovering from being off line.60 streets × 320 ticks ≈ 19,200 framesthe expert's steering at every state actually visited
- 02Overtake oversamplingOvertake windows are rare and decisive, so their frames are weighted three to one in every batch.weight 3× on overtake tickssame labels, higher sampling weight
- 03DAgger roundsThe brain drives the training streets on its own; the expert annotates what it should have done at every state the brain reached; training continues on the union of both sets.3 rounds, all 60 training streets eachexpert relabels of the brain's own states
- 04Held-out evaluationStreets generated from seeds the training never used, scored closed loop. A street counts only when all 500 decisions, 200 metres, are free of collisions and off-road events.3 sets × 20 streets × 500 ticksnone; pass or fail per street, failure type, lane offset
The six steps
- 01Demonstrations
An expert that can see everything drives 60 training streets
A pure-pursuit controller reads the true state, lane geometry, every car, every crosser, and drives with random noise injected into its steering so the recordings include getting back on line, not just staying on it. Overtake moments are oversampled three to one because they are rare and decisive.
- Training streets
- 60
- Ticks per street
- 320
- 02Behaviour cloning
Fit the gains and leaks to the expert's steering
Truncated backpropagation through time over windows of 16 decisions, batch 4, Adam at a learning rate of 0.04, mean-squared error on the normalised steering command. 1,200 updates, about 35 minutes on an H100 and 100 on an Apple M4.
This alone gave 12 of 20 held-out streets with traffic and a recognisable failure: timid steering of one or two degrees where the expert commanded eight to eighteen. Small errors compound, the car drifts, and it ends up somewhere the demonstrations never covered.
- Window
- 16 decisions
- Updates
- 1,200
- Held-out result
- 12 / 20
- 03DAgger
Let the brain drive, have the expert relabel its mistakes
Three rounds of DAgger: the connectome drives the training streets on its own, the expert annotates what it should have done at every state the brain actually reached, and training continues on the union. Five hundred updates a round. This closes the drift because the model now learns from its own errors rather than the expert's smooth path.
- Rounds
- 3 × 500 updates
- Held-out result
- 20 / 20
- 04Evaluation
Three independent sets of twenty streets it never saw
Each street is generated from a seed the training never used. One set is not enough, so the same checkpoint is scored on three, with and without traffic, and the lane offset is compared with the expert's on the same streets. Every number on this site is from held-out streets.
- Traffic
- 20 · 19 · 19 of 20
- No traffic
- 20 · 20 · 19 of 20
- Lane offset vs expert
- within 4 cm
- 05Controls
Baselines that see the same pixels, and a rewired fly
A linear map from pixels to steering and a small MLP with one hidden layer of 40 units are trained on the same demonstrations. Neither has memory. The rewired control keeps every neuron's number of inputs and the total edge count, and shuffles only which neuron connects to which, then trains with the identical recipe. It is the one comparison that asks whether the fly's specific wiring matters.
- Linear map
- 12 / 20
- MLP-40
- 18 / 20
- Rewired fly
- 16 / 20
- 06Second output
Add speed control, pedestrians and dogs
The same recipe with a target-speed readout, crossers that step off the kerb when the car is 26 to 38 metres away, and an expert that brakes for them. The fly completes 53 of 60 streets and the rewired graph collapses to 5 of 20. But the recorded episodes show the fly never brakes: crossers always cleared the road before it arrived, so the demonstrations barely contained a stop, and the one time a stop was needed it hit the pedestrian. The next run moves the step-off closer so stopping is required.
- Fly, three sets
- 18 · 19 · 16 of 20
- Rewired fly
- 5 / 20
- Minimum speed observed
- 7.1 m/s
Detailed recipe
Model
- Graph
- MaleCNS traced neurons, 165,122 nodes, 25,563,197 directed edges, adjacency frozen
- Neuron
- signed rate state with a learned leak; 4 graph updates per 50 ms decision; state carried between decisions
- Trainable
- 25,563,197 edge gains + 165,122 leaks = 25,728,319 parameters
- Sensory interface
- each of 4,114 optic-lobe neurons reads one random pixel with a random sign; pixels normalised to (value − 0.5) × 2 and clamped to ±2; frozen
- Motor interface
- fixed random weights over 708 VNC motor neurons; steering = 0.5 rad × tanh(·); target speed = 14 m/s × sigmoid(·) in the crossers task; frozen
Environment
- Camera
- 64 × 32 grayscale, ray cast against road, verges, buildings, cars and crossers; 20 frames per second
- Car
- kinematic bicycle, 2.6 m wheelbase, fixed 8 m/s (speed control in the crossers task), steering slew ≤ 4 rad/s, ±0.5 rad
- Street
- two 3.5 m lanes on a sinusoidal centreline, buildings and lamp posts, parked cars intruding into the lane, oncoming cars at 8 m/s, a slow car at 3 m/s to overtake; crossers step off the kerb at 26 to 38 m in the later task
- Failure
- off road, parked car, oncoming car, slow car, crosser; any one ends the street
Optimisation
- Objective
- mean-squared error between the model's steering and the expert's, normalised by the 0.5 rad range; speed added as a second term in the crossers task
- Backprop
- truncated through time over windows of 16 decisions, batch 4 streets
- Optimiser
- Adam, learning rate 0.04 on gains and leaks
- Schedule
- 1,200 cloning updates, then 3 DAgger rounds of 500 updates
- Hardware
- Apple M4 Mac mini for the first runs (about 100 min per run); NVIDIA H100 for the sweeps (about 35 min per run); total GPU rental under $25
Controls
- Linear map
- 2,048 pixels to steering, 1,153 parameters, same demonstrations, no memory
- MLP-40
- one hidden layer of 40 units, 46k parameters, same demonstrations, no memory
- Rewired fly
- presynaptic index permuted over all edges; every neuron keeps its in-degree and the edge count is unchanged; identical recipe
- Seeds
- one training seed per condition (a second seed of the crossers model scored 13 of 20); three independent street sets for the best models
What it produced: held-out streets completed
Twenty streets per set, none seen in training. Three numbers means the same checkpoint was scored on three independent sets. The baselines see the same pixels; the rewired fly is the control that asks whether the wiring matters.
- 01Expert (pure pursuit, reads the true state)20 / 20–20 / 20
- 02Steer straight0 / 20off road at 35 m0 / 20
- 03Linear map, pixels to steering (1,153 parameters)12 / 20oncoming, slow car20 / 20
- 04MLP, one hidden layer of 40 (46k parameters)18 / 20oncoming20 / 20
- 05Fly connectome, behaviour cloning only12 / 208 oncoming20 / 20
- 06Fly connectome + DAgger ×3, window 1620 · 19 · 19 of 202 off road20 · 20 · 19
- 07Same recipe, randomly rewired graph16 / 204 off road18 / 20
- 01Expert20 / 20–20 / 20
- 02Fly connectome, two outputs, DAgger ×318 · 19 · 16 of 201 off road, 2 parked, 2 oncoming, 2 crossers20 · 20 · 19
- 03Same recipe, second training seed13 / 206 in the wait-for-a-gap overtake–
- 04Same recipe, randomly rewired graph5 / 20largely fails to learn5 / 20
Read the failures too
It never brakes
In the crossers task the fly's speed never dropped below 7.1 m/s in any of 20 recorded episodes. The one time braking was genuinely needed it hit the pedestrian. The demonstrations almost never contained a stop, so the speed output learned to cruise.
The published demo was instructed steering, not driving
The flyhard videos that started this show a connectome holding a requested wheel angle while a scripted car drives through CARLA; its author says the policy has no camera input. We reproduced that wheel task on a Mac mini first, 100 of 100 held-out targets after training, before asking whether the brain could see.
Memoryless, the fly wiring lost to a linear map
On a flat road with obstacles, run with the state wiped before every decision as flyhard does, the connectome completed 3 of 20 roads and a 1,153-parameter linear readout did 5 to 9. Carrying the state took the same graph to 17 of 20.
One seed, twenty streets a set
The gap between the measured and rewired graphs rests on one training seed per condition and twenty streets per set. It is a gap, not a verdict. Nothing here is a claim about a fly's mind or about biological learning; what is measured is the topology, what is learned is how loudly each synapse speaks.