Fly Self Driving.
Code on GitHub

How does it work

Pixels in. Steering out. A real fly’s wiring in between.

The driver is not a neural network designed for driving. It is the measured wiring diagram of a fruit fly, run as a recurrent network. A camera frame enters through the fly’s own optic-lobe neurons, travels through 25.56 million measured synapses, and comes out as a steering angle read from its leg motor neurons. Nothing we engineered can carry the signal on its own.

Model architecture

Camera frame to random projection to optic-lobe sensory neurons to the measured connectome, whose state carries between decisions, to motor neurons to frozen readouts for steering and speed, which move the car and produce the next frame.INPUTCamera frame64 × 32 grey · every 50 msENGINEERED · FROZENRandom pixel mapone pixel, one signper sensory neuronMEASURED · MALECNS CONNECTOME165,122 neurons · 25,563,197 synapseswhich neuron talks to which is fixed by the microscopeOptic lobe4,114 sensoryneuronsVentral cord708 motorneuronsLEARNEDone gain per synapse · one leak per neuronsigned rate state · 4 updates per decision25,728,319 trainable parametersstate carried to the next decisionthe only path from pixels to the wheelENGINEERED · FROZENSteering readoutfixed weights · tanhENGINEERED · FROZENSpeed readoutfixed weights · sigmoidspeed only in the crossers taskcar8 m/s bicycle modelthe car moves, the next frame is rendered
Solid boxes are measured, the dashed ones are engineered and frozen, and only the shaded core learns. The state loop is the change from the original recipe: the network is never reset between decisions.
165,122 neurons25.56 M synapses

What is measured, what is learned, what we engineered

Measured: which neuron connects to which. Learned: how loudly each connection speaks and how fast each neuron forgets. Engineered and then frozen: the random map from pixels to sensory neurons, the random readout from motor neurons to the wheel, and the car itself. The question underneath the whole project is whether the measured part does anything, so every result is paired with the same graph rewired at random.

165,122
neurons, all measured
25,563,197
connections, all measured
25.7 M
learned gains and leaks
4,114
sensory neurons see pixels
708
motor neurons steer
20 Hz
decisions, state carried

Six stages, from the camera to the wheel

Each stage is tagged by where it came from: measured by a microscope, learned by training, or engineered by us and then frozen.

  1. 01Engineered

    A 64 by 32 grayscale frame every 50 milliseconds

    The car has a single forward camera rendered by ray casting: road bright, verges dark, cars and crossers as blocks. That is the entire sensory world. No speed, no position, no map.

    Pixels
    2,048
    Decisions per second
    20
  2. 02Engineered, frozen

    Pixels land on the fly's optic-lobe sensory neurons

    Each of the 4,114 optic-lobe sensory neurons is wired to one random pixel with a random sign. The map is drawn once and never trained, so the network cannot learn a convenient camera; it has to make do with whatever its photoreceptors happen to see.

    Sensory neurons
    4,114
  3. 03Measured

    The male fruit-fly connectome, as a recurrent network

    Every traced neuron in the MaleCNS release is a unit with a signed rate state; every measured neuron-to-neuron connection is an edge. Which neuron talks to which is fixed by the microscope, not by training. Four graph updates run per decision, and the state carries over to the next one.

    That last point is the biggest departure from the original flyhard recipe, which wiped the state before every decision. A real fly never resets; its circuits hold heading and short-term memory. Carrying state is what let the same graph go from 3 of 20 roads to 17 of 20 on the first driving task, and it is what lets it remember a car it is overtaking after it leaves the frame.

    Neurons
    165,122
    Synaptic connections
    25,563,197
    Graph updates per decision
    4
  4. 04Learned

    One gain per synapse, one leak per neuron

    Training adjusts how loudly each measured connection speaks and how fast each neuron forgets. It never adds, removes or rewires a connection. That is 25.7 million parameters, but all of them are constrained to the fly's wiring diagram, which is why a randomly rewired graph with the same number of parameters is the control that matters.

    Trainable parameters
    25,728,319
  5. 05Engineered, frozen

    Steering is read from the leg motor neurons

    A fixed random weight vector over the 708 motor neurons of the ventral nerve cord produces the steering angle, through a tanh so it cannot exceed the wheel's range. In the crossers task a second fixed vector over the same neurons produces a target speed. The readout is never trained either: if the motor neurons do not carry the answer, nothing downstream can invent it.

    Motor neurons
    708
    Outputs
    steering, and later speed
  6. 06Engineered

    A kinematic bicycle at 8 metres per second

    The wheel slews at most 4 radians per second, so a lane change takes about a second whatever the brain asks for. The street is two lanes with buildings, lamp posts, parked cars intruding into the lane, oncoming cars, and a slow car ahead that has to be overtaken through the oncoming lane. Failure is any collision or leaving the road; a street counts only when all 500 decisions, 200 metres, are clean.