<- back to blog / ryanhub

neural voxel engine

replacing analytic voxel physics with a tiny neural network

by Ryan Alport Jun 21, 2026 repo

static void move_axis(Player *p, int axis,
                      float amount) {
  float old = get_axis(p->pos, axis);
  set_axis(&p->pos, axis, old + amount);

  if (player_collides(p)) {
    set_axis(&p->pos, axis, old);
    if (axis == 0) p->vel.x = 0;
    else if (axis == 1) {
      if (p->vel.y < 0) p->grounded = 1;
      p->vel.y = 0;
    } else p->vel.z = 0;
  }
}

the question

can AI be used to completely replace a physics engine?

I say why the hell not.

this page is an examination of my process for implementing this idea from top to bottom while frequently stopping to analyze the results and why things look the way they do.

starting with the game

step one was to build our world. naturally, I chose C and Raylib for my stack here. we simulate a 16x16x16 cube where each cell can be one of 4 types

• air
• regular tile (grey)
• goal block (green)
• starting block (blue)

our character is the orange box

with a playable world, we can extract a state each time step and generate ourselves some training data to use in our model.

gamestate object and model architecture

the game executable has a headless data capture mode that grabs current inputs, current position (xyz), velocity (xyz), a grounded flag and the 9x9x9 voxels surrounding the player.

we trained and will examine a number of models but we will mostly focus on ones that used 3x3x3 voxel observation as input.

after a little testing, I determined that two layers is sufficient for this task while keeping the models very small and fast. we could push the fidelity with more layers but for now, two works. we will examine different layer sizes shortly...

rules for training, inference, and benchmarks

• pure neural at runtime. no analytic fallback or corrections ever
• inference is done in the main thread in C. no pytorch outside of training
• fixed dt = 1/60. same in recording, training, and inference
• analytic is the reference. all comparisons should be considered a multiple of the analytic physics
• one step validation is not enough. we must study the divergence across multiple frames to get a true benchmark

these rules are meant to keep this a super honest evaluation of the models we build. in some early generations, before clarifying these rules, Cursor found all kinds of ways to cheat and artificially inflate the benchmark results. I recommend really thinking through your success criteria and considering how models might cheat or mislead your benchmark results.

when training AI we always deal with the models themselves finding ways to misbehave but win on benchmarks, agents will do the same thing unless you build and enforce an intelligent system of rules.

training basics

training follows the standard supervised training pipeline. we gather data from the actual game's physics engine and use pytorch to train a MLP.

loss is computed using MSE on the seven outputs, velocity, position, and grounded, but, position error is multiplied by 20, velocity by 2, grounded by 5.

these numbers were tuned during our evolution in addition to modifications to the dataset to represent certain cases more or less.

we also aim to train across multiple frames of a single simulation world. for a single step it was fairly easy to push accuracy as high as 99%. across a rollout however, it is much more difficult and almost certainly impossible if we only see one frame at a time.

single step accuracy is still an important metric for ensuring training has not collapsed but it tells us very very little about how the neural physics will actually play.

first stable models

the first model worth taking seriously was models/model.bin trained to see just one step at a time and the full 9x9x9.

• input - 9x9x9 voxels, position, velocity, grounded flag, WASD+jump (741 floats)
• architecture - 128×128 MLP, ReLU, 7 outputs (Δpos, next velocity, next grounded)
• loss - one-step MSE per frame: position x20, velocity x2, grounded x5
• export - fp32 weights + normalization stats to model.bin

this looked great from the benchmarks, tiny positional error, 99% grounded accuracy but it played pretty bad...

this led to the development of the rollout benchmark. we needed to better measure how the game would actually play.

this shows a more accurate story, over 300 steps, we average 0.68 voxels off and 941 / 12475 (7.5%) of steps, we are inside a block and not grounded.

this matched the actual gameplay where we sink into terrain and jitter or slide when moving.

V2, 256x256 trained on 8-steps

in two steps, this iteration scaled the model up and implemented the multi step training I already detailed.

rollout training moved every metric, pos error 0.68 -> 0.46, tunnelling 941 -> 357, grounded mismatch cut nearly in half. just to illustrate again the invalidity of the one-step benchmarks, the val RMSE hardly moved. the cost of this new model was speed: ~153 µs/step vs ~66 µs for the smaller one, my focus was first to push the model to high fidelity, then optimize.

this was also a huge step for the gameplay quality, things were now pretty stable but not perfect, the jitter and phasing are still present especially around edge cases.

V3, better dataset coverage

models/model_rollout_v2.bin - same 256×256, 8-step rollout as v2, but trained on a modified training set.

I figured that a lot of the jitter and sliding when standing still was due to a lack of just standing in the dataset, despite real human gameplay having lots of idle time.

I also theorized that the small upward velocity correction when standing on the ground has some effect on the character bouncing constantly during gameplay.

I added direct idle gameplay coverage to the data collection spread, as well as a tweak when recording so that we did not report the micro adjustments when grounded.

I did not know what to expect as far as the effect on the benchmarks for these changes. that said, we see solid improvements on everything except for the tunneling

in gameplay this is also much stronger, still some sinking but this is very playable now.

At this point there were a few upgrades I thought about for the dataset along with a size increase but the training duration was far too long and beginning to hit memory limits on my laptop. I created a V3 dataset detailed in the chart below but we will save it for our next generation. down the next rabbit hole...

patch size sweep

one thing I thought we could very easily optimize was the patch size (number of voxels input) we give to the network at each step. this was making training super slow and theoretically, it makes very little sense that we would ever need to know anything about voxels besides the ones we could theoretically touch in this or the next frame.

so I went and trained 128x128 models at every patch size from 9x9x9 down to 2x2x2; (15 epochs, 200k samples).

the goal was just to evidence that we wouldnt have steep performance drop offs while we made each model exponentially smaller and easier to train. below are the somewhat interesting results.

I wanted to investigate why the error was not in line with my expectations. the amount of noise in the sweep led me to believe that there were some dials we could play with to at least find a real pattern here.

my main suspicion was that my initial sweep was underfitted so I bumped the training steps, I also increased the dimensionality of each model to ensure that wasnt the issue.

my current 'situation' does not include an easy to access graphics card so I only ran this for a few of these models to save some time, the results are below.

with this making sense, the fidelity being pretty damn good, and a known good recipe, we can move on to optimizations and pushing these models to their limit.

pushing the 3x3x3

I chose the 3x3x3 as the model to continue pushing. there are a few goals,

• see how fast we can go
• see how accurate we can get
• see how small we can make it

I set off a few training runs at different sizes and architectures, all using the v3 dataset, and here are the successful models.

models to highlight:

• patch3/256_rollout8_idle4.bin - fidelity anchor. 0.335 pos, 9.9% grounded mismatch, 231 tunnel steps. plays great.
• patch3/64_rollout4_fast.bin - speed floor. 0.476 pos at 5.7µs/step. this one still beats the old 9x9x9 baseline in accuracy.

final optimization push

there are a few things we can do in C to push inference speeds for all of our models without touching weights or retraining.

fused matmul + compiler optimizations

layer-1 normalized into a temp buffer then take a dot product with w1. we can fold normalization calculation right into that calculation to remove the entire whole pass and we can do it on the denormalization step as well. we also prefetch the next row with this cool macro:
#define NNPREFETCH(p) __builtin_prefetch((p), 0, 3)
of course we also flip the compiler to Release
-O3 -ffast-math -funroll-loops -march=native

this is basically free and it results in 72 µs → 11.9 µs (~6×) speed increase for the 256_rollout8_idle4 model.

AVX2 SIMD

this is over the top but I went ahead and implemented AVX2-FMA kernels: 8-wide FP32 dots with runtime dispatch via __builtin_cpu_supports.

I briefly tried VNNI int8 but rollout instantly broke and I could not fix it. real int8 speed probably needs QAT and integer dots. our post training quantization is not compatible, more on that in the next section.

verdict: absolutely worth doing the fused matmul pass, free speedup and not too much work. the AVX2 is questionable, more worthwhile the bigger your model is but is caused all kinds of issues and for the smaller models I was targeting, only a small speedup.

quantization

this part is pretty fun, we will consider the following five schemes: fp32, int8_row, int8_layer, int4_row, fp16.

we are quantizing after training here rather than during. from my research this does introduce potential for greater cumulative errors and I think this is the issue that prevented the VNNI tests from working.

final comparisons

this is every model we trained. the rollout benchmark runs 50 ep × 300 steps, post–AVX2 step times unless noted.

quantized versions are not shown.

idle4 anchor is probably the sweet spot ~10 µs, 0.335 pos, 304 KB.

4³–6³ idle4 retrains cluster right next to it (0.35–0.39 pos at similar speed).

64x64 is our observed speed floor (~1 µs) with high accuracy still.

512×256 is the accuracy ceiling on 3³.

9³ v2 loses on both axes we care about.

sweep models are noise, some score great, some bad, none actually useful.

building with cursor

I wrote a single spec.md and drove ~16 rough phases through Cursor agents.

agents are awesome and they push the speed of development unreasonably fast, but you need to stay on top of them. Cursor tried to cheat on the benchmarks, implement weird overrides, train models that made no sense, and otherwise just screw around.

the biggest benefit is the ability to come up with a hypothesis and tell cursor to go test it with N subagents. I can also ask for suggestions or benchmarking scripts or really anything I want and its basically done. I also would not have been able to generate all of the cute charts and graphs you see in this post without AI, too tedious for me...

keeping your agents on track and doing what you want is a skill, especially when you manage lots of them at a time. you need to write intelligent prompts as well as repo level rules to enforce your standards.

if I knew nothing about machine learning or physics engines, I dont think I could have directed any AI model through this thing. Taste and actual knowhow are still important skills in the age of AI and those who have those things and know how to intelligently use tools like Cursor or Claude Code will own the day.

whats next

I loved this project! Im probably not done with it yet and I think different architectures or better training could push the loss a lot lower.

to our initial question: "can AI completely replace a physics engine" I say yes. I do not say that it will be as performant or robust in all edge cases but it is not impossible. suppose a company that has millions of hours of training data in their game engine, there is a ton of potential there. (server sized GPU could do inference for an entire playerbase in batches. good enough and avoids expensive server side CPU physics, you could train an even cheaper classifier model to just detect people exploiting movement ... food for thought)

my neural physics engine is now definitely playable but also very quirky, sometimes you can spam jump forever or phase through the map, but on average, we can move and interact with the voxel world semi convincingly.

the idea that neural networks are universal function approximators is very interesting. for all tasks, including this one, there theoretically is a network that can perfectly approximate the algorithm. building a training pipeline is a whole other question but as shown here, it can be done.

a project Ive had in mind forever is to simulate a little terrarium of some sort and slowly swap out algorithmic systems for neural ones. imagine a mini food chain that follows rules but creature by creature, I swap in a tiny neural network. This all may even build to unique little world models that simulate every interaction in the terrarium one forward pass at a time. very cool!

if you found this interesting or have some idea about it you want to share, please email me @ [email protected] I love getting emails.

find the repo here