Comparing UchenML, TensorFlow.js and LiteRT.js: Language Detection

Benchmarks on my machine.

The native build is here only because I was curious how much WASM costs over machine code. Same C++, compiled for ARM Neon instead of WASM SIMD.

TensorFlow.js is the outlier - the model is small, so most of a call runs a general-purpose graph of string operations that split the text into byte pairs, hash them and add up the results, one JavaScript kernel at a time. Uchen does the same step as one short loop of compiled C++ over the first 10,000 byte pairs, then gets on with the model's own math.

LiteRT.js is Uchen's real peer: the same general approach, optimized kernels compiled to WASM SIMD. It runs only the numeric layers; its byte pairs are hashed in JavaScript before every call. It stays close on short input, and on the big file it falls behind mostly inside those numeric layers, not in the JavaScript. The bigger difference is flexibility: Uchen can't yet define a model from JavaScript — the graph is compiled into the C++.

The sizes follow from what each one ships. Uchen's single file is built for this one model. Nothing stops it from holding more, and they would share the code already there. TensorFlow.js and LiteRT.js ship general engines that run any model, and LiteRT's engine is most of its 9,670 KB. Uchen and TensorFlow.js can read the same uint8-quantized weights, while I had to convert the weights to fp32 for LiteRT, which made it almost three times the size of the TF.js files.

Ported in one prompt.

Moving the detector onto Uchen took a single prompt:

/goal Build a drop-in replacement NPM module for https://github.com/microsoft/vscode-languagedetection.git based on Uchen.

Uchen: @../uchen-core/ Use Dots @../../dots/ as an example.
1. Use Bazel with submodules (like Dots).
2. Fan out agents.
3. Do as little changes to Uchen as possible - new layers and such should stay in the new module to prove Uchen is extensible.
4. Add support for loading TFlow parameters from WASM, do not preconvert.

@"systems-architect (agent)" makes the decisions. Work autonomously.

Below is the network definition the agents came up with. It is a full copy of the original one, so the parameters work as is. As usual, it is a constexpr value with the forward pass configured at compile time. I did not train this model — the weights were downloaded from the VS Code GitHub repository.

inline constexpr size_t kEmbedding = 70;
inline constexpr size_t kHidden0 = 512;
inline constexpr size_t kHidden1 = 32;
inline constexpr size_t kParameterCount = 674604;

inline constexpr auto kModel =
    uchen::layers::FloatModel<kBuckets> | uchen::layers::Fork<2> |
    uchen::layers::Parallel(
        uchen::layers::Linear<kClasses>,
        layers::EmbeddingBagMean<kEmbedding> | uchen::layers::Linear<kHidden0> |
            uchen::layers::Relu | uchen::layers::Linear<kHidden1> |
            uchen::layers::Relu | uchen::layers::Linear<kClasses>) |
    uchen::layers::Join | uchen::layers::Reduce<uchen::PlusOp, 2> |
    uchen::layers::Logits;

static_assert(kModel.all_parameters_count() == kParameterCount);

What's running in the editor.

langguess-demo.js: 422.180 KB
model/model.json: 242.642 KB
model/model.json sha256: 100ce176367e7311e37ced0695057452991a8692029a79340a25e622893e7983
model/group1-shard1of1.bin: 715.908 KB
model/group1-shard1of1.bin sha256: fab6442698f64d5b1d2df052061d12bafd570330556819d29f48c7bcbb5889f7
in the background — langguess-editor.js: 536.955 KB
in the background — tfjs-race.js: 522.781 KB
in the background — litert-race.js: 28.647 KB
in the background — litert/litert_wasm_internal.js: 273.035 KB
in the background — litert/litert_wasm_internal.wasm: 9,367.934 KB
in the background — litert/guesslang_sparse.tflite: 2,702.260 KB
on request — large.ts.txt: 201.420 KB