Searching English text in Spanish, and why "balompié" goes skiing

I have a small database of 52 short English sentences. “How to bake bread at home”. “Mountain biking in the Dolomites”. “I miss my family”.

I searched it for “cómo hacer pan”, which is Spanish for “how to make bread”. There isn’t a word of Spanish in the database, and the first result was “How to bake bread at home”.

Then I searched for “balompié”, a formal Spanish word for football. The first result was “Skiing in the Pyrenees”. The sentence about football came 9th out of 52.

Same database, same code. This post looks at why the first search worked and the second didn’t, using 2 models: OpenAI’s text-embedding-3-small, and Qwen3-Embedding-8B running on my own machine. You don’t need to know how either works to follow along.

Embeddings in 3 paragraphs

An embedding model turns a piece of text into a long list of numbers, called a vector. OpenAI’s model gives back 1536 numbers for any text you send it, whether it’s 1 word or a paragraph.

The model is trained so that texts with similar meanings get similar lists of numbers. “How to bake bread” and “a recipe for a loaf” end up close together. “How to bake bread” and “renting a car” end up far apart.

To search, you turn the search text into a vector too, then compare it with every stored vector and sort by how close they are. Closeness is measured with a similarity score. Identical texts score 1, unrelated texts score close to 0, and a good match in this post scores somewhere around 0.4 to 0.85.

The setup

The demo is a small Node.js server with a web page in front of it. The sentences live in SQLite, the database that sits in a single file. An extension called sqlite-vec adds vector search to it. Node 24 has SQLite built in and can load extensions, so there’s nothing extra to install or compile:

import { DatabaseSync } from 'node:sqlite';
import * as sqliteVec from 'sqlite-vec';

const db = new DatabaseSync('embeddings.db', { allowExtension: true });
sqliteVec.load(db);

db.exec(`
  CREATE TABLE IF NOT EXISTS documents (
    id    INTEGER PRIMARY KEY,
    topic TEXT NOT NULL,
    text  TEXT NOT NULL
  );

  CREATE VIRTUAL TABLE IF NOT EXISTS vec_documents USING vec0(
    embedding float[1536] distance_metric=cosine
  );
`);

There are 2 tables. documents holds the text. vec_documents holds each sentence’s vector: 1536 float32 numbers, which is 6,144 bytes per sentence. Both rows share an id, so a vector can always be matched back to its sentence. Qwen3 gets a second vector table of its own, which comes up later.

To fill the tables, I send all 52 sentences to OpenAI in one request and store what comes back:

const res = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: documents.map((d) => d.text),
});

// sqlite-vec stores a vector as raw bytes
const toBlob = (f32) => new Uint8Array(f32.buffer, f32.byteOffset, f32.byteLength);

db.exec('BEGIN');
res.data.forEach(({ embedding }, i) => {
  const id = BigInt(i + 1); // vec0 needs an integer rowid
  insertDoc.run(id, documents[i].topic, documents[i].text);
  insertVec.run(id, toBlob(new Float32Array(embedding)));
});
db.exec('COMMIT');

A search is one SQL query. You pass in the search text’s vector, and k says how many results you want back:

SELECT d.id, d.text, v.distance
FROM vec_documents v
JOIN documents d ON d.id = v.rowid
WHERE v.embedding MATCH ? AND k = ?
ORDER BY v.distance

sqlite-vec reports a distance, where smaller means closer. The similarity score is 1 - distance, and that’s the number used everywhere below.

The sentences are short on purpose, one idea each, so every search has one obvious right answer. They cover sport, cooking, travel, weather, health, home life and tech. A few are traps. “I am pregnant” sits next to “I feel embarrassed”, and “Buying books at a bookshop” sits next to “Borrowing books from the library”. If you speak Spanish, you can probably guess why. We’ll get to them.

What the model reads

A model can’t read letters. Before anything else, a tokenizer cuts the text into small pieces called tokens, and swaps each piece for a number (its id). A token can be a whole word, part of a word, or a single letter.

OpenAI’s model uses a tokenizer called cl100k_base. I checked this by comparing its token counts with the counts OpenAI’s API reports, and they matched for every test string. You can run the same tokenizer yourself with js-tiktoken:

import { getEncoding } from 'js-tiktoken';

const enc = getEncoding('cl100k_base');
enc.encode('montar en bici en las montañas').map((id) => [id, enc.decode([id])]);
// [[21047, 'mont'], [277, 'ar'], [665, ' en'], [293, ' b'], [3457, 'ici'], …]

One detail before the figure. The space in front of a word is part of the token. Spaces are invisible, so I draw that space as a dot. ·pan means ” pan” with a space (token 7363), and that’s a different token from pan without one (token 857).

Pick a pair to see how each side gets cut up:

3 things stand out.

Spanish takes more tokens. The 8 Spanish searches I use later add up to 49 tokens, and their English translations to 28. That’s 1.75 times as many. My guess is that the tokenizer was built mostly from English text, so Spanish words get chopped into smaller pieces.

Matching doesn’t depend on shared tokens. “montañas” looks a lot like “mountain” to us, but the model only ever sees the numbers. Mountain is token 92269 and mont is 21047, which have nothing in common. The same goes for fútbol (4 pieces) and ·football (1 piece). The model learned that these mean the same thing during training, by seeing them used in similar sentences millions of times.

The one shared token is misleading. ·pan is token 7363 in “cómo hacer pan” and also in “Frying eggs in a pan”. To the model, Spanish bread and an English frying pan start out as the exact same token, and it has only the 3 tokens around it to tell them apart.

What comes back

The model reads those tokens and returns its 1536 numbers. For “montar en bici en las montañas” they start 0.0338, -0.0077, 0.0092, 0.0017, …, and they’re all between −0.084 and 0.086.

Comparing 2 vectors is simple arithmetic. Multiply the 2 first numbers together, then the 2 second numbers, and so on for all 1536 positions, then add the results up. That sum is the similarity score. (OpenAI scales every vector so this shortcut gives the proper score.)

You can think of each of those 1536 products as a tiny vote for or against a match. Here are all of them for the bike search, against the sentence it should find and against one it shouldn’t:

No single position means “bike”, “mountain” or “Spanish”. The biggest single vote is 0.006, and 454 of the 1536 positions vote against the right answer. A match is a pattern spread across hundreds of positions.

Reading a ranking

From here until the section on the second model, every score comes from OpenAI’s text-embedding-3-small.

A search normally shows you the top few results. It’s more useful to look at all 52 scores at once. In the figure below, each dot is one sentence, and the big blue dot is the one the search should find.

The shaded band runs from the lowest score to the highest. I’ll call its width the spread. Keep an eye on it, because it turns out to be the most useful number in this post.

Every one of these finds the right sentence first. Every Spanish version also scores lower than its English version. “how to bake bread” gets 0.841, and “cómo hacer pan” gets 0.536 for the same sentence.

The less the Spanish looks like the usual way of saying it, the lower the score. “montar en bici en las montañas” gets 0.581, “andar en bicicleta por la sierra” 0.448, and “pedalear por el monte” (pedal through the hills) 0.415. All 3 still come first, but by less each time. The first one beats “Snow in the mountains” by 0.090, and the last one by only 0.040.

“cómo hornear pan” swaps hacer (make) for hornear (bake). The right sentence goes up a little, to 0.550, and “Frying eggs in a pan” jumps to second place at 0.419. Baking plus ·pan sounds like kitchen equipment.

Where it falls apart

Spanish has many words for football. Some are everywhere, and some are only used in one country.

Fútbol was borrowed from English, and Spanish text uses it all the time. It comes first at 0.477.

Balompié means the same thing. It’s built from balón (ball) and pie (foot), and you’ll mostly see it in club names like Real Betis Balompié. It puts the football sentence 9th. Its top 3 are skiing, mountain biking and climbing, so the model understood “some kind of sport” and no more.

Add a few words around it and it recovers. “jugar al balompié” (play balompié) comes first at 0.381, and “balompié con amigos” (with friends) comes first at 0.465. The extra words give the model something it recognises.

Regional slang is where it breaks. Fulbito is what Argentines and Uruguayans call a casual game. Its spread is only 0.15, meaning every sentence scored somewhere between 0.02 and 0.17. When the scores are that close together, the order is close to random. That’s how “Mountain biking in the Dolomites” ends up first and football 17th. Mexico’s pambol (football) and cascarita (a pickup game) do no better. Their top results are “Cooking paella with seafood” and “I feel embarrassed”.

So what matters is how often the model saw a word during training. Spanish is one of the most common languages on the internet, and that didn’t help with these 3.

This is why the spread matters. When a search works, the right sentence pulls away from the rest and the spread is wide, between 0.35 and 0.8 in these examples. When the model doesn’t know the word, the top score stays below about 0.25, everything bunches up underneath, and the first result is noise that still comes with a score attached.

The false friends

Now the 2 traps from the setup. Embarazada means pregnant, but looks like “embarrassed”. Librería means bookshop, but looks like “library”. Both catch out English speakers learning Spanish, and the database has the real meaning and the lookalike side by side.

OpenAI’s model gets both right. “estoy embarazada” finds “I am pregnant” at 0.668, and “I feel embarrassed” comes 4th at 0.276, behind 2 sentences about being ill. The tokens don’t overlap. The Spanish is ·embar + az + ada, and ·embarrassed is a single token.

The library is closer. “voy a la librería” (I’m going to the bookshop) puts the bookshop first at 0.486 and the library second at 0.415. Some of that is fair, because both sentences are about books, and 3rd place is far behind at 0.334. This test can’t tell how much of the 0.415 comes from the lookalike spelling. The Spanish does start with ·lib, like the English word, though it’s a different token.

The language penalty

Even when a Spanish search finds the right sentence, it scores 0.09 to 0.30 lower than the same search in English. (I’m leaving out the 2 English searches that are exact copies of a sentence, since those score a perfect 1.)

Part of that is meaning, because a translation never matches perfectly. But part of it is something else. Every vector also carries a signal that says which language the text is written in.

You can find that signal with 8 pairs of translations. Average the 8 Spanish vectors, average the 8 English ones, and subtract. What’s left points from “English” toward “Spanish”. Then you can measure how far along that line any vector sits, or remove it:

// average(Spanish) − average(English), then scaled to length 1
const dir = new Float32Array(dims);
for (const [es, en] of pairs) {
  for (let i = 0; i < dims; i++) dir[i] += (es[i] - en[i]) / pairs.length;
}
const len = Math.hypot(...dir);
for (let i = 0; i < dims; i++) dir[i] /= len;

// how "Spanish" a vector is
const along = (v) => v.reduce((s, x, i) => s + x * dir[i], 0);

// the same vector with the language signal taken out
const removeLanguage = (q) => {
  const k = along(q);
  return q.map((x, i) => x - k * dir[i]);
};

To keep the test fair, each pair is checked against a signal built from the other 7 pairs, never from itself.

With OpenAI’s model, every Spanish search lands between +0.23 and +0.29 on this line. The English sentences in the database average −0.06. So every Spanish search starts on the far side from everything it’s compared with.

Removing the language signal raises the Spanish scores by 0.02 to 0.07. The order of the results stays the same, and the gap to English gets smaller without closing.

This matters in practice. A rule like “only show results that score above 0.5” hides correct Spanish results. “andar en bicicleta por la sierra” finds the right sentence with 0.448, and that rule would throw it away.

A second model

Qwen3-Embedding-8B is a free, open model built for searching in many languages. I ran it on my own computer, an M5 Mac with 32 GB of memory, using LM Studio, an app for running models locally.

The file I used is Qwen3-Embedding-8B-Q4_K_M.gguf. GGUF is a common file format for local models, and Q4 means the model was compressed to around 4 bits per number, which brings it down to 4.68 GB. LM Studio offers the same kind of API as OpenAI, so the same code works with a different address:

import OpenAI from 'openai';

const qwen = new OpenAI({ baseURL: 'http://localhost:1234/v1', apiKey: 'lm-studio' });
const res = await qwen.embeddings.create({
  model: 'text-embedding-qwen3-embedding-8b',
  input: texts,
});

Turning all 52 sentences into vectors took about 6 seconds. 3 things work differently from OpenAI.

The vectors are longer. Qwen3 gives back 4096 numbers, so it gets its own table. Vectors from 2 different models can’t be compared with each other anyway, and sqlite-vec refuses if you try to search a 4096-number table with a 1536-number vector.

export const MODELS = {
  openai: { model: 'text-embedding-3-small', dims: 1536, table: 'vec_documents', queryPrefix: '' },
  qwen3: {
    model: 'text-embedding-qwen3-embedding-8b',
    dims: 4096,
    table: 'vec_documents_qwen3',
    queryPrefix: 'Instruct: Given a search query, retrieve relevant passages that answer the query\nQuery: ',
  },
};

Searches get an instruction in front. Qwen3 was trained with a short task description before every search text, and none before the stored sentences. That’s the queryPrefix above, and it only goes on searches.

The model needs an end marker. Qwen3 builds its vector from a special token, <|endoftext|>, at the very end of the text. If the software running the model forgets to add it, you still get a vector, just a worse one, and nothing tells you. The model file says whether the marker gets added, and you can read that setting yourself:

import { gguf } from '@huggingface/gguf';

const { metadata } = await gguf(modelPath, { allowLocalFile: true });
metadata['qwen3.embedding_length'];       // 4096 numbers per vector
metadata['qwen3.pooling_type'];           // 3 = build the vector from the last token
metadata['tokenizer.ggml.add_eos_token']; // true = the end marker gets added

Both settings are right, so the app can send plain text.

The tokenizer is the same as OpenAI’s, near enough. The Qwen technical report says Qwen started from cl100k_base and added to it. For every example in the token figure above, both tokenizers cut the text in exactly the same places, and only the id numbers differ. (Flip the toggle in that figure to see it.) Whatever Qwen3 does differently, it starts from the same pieces.

Here are the hard searches with both models side by side. Each strip shows all 52 sentences, and the blue mark is the right one:

Qwen3 handles the rarer words. Fulbito goes from 17th to 1st at 0.441, and balompié from 9th to 1st. “balompié con amigos” scores 0.614, compared with OpenAI’s 0.465. Reworded searches score higher too. “pedalear por el monte” gets 0.538 and a lead of 0.073, where OpenAI gave it 0.415 and a lead of 0.040.

Then it falls for the library trap. OpenAI put the bookshop first (0.486 against 0.415), and Qwen3 picks the library (0.543 against 0.519). Embarazada still works, with “I am pregnant” first at 0.647, but “I feel embarrassed” moves up from 4th to 2nd. Estoy constipado (I have a cold) still comes first, but only just, with “I am pregnant” and “I feel embarrassed” close behind.

The Mexican slang doesn’t work either. Qwen3 puts pambol with the food, with paella, pasta and tomato soup on top and football 29th. Cascarita puts football 52nd, dead last. My guess is that it reads the word like cáscara, a peel or a shell, which would explain the soup.

The library result worries me more than the slang. When the model doesn’t know a word, the spread is narrow and you can catch it. Qwen3’s library answer has a wide spread of 0.39, which looks just like a correct answer.

Also look at where the marks sit. Qwen3’s lowest score on any search is 0.09, while OpenAI’s goes below zero. The 2 models use the score range differently, so a 0.4 from one doesn’t mean the same as a 0.4 from the other. When comparing models, compare the order of the results.

The language signal is weaker in Qwen3 as well. Switch the language figure to Qwen3 and the Spanish searches sit at +0.07 to +0.19 instead, and removing the signal changes scores by 0.010 at most. The gap that’s left is probably real differences in meaning between the languages.

What I’d take from this

If you’re building search for people who type in more than one language:

• Don’t use one minimum score for every language. With text-embedding-3-small, a Spanish search loses 0.02 to 0.07 to the language signal alone.
• Check the spread, or how far first place is ahead of the rest, before treating a result as a match. Here, anything with a spread under about 0.2 was a guess.
• Test the words your users type. Fútbol worked and balompié didn’t, and they’re the same language. Collect real searches, including formal, regional and slang words, and try those.
• A bigger model that’s built for many languages fixed the rare words, but got a false friend wrong that the smaller model got right. Test your own list with both before you switch.
• Give each model its own table. The vector sizes differ, the scores mean different things, and a stored vector from one model is useless to the other.

Before trusting any of this on a real product, I’d want 2 more things: a database of thousands of sentences instead of 52, and a few hundred real searches instead of the ones I made up. The ranking figures would be the first thing I’d look at again.