Skip to main content

Content Similarity Scorer

The createContentSimilarityScorer() function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.

Parameters​

The createContentSimilarityScorer() function accepts a single options object with the following properties:

ignoreCase:

boolean
= true
Whether to ignore case differences when comparing strings.

ignoreWhitespace:

boolean
= true
Whether to normalize whitespace when comparing strings.

This function returns an instance of the MastraScorer class. See the MastraScorer reference for details on the .run() method and its input/output.

.run() Returns​

runId:

string
The id of the run (optional).

preprocessStepResult:

object
Object with processed input and output: { processedInput: string, processedOutput: string }

analyzeStepResult:

object
Object with similarity: { similarity: number }

score:

number
Similarity score (0-1) where 1 indicates perfect similarity.

Scoring Details​

The scorer evaluates textual similarity through character-level matching and configurable text normalization.

Scoring Process​

  1. Normalizes text:
    • Case normalization (if ignoreCase: true)
    • Whitespace normalization (if ignoreWhitespace: true)
  2. Compares processed strings using string-similarity algorithm:
    • Analyzes character sequences
    • Aligns word boundaries
    • Considers relative positions
    • Accounts for length differences

Final score: similarity_value * scale

Examples​

High similarity example​

In this example, the response closely resembles the query in both structure and meaning. Minor differences in tense and phrasing do not significantly affect the overall similarity.

src/example-high-similarity.ts
import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";

const scorer = createContentSimilarityScorer();

const query = "The quick brown fox jumps over the lazy dog.";
const response = "A quick brown fox jumped over a lazy dog.";

const result = await scorer.run({
  input: [{ role: "user", content: query }],
  output: { text: response },
});

console.log(result);

High similarity output​

The output receives a high score because the response preserves the intent and content of the query with only subtle wording changes.

{
  score: 0.7761194029850746,
  analyzeStepResult: {
    similarity: 0.7761194029850746
  },
}

Moderate similarity example​

In this example, the response shares some conceptual overlap with the query but diverges in structure and wording. Key elements remain present, but the phrasing introduces moderate variation.

src/example-moderate-similarity.ts
import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";

const scorer = createContentSimilarityScorer();

const query = "A brown fox quickly leaps across a sleeping dog.";
const response = "The quick brown fox jumps over the lazy dog.";

const result = await scorer.run({
  input: [{ role: "user", content: query }],
  output: { text: response },
});

console.log(result);

Moderate similarity output​

The output receives a mid-range score because the response captures the general idea of the query, though it differs enough in wording to reduce overall similarity.

{
  score: 0.40540540540540543,
  analyzeStepResult: {
    similarity: 0.40540540540540543
  }
}

Low similarity example​

In this example, the response and query are unrelated in meaning, despite having a similar grammatical structure. There is little to no shared content overlap.

src/example-low-similarity.ts
import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";

const scorer = createContentSimilarityScorer();

const query = "The cat sleeps on the windowsill.";
const response = "The quick brown fox jumps over the lazy dog.";

const result = await scorer.run({
  input: [{ role: "user", content: query }],
  output: { text: response },
});

console.log(result);

Low similarity output​

The output receives a low score because the response does not align with the content or intent of the query.

{
  score: 0.25806451612903225,
  analyzeStepResult: {
    similarity: 0.25806451612903225
  },
}

Score interpretation​

A similarity score between 0 and 1:

  • 1.0: Perfect match – content is nearly identical.
  • 0.7–0.9: High similarity – minor differences in word choice or structure.
  • 0.4–0.6: Moderate similarity – general overlap with noticeable variation.
  • 0.1–0.3: Low similarity – few common elements or shared meaning.
  • 0.0: No similarity – completely different content.

On this page