Class TokenizerOptions
Tokenizer configuration
public class TokenizerOptions
- Inheritance
-
TokenizerOptions
- Inherited Members
Properties
Arguments
Additional tokenizer arguments
[JsonPropertyName("args")]
public Dictionary<string, object> Arguments { get; set; }
Property Value
Name
Name of the tokenizer (e.g., "standard")
[JsonPropertyName("name")]
public string Name { get; set; }