Skip to content

Overview

Polyglot is built on a modular, layered architecture that separates concerns and promotes extensibility. Each layer has a clear responsibility, and dependencies flow in one direction -- from the public API down to the HTTP transport.

Understanding these layers will help you extend the library, contribute to its development, or build your own integrations with new LLM providers.

The Four Layers

Public Layer

This is what application code usually touches. Two facade classes provide a unified interface for all provider interactions:

  • Inference -- for chat completions and text generation
  • Embeddings -- for generating vector embeddings

These facades build request objects, delegate execution to runtimes, and return normalized responses regardless of the underlying provider. Both facades follow an immutable, fluent interface pattern -- every method that modifies state returns a new instance, so you can safely branch configurations from a shared base.

Runtime Layer

Runtimes assemble the moving parts needed for a provider call. They wire together the configuration, driver, HTTP client, and event dispatcher, and they own the execution lifecycle including retry logic and response caching.

The key classes are:

  • InferenceRuntime -- coordinates inference execution and creates PendingInference handles
  • EmbeddingsRuntime -- coordinates embeddings execution and creates PendingEmbeddings handles

Each runtime can be constructed from a config object, a provider, or injected directly. When no HTTP client is provided, the runtime builds a default one via HttpClientBuilder. Runtimes also expose onEvent() and wiretap() methods for hooking into the event system.

Request and Response Layer

Requests and responses are normalized into package data objects that are provider-agnostic:

  • InferenceRequest -- messages, model, tools, tool choice, response format, options, cached context, retry policy, response cache policy
  • InferenceResponse -- content, reasoning content, tool calls, usage, finish reason, raw HTTP response data
  • PartialInferenceDelta -- a single streaming event delta with content, reasoning content, tool call fragments, finish reason, and usage
  • EmbeddingsRequest -- input texts, model, options, retry policy
  • EmbeddingsResponse -- vectors and usage

These objects isolate your application from provider-specific response shapes. Both request types support immutable with*() mutators for building modified copies.

PendingInference is the laziness boundary -- nothing is sent until you ask for a result. Behind it, InferenceExecutionSession drives one request to one response and divides that job with two per-execution collaborators, both in Inference/Core/:

  • InferenceLifecycleEmitter -- every lifecycle event, the execution and attempt stopwatches, the attempt counter, and per-attempt telemetry correlation
  • InferenceRetryLoop -- the attempt and length-recovery budgets, backoff delay, and length-recovery request rewriting

There is deliberately no response cache. Calling response() repeatedly returns the identical instance because the session reads it back off the InferenceExecution it already holds -- which works for every ResponseCachePolicy, not only Memory. ResponseCachePolicy still matters, but at the HTTP layer: it becomes the StreamCachePolicy that decides whether a stream is replayable.

See lifecycle.md for the event sequence each path emits.

Driver Layer

Drivers translate Polyglot requests into provider-native HTTP payloads and normalize the results back. Each driver implements CanProcessInferenceRequest (for inference) or CanHandleVectorization (for embeddings) and is composed of smaller adapter responsibilities:

  • Request adapters (CanTranslateInferenceRequest) -- convert InferenceRequest into an HttpRequest
  • Response adapters (CanTranslateInferenceResponse) -- convert raw HttpResponse data into InferenceResponse or stream of PartialInferenceDelta
  • Message formatters (CanMapMessages) -- map typed Messages to provider-specific structures, composing a MessageMapper utility for iteration
  • Body formatters (CanMapRequestBody) -- assemble the full request body with mode-specific adjustments
  • Usage formatters (CanMapUsage) -- extract token usage from provider responses

All inference drivers extend BaseInferenceRequestDriver, which provides the standard HTTP execution flow and stream handling. Every bundled provider is declared as an InferenceDriverSpec -- a row naming its adapters and formatters -- and built by the single SpecifiedInferenceDriver. Providers that assemble their own URL or headers select bespoke request adapters in that row; the provider-specific behavior does not require a driver class.

Shared Support

Inference and embeddings are separate subsystems with parallel class hierarchies, but a few primitives belong to neither. They live under Cognesy\Polyglot\Support\:

  • Support\Redaction\SensitiveDataRedactor -- decides what is sensitive: which header names, query parameters and option keys are masked before a value can reach an event payload or an exception message.
  • Support\Redaction\RedactsHttpPayloads -- decides where the drivers apply it. BaseInferenceRequestDriver and BaseEmbedDriver both use this trait, so a tightened rule takes effect on both sides at once. Redaction runs on error paths only; nothing here executes on a successful request.
  • Support\Retry\RetryBackoff, RetryJitter, RetryPolicyInvariants -- the delay computation, jitter strategies and constructor invariants shared by InferenceRetryPolicy and EmbeddingsRetryPolicy.
  • Support\Pricing\Cost -- the value object every cost calculator returns, on both sides. Note that the calculators themselves are not shared: Inference\Pricing\FlatRateCostCalculator prices five token categories and Embeddings\Pricing\FlatRateCostCalculator prices one, and InferencePricing and EmbeddingsPricing carry different rate fields. They share a name, not a signature.

The retry and redaction classes previously lived under Inference\, which forced every embeddings file that needed them to import from the inference subsystem. Cost was already neutral, at a top-level Polyglot\Pricing\; it moved here so that Support\ is the package's single answer to "where do things neither subsystem owns live?" rather than one of two competing conventions.

The old fully-qualified names all still resolve, through a lazily-registered alias in src/deprecated_aliases.php, and will be removed in the next major version.

How the Layers Connect

+---------------------+    +---------------------+
|      Inference      |    |     Embeddings      |     Public Layer
+---------------------+    +---------------------+
          |                          |
+---------------------+    +---------------------+
|  InferenceRuntime   |    | EmbeddingsRuntime   |     Runtime Layer
+---------------------+    +---------------------+
          |                          |
+---------------------+    +---------------------+
|  InferenceRequest   |    | EmbeddingsRequest   |
|  PendingInference   |    | PendingEmbeddings   |     Request/Response
|  InferenceResponse  |    | EmbeddingsResponse  |     Layer
+---------------------+    +---------------------+
          |                          |
+---------------------+    +---------------------+
|  Inference Drivers  |    |  Embeddings Drivers |     Driver Layer
| (OpenAI, Anthropic, |    | (OpenAI, Cohere,    |
|  Gemini, etc.)      |    |  Gemini, etc.)      |
+---------------------+    +---------------------+
          |                          |
+------------------------------------------------+
|             HTTP Client (shared)               |     Transport
+------------------------------------------------+
// @doctest id="1662"

The public facade creates a request and hands it to the runtime. The runtime delegates to a driver, which translates the request into an HTTP call and normalizes the response. Events are dispatched at each stage for observability. The result flows back up as a normalized data object.

Key Design Decisions

Immutability. Both the public facades and the request/response objects are immutable. Calling withMessages() or withModel() always returns a new instance rather than modifying the original. This makes it safe to reuse a configured Inference or Embeddings instance across multiple concurrent calls.

Lazy execution. Calling create() on a facade returns a PendingInference or PendingEmbeddings handle without triggering the HTTP call. Execution is deferred until the application reads from the handle via get(), response(), or stream().

Driver registry. Inference drivers are resolved through InferenceDriverRegistry, which maps string names (like 'openai' or 'anthropic') to driver factory functions. Embeddings drivers use EmbeddingsDriverFactory with a similar pattern. Both support registering custom drivers at runtime.

Provider-agnostic data. The InferenceResponse and EmbeddingsResponse objects present a uniform shape regardless of which provider produced them. Provider-specific details are accessible through responseData() when needed, but the primary accessors (content(), toolCalls(), usage(), etc.) work identically across all providers.