# Creating DataFrames with an Explicit Schema

Canonical HTML: https://docs.fenic.ai/latest/topics/create_dataframe_schema/

This page describes how to make a fenic `Schema` authoritative when you build a
DataFrame with `Session.create_dataframe`, and the capabilities that unlocks:
logical string-backed types, fixed-size embedding preservation, and typed empty
frames.

## Overview

By default, `create_dataframe(data)` infers a schema from the input. Inference is
convenient, but it cannot know that a string column is really JSON or Markdown,
and it normalizes fixed-size arrays to variable-length lists.

Passing an explicit `schema` makes a complete, top-level `Schema` the source of
truth for the resulting DataFrame:

- **Field names are authoritative** — extra columns are rejected; column-oriented
  inputs (a dict, a Polars/pandas DataFrame, or a PyArrow table) must supply every
  schema column, while row-oriented `list[dict]` rows may omit fields (filled with
  null). See [The Schema Contract](#the-schema-contract) below.
- **Columns are reordered** to the schema's field order.
- **Values are physically coerced** to the schema's representation.
- **The logical schema is preserved exactly**, including string-backed types.

```
import fenic as fc

session = fc.Session.get_or_create(fc.SessionConfig(app_name="schema_intro"))

schema = fc.Schema([
    fc.ColumnField("age", fc.IntegerType),
    fc.ColumnField("name", fc.StringType),
])

# Input columns can be in any order; values are coerced to the schema.
df = session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema)

df.schema
# Schema(
#   ColumnField(name='age', data_type=IntegerType)
#   ColumnField(name='name', data_type=StringType)
# )
```

Reach for an explicit schema when you need to seed logical string-backed types,
preserve embeddings, or guarantee column order and dtypes; rely on inference for
quick, ad-hoc frames.

## The Schema Contract

When you pass `schema`, `create_dataframe` enforces a strict top-level contract:

- **No unknown columns.** The data may not contain columns outside the schema.
  Column *order* in the input does not matter — the result is ordered by the
  schema.
- **Missing columns depend on input shape.** For column-oriented inputs (a
  `dict`, a Polars/pandas DataFrame, or a PyArrow table) the columns must match
  the schema exactly. For row-oriented input (a `list[dict]`), any schema fields
  absent from the rows are backfilled with nulls.
- **Physical coercion.** Each column is cast to the schema's Polars
  representation (for example `IntegerType` → `Int64`, `EmbeddingType` →
  `Array(Float32, dimensions)`).
- **Typed errors.** Shape problems raise `ValidationError`; coercion or
  schema-validity problems raise `PlanError`.

```
from fenic.core.error import ValidationError, PlanError

# A column outside the schema (here `schema` is the age/name schema above)
# -> ValidationError
try:
    session.create_dataframe({"name": ["A"], "age": ["1"], "extra": [9]}, schema=schema)
except ValidationError as e:
    print(e)
    # Data columns must match the provided schema exactly; extra columns: ['extra']

# Duplicate schema field names -> PlanError
dup = fc.Schema([
    fc.ColumnField("x", fc.IntegerType),
    fc.ColumnField("x", fc.StringType),
])
try:
    session.create_dataframe({"x": [1]}, schema=dup)
except PlanError as e:
    print(e)  # Duplicate column names found: 'x'. Column names must be unique. ...
```

### Typed empty frames

With a schema, empty inputs (`[]`, `{}`, `pl.DataFrame()`, `pd.DataFrame()`,
`pa.table({})`) produce a zero-row DataFrame that still carries the requested
schema — useful for seeding a typed, empty starting point.

```
empty = session.create_dataframe([], schema=schema)
empty.schema.column_names()          # ['age', 'name']
empty.to_polars().height             # 0
```

Without a schema, an empty list is rejected — the no-schema path cannot infer
types from nothing.

## Logical String-Backed Types (JSON / Markdown)

fenic stores `JsonType`, `MarkdownType`, and `HtmlType` as physical strings while
tracking their logical type so type-specific functions (JQ, markdown parsing,
etc.) are available. Declaring these types in the schema tags the columns at
ingestion, so you don't need a follow-up `.cast()`:

```
schema = fc.Schema([
    fc.ColumnField("payload", fc.JsonType),
    fc.ColumnField("notes", fc.MarkdownType),
])

df = session.create_dataframe(
    {"payload": ['{"a": 1}'], "notes": ["# Title"]},
    schema=schema,
)

df.schema
# ColumnField(name='payload', data_type=JsonType)
# ColumnField(name='notes',   data_type=MarkdownType)

df.to_polars()["payload"].dtype      # String  (logical type, physical string)
```

The logical type stays visible in `df.schema` and drives downstream validation,
while the value is stored as a plain string.

**No content validation.** An explicit schema tags a column's *logical type*; it
does not validate the *content*. A `JsonType` column is not parsed or checked for
well-formed JSON at ingestion — malformed values surface when a JSON operation
runs on them.

For columns produced mid-pipeline (where no ingestion schema exists), continue to
use `.cast(fc.JsonType)` / `.cast(fc.MarkdownType)` as shown in the
[JSON Processing](https://docs.fenic.ai/latest/examples/json_processing/index.md) and
[Markdown Processing](https://docs.fenic.ai/latest/examples/markdown_processing/index.md) examples.

## Embedding Preservation

fenic represents `EmbeddingType` as a fixed-size Polars array
(`Array(Float32, dimensions)`). The schema-free ingestion path normalizes all
fixed-size arrays to variable-length lists (`pl.List`), which loses the
fixed-width guarantee. An explicit schema preserves the fixed-size array through
local execution all the way to `to_polars()`:

```
emb_schema = fc.Schema([
    fc.ColumnField("id", fc.IntegerType),
    fc.ColumnField("embedding", fc.EmbeddingType(dimensions=3, embedding_model="example/model")),
])

edf = session.create_dataframe(
    {"id": [1, 2], "embedding": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]},
    schema=emb_schema,
)

edf.to_polars()["embedding"].dtype   # Array(Float32, shape=(3,))
```

This lets you bring precomputed embeddings into fenic without losing their
fixed-size representation. (Constructing the frame does not call an embedding
provider — the `embedding_model` on the type is metadata.)

## Relationship to Schema-Free Ingestion

The no-schema path retains its existing schema semantics:

- Empty lists are still rejected.
- Types are still inferred from the data.
- Fixed-size arrays are still normalized to `pl.List`.

Its object-identity behavior differs for schema-free Polars inputs: when the
input already has Fenic-compatible physical dtypes, local execution can retain
and return the same Polars frame instead of copying it. If ingestion must
normalize an array, timestamp, or nested dtype, it materializes a new frame.
Supplying an explicit schema also applies its authoritative ordering and casts,
so it does not use this schema-free pass-through path.

Only supplying a `schema` opts you into authoritative names, ordering, coercion,
and logical-type/embedding preservation. For related ingestion-normalization
behavior on temporal columns, see
[Timestamps and Dates](https://docs.fenic.ai/latest/topics/timestamps_and_dates/index.md).