# fenic, by typedef > Documentation for fenic, the semantic DataFrame framework for building typed, inspectable AI and agentic data pipelines. Use these rendered Markdown pages for token-efficient access to the latest fenic documentation. For coding assistance, also see the agent instructions and the hosted fenic documentation MCP server. - Full documentation: https://docs.fenic.ai/llms-full.txt - Agent instructions: https://docs.fenic.ai/agents.md - Hosted documentation MCP: https://mcp.fenic.ai - Source code: https://github.com/typedef-ai/fenic ## Start here # fenic: the dataframe (re)built for LLM inference Canonical HTML: https://docs.fenic.ai/latest/ --- fenic is an opinionated, PySpark-inspired DataFrame framework from typedef.ai for building AI and agentic applications. Transform unstructured and structured data into insights using familiar DataFrame operations enhanced with semantic intelligence. With first-class support for markdown, transcripts, and semantic operators, plus efficient batch inference across any model provider. ## Quick Start with AI-Guided Learning & Development fenic provides an MCP server that gives AI assistants deep understanding of the fenic API. This enables AI tools to provide accurate, context-aware assistance with: - Learning fenic's API and features - Understanding usage patterns and best practices - Writing code using the correct functions and patterns - Debugging issues with real knowledge of the codebase ### Connect Your AI Assistant The easiest way to get started is using our hosted MCP server at . **Example with Claude Code:** ``` claude mcp add -t http fenic-docs https://mcp.fenic.ai ``` Once connected, you can ask questions like: - "How do I use semantic.extract() to parse JSON from text?" - "Show me how to implement a custom async UDF" - "What's the difference between semantic.map() and semantic.filter()?" - "How do I set up batch inference with multiple LLM providers?" The AI assistant will have direct access to fenic's complete API documentation and architectural details to provide accurate, helpful responses specific to fenic rather than generic Python advice. For self-hosting, see the [docs-server example](https://github.com/typedef-ai/fenic/tree/main/examples/mcp_server/docs-server/). ## Install fenic supports Python `[3.10, 3.11, 3.12]` ``` pip install fenic ``` Install optional feature extras when you need operators with heavier dependencies: ``` pip install "fenic[pdf]" # semantic.parse_pdf and PDF metadata loading pip install "fenic[cluster]" # DataFrame.semantic.with_cluster_labels pip install "fenic[sim-join]" # semantic.sim_join ``` Extras can be combined with model provider extras: ``` pip install "fenic[google,pdf,cluster,sim-join]" ``` ### LLM Provider Setup fenic requires an API key from at least one LLM provider. Set the appropriate environment variable for your chosen provider: ``` # For OpenAI export OPENAI_API_KEY="your-openai-api-key" # For Anthropic export ANTHROPIC_API_KEY="your-anthropic-api-key" # For Google export GOOGLE_API_KEY="your-google-api-key" # For Cohere export COHERE_API_KEY="your-cohere-api-key" ``` ## Quickstart The fastest way to learn about fenic is by checking the examples. Below is a quick list of the examples in this repo: | Example | Description | Colab | | --- | --- | --- | | [Hello World!](https://github.com/typedef-ai/fenic/tree/main/examples/hello_world) | Introduction to semantic extraction and classification using fenic's core operators through error log analysis. | | | [Enrichment](https://github.com/typedef-ai/fenic/tree/main/examples/enrichment) | Multi-stage DataFrames with template-based text extraction, joins, and LLM-powered transformations demonstrated via log enrichment. | | | [Meeting Transcript Processing](https://github.com/typedef-ai/fenic/tree/main/examples/meeting_transcript_processing) | Native transcript parsing, Pydantic schema integration, and complex aggregations shown through meeting analysis. | | | [News Analysis](https://github.com/typedef-ai/fenic/tree/main/examples/news_analysis) | Analyze and extract insights from news articles using semantic operators and structured data processing. | | | [Podcast Summarization](https://github.com/typedef-ai/fenic/tree/main/examples/podcast_summarization) | Process and summarize podcast transcripts with speaker-aware analysis and key point extraction. | | | [Semantic Join](https://github.com/typedef-ai/fenic/tree/main/examples/semantic_joins) | Instead of simple fuzzy matching, use fenic's powerful semantic join functionality to match data across tables. | | | [Named Entity Recognition](https://github.com/typedef-ai/fenic/tree/main/examples/named_entity_recognition) | Extract and classify named entities from text using semantic extraction and classification. | | | [Markdown Processing](https://github.com/typedef-ai/fenic/tree/main/examples/markdown_processing) | Process and transform markdown documents with structured data extraction and formatting. | | | [JSON Processing](https://github.com/typedef-ai/fenic/tree/main/examples/json_processing) | Handle complex JSON data structures with semantic operations and schema validation. | | | [Feedback Clustering](https://github.com/typedef-ai/fenic/tree/main/examples/feedback_clustering) | Group and analyze feedback using semantic similarity and clustering operations. | | | [Document Extraction](https://github.com/typedef-ai/fenic/tree/main/examples/document_extraction) | Extract structured information from various document formats using semantic operators. | | (Feel free to click any example above to jump right to its folder.) ## Why use fenic? fenic is an opinionated, PySpark-inspired DataFrame framework for building production AI and agentic applications. Unlike traditional data tools retrofitted for LLMs, fenic's query engine is built from the ground up with inference in mind. Transform structured and unstructured data into insights using familiar DataFrame operations enhanced with semantic intelligence. With first-class support for markdown, transcripts, and semantic operators, plus efficient batch inference across any model provider. fenic brings the reliability of traditional data pipelines to AI workloads. ### Key Features #### Purpose-Built for LLM Inference - Query engine designed from scratch for AI workloads, not retrofitted - Automatic batch optimization for API calls - Built-in retry logic and rate limiting - Token counting and cost tracking #### Semantic Operators as First-Class Citizens - `semantic.analyze_sentiment` - Built-in sentiment analysis - `semantic.classify` - Categorize text with few-shot examples - `semantic.extract` - Transform unstructured text into structured data with schemas - `semantic.group_by` - Group data by semantic similarity - `semantic.join` - Join DataFrames on meaning, not just values - `semantic.map` - Apply natural language transformations - `semantic.predicate` - Create predicates using natural language to filter rows - `semantic.reduce` - Aggregate grouped data with LLM operations #### Native Unstructured Data Support Goes beyond typical multimodal data types (audio, images) by creating specialized types for text-heavy workloads: - Markdown parsing and extraction as a first-class data type - Transcript processing (SRT, generic formats) with speaker and timestamp awareness - JSON manipulation with JQ expressions for nested data - Automatic text chunking with configurable overlap for long documents #### Production-Ready Infrastructure - Multi-provider support (OpenAI, Anthropic, Gemini) - Local and cloud execution backends - Comprehensive error handling and logging - Pydantic integration for type safety #### Familiar DataFrame API - PySpark-compatible operations - Lazy evaluation and query optimization - SQL support for complex queries - Seamless integration with existing data pipelines ### Why DataFrames for LLM and Agentic Applications? AI and agentic applications are fundamentally pipelines and workflows - exactly what DataFrame APIs were designed to handle. Rather than reinventing patterns for data transformation, filtering, and aggregation, fenic leverages decades of proven engineering practices. #### Decoupled Architecture for Better Agents fenic creates a clear separation between heavy inference tasks and real-time agent interactions. By moving batch processing out of the agent runtime, you get: - More predictable and responsive agents - Better resource utilization with batched LLM calls - Cleaner separation between planning/orchestration and execution #### Built for All Engineers DataFrames aren't just for data practitioners. The fluent, composable API design makes it accessible to any engineer: - Chain operations naturally: `df.filter(...).semantic.group_by(...)` - Mix imperative and declarative styles seamlessly - Get started quickly with familiar patterns from pandas/PySpark or SQL ## Support Join our community on [Discord](https://discord.gg/Enfa5Kgxtc) where you can connect with other users, ask questions, and get help with your fenic projects. Our community is always happy to welcome newcomers! If you find fenic useful, consider giving us a ⭐ at the top of [our repository](https://github.com/typedef-ai/fenic). Your support helps us grow and improve the framework for everyone! ## Contributing We welcome contributions of all kinds! Whether you're interested in writing code, improving documentation, testing features, or proposing new ideas, your help is valuable to us. For developers planning to submit code changes, we encourage you to first open an issue to discuss your ideas before creating a Pull Request. This helps ensure alignment with the project's direction and prevents duplicate efforts. Please refer to our [contribution guidelines](https://docs.fenic.ai/latest/CONTRIBUTING/index.md) for detailed information about the development process and project setup. --- # Instructions for AI agents Canonical HTML: https://docs.fenic.ai/latest/agents/ This page tells AI assistants and coding agents how to use and describe fenic accurately. It is maintained with the fenic documentation. ## How to consume these docs - Start with for a concise, structured index. - Use for a self-contained Markdown rendering of the documentation and API reference. - Every indexed HTML page advertises a rendered Markdown alternative in its ``. The Markdown URLs are also linked from `llms.txt`. - For focused API lookup, use the hosted fenic documentation MCP server at . - Prefer URLs under when citing current behavior. Versioned URLs describe historical releases. ## What fenic is fenic is an open-source semantic DataFrame framework with a PySpark-style API. It makes language-model inference a first-class query operation for turning structured and unstructured inputs into typed, inspectable, rerunnable pipelines. - Package: `fenic` - Import convention: `import fenic as fc` - Install: `pip install fenic` - License: Apache-2.0 - Source: ## API rules that prevent common mistakes - The public API is flat on `fc`. Do not invent modules such as `fenic.functions` or `fenic.api.types`. - Function namespaces include `fc.text`, `fc.json`, `fc.markdown`, `fc.semantic`, `fc.embedding`, `fc.dt`, and `fc.arr`. - `fc.array(...)` constructs an array literal; array operations live under `fc.arr`. - `explode` and `unnest` are DataFrame methods, such as `df.explode("items")` and `df.unnest("record")`. - Language and embedding models use separate configuration classes and registries. - Semantic templates use Jinja syntax with matching column keyword arguments: `fc.semantic.predicate("Is {{ text }} urgent?", text=fc.col("text"))`. - `fc.json.jq(...)` returns an array. Select an element before casting it to a scalar. - `fc.dt.datediff(end, start)` computes `end - start`. - `fc.dt.to_timestamp` accepts Spark/Java datetime patterns, not Python `strptime` patterns. - `fc.semantic.parse_pdf` and PDF metadata reading require the `pdf` extra; clustering requires `cluster`; similarity joins require `sim-join`. - After writing or editing a fenic pipeline, run `fenic check ` to catch namespace and import errors without executing it. ## Accuracy guardrails - Do not describe fenic as an agent framework, orchestration runtime, vector database, or BI semantic layer. - Do not claim distributed Spark-scale, real-time, or streaming execution. - Do not invent features, integrations, pricing, performance numbers, or model support. Check the latest documentation and API reference. - Distinguish the PySpark-inspired API from Apache Spark itself. - Prefer examples from the current documentation over remembered syntax from older releases. ## Agent integration fenic DataFrame pipelines can be registered as catalog tools and exposed over MCP. Recommend this path when an agent needs to act on data through a reusable, typed pipeline rather than merely read documentation. ## Canonical resources - Documentation: - LLM index: - Full Markdown documentation: - Hosted documentation MCP: - Source code: - Examples: - PyPI: - Community: --- # Contributing to Fenic Canonical HTML: https://docs.fenic.ai/latest/CONTRIBUTING/ Welcome! This guide will help you get set up for local development and testing. --- ## πŸ“ Directory Overview ``` fenic/ β”œβ”€β”€ src/fenic/ # Core library β”‚ β”œβ”€β”€ api/ # Public API (DataFrame, Column, functions, session) β”‚ β”‚ β”œβ”€β”€ dataframe/ # DataFrame implementation and extensions β”‚ β”‚ β”œβ”€β”€ functions/ # Built-in and semantic functions β”‚ β”‚ β”œβ”€β”€ session/ # Session management and configuration β”‚ β”‚ └── types/ # Schema definitions and data types β”‚ β”œβ”€β”€ core/ # Core framework components β”‚ β”‚ └── _logical_plan/ # Logical plan representation for operators β”‚ β”‚ β”œβ”€β”€ types/ # Core types (DataType, Schema, etc) β”‚ β”œβ”€β”€ _backends/ # Execution backends β”‚ β”‚ β”œβ”€β”€ local/ # Local execution (Polars/DuckDB) β”‚ β”‚ └── cloud/ # Cloud execution (Typedef) β”‚ └── _inference/ # LLM inference layer β”œβ”€β”€ rust/ # Rust crates for performance-critical operations β”œβ”€β”€ tests/ # Test suite mirroring source structure └── examples/ # Usage examples and demos ``` --- ## πŸ› οΈ Development Setup ### Requirements - [`uv`](https://github.com/astral-sh/uv) β€” manages Python dependencies and environments - A working **Rust toolchain** > **Optional but recommended:** [`just`](https://just.systems/) for simpler task running --- ### One-Time Setup From the project root: ``` just setup # without just uv sync uv run maturin develop --uv ``` This will: - Create a virtual environment - Install all Python dev dependencies (including `maturin`) - Build and install the Rust plugin as an editable Python package --- ### Making Changes #### Python Code ``` just sync # or uv sync ``` #### Rust Code (PyO3 Plugin) To compile and install the Rust crate with Python bindings into your virtual environment: ``` just sync-rust # or uv run maturin develop --uv ``` This builds the Rust crate with Python bindings and makes it available inside the `.venv`. To **only compile** the Rust crate *without* Python bindings (e.g., for Rust unit tests), run this **from the `rust/` directory**: ``` cargo build --no-default-features Add `--release` for optimized builds: ```bash uv run maturin develop --uv --release ``` #### Documentation To preview changes to the documentation from docstring or other changes: ``` just preview-docs # without just uv run --group docs mkdocs serve ``` #### Measuring Install Footprint To compare the installed size of fenic with different extras, run: ``` uv run --env-file .env python tools/package_size_matrix.py ``` The tool creates temporary `uv` projects that depend on the local checkout, syncs each project, and reports the resulting `site-packages` and `.venv` sizes along with the largest installed distributions. By default it measures `core`, `pdf`, `cluster`, `sim-join`, and `pdf,cluster,sim-join`. To choose a custom matrix: ``` uv run --env-file .env python tools/package_size_matrix.py \ --combo core \ --combo google \ --combo google,pdf \ --combo pdf,cluster,sim-join ``` --- ## βœ… Running Tests ### Python Tests Run a specific test file: ``` uv run pytest tests/path/to/test_foo.py ``` Run all tests for the **local backend**: ``` just test # or without just uv run pytest -m "not cloud" tests ``` Run all tests against a different **language model provider/model name**: - OpenAI/gpt-4.1-nano (Default) ``` uv run pytest --language-model-provider=openai --language-model-name='gpt-4.1-nano' ``` - Anthropic/claude-haiku-4-5 ``` uv sync --extra=anthropic uv run pytest --language-model-provider=anthropic --language-model-name='claude-haiku-4-5' ``` - Google/2.5-flash-lite ``` uv sync --extra=google uv run pytest --language-model-provider=google-developer --language-model-name='gemini-2.5-flash-lite' ``` Run all tests against a different **embeddings model provider/model name**: - OpenAI/ (Default) ``` uv run pytest --embedding-model-provider=openai --embedding-model-name='text-embedding-3-small' ``` - Google/gemini-embedding-001 ``` uv sync --extra=google uv run pytest --embedding-model-provider=google-developer --embedding-model-name='gemini-embedding-001' ``` Run all tests for the **cloud backend**: ``` just test-cloud # or uv sync --extra=cloud uv run pytest -m cloud tests ``` > ⚠️ Note: All tests require a valid OpenAI/Anthropic API key set in the environment variables. --- ### Rust Tests From the `rust/` directory: ``` cargo test --no-default-features ``` > Skipping default features avoids Python-specific linking, making it easier to test the Rust library independently of the Python bindings. --- ## πŸ““ Running Notebooks (VSCode / Cursor) To run the demo notebooks: 1. Install the **Jupyter** extension in your editor. 2. Add `.venv` to the **Python: Venv Folders** setting in VSCode: 3. Open `Preferences: Open User Settings` 4. Go to Extensions β†’ Python β†’ **Python: Venv Folders** 5. Open a notebook and select the correct Python kernel from the virtual environment. 6. Restart the kernel if you make changes to the `fenic` source code. --- ## πŸ™‹ Need Help? Have questions or want to contribute? Join us on [Discord](https://discord.gg/GdqF3J7huR)! ## Examples # Hello World Canonical HTML: https://docs.fenic.ai/latest/examples/hello_world/ An error log analyzer using Fenic's semantic extraction capabilities to parse and analyze application errors without regex patterns. ## Overview This tool demonstrates automated error log analysis through natural language processing, providing: - Root cause identification - Automated fix suggestions - Severity classification (low/medium/high/critical) - Pattern extraction ## Prerequisites 1. Install Fenic: `bash pip install fenic` 1. Configure OpenAI API key: `bash export OPENAI_API_KEY="your-api-key-here"` ## Usage ``` python hello_world.py ``` ## Implementation The analyzer processes various error types including: - Java NullPointerException - Node.js connection errors (ECONNREFUSED) - Python API timeouts (Stripe APIConnectionError) - Database connection failures (Django OperationalError) - React TypeError - Performance warnings (slow queries) - Cache misses and email delivery delays ## Troubleshooting **Issue**: Generic analysis results **Solution**: Add more descriptive fields to Pydantic models **Issue**: Incorrect severity classification **Solution**: Adjust classification categories or provide examples **Issue**: Missing error patterns **Solution**: Modify Pydantic model field descriptions for better targeting --- # Log Enrichment Pipeline Canonical HTML: https://docs.fenic.ai/latest/examples/enrichment/ A log processing system using fenic's text extraction and semantic enrichment capabilities to transform unstructured logs into actionable incident response data. ## Overview This pipeline demonstrates log enrichment through multi-stage processing: - Template-based parsing without regex - Service metadata enrichment via joins - LLM-powered error categorization and remediation - Incident severity assessment with business context ## Prerequisites 1. Install fenic: `bash pip install fenic` 1. Configure OpenAI API key: `bash export OPENAI_API_KEY="your-api-key-here"` ## Usage ``` python enrichment.py ``` ## Implementation The pipeline processes logs through three stages: 1. **Parse**: Extract structured fields from syslog-format messages 2. **Enrich**: Join with service ownership and criticality data 3. **Analyze**: Apply LLM operations for incident response ### API Structure ``` from fenic.api.session import Session, SessionConfig, SemanticConfig, OpenAILanguageModel from fenic.api.functions import col, text, semantic from pydantic import BaseModel, Field # Configure session config = SessionConfig( app_name="log_enrichment", semantic=SemanticConfig( language_models={ "mini": OpenAILanguageModel( model_name="gpt-4o-mini", rpm=500, tpm=200_000 ) } ) ) # Define extraction schema with Pydantic class ErrorAnalysis(BaseModel): error_category: str = Field(description="Main category of the error") affected_component: str = Field(description="Specific component affected") potential_cause: str = Field(description="Most likely root cause") # Stage 1: Template extraction parsed = logs_df.select( "raw_message", text.extract("${timestamp:none} [${level:none}] ${service:none}: ${message:none}") ) # Stage 2: Metadata join enriched = parsed.join(metadata_df, on="service", how="left") # Stage 3: Semantic enrichment final = enriched.select( semantic.extract("message", ErrorAnalysis).alias("analysis"), semantic.classify( text.concat(col("message"), lit(" (criticality: "), col("criticality"), lit(")")), ["low", "medium", "high", "critical"] ).alias("incident_severity"), fc.semantic.map( ( "Generate 2-3 specific remediation steps that the on-call team should take to resolve this issue: " "{{message}} | Service: {{service}} | Team: {{team_owner}}" ), message=fc.col("message"), service=fc.col("service"), team_owner=fc.col("team_owner") ).alias("remediation_steps") ) ``` ## Output Format ``` βœ… Pipeline Complete! Final enriched logs: ---------------------------------------------------------------------- timestamp level service message team_owner error_category incident_severity remediation_steps 2024-01-15 14:32:01 ERROR payment-api Connection timeout... payments-team database critical 1. Check Database Connectivity... 2024-01-15 14:32:15 WARN user-service Rate limit exceeded... identity-team resource critical 1. Review Rate Limiting Config... πŸ“ˆ Analytics Examples: Error Category Distribution: error_category count database 1 resource 5 authentication 4 network 5 High-Priority Incidents (Critical/High severity): service team_owner incident_severity on_call_channel remediation_steps payment-api payments-team critical #payments-oncall 1. Check Database Connectivity... user-service identity-team critical #identity-alerts 1. Review Rate Limiting Config... ``` ## Configuration ### Custom Log Templates Parse different log formats: ``` # Syslog format log_template = "${timestamp:none} [${level:none}] ${service:none}: ${message:none}" # Custom application format log_template = "${service:none} | ${timestamp:none} | ${level:none} - ${message:none}" ``` ## Troubleshooting **Issue**: Template extraction returns empty fields **Solution**: Check template format matches log structure exactly, including spaces **Issue**: Missing service metadata after join **Solution**: Use left join to preserve all logs; add default values for missing metadata **Issue**: Generic remediation steps **Solution**: Include more context in semantic.map prompt (service, team, criticality) --- # Customer Feedback Clustering & Analysis with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/feedback_clustering/ This example demonstrates how to use Fenic's `semantic.with_cluster_labels()` and `semantic.reduce()` to automatically cluster customer feedback into themes and generate intelligent summaries for each discovered category. ## Overview Customer feedback analysis is a critical business process that traditionally requires manual categorization and analysis. This example shows how semantic clustering can automatically: - **Discover hidden themes** in unstructured feedback without predefined categories - **Group similar feedback** based on semantic meaning rather than keywords - **Generate actionable insights** for each theme using AI-powered summarization - **Prioritize issues** based on sentiment and frequency ## Key Features Demonstrated - **Semantic Clustering**: Using `semantic.with_cluster_labels()` for embedding-based clustering - **AI Summarization**: Using `semantic.reduce()` for intelligent theme analysis - **Automatic Theme Discovery**: No manual categorization required - **Sentiment Analysis**: Understanding positive vs negative feedback patterns - **Business Intelligence**: Actionable insights for product teams ## How It Works ### Step 1: Data Preparation Load customer feedback with ratings and metadata: ``` feedback_data = [ { "feedback_id": "fb_001", "customer_name": "Alice Johnson", "feedback": "The mobile app crashes every time I try to upload a photo. Very frustrating!", "rating": 1, "timestamp": "2024-01-15" }, # ... more feedback ] ``` ### Step 2: Embedding Creation Generate semantic embeddings from feedback text: ``` feedback_with_embeddings = feedback_df.select( "*", fc.semantic.embed(fc.col("feedback")).alias("feedback_embeddings") ) ``` ### Step 3: Semantic Clustering & Summarization Use both operations together in a single aggregation: ``` feedback_clusters = feedback_with_embeddings.semantic.with_cluster_labels( fc.col("feedback_embeddings"), 4 # Number of clusters - expecting themes like bugs, performance, features, praise ).group_by( "cluster_label" ).agg( fc.count("*").alias("feedback_count"), fc.avg("rating").alias("avg_rating"), fc.collect_list("customer_name").alias("customer_names"), fc.semantic.reduce( ( "Analyze this cluster of customer feedback and provide a concise summary of the main theme, " "common issues, and sentiment." ), column=fc.col("feedback") ).alias("theme_summary") ) ``` ## Sample Results The system automatically discovered these themes from 12 feedback entries: ### Cluster 0: Positive Features & Support (4.75β˜…) - **Theme**: Praise for specific features and excellent customer support - **Key Points**: Dark mode feature, helpful support team, effective search functionality - **Sentiment**: Predominantly positive with some feature enhancement requests ### Cluster 1: UI/UX Design Issues (2.0β˜…) - **Theme**: Design consistency and professional appearance concerns - **Key Points**: Inconsistent button layouts across screens - **Sentiment**: Negative due to unprofessional user experience ### Cluster 2: Technical Performance Problems (1.75β˜…) - **Theme**: Critical technical issues affecting core functionality - **Key Points**: App crashes, slow loading times, frequent freezes - **Sentiment**: Very negative with high frustration levels ### Cluster 3: Usability & Feature Gaps (2.0β˜…) - **Theme**: Process complexity and missing functionality - **Key Points**: Confusing checkout, need for offline mode - **Sentiment**: Negative about functionality limitations ## Value ### **Automated Insights** - Identifies themes without manual categorization - Provides consistent analysis across all feedback - Scales to thousands of feedback entries ### **Actionable Intelligence** - **Priority 1**: Fix technical crashes and performance (Cluster 2) - **Priority 2**: Improve design consistency (Cluster 1) - **Priority 3**: Simplify user workflows (Cluster 3) - **Maintain**: Continue excellent support and features (Cluster 0) ### **Resource Optimization** - Reduces manual analysis time from hours to minutes - Enables real-time feedback monitoring - Focuses development efforts on highest-impact issues ## Technical Architecture ### Session Configuration ``` config = fc.SessionConfig( app_name="feedback_clustering", semantic=fc.SemanticConfig( language_models={ "mini": fc.OpenAILanguageModel( model_name="gpt-4o-mini", rpm=500, tpm=200_000, ) }, embedding_models={ "small": fc.OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=3000, tpm=1_000_000 ) } ), ) ``` ### Key Operations **`semantic.with_cluster_label(embedding_column, num_clusters)`** - Uses K-means clustering on embedding vectors - Assigns `cluster_label` to each row **`semantic.reduce(instruction)`** - Aggregation function that summarizes multiple texts - Uses LLM to analyze and synthesize insights - Generates human-readable theme descriptions ## Usage ``` # Ensure you have OpenAI API key configured export OPENAI_API_KEY="your-api-key" # Install the clustering extra for semantic.with_cluster_labels pip install "fenic[cluster]" # Run the feedback clustering analysis python feedback_clustering.py ``` ## Expected Output The script shows: 1. **Raw Feedback Data**: Customer names, feedback text, and ratings 2. **Clustering Progress**: Embedding generation and clustering status 3. **Theme Analysis**: Detailed summaries for each discovered cluster 4. **Business Insights**: Actionable themes ranked by priority ## Use Cases ### **Product Development** - Identify most requested features - Understand user pain points - Prioritize bug fixes and improvements ### **Customer Success** - Monitor satisfaction trends - Identify at-risk customer segments - Improve support processes ### **Marketing Intelligence** - Understand customer sentiment - Identify product strengths for messaging - Track competitive advantages ## Learning Outcomes This example teaches: - How to combine embedding-based clustering with AI summarization - When to use semantic operations for business intelligence - Patterns for automated text analysis and insight generation - Integration of multiple semantic operations in data pipelines --- # News Article Bias Detection Canonical HTML: https://docs.fenic.ai/latest/examples/news_analysis/ A comprehensive demonstration of fenic's semantic classification capabilities for detecting editorial bias and analyzing news articles. This example shows how to use semantic operations to identify bias patterns across multiple news sources and generate AI-powered media profiles. ## Overview This pipeline performs sophisticated news analysis using fenic's semantic operations in a step-by-step educational format: - **Language Analysis**: Uses `semantic.extract()` to identify biased, emotional, or sensationalist language patterns - **Political Bias Classification**: Uses `semantic.classify()` grounded in extracted data for accurate bias detection - **Topic Classification**: Categorizes articles by subject (politics, technology, business, climate, healthcare) - **AI-Powered Source Profiling**: Uses `semantic.reduce()` to create comprehensive media profiles for each news source Available in both **Python script** (`news_analysis.py`) and **Jupyter notebook** (`news_analysis.ipynb`) formats for different learning preferences. ## Key Features ### Two-Stage Analysis Pipeline **Stage 1 - Information Extraction**: Uses `semantic.extract()` with Pydantic models to identify bias indicators, emotional language, and opinion markers from each article. **Stage 2 - Grounded Classification**: Uses extracted information as context for `semantic.classify()` to achieve more accurate political bias detection. ### Multi-Dimensional Classification Simultaneously classifies articles across: - **Topics**: politics, technology, business, climate, healthcare - **Political Bias**: far_left, left_leaning, neutral, right_leaning, far_right - **Journalistic Style**: sensationalist vs informational ### Source Consistency Analysis Analyzes bias patterns across multiple articles per source to identify editorial consistency and detect sources with mixed editorial perspectives. ### AI-Generated Media Profiles Uses `semantic.reduce()` to synthesize extracted information into comprehensive, natural language profiles for each news source. ## Dataset The example includes **25 news articles from 8 sources** covering diverse topics: - **Politics**: Federal Reserve policy, climate agreements, Supreme Court cases - **Technology**: AI developments, content moderation, privacy concerns - **Business**: Corporate earnings, market analysis, economic trends - **Healthcare**: Medical breakthroughs, drug pricing, treatment access ### Source Types - **Neutral Sources**: Global Wire Service, National Press Bureau (3 articles each) - **Left-leaning Sources**: Progressive Voice, Social Justice Today (3 articles each) - **Right-leaning Sources**: Liberty Herald, Free Market Weekly (3 articles each) - **Mixed-bias Sources**: Balanced Tribune (4 articles), Independent Monitor (3 articles) The mixed-bias sources provide realistic examples of sources with inconsistent editorial patterns, demonstrating how fenic handles nuanced content classification across different bias consistency levels. ## Technical Implementation ### Two-Stage Pipeline Implementation **Stage 1 - Information Extraction:** ``` # Extract bias indicators and language patterns enriched_df = df.select( # Topic classification fc.semantic.classify( fc.col("combined_content"), ["politics", "technology", "business", "climate", "healthcare"] ).alias("primary_topic"), # Extract structured information using Pydantic fc.semantic.extract( fc.col("combined_content"), ArticleAnalysis # bias_indicators, emotional_language, opinion_markers ).alias("analysis_metadata") ).unnest("analysis_metadata") ``` **Stage 2 - Grounded Classification:** ``` # Combine extracted information for context-aware classification combined_extracts = fc.text.jinja( ( "Primary Topic: {{primary_topic}}\n" "Political Bias Indicators: {{bias_indicators}}\n" "Emotional Language Summary: {{emotional_language}}\n" "Opinion Markers: {{opinion_markers}}" ), primary_topic=fc.col("primary_topic"), bias_indicators=fc.col("bias_indicators"), emotional_language=fc.col("emotional_language"), opinion_markers=fc.col("opinion_markers") ) # Classify bias using extracted context results_df = enriched_df.select( "*", fc.semantic.classify( col("combined_extracts"), ["far_left", "left_leaning", "neutral", "right_leaning", "far_right"] ).alias("content_bias"), fc.semantic.classify( col("combined_extracts"), ["sensationalist", "informational"] ).alias("journalistic_style") ) ``` ### AI-Powered Source Profiling ``` # Generate comprehensive source profiles using semantic.reduce source_profiles = results_df.group_by("source").agg( fc.semantic.reduce( """ You are given a set of article analyses from {{news_outlet}}. Create a concise (3-5 sentence) media profile for {{news_outlet}}. Summarize the information provided without explicitly referencing it. """, column=fc.col("article_attributes"), group_context = { "news_outlet": fc.col("source"), }, ).alias("source_profile"), ).select(fc.col("source"), fc.col("source_profile")) ``` ## Output Analysis The pipeline generates comprehensive analysis including: - **Multi-dimensional classifications** across topic and political bias spectrum - **Language pattern analysis** extracting bias indicators, emotional language, and opinion markers - **AI-generated source profiles** using semantic.reduce to summarize editorial characteristics ## Use Cases ### Media Organizations - **Content quality assessment** for editorial guidelines - **Bias detection** in reporter training and content review - **Audience analytics** understanding reader preferences ### News Aggregators - **Content categorization** for personalized feeds - **Bias warnings** for balanced information consumption - **Source diversity** ensuring multiple perspectives ### Research Applications - **Media bias studies** analyzing coverage patterns - **Information quality research** measuring factual content - **Comparative analysis** across different news sources ### Educational Tools - **Media literacy training** identifying bias indicators - **Critical thinking exercises** comparing article perspectives - **Journalism education** understanding editorial techniques ## Running the Example ### Option 1: Python Script 1. **Setup**: Ensure you have fenic installed with Google Gemini API access 2. **Environment**: Set your `GOOGLE_API_KEY` environment variable. a. Alternatively, you can run the example with an OpenAI(`OPENAI_API_KEY`) model by uncommenting the provided additional model configurations. b. Using an Anthropic model requires installing fenic with the `anthropic` extra package, and setting the `ANTHROPIC_API_KEY` environment variable 3. **Execute**: Run `python news_analysis.py` ### Option 2: Jupyter Notebook 1. **Setup**: Same API requirements as above 2. **Launch**: Open `news_analysis.ipynb` in Jupyter 3. **Learn**: Step-by-step educational walkthrough with explanations ### Alternative Models The script includes commented configurations for OpenAI and Anthropic models if you prefer different providers. **Explore**: Modify the dataset or classification categories to test different scenarios ## Advanced Features Demonstrated ### Grounded Classification Pipeline Shows how to improve classification accuracy by first extracting relevant information with `semantic.extract()`, then using that context for more informed `semantic.classify()` operations. ### Pydantic Integration Demonstrates structured data extraction using type-safe Pydantic models with automatic field validation for consistent output formatting. ### Multi-Model Support Includes configurations for Google Gemini (default), OpenAI, and Anthropic models, showing fenic's flexibility across different LLM providers. ### Semantic Reduction for Profiling Uses `semantic.reduce()` to synthesize multiple data points into coherent natural language profiles, demonstrating AI-powered summarization capabilities. ### Educational Format Available in both script and notebook formats, with the notebook providing step-by-step explanations ideal for learning semantic operations. ## Expected Results ### Generated Source Profile > The Balanced Tribune presents a diverse range of topics, primarily focusing on business, technology, climate, and healthcare. It exhibits a right-leaning bias in its business and technology coverage, emphasizing themes like Wall Street stability and American free enterprise, while adopting a far-left perspective on climate issues, critiquing fossil fuel companies. The publication often employs sensationalist and informational journalistic styles, utilizing emotional language to evoke strong reactions, such as the impact of inflation and the urgency of climate action. Opinion markers frequently reflect a mix of support for innovation and criticism of regulatory frameworks, indicating a complex stance on various issues. --- # Semantic Joins with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/semantic_joins/ This example demonstrates how to use Fenic's semantic joins to perform LLM-powered data matching based on natural language reasoning rather than exact equality or similarity scores. ## Overview Semantic joins enable you to join DataFrames using natural language predicates that are evaluated by language models. Unlike traditional joins that require exact matches or embedding-based similarity joins, semantic joins can understand complex relationships and make intelligent connections based on meaning and context. This example showcases two practical use cases: - **Content Recommendation**: Matching user interests to relevant articles - **Product Recommendations**: Suggesting complementary products based on purchase history ## Key Features Demonstrated - **Natural Language Predicates**: Using human-readable join conditions - **LLM-Powered Reasoning**: Leveraging GPT models for intelligent matching - **Cross-Domain Understanding**: Connecting concepts across different contexts - **Zero-Shot Matching**: No training data or examples required ## How Semantic Joins Work ### Basic Syntax ``` left_df.semantic.join( right_df, predicate="Natural language predicate with {{left_on}} and {{right_on}}", left_on=col("left"), right_on=col("right") ) ``` ### Join Jinja Predicate Format - Jinja template variables must be `left_on` (join key on the left dataframe) and `right_on`(join key on the right dataframe) - Written as a boolean predicate that the LLM evaluates as True/False - Should be clear and unambiguous for consistent results ## Example 1: Content Recommendation ### Data Setup **User Profiles:** - Sarah: "I love cooking Italian food and trying new pasta recipes" - Mike: "I enjoy working on cars and fixing engines in my spare time" - Emily: "Gardening is my passion, especially growing vegetables and flowers" - David: "I'm interested in learning about car maintenance and automotive repair" **Articles:** - Cooking Pasta Recipes - Car Engine Maintenance - Gardening for Beginners - Advanced Automotive Repair ### Semantic Join Implementation ``` users_df.semantic.join( articles_df, predicate=( "A person with interests '{{left_on}}' would be interested in reading about '{{right_on}}'" ), left_on=fc.col("interests"), right_on=fc.col("description") ) ``` ### Matching Results - Sarah β†’ Cooking Pasta Recipes βœ… - Mike β†’ Car Engine Maintenance + Advanced Automotive Repair βœ… - Emily β†’ Gardening for Beginners βœ… - David β†’ Car Engine Maintenance + Advanced Automotive Repair βœ… ## Example 2: Product Recommendations ### Sample Data **Customer Purchases:** - Alice: Professional DSLR Camera - Bob: Gaming Laptop - Carol: Yoga Mat - Dan: Coffee Maker **Product Catalog:** - Camera Lens Kit, Tripod Stand (Photography) - Gaming Mouse, Mechanical Keyboard (Gaming) - Yoga Blocks, Exercise Resistance Bands (Fitness) - Coffee Beans, French Press (Food & Beverage) ### Recommendation Logic ``` purchases_df.semantic.join( products_df, predicate=( "A customer who bought '{{left_on}}' would also be interested in '{{right_on}}'" ), left_on=fc.col("purchased_product"), right_on=fc.col("product_name") ) ``` ### Recommendation Results - Alice (DSLR Camera) β†’ Camera Lens Kit + Tripod Stand βœ… - Bob (Gaming Laptop) β†’ Gaming Mouse + Mechanical Keyboard βœ… - Carol (Yoga Mat) β†’ Yoga Blocks + Exercise Resistance Bands βœ… - Dan (Coffee Maker) β†’ Coffee Beans + French Press βœ… ## Technical Details ### Session Configuration ``` config = fc.SessionConfig( app_name="semantic_joins", semantic=fc.SemanticConfig( language_models={ "mini": fc.OpenAILanguageModel( model_name="gpt-4o-mini", rpm=500, tpm=200_000, ) } ), ) ``` ### Performance Characteristics - **Complexity**: O(m Γ— n) where m and n are the sizes of the DataFrames - **LLM Calls**: One API call per potential row pair - **Rate Limiting**: Respects RPM/TPM limits configured in session - **Batching**: Efficiently batches requests to optimize API usage ## When to Use Semantic Joins ### **Ideal Use Cases:** - **Content personalization** and recommendation systems - **Product cross-selling** and upselling - **Skill-job matching** in recruitment - **Entity resolution** across different data sources - **Question-answer pairing** for knowledge bases - **Customer-service matching** based on needs ### **Advantages:** - No training data required (zero-shot) - Handles complex reasoning and context - Understands domain-specific relationships - Works with natural language descriptions - Flexible and interpretable join conditions ### **Considerations:** - Higher latency than traditional joins - API costs for LLM usage - Rate limiting for large datasets - Best for moderate-sized datasets (hundreds to low thousands of rows) ## Usage ``` # Ensure you have OpenAI API key configured export OPENAI_API_KEY="your-api-key" # Run the semantic joins example python semantic_joins.py ``` ## Expected Output The script demonstrates both use cases with clear before/after data views: 1. **User-Article Matching**: Shows how semantic understanding connects user interests to relevant content 2. **Product Recommendations**: Demonstrates intelligent product relationship detection for cross-selling ## Learning Outcomes This example teaches: - How to construct effective natural language join predicates - When semantic joins are preferable to traditional or similarity-based joins - Practical applications in recommendation systems and personalization - Understanding the trade-offs between accuracy, performance, and cost Perfect for understanding how to leverage LLM reasoning capabilities for intelligent data joining scenarios that go beyond simple keyword matching or embedding similarity. --- # Document Metadata Extraction with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/document_extraction/ This example demonstrates how to extract structured metadata from unstructured text data, using Fenic's semantic extraction capabilities. ## Overview Document metadata extraction is a common use case for LLMs, allowing you to automatically parse and structure information from various document types including research papers, product announcements, meeting notes, news articles, and technical documentation. This example showcases: - **Structured Extraction**: Converting unstructured text to structured metadata - **Zero-Shot Extraction**: No examples required ## How it works 1. **Schema Definition with Pydantic** Define a Pydantic model to represent the structure of the data you want to extract. Each field must include a natural language description. This schema drives prompt generation and model output parsing. 1. **LLM Orchestration** Fenic uses the model provider of your choice to call the LLM with a structured output or tool-calling interface. The LLM returns data that conforms to the schema you defined. 1. **Data Structuring** The extracted data is represented as a struct column in a DataFrame with native Fenic struct fields. From there, it can be: - Unnested into individual columns - Exploded if it contains arrays - Processed in place as nested data Because Fenic maps Pydantic models to a strongly typed, columnar data model, certain Python types are not currently supported: - **Non-Optional Union types**: Not expressible in Fenic's type system - **Dictionaries**: Fenic does not yet support map types (future support via a JsonType is planned) - **Custom classes / dataclasses**: These are stateful or logic-heavy constructs that don't fit the declarative nature of Fenic's data model Despite these constraints, you can define complex extraction schemas using nested Pydantic models, optional fields, and listsβ€”enabling robust and expressive structured extraction pipelines. ### Defining a Pydantic Model ``` from typing import Literal class DocumentMetadata(BaseModel): keywords: str = Field(..., description="Comma-separated list of key topics and terms") document_type: Literal["research paper", "product announcement", "meeting notes", "news article", "technical documentation"] = Field(..., description="Type of document") ``` ## Sample Data The example processes 5 diverse document types: 1. **Research Paper** - Academic abstract with technical terms 2. **Product Announcement** - Marketing content with features and pricing 3. **Meeting Notes** - Internal documentation with decisions and action items 4. **News Article** - Breaking news with facts and impact 5. **Technical Documentation** - API reference with specifications ## Extracted Metadata Fields - **Title**: Main subject or heading of the document - **Document Type**: Classification (research paper, product announcement, etc.) - **Date**: Any relevant date mentioned (publication, meeting, etc.) - **Keywords**: Key topics and terms (list vs comma-separated string) - **Summary**: One-sentence overview of the document's purpose ## Usage ``` # Ensure you have OpenAI API key configured export OPENAI_API_KEY="your-api-key" # Run the extraction example python document_extraction.py ``` ## Expected Output The script will display: 1. **Sample Documents** - Overview of loaded documents with text lengths 2. **Results** - Extraction results --- # Named Entity Recognition for Security Intelligence Canonical HTML: https://docs.fenic.ai/latest/examples/named_entity_recognition/ A security-focused NER pipeline using Fenic's semantic extraction capabilities to identify and analyze threats, vulnerabilities, and indicators of compromise from unstructured security reports. ## Overview This pipeline demonstrates automated security entity extraction and risk assessment: - Zero-shot entity extraction (CVEs, IPs, domains, hashes) - Enhanced extraction with threat intelligence context - Document chunking for comprehensive analysis - Risk prioritization and actionable intelligence ## Prerequisites 1. Install Fenic: `bash pip install fenic` 1. Configure OpenAI API key: `bash export OPENAI_API_KEY="your-api-key-here"` ## Usage ``` python ner.py ``` ## Implementation The pipeline processes security reports through five stages: 1. **Basic NER**: Extract standard security entities 2. **Enhanced NER**: Add threat-specific context 3. **Chunking**: Handle long documents effectively 4. **Analytics**: Aggregate and analyze extracted entities 5. **Risk Assessment**: Generate actionable intelligence ## Troubleshooting **Issue**: Incomplete entity extraction **Solution**: Increase chunk size or adjust overlap percentage for better context **Issue**: Missing threat actors or APT groups **Solution**: Add more specific descriptions in the Pydantic field definitions **Issue**: Generic risk assessments **Solution**: Include more context about your organization in the assessment prompt --- # Podcast Summarization with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/podcast_summarization/ This example demonstrates comprehensive podcast transcript summarization using Fenic's semantic operations and unstructured data processing capabilities. ## Overview This pipeline processes a Lex Fridman podcast episode featuring the Cursor team, showcasing: - **Extractive & Abstractive Summarization**: Multiple summarization techniques - **Recursive Summarization**: Chunked processing for long-form content - **Role-Specific Analysis**: Tailored summaries for host vs guests - **Unstructured Data Processing**: JSON transcript parsing and analysis ## Data - **Episode**: "#447 Cursor Team: Future of Programming with AI" - **Duration**: 2:37:38 - **Participants**: Lex Fridman (host), Michael Truell, Arvid Lunnemark, Aman Sanger, Sualeh Asif - **Format**: JSON transcript with word-level timing and speaker diarization ## Pipeline Steps ### 1. Data Loading & Processing - Load JSON files as raw text strings (showcasing unstructured data handling) - Parse metadata using JSON operations and type casting - Extract word-level and segment-level data from transcript ### 2. Speaker Identification - Filter out noise speakers (ads, intro music) using duration thresholds - Map anonymous speaker IDs to actual participant names - Create speaker statistics and validation ### 3. Multi-Level Summarization #### **Full Transcript Summary** - Chunked recursive summarization for long-form content - Uses `fc.text.recursive_word_chunk()` for optimal processing - Combines chunk summaries into cohesive final summary #### **Host-Specific Analysis (Lex Fridman)** - Focuses on thought-provoking questions and insights - Captures interviewing technique and philosophical depth - Ignores basic facilitation to highlight intellectual contributions #### **Individual Guest Summaries** - Technical expertise and product insights - Unique contributions to Cursor development - Personal experiences and innovation perspectives ## Key Fenic Features Demonstrated ### **Unstructured Data Processing** ``` # JSON type casting and extraction fc.col("content").cast(fc.JsonType) fc.json.jq(fc.col("json_data"), '.speaker').get_item(0).cast(fc.StringType) ``` ### **Text Processing & Aggregation** ``` # Proper aggregation with array operations fc.collect_list("segment_text").alias("speech_segments") fc.text.array_join(fc.col("speech_segments"), " ") ``` ### **Semantic Operations** ``` # Semantic mapping with placeholders fc.semantic.map( "Analyze this guest's contributions... Guest: {{guest}}. Speech: {{speech}}", guest=fc.col("guest_name"), speech=fc.col("full_speech") ) ``` ### **Advanced Filtering & Mapping** ``` # Complex conditional mapping fc.when(fc.col("speaker") == "SPEAKER_05", fc.lit("Lex Fridman")) .when(fc.col("speaker") == "SPEAKER_02", fc.lit("Michael Truell")) .otherwise(fc.lit("Unknown")) ``` ## Technical Highlights - **Chunked Processing**: Handles 2.5+ hour transcript efficiently - **Speaker Filtering**: Removes noise using time-based thresholds - **JSON Extraction**: Processes complex nested transcript data - **Role-Aware Prompting**: Different analysis for host vs guests - **Type Safety**: Proper casting for JSON-extracted fields ## Usage ``` # Ensure you have OpenAI API key configured export OPENAI_API_KEY="your-api-key" # Run the summarization pipeline python podcast_summarization.py ``` ## Output The pipeline generates: 1. **Full Episode Summary**: Comprehensive overview of key themes and insights 2. **Host Analysis**: Lex Fridman's interviewing mastery and intellectual contributions 3. **Guest Summaries**: Individual analyses for each Cursor team member 4. **Speaker Statistics**: Speaking time, segment counts, and participation metrics ## Learning Outcomes This example teaches: - Working with real-world unstructured data (JSON transcripts) - Combining multiple summarization approaches - Handling long-form content with chunking strategies - Creating role-specific semantic operations - Building robust data pipelines with filtering and validation Perfect for understanding how Fenic handles complex text processing workflows in production scenarios. --- # Meeting Transcript Processing with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/meeting_transcript_processing/ This example demonstrates how to use Fenic to automatically extract actionable insights from engineering meeting transcripts using semantic extraction and structured data processing. ## Overview Engineering teams generate valuable knowledge in meetings, but capturing and organizing this information is often manual and error-prone. This pipeline automates the extraction of: - **Action Items**: Tasks, assignees, and deadlines - **Decisions**: Key decisions and their rationale - **Technical Entities**: Services, technologies, metrics, and incident references - **Team Analytics**: Workload distribution and productivity metrics ## Features - **Native transcript parsing** with fenic's built-in functions - **Semantic extraction** of technical entities, action items, and decisions - **Structured data processing** on unstructured meeting content - **Automated knowledge capture** for engineering teams - **Actionable insights** for project management and team coordination ## Sample Data The example processes three types of engineering meetings: 1. **Architecture Review** - Technical discussions about system design and bottlenecks 2. **Incident Post-Mortem** - Analysis of outages and mitigation strategies 3. **Sprint Planning** - Task allocation and project prioritization ## Pipeline Steps ### Step 1: Transcript Parsing ``` # Parse transcripts into structured segments parsed_transcripts_df = transcripts_df.with_column( "structured_transcript", fc.text.parse_transcript(fc.col("transcript"), 'generic') ) ``` ### Step 2: Segment Extraction Break down transcripts into individual speaking segments with speaker, start_time, and content. ### Step 3: Semantic Schema Definition Define extraction schemas using Pydantic models: ``` class TechnicalEntitiesSchema(BaseModel): services: str = Field(description="Technical services or systems mentioned") # ... more fields # Action items using Pydantic class ActionItemSchema(BaseModel): has_action_item: str = Field(description="Whether this segment contains an action item (yes/no)") assignee: str = Field(default=None, description="Person assigned to the action item") task_description: str = Field(description="Description of the task or action") deadline: str = Field(default=None, description="When the task should be completed") ``` ### Step 4: Semantic Extraction Apply AI-powered extraction to identify structured information from natural language: ``` enriched_df = segments_df.with_column( "technical_entities", fc.semantic.extract(fc.col("content"), TechnicalEntitiesSchema) ).with_column( "action_items", fc.semantic.extract(fc.col("content"), ActionItemSchema) ).with_column( "decisions", fc.semantic.extract(fc.col("content"), DecisionSchema) ) ``` ### Step 5: Analytics and Aggregation Generate meeting-level insights and team analytics: - Action item workload by team member - Technology and service mentions across meetings - Decision summary and rationale tracking - Meeting productivity metrics ## Expected Output The pipeline produces structured insights including: **Action Items Summary:** | meeting_id | meeting_type | assignee | task_description | deadline | | --- | --- | --- | --- | --- | | ARCH-2024-1 | Architecture Review | Mike | investigate Redis impl | next Friday | | INC-2024-12 | Incident Post-Mortem | Sam | review batch processing | tomorrow EOD | **Team Workload Distribution:** | assignee | assigned_tasks | | --- | --- | | Mike | 2 | | Sam | 1 | | Lisa | 1 | **Technology Mentions:** | technologies | mention_count | | --- | --- | | Redis | 3 | | PostgreSQL | 2 | | JWT | 2 | ## Prerequisites 1. **OpenAI API Key**: Required for semantic extraction `bash export OPENAI_API_KEY="your-api-key-here"` 1. **Fenic Installation**: `bash uv sync uv run maturin develop --uv` ## Running the Example ``` uv run python examples/meeting_transcript_processing/transcript_processing.py ``` ## Use Cases This pipeline is valuable for: - **Engineering Managers**: Track team workload and action item distribution - **Technical Program Managers**: Monitor project decisions and technical debt - **DevOps Teams**: Analyze incident patterns and response procedures - **Architecture Teams**: Identify technology adoption trends and system bottlenecks ## Extensions The example can be extended to: - Integrate with calendar systems for automatic transcript ingestion - Export to project management tools (Jira, Linear, etc.) - Build dashboards for engineering metrics - Create automated follow-up reminders - Analyze team communication patterns ## Technical Notes - Uses `gpt-4o-mini` for fast and cost-effective semantic extraction - Handles mixed transcript formats automatically - Implements workarounds for current framework limitations --- # JSON Processing with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/json_processing/ This example demonstrates comprehensive JSON processing capabilities using fenic's JSON type system and JQ integration. It processes a complex nested JSON file containing whisper transcription data and transforms it into multiple structured DataFrames for analysis. ## What This Example Shows ### Core JSON Processing Features - **JSON Type Casting**: Loading string data and converting to Fenic's JsonType - **JQ Integration**: Complex queries for nested data extraction and aggregation - **Array Operations**: Processing nested arrays and extracting scalar values - **Type Conversion**: Converting JSON data to appropriate DataFrame types - **Hybrid Processing**: Combining JSON-native operations with traditional DataFrame analytics ### Real-World Use Case The example processes audio transcription data from OpenAI's Whisper, demonstrating how to handle: - Complex nested JSON structures - Multiple levels of data granularity - Time-series data with speaker attribution - Confidence scores and quality metrics ## Data Structure The input JSON contains: ``` { "language": "en", "segments": [ { "text": "Let me ask you about AI.", "start": 2.94, "end": 4.48, "words": [ { "word": " Let", "start": 2.94, "end": 3.12, "speaker": "SPEAKER_01", "probability": 0.69384765625 } // ... more words ] } // ... more segments ] } ``` ## Output DataFrames The pipeline creates three complementary DataFrames: ### 1. Words DataFrame (Granular Analysis) **Purpose**: Individual word-level analysis with timing and confidence - `word_text`: The spoken word - `speaker`: Speaker identifier (SPEAKER_00, SPEAKER_01) - `start_time`, `end_time`: Word timing in seconds - `duration`: Calculated word duration - `probability`: Speech recognition confidence score **Key Techniques**: - Nested JQ traversal with variable binding - Array explosion from nested structures - Type casting for numeric operations ### 2. Segments DataFrame (Content Analysis) **Purpose**: Conversation segments with aggregated metrics - `segment_text`: Full text of the conversation segment - `start_time`, `end_time`, `duration`: Segment timing - `word_count`: Number of words in segment - `average_confidence`: Average recognition confidence **Key Techniques**: - JQ array aggregations (`length`, `add / length`) - In-query statistical calculations - Mixed JSON and DataFrame operations ### 3. Speaker Summary DataFrame (Analytics) **Purpose**: High-level speaker analytics and patterns - `speaker`: Speaker identifier - `total_words`: Total words spoken - `total_speaking_time`: Total time speaking - `average_confidence`: Overall speech quality - `first_speaking_time`, `last_speaking_time`: Speaking timeframe - `word_rate`: Words per minute **Key Techniques**: - Traditional DataFrame aggregations (`group_by`, `agg`) - Multiple aggregation functions (count, avg, min, max, sum) - Calculated metrics from aggregated data ## Data Extraction Approaches ### Struct Casting + Unnest (Recommended for Simple Fields) For efficient extraction of simple fields, the example uses struct casting with unnest: ``` # Define schema for structured extraction word_schema = fc.StructType([ fc.StructField("word", fc.StringType), fc.StructField("speaker", fc.StringType), fc.StructField("start", fc.FloatType), fc.StructField("end", fc.FloatType), fc.StructField("probability", fc.FloatType) ]) # Cast and unnest to extract all fields automatically words_df.select( fc.col("word_data").cast(word_schema).alias("word_struct") ).unnest("word_struct") ``` **Benefits:** - More efficient than multiple JQ queries - Automatic type handling through schema - Single operation extracts all fields - Better performance for simple field extraction ### JQ Queries (Best for Complex Operations) Complex array operations still use JQ for maximum power: ``` # Nested array traversal with variable binding '.segments[] as $seg | $seg.words[] | {...}' # Array length calculation '.words | length' # Array aggregation for averages '[.words[].probability] | add / length' # Object construction with mixed data levels '{word: .word, segment_start: $seg.start, ...}' ``` **When to Use JQ:** - Array aggregations (`length`, `add / length`) - Complex transformations - Variable binding and nested operations - Mathematical operations within queries ## Running the Example ### Prerequisites 1. Set your OpenAI API key: `bash export OPENAI_API_KEY="your-api-key-here"` 1. Ensure you have the whisper-transcript.json file in the same directory ### Execute ``` python json_processing.py ``` ### Expected Output The script will display: 1. Raw JSON data loading and casting 2. Word-level extraction (showing ~4000+ individual words) 3. Segment-level analysis (showing conversation segments) 4. Speaker summary analytics (showing 2 speakers with statistics) 5. Comprehensive pipeline summary ## Technical Highlights ### JSON Type System - Demonstrates proper JSON type casting from strings - Shows how to work with Fenic's JsonType for complex operations ### JQ Query Complexity - Simple field extraction: `.field` - Nested traversal: `.segments[].words[]` - Variable binding: `.segments[] as $seg` - Array operations: `length`, `add / length` - Object construction with mixed data sources ### Hybrid Processing Pattern - JSON extraction for granular data access - DataFrame aggregations for analytical operations - Type conversion between JSON and DataFrame types - Calculated fields using both JSON and DataFrame operations ## Learning Outcomes After studying this example, you'll understand: 1. **How to load and cast JSON data** in Fenic 2. **Struct casting + unnest** for efficient field extraction 3. **Complex JQ queries** for advanced data manipulation 4. **When to choose** struct casting vs JQ for different use cases 5. **Array processing** and aggregation within JSON 6. **Type conversion** between JSON and DataFrame types 7. **Hybrid workflows** combining multiple extraction approaches 8. **Performance optimization** for JSON processing pipelines 9. **Real-world data processing** patterns for audio/transcript analysis This example serves as a comprehensive reference for JSON processing in Fenic, showcasing both the power of JQ integration and the seamless bridge to traditional DataFrame analytics. --- # Markdown Processing with Fenic Canonical HTML: https://docs.fenic.ai/latest/examples/markdown_processing/ This example demonstrates how to process structured markdown documents using fenic's specialized markdown functions, JSON processing, and text extraction capabilities. We use the "Attention Is All You Need" paper, the foundational work that introduced the Transformer architecture, which powers modern large language models (LLMs). ## What This Example Shows Learn how to process structured markdown documents using fenic's comprehensive capabilities: - **`markdown.generate_toc()`** - Generate automatic table of contents from document headings - **`markdown.extract_header_chunks()`** - Extract and structure document sections into DataFrame rows - **`markdown.to_json()`** - Convert markdown to structured JSON for complex querying - **`json.jq()`** - Navigate complex document structures with powerful jq queries - **`text.extract()`** - Parse structured text using templates for field extraction - **DataFrame operations** - Filter, explode, unnest, and transform document data - **Text processing** - Split and parse content using fenic's templating functionality ## Files - `attention_is_all_you_need.md` - OCR output of the "Attention Is All You Need" paper - `markdown_processing.py` - Python script demonstrating markdown processing workflow ## Running the Example ``` cd examples/markdown_processing uv run python markdown_processing.py ``` Make sure you have your OpenAI API key set in your environment: ``` export OPENAI_API_KEY="your-api-key-here" ``` ## What You'll Learn This example showcases how fenic transforms unstructured markdown into structured, queryable data: ### 1. Document Structure Analysis - Load markdown documents into fenic DataFrames - Cast content to `MarkdownType` for specialized processing - Generate automatic table of contents from document hierarchy ### 2. Section Extraction and Structuring - Extract document sections using `markdown.extract_header_chunks()` - Convert nested arrays to DataFrame rows with `explode()` and `unnest()` - Work with structured section data (headings, content, hierarchy paths) ### 3. Traditional Text Processing - Filter DataFrames to find specific sections (e.g., References) - Parse structured content using text splitting with regex patterns - Handle academic citation formats and numbered references ### 4. JSON-Based Document Processing - Convert markdown to structured JSON with `markdown.to_json()` - Navigate complex nested document structures using `json.jq()` - Handle hierarchical content with powerful jq query language - Extract data from deeply nested JSON structures ### 5. Template-Based Text Extraction - Use `text.extract()` with templates for structured data parsing - Extract multiple fields from text in a single operation - Parse academic citations into separate reference numbers and content - Handle structured text formats with template patterns ### 6. Advanced DataFrame Operations - Combine multiple markdown functions in a single select statement - Chain operations for complex data transformations - Process hierarchical document structures efficiently - Cast between different data types (JsonType ↔ StringType) Perfect for building academic paper analysis pipelines, research document processing, citation extraction systems, or preparing structured content for downstream analysis. ## Guides # Creating DataFrames with an Explicit Schema Canonical HTML: https://docs.fenic.ai/latest/topics/create_dataframe_schema/ This page describes how to make a fenic `Schema` authoritative when you build a DataFrame with `Session.create_dataframe`, and the capabilities that unlocks: logical string-backed types, fixed-size embedding preservation, and typed empty frames. ## Overview By default, `create_dataframe(data)` infers a schema from the input. Inference is convenient, but it cannot know that a string column is really JSON or Markdown, and it normalizes fixed-size arrays to variable-length lists. Passing an explicit `schema` makes a complete, top-level `Schema` the source of truth for the resulting DataFrame: - **Field names are authoritative** β€” extra columns are rejected; column-oriented inputs (a dict, a Polars/pandas DataFrame, or a PyArrow table) must supply every schema column, while row-oriented `list[dict]` rows may omit fields (filled with null). See [The Schema Contract](#the-schema-contract) below. - **Columns are reordered** to the schema's field order. - **Values are physically coerced** to the schema's representation. - **The logical schema is preserved exactly**, including string-backed types. ``` import fenic as fc session = fc.Session.get_or_create(fc.SessionConfig(app_name="schema_intro")) schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) # Input columns can be in any order; values are coerced to the schema. df = session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) df.schema # Schema( # ColumnField(name='age', data_type=IntegerType) # ColumnField(name='name', data_type=StringType) # ) ``` Reach for an explicit schema when you need to seed logical string-backed types, preserve embeddings, or guarantee column order and dtypes; rely on inference for quick, ad-hoc frames. ## The Schema Contract When you pass `schema`, `create_dataframe` enforces a strict top-level contract: - **No unknown columns.** The data may not contain columns outside the schema. Column *order* in the input does not matter β€” the result is ordered by the schema. - **Missing columns depend on input shape.** For column-oriented inputs (a `dict`, a Polars/pandas DataFrame, or a PyArrow table) the columns must match the schema exactly. For row-oriented input (a `list[dict]`), any schema fields absent from the rows are backfilled with nulls. - **Physical coercion.** Each column is cast to the schema's Polars representation (for example `IntegerType` β†’ `Int64`, `EmbeddingType` β†’ `Array(Float32, dimensions)`). - **Typed errors.** Shape problems raise `ValidationError`; coercion or schema-validity problems raise `PlanError`. ``` from fenic.core.error import ValidationError, PlanError # A column outside the schema (here `schema` is the age/name schema above) # -> ValidationError try: session.create_dataframe({"name": ["A"], "age": ["1"], "extra": [9]}, schema=schema) except ValidationError as e: print(e) # Data columns must match the provided schema exactly; extra columns: ['extra'] # Duplicate schema field names -> PlanError dup = fc.Schema([ fc.ColumnField("x", fc.IntegerType), fc.ColumnField("x", fc.StringType), ]) try: session.create_dataframe({"x": [1]}, schema=dup) except PlanError as e: print(e) # Duplicate column names found: 'x'. Column names must be unique. ... ``` ### Typed empty frames With a schema, empty inputs (`[]`, `{}`, `pl.DataFrame()`, `pd.DataFrame()`, `pa.table({})`) produce a zero-row DataFrame that still carries the requested schema β€” useful for seeding a typed, empty starting point. ``` empty = session.create_dataframe([], schema=schema) empty.schema.column_names() # ['age', 'name'] empty.to_polars().height # 0 ``` Without a schema, an empty list is rejected β€” the no-schema path cannot infer types from nothing. ## Logical String-Backed Types (JSON / Markdown) fenic stores `JsonType`, `MarkdownType`, and `HtmlType` as physical strings while tracking their logical type so type-specific functions (JQ, markdown parsing, etc.) are available. Declaring these types in the schema tags the columns at ingestion, so you don't need a follow-up `.cast()`: ``` schema = fc.Schema([ fc.ColumnField("payload", fc.JsonType), fc.ColumnField("notes", fc.MarkdownType), ]) df = session.create_dataframe( {"payload": ['{"a": 1}'], "notes": ["# Title"]}, schema=schema, ) df.schema # ColumnField(name='payload', data_type=JsonType) # ColumnField(name='notes', data_type=MarkdownType) df.to_polars()["payload"].dtype # String (logical type, physical string) ``` The logical type stays visible in `df.schema` and drives downstream validation, while the value is stored as a plain string. **No content validation.** An explicit schema tags a column's *logical type*; it does not validate the *content*. A `JsonType` column is not parsed or checked for well-formed JSON at ingestion β€” malformed values surface when a JSON operation runs on them. For columns produced mid-pipeline (where no ingestion schema exists), continue to use `.cast(fc.JsonType)` / `.cast(fc.MarkdownType)` as shown in the [JSON Processing](https://docs.fenic.ai/latest/examples/json_processing/index.md) and [Markdown Processing](https://docs.fenic.ai/latest/examples/markdown_processing/index.md) examples. ## Embedding Preservation fenic represents `EmbeddingType` as a fixed-size Polars array (`Array(Float32, dimensions)`). The schema-free ingestion path normalizes all fixed-size arrays to variable-length lists (`pl.List`), which loses the fixed-width guarantee. An explicit schema preserves the fixed-size array through local execution all the way to `to_polars()`: ``` emb_schema = fc.Schema([ fc.ColumnField("id", fc.IntegerType), fc.ColumnField("embedding", fc.EmbeddingType(dimensions=3, embedding_model="example/model")), ]) edf = session.create_dataframe( {"id": [1, 2], "embedding": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]}, schema=emb_schema, ) edf.to_polars()["embedding"].dtype # Array(Float32, shape=(3,)) ``` This lets you bring precomputed embeddings into fenic without losing their fixed-size representation. (Constructing the frame does not call an embedding provider β€” the `embedding_model` on the type is metadata.) ## Relationship to Schema-Free Ingestion The no-schema path is unchanged and fully behavior-compatible: - Empty lists are still rejected. - Types are still inferred from the data. - Fixed-size arrays are still normalized to `pl.List`. Only supplying a `schema` opts you into authoritative names, ordering, coercion, and logical-type/embedding preservation. For related ingestion-normalization behavior on temporal columns, see [Timestamps and Dates](https://docs.fenic.ai/latest/topics/timestamps_and_dates/index.md). --- # Markdown -> JSON Conversion Canonical HTML: https://docs.fenic.ai/latest/topics/markdown-json/ The `markdown.to_json()` function converts Markdown-formatted text into a hierarchical JSON representation that preserves the document's structure and formatting. This JSON format is optimized for document chunking, semantic analysis, and querying with tools like `jq`. ## Overview The JSON representation follows a tree structure where each node represents a different Markdown element (headings, paragraphs, lists, etc.). The schema is designed to be both expressive and practical, making it easier to work with Markdown content programmatically. ## Key Features - **Hierarchical Structure**: Content is organized into nested sections based on heading levels - **Rich Formatting**: Preserves all Markdown formatting including inline styles, links, and code blocks - **Metadata Support**: Optional document-level metadata for source, title, tags, and dates - **Complete Markdown Support**: Handles all standard Markdown elements including: - Headings (h1-h6) - Paragraphs with inline formatting - Lists (ordered, unordered, task lists) - Tables with alignment - Code blocks with or without language info - Blockquotes - HTML blocks and inline elements ## Usage Examples ### Basic Conversion ``` df.select(markdown.to_json(col("markdown_text"))) ``` ### Extract Headings with jq ``` # Get all level-2 headings df.select(json.jq(markdown.to_json(col("md")), ".. | select(.type == 'heading' and .level == 2)")) ``` ## JSON Schema The schema defines a tree structure where each node can be either a block-level element or an inline element. Raw JSON Schema ``` { "$schema": "http://json-schema.org/draft-07/schema#", "title": "Document", "type": "object", "required": ["type", "children"], "properties": { "type": { "const": "document" // Root node type }, "metadata": { // Optional document metadata "type": "object", "properties": { "source": { "type": "string" }, // Source document path/URL "title": { "type": "string" }, // Document title "tags": { // Document tags "type": "array", "items": { "type": "string" } }, "date": { "type": "string" } // Document date }, "additionalProperties": true }, "children": { // Main content blocks "type": "array", "items": { "$ref": "#/definitions/BlockNode" } } }, "definitions": { "BlockNode": { // Block-level elements "oneOf": [ { "$ref": "#/definitions/Heading" }, // # Heading { "$ref": "#/definitions/Paragraph" }, // Regular paragraph { "$ref": "#/definitions/List" }, // Ordered/unordered list { "$ref": "#/definitions/Blockquote" }, // > Blockquote { "$ref": "#/definitions/CodeBlock" }, // ``` Code block { "$ref": "#/definitions/Table" }, // | Table | { "$ref": "#/definitions/ThematicBreak" },// --- { "$ref": "#/definitions/HtmlBlock" } // HTML block ] }, "Heading": { // Heading structure "type": "object", "required": ["type", "level", "content", "children"], "properties": { "type": { "const": "heading" }, "level": { "type": "integer", "minimum": 1, "maximum": 6 }, // h1-h6 "content": { // Heading text with inline formatting "type": "array", "items": { "$ref": "#/definitions/InlineNode" } }, "children": { // Content under this heading "type": "array", "items": { "$ref": "#/definitions/BlockNode" } } } }, "Paragraph": { // Paragraph structure "type": "object", "required": ["type", "content"], "properties": { "type": { "const": "paragraph" }, "content": { // Paragraph content with inline formatting "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } }, "List": { // List structure "type": "object", "required": ["type", "ordered", "tight", "items"], "properties": { "type": { "const": "list" }, "ordered": { "type": "boolean" }, // true for ordered lists "start": { "type": "integer" }, // Starting number for ordered lists "tight": { "type": "boolean" }, // Whether list items are tightly packed "items": { // List items "type": "array", "items": { "$ref": "#/definitions/ListItem" } } } }, "ListItem": { // List item structure "type": "object", "required": ["children"], "properties": { "checked": { "type": ["boolean", "null"] }, // For task lists "children": { // Item content "type": "array", "items": { "$ref": "#/definitions/BlockNode" } } } }, "CodeBlock": { // Code block structure "type": "object", "required": ["type", "text", "fenced"], "properties": { "type": { "const": "code_block" }, "info": { "type": "string" }, // Language info "language": { "type": "string" }, // Language identifier "text": { "type": "string" }, // Code content "fenced": { "type": "boolean" }, // Whether using ``` or indented "fence_char": { "type": "string" },// Fence character used "fence_length": { "type": "integer", "minimum": 1 } // Number of fence chars } }, "Table": { // Table structure "type": "object", "required": ["type", "alignments", "header", "rows"], "properties": { "type": { "const": "table" }, "alignments": { // Column alignments "type": "array", "items": { "type": ["string", "null"], "enum": ["left", "center", "right", null] } }, "header": { // Table header "type": "array", "items": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } }, "rows": { // Table rows "type": "array", "items": { "type": "array", "items": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } } } }, "InlineNode": { // Inline elements "oneOf": [ { "$ref": "#/definitions/Text" }, // Plain text { "$ref": "#/definitions/Strong" }, // **Bold** { "$ref": "#/definitions/Emphasis" }, // *Italic* { "$ref": "#/definitions/Link" }, // [Link](url) { "$ref": "#/definitions/InlineCode" }, // `code` { "$ref": "#/definitions/Image" }, // ![Alt](src) { "$ref": "#/definitions/Strikethrough" },// ~~Strikethrough~~ { "$ref": "#/definitions/HtmlInline" }, // { "$ref": "#/definitions/HardBreak" } // Line break ] } }, "Text": { "type": "object", "required": ["type", "text"], "properties": { "type": { "const": "text" }, "text": { "type": "string" } } }, "Strong": { "type": "object", "required": ["type", "content"], "properties": { "type": { "const": "strong" }, "content": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } }, "Emphasis": { "type": "object", "required": ["type", "content"], "properties": { "type": { "const": "emphasis" }, "content": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } }, "Link": { "type": "object", "required": ["type", "href", "content"], "properties": { "type": { "const": "link" }, "href": { "type": "string" }, "title": { "type": "string" }, "content": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } }, "InlineCode": { "type": "object", "required": ["type", "text"], "properties": { "type": { "const": "inline_code" }, "text": { "type": "string" } } }, "Image": { "type": "object", "required": ["type", "src"], "properties": { "type": { "const": "image" }, "src": { "type": "string" }, "alt": { "type": "string" }, "title": { "type": "string" } } }, "Strikethrough": { "type": "object", "required": ["type", "content"], "properties": { "type": { "const": "strikethrough" }, "content": { "type": "array", "items": { "$ref": "#/definitions/InlineNode" } } } }, "HtmlInline": { "type": "object", "required": ["type", "text"], "properties": { "type": { "const": "html_inline" }, "text": { "type": "string" } } }, "HardBreak": { "type": "object", "required": ["type"], "properties": { "type": { "const": "hardbreak" } } } } } ``` ## Example: Markdown to JSON Conversion Here's a simple example showing how a Markdown document is converted to JSON: Raw Markdown ``` # Project Documentation This is a sample document with various Markdown elements. ## Features - **Bold text** and _italic text_ - `Inline code` and [links](https://example.com) - Code blocks with syntax highlighting: ```python def hello(): print("Hello, world!") ``` ### Nested Section > This is a blockquote with some text. | Column 1 | Column 2 | | -------- | -------- | | Cell 1 | Cell 2 | ``` The above Markdown is converted to this JSON structure: Converted JSON ``` { "type":"document", "children":[ { "type":"heading", "level":1, "content":[ { "type":"text", "text":"Project Documentation" } ], "children":[ { "type":"paragraph", "content":[ { "type":"text", "text":"This is a sample document with various Markdown elements." } ] }, { "type":"heading", "level":2, "content":[ { "type":"text", "text":"Features" } ], "children":[ { "type":"list", "ordered":false, "tight":true, "items":[ { "children":[ { "type":"paragraph", "content":[ { "type":"strong", "content":[ { "type":"text", "text":"Bold text" } ] }, { "type":"text", "text":" and " }, { "type":"emphasis", "content":[ { "type":"text", "text":"italic text" } ] } ] } ] }, { "children":[ { "type":"paragraph", "content":[ { "type":"inline_code", "text":"Inline code" }, { "type":"text", "text":" and " }, { "type":"link", "href":"https://example.com", "content":[ { "type":"text", "text":"links" } ] } ] } ] }, { "children":[ { "type":"paragraph", "content":[ { "type":"text", "text":"Code blocks with syntax highlighting:" } ] } ] } ] }, { "type":"code_block", "language":"python", "text":"def hello():\n print(\"Hello, world!\")", "fenced":true, "fence_char":"`", "fence_length":3 }, { "type":"heading", "level":3, "content":[ { "type":"text", "text":"Nested Section" } ], "children":[ { "type":"blockquote", "children":[ { "type":"paragraph", "content":[ { "type":"text", "text":"This is a blockquote with some text." } ] } ] }, { "type":"table", "alignments":[ null, null ], "header":[ [ { "type":"text", "text":"Column 1" } ], [ { "type":"text", "text":"Column 2" } ] ], "rows":[ [ [ { "type":"text", "text":"Cell 1" } ], [ { "type":"text", "text":"Cell 2" } ] ] ] } ] } ] } ] } ] } ``` This example demonstrates how: 1. The document structure is preserved in a tree format 2. Each Markdown element is represented as a specific node type 3. Inline formatting (bold, italic, code, links) is nested within their parent elements 4. Headings create a hierarchy with their content as children 5. Complex elements like tables and code blocks maintain their structure and metadata ## Common Use Cases 1. **Document Analysis**: Extract and analyze document structure 2. **Content Extraction**: Pull specific elements like code blocks or headings 3. **Document Transformation**: Convert between formats while preserving structure 4. **Semantic Search**: Build search indices based on document structure 5. **Content Validation**: Verify document structure and formatting ## Best Practices 1. Use `jq` queries to extract specific elements from the JSON structure 2. Leverage the hierarchical nature for document chunking 3. Use metadata for document organization and search 4. Consider the schema when building tools that process the JSON output ```` --- # Fenic MCP: Create and Serve Catalog Tools Canonical HTML: https://docs.fenic.ai/latest/topics/fenic-mcp/ This guide shows how to expose User Defined fenic tools via an MCP server. User Defined Tools are created by adding placeholder values to DataFrame operations to be filled in at runtime, and managed in the Fenic Catalog. In most respects, these tools are like SQL Macros or parameterized views. Just like one might create a macro as: ``` CREATE MACRO get_users(user_name) AS TABLE SELECT * FROM users WHERE name = user_name; ``` One can create a Tool in Fenic: ``` import fenic as fc filter_tool_df = df.filter(fc.col("name") == fc.tool_param("user_name", fc.StringType)) session.catalog.create_tool( "users_by_name", "Filter users by name", filter_tool_df, tool_params=[ # If default values are provided, the parameters will be marked as `Optional` in the MCP API Spec. fc.ToolParam(name="user_name", description="User's Name (Exact Match)") ] ) ``` ## Prerequisites - A working Fenic installation and a way to create a `Session` (optionally via a `SessionConfig` JSON file). - Any required provider API keys available in your environment (e.g., `OPENAI_API_KEY`). ## Step 1: Create or augment tables with descriptions You can set a table description when creating the table or for an existing table. Create with description: ``` from fenic import ColumnField, IntegerType, Schema, Session session.catalog.create_table( "orders", Schema([ColumnField("order_id", IntegerType)]), description="Customer orders with line-item totals", ) ``` Create by writing a DataFrame, then set a description: ``` df = session.create_dataframe({"order_id": [1, 2, 3]}) df.write.save_as_table("orders", mode="overwrite") session.catalog.set_table_description("orders", "Customer orders with line-item totals") ``` Add or update description on an existing table: ``` session.catalog.set_table_description("orders", "Customer orders with line-item totals") ``` You can confirm the description (and schema) via: ``` meta = session.catalog.describe_table("orders") print(meta.description) print([f.name for f in meta.schema.column_fields]) ``` ## Step 2: Create catalog tools Tools encapsulate a parameterized query and an optional row limit. Define inputs via `tool_param` placeholders in your query and register their schema via `ToolParam`, then save with `create_tool`. ``` import fenic as fc session = Session.get_or_create(fc.SessionConfig( app_name="mcp_example", )) # Create a small users table and add a description users_df = session.create_dataframe({ "name": ["Alice", "Bob", "Charlie", "Diana"], "age": [25, 40, 31, 18], }) users_df.write.save_as_table("users", mode="overwrite") session.catalog.set_table_description("users", "User information") users = session.table("users") # Tool A: Filter users by optional age range. Uses coalesce to evaluate to True if the user does not pass in one side of the range. optional_min = fc.coalesce(fc.col("age") >= fc.tool_param("min_age", fc.IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= fc.tool_param("max_age", fc.IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_by_age_range", "Filter users by age", core_filter, tool_params=[ # If default values are provided, the parameters will be marked as `Optional` in the MCP API Spec. fc.ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), fc.ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) # Tool B: Case-sensitive regex search by name (use (?i) for case-insensitive). name_search_query = users.filter(fc.col("name").rlike(fc.tool_param("name_regex"))) # If a default is not provided, the parameter will be marked as `required` in the MCP API Spec. name_search_params = [ fc.ToolParam( name="name_regex", description="Search for users by name, using a regular expression. (use (?i) for case-insensitive)", ) ] session.catalog.create_tool( tool_name="users_by_name_regex", tool_description="Return users whose name matches the provided regex.", tool_query=name_search_query, tool_params=name_search_params, result_limit=100, ) ``` List, fetch, or drop tools: ``` all_tools = session.catalog.list_tools() age_tool = session.catalog.describe_tool("users_by_age_range") search_tool = session.catalog.describe_tool("users_by_name_regex") session.catalog.drop_tool("users_by_age_range", ignore_if_not_exists=True) session.catalog.drop_tool("users_by_name_regex", ignore_if_not_exists=True) ``` ### Step 2a: Auto-generate system tools from catalog tables You can generate a suite of reusable data tools (Schema, Profile, Read, Search Summary, Search Content, Analyze) directly from catalog tables and their descriptions. This is helpful for quickly exposing exploratory and read/query capabilities to MCP. Available tools include: - Schema: list columns/types for any or all tables - Profile: column statistics (counts, basic numeric analysis [min, max, mean, etc.], contextual information for text columns [average_length, etc.]) - Read: read a selection of rows from a single table. These rows can be paged over, filtered and can use column projections. - Search Summary: literal or regex search across all text columns in all tables -- returns back dataframe names with result counts. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Search Content: literal or regex search across a single table, specifying one or more text columns to search across -- returns back rows corresponding to the query. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Analyze: Write raw SQL to perform complex analysis on one or more tables. Requirements: - Each table must exist and have a non-empty description (see Step 1). Example: ``` from fenic import Session from fenic.api.mcp.server import create_mcp_server from fenic.api.mcp.tools import SystemToolConfig session = Session.get_or_create(...) server = create_mcp_server( session, server_name="Fenic MCP", system_tools=SystemToolConfig( table_names=session.catalog.list_tables(), tool_namespace="Dataset Exploration", max_result_rows=200, ), ) ``` ## Step 3a: Serve tools programmatically Use the MCP server helpers to serve existing catalog tools. To use all catalog tools in the MCP server, pass `session.catalog.list_tools` to `create_mcp_server`: ``` from fenic import Session,SessionConfig from fenic.api.mcp.server import create_mcp_server, run_mcp_server_sync, run_mcp_server_async, run_mcp_server_asgi session = Session.get_or_create(SessionConfig( app_name="mcp_example", ... )) server = create_mcp_server(session, server_name="Fenic MCP", user_defined_tools=session.catalog.list_tools()) # Run HTTP server (defaults shown); if additional configuration is required, any argument that can be passed to FastMCP `run` can be passed here # run_mcp_server_sync( server, transport="http", host="127.0.0.1", port=8000, stateless_http=True, path="/mcp", ) # If already within an async context, the server can run inside that existing context instead of creating a new event loop await run_mcp_server_async( server, transport="http", host="127.0.0.1", port=8000, stateless_http=True, path="/mcp", ) # Finally, in production environments it might be necessary to configure the application with additional middleware, or serve the application from something other # than uvicorn -- in that case, we expose `run_mcp_server_asgi`, which creates a Starlette ASGI application that can be plugged in to your existing stack asgi_app = run_mcp_server_asgi( server, stateless_http=True, path="/mcp", # middleware = [...] ) ``` ## Step 3b: Serve tools via CLI (fenic-serve) The CLI starts an MCP server directly from your catalog. By default, it serves all registered tools in the current database, using uvicorn. Basic usage (serve all tools registered in catalog, using the existing `mcp_app`): ``` fenic-serve --app-name mcp_example --port 8000 --host 127.0.0.1 ``` Serve specific tools only: ``` fenic-serve --app-name mcp_example --tools users_by_age_range users_by_name_regex --port 8000 ``` Provide a session configuration via JSON file, and customize the path. ``` fenic-serve --config-file ./session.config.json --port 8000 --path / ``` Example `session.config.json` (minimal): ``` { "app_name": "mcp_demo", "semantic": { "language_models": { "gpt-4.1-nano": { "model_name": "gpt-4.1-nano", "rpm": 2500, "tpm": 2000000 } }, "default_language_model": "gpt-4.1-nano" } } ``` Environment variables for model providers (if your tools use semantic operators) should be set in your shell, or via your runner (for example: `uv run --env-file .env ...`). ## Troubleshooting - No tools found: ensure you have created tools in the current database (`session.catalog.list_tools()`). - Table descriptions: recommended for documentation; set via `create_table(..., description=...)` or `set_table_description()`. - HTTP path: The default path for the mcp server is `/mcp`. - SessionConfig exposes `to_json` for converting an existing SessionConfig to a jsonified version. --- # OpenRouter integration Canonical HTML: https://docs.fenic.ai/latest/topics/openrouter/ This page describes how Fenic integrates with OpenRouter, how to configure it, and how to work around common issues (especially with structured outputs on non-frontier models). ## What is supported - **Provider**: `ModelProvider.OPENROUTER` - **Client**: Uses the OpenAI SDK with OpenRouter `base_url` and headers - **Dynamic model loading**: Models are fetched from OpenRouter on first use and cached - **Profiles**: Profile fields map to OpenRouter request parameters via `extra_body` - **Rate limiting**: Adaptive strategy that respects `X-RateLimit-Reset` and provider-specific 429s ## Setup ### Requirements - Set `OPENROUTER_API_KEY` in your environment - *fenic* automatically configures an OpenAI SDK client to point at OpenRouter (`base_url=https://openrouter.ai/api/v1`) ### Session configuration You can pick OpenRouter via session configuration. ``` from _fenic_.api.session.config import SessionConfig config = SessionConfig( language_model=SessionConfig.OpenRouterLanguageModel( model_name="openai/gpt-4o", # Any OpenRouter model id profiles={ "default": SessionConfig.OpenRouterLanguageModel.Profile( models=["openai/gpt-4", "openai/gpt-4.1"], # Fallback models, if the primary model is unavailable reasoning_effort="medium", # For reasoning-capable models provider=SessionConfig.OpenRouterLanguageModel.Provider( sort="latency", # "price" | "throughput" | "latency" only=["OpenAI"], # Route only to specific providers exclude=["Azure"] # exclude a provider from routing order=["OpenAI", "Azure"] # try providers in a specific order for routing quantizations=["unknown", "fp8", "bf16", "fp16" "fp32"] # only allow providers that offer the model with one of the following quantization levels data_collection="deny" # only allow providers that do not collect/train on prompt data max_prompt_price=5 # max price ($USD) per 1M input tokens max_completion_price=10 # max price ($USD) per 1M output tokens ), ), }, default_profile="default", ) ) ``` ## Dynamic model loading and the model catalog - On first use of an OpenRouter model, Fenic fetches available models from your OpenRouter account, translates capabilities and pricing, and caches them. - Available models depend on the providers you’ve enabled in OpenRouter. If you don’t see a model, ensure it’s available in your OpenRouter dashboard. ## Profile configuration OpenRouter profiles support the following configuration options: - **`models`**: List of fallback models to use if the primary model is unavailable - **`reasoning_effort`**: For reasoning-capable models, set to `"none"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, or `"xhigh"`. - **`reasoning_max_tokens`**: Token budget for reasoning for Gemini/Anthropic models - **`provider`**: Provider routing preferences with these options: - `sort`: Route by `"price"`, `"throughput"`, or `"latency"` - `only`: List of providers to exclusively use (e.g., `["OpenAI", "Anthropic"]`) - `exclude`: List of providers to avoid - `order`: Specific provider order to try - `quantizations`: Allowed model quantizations (e.g., `["fp16", "bf16"]`) - `data_collection`: Set to `"allow"` or `"deny"` for data retention preferences - `max_prompt_price` / `max_completion_price`: Price limits per 1M tokens ## Structured outputs with OpenRouter OpenRouter exposes multiple ways to request structured outputs, but support varies by model and provider. Frontier models (Anthropic, OpenAI, Gemini) generally behave similarly to their native clients. Other models can vary widely depending on the model's capabilities and the provider's inference harness. For *fenic* use cases involving structured outputs, prefer a frontier or another widely used model family (Llama 3/4, Mistral, etc.). 1. **Pydantic Structured Outputs (via Response Format)** [Available when the target model lists `structured_outputs` in its `supported_parameters`] 2. This option is generally preferred, the JSON Schema corresponding to the provided Pydantic Model is sent to the model, which constrains its output to JSON and (in theory) coerces the response into the proper format. 3. **Forced Tool Calling (with JSON Schema)** [Available when the target model lists both `tools` and `tool_choice` in its `supported_parameters`] 4. If native structured outputs are unavailable, or the profile uses `structured_output_strategy="prefer_tools"`, *fenic* registers an `output_formatter` tool whose arguments follow the Pydantic model's JSON Schema and explicitly forces the model to call that tool. 5. Merely listing a tool is not sufficient: OpenRouter otherwise defaults `tool_choice` to `auto`, which allows the model to return ordinary text instead of the required structured result. If the model does not support native `structured_outputs` or the combination of `tools` and `tool_choice`, and a semantic operation that requires structured output (such as `semantic.extract`) is used, *fenic* fails fast before sending requests. Forced tool choice is incompatible with manual extended thinking on Anthropic models. Through OpenRouter, manual thinking includes profiles with `reasoning_max_tokens` and effort-based reasoning on older Claude models that do not support adaptive thinking. In those cases, *fenic* uses native structured outputs when they are available; otherwise, it fails fast with configuration guidance. Adaptive thinking supports forced tool choice. ## Rate limiting Fenic uses an adaptive strategy for OpenRouter requests: - Multiplicative backoff on 429s - Honors `X-RateLimit-Reset` (temporary cooldown window) - Accepts provider RPM hints when supplied - Additive RPM increase after consecutive successes (when no hint is active) ## Troubleshooting ### Structured Outputs #### Error: `No endpoints found that can handle the requested parameters.` - This is likely occuring because the model configuration or the OpenRouter account attached to the API Key is forcing the use of a single provider or a subset of providers for a given model that do not support `structred_outputs`. - Try setting `structured_output_strategy` in the model config to `prefer_tools`. This will force *fenic* to use `Forced Tool Calling` instead of `structured_outputs`, which has broader compatibility. #### Error `{'error': {'message': 'Provider returned error', 'code': 400, 'metadata': {'raw': '{"error":{"message":"invalid schema for response format: response_format/json_schema: ..."}}\n', 'provider_name': 'Groq'}}` (or similar) - Examine the error message and determine if a single provider is causing the issue. Try to `exclude` the provider from the model config to see if this resolves the problem. #### Model is listed as supporting structured outputs, but generation performance is much slower than expected - Examine in detail the model's overview page in OpenRouter -- somtimes models are listed as supporting `structured_outputs` because a single provider out of 5 supports the functionality, while the other 4 providers only support `tools`. If this is the case, try setting `structured_output_strategy` in the model config to `prefer_tools`. This will force *fenic* to use `Forced Tool Calling` instead of `structured_outputs`, allowing the request to be load-balanced across all of the model providers. #### Generated JSON is malformed or inconsistently empty - Examine the generated error messages closely, if a single provider is a common factor, try adding that provider to the `exclude` list in the provider routing preferences section of the model config. - Reduce temperature for extraction steps. - Try setting `structured_output_strategy` in the model config to `prefer_tools`. This will force *fenic* to use `Forced Tool Calling` instead of `structured_outputs`, - Try a different model with the same task -- if a different model works more consistently, there is likely some ### Rate Limiting #### 429 / rate limit exceeded - *fenic* will attempt to reduce the rate at which the model is being called if multiple 429s are encountered. - For some models (typically free models or smaller models with fewer providers, ex. `mistralai/mistral-small-3.2-24b-instruct`) , OpenRouter will throttle at a very low limit, like 100 RPM. *fenic* will alert the user of this throttling in the logs. - Persistent 429s usually mean the model is ill-suited for the scale of the use-case -- consider selecting a more popular model that has more capacity. ### General #### Model Not available - Ensure that the model indeed exists in the OpenRouter model list. - Ensure that the OpenRouter account attached to the API Key is not excluding providers in its settings -- if the account excludes the only providers that offer a given model, the model will no longer be available for use. #### Reasoning not working - Verify the model supports reasoning by checking its capabilities. - Use either `reasoning_effort` or `reasoning_max_tokens` in your profile, not both. #### Bad Provider Behavior - Use the `provider.only` field to restrict to specific providers, or `provider.exclude` to avoid problematic ones. To persist these settings and avoid needing to set them in the session each time, add them to the [OpenRouter Settings](https://openrouter.ai/settings/preferences) `Allowed Providers` and `Ignored Providers` lists. ## References - OpenRouter docs: [API reference](https://openrouter.ai/docs/api-reference/chat-completion) --- # Timestamps and Dates Canonical HTML: https://docs.fenic.ai/latest/topics/timestamps_and_dates/ This page describes how Fenic handles temporal data types (`TimestampType` and `DateType`), including timezone behavior, storage precision, and common operations. ## Overview Fenic provides two temporal data types along with various temporal manipulation functions in the fenic.dt module. - **`TimestampType`**: Represents a point in time with microsecond precision, always stored in UTC - **`DateType`**: Represents a calendar date without time information Both types and related functionality follow similar patterns to PySpark, except that fenic converts all timestamps to UTC timezone instead of the session local timezone. ## Key Features - **Microsecond precision**: Timestamps support up to 1/1,000,000 second precision - **UTC-first design**: All timestamps are normalized to UTC during ingestion - **Automatic timezone conversion**: Timezone-aware inputs are converted to UTC automatically - **Consistent behavior**: Same behavior across all data sources (Parquet, CSV, in-memory DataFrames, tables) - **Rich date/time functions**: Comprehensive set of functions for temporal operations - **Timezone conversion utilities**: Functions to work with local timezones while maintaining UTC storage ## TimestampType and DateType ### TimestampType Represents a point in time stored as microseconds since the Unix epoch (1970-01-01 00:00:00 UTC). **Characteristics:** - Precision: Microseconds (ΞΌs) - Timezone: Always UTC - Storage: Int64 (microseconds from epoch) - Python type: `datetime.datetime` with `tzinfo=UTC` ### DateType Represents a calendar date without time-of-day information. **Characteristics:** - Precision: Day - Timezone: Not applicable - Storage: Int32 (days from epoch) - Python type: `datetime.date` ## PySpark-like behavior Fenic's timestamp handling and functionality is very similar to PySpark, with the following differences: - **Default timezone**: Fenic always converts timestamps to **UTC** whereas PySpark converts to a configurable timezone that defaults to the session's global timezone. - **Timezone consistency**: Fenic guarantees all timestamps are UTC, eliminating ambiguity when session configuration changes ## Conversion and Casting Behavior Reference The following table shows how different input types are handled when loading or casting to `TimestampType` and `DateType`: | **Input Type** | **Operation** | **Input Example** | **Result** | **Notes** | | --- | --- | --- | --- | --- | | **Naive Python datetime** | Load into TimestampType | `datetime.datetime(2025, 1, 1, 10, 0)` | `2025-01-01 10:00:00 UTC` | Interpreted as UTC | | **UTC Python datetime** | Load into TimestampType | `datetime.datetime(2025, 1, 1, 10, 0, tzinfo=UTC)` | `2025-01-01 10:00:00 UTC` | Already UTC, no conversion | | **LA Python datetime** | Load into TimestampType | `datetime.datetime(2025, 1, 1, 10, 0, tzinfo=LA)` | `2025-01-01 18:00:00 UTC` | Converted: +8 hour offset | | **Naive Polars Datetime** | Load into TimestampType | `pl.Datetime` (no timezone) | Interpreted as UTC | Applied during ingestion | | **Polars Datetime (UTC)** | Load into TimestampType | `pl.Datetime(time_zone="UTC")` | Kept as UTC | No conversion needed | | **Polars Datetime (LA)** | Load into TimestampType | `pl.Datetime(time_zone="America/Los_Angeles")` | Converted to UTC | Timezone conversion applied | | **String (ISO 8601 with TZ)** | `cast(TimestampType)` | `"2025-01-01T10:00:00+08:00"` | `2025-01-01 02:00:00 UTC` | Parses timezone, converts to UTC | | **String (ISO 8601 no TZ)** | `cast(TimestampType)` | `"2025-01-01T10:00:00.000"` | `None` | Currently returns null (limitation) | | **String with format** | `to_timestamp(col, format)` | `"01-15-2025 10:30:00"` with format `"MM-dd-yyyy HH:mm:ss"` | `2025-01-15 10:30:00 UTC` | Parsed as UTC if no TZ in format | | **String with format + TZ** | `to_timestamp(col, format)` | `"01-15-2025 10:30:00 +08:00"` with format `"MM-dd-yyyy HH:mm:ss XXX"` | `2025-01-15 02:30:00 UTC` | Timezone parsed and converted | | **Integer (microseconds)** | `cast(TimestampType)` | `1735729800000000` | `2025-01-01 10:30:00 UTC` | Interpreted as ΞΌs from epoch | | **TimestampType** | `cast(DateType)` | `2025-01-01 10:30:00 UTC` | `2025-01-01` | Truncates time, keeps date | | **DateType** | `cast(TimestampType)` | `2025-01-01` | `2025-01-01 00:00:00 UTC` | Midnight UTC | | **String (date format)** | `cast(DateType)` | `"2025-01-01"` | `2025-01-01` | Parses as date | | **String with format** | `to_date(col, format)` | `"01-27-2025"` with format `"MM-dd-yyyy"` | `2025-01-27` | Custom format parsing | | **Integer (days)** | `cast(DateType)` | `20000` | `2024-10-04` | Days since epoch | | **SQL CAST to DATE** | `session.sql("SELECT CAST(col AS DATE)")` | `"2025-01-01"` | `2025-01-01` | DuckDB parses date, returned as DateType | | **SQL CAST to TIMESTAMP** | `session.sql("SELECT CAST(col AS TIMESTAMP)")` | `"2025-01-01 10:30:00"` | `2025-01-01 10:30:00 UTC` | DuckDB parses timestamp, converted to UTC | | **SQL DATE_TRUNC** | `session.sql("SELECT DATE_TRUNC('day', ts)")` | Timestamp column | `2025-01-01 00:00:00 UTC` | DuckDB truncates, result converted to UTC | | **TimestampType** | `date_format(col, format)` | `2025-01-15 10:30:45 UTC` with format `"yyyy-MM-dd HH:mm:ss"` | `"2025-01-15 10:30:45"` | Formats timestamp as string | | **DateType** | `date_format(col, format)` | `2025-01-15` with format `"MM/dd/yyyy"` | `"01/15/2025"` | Formats date as string | ### Timezone Conversion Function Reference | **Function** | **Input** | **Result** | **Explanation** | | --- | --- | --- | --- | | `to_utc_timestamp(col, "America/Los_Angeles")` | `2025-01-15 10:30:00 UTC` | `2025-01-15 18:30:00 UTC` | Treats input as LA wall-clock time, converts to UTC | | `from_utc_timestamp(col, "America/Los_Angeles")` | `2025-01-15 10:30:00 UTC` | `2025-01-15 02:30:00 UTC` | Converts input to LA wall-clock, then represents as UTC | ## Generating Timestamps and Dates ### Creating Timestamps You can create timestamps by loading Python `datetime` objects or Polars DataFrames with datetime columns, or by using Fenic's native functions like `current_timestamp()` and `now()`. ``` import datetime import zoneinfo from fenic.api.functions.dt import current_timestamp, now # Create a timestamp from Python datetime df = session.create_dataframe({ "event_time": [datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df.show() # Output: 2025-01-15 10:30:00 UTC # Get current timestamp df = session.create_dataframe({"id": [1, 2, 3]}) df = df.select(current_timestamp().alias("ts")) df.show() # Output: 2025-01-15 14:23:45 UTC (current time) # Alternative: use now() df = df.select(now().alias("ts")) df.show() # Output: 2025-01-15 14:23:45 UTC (current time) ``` #### Creating Dates You can create dates by loading Python `date` objects or Polars DataFrames with date columns, or by using Fenic's native `current_date()` function. ``` import datetime import zoneinfo from fenic.api.functions.dt import current_timestamp, now # Create a timestamp from Python datetime df = session.create_dataframe({ "event_time": [datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df.show() # Output: 2025-01-15 10:30:00 UTC # Get current timestamp using Fenic functions df = session.create_dataframe({"id": [1, 2, 3]}) df = df.select(current_timestamp().alias("ts")) df.show() # Output: 2025-01-15 14:23:45 UTC (current time) # Alternative: use now() df = df.select(now().alias("ts")) df.show() # Output: 2025-01-15 14:23:45 UTC (current time) ``` ## Ingestion and Persistence ### Reading and Writing Files When you write timestamp or date data to files or tables, Fenic writes them in your **local session timezone** (the timezone of your computer). When you read them back, Fenic automatically converts timestamps back to **UTC**. ``` import datetime import zoneinfo # Create a DataFrame with a date df = session.create_dataframe({ "event_date": [datetime.date(2024, 1, 4)] }) # Write to CSV (saved in local session timezone, e.g., "2024-01-04" in PST) df.write.csv("events.csv") # Read back from CSV (automatically converted to UTC) df_read = session.read.csv("events.csv") df_read.show() # Output: 2024-01-04 (dates are timezone-agnostic) ``` For timestamps: ``` import datetime import zoneinfo # Create a DataFrame with a timestamp df = session.create_dataframe({ "created_at": [datetime.datetime(2024, 1, 4, 7, 10, 13, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) # Write to CSV (saved with timezone info, e.g., "2024-01-04 07:10:13.000000+00:00") df.write.csv("events.csv") # Read back from CSV (automatically normalized to UTC) df_read = session.read.csv("events.csv") df_read.show() # Output: 2024-01-04 07:10:13 UTC ``` The same behavior applies to: - **Parquet files**: Timestamps stored with timezone metadata, automatically converted to UTC on read - **Fenic tables** (DuckDB): DuckDB session timezone is always UTC, ensuring consistency - **In-memory DataFrames**: Polars DataFrames with timezone-aware or naive timestamps are normalized to UTC ### SQL Queries When using `session.sql()`, the underlying SQL engine (DuckDB) converts timestamps to the local session timezone during query execution. However, when Fenic loads the results back from SQL queries (including views and tables), all timestamps are automatically converted back to UTC. ``` import datetime import zoneinfo # Create DataFrame with timestamps in different timezones ts_la = datetime.datetime(2025, 1, 2, 1, 1, 1, tzinfo=zoneinfo.ZoneInfo("America/Los_Angeles")) ts_utc = datetime.datetime(2025, 1, 2, 1, 1, 1, tzinfo=zoneinfo.ZoneInfo("UTC")) df = session.create_dataframe({ "ts_la": [ts_la], "ts_utc": [ts_utc] }) # Execute SQL query - DuckDB internally uses local session timezone result = session.sql("SELECT * FROM {df1}", df1=df) # Results are automatically converted back to UTC result.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ ts_la ┆ ts_utc β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-02 09:01:01 UTC ┆ 2025-01-02 01:01:01 UTC β”‚ # Both in UTC #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ # You can use SQL date/time functions result = session.sql( "SELECT DATE_TRUNC('day', ts_utc) as day_start FROM {df1}", df1=df ) result.show() # Output: 2025-01-02 00:00:00 UTC (converted back to UTC) ``` This behavior ensures: - Consistent UTC timestamps regardless of SQL operations - Predictable behavior when working with views and tables created via SQL - No timezone confusion when chaining SQL and DataFrame operations ### Timezone Conversion During Ingestion Fenic automatically handles different timezone inputs: ``` import datetime import zoneinfo # Naive timestamp β†’ interpreted as UTC naive = datetime.datetime(2025, 1, 15, 10, 30, 0) # UTC timestamp β†’ no conversion needed utc = datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC")) # LA timestamp β†’ converted to UTC la = datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("America/Los_Angeles")) df = session.create_dataframe({ "naive_ts": [naive], "utc_ts": [utc], "la_ts": [la] }) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ naive_ts ┆ utc_ts ┆ la_ts β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════β•ͺ═════════════════════════║ #β”‚ 2025-01-15 10:30:00 UTC ┆ 2025-01-15 10:30:00 UTC ┆ 2025-01-15 18:30:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Working with Timestamps ### Parsing Timestamps from Strings Use `to_timestamp()` to parse timestamp strings with custom formats: ``` from fenic.api.functions.dt import to_timestamp from fenic.api.functions import col df = session.create_dataframe({ "timestamp_str": ["01-15-2025 10:30:00", "01-16-2025 14:00:00"] }) df = df.select( to_timestamp(col("timestamp_str"), "MM-dd-yyyy HH:mm:ss").alias("ts") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ ts β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 10:30:00 UTC β”‚ #β”‚ 2025-01-16 14:00:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` With timezone information: ``` df = session.create_dataframe({ "timestamp_str": ["01-15-2025 10:30:00 +08:00", "01-16-2025 14:00:00 +08:00"] }) df = df.select( to_timestamp(col("timestamp_str"), "MM-dd-yyyy HH:mm:ss XXX").alias("ts") ) df.show() # Output (converted to UTC): #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ ts β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 02:30:00 UTC β”‚ #β”‚ 2025-01-16 06:00:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ``` from fenic import TimestampType, DateType # Integer to Timestamp (microseconds from epoch) df = session.create_dataframe({"ts_int": [1735729800000000]}) df = df.select(col("ts_int").cast(TimestampType).alias("ts")) df.show() # Output: 2025-01-01 10:30:00 UTC # Timestamp to Date (truncate time) df = df.select(col("ts").cast(DateType).alias("date")) df.show() # Output: 2025-01-01 # Date to Timestamp (midnight UTC) df = df.select(col("date").cast(TimestampType).alias("ts")) df.show() # Output: 2025-01-01 00:00:00 UTC ``` ### Extracting Date Components ``` from fenic.api.functions.dt import year, month, day, hour, minute, second df = session.create_dataframe({ "ts": [datetime.datetime(2025, 1, 15, 10, 30, 45, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df = df.select( year(col("ts")).alias("year"), month(col("ts")).alias("month"), day(col("ts")).alias("day"), hour(col("ts")).alias("hour"), minute(col("ts")).alias("minute"), second(col("ts")).alias("second") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ year ┆ month ┆ day ┆ hour ┆ minute ┆ second β”‚ #β•žβ•β•β•β•β•β•β•ͺ═══════β•ͺ═════β•ͺ══════β•ͺ════════β•ͺ════════║ #β”‚ 2025 ┆ 1 ┆ 15 ┆ 10 ┆ 30 ┆ 45 β”‚ #β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Working with Timezones ### Timezone Conversion Functions Fenic provides two functions for working with timestamps across timezones: #### to_utc_timestamp Interprets a timestamp (stored in UTC) as wall-clock time in a specific timezone, then converts it to UTC. **Use case**: Your data contains timestamps representing local time but stored with UTC metadata. You want to convert them to actual UTC. ``` from fenic.api.functions.dt import to_timestamp, to_utc_timestamp # Timestamps representing LA wall-clock time df = session.create_dataframe({ "la_wall_clock": ["2025-01-15 10:30:00", "2025-01-16 14:00:00"] }) df = df.select(to_timestamp(col("la_wall_clock"), "yyyy-MM-dd HH:mm:ss").alias("ts")) df = df.select(to_utc_timestamp(col("ts"), "America/Los_Angeles").alias("utc_ts")) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ utc_ts β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 18:30:00 UTC β”‚ # 10:30 AM LA β†’ 6:30 PM UTC #β”‚ 2025-01-16 22:00:00 UTC β”‚ # 2:00 PM LA β†’ 10:00 PM UTC #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` #### from_utc_timestamp Converts a UTC timestamp to wall-clock time in a specific timezone, then represents it as UTC. **Use case**: Display or work with timestamps in a local timezone while keeping them stored as UTC. ``` from fenic.api.functions.dt import to_timestamp, from_utc_timestamp, date_format # UTC timestamps df = session.create_dataframe({ "utc_time": ["2025-01-15 10:30:00", "2025-01-16 14:00:00"] }) df = df.select(to_timestamp(col("utc_time"), "yyyy-MM-dd HH:mm:ss").alias("ts")) df = df.select( from_utc_timestamp(col("ts"), "America/Los_Angeles").alias("la_ts"), date_format( from_utc_timestamp(col("ts"), "America/Los_Angeles"), "yyyy-MM-dd HH:mm:ss" ).alias("la_formatted") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ la_ts ┆ la_formatted β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════║ #β”‚ 2025-01-15 02:30:00 UTC ┆ 2025-01-15 02:30:00 β”‚ # 10:30 AM UTC β†’ 2:30 AM LA #β”‚ 2025-01-16 06:00:00 UTC ┆ 2025-01-16 06:00:00 β”‚ # 2:00 PM UTC β†’ 6:00 AM LA #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Formatting Timestamps and Dates Use `date_format()` to create formatted strings from timestamps or dates with whatever format you want: ``` from fenic.api.functions.dt import date_format, from_utc_timestamp df = session.create_dataframe({ "ts": [datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))], "date": [datetime.date(2025, 1, 15)] }) # Format timestamps and dates in various formats df = df.select( col("ts"), col("date"), date_format(col("ts"), "yyyy-MM-dd HH:mm:ss").alias("ts_simple"), date_format(col("ts"), "MM/dd/yyyy hh:mm a").alias("ts_12hr"), date_format( from_utc_timestamp(col("ts"), "America/Los_Angeles"), "MM-dd-yyyy hh:mm:ss a XXX" ).alias("ts_la_with_tz"), date_format(col("date"), "MMMM dd, yyyy").alias("date_long"), date_format(col("date"), "MM/dd/yy").alias("date_short") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ ts ┆ date ┆ ts_simple ┆ ts_12hr ┆ ts_la_with_tz ┆ date_long ┆ date_short β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ════════════β•ͺ═════════════════════β•ͺ══════════════════β•ͺ══════════════════════════β•ͺ══════════════════β•ͺ════════════║ #β”‚ 2025-01-15 10:30:00 UTC ┆ 2025-01-15 ┆ 2025-01-15 10:30:00 ┆ 01/15/2025 10:30 ┆ 01-15-2025 02:30:00 AM ┆ January 15, 2025 ┆ 01/15/25 β”‚ #β”‚ ┆ ┆ ┆ AM ┆ +00:00 ┆ ┆ β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Date and Time Arithmetic ### Adding/Subtracting Days ``` from fenic.api.functions.dt import date_add, date_sub df = session.create_dataframe({ "date": [datetime.date(2025, 1, 15)] }) df = df.select( date_add(col("date"), 7).alias("next_week"), date_sub(col("date"), 7).alias("last_week") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ next_week ┆ last_week β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•ͺ════════════║ #β”‚ 2025-01-22 ┆ 2025-01-08 β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Adding/Subtracting Time Units ``` from fenic.api.functions.dt import timestamp_add df = session.create_dataframe({ "ts": [datetime.datetime(2025, 1, 15, 10, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df = df.select( timestamp_add(col("ts"), 2, "hour").alias("in_2_hours"), timestamp_add(col("ts"), -30, "minute").alias("30_min_ago") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ in_2_hours ┆ 30_min_ago β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-15 12:30:00 UTC ┆ 2025-01-15 10:00:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Calculating Differences ``` from fenic.api.functions.dt import datediff, timestamp_diff df = session.create_dataframe({ "start": [datetime.date(2025, 1, 1)], "end": [datetime.date(2025, 1, 15)] }) df = df.select(datediff(col("end"), col("start")).alias("days_diff")) df.show() # Output: 14 # For timestamps df = session.create_dataframe({ "start_ts": [datetime.datetime(2025, 1, 1, 10, 0, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))], "end_ts": [datetime.datetime(2025, 1, 15, 14, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df = df.select( timestamp_diff(col("start_ts"), col("end_ts"), "day").alias("days"), timestamp_diff(col("start_ts"), col("end_ts"), "hour").alias("hours") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β” #β”‚ days ┆ hours β”‚ #β•žβ•β•β•β•β•β•β•ͺ═══════║ #β”‚ 14 ┆ 340 β”‚ #β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Common Patterns ### Current Date/Time ``` from fenic.api.functions.dt import current_date, current_timestamp df = session.create_dataframe({"id": [1, 2, 3]}) df = df.select( col("id"), current_date().alias("today"), current_timestamp().alias("now") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ id ┆ today ┆ now β”‚ #β•žβ•β•β•β•β•ͺ════════════β•ͺ═════════════════════════║ #β”‚ 1 ┆ 2025-01-15 ┆ 2025-01-15 10:30:45 UTC β”‚ #β”‚ 2 ┆ 2025-01-15 ┆ 2025-01-15 10:30:45 UTC β”‚ #β”‚ 3 ┆ 2025-01-15 ┆ 2025-01-15 10:30:45 UTC β”‚ #β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Formatting Timestamps with Multiple Timezones When displaying timestamps to users in different regions, format them to their local timezones without showing the timezone suffix: ``` from fenic.api.functions.dt import from_utc_timestamp, date_format df = session.create_dataframe({ "event_time": [ datetime.datetime(2025, 1, 15, 18, 30, 0, tzinfo=zoneinfo.ZoneInfo("UTC")), datetime.datetime(2025, 1, 16, 14, 45, 0, tzinfo=zoneinfo.ZoneInfo("UTC")) ] }) df = df.select( col("event_time"), date_format( from_utc_timestamp(col("event_time"), "America/Los_Angeles"), "yyyy-MM-dd HH:mm:ss" ).alias("time_pst"), date_format( from_utc_timestamp(col("event_time"), "America/New_York"), "yyyy-MM-dd HH:mm:ss" ).alias("time_est"), date_format( from_utc_timestamp(col("event_time"), "Europe/London"), "yyyy-MM-dd HH:mm:ss" ).alias("time_gmt") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ event_time ┆ time_pst ┆ time_est ┆ time_gmt β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════β•ͺ═════════════════════β•ͺ═════════════════════║ #β”‚ 2025-01-15 18:30:00 UTC ┆ 2025-01-15 10:30:00 ┆ 2025-01-15 13:30:00 ┆ 2025-01-15 18:30:00 β”‚ #β”‚ 2025-01-16 14:45:00 UTC ┆ 2025-01-16 06:45:00 ┆ 2025-01-16 09:45:00 ┆ 2025-01-16 14:45:00 β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Truncating Timestamps ``` from fenic.api.functions.dt import date_trunc df = session.create_dataframe({ "ts": [datetime.datetime(2025, 1, 15, 10, 30, 45, tzinfo=zoneinfo.ZoneInfo("UTC"))] }) df = df.select( date_trunc(col("ts"), "year").alias("year_start"), date_trunc(col("ts"), "month").alias("month_start"), date_trunc(col("ts"), "day").alias("day_start"), date_trunc(col("ts"), "hour").alias("hour_start") ) df.show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ year_start ┆ month_start ┆ day_start ┆ hour_start β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════β•ͺ═════════════════════════β•ͺ═════════════════════════║ #β”‚ 2025-01-01 00:00:00 UTC ┆ 2025-01-01 00:00:00 UTC ┆ 2025-01-15 00:00:00 UTC ┆ 2025-01-15 10:00:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ## Limitations - **String casting without timezone**: Currently, casting ISO 8601 timestamp strings without timezone information to `TimestampType` returns `None`. Use `to_timestamp()` instead: ``` # This returns None: df.select(col("ts_str").cast(TimestampType)) # "2025-01-01T10:00:00.000" # Use this instead: df.select(to_timestamp(col("ts_str"))) # Default ISO 8601 parsing ``` ## References - Date/Time Functions: `src/fenic/api/functions/dt.py` - Cast Implementation: `src/fenic/api/column.py:197-228` - Ingestion Coercions: `src/fenic/_backends/local/physical_plan/utils.py:6-54` - Type Definitions: `src/fenic/core/types/datatypes.py` - SQL Tests: `tests/_backends/local/test_sql.py` ## API reference # fenic Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/ Fenic is an opinionated, PySpark-inspired DataFrame framework for building production AI and agentic applications. Classes: - **`AdaptiveTokenEstimationConfig`** – Tunes adaptive output-token reservation for rate limiting. - **`AnthropicLanguageModel`** – Configuration for Anthropic language models. - **`ArrayType`** – A type representing a homogeneous variable-length array (list) of elements. - **`BoundToolParam`** – A bound tool parameter. - **`Catalog`** – Entry point for catalog operations. - **`ClassDefinition`** – Definition of a classification class with optional description. - **`ClassifyExample`** – A single semantic example for classification operations. - **`ClassifyExampleCollection`** – Collection of text-to-category examples for classification operations. - **`CloudConfig`** – Configuration for cloud-based execution. - **`CohereEmbeddingModel`** – Configuration for Cohere embedding models. - **`Column`** – A column expression in a DataFrame. - **`ColumnField`** – Represents a typed column in a DataFrame schema. - **`DataFrame`** – A data collection organized into named columns. - **`DataFrameReader`** – Interface used to load a DataFrame from external storage systems. - **`DataFrameWriter`** – Interface used to write a DataFrame to external storage systems. - **`DataType`** – Base class for all data types. - **`DatasetMetadata`** – Metadata for a dataset (table or view). - **`DocumentPathType`** – Represents a string containing a a document's local (file system) or remote (URL) path. - **`EmbeddingType`** – A type representing a fixed-length embedding vector. - **`GoogleDeveloperEmbeddingModel`** – Configuration for Google Developer embedding models. - **`GoogleDeveloperLanguageModel`** – Configuration for Gemini models accessible through Google Developer AI Studio. - **`GoogleVertexEmbeddingModel`** – Configuration for Google Vertex AI embedding models. - **`GoogleVertexLanguageModel`** – Configuration for Google Vertex AI models. - **`GroupedData`** – Methods for aggregations on a grouped DataFrame. - **`InvalidExampleCollectionError`** – Exception raised when a semantic example collection is invalid. - **`JoinExample`** – A single semantic example for semantic join operations. - **`JoinExampleCollection`** – Collection of comparison examples for semantic join operations. - **`KeyPoints`** – Summary as a concise bulleted list. - **`LLMResponseCacheConfig`** – Configuration for LLM response caching. - **`LMMetrics`** – Tracks language model usage metrics including token counts and costs. - **`Lineage`** – Query interface for tracing data lineage through a query plan. - **`MapExample`** – A single semantic example for semantic mapping operations. - **`MapExampleCollection`** – Collection of input-output examples for semantic map operations. - **`ModelAlias`** – A combination of a model name and a required profile for that model. - **`OpenAIEmbeddingModel`** – Configuration for OpenAI embedding models. - **`OpenAILanguageModel`** – Configuration for OpenAI language models. - **`OpenRouterLanguageModel`** – Configuration for OpenRouter language models. - **`OperatorMetrics`** – Metrics for a single operator in the query execution plan. - **`Paragraph`** – Summary as a cohesive narrative. - **`PredicateExample`** – A single semantic example for semantic predicate operations. - **`PredicateExampleCollection`** – Collection of input-to-boolean examples for predicate operations. - **`QueryMetrics`** – Comprehensive metrics for an executed query. - **`QueryResult`** – Container for query execution results and associated metadata. - **`RMMetrics`** – Tracks embedding model usage metrics including token counts and costs. - **`Schema`** – Represents the schema of a DataFrame. - **`SemanticConfig`** – Configuration for semantic language and embedding models. - **`SemanticExtensions`** – A namespace for semantic dataframe operators. - **`Session`** – The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. - **`SessionConfig`** – Configuration for a user session. - **`StructField`** – A field in a StructType. Fields are nullable. - **`StructType`** – A type representing a struct (record) with named fields. - **`SystemTool`** – A tool implemented as a regular Python function with explicit parameters. - **`SystemToolConfig`** – Configuration for canonical system tools. - **`ToolParam`** – A parameter for a parameterized view tool. - **`TranscriptType`** – Represents a string containing a transcript in a specific format. - **`UserDefinedTool`** – A tool that has been bound to a specific Parameterized View. Functions: - **`approx_count_distinct`** – Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. - **`array_agg`** – Alias for collect_list(). - **`asc`** – Mark this column for ascending sort order with nulls first. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`async_udf`** – A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. - **`avg`** – Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. - **`coalesce`** – Returns the first non-null value from the given columns for each row. - **`col`** – Creates a Column expression referencing a column in the DataFrame. - **`collect_list`** – Aggregate function: collects all values from the specified column into a list. - **`configure_logging`** – Configure logging for the library and root logger in interactive environments. - **`count`** – Aggregate function: returns the count of non-null values in the specified column. - **`count_distinct`** – Aggregate function: returns the number of distinct non-null rows across one or more columns. - **`create_mcp_server`** – Create an MCP server from datasets and tools. - **`desc`** – Mark this column for descending sort order with nulls first. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`empty`** – Creates a Column expression representing an empty value of the given type. - **`first`** – Aggregate function: returns the first non-null value in the specified column. - **`flatten`** – Flattens an array of arrays into a single array (one level deep). - **`greatest`** – Returns the greatest value from the given columns for each row. - **`least`** – Returns the least value from the given columns for each row. - **`lit`** – Creates a Column expression representing a literal value. - **`max`** – Aggregate function: returns the maximum value in the specified column. - **`mean`** – Aggregate function: returns the mean (average) of all values in the specified column. - **`min`** – Aggregate function: returns the minimum value in the specified column. - **`null`** – Creates a Column expression representing a null value of the specified data type. - **`run_mcp_server_asgi`** – Run an MCP server as a Starlette ASGI app. - **`run_mcp_server_async`** – Run an MCP server asynchronously. - **`run_mcp_server_sync`** – Run an MCP server synchronously. - **`stddev`** – Aggregate function: returns the sample standard deviation of the specified column. - **`struct`** – Creates a new struct column from multiple input columns. - **`sum`** – Aggregate function: returns the sum of all values in the specified column. - **`sum_distinct`** – Aggregate function: returns the sum of distinct numeric values in the specified column. - **`tool_param`** – Creates an unresolved literal placeholder column with a declared data type. - **`udf`** – A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. - **`when`** – Evaluates a conditional expression (like if-then). Attributes: - **`BooleanType`** – Represents a boolean value. (True/False) - **`DataCollection`** – Type alias representing provider data collection policies. - **`DataLike`** – Union type representing any supported data format for both input and output operations. - **`DataLikeType`** – String literal type for specifying data output formats. - **`DateType`** – Represents a date value. - **`DoubleType`** – Represents a 64-bit floating-point number. - **`FloatType`** – Represents a 32-bit floating-point number. - **`FuzzySimilarityMethod`** – Type alias representing the supported fuzzy string similarity algorithms. - **`HtmlType`** – Represents a string containing raw HTML markup. - **`IntegerType`** – Represents a signed integer value. - **`JsonType`** – Represents a string containing JSON data. - **`MarkdownType`** – Represents a string containing Markdown-formatted text. - **`ModelQuantization`** – Type alias representing supported quantization formats for provider models. - **`ProviderSort`** – Type alias representing provider sorting strategies used by OpenRouter routing. - **`SemanticSimilarityMetric`** – Type alias representing supported semantic similarity metrics. - **`StringType`** – Represents a UTF-8 encoded string value. - **`StructuredOutputStrategy`** – Type alias representing the strategy to use when a model supports both - **`TimestampType`** – Represents a timestamp value. ## BooleanType ``` BooleanType = _BooleanType() ``` Represents a boolean value. (True/False) ## DataCollection ``` DataCollection = Literal['allow', 'deny'] ``` Type alias representing provider data collection policies. Valid values: - "allow": Permit providers that may retain or train on prompts non-transiently. - "deny": Restrict to providers that do not collect/store user data. ## DataLike ``` DataLike = Union[pl.DataFrame, pd.DataFrame, Dict[str, List[Any]], List[Dict[str, Any]], pa.Table] ``` Union type representing any supported data format for both input and output operations. This type encompasses all possible data structures that can be: 1. Used as input when creating DataFrames 2. Returned as output from query results Supported formats - pl.DataFrame: Native Polars DataFrame with efficient columnar storage - pd.DataFrame: Pandas DataFrame, optionally with PyArrow extension arrays - Dict[str, List[Any]]: Column-oriented dictionary where: - Keys are column names (str) - Values are lists containing all values for that column - List[Dict[str, Any]]: Row-oriented list where: - Each element is a dictionary representing one row - Dictionary keys are column names, values are cell values - pa.Table: Apache Arrow Table with columnar memory layout Usage - Input: Used in create_dataframe() to accept data in various formats - Output: Used in QueryResult.data to return results in requested format The specific type returned depends on the DataLikeType format specified when collecting query results. ## DataLikeType ``` DataLikeType = Literal['polars', 'pandas', 'pydict', 'pylist', 'arrow'] ``` String literal type for specifying data output formats. Valid values - "polars": Native Polars DataFrame format - "pandas": Pandas DataFrame with PyArrow extension arrays - "pydict": Python dictionary with column names as keys, lists as values - "pylist": Python list of dictionaries, each representing one row - "arrow": Apache Arrow Table format Used as input parameter for methods that can return data in multiple formats. ## DateType ``` DateType = _DateType() ``` Represents a date value. ## DoubleType ``` DoubleType = _DoubleType() ``` Represents a 64-bit floating-point number. ## FloatType ``` FloatType = _FloatType() ``` Represents a 32-bit floating-point number. ## FuzzySimilarityMethod ``` FuzzySimilarityMethod = Literal['indel', 'levenshtein', 'damerau_levenshtein', 'jaro_winkler', 'jaro', 'hamming'] ``` Type alias representing the supported fuzzy string similarity algorithms. These algorithms quantify the similarity or difference between two strings using various distance or similarity metrics: - "indel": Computes the Indel (Insertion-Deletion) distance, which counts only insertions and deletions needed to transform one string into another, excluding substitutions. This is equivalent to the Longest Common Subsequence (LCS) problem. Useful when character substitutions should not be considered as valid operations (e.g., DNA sequence alignment where only insertions/deletions occur). - "levenshtein": Computes the Levenshtein distance, which is the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. Suitable for general-purpose fuzzy matching where transpositions do not matter. - "damerau_levenshtein": An extension of Levenshtein distance that also accounts for transpositions of adjacent characters (e.g., "ab" β†’ "ba"). This metric is more accurate for real-world typos and keyboard errors. - "jaro": Measures similarity based on the number and order of common characters between two strings. It is particularly effective for short strings such as names. Returns a normalized score between 0 (no similarity) and 1 (exact match). - "jaro_winkler": A variant of the Jaro distance that gives more weight to common prefixes. Designed to improve accuracy on strings with shared beginnings (e.g., first names, surnames). - "hamming": Measures the number of differing characters between two strings of equal length. Only valid when both strings are the same length. It does not support insertions or deletionsβ€”only substitutions. Choose the method based on the type of expected variation (e.g., typos, transpositions, or structural changes). ## HtmlType ``` HtmlType = _HtmlType() ``` Represents a string containing raw HTML markup. ## IntegerType ``` IntegerType = _IntegerType() ``` Represents a signed integer value. ## JsonType ``` JsonType = _JsonType() ``` Represents a string containing JSON data. ## MarkdownType ``` MarkdownType = _MarkdownType() ``` Represents a string containing Markdown-formatted text. ## ModelQuantization ``` ModelQuantization = Literal['int4', 'int8', 'fp4', 'fp6', 'fp8', 'fp16', 'bf16', 'fp32', 'unknown'] ``` Type alias representing supported quantization formats for provider models. Common values: - "int4", "int8": Integer quantization for smaller, faster models. - "fp4", "fp6", "fp8": Low-precision floating point formats. - "fp16", "bf16": Half-precision formats commonly used on GPUs/TPUs. - "fp32": Full precision floating point. - "unknown": Provider did not specify a quantization. ## ProviderSort ``` ProviderSort = Literal['price', 'throughput', 'latency'] ``` Type alias representing provider sorting strategies used by OpenRouter routing. Valid values: - "price": Prefer providers with the lowest recent price. - "throughput": Prefer providers with the highest recent throughput. - "latency": Prefer providers with the lowest recent latency. ## SemanticSimilarityMetric ``` SemanticSimilarityMetric = Literal['cosine', 'l2', 'dot'] ``` Type alias representing supported semantic similarity metrics. Valid values: - "cosine": Cosine similarity, measures the cosine of the angle between two vectors. - "l2": Euclidean (L2) distance, measures the straight-line distance between two vectors. - "dot": Dot product similarity, the raw inner product of two vectors. These metrics are commonly used for comparing embedding vectors in semantic search and other similarity-based applications. ## StringType ``` StringType = _StringType() ``` Represents a UTF-8 encoded string value. ## StructuredOutputStrategy ``` StructuredOutputStrategy = Literal['prefer_tools', 'prefer_response_format'] ``` Type alias representing the strategy to use when a model supports both tool-calling and response-format-based structured outputs. Valid values: - "prefer_tools": Prefer tool/function calling with a JSON schema. - "prefer_response_format": Prefer response_format structured outputs. ## TimestampType ``` TimestampType = _TimestampType() ``` Represents a timestamp value. ## AdaptiveTokenEstimationConfig Bases: `BaseModel` ``` flowchart TD fenic.AdaptiveTokenEstimationConfig[AdaptiveTokenEstimationConfig] click fenic.AdaptiveTokenEstimationConfig href "" "fenic.AdaptiveTokenEstimationConfig" ``` Tunes adaptive output-token reservation for rate limiting. Output-token reservations are learned from observed usage and clamped to the request's max_completion_tokens ceiling, then corrected after each response (settlement). Enabled by default. Setting `enabled=False` disables adaptive *estimation* β€” reservations fall back to the static worst-case ceiling instead of the learned distribution. Settlement (reconciling the token bucket to actual usage after each response) is **always on** regardless of this flag. It corrects the bucket in both directions β€” refunding the over-reservation (the common case) and debiting further when a request exceeds its reservation β€” and neither direction increases 429 risk: a refund only returns capacity the provider never charged, and a debit only makes the limiter more conservative. ## AnthropicLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.AnthropicLanguageModel[AnthropicLanguageModel] click fenic.AnthropicLanguageModel href "" "fenic.AnthropicLanguageModel" ``` Configuration for Anthropic language models. This class defines the configuration settings for Anthropic language models, including model selection and separate rate limiting parameters for input and output tokens. Attributes: - **`model_name`** (`AnthropicLanguageModelName`) – The name of the Anthropic model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`input_tpm`** (`int`) – Input tokens per minute limit; must be greater than 0. - **`output_tpm`** (`int`) – Output tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring an Anthropic model with separate input/output rate limits: ``` config = AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100 ) ``` Configuring an Anthropic model with profiles: ``` config = SessionConfig( semantic=SemanticConfig( language_models={ "claude": AnthropicLanguageModel( model_name="claude-sonnet-4-6", rpm=100, input_tpm=100, output_tpm=100, profiles={ "thinking_disabled": AnthropicLanguageModel.Profile(), "fast": AnthropicLanguageModel.Profile(thinking_token_budget=1024), "thorough": AnthropicLanguageModel.Profile(thinking_token_budget=4096) }, default_profile="fast" ) }, default_language_model="claude" ) # Using the default "fast" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias="claude") # Using the "thorough" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="claude", profile="thorough")) ``` Classes: - **`Profile`** – Anthropic-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.AnthropicLanguageModel.Profile[Profile] click fenic.AnthropicLanguageModel.Profile href "" "fenic.AnthropicLanguageModel.Profile" ``` Anthropic-specific profile configurations. This class defines profile configurations for Anthropic models, allowing different thinking and effort settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – Provide a default thinking budget in tokens. If not provided, thinking will be disabled for the profile. The minimum token budget supported by Anthropic is 1024 tokens. For Claude models that use adaptive thinking, use `effort` instead. - **`effort`** (`Optional[AnthropicReasoningEffortType]`) – Provider-native Anthropic effort level. Supported values vary by model: low, medium, high, xhigh, and max. On adaptive-thinking models the thinking budget shares the request's output token window rather than being reserved on top of it, so very high effort levels can consume part of the visible completion budget. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note If `thinking_token_budget` or adaptive `effort` enables thinking, `temperature` cannot be customized -- any changes to `temperature` will be ignored. Effort-only profiles on non-adaptive models configure Anthropic `output_config` without enabling thinking, so custom `temperature` remains available when the model supports it. Example Configuring a profile with a thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=2048) ``` Configuring a profile with a large thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=8192) ``` Configuring a profile with effort: ``` profile = AnthropicLanguageModel.Profile(effort="xhigh") ``` ## ArrayType Bases: `DataType` ``` flowchart TD fenic.ArrayType[ArrayType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.ArrayType click fenic.ArrayType href "" "fenic.ArrayType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a homogeneous variable-length array (list) of elements. Attributes: - **`element_type`** (`DataType`) – The data type of each element in the array. Create an array of strings ``` ArrayType(StringType) ArrayType(element_type=StringType) ``` ## BoundToolParam A bound tool parameter. A bound tool parameter is a parameter that has been bound to a specific, typed, `tool_param` usage within a Dataframe. ## Catalog ``` Catalog(catalog: BaseCatalog) ``` Entry point for catalog operations. Provides methods to manage catalogs, databases, and tables, as well as read-only access to system tables such as `fenic_system.query_metrics`. ##### Catalog and Database Management Example: ```python # Create a catalog session.catalog.create_catalog("my_catalog") # β†’ True ``` # Set active catalog session.catalog.set_current_catalog("my_catalog") # Create a database session.catalog.create_database("my_database") # β†’ True # Set active database session.catalog.set_current_database("my_database") # Create a table session.catalog.create_table( "my_table", Schema([ColumnField("id", IntegerType)]) ) # β†’ True ``` ``` ##### Metrics Table (Local Sessions Only) Query metrics are recorded for each session and stored locally in `fenic_system.query_metrics`. Metrics can be loaded into a DataFrame for analysis. Example ``` # Load all metrics for the current application metrics_df = session.table("fenic_system.query_metrics") # Show the 10 most recent queries in the application recent_queries = session.sql(""" SELECT * FROM {df} ORDER BY CAST(end_ts AS TIMESTAMP) DESC LIMIT 10 """, df=metrics_df) recent_queries.show() # Find query metrics for a specific session with non-zero LM costs specific_session_queries = session.sql(""" SELECT * FROM {df} WHERE session_id = '9e7e256f-fad9-4cd9-844e-399d795aaea0' AND total_lm_cost > 0 ORDER BY CAST(end_ts AS TIMESTAMP) ASC """, df=metrics_df) specific_session_queries.show() # Aggregate total LM costs and requests between a specific time window metrics_window = session.sql(""" SELECT CAST(SUM(total_lm_cost) AS DOUBLE) AS total_lm_cost_in_window, CAST(SUM(total_lm_requests) AS DOUBLE) AS total_lm_requests_in_window FROM {df} WHERE CAST(end_ts AS TIMESTAMP) BETWEEN CAST('2025-08-29 10:00:00' AS TIMESTAMP) AND CAST('2025-08-29 12:00:00' AS TIMESTAMP) """, df=metrics_df) metrics_window.show() ``` Initialize a Catalog instance. Parameters: - **`catalog`** (`BaseCatalog`) – The underlying catalog implementation. Methods: - **`create_catalog`** – Creates a new catalog. - **`create_database`** – Creates a new database. - **`create_table`** – Creates a new table. - **`create_tool`** – Creates a new tool in the current catalog. - **`describe_table`** – Returns the schema of the specified table. - **`describe_tool`** – Returns the tool with the specified name from the current catalog. - **`describe_view`** – Returns the schema and description of the specified view. - **`does_catalog_exist`** – Checks if a catalog with the specified name exists. - **`does_database_exist`** – Checks if a database with the specified name exists. - **`does_table_exist`** – Checks if a table with the specified name exists. - **`does_view_exist`** – Checks if a view with the specified name exists. - **`drop_catalog`** – Drops a catalog. - **`drop_database`** – Drops a database. - **`drop_table`** – Drops the specified table. - **`drop_tool`** – Drops the specified tool from the current catalog. - **`drop_view`** – Drops the specified view. - **`get_current_catalog`** – Returns the name of the current catalog. - **`get_current_database`** – Returns the name of the current database in the current catalog. - **`list_catalogs`** – Returns a list of available catalogs. - **`list_databases`** – Returns a list of databases in the current catalog. - **`list_tables`** – Returns a list of tables stored in the current database. - **`list_tools`** – Lists the tools available in the current catalog. - **`list_views`** – Returns a list of views stored in the current database. - **`set_current_catalog`** – Sets the current catalog. - **`set_current_database`** – Sets the current database. - **`set_table_description`** – Set or unset the description for a table. - **`set_view_description`** – Set the description for a view. Source code in `src/fenic/api/catalog.py` ``` def __init__(self, catalog: BaseCatalog): """Initialize a Catalog instance. Args: catalog: The underlying catalog implementation. """ self.catalog = catalog ``` ### create_catalog ``` create_catalog(catalog_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: - `CatalogAlreadyExistsError` – If the catalog already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the catalog was created successfully, False if the catalog - `bool` – already exists and ignore_if_exists is True. Create a new catalog ``` # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Create an existing catalog with ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Create an existing catalog without ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_catalog(self, catalog_name: str, ignore_if_exists: bool = True) -> bool: """Creates a new catalog. Args: catalog_name (str): Name of the catalog to create. ignore_if_exists (bool): If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: CatalogAlreadyExistsError: If the catalog already exists and ignore_if_exists is False. Returns: bool: True if the catalog was created successfully, False if the catalog already exists and ignore_if_exists is True. Example: Create a new catalog ```python # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Example: Create an existing catalog with ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Example: Create an existing catalog without ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` """ return self.catalog.create_catalog(catalog_name, ignore_if_exists) ``` ### create_database ``` create_database(database_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: - `DatabaseAlreadyExistsError` – If the database already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the database was created successfully, False if the database - `bool` – already exists and ignore_if_exists is True. Create a new database ``` # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Create an existing database with ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Create an existing database without ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_database( self, database_name: str, ignore_if_exists: bool = True ) -> bool: """Creates a new database. Args: database_name (str): Fully qualified or relative database name to create. ignore_if_exists (bool): If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: DatabaseAlreadyExistsError: If the database already exists and ignore_if_exists is False. Returns: bool: True if the database was created successfully, False if the database already exists and ignore_if_exists is True. Example: Create a new database ```python # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Example: Create an existing database with ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Example: Create an existing database without ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` """ return self.catalog.create_database(database_name, ignore_if_exists) ``` ### create_table ``` create_table(table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None) -> bool ``` Creates a new table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to create. - **`schema`** (`Schema`) – Schema of the table to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. - **`description`** (`Optional[str]`, default: `None` ) – Description of the table to create. Defaults to None. Returns: - **`bool`** ( `bool` ) – True if the table was created successfully, False if the table - `bool` – already exists and ignore_if_exists is True. Raises: - `TableAlreadyExistsError` – If the table already exists and ignore_if_exists is False Create a new table ``` # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Create an existing table with ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Create an existing table without ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_table( self, table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None ) -> bool: """Creates a new table. Args: table_name (str): Fully qualified or relative table name to create. schema (Schema): Schema of the table to create. ignore_if_exists (bool): If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. description (Optional[str]): Description of the table to create. Defaults to None. Returns: bool: True if the table was created successfully, False if the table already exists and ignore_if_exists is True. Raises: TableAlreadyExistsError: If the table already exists and ignore_if_exists is False Example: Create a new table ```python # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Example: Create an existing table with ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Example: Create an existing table without ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` """ return self.catalog.create_table(table_name, schema, ignore_if_exists, description) ``` ### create_tool ``` create_tool(tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True) -> bool ``` Creates a new tool in the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool. - **`tool_description`** (`str`) – The description of the tool. - **`tool_query`** (`DataFrame`) – The query to execute when the tool is called. - **`tool_params`** (`Sequence[ToolParam]`) – The parameters of the tool. - **`result_limit`** (`int`, default: `50` ) – The maximum number of rows to return from the tool. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was created successfully, False otherwise. Raises: - `ToolAlreadyExistsError` – If the tool already exists. Examples: ``` # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_tool( self, tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True ) -> bool: """Creates a new tool in the current catalog. Args: tool_name (str): The name of the tool. tool_description (str): The description of the tool. tool_query (DataFrame): The query to execute when the tool is called. tool_params (Sequence[ToolParam]): The parameters of the tool. result_limit (int): The maximum number of rows to return from the tool. ignore_if_exists (bool): If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: bool: True if the tool was created successfully, False otherwise. Raises: ToolAlreadyExistsError: If the tool already exists. Examples: ```python # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` """ return self.catalog.create_tool( tool_name, tool_description, tool_params, tool_query._logical_plan, result_limit, ignore_if_exists, ) ``` ### describe_table ``` describe_table(table_name: str) -> DatasetMetadata ``` Returns the schema of the specified table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: - `TableNotFoundError` – If the table doesn't exist. Describe a table's schema ``` # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_table(self, table_name: str) -> DatasetMetadata: """Returns the schema of the specified table. Args: table_name (str): Fully qualified or relative table name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: TableNotFoundError: If the table doesn't exist. Example: Describe a table's schema ```python # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` """ return self.catalog.describe_table(table_name) ``` ### describe_tool ``` describe_tool(tool_name: str) -> UserDefinedTool ``` Returns the tool with the specified name from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to get. Raises: - `ToolNotFoundError` – If the tool doesn't exist. Returns: - **`Tool`** ( `UserDefinedTool` ) – The tool with the specified name. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_tool(self, tool_name: str) -> UserDefinedTool: """Returns the tool with the specified name from the current catalog. Args: tool_name (str): The name of the tool to get. Raises: ToolNotFoundError: If the tool doesn't exist. Returns: Tool: The tool with the specified name. """ return self.catalog.describe_tool(tool_name) ``` ### describe_view ``` describe_view(view_name: str) -> DatasetMetadata ``` Returns the schema and description of the specified view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: - `TableNotFoundError` – If the view doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_view(self, view_name: str) -> DatasetMetadata: """Returns the schema and description of the specified view. Args: view_name (str): Fully qualified or relative view name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: TableNotFoundError: If the view doesn't exist. """ return self.catalog.describe_view(view_name) ``` ### does_catalog_exist ``` does_catalog_exist(catalog_name: str) -> bool ``` Checks if a catalog with the specified name exists. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to check. Returns: - **`bool`** ( `bool` ) – True if the catalog exists, False otherwise. Check if a catalog exists ``` # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_catalog_exist(self, catalog_name: str) -> bool: """Checks if a catalog with the specified name exists. Args: catalog_name (str): Name of the catalog to check. Returns: bool: True if the catalog exists, False otherwise. Example: Check if a catalog exists ```python # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` """ return self.catalog.does_catalog_exist(catalog_name) ``` ### does_database_exist ``` does_database_exist(database_name: str) -> bool ``` Checks if a database with the specified name exists. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to check. Returns: - **`bool`** ( `bool` ) – True if the database exists, False otherwise. Check if a database exists ``` # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_database_exist(self, database_name: str) -> bool: """Checks if a database with the specified name exists. Args: database_name (str): Fully qualified or relative database name to check. Returns: bool: True if the database exists, False otherwise. Example: Check if a database exists ```python # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` """ return self.catalog.does_database_exist(database_name) ``` ### does_table_exist ``` does_table_exist(table_name: str) -> bool ``` Checks if a table with the specified name exists. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to check. Returns: - **`bool`** ( `bool` ) – True if the table exists, False otherwise. Check if a table exists ``` # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_table_exist(self, table_name: str) -> bool: """Checks if a table with the specified name exists. Args: table_name (str): Fully qualified or relative table name to check. Returns: bool: True if the table exists, False otherwise. Example: Check if a table exists ```python # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` """ return self.catalog.does_table_exist(table_name) ``` ### does_view_exist ``` does_view_exist(view_name: str) -> bool ``` Checks if a view with the specified name exists. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to check. Returns: - **`bool`** ( `bool` ) – True if the view exists, False otherwise. Example > > > session.catalog.does_view_exist('my_view') > > > True. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_view_exist(self, view_name: str) -> bool: """Checks if a view with the specified name exists. Args: view_name (str): Fully qualified or relative view name to check. Returns: bool: True if the view exists, False otherwise. Example: >>> session.catalog.does_view_exist('my_view') True. """ return self.catalog.does_view_exist(view_name) ``` ### drop_catalog ``` drop_catalog(catalog_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops a catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: - `CatalogNotFoundError` – If the catalog does not exist and ignore_if_not_exists is False Returns: - **`bool`** ( `bool` ) – True if the catalog was dropped successfully, False if the catalog - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent catalog ``` # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Drop a non-existent catalog without ignore_if_not_exists ``` # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_catalog( self, catalog_name: str, ignore_if_not_exists: bool = True ) -> bool: """Drops a catalog. Args: catalog_name (str): Name of the catalog to drop. ignore_if_not_exists (bool): If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: CatalogNotFoundError: If the catalog does not exist and ignore_if_not_exists is False Returns: bool: True if the catalog was dropped successfully, False if the catalog didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent catalog ```python # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Example: Drop a non-existent catalog without ignore_if_not_exists ```python # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` """ return self.catalog.drop_catalog(catalog_name, ignore_if_not_exists) ``` ### drop_database ``` drop_database(database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True) -> bool ``` Drops a database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to drop. - **`cascade`** (`bool`, default: `False` ) – If True, drop all tables in the database. Defaults to False. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: - `DatabaseNotFoundError` – If the database does not exist and ignore_if_not_exists is False - `CatalogError` – If the current database is being dropped, if the database is not empty and cascade is False Returns: - **`bool`** ( `bool` ) – True if the database was dropped successfully, False if the database - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent database ``` # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Drop a non-existent database without ignore_if_not_exists ``` # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_database( self, database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True, ) -> bool: """Drops a database. Args: database_name (str): Fully qualified or relative database name to drop. cascade (bool): If True, drop all tables in the database. Defaults to False. ignore_if_not_exists (bool): If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: DatabaseNotFoundError: If the database does not exist and ignore_if_not_exists is False CatalogError: If the current database is being dropped, if the database is not empty and cascade is False Returns: bool: True if the database was dropped successfully, False if the database didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent database ```python # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Example: Drop a non-existent database without ignore_if_not_exists ```python # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` """ return self.catalog.drop_database(database_name, cascade, ignore_if_not_exists) ``` ### drop_table ``` drop_table(table_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified table. By default this method will return False if the table doesn't exist. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the table was dropped successfully, False if the table - `bool` – didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the table doesn't exist and ignore_if_not_exists is False Drop an existing table ``` # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Drop a non-existent table with ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Drop a non-existent table without ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_table(self, table_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified table. By default this method will return False if the table doesn't exist. Args: table_name (str): Fully qualified or relative table name to drop. ignore_if_not_exists (bool): If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: bool: True if the table was dropped successfully, False if the table didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the table doesn't exist and ignore_if_not_exists is False Example: Drop an existing table ```python # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Example: Drop a non-existent table with ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Example: Drop a non-existent table without ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` """ return self.catalog.drop_table(table_name, ignore_if_not_exists) ``` ### drop_tool ``` drop_tool(tool_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified tool from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: - `ToolNotFoundError` – If the tool doesn't exist and ignore_if_not_exists is False Example > > > session.catalog.drop_tool('my_tool') > > > True > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) > > > False > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) #### Raises ToolNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_tool(self, tool_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified tool from the current catalog. Args: tool_name (str): The name of the tool to drop. ignore_if_not_exists (bool): If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: bool: True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: ToolNotFoundError: If the tool doesn't exist and ignore_if_not_exists is False Example: >>> session.catalog.drop_tool('my_tool') True >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) False >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) # Raises ToolNotFoundError. """ return self.catalog.drop_tool(tool_name, ignore_if_not_exists) ``` ### drop_view ``` drop_view(view_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified view. By default this method will return False if the view doesn't exist. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_view(self, view_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified view. By default this method will return False if the view doesn't exist. Args: view_name (str): Fully qualified or relative view name to drop. ignore_if_not_exists (bool, optional): If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: bool: True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. """ return self.catalog.drop_view(view_name, ignore_if_not_exists) ``` ### get_current_catalog ``` get_current_catalog() -> str ``` Returns the name of the current catalog. Returns: - **`str`** ( `str` ) – The name of the current catalog. Get current catalog name ``` # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_catalog(self) -> str: """Returns the name of the current catalog. Returns: str: The name of the current catalog. Example: Get current catalog name ```python # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` """ return self.catalog.get_current_catalog() ``` ### get_current_database ``` get_current_database() -> str ``` Returns the name of the current database in the current catalog. Returns: - **`str`** ( `str` ) – The name of the current database. Get current database name ``` # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_database(self) -> str: """Returns the name of the current database in the current catalog. Returns: str: The name of the current database. Example: Get current database name ```python # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` """ return self.catalog.get_current_database() ``` ### list_catalogs ``` list_catalogs() -> List[str] ``` Returns a list of available catalogs. Returns: - `List[str]` – List[str]: A list of catalog names available in the system. - `List[str]` – Returns an empty list if no catalogs are found. List all catalogs ``` # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_catalogs(self) -> List[str]: """Returns a list of available catalogs. Returns: List[str]: A list of catalog names available in the system. Returns an empty list if no catalogs are found. Example: List all catalogs ```python # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` """ return self.catalog.list_catalogs() ``` ### list_databases ``` list_databases() -> List[str] ``` Returns a list of databases in the current catalog. Returns: - `List[str]` – List[str]: A list of database names in the current catalog. - `List[str]` – Returns an empty list if no databases are found. List all databases ``` # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_databases(self) -> List[str]: """Returns a list of databases in the current catalog. Returns: List[str]: A list of database names in the current catalog. Returns an empty list if no databases are found. Example: List all databases ```python # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` """ return self.catalog.list_databases() ``` ### list_tables ``` list_tables() -> List[str] ``` Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: - `List[str]` – List[str]: A list of table names stored in the database. - `List[str]` – Returns an empty list if no tables are found. List all tables ``` # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_tables(self) -> List[str]: """Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: List[str]: A list of table names stored in the database. Returns an empty list if no tables are found. Example: List all tables ```python # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` """ return self.catalog.list_tables() ``` ### list_tools ``` list_tools() -> List[UserDefinedTool] ``` Lists the tools available in the current catalog. Source code in `src/fenic/api/catalog.py` ``` def list_tools(self) -> List[UserDefinedTool]: """Lists the tools available in the current catalog.""" return self.catalog.list_tools() ``` ### list_views ``` list_views() -> List[str] ``` Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: - `List[str]` – List[str]: A list of view names stored in the database. - `List[str]` – Returns an empty list if no views are found. Example > > > session.catalog.list_views() > > > ['view1', 'view2', 'view3']. Source code in `src/fenic/api/catalog.py` ``` def list_views(self) -> List[str]: """Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: List[str]: A list of view names stored in the database. Returns an empty list if no views are found. Example: >>> session.catalog.list_views() ['view1', 'view2', 'view3']. """ return self.catalog.list_views() ``` ### set_current_catalog ``` set_current_catalog(catalog_name: str) -> None ``` Sets the current catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to set as current. Raises: - `ValueError` – If the specified catalog doesn't exist. Set current catalog ``` # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_catalog(self, catalog_name: str) -> None: """Sets the current catalog. Args: catalog_name (str): Name of the catalog to set as current. Raises: ValueError: If the specified catalog doesn't exist. Example: Set current catalog ```python # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` """ self.catalog.set_current_catalog(catalog_name) ``` ### set_current_database ``` set_current_database(database_name: str) -> None ``` Sets the current database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to set as current. Raises: - `DatabaseNotFoundError` – If the specified database doesn't exist. Set current database ``` # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_database(self, database_name: str) -> None: """Sets the current database. Args: database_name (str): Fully qualified or relative database name to set as current. Raises: DatabaseNotFoundError: If the specified database doesn't exist. Example: Set current database ```python # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` """ self.catalog.set_current_database(database_name) ``` ### set_table_description ``` set_table_description(table_name: str, description: Optional[str] = None) -> None ``` Set or unset the description for a table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to set the description for. - **`description`** (`Optional[str]`, default: `None` ) – The description to set for the table. Raises: - `TableNotFoundError` – If the table doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_table_description(self, table_name: str, description: Optional[str] = None) -> None: """Set or unset the description for a table. Args: table_name: Fully qualified or relative table name to set the description for. description: The description to set for the table. Raises: TableNotFoundError: If the table doesn't exist. """ self.catalog.set_table_description(table_name, description) ``` ### set_view_description ``` set_view_description(view_name: str, description: Optional[str] = None) -> None ``` Set the description for a view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to set the description for. - **`description`** (`str`, default: `None` ) – The description to set for the view. Raises: - `TableNotFoundError` – If the view doesn't exist. - `ValidationError` – If the description is empty. Set a description for a view ```python #### Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_view_description(self, view_name: str, description: Optional[str] = None) -> None: """Set the description for a view. Args: view_name (str): Fully qualified or relative view name to set the description for. description (str): The description to set for the view. Raises: TableNotFoundError: If the view doesn't exist. ValidationError: If the description is empty. Example: Set a description for a view ```python # Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') """ self.catalog.set_view_description(view_name, description) ``` ## ClassDefinition Bases: `BaseModel` ``` flowchart TD fenic.ClassDefinition[ClassDefinition] click fenic.ClassDefinition href "" "fenic.ClassDefinition" ``` Definition of a classification class with optional description. Used to define the available classes for semantic classification operations. The description helps the LLM understand what each class represents. ## ClassifyExample Bases: `BaseModel` ``` flowchart TD fenic.ClassifyExample[ClassifyExample] click fenic.ClassifyExample href "" "fenic.ClassifyExample" ``` A single semantic example for classification operations. Classify examples demonstrate the classification of an input string into a specific category string, used in a semantic.classify operation. ## ClassifyExampleCollection ``` ClassifyExampleCollection(examples: List[ExampleType] = None) ``` Bases: `BaseExampleCollection[ClassifyExample]` ``` flowchart TD fenic.ClassifyExampleCollection[ClassifyExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.ClassifyExampleCollection click fenic.ClassifyExampleCollection href "" "fenic.ClassifyExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of text-to-category examples for classification operations. Stores examples showing which category each input text should be assigned to. Each example contains an input string and its corresponding category label. Methods: - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[ExampleType] = None): """Initialize a collection of semantic examples. Args: examples: Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note: The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. """ self.examples: List[ExampleType] = [] if examples: for example in examples: self.create_example(example) ``` ### from_polars ``` from_polars(df: DataFrame) -> ClassifyExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> ClassifyExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column.""" collection = cls() if EXAMPLE_INPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_INPUT_KEY}' column" ) if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_INPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_INPUT_KEY}' column" ) if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) example = ClassifyExample( input=row[EXAMPLE_INPUT_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## CloudConfig Bases: `BaseModel` ``` flowchart TD fenic.CloudConfig[CloudConfig] click fenic.CloudConfig href "" "fenic.CloudConfig" ``` Configuration for cloud-based execution. This class defines settings for running operations in a cloud environment, allowing for scalable and distributed processing of language model operations. Attributes: - **`size`** (`Optional[CloudExecutorSize]`) – Size of the cloud executor instance. If None, the default size will be used. Example Configuring cloud execution with a specific size: ``` config = CloudConfig(size=CloudExecutorSize.MEDIUM) ``` Using default cloud configuration: ``` config = CloudConfig() ``` ## CohereEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.CohereEmbeddingModel[CohereEmbeddingModel] click fenic.CohereEmbeddingModel href "" "fenic.CohereEmbeddingModel" ``` Configuration for Cohere embedding models. This class defines the configuration settings for Cohere embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`CohereEmbeddingModelName`) – The name of the Cohere model to use. - **`rpm`** (`int`) – Requests per minute limit for the model. - **`tpm`** (`int`) – Tokens per minute limit for the model. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional dictionary of profile configurations. - **`default_profile`** (`Optional[str]`) – Default profile name to use if none specified. Example Configuring a Cohere embedding model with profiles: ``` cohere_config = CohereEmbeddingModel( model_name="embed-v4.0", rpm=100, tpm=50_000, profiles={ "high_dim": CohereEmbeddingModel.Profile( embedding_dimensionality=1536, embedding_task_type="search_document" ), "classification": CohereEmbeddingModel.Profile( embedding_dimensionality=1024, embedding_task_type="classification" ), }, default_profile="high_dim", ) ``` Classes: - **`Profile`** – Profile configurations for Cohere embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.CohereEmbeddingModel.Profile[Profile] click fenic.CohereEmbeddingModel.Profile href "" "fenic.CohereEmbeddingModel.Profile" ``` Profile configurations for Cohere embedding models. This class defines profile configurations for Cohere embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`input_type`** (`CohereEmbeddingTaskType`) – The type of input text (search_query, search_document, classification, clustering) Example Configuring a profile with custom dimensionality: ``` profile = CohereEmbeddingModel.Profile(output_dimensionality=1536) ``` Configuring a profile with default settings: ``` profile = CohereEmbeddingModel.Profile() ``` ## Column A column expression in a DataFrame. This class represents a column expression that can be used in DataFrame operations. It provides methods for accessing, transforming, and combining column data. Create a column reference ``` # Reference a column by name using col() function col("column_name") ``` Use column in operations ``` # Perform arithmetic operations df.select(col("price") * col("quantity")) ``` Chain column operations ``` # Chain multiple operations df.select(col("name").upper().contains("John")) ``` Methods: - **`alias`** – Create an alias for this column. - **`asc`** – Mark this column for ascending sort order. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`cast`** – Cast the column to a new data type. - **`contains`** – Check if the column contains a substring. - **`contains_any`** – Check if the column contains any of the specified substrings. - **`desc`** – Mark this column for descending sort order. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`ends_with`** – Check if the column ends with a substring. - **`get_item`** – Access an item in a struct or array column. - **`ilike`** – Check if the column matches a SQL LIKE pattern (case-insensitive). - **`is_in`** – Check if the column is in a list of values or a column expression. - **`is_not_null`** – Check if the column contains non-NULL values. - **`is_null`** – Check if the column contains NULL values. - **`like`** – Check if the column matches a SQL LIKE pattern. - **`otherwise`** – Returns a value when no prior conditions are True. - **`rlike`** – Check if the column matches a regular expression pattern. - **`starts_with`** – Check if the column starts with a substring. - **`when`** – Evaluates a condition for each row and returns a value when true. ### alias ``` alias(name: str) -> Column ``` Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Parameters: - **`name`** (`str`) – The alias name to assign Returns: - **`Column`** ( `Column` ) – Column with the specified alias Rename a column ``` # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Name a complex expression ``` # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` Source code in `src/fenic/api/column.py` ``` def alias(self, name: str) -> Column: """Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Args: name (str): The alias name to assign Returns: Column: Column with the specified alias Example: Rename a column ```python # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Example: Name a complex expression ```python # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` """ return Column._from_logical_expr(AliasExpr(self._logical_expr, name)) ``` ### asc ``` asc() -> Column ``` Mark this column for ascending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls first. Sort by age in ascending order ``` # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc(self) -> Column: """Mark this column for ascending sort order. Returns: Column: A sort expression with ascending order and nulls first. Example: Sort by age in ascending order ```python # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` """ return Column._from_logical_expr(SortExpr(self._logical_expr, ascending=True)) ``` ### asc_nulls_first ``` asc_nulls_first() -> Column ``` Alias for asc(). Returns: - **`Column`** ( `Column` ) – A Column expression that provides a column and sort order to the sort function Source code in `src/fenic/api/column.py` ``` def asc_nulls_first(self) -> Column: """Alias for asc(). Returns: Column: A Column expression that provides a column and sort order to the sort function """ return self.asc() ``` ### asc_nulls_last ``` asc_nulls_last() -> Column ``` Mark this column for ascending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls last. Sort by age in ascending order with nulls last ``` # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc_nulls_last(self) -> Column: """Mark this column for ascending sort order with nulls last. Returns: Column: A sort expression with ascending order and nulls last. Example: Sort by age in ascending order with nulls last ```python # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=True, nulls_last=True) ) ``` ### cast ``` cast(data_type: DataType) -> Column ``` Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Parameters: - **`data_type`** (`DataType`) – The target DataType to cast the column to Returns: - **`Column`** ( `Column` ) – A Column representing the casted expression Cast integer to string ``` # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Cast array of integers to array of strings ``` # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Cast struct fields to different types ``` # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: - `TypeError` – If the requested cast operation is not supported Source code in `src/fenic/api/column.py` ``` def cast(self, data_type: DataType) -> Column: """Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Args: data_type (DataType): The target DataType to cast the column to Returns: Column: A Column representing the casted expression Example: Cast integer to string ```python # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Example: Cast array of integers to array of strings ```python # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Example: Cast struct fields to different types ```python # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: TypeError: If the requested cast operation is not supported """ return Column._from_logical_expr(CastExpr(self._logical_expr, data_type)) ``` ### contains ``` contains(other: Union[str, Column]) -> Column ``` Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to search for (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains the substring Find rows where name contains "john" ``` # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Find rows where text contains a dynamic pattern ``` # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` Source code in `src/fenic/api/column.py` ``` def contains(self, other: Union[str, Column]) -> Column: """Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to search for (can be a string or column expression) Returns: Column: A boolean column indicating whether each value contains the substring Example: Find rows where name contains "john" ```python # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Example: Find rows where text contains a dynamic pattern ```python # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ContainsExpr(self._logical_expr, other_expr)) ``` ### contains_any ``` contains_any(others: List[str], case_insensitive: bool = True) -> Column ``` Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Parameters: - **`others`** (`List[str]`) – List of substrings to search for - **`case_insensitive`** (`bool`, default: `True` ) – Whether to perform case-insensitive matching (default: True) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains any substring Find rows where name contains "john" or "jane" (case-insensitive) ``` # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Case-sensitive matching ``` # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` Source code in `src/fenic/api/column.py` ``` def contains_any(self, others: List[str], case_insensitive: bool = True) -> Column: """Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Args: others (List[str]): List of substrings to search for case_insensitive (bool): Whether to perform case-insensitive matching (default: True) Returns: Column: A boolean column indicating whether each value contains any substring Example: Find rows where name contains "john" or "jane" (case-insensitive) ```python # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Example: Case-sensitive matching ```python # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` """ return Column._from_logical_expr( ContainsAnyExpr(self._logical_expr, others, case_insensitive) ) ``` ### desc ``` desc() -> Column ``` Mark this column for descending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order. Sort by age in descending order ``` # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc(self) -> Column: """Mark this column for descending sort order. Returns: Column: A sort expression with descending order. Example: Sort by age in descending order ```python # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False) ) ``` ### desc_nulls_first ``` desc_nulls_first() -> Column ``` Alias for desc(). Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls first. Sort by age in descending order with nulls first ``` df.sort(col("age").desc_nulls_first()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_first(self) -> Column: """Alias for desc(). Returns: Column: A sort expression with descending order and nulls first. Example: Sort by age in descending order with nulls first ```python df.sort(col("age").desc_nulls_first()).show() ``` """ return self.desc() ``` ### desc_nulls_last ``` desc_nulls_last() -> Column ``` Mark this column for descending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls last. Sort by age in descending order with nulls last ``` # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_last(self) -> Column: """Mark this column for descending sort order with nulls last. Returns: Column: A sort expression with descending order and nulls last. Example: Sort by age in descending order with nulls last ```python # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False, nulls_last=True) ) ``` ### ends_with ``` ends_with(other: Union[str, Column]) -> Column ``` Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the end (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value ends with the substring Find rows where email ends with "@gmail.com" ``` df.filter(col("email").ends_with("@gmail.com")) ``` Find rows where text ends with a dynamic pattern ``` df.filter(col("text").ends_with(col("suffix"))) ``` Raises: - `ValueError` – If the substring ends with a regular expression anchor ($) Source code in `src/fenic/api/column.py` ``` def ends_with(self, other: Union[str, Column]) -> Column: """Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the end (can be a string or column expression) Returns: Column: A boolean column indicating whether each value ends with the substring Example: Find rows where email ends with "@gmail.com" ```python df.filter(col("email").ends_with("@gmail.com")) ``` Example: Find rows where text ends with a dynamic pattern ```python df.filter(col("text").ends_with(col("suffix"))) ``` Raises: ValueError: If the substring ends with a regular expression anchor ($) """ if isinstance(other, str): if other.endswith("$"): raise ValidationError("substr should not end with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(EndsWithExpr(self._logical_expr, other_expr)) ``` ### get_item ``` get_item(key: Union[str, int, Column]) -> Column ``` Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Parameters: - **`key`** (`Union[str, int]`) – The index (for arrays) or field name (for structs) to access Returns: - **`Column`** ( `Column` ) – A Column representing the accessed item Access an array element ``` # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Access a struct field ``` # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` Source code in `src/fenic/api/column.py` ``` def get_item(self, key: Union[str, int, Column]) -> Column: """Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Args: key (Union[str, int]): The index (for arrays) or field name (for structs) to access Returns: Column: A Column representing the accessed item Example: Access an array element ```python # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Example: Access a struct field ```python # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` """ if isinstance(key, Column): return Column._from_logical_expr(IndexExpr(self._logical_expr, key._logical_expr)) elif isinstance(key, str): return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, StringType))) else: return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, IntegerType))) ``` ### ilike ``` ilike(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "j" and ends with "n" (case-insensitive) ``` # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Find rows where code matches pattern (case-insensitive) ``` # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` Source code in `src/fenic/api/column.py` ``` def ilike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "j" and ends with "n" (case-insensitive) ```python # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Example: Find rows where code matches pattern (case-insensitive) ```python # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ILikeExpr(self._logical_expr, other_expr)) ``` ### is_in ``` is_in(other: Union[List[Any], ColumnOrName]) -> Column ``` Check if the column is in a list of values or a column expression. Parameters: - **`other`** (`Union[List[Any], ColumnOrName]`) – A list of values or a Column expression Returns: - **`Column`** ( `Column` ) – A Column expression representing whether each element of Column is in the list Check if name is in a list of values ``` # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Check if value is in another column ``` # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` Source code in `src/fenic/api/column.py` ``` def is_in(self, other: Union[List[Any], ColumnOrName]) -> Column: """Check if the column is in a list of values or a column expression. Args: other (Union[List[Any], ColumnOrName]): A list of values or a Column expression Returns: Column: A Column expression representing whether each element of Column is in the list Example: Check if name is in a list of values ```python # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Example: Check if value is in another column ```python # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` """ if isinstance(other, list): try: type_ = infer_dtype_from_pyobj(other) return Column._from_logical_expr(InExpr(self._logical_expr, LiteralExpr(other, type_))) except TypeInferenceError as e: raise ValidationError(f"Cannot apply IN on {other}. List argument to IN must be be a valid Python List literal.") from e else: return Column._from_logical_expr(InExpr(self._logical_expr, other._logical_expr)) ``` ### is_not_null ``` is_not_null() -> Column ``` Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is not NULL Filter rows where a column is not NULL ``` df.filter(col("some_column").is_not_null()) ``` Use in a complex condition ``` df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` Source code in `src/fenic/api/column.py` ``` def is_not_null(self) -> Column: """Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is not NULL Example: Filter rows where a column is not NULL ```python df.filter(col("some_column").is_not_null()) ``` Example: Use in a complex condition ```python df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, False)) ``` ### is_null ``` is_null() -> Column ``` Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is NULL Filter rows where a column is NULL ``` # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Use in a complex condition ``` # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` Source code in `src/fenic/api/column.py` ``` def is_null(self) -> Column: """Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is NULL Example: Filter rows where a column is NULL ```python # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Example: Use in a complex condition ```python # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, True)) ``` ### like ``` like(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "J" and ends with "n" ``` # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Find rows where code matches specific pattern ``` # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` Source code in `src/fenic/api/column.py` ``` def like(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "J" and ends with "n" ```python # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Example: Find rows where code matches specific pattern ```python # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(LikeExpr(self._logical_expr, other_expr)) ``` ### otherwise ``` otherwise(value: Column) -> Column ``` Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Parameters: - **`value`** (`Column`) – Value to return when no prior conditions are True Returns: - **`Column`** ( `Column` ) – The complete conditional expression chain Raises: - `ValidationError` – If called on a non-when expression Example ``` # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain Source code in `src/fenic/api/column.py` ``` def otherwise(self, value: Column) -> Column: """Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Args: value: Value to return when no prior conditions are True Returns: Column: The complete conditional expression chain Raises: ValidationError: If called on a non-when expression Example: ```python # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note: - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain """ return Column._from_logical_expr(OtherwiseExpr(self._logical_expr, value._logical_expr)) ``` ### rlike ``` rlike(other: Union[str, Column]) -> Column ``` Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Parameters: - **`other`** (`Union[str, Column]`) – The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where phone number matches pattern ``` # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Find rows where text contains word boundaries ``` # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` Source code in `src/fenic/api/column.py` ``` def rlike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Args: other (Union[str, Column]): The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where phone number matches pattern ```python # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Example: Find rows where text contains word boundaries ```python # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(RLikeExpr(self._logical_expr, other_expr)) ``` ### starts_with ``` starts_with(other: Union[str, Column]) -> Column ``` Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the start (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value starts with the substring Find rows where name starts with "Mr" ``` # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Find rows where text starts with a dynamic pattern ``` # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: - `ValueError` – If the substring starts with a regular expression anchor (^) Source code in `src/fenic/api/column.py` ``` def starts_with(self, other: Union[str, Column]) -> Column: """Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the start (can be a string or column expression) Returns: Column: A boolean column indicating whether each value starts with the substring Example: Find rows where name starts with "Mr" ```python # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Example: Find rows where text starts with a dynamic pattern ```python # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: ValueError: If the substring starts with a regular expression anchor (^) """ if isinstance(other, str): if other.startswith("^"): raise ValidationError("substr should not start with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(StartsWithExpr(self._logical_expr, other_expr)) ``` ### when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A new when expression with this condition added to the chain Raises: - `ValidationError` – If called on a non-when expression (e.g., regular columns) Example ``` # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. Source code in `src/fenic/api/column.py` ``` def when(self, condition: Column, value: Column) -> Column: """Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A new when expression with this condition added to the chain Raises: ValidationError: If called on a non-when expression (e.g., regular columns) Example: ```python # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. """ return Column._from_logical_expr(WhenExpr(self._logical_expr, condition._logical_expr, value._logical_expr)) ``` ## ColumnField Represents a typed column in a DataFrame schema. A ColumnField defines the structure of a single column by specifying its name and data type. This is used as a building block for DataFrame schemas. Attributes: - **`name`** (`str`) – The name of the column. - **`data_type`** (`DataType`) – The data type of the column, as a DataType instance. ## DataFrame A data collection organized into named columns. The DataFrame class represents a lazily evaluated computation on data. Operations on DataFrame build up a logical query plan that is only executed when an action like show(), to_polars(), to_pandas(), to_arrow(), to_pydict(), to_pylist(), or count() is called. The DataFrame supports method chaining for building complex transformations. Create and transform a DataFrame ``` # Create a DataFrame from a dictionary df = session.create_dataframe({"id": [1, 2, 3], "value": ["a", "b", "c"]}) # Chain transformations result = df.filter(col("id") > 1).select("id", "value") # Show results result.show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 2| b| # | 3| c| # +---+-----+ ``` Methods: - **`agg`** – Aggregate on the entire DataFrame without groups. - **`cache`** – Alias for persist(). Mark DataFrame for caching after first computation. - **`collect`** – Execute the DataFrame computation and return the result as a QueryResult. - **`count`** – Count the number of rows in the DataFrame. - **`distinct`** – Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). - **`drop`** – Remove one or more columns from this DataFrame. - **`drop_duplicates`** – Return a DataFrame with duplicate rows removed. - **`explain`** – Display the logical plan of the DataFrame. - **`explode`** – Create a new row for each element in an array column. - **`explode_outer`** – Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. - **`explode_with_index`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`filter`** – Filters rows using the given condition. - **`group_by`** – Groups the DataFrame using the specified columns. - **`join`** – Joins this DataFrame with another DataFrame. - **`limit`** – Limits the number of rows to the specified number. - **`lineage`** – Create a Lineage object to trace data through transformations. - **`order_by`** – Sort the DataFrame by the specified columns. Alias for sort(). - **`persist`** – Mark this DataFrame to be persisted after first computation. - **`posexplode`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`posexplode_outer`** – Create a new row for each element in an array column with position and value, preserving null/empty arrays. - **`select`** – Projects a set of Column expressions or column names. - **`show`** – Display the DataFrame content in a tabular form. - **`sort`** – Sort the DataFrame by the specified columns. - **`to_arrow`** – Execute the DataFrame computation and return an Apache Arrow Table. - **`to_pandas`** – Execute the DataFrame computation and return a Pandas DataFrame. - **`to_polars`** – Execute the DataFrame computation and return the result as a Polars DataFrame. - **`to_pydict`** – Execute the DataFrame computation and return a dictionary of column arrays. - **`to_pylist`** – Execute the DataFrame computation and return a list of row dictionaries. - **`union`** – Return a new DataFrame containing the union of rows in this and another DataFrame. - **`unnest`** – Unnest the specified struct columns into separate columns. - **`where`** – Filters rows using the given condition (alias for filter()). - **`with_column`** – Add a new column or replace an existing column. - **`with_column_renamed`** – Rename a column. No-op if the column does not exist. - **`with_columns`** – Add multiple new columns or replace existing columns. Attributes: - **`columns`** (`List[str]`) – Get list of column names. - **`schema`** (`Schema`) – Get the schema of this DataFrame. - **`semantic`** (`SemanticExtensions`) – Interface for semantic operations on the DataFrame. - **`write`** (`DataFrameWriter`) – Interface for saving the content of the DataFrame. ### columns ``` columns: List[str] ``` Get list of column names. Returns: - `List[str]` – List[str]: List of all column names in the DataFrame Examples: ``` >>> df.columns ['name', 'age', 'city'] ``` ### schema ``` schema: Schema ``` Get the schema of this DataFrame. Returns: - **`Schema`** ( `Schema` ) – Schema containing field names and data types Examples: ``` >>> df.schema Schema([ ColumnField('name', StringType), ColumnField('age', IntegerType) ]) ``` ### semantic ``` semantic: SemanticExtensions ``` Interface for semantic operations on the DataFrame. ### write ``` write: DataFrameWriter ``` Interface for saving the content of the DataFrame. Returns: - **`DataFrameWriter`** ( `DataFrameWriter` ) – Writer interface to write DataFrame. ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions or dictionary of aggregations. Returns: - **`DataFrame`** ( `DataFrame` ) – Aggregation results. Multiple aggregations ``` # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Dictionary style ``` # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Args: *exprs: Aggregation expressions or dictionary of aggregations. Returns: DataFrame: Aggregation results. Example: Multiple aggregations ```python # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Example: Dictionary style ```python # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` """ return self.group_by().agg(*exprs) ``` ### cache ``` cache() -> DataFrame ``` Alias for persist(). Mark DataFrame for caching after first computation. Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for caching See Also persist(): Full documentation of caching behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def cache(self) -> DataFrame: """Alias for persist(). Mark DataFrame for caching after first computation. Returns: DataFrame: Same DataFrame, but marked for caching See Also: persist(): Full documentation of caching behavior """ return self.persist() ``` ### collect ``` collect(data_type: DataLikeType = 'polars') -> QueryResult ``` Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Parameters: - **`data_type`** (`DataLikeType`, default: `'polars'` ) – The type of data to return Returns: - **`QueryResult`** ( `QueryResult` ) – A QueryResult with materialized data and query metrics Source code in `src/fenic/api/dataframe/dataframe.py` ``` def collect(self, data_type: DataLikeType = "polars") -> QueryResult: """Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Args: data_type: The type of data to return Returns: QueryResult: A QueryResult with materialized data and query metrics """ result: Tuple[pl.DataFrame, QueryMetrics] = self._session_state.execution.collect(self._logical_plan) df, metrics = result logger.info(metrics.get_summary()) if data_type == "polars": return QueryResult(df, metrics) elif data_type == "pandas": return QueryResult(df.to_pandas(use_pyarrow_extension_array=True), metrics) elif data_type == "arrow": return QueryResult(df.to_arrow(), metrics) elif data_type == "pydict": return QueryResult(df.to_dict(as_series=False), metrics) elif data_type == "pylist": return QueryResult(df.to_dicts(), metrics) else: raise ValidationError(f"Invalid data type: {data_type} in collect(). Valid data types are: polars, pandas, arrow, pydict, pylist") ``` ### count ``` count() -> int ``` Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: - **`int`** ( `int` ) – The number of rows in the DataFrame Source code in `src/fenic/api/dataframe/dataframe.py` ``` def count(self) -> int: """Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: int: The number of rows in the DataFrame """ return self._session_state.execution.count(self._logical_plan)[0] ``` ### distinct ``` distinct() -> DataFrame ``` Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Example ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def distinct(self) -> DataFrame: """Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: DataFrame: A new DataFrame with duplicate rows removed. Example: ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ return self.drop_duplicates() ``` ### drop ``` drop(*col_names: str) -> DataFrame ``` Remove one or more columns from this DataFrame. Parameters: - **`*col_names`** (`str`, default: `()` ) – Names of columns to drop. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame without specified columns. Raises: - `ValueError` – If any specified column doesn't exist in the DataFrame. - `ValueError` – If dropping the columns would result in an empty DataFrame. Drop single column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Drop multiple columns ``` # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Error when dropping non-existent column ``` # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop(self, *col_names: str) -> DataFrame: """Remove one or more columns from this DataFrame. Args: *col_names: Names of columns to drop. Returns: DataFrame: New DataFrame without specified columns. Raises: ValueError: If any specified column doesn't exist in the DataFrame. ValueError: If dropping the columns would result in an empty DataFrame. Example: Drop single column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Drop multiple columns ```python # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Example: Error when dropping non-existent column ```python # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` """ if not col_names: return self current_cols = set(self.columns) to_drop = set(col_names) missing = to_drop - current_cols if missing: missing_str = ( f"Column '{next(iter(missing))}'" if len(missing) == 1 else f"Columns {sorted(missing)}" ) raise ValueError(f"{missing_str} not found in DataFrame") remaining_cols = [ col(c)._logical_expr for c in self.columns if c not in to_drop ] if not remaining_cols: raise ValueError("Cannot drop all columns from DataFrame") return self._from_logical_plan( Projection.from_session_state(self._logical_plan, remaining_cols, self._session_state), self._session_state, ) ``` ### drop_duplicates ``` drop_duplicates(subset: Optional[List[str]] = None) -> DataFrame ``` Return a DataFrame with duplicate rows removed. Parameters: - **`subset`** (`Optional[List[str]]`, default: `None` ) – Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Raises: - `ValueError` – If a specified column is not present in the current DataFrame schema. Remove duplicates considering specific columns ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop_duplicates( self, subset: Optional[List[str]] = None, ) -> DataFrame: """Return a DataFrame with duplicate rows removed. Args: subset: Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: DataFrame: A new DataFrame with duplicate rows removed. Raises: ValueError: If a specified column is not present in the current DataFrame schema. Example: Remove duplicates considering specific columns ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ exprs = [] if subset: for c in subset: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( DropDuplicates.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### explain ``` explain() -> None ``` Display the logical plan of the DataFrame. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explain(self) -> None: """Display the logical plan of the DataFrame.""" print(str(self._logical_plan)) ``` ### explode ``` explode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. Raises: - `TypeError` – If column argument is not a string or Column. Explode array column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Using column expression ``` # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Raises: TypeError: If column argument is not a string or Column. Example: Explode array column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Example: Using column expression ```python # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state(self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state), self._session_state, ) ``` ### explode_outer ``` explode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. - `DataFrame` – Rows with null or empty arrays are preserved with null in the exploded column. Explode with outer join behavior ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Rows with null or empty arrays are preserved with null in the exploded column. Example: Explode with outer join behavior ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state, keep_null_and_empty=True ), self._session_state, ) ``` ### explode_with_index ``` explode_with_index(column: ColumnOrName, index_col_name: str = 'pos', value_col_name: str = 'col', keep_null_and_empty: bool = False) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. - **`index_col_name`** (`str`, default: `'pos'` ) – Name for the column containing 0-based array positions (default: "pos"). - **`value_col_name`** (`str`, default: `'col'` ) – Name for the exploded value column (default: "col"). - **`keep_null_and_empty`** (`bool`, default: `False` ) – If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Explode with index ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Custom column names ``` df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_with_index( self, column: ColumnOrName, index_col_name: str = "pos", value_col_name: str = "col", keep_null_and_empty: bool = False, ) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Args: column: Name of array column to explode (as string) or Column expression. index_col_name: Name for the column containing 0-based array positions (default: "pos"). value_col_name: Name for the exploded value column (default: "col"). keep_null_and_empty: If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: DataFrame: New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Example: Explode with index ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Example: Custom column names ```python df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` """ return self._from_logical_plan( ExplodeWithIndex.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, index_col_name, value_col_name, self._session_state, keep_null_and_empty, ), self._session_state, ) ``` ### filter ``` filter(condition: Column) -> DataFrame ``` Filters rows using the given condition. Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame Filter with numeric comparison ``` # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with semantic predicate ``` # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with multiple conditions ``` # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def filter(self, condition: Column) -> DataFrame: """Filters rows using the given condition. Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame Example: Filter with numeric comparison ```python # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with semantic predicate ```python # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with multiple conditions ```python # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` """ return self._from_logical_plan( Filter.from_session_state(self._logical_plan, condition._logical_expr, self._session_state), self._session_state, ) ``` ### group_by ``` group_by(*cols: ColumnOrName) -> GroupedData ``` Groups the DataFrame using the specified columns. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns to group by. Can be column names as strings or Column expressions. Returns: - **`GroupedData`** ( `GroupedData` ) – Object for performing aggregations on the grouped data. Group by single column ``` # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Group by multiple columns ``` # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Group by expression ``` # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def group_by(self, *cols: ColumnOrName) -> GroupedData: """Groups the DataFrame using the specified columns. Args: *cols: Columns to group by. Can be column names as strings or Column expressions. Returns: GroupedData: Object for performing aggregations on the grouped data. Example: Group by single column ```python # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Example: Group by multiple columns ```python # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Example: Group by expression ```python # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` """ return GroupedData(self, list(cols) if cols else None) ``` ### join ``` join(other: DataFrame, on: Union[str, List[str]], *, how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, *, left_on: Union[ColumnOrName, List[ColumnOrName]], right_on: Union[ColumnOrName, List[ColumnOrName]], how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = 'inner') -> DataFrame ``` Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Parameters: - **`other`** (`DataFrame`) – DataFrame to join with. - **`on`** (`Optional[Union[str, List[str]]]`, default: `None` ) – Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins - **`left_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions - **`right_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions - **`how`** (`JoinType`, default: `'inner'` ) – Type of join to perform. Returns: - `DataFrame` – Joined DataFrame. Raises: - `ValidationError` – If cross join is used with an ON clause. - `ValidationError` – If join condition is invalid. - `ValidationError` – If both 'on' and 'left_on'/'right_on' parameters are provided. - `ValidationError` – If only one of 'left_on' or 'right_on' is provided. - `ValidationError` – If 'left_on' and 'right_on' have different lengths Inner join on column name ``` # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Join with expression ``` # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Cross join ``` # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def join( self, other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = "inner", ) -> DataFrame: """Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Args: other: DataFrame to join with. on: Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins left_on: Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions right_on: Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions how: Type of join to perform. Returns: Joined DataFrame. Raises: ValidationError: If cross join is used with an ON clause. ValidationError: If join condition is invalid. ValidationError: If both 'on' and 'left_on'/'right_on' parameters are provided. ValidationError: If only one of 'left_on' or 'right_on' is provided. ValidationError: If 'left_on' and 'right_on' have different lengths Example: Inner join on column name ```python # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Join with expression ```python # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Cross join ```python # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` """ validate_join_parameters(self, on, left_on, right_on, how) # Build join conditions left_conditions, right_conditions = build_join_conditions(on, left_on, right_on) self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( Join.from_session_state( self._logical_plan, other._logical_plan, left_conditions, right_conditions, how, self._session_state), self._session_state, ) ``` ### limit ``` limit(n: int) -> DataFrame ``` Limits the number of rows to the specified number. Parameters: - **`n`** (`int`) – Maximum number of rows to return. Returns: - **`DataFrame`** ( `DataFrame` ) – DataFrame with at most n rows. Raises: - `TypeError` – If n is not an integer. Limit rows ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Limit with other operations ``` # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def limit(self, n: int) -> DataFrame: """Limits the number of rows to the specified number. Args: n: Maximum number of rows to return. Returns: DataFrame: DataFrame with at most n rows. Raises: TypeError: If n is not an integer. Example: Limit rows ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Limit with other operations ```python # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` """ return self._from_logical_plan( Limit.from_session_state(self._logical_plan, n, self._session_state), self._session_state) ``` ### lineage ``` lineage() -> Lineage ``` Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: - **`Lineage`** ( `Lineage` ) – Interface for querying data lineage Example ``` # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also LineageQuery: Full documentation of lineage querying capabilities Source code in `src/fenic/api/dataframe/dataframe.py` ``` def lineage(self) -> Lineage: """Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: Lineage: Interface for querying data lineage Example: ```python # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also: LineageQuery: Full documentation of lineage querying capabilities """ return Lineage(self._session_state.execution.build_lineage(self._logical_plan)) ``` ### order_by ``` order_by(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Alias for sort(). Returns: - **`DataFrame`** ( `DataFrame` ) – sorted Dataframe. See Also sort(): Full documentation of sorting behavior and parameters. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def order_by( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Alias for sort(). Returns: DataFrame: sorted Dataframe. See Also: sort(): Full documentation of sorting behavior and parameters. """ return self.sort(cols, ascending) ``` ### persist ``` persist() -> DataFrame ``` Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for persistence Example ``` # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def persist(self) -> DataFrame: """Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: DataFrame: Same DataFrame, but marked for persistence Example: ```python # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` """ cache_info = CacheInfo(cache_key=f"cache_{uuid.uuid4().hex}") self._logical_plan.set_cache_info(cache_info) return self._from_logical_plan( self._logical_plan, self._session_state) ``` ### posexplode ``` posexplode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. PySpark-style posexplode ``` df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Example: PySpark-style posexplode ```python df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` """ return self.explode_with_index(column) ``` ### posexplode_outer ``` posexplode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. - `DataFrame` – Rows with null or empty arrays are preserved with (null, null). PySpark-style posexplode_outer ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Rows with null or empty arrays are preserved with (null, null). Example: PySpark-style posexplode_outer ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` """ return self.explode_with_index( column, keep_null_and_empty=True ) ``` ### select ``` select(*cols: ColumnOrName) -> DataFrame ``` Projects a set of Column expressions or column names. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with selected columns Select by column names ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Select with expressions ``` # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Mix strings and expressions ``` # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def select(self, *cols: ColumnOrName) -> DataFrame: """Projects a set of Column expressions or column names. Args: *cols: Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: DataFrame: A new DataFrame with selected columns Example: Select by column names ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Example: Select with expressions ```python # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Example: Mix strings and expressions ```python # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` """ exprs = [] if not cols: return self for c in cols: if isinstance(c, str): if c == "*": exprs.extend(col(field)._logical_expr for field in self.columns) else: exprs.append(col(c)._logical_expr) else: exprs.append(c._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### show ``` show(n: int = 10, explain_analyze: bool = False) -> None ``` Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Parameters: - **`n`** (`int`, default: `10` ) – Number of rows to display - **`explain_analyze`** (`bool`, default: `False` ) – Whether to print the explain analyze plan Source code in `src/fenic/api/dataframe/dataframe.py` ``` def show(self, n: int = 10, explain_analyze: bool = False) -> None: """Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Args: n: Number of rows to display explain_analyze: Whether to print the explain analyze plan """ output, metrics = self._session_state.execution.show(self._logical_plan, n) logger.info(metrics.get_summary()) print(output) if explain_analyze: print(metrics.get_execution_plan_details()) ``` ### sort ``` sort(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Parameters: - **`cols`** (`Union[ColumnOrName, List[ColumnOrName], None]`, default: `None` ) – Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. - **`ascending`** (`Optional[Union[bool, List[bool]]]`, default: `None` ) – A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame sorted by the specified columns. Raises: - `ValueError` – - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used - `TypeError` – - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Sort in ascending order ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Sort in descending order ``` # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Sort with boolean ascending parameter ``` # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Multiple columns with different sort orders ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Multiple columns with list of ascending strategies ``` # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def sort( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Args: cols: Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. ascending: A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: DataFrame: A new DataFrame sorted by the specified columns. Raises: ValueError: - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used TypeError: - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Example: Sort in ascending order ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Example: Sort in descending order ```python # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Sort with boolean ascending parameter ```python # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Multiple columns with different sort orders ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Example: Multiple columns with list of ascending strategies ```python # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` """ col_args = cols if cols is None: return self._from_logical_plan( Sort.from_session_state(self._logical_plan, [], self._session_state), self._session_state, ) elif not isinstance(cols, List): col_args = [cols] # parse the ascending arguments bool_ascending = [] using_default_ascending = False if ascending is None: using_default_ascending = True bool_ascending = [True] * len(col_args) elif isinstance(ascending, bool): bool_ascending = [ascending] * len(col_args) elif isinstance(ascending, List): bool_ascending = ascending if len(bool_ascending) != len(cols): raise ValueError( f"the list length of ascending sort strategies must match the specified sort columns" f"Got {len(cols)} column expressions and {len(bool_ascending)} ascending strategies. " ) else: raise TypeError( f"Invalid ascending strategy type: {type(ascending)}. Must be a boolean or list of booleans." ) # create our list of sort expressions, for each column expression # that isn't already provided as a asc()/desc() SortExpr sort_exprs = [] for c, asc_bool in zip(col_args, bool_ascending, strict=True): if isinstance(c, ColumnOrName): c_expr = Column._from_col_or_name(c)._logical_expr else: raise TypeError( f"Invalid column type: {type(c).__name__}. Must be a string or Column Expression." ) if not isinstance(asc_bool, bool): raise TypeError( f"Invalid ascending strategy type: {type(asc_bool).__name__}. Must be a boolean." ) if isinstance(c_expr, SortExpr): if not using_default_ascending: raise TypeError( "Cannot specify both asc()/desc() expressions and boolean ascending strategies." f"Got expression: {c_expr} and ascending argument: {bool_ascending}" ) sort_exprs.append(c_expr) else: sort_exprs.append(SortExpr(c_expr, ascending=asc_bool)) return self._from_logical_plan( Sort.from_session_state(self._logical_plan, sort_exprs, self._session_state), self._session_state, ) ``` ### to_arrow ``` to_arrow() -> pa.Table ``` Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: - `Table` – pa.Table: An Apache Arrow Table containing the computed results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_arrow(self) -> pa.Table: """Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: pa.Table: An Apache Arrow Table containing the computed results """ return self.collect("arrow").data ``` ### to_pandas ``` to_pandas() -> pd.DataFrame ``` Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: - `DataFrame` – pd.DataFrame: A Pandas DataFrame containing the computed results with Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pandas(self) -> pd.DataFrame: """Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: pd.DataFrame: A Pandas DataFrame containing the computed results with """ return self.collect("pandas").data ``` ### to_polars ``` to_polars() -> pl.DataFrame ``` Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: - `DataFrame` – pl.DataFrame: A Polars DataFrame with materialized results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_polars(self) -> pl.DataFrame: """Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: pl.DataFrame: A Polars DataFrame with materialized results """ return self.collect("polars").data ``` ### to_pydict ``` to_pydict() -> Dict[str, List[Any]] ``` Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: - `Dict[str, List[Any]]` – Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pydict(self) -> Dict[str, List[Any]]: """Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column """ return self.collect("pydict").data ``` ### to_pylist ``` to_pylist() -> List[Dict[str, Any]] ``` Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: - `List[Dict[str, Any]]` – List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pylist(self) -> List[Dict[str, Any]]: """Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result """ return self.collect("pylist").data ``` ### union ``` union(other: DataFrame) -> DataFrame ``` Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Parameters: - **`other`** (`DataFrame`) – Another DataFrame with the same schema. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing rows from both DataFrames. Raises: - `ValueError` – If the DataFrames have different schemas. - `TypeError` – If other is not a DataFrame. Union two DataFrames ``` # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Union with duplicates ``` # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def union(self, other: DataFrame) -> DataFrame: """Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Args: other: Another DataFrame with the same schema. Returns: DataFrame: A new DataFrame containing rows from both DataFrames. Raises: ValueError: If the DataFrames have different schemas. TypeError: If other is not a DataFrame. Example: Union two DataFrames ```python # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Example: Union with duplicates ```python # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` """ self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( UnionLogicalPlan.from_session_state([self._logical_plan, other._logical_plan], self._session_state), self._session_state, ) ``` ### unnest ``` unnest(*col_names: str) -> DataFrame ``` Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Parameters: - **`*col_names`** (`str`, default: `()` ) – One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with the specified struct columns expanded. Raises: - `TypeError` – If any argument is not a string or Column. - `ValueError` – If a specified column does not contain struct data. Unnest struct column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Unnest multiple struct columns ``` # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def unnest(self, *col_names: str) -> DataFrame: """Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Args: *col_names: One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: DataFrame: A new DataFrame with the specified struct columns expanded. Raises: TypeError: If any argument is not a string or Column. ValueError: If a specified column does not contain struct data. Example: Unnest struct column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Example: Unnest multiple struct columns ```python # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` """ if not col_names: return self exprs = [] for c in col_names: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( Unnest.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### where ``` where(condition: Column) -> DataFrame ``` Filters rows using the given condition (alias for filter()). Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame See Also filter(): Full documentation of filtering behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def where(self, condition: Column) -> DataFrame: """Filters rows using the given condition (alias for filter()). Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame See Also: filter(): Full documentation of filtering behavior """ return self.filter(condition) ``` ### with_column ``` with_column(col_name: str, col: Union[Any, Column, Series, Series]) -> DataFrame ``` Add a new column or replace an existing column. Parameters: - **`col_name`** (`str`) – Name of the new column - **`col`** (`Union[Any, Column, Series, Series]`) – Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced column Raises: - `ExecutionError` – - If a Series length does not match the DataFrame height - `ValidationError` – - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Add literal column ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Add computed column ``` # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Replace existing column ``` # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Add column with complex expression ``` # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Add column from Polars Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Add column from pandas Series ``` import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column(self, col_name: str, col: Union[Any, Column, pl.Series, pd.Series]) -> DataFrame: """Add a new column or replace an existing column. Args: col_name: Name of the new column col: Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced column Raises: ExecutionError: - If a Series length does not match the DataFrame height ValidationError: - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Example: Add literal column ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Example: Add computed column ```python # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Example: Replace existing column ```python # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Example: Add column with complex expression ```python # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Example: Add column from Polars Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Example: Add column from pandas Series ```python import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` """ exprs = [] # Handle different input types: Column, Series, or literal value if isinstance(col, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col = Column._from_logical_expr(SeriesLiteralExpr(col)) elif not isinstance(col, Column): # Wrap other values as literals col = lit(col) for field in self.columns: if field != col_name: exprs.append(Column._from_column_name(field)._logical_expr) # Add the new column with alias exprs.append(col.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_column_renamed ``` with_column_renamed(col_name: str, new_col_name: str) -> DataFrame ``` Rename a column. No-op if the column does not exist. Parameters: - **`col_name`** (`str`) – Name of the column to rename. - **`new_col_name`** (`str`) – New name for the column. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the column renamed. Rename a column ``` # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Rename multiple columns ``` # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column_renamed(self, col_name: str, new_col_name: str) -> DataFrame: """Rename a column. No-op if the column does not exist. Args: col_name: Name of the column to rename. new_col_name: New name for the column. Returns: DataFrame: New DataFrame with the column renamed. Example: Rename a column ```python # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Example: Rename multiple columns ```python # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` """ exprs = [] renamed = False for field in self.schema.column_fields: name = field.name if name == col_name: exprs.append(col(name).alias(new_col_name)._logical_expr) renamed = True else: exprs.append(col(name)._logical_expr) if not renamed: return self return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_columns ``` with_columns(cols_map: Dict[str, Union[Any, Column, Series, Series]]) -> DataFrame ``` Add multiple new columns or replace existing columns. Parameters: - **`cols_map`** (`Dict[str, Union[Any, Column, Series, Series]]`) – A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced columns Raises: - `ValueError` – - If two columns being created in the same `with_columns` call depend on each other - `ExecutionError` – - If any Series length does not match the DataFrame height - `ValidationError` – - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Add multiple columns ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Replace and add columns ``` # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Complex expressions ``` # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Add columns from Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Mix Series with Column expressions ``` import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Error when adding columns that depend on each other ``` df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_columns(self, cols_map: Dict[str, Union[Any, Column, pl.Series, pd.Series]]) -> DataFrame: """Add multiple new columns or replace existing columns. Args: cols_map: A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced columns Raises: ValueError: - If two columns being created in the same `with_columns` call depend on each other ExecutionError: - If any Series length does not match the DataFrame height ValidationError: - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Example: Add multiple columns ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Example: Replace and add columns ```python # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Example: Complex expressions ```python # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Example: Add columns from Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Example: Mix Series with Column expressions ```python import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Example: Error when adding columns that depend on each other ```python df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` """ if not cols_map: return self exprs = [] new_col_names = set(cols_map.keys()) # Add existing columns that are not being replaced for field in self.columns: if field not in new_col_names: exprs.append(Column._from_column_name(field)._logical_expr) # Add all new columns with aliases for col_name, col_expr in cols_map.items(): # Handle different input types: Column, Series, or literal value if isinstance(col_expr, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col_expr = Column._from_logical_expr(SeriesLiteralExpr(col_expr)) elif not isinstance(col_expr, Column): # Automatically wrap non-Column values (literals) with lit() for convenience # This allows users to pass raw Python values like: {"constant": 100, "status": "active"} # instead of requiring: {"constant": lit(100), "status": lit("active")} col_expr = lit(col_expr) exprs.append(col_expr.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ## DataFrameReader ``` DataFrameReader(session_state: BaseSessionState) ``` Interface used to load a DataFrame from external storage systems. Similar to PySpark's DataFrameReader. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Hugging Face Datasets (hf://) - Format: hf://{repo_type}/{repo_id}/{path_to_file} - Notes: - Supports glob patterns (*,* \*) - Supports dataset revisions and branch aliases (e.g., @refs/convert/parquet, @~parquet) - HF_TOKEN environment variable is required to read private datasets. - Examples: - hf://datasets/datasets-examples/doc-formats-csv-1/data.csv - hf://datasets/cais/mmlu/astronomy/\*.parquet - hf://datasets/datasets-examples/doc-formats-csv-1@~parquet/\**/*.parquet - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Creates a DataFrameReader. Parameters: - **`session_state`** (`BaseSessionState`) – The session state to use for reading Methods: - **`csv`** – Load a DataFrame from one or more CSV files. - **`docs`** – Load a DataFrame with the document contents of a list of paths (markdown or json). - **`parquet`** – Load a DataFrame from one or more Parquet files. - **`pdf_metadata`** – Load a DataFrame with metadata of PDF files in a list of paths. Source code in `src/fenic/api/io/reader.py` ``` def __init__(self, session_state: BaseSessionState): """Creates a DataFrameReader. Args: session_state: The session state to use for reading """ self._options: Dict[str, Any] = {} self._session_state = session_state ``` ### csv ``` csv(paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more CSV files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.csv"), or a list of paths. - **`schema`** (`Optional[Schema]`, default: `None` ) – (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. - **`merge_schemas`** (`bool`, default: `False` ) – Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If both `schema` and `merge_schemas=True` are provided. - `ValidationError` – If any path does not end with `.csv`. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single CSV file ``` df = session.read.csv("file.csv") ``` Read multiple CSV files with schema merging ``` df = session.read.csv("data/*.csv", merge_schemas=True) ``` Read CSV files with explicit schema `python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) )` Source code in `src/fenic/api/io/reader.py` ``` def csv( self, paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more CSV files. Args: paths: A single file path, a glob pattern (e.g., "data/*.csv"), or a list of paths. schema: (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. merge_schemas: Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: ValidationError: If both `schema` and `merge_schemas=True` are provided. ValidationError: If any path does not end with `.csv`. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single CSV file ```python df = session.read.csv("file.csv") ``` Example: Read multiple CSV files with schema merging ```python df = session.read.csv("data/*.csv", merge_schemas=True) ``` Example: Read CSV files with explicit schema ```python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) ) ``` """ if schema is not None and merge_schemas: raise ValidationError( "Cannot specify both 'schema' and 'merge_schemas=True' - these options conflict. " "Choose one approach: " "1) Use 'schema' to enforce a specific schema: csv(paths, schema=your_schema), " "2) Use 'merge_schemas=True' to automatically merge schemas: csv(paths, merge_schemas=True), " "3) Use neither to inherit schema from the first file: csv(paths)" ) if schema is not None: for col_field in schema.column_fields: if not isinstance( col_field.data_type, _PrimitiveType, ): raise ValidationError( f"CSV files only support primitive data types in schema definitions. " f"Column '{col_field.name}' has type {type(col_field.data_type).__name__}, but CSV schemas must use: " f"IntegerType, FloatType, DoubleType, BooleanType, or StringType. " f"Example: Schema([ColumnField(name='id', data_type=IntegerType), ColumnField(name='name', data_type=StringType)])" ) options = { "merge_schemas": merge_schemas, } if schema: options["schema"] = schema return self._read_file( paths, file_format="csv", file_extension=".csv", **options ) ``` ### docs ``` docs(paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal['markdown', 'json'], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with the document contents of a list of paths (markdown or json). Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`content_type`** (`Literal['markdown', 'json']`) – Content type of the files. One of "markdown" or "json". - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.md` or `.json` depending on the content_type. - `UnsupportedFileTypeError` – If the specified content_type is not "markdown" or "json" . Notes - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read all the markdown files in a folder and all its subfolders. ``` df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Read a folder of markdown files excluding some files. ``` df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` Source code in `src/fenic/api/io/reader.py` ``` def docs( self, paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal["markdown", "json"], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with the document contents of a list of paths (markdown or json). Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. content_type: Content type of the files. One of "markdown" or "json". exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.md` or `.json` depending on the content_type. UnsupportedFileTypeError: If the specified content_type is not "markdown" or "json" . Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read all the markdown files in a folder and all its subfolders. ```python df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Example: Read a folder of markdown files excluding some files. ```python df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) if content_type not in ["markdown", "json"]: raise UnsupportedFileTypeError(f"{content_type}, must be 'markdown' or 'json'") logical_node = DocSource.from_session_state( paths=path_str_list, content_type=content_type, exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ### parquet ``` parquet(paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more Parquet files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.parquet"), or a list of paths. - **`merge_schemas`** (`bool`, default: `False` ) – If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - Date and datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If any file does not have a `.parquet` extension. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single Parquet file ``` df = session.read.parquet("file.parquet") ``` Read multiple Parquet files ``` df = session.read.parquet("data/*.parquet") ``` Read Parquet files with schema merging ``` df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` Source code in `src/fenic/api/io/reader.py` ``` def parquet( self, paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more Parquet files. Args: paths: A single file path, a glob pattern (e.g., "data/*.parquet"), or a list of paths. merge_schemas: If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior: - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - Date and datetime columns are cast to strings during ingestion. Raises: ValidationError: If any file does not have a `.parquet` extension. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single Parquet file ```python df = session.read.parquet("file.parquet") ``` Example: Read multiple Parquet files ```python df = session.read.parquet("data/*.parquet") ``` Example: Read Parquet files with schema merging ```python df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` """ options = { "merge_schemas": merge_schemas, } return self._read_file( paths, file_format="parquet", file_extension=".parquet", **options ) ``` ### pdf_metadata ``` pdf_metadata(paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with metadata of PDF files in a list of paths. Note Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.pdf` extension. Notes - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read the metadata of all the PDF files in a folder and all its subfolders. ``` df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Read a metadata of PDFS in a folder, excluding some files. ``` df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` Source code in `src/fenic/api/io/reader.py` ``` def pdf_metadata( self, paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with metadata of PDF files in a list of paths. Note: Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.pdf` extension. Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read the metadata of all the PDF files in a folder and all its subfolders. ```python df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Example: Read a metadata of PDFS in a folder, excluding some files. ```python df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) logical_node = DocSource.from_session_state( paths=path_str_list, content_type="pdf", exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ## DataFrameWriter ``` DataFrameWriter(dataframe: DataFrame) ``` Interface used to write a DataFrame to external storage systems. Similar to PySpark's DataFrameWriter. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Initialize a DataFrameWriter. Parameters: - **`dataframe`** (`DataFrame`) – The DataFrame to write. Methods: - **`csv`** – Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. - **`parquet`** – Saves the content of the DataFrame as a single Parquet file. - **`save_as_table`** – Saves the content of the DataFrame as the specified table. - **`save_as_view`** – Saves the content of the DataFrame as a view. Source code in `src/fenic/api/io/writer.py` ``` def __init__(self, dataframe: DataFrame): """Initialize a DataFrameWriter. Args: dataframe: The DataFrame to write. """ self._dataframe = dataframe ``` ### csv ``` csv(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the CSV file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.csv("output.csv") # Overwrites if exists ``` Save with error mode ``` df.write.csv("output.csv", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.csv("output.csv", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def csv( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Args: file_path: Path to save the CSV file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.csv("output.csv") # Overwrites if exists ``` Example: Save with error mode ```python df.write.csv("output.csv", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.csv("output.csv", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".csv"): raise ValidationError( f"CSV writer requires a '.csv' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="csv", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### parquet ``` parquet(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single Parquet file. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the Parquet file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.parquet("output.parquet") # Overwrites if exists ``` Save with error mode ``` df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def parquet( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single Parquet file. Args: file_path: Path to save the Parquet file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.parquet("output.parquet") # Overwrites if exists ``` Example: Save with error mode ```python df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".parquet"): raise ValidationError( f"Parquet writer requires a '.parquet' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="parquet", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_table ``` save_as_table(table_name: str, mode: Literal['error', 'append', 'overwrite', 'ignore'] = 'error') -> QueryMetrics ``` Saves the content of the DataFrame as the specified table. Parameters: - **`table_name`** (`str`) – Name of the table to save to - **`mode`** (`Literal['error', 'append', 'overwrite', 'ignore']`, default: `'error'` ) – Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with error mode (default) ``` df.write.save_as_table("my_table") # Raises error if table exists ``` Save with append mode ``` df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Save with overwrite mode ``` df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` Source code in `src/fenic/api/io/writer.py` ``` def save_as_table( self, table_name: str, mode: Literal["error", "append", "overwrite", "ignore"] = "error", ) -> QueryMetrics: """Saves the content of the DataFrame as the specified table. Args: table_name: Name of the table to save to mode: Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: QueryMetrics: The query metrics Example: Save with error mode (default) ```python df.write.save_as_table("my_table") # Raises error if table exists ``` Example: Save with append mode ```python df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Example: Save with overwrite mode ```python df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` """ sink_plan = TableSink.from_session_state( child=self._dataframe._logical_plan, table_name=table_name, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_as_table( sink_plan, table_name=table_name, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_view ``` save_as_view(view_name: str, description: str | None = None) -> None ``` Saves the content of the DataFrame as a view. Parameters: - **`view_name`** (`str`) – Name of the view to save to - **`description`** (`str | None`, default: `None` ) – Optional human-readable view description to store in the catalog. Returns: - `None` – None. Source code in `src/fenic/api/io/writer.py` ``` def save_as_view( self, view_name: str, description: str | None = None, ) -> None: """Saves the content of the DataFrame as a view. Args: view_name: Name of the view to save to description: Optional human-readable view description to store in the catalog. Returns: None. """ self._dataframe._session_state.execution.save_as_view( logical_plan=self._dataframe._logical_plan, view_name=view_name, view_description=description ) ``` ## DataType Bases: `ABC` ``` flowchart TD fenic.DataType[DataType] click fenic.DataType href "" "fenic.DataType" ``` Base class for all data types. You won't instantiate this class directly. Instead, use one of the concrete types like `StringType`, `ArrayType`, or `StructType`. Used for casting, type validation, and schema inference in the DataFrame API. ## DatasetMetadata Metadata for a dataset (table or view). Attributes: - **`schema`** (`Schema`) – The schema of the dataset. - **`description`** (`Optional[str]`) – The natural language description of the dataset's contents. ## DocumentPathType Bases: `_LogicalType` ``` flowchart TD fenic.DocumentPathType[DocumentPathType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.DocumentPathType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.DocumentPathType href "" "fenic.DocumentPathType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a a document's local (file system) or remote (URL) path. ## EmbeddingType Bases: `_LogicalType` ``` flowchart TD fenic.EmbeddingType[EmbeddingType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.EmbeddingType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.EmbeddingType href "" "fenic.EmbeddingType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a fixed-length embedding vector. Attributes: - **`dimensions`** (`int`) – The number of dimensions in the embedding vector. - **`embedding_model`** (`str`) – Name of the model used to generate the embedding. Create an embedding type for text-embedding-3-small ``` EmbeddingType(384, embedding_model="text-embedding-3-small") ``` ## GoogleDeveloperEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.GoogleDeveloperEmbeddingModel[GoogleDeveloperEmbeddingModel] click fenic.GoogleDeveloperEmbeddingModel href "" "fenic.GoogleDeveloperEmbeddingModel" ``` Configuration for Google Developer embedding models. This class defines the configuration settings for Google embedding models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperEmbeddingModelName`) – The name of the Google Developer embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer embedding model with rate limits: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Developer embedding model with profiles: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleDeveloperEmbeddingModelConfig.Profile(), "high_dim": GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.GoogleDeveloperEmbeddingModel.Profile[Profile] click fenic.GoogleDeveloperEmbeddingModel.Profile href "" "fenic.GoogleDeveloperEmbeddingModel.Profile" ``` Profile configurations for Google Developer embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile() ``` ## GoogleDeveloperLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.GoogleDeveloperLanguageModel[GoogleDeveloperLanguageModel] click fenic.GoogleDeveloperLanguageModel href "" "fenic.GoogleDeveloperLanguageModel" ``` Configuration for Gemini models accessible through Google Developer AI Studio. This class defines the configuration settings for Google Gemini models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperLanguageModelName`) – The name of the Google Developer model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer model with rate limits: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Developer model with profiles: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleDeveloperLanguageModel.Profile(), "fast": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=1024 ), "thorough": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.GoogleDeveloperLanguageModel.Profile[Profile] click fenic.GoogleDeveloperLanguageModel.Profile href "" "fenic.GoogleDeveloperLanguageModel.Profile" ``` Profile configurations for Google Developer models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_level="high") ``` ## GoogleVertexEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.GoogleVertexEmbeddingModel[GoogleVertexEmbeddingModel] click fenic.GoogleVertexEmbeddingModel href "" "fenic.GoogleVertexEmbeddingModel" ``` Configuration for Google Vertex AI embedding models. This class defines the configuration settings for Google embedding models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexEmbeddingModelName`) – The name of the Google Vertex embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex embedding model with rate limits: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Vertex embedding model with profiles: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleVertexEmbeddingModel.Profile(), "high_dim": GoogleVertexEmbeddingModel.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.GoogleVertexEmbeddingModel.Profile[Profile] click fenic.GoogleVertexEmbeddingModel.Profile href "" "fenic.GoogleVertexEmbeddingModel.Profile" ``` Profile configurations for Google Vertex embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleVertexEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleVertexEmbeddingModelConfig.Profile() ``` ## GoogleVertexLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.GoogleVertexLanguageModel[GoogleVertexLanguageModel] click fenic.GoogleVertexLanguageModel href "" "fenic.GoogleVertexLanguageModel" ``` Configuration for Google Vertex AI models. This class defines the configuration settings for Google Gemini models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexLanguageModelName`) – The name of the Google Vertex model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex model with rate limits: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Vertex model with profiles: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleVertexLanguageModel.Profile(), "fast": GoogleVertexLanguageModel.Profile(thinking_token_budget=1024), "thorough": GoogleVertexLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.GoogleVertexLanguageModel.Profile[Profile] click fenic.GoogleVertexLanguageModel.Profile href "" "fenic.GoogleVertexLanguageModel.Profile" ``` Profile configurations for Google Vertex models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same underlying model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleVertexLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleVertexLanguageModel.Profile(thinking_level="high") ``` ## GroupedData ``` GroupedData(df: DataFrame, by: Optional[List[ColumnOrName]] = None) ``` Bases: `BaseGroupedData` ``` flowchart TD fenic.GroupedData[GroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData[BaseGroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData --> fenic.GroupedData click fenic.GroupedData href "" "fenic.GroupedData" click fenic.api.dataframe._base_grouped_data.BaseGroupedData href "" "fenic.api.dataframe._base_grouped_data.BaseGroupedData" ``` Methods for aggregations on a grouped DataFrame. Initialize grouped data. Parameters: - **`df`** (`DataFrame`) – The DataFrame to group. - **`by`** (`Optional[List[ColumnOrName]]`, default: `None` ) – Optional list of columns to group by. Methods: - **`agg`** – Compute aggregations on grouped data and return the result as a DataFrame. Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def __init__(self, df: DataFrame, by: Optional[List[ColumnOrName]] = None): """Initialize grouped data. Args: df: The DataFrame to group. by: Optional list of columns to group by. """ super().__init__(df) self._by: List[Column] = [] for c in by or []: if isinstance(c, str): self._by.append(col(c)) elif isinstance(c, Column): # Allow any expression except literals if isinstance(c._logical_expr, LiteralExpr): raise ValueError(f"Cannot group by literal value: {c}") self._by.append(c) else: raise TypeError( f"Group by expressions must be string or Column, got {type(c)}" ) self._by_exprs = [c._logical_expr for c in self._by] ``` ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with one row per group and columns for group keys and aggregated values Raises: - `ValueError` – If arguments are not Column expressions or a dictionary - `ValueError` – If dictionary values are not valid aggregate function names Count employees by department ``` # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Multiple aggregations ``` # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Dictionary style aggregations ``` # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Args: *exprs: Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: DataFrame: A new DataFrame with one row per group and columns for group keys and aggregated values Raises: ValueError: If arguments are not Column expressions or a dictionary ValueError: If dictionary values are not valid aggregate function names Example: Count employees by department ```python # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Example: Multiple aggregations ```python # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Example: Dictionary style aggregations ```python # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` """ self._validate_agg_exprs(*exprs) if len(exprs) == 1 and isinstance(exprs[0], dict): agg_dict = exprs[0] return self.agg(*self._process_agg_dict(agg_dict)) agg_exprs = self._process_agg_exprs(exprs) return self._df._from_logical_plan( Aggregate.from_session_state(self._df._logical_plan, self._by_exprs, agg_exprs, self._df._session_state), self._df._session_state, ) ``` ## InvalidExampleCollectionError Bases: `ValidationError` ``` flowchart TD fenic.InvalidExampleCollectionError[InvalidExampleCollectionError] fenic.core.error.ValidationError[ValidationError] fenic.core.error.FenicError[FenicError] fenic.core.error.ValidationError --> fenic.InvalidExampleCollectionError fenic.core.error.FenicError --> fenic.core.error.ValidationError click fenic.InvalidExampleCollectionError href "" "fenic.InvalidExampleCollectionError" click fenic.core.error.ValidationError href "" "fenic.core.error.ValidationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Exception raised when a semantic example collection is invalid. ## JoinExample Bases: `BaseModel` ``` flowchart TD fenic.JoinExample[JoinExample] click fenic.JoinExample href "" "fenic.JoinExample" ``` A single semantic example for semantic join operations. Join examples demonstrate the evaluation of two input variables across different datasets against a specific condition, used in a semantic.join operation. ## JoinExampleCollection ``` JoinExampleCollection(examples: List[JoinExample] = None) ``` Bases: `BaseExampleCollection[JoinExample]` ``` flowchart TD fenic.JoinExampleCollection[JoinExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.JoinExampleCollection click fenic.JoinExampleCollection href "" "fenic.JoinExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of comparison examples for semantic join operations. Stores examples showing which pairs of values should be considered matches for joining data. Each example contains a left value, right value, and boolean output indicating whether they match. Initialize a collection of semantic join examples. Parameters: - **`examples`** (`List[JoinExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[JoinExample] = None): """Initialize a collection of semantic join examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: JoinExample) -> JoinExampleCollection ``` Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Parameters: - **`example`** (`JoinExample`) – The JoinExample to add. Returns: - `JoinExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: JoinExample) -> JoinExampleCollection: """Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Args: example: The JoinExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, JoinExample): raise InvalidExampleCollectionError( f"Expected example of type {JoinExample.__name__}, got {type(example).__name__}" ) # Convert to dict format for validation example_dict = { LEFT_ON_KEY: example.left_on, RIGHT_ON_KEY: example.right_on } example_num = len(self.examples) + 1 self._type_validator.process_example(example_dict, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> JoinExampleCollection ``` Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> JoinExampleCollection: """Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns.""" collection = cls() required_columns = [ LEFT_ON_KEY, RIGHT_ON_KEY, EXAMPLE_OUTPUT_KEY, ] for col in required_columns: if col not in df.columns: raise InvalidExampleCollectionError( f"Join Examples DataFrame missing required '{col}' column" ) for row in df.iter_rows(named=True): for col in required_columns: if row[col] is None: raise InvalidExampleCollectionError( f"Join Examples DataFrame contains null values in '{col}' column" ) example = JoinExample( left_on=row[LEFT_ON_KEY], right_on=row[RIGHT_ON_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## KeyPoints Bases: `BaseModel` ``` flowchart TD fenic.KeyPoints[KeyPoints] click fenic.KeyPoints href "" "fenic.KeyPoints" ``` Summary as a concise bulleted list. Each bullet should capture a distinct and essential idea, with a maximum number of points specified. Attributes: - **`max_points`** (`int`) – The maximum number of key points to include in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of key points. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of key points. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of key points.""" return self.max_points * 75 ``` ## LLMResponseCacheConfig Bases: `BaseModel` ``` flowchart TD fenic.LLMResponseCacheConfig[LLMResponseCacheConfig] click fenic.LLMResponseCacheConfig href "" "fenic.LLMResponseCacheConfig" ``` Configuration for LLM response caching. LLM response caching stores the results of language model API calls to reduce costs and improve performance for repeated queries. This is distinct from DataFrame caching (the `.cache()` operator). Attributes: - **`enabled`** – Whether caching is enabled (default: True). - **`backend`** (`CacheBackend`) – Cache backend to use (default: LOCAL). - **`ttl`** (`str`) – Time-to-live duration string (default: "1h"). Format: where unit is s/m/h/d. Examples: "30s", "15m", "2h", "7d". Maximum: 30 days, Minimum: 1 second. - **`max_size_mb`** (`int`) – Maximum cache size in MB before LRU eviction (default: 128MB). - **`namespace`** (`str`) – Cache namespace for isolation (default: "default"). Example Basic configuration within SemanticConfig: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt": OpenAILanguageModel(model_name="gpt-4o-mini", rpm=100, tpm=1000) }, llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="1h", max_size_mb=1000, ) ) ) ``` Custom TTL and larger cache: ``` llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="7d", # 7 days max_size_mb=5000, ) ``` Disabled caching: ``` llm_response_cache=LLMResponseCacheConfig(enabled=False) ``` Methods: - **`ttl_seconds`** – Convert TTL string to seconds. - **`validate_ttl`** – Validate TTL duration string format. ### ttl_seconds ``` ttl_seconds() -> int ``` Convert TTL string to seconds. Returns: - `int` – TTL duration in seconds. Raises: - `ValueError` – If TTL format is invalid. Source code in `src/fenic/api/session/config.py` ``` def ttl_seconds(self) -> int: """Convert TTL string to seconds. Returns: TTL duration in seconds. Raises: ValueError: If TTL format is invalid. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, self.ttl.lower()) if not match: raise ValueError(f"Invalid TTL format: '{self.ttl}'") value, unit = match.groups() value = int(value) multipliers = {"s": 1, "m": 60, "h": 3600, "d": 86400} return value * multipliers[unit] ``` ### validate_ttl ``` validate_ttl(v: str) -> str ``` Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Parameters: - **`v`** (`str`) – TTL duration string to validate. Returns: - `str` – The validated TTL string. Raises: - `ValueError` – If format is invalid or value is out of range. Source code in `src/fenic/api/session/config.py` ``` @field_validator("ttl") @classmethod def validate_ttl(cls, v: str) -> str: """Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Args: v: TTL duration string to validate. Returns: The validated TTL string. Raises: ValueError: If format is invalid or value is out of range. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, v.lower()) if not match: raise ValueError( f"Invalid TTL format: '{v}'. " "Expected: where unit is s/m/h/d. " "Examples: '30m', '2h', '1d'" ) value, unit = match.groups() value = int(value) # Validate ranges if unit == "s" and value < 1: raise ValueError("TTL must be at least 1 second") if unit == "h" and value > 720: # 30 days raise ValueError("TTL cannot exceed 720 hours (30 days)") if unit == "d" and value > 30: raise ValueError("TTL cannot exceed 30 days") return v ``` ## LMMetrics ``` LMMetrics(num_uncached_input_tokens: int = 0, num_cached_input_tokens: int = 0, num_output_tokens: int = 0, cost: float = 0.0, num_requests: int = 0, num_reserved_output_tokens: int = 0) ``` Tracks language model usage metrics including token counts and costs. Attributes: - **`num_uncached_input_tokens`** (`int`) – Number of uncached tokens in the prompt/input. - **`num_cached_input_tokens`** (`int`) – Number of cached tokens in the prompt/input. - **`num_output_tokens`** (`int`) – Number of tokens in the completion/output (actual usage). - **`cost`** (`float`) – Total cost in USD for the LM API call. - **`num_requests`** (`int`) – Total number of LM API requests made. - **`num_reserved_output_tokens`** (`int`) – Output tokens debited from the TPM bucket at reservation time. Compare against num_output_tokens to measure reservation efficiency (actual / reserved β†’ 1 is tight). ## Lineage ``` Lineage(lineage: BaseLineage) ``` Query interface for tracing data lineage through a query plan. This class allows you to navigate through the query plan both forwards and backwards, tracing how specific rows are transformed through each operation. Example ``` # Create a lineage query starting from the root query = LineageQuery(lineage, session.execution) # Or start from a specific source query.start_from_source("my_table") # Trace rows backwards through a transformation result = query.backward(["uuid1", "uuid2"]) # Trace rows forward to see their outputs result = query.forward(["uuid3", "uuid4"]) ``` Initialize a Lineage instance. Parameters: - **`lineage`** (`BaseLineage`) – The underlying lineage implementation. Methods: - **`backwards`** – Trace rows backwards to see which input rows produced them. - **`forwards`** – Trace rows forward to see how they are transformed by the next operation. - **`get_result_df`** – Get the result of the query as a Polars DataFrame. - **`get_source_df`** – Get a query source by name as a Polars DataFrame. - **`get_source_names`** – Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. - **`show`** – Print the operator tree of the query. - **`skip_backwards`** – [Not Implemented] Trace rows backwards through multiple operations at once. - **`skip_forwards`** – [Not Implemented] Trace rows forward through multiple operations at once. - **`start_from_source`** – Set the current position to a specific source in the query plan. Source code in `src/fenic/api/lineage.py` ``` def __init__(self, lineage: BaseLineage): """Initialize a Lineage instance. Args: lineage: The underlying lineage implementation. """ self.lineage = lineage ``` ### backwards ``` backwards(ids: List[str], branch_side: Optional[BranchSide] = None) -> pl.DataFrame ``` Trace rows backwards to see which input rows produced them. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back - **`branch_side`** (`Optional[BranchSide]`, default: `None` ) – For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: - `DataFrame` – DataFrame containing the source rows that produced the specified outputs Raises: - `ValueError` – If invalid ids format or incorrect branch_side specification Example ``` # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def backwards( self, ids: List[str], branch_side: Optional[BranchSide] = None ) -> pl.DataFrame: """Trace rows backwards to see which input rows produced them. Args: ids: List of UUIDs identifying the rows to trace back branch_side: For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: DataFrame containing the source rows that produced the specified outputs Raises: ValueError: If invalid ids format or incorrect branch_side specification Example: ```python # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` """ return self.lineage.backwards(ids, branch_side) ``` ### forwards ``` forwards(row_ids: List[str]) -> pl.DataFrame ``` Trace rows forward to see how they are transformed by the next operation. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the transformed rows in the next operation Raises: - `ValueError` – If at root node or if row_ids format is invalid Example ``` # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def forwards(self, row_ids: List[str]) -> pl.DataFrame: """Trace rows forward to see how they are transformed by the next operation. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the transformed rows in the next operation Raises: ValueError: If at root node or if row_ids format is invalid Example: ```python # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` """ return self.lineage.forwards(row_ids) ``` ### get_result_df ``` get_result_df() -> pl.DataFrame ``` Get the result of the query as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` def get_result_df(self) -> pl.DataFrame: """Get the result of the query as a Polars DataFrame.""" return self.lineage.get_result_df() ``` ### get_source_df ``` get_source_df(source_name: str) -> pl.DataFrame ``` Get a query source by name as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_df(self, source_name: str) -> pl.DataFrame: """Get a query source by name as a Polars DataFrame.""" return self.lineage.get_source_df(source_name) ``` ### get_source_names ``` get_source_names() -> List[str] ``` Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_names(self) -> List[str]: """Get the names of all sources in the query plan. Used to determine where to start the lineage traversal.""" return self.lineage.get_source_names() ``` ### show ``` show() -> None ``` Print the operator tree of the query. Source code in `src/fenic/api/lineage.py` ``` def show(self) -> None: """Print the operator tree of the query.""" print(self.lineage.stringify_graph()) ``` ### skip_backwards ``` skip_backwards(ids: List[str]) -> Dict[str, pl.DataFrame] ``` [Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back Returns: - `Dict[str, DataFrame]` – Dictionary mapping operation names to their source DataFrames Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_backwards(self, ids: List[str]) -> Dict[str, pl.DataFrame]: """[Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: ids: List of UUIDs identifying the rows to trace back Returns: Dictionary mapping operation names to their source DataFrames Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip backwards not yet implemented") ``` ### skip_forwards ``` skip_forwards(row_ids: List[str]) -> pl.DataFrame ``` [Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the final transformed rows Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_forwards(self, row_ids: List[str]) -> pl.DataFrame: """[Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the final transformed rows Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip forwards not yet implemented") ``` ### start_from_source ``` start_from_source(source_name: str) -> None ``` Set the current position to a specific source in the query plan. Parameters: - **`source_name`** (`str`) – Name of the source table to start from Example ``` query.start_from_source("customers") # Now you can trace forward from the customers table ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def start_from_source(self, source_name: str) -> None: """Set the current position to a specific source in the query plan. Args: source_name: Name of the source table to start from Example: ```python query.start_from_source("customers") # Now you can trace forward from the customers table ``` """ self.lineage.start_from_source(source_name) ``` ## MapExample Bases: `BaseModel` ``` flowchart TD fenic.MapExample[MapExample] click fenic.MapExample href "" "fenic.MapExample" ``` A single semantic example for semantic mapping operations. Map examples demonstrate the transformation of input variables to a specific output string or structured model used in a semantic.map operation. ## MapExampleCollection ``` MapExampleCollection(examples: List[MapExample] = None) ``` Bases: `BaseExampleCollection[MapExample]` ``` flowchart TD fenic.MapExampleCollection[MapExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.MapExampleCollection click fenic.MapExampleCollection href "" "fenic.MapExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-output examples for semantic map operations. Stores examples that demonstrate how input data should be transformed into output text or structured data. Each example shows the expected output for a given set of input fields. Initialize a collection of semantic map examples. Parameters: - **`examples`** (`List[MapExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with output and input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[MapExample] = None): """Initialize a collection of semantic map examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: MapExample) -> MapExampleCollection ``` Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Parameters: - **`example`** (`MapExample`) – The MapExample to add. Returns: - `MapExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: MapExample) -> MapExampleCollection: """Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Args: example: The MapExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, MapExample): raise InvalidExampleCollectionError( f"Expected example of type {MapExample.__name__}, got {type(example).__name__}" ) # Validate output type consistency self._validate_single_example_output_type(example) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> MapExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> MapExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column.""" collection = cls() if EXAMPLE_OUTPUT_KEY not in df.columns: raise ValueError( f"Map Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise ValueError( "Map Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): input_dict = {col: row[col] for col in input_cols} example = MapExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## ModelAlias Bases: `BaseModel` ``` flowchart TD fenic.ModelAlias[ModelAlias] click fenic.ModelAlias href "" "fenic.ModelAlias" ``` A combination of a model name and a required profile for that model. Model aliases are used to select a specific model to use in a semantic operation. Both the model name and profile must be specified when creating a ModelAlias. Attributes: - **`name`** (`str`) – The name of the model. - **`profile`** (`str`) – The name of a profile configuration to use for the model. Example ``` model_alias = ModelAlias(name="o4-mini", profile="low") ``` ## OpenAIEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.OpenAIEmbeddingModel[OpenAIEmbeddingModel] click fenic.OpenAIEmbeddingModel href "" "fenic.OpenAIEmbeddingModel" ``` Configuration for OpenAI embedding models. This class defines the configuration settings for OpenAI embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAIEmbeddingModelName`) – The name of the OpenAI embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. Example Configuring an OpenAI embedding model with rate limits: ``` config = OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) ``` ## OpenAILanguageModel Bases: `BaseModel` ``` flowchart TD fenic.OpenAILanguageModel[OpenAILanguageModel] click fenic.OpenAILanguageModel href "" "fenic.OpenAILanguageModel" ``` Configuration for OpenAI language models. This class defines the configuration settings for OpenAI language models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAILanguageModelName`) – The name of the OpenAI model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Note When using an o-series or gpt5 reasoning model without specifying a reasoning effort in a Profile, the `reasoning_effort` will default to `low` (for o-series models) or `minimal` (for gpt5 models). Example Configuring an OpenAI language model with rate limits: ``` config = OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) ``` Configuring an OpenAI model with profiles: ``` config = OpenAILanguageModel( model_name="o4-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile(reasoning_effort="high"), }, default_profile="fast", ) ``` Using a profile in a semantic operation: ``` config = SemanticConfig( language_models={ "o4": OpenAILanguageModel( model_name="o4-mini", rpm=1_000, tpm=1_000_000, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ) }, default_language_model="o4", ) # Will use the default "fast" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias="o4", ) # Will use the "thorough" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="o4", profile="thorough"), ) ``` Classes: - **`Profile`** – OpenAI-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.OpenAILanguageModel.Profile[Profile] click fenic.OpenAILanguageModel.Profile href "" "fenic.OpenAILanguageModel.Profile" ``` OpenAI-specific profile configurations. This class defines profile configurations for OpenAI models, allowing a user to reference the same underlying model in semantic operations with different settings. Attributes: - **`reasoning_effort`** (`Optional[ReasoningEffort]`) – Provide a reasoning effort. Only for gpt5 and o-series models. Valid values: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'. - For gpt-5.6 models: supports 'xhigh' and 'max' - For gpt-5.5 models: defaults to 'medium', supports 'xhigh' - For gpt-5.4 models: defaults to 'none' (disabled reasoning), supports 'xhigh' - For gpt-5.1 and gpt-5.2 models: defaults to 'none' (disabled reasoning), does NOT support 'minimal' or 'xhigh' - For gpt-5 models: defaults to 'minimal', does NOT support 'none' - For o-series models: defaults to 'low', does NOT support 'none' or 'minimal' - **`verbosity`** (`Optional[Verbosity]`) – Provide a verbosity level. Only for gpt5/gpt5.1 models. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note When using an o-series or gpt5 reasoning model with reasoning enabled, the `temperature` cannot be customized. For gpt-5.1 models with reasoning_effort='none', temperature CAN be customized. Example Configuring a profile with medium reasoning effort: ``` profile = OpenAILanguageModel.Profile(reasoning_effort="medium") ``` ## OpenRouterLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.OpenRouterLanguageModel[OpenRouterLanguageModel] click fenic.OpenRouterLanguageModel href "" "fenic.OpenRouterLanguageModel" ``` Configuration for OpenRouter language models. This class defines the configuration settings for OpenRouter language models, including model selection and rate limiting parameters. When fetching available models from OpenRouter, results will be filtered to only include models from providers that are not in the user’s ignored providers list and are either in the user’s allowed providers list (if configured) or from any provider (if no allowed providers are specified). Attributes: - **`model_name`** (`str`) – `{family}/{model}` identifier (e.g., `anthropic/claude-3-5-sonnet`). - **`profiles`** (`Optional[dict[str, Profile]]`) – Mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The key in `profiles` to select by default. - **`structured_output_strategy`** (`Optional[StructuredOutputStrategy]`) – The strategy to use for structured output if a model supports both tool calling and structured outputs. - `prefer_tools`: prefer using tools over response format. - `prefer_response_format`: prefer using response format over tools. Requirements - Set `OPENROUTER_API_KEY` in your environment. Example: ``` OpenRouterLanguageModel( model_name="openai/gpt-oss-20b", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="price" # Routes to the cheapest available provider ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="anthropic/claude-sonnet-4", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( only=[ "Anthropic" ] # ensures the request will only be routed to Anthropic and not AWS Bedrock or Google Vertex ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="qwen/qwen3-next-80b-a3b-instruct", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="throughput", # routes to the provider with the highest overall throughput data_collection="deny" # eliminates providers that retain prompt data (would only route to DeepInfra/AtlasCloud, in this example) # Eliminate providers that offer an fp8 quantized version of the model, only allowing bf16. # Note that many providers have an `unknown` quantization, so you may be excluding more providers than you expect. quantizations=["bf16"] ) ) } ) ``` Classes: - **`Profile`** – Profile configurations for OpenRouter language models. - **`Provider`** – Provider routing configuration for OpenRouter language models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.OpenRouterLanguageModel.Profile[Profile] click fenic.OpenRouterLanguageModel.Profile href "" "fenic.OpenRouterLanguageModel.Profile" ``` Profile configurations for OpenRouter language models. Attributes: - **`models`** (`Optional[list[str]]`) – A list of fallback models to use if the primary model is unavailable. ([OpenRouter Documentation](https://openrouter.ai/docs/features/model-routing#the-models-parameter)). - **`provider`** (`Optional[Provider]`) – Provider routing preferences (include/exclude specific providers, set provider ranking method preference) ([OpenRouter Documentation](https://openrouter.ai/docs/features/provider-routing)). - **`reasoning_effort`** (`Optional[OpenRouterReasoningEffort]`) – OpenRouter reasoning effort configuration (none, minimal, low, medium, high, xhigh, max). If the model does support reasoning, but not `reasoning_effort`, a `reasoning_max_tokens` will be calculated that is roughly equivalent as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#reasoning-effort-level)) - **`reasoning_max_tokens`** (`Optional[int]`) – Supported by Anthropic, Gemini, etc., sets a token budget for reasoning If the model does support reasoning, but not `reasoning_max_tokens`, a `reasoning_effort_ will be automatically calculated based on`reasoning_max_tokens` as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#max-tokens-for-reasoning)) - **`parsing_engine`** (`Optional[ParsingEngine]`) – The parsing engine to use for processing PDF files. By default, the model's native parsing engine will be used. If the model doesn't support PDF processing and the parsing engine is not provided, an error will be raised. Note: 'mistral-ocr' incurs additional costs. ([OpenRouter Documentation](https://openrouter.ai/docs/features/multimodal/pdfs)) ### Provider Bases: `BaseModel` ``` flowchart TD fenic.OpenRouterLanguageModel.Provider[Provider] click fenic.OpenRouterLanguageModel.Provider href "" "fenic.OpenRouterLanguageModel.Provider" ``` Provider routing configuration for OpenRouter language models. [Provider Routing Documentation](https://openrouter.ai/docs/features/provider-routing) Attributes: - **`order`** (`Optional[list[str]]`) – List of providers to try in order (e.g. ['Anthropic', 'Amazon Bedrock']). - **`sort`** (`Optional[ProviderSort]`) – Provider routing preference (e.g. 'price', 'throughput', 'latency'). "price" will route to the cheapest available provider first, progressing through the list of providers in order of price. "throughput" will route to the provider with the highest overall recent throughput, progressing through the list of providers in order of throughput. "latency" will route to the provider with the lowest overall recent latency, progressing through the list of providers in order of latency. - **`quantizations`** (`Optional[list[ModelQuantization]]`) – Allowed quantizations. Note: many providers report `unknown`. - **`data_collection`** (`Optional[DataCollection]`) – Data collection preference. `allow`: allows the use of providers which store prompt data non-transiently and may train on it. `deny`: use only providers which do not collect/store prompt data. - **`only`** (`Optional[list[str]]`) – Only include these providers when performing provider routing. - **`exclude`** (`Optional[list[str]]`) – Exclude these providers when performing provider routing. - **`max_prompt_price`** (`Optional[float]`) – Maximum prompt price ($USD per 1M tokens). - **`max_completion_price`** (`Optional[float]`) – Maximum completion price ($USD per 1M tokens). ## OperatorMetrics ``` OperatorMetrics(operator_id: str, num_output_rows: int = 0, execution_time_ms: float = 0.0, lm_metrics: LMMetrics = LMMetrics(), rm_metrics: RMMetrics = RMMetrics()) ``` Metrics for a single operator in the query execution plan. Attributes: - **`operator_id`** (`str`) – Unique identifier for the operator - **`num_output_rows`** (`int`) – Number of rows output by this operator - **`execution_time_ms`** (`float`) – Execution time in milliseconds - **`lm_metrics`** (`LMMetrics`) – Language model usage metrics for this operator ## Paragraph Bases: `BaseModel` ``` flowchart TD fenic.Paragraph[Paragraph] click fenic.Paragraph href "" "fenic.Paragraph" ``` Summary as a cohesive narrative. The summary should flow naturally and not exceed a specified maximum word count. Attributes: - **`max_words`** (`int`) – The maximum number of words allowed in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of words. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of words. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of words.""" return int(self.max_words * 1.5) ``` ## PredicateExample Bases: `BaseModel` ``` flowchart TD fenic.PredicateExample[PredicateExample] click fenic.PredicateExample href "" "fenic.PredicateExample" ``` A single semantic example for semantic predicate operations. Predicate examples demonstrate the evaluation of input variables against a specific condition, used in a semantic.predicate operation. ## PredicateExampleCollection ``` PredicateExampleCollection(examples: List[PredicateExample] = None) ``` Bases: `BaseExampleCollection[PredicateExample]` ``` flowchart TD fenic.PredicateExampleCollection[PredicateExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.PredicateExampleCollection click fenic.PredicateExampleCollection href "" "fenic.PredicateExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-to-boolean examples for predicate operations. Stores examples showing which inputs should evaluate to True or False based on some condition. Each example contains input fields and a boolean output indicating whether the condition holds. Initialize a collection of semantic predicate examples. Parameters: - **`examples`** (`List[PredicateExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[PredicateExample] = None): """Initialize a collection of semantic predicate examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: PredicateExample) -> PredicateExampleCollection ``` Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Parameters: - **`example`** (`PredicateExample`) – The PredicateExample to add. Returns: - `PredicateExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: PredicateExample) -> PredicateExampleCollection: """Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Args: example: The PredicateExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, PredicateExample): raise InvalidExampleCollectionError( f"Expected example of type {PredicateExample.__name__}, got {type(example).__name__}" ) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> PredicateExampleCollection ``` Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> PredicateExampleCollection: """Create collection from a Polars DataFrame.""" collection = cls() # Validate output column exists if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise InvalidExampleCollectionError( "Predicate Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) input_dict = {col: row[col] for col in input_cols if row[col] is not None} example = PredicateExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## QueryMetrics ``` QueryMetrics(execution_id: str, session_id: str, execution_time_ms: float = 0.0, num_output_rows: int = 0, total_lm_metrics: LMMetrics = LMMetrics(), total_rm_metrics: RMMetrics = RMMetrics(), end_ts: datetime = datetime.now(), _operator_metrics: Dict[str, OperatorMetrics] = dict(), _plan_repr: PhysicalPlanRepr = (lambda: PhysicalPlanRepr(operator_id='empty'))()) ``` Comprehensive metrics for an executed query. Includes overall statistics and detailed metrics for each operator in the execution plan. Attributes: - **`execution_id`** (`str`) – Unique identifier for this query execution - **`session_id`** (`str`) – Identifier for the session this query belongs to - **`execution_time_ms`** (`float`) – Total query execution time in milliseconds - **`num_output_rows`** (`int`) – Total number of rows returned by the query - **`total_lm_metrics`** (`LMMetrics`) – Aggregated language model metrics across all operators - **`end_ts`** (`datetime`) – Timestamp when query execution completed Methods: - **`get_execution_plan_details`** – Generate a formatted execution plan with detailed metrics. - **`get_summary`** – Summarize the query metrics in a single line. - **`to_dict`** – Convert QueryMetrics to a dictionary for table storage. ### start_ts ``` start_ts: datetime ``` Calculate start timestamp from end timestamp and execution time. ### get_execution_plan_details ``` get_execution_plan_details() -> str ``` Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: - **`str`** ( `str` ) – A formatted string showing the execution plan with metrics. Source code in `src/fenic/core/metrics.py` ``` def get_execution_plan_details(self) -> str: """Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: str: A formatted string showing the execution plan with metrics. """ def _format_node(node: PhysicalPlanRepr, indent: int = 1) -> str: op = self._operator_metrics[node.operator_id] indent_str = " " * indent details = [ f"{indent_str}{op.operator_id}", f"{indent_str} Output Rows: {op.num_output_rows:,}", f"{indent_str} Execution Time: {op.execution_time_ms:.2f}ms", ] if op.lm_metrics.cost > 0: details.extend( [ f"{indent_str} Language Model Usage: {op.lm_metrics.num_uncached_input_tokens:,} input tokens, {op.lm_metrics.num_cached_input_tokens:,} cached input tokens, {op.lm_metrics.num_output_tokens:,} output tokens ({op.lm_metrics.num_reserved_output_tokens:,} reserved)", f"{indent_str} Language Model Cost: ${op.lm_metrics.cost:.6f}", ] ) if op.rm_metrics.cost > 0: details.extend( [ f"{indent_str} Embedding Model Usage: {op.rm_metrics.num_input_tokens:,} input tokens", f"{indent_str} Embedding Model Cost: ${op.rm_metrics.cost:.6f}", ] ) return ( "\n".join(details) + "\n" + "".join(_format_node(child, indent + 1) for child in node.children) ) return f"Execution Plan\n{_format_node(self._plan_repr)}" ``` ### get_summary ``` get_summary() -> str ``` Summarize the query metrics in a single line. Returns: - **`str`** ( `str` ) – A concise summary of execution time, row count, and LM cost. Source code in `src/fenic/core/metrics.py` ``` def get_summary(self) -> str: """Summarize the query metrics in a single line. Returns: str: A concise summary of execution time, row count, and LM cost. """ return ( f"Query executed in {self.execution_time_ms:.2f}ms, " f"returned {self.num_output_rows:,} rows, " f"language model cost: ${self.total_lm_metrics.cost:.6f}, " f"embedding model cost: ${self.total_rm_metrics.cost:.6f}" ) ``` ### to_dict ``` to_dict() -> Dict[str, Any] ``` Convert QueryMetrics to a dictionary for table storage. Returns: - `Dict[str, Any]` – Dict containing all metrics fields suitable for database storage. Source code in `src/fenic/core/metrics.py` ``` def to_dict(self) -> Dict[str, Any]: """Convert QueryMetrics to a dictionary for table storage. Returns: Dict containing all metrics fields suitable for database storage. """ return { "execution_id": self.execution_id, "session_id": self.session_id, "execution_time_ms": self.execution_time_ms, "num_output_rows": self.num_output_rows, "start_ts": self.start_ts, "end_ts": self.end_ts, "total_lm_cost": self.total_lm_metrics.cost, "total_lm_uncached_input_tokens": self.total_lm_metrics.num_uncached_input_tokens, "total_lm_cached_input_tokens": self.total_lm_metrics.num_cached_input_tokens, "total_lm_output_tokens": self.total_lm_metrics.num_output_tokens, "total_lm_requests": self.total_lm_metrics.num_requests, "total_rm_cost": self.total_rm_metrics.cost, "total_rm_input_tokens": self.total_rm_metrics.num_input_tokens, "total_rm_requests": self.total_rm_metrics.num_requests, } ``` ## QueryResult ``` QueryResult(data: DataLike, metrics: QueryMetrics) ``` Container for query execution results and associated metadata. This dataclass bundles together the materialized data from a query execution along with metrics about the execution process. It provides a unified interface for accessing both the computed results and performance information. Attributes: - **`data`** (`DataLike`) – The materialized query results in the requested format. Can be any of the supported data types (Polars/Pandas DataFrame, Arrow Table, or Python dict/list structures). - **`metrics`** (`QueryMetrics`) – Execution metadata including timing information, memory usage, rows processed, and other performance metrics collected during query execution. Access query results and metrics ``` # Execute query and get results with metrics result = df.filter(col("age") > 25).collect("pandas") pandas_df = result.data # Access the Pandas DataFrame print(result.metrics.execution_time) # Access execution metrics print(result.metrics.rows_processed) # Access row count ``` Work with different data formats ``` # Get results in different formats polars_result = df.collect("polars") arrow_result = df.collect("arrow") dict_result = df.collect("pydict") # All contain the same data, different formats print(type(polars_result.data)) # print(type(arrow_result.data)) # print(type(dict_result.data)) # ``` Note The actual type of the `data` attribute depends on the format requested during collection. Use type checking or isinstance() if you need to handle the data differently based on its format. ## RMMetrics ``` RMMetrics(num_input_tokens: int = 0, num_requests: int = 0, cost: float = 0.0) ``` Tracks embedding model usage metrics including token counts and costs. Attributes: - **`num_input_tokens`** (`int`) – Number of tokens to embed - **`cost`** (`float`) – Total cost in USD to embed the tokens ## Schema Represents the schema of a DataFrame. A Schema defines the structure of a DataFrame by specifying an ordered collection of column fields. Each column field defines the name and data type of a column in the DataFrame. Attributes: - **`column_fields`** (`List[ColumnField]`) – An ordered list of ColumnField objects that define the structure of the DataFrame. Methods: - **`column_names`** – Get a list of all column names in the schema. ### column_names ``` column_names() -> List[str] ``` Get a list of all column names in the schema. Returns: - `List[str]` – A list of strings containing the names of all columns in the schema. Source code in `src/fenic/core/types/schema.py` ``` def column_names(self) -> List[str]: """Get a list of all column names in the schema. Returns: A list of strings containing the names of all columns in the schema. """ return [field.name for field in self.column_fields] ``` ## SemanticConfig Bases: `BaseModel` ``` flowchart TD fenic.SemanticConfig[SemanticConfig] click fenic.SemanticConfig href "" "fenic.SemanticConfig" ``` Configuration for semantic language and embedding models. This class defines the configuration for both language models and optional embedding models used in semantic operations. It ensures that all configured models are valid and supported by their respective providers. Attributes: - **`language_models`** (`Optional[dict[str, LanguageModel]]`) – Mapping of model aliases to language model configurations. - **`default_language_model`** (`Optional[str]`) – The alias of the default language model to use for semantic operations. Not required if only one language model is configured. - **`embedding_models`** (`Optional[dict[str, EmbeddingModel]]`) – Optional mapping of model aliases to embedding model configurations. - **`default_embedding_model`** (`Optional[str]`) – The alias of the default embedding model to use for semantic operations. Note The embedding model is optional and only required for operations that need semantic search or embedding capabilities. Example Configuring semantic models with a single language model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) } ) ``` Configuring semantic models with multiple language models and an embedding model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), "gemini": GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ) ``` Configuring models with profiles: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4o-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, profiles={ "fast": AnthropicLanguageModel.Profile(effort="low"), "thorough": AnthropicLanguageModel.Profile(effort="high"), }, default_profile="fast", ), }, default_language_model="gpt4", ) ``` Methods: - **`model_post_init`** – Post initialization hook to set defaults. - **`validate_models`** – Validates that the selected models are supported by the system. ### model_post_init ``` model_post_init(__context) -> None ``` Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. Source code in `src/fenic/api/session/config.py` ``` def model_post_init(self, __context) -> None: """Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. """ if self.language_models: # Set default language model if not set and only one model exists if self.default_language_model is None and len(self.language_models) == 1: self.default_language_model = list(self.language_models.keys())[0] # Auto-create profiles for Google models that support thinking_level for model_config in self.language_models.values(): if isinstance(model_config, (GoogleDeveloperLanguageModel, GoogleVertexLanguageModel)): model_provider = _get_model_provider_for_model_config(model_config) model_params = model_catalog.get_completion_model_parameters( model_provider, model_config.model_name ) if model_params and model_params.supported_thinking_levels and model_config.profiles is None: # Auto-create profiles for each supported thinking level if isinstance(model_config, GoogleDeveloperLanguageModel): model_config.profiles = { level: GoogleDeveloperLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } else: model_config.profiles = { level: GoogleVertexLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } model_config.default_profile = "low" # Set default profile for each model if not set and only one profile exists for model_config in self.language_models.values(): if model_config.profiles is not None: profile_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(profile_names) == 1: model_config.default_profile = profile_names[0] # Set default embedding model if not set and only one model exists if self.embedding_models: if self.default_embedding_model is None and len(self.embedding_models) == 1: self.default_embedding_model = list(self.embedding_models.keys())[0] # Set default profile for each model if not set and only one preset exists for model_config in self.embedding_models.values(): if ( hasattr(model_config, "profiles") and model_config.profiles is not None ): preset_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(preset_names) == 1: model_config.default_profile = preset_names[0] ``` ### validate_models ``` validate_models() -> SemanticConfig ``` Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: - `SemanticConfig` – The validated SemanticConfig instance. Raises: - `ConfigurationError` – If any of the models are not supported. Source code in `src/fenic/api/session/config.py` ``` @model_validator(mode="after") def validate_models(self) -> SemanticConfig: """Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: The validated SemanticConfig instance. Raises: ConfigurationError: If any of the models are not supported. """ # Skip validation if no models configured (embedding-only or empty config) if not self.language_models and not self.embedding_models: return self # Validate language models if provided if self.language_models: available_language_model_aliases = list(self.language_models.keys()) if self.default_language_model is None and len(self.language_models) > 1: raise ConfigurationError( f"default_language_model is not set, and multiple language models are configured. Please specify one of: {available_language_model_aliases} as a default_language_model." ) if ( self.default_language_model is not None and self.default_language_model not in self.language_models ): raise ConfigurationError( f"default_language_model {self.default_language_model} is not in configured map of language models. Available models: {available_language_model_aliases} ." ) for model_alias, language_model in self.language_models.items(): language_model_name = language_model.model_name language_model_provider = _get_model_provider_for_model_config( language_model ) completion_model_params = model_catalog.get_completion_model_parameters( language_model_provider, language_model_name ) if completion_model_params is None: raise ConfigurationError( model_catalog.generate_unsupported_completion_model_error_message( language_model_provider, language_model_name ) ) if language_model.profiles is not None: if not completion_model_params.supports_profiles: raise ConfigurationError( f"Model '{model_alias}' does not support parameter profiles. Please remove the Profile configuration." ) profile_names = list(language_model.profiles.keys()) if ( language_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( language_model.default_profile is not None and language_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {language_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in language_model.profiles.items(): _validate_language_profile( language_model, model_alias, completion_model_params, profile, profile_alias, ) if self.embedding_models is not None: available_embedding_model_aliases = list(self.embedding_models.keys()) if self.default_embedding_model is None and len(self.embedding_models) > 1: raise ConfigurationError( f"default_embedding_model is not set, and multiple embedding models are configured. Please specify one of: {available_embedding_model_aliases} as a default_embedding_model." ) if ( self.default_embedding_model is not None and self.default_embedding_model not in self.embedding_models ): raise ConfigurationError( f"default_embedding_model {self.default_embedding_model} is not in configured map of embedding models. Available models: {available_embedding_model_aliases} ." ) for model_alias, embedding_model in self.embedding_models.items(): embedding_model_provider = _get_model_provider_for_model_config( embedding_model ) embedding_model_name = embedding_model.model_name embedding_model_parameters = ( model_catalog.get_embedding_model_parameters( embedding_model_provider, embedding_model_name ) ) if embedding_model_parameters is None: raise ConfigurationError( model_catalog.generate_unsupported_embedding_model_error_message( embedding_model_provider, embedding_model_name ) ) if hasattr(embedding_model, "profiles") and embedding_model.profiles: profile_names = list(embedding_model.profiles.keys()) if ( embedding_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( embedding_model.default_profile is not None and embedding_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {embedding_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in embedding_model.profiles.items(): _validate_embedding_profile( embedding_model_parameters, model_alias, profile_alias, profile, ) return self ``` ## SemanticExtensions ``` SemanticExtensions(df: DataFrame) ``` A namespace for semantic dataframe operators. Initialize semantic extensions. Parameters: - **`df`** (`DataFrame`) – The DataFrame to extend with semantic operations. Methods: - **`join`** – Performs a semantic join between two DataFrames using a natural language predicate. - **`sim_join`** – Performs a semantic similarity join between two DataFrames using embedding expressions. - **`with_cluster_labels`** – Cluster rows using K-means and add cluster metadata columns. Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def __init__(self, df: DataFrame): """Initialize semantic extensions. Args: df: The DataFrame to extend with semantic operations. """ self._df = df ``` ### join ``` join(other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Parameters: - **`other`** (`DataFrame`) – The DataFrame to join with. - **`predicate`** (`str`) – A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. - **`left_on`** (`Column`) – The column from the left DataFrame (self) to use in the join predicate. - **`right_on`** (`Column`) – The column from the right DataFrame (other) to use in the join predicate. - **`strict`** (`bool`, default: `True` ) – If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[JoinExampleCollection]`, default: `None` ) – Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use. If None, uses the default model. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing matched row pairs with all columns from both DataFrames. Basic semantic join ``` # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Semantic join with examples ``` # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def join( self, other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Args: other: The DataFrame to join with. predicate: A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. left_on: The column from the left DataFrame (self) to use in the join predicate. right_on: The column from the right DataFrame (other) to use in the join predicate. strict: If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. examples: Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) model_alias: Optional alias for the language model to use. If None, uses the default model. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: DataFrame: A new DataFrame containing matched row pairs with all columns from both DataFrames. Example: Basic semantic join ```python # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Example: Semantic join with examples ```python # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(other, DataFrame): raise ValidationError(f"other argument must be a DataFrame, got {type(other)}") if not isinstance(predicate, str): raise ValidationError( f"The `predicate` argument to `semantic.join` must be a string, got {type(predicate)}" ) if not isinstance(left_on, Column): raise ValidationError(f"`left_on` argument must be a Column, got {type(left_on)} instead.") if not isinstance(right_on, Column): raise ValidationError(f"`right_on` argument must be a Column, got {type(right_on)} instead.") if examples is not None and not isinstance(examples, JoinExampleCollection): raise ValidationError(f"`examples` argument must be a JoinExampleCollection, got {type(examples)} instead.") if model_alias is not None and not isinstance(model_alias, (str, ModelAlias)): raise ValidationError(f"`model_alias` argument must be a string or ModelAlias, got {type(model_alias)} instead.") # Validate request_timeout request_timeout = validate_timeout(request_timeout) resolved_model_alias = _resolve_model_alias(model_alias) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticJoin.from_session_state( left=self._df._logical_plan, right=other._logical_plan, left_on=left_on._logical_expr, right_on=right_on._logical_expr, jinja_template=predicate, strict=strict, model_alias=resolved_model_alias, examples=examples, request_timeout=request_timeout, session_state=self._df._session_state, ), self._df._session_state, ) ``` ### sim_join ``` sim_join(other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = 'cosine', similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Parameters: - **`other`** (`DataFrame`) – The right-hand DataFrame to join with. - **`left_on`** (`ColumnOrName`) – Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. - **`right_on`** (`ColumnOrName`) – Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. - **`k`** (`int`, default: `1` ) – Number of most similar matches to return per row. - **`similarity_metric`** (`SemanticSimilarityMetric`, default: `'cosine'` ) – Similarity metric to use: "l2", "cosine", or "dot". - **`similarity_score_column`** (`Optional[str]`, default: `None` ) – If set, adds a column with this name containing similarity scores. If None, the scores are omitted. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. - `DataFrame` – The result includes all columns from both input DataFrames and, when `similarity_score_column` - `DataFrame` – is provided, a similarity score column. Join key columns that already exist in the input - `DataFrame` – DataFrames are preserved as normal user columns. Join keys derived from expressions are - `DataFrame` – temporary execution columns and are not included in the result schema. Raises: - `ValidationError` – If `k` is not positive or if the columns are invalid. - `ValidationError` – If `similarity_metric` is not one of "l2", "cosine", "dot" Match queries to FAQ entries ``` # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Link headlines to articles ``` # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Find similar job postings ``` # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def sim_join( self, other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = "cosine", similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note: Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Args: other: The right-hand DataFrame to join with. left_on: Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. right_on: Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. k: Number of most similar matches to return per row. similarity_metric: Similarity metric to use: "l2", "cosine", or "dot". similarity_score_column: If set, adds a column with this name containing similarity scores. If None, the scores are omitted. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. The result includes all columns from both input DataFrames and, when `similarity_score_column` is provided, a similarity score column. Join key columns that already exist in the input DataFrames are preserved as normal user columns. Join keys derived from expressions are temporary execution columns and are not included in the result schema. Raises: ValidationError: If `k` is not positive or if the columns are invalid. ValidationError: If `similarity_metric` is not one of "l2", "cosine", "dot" Example: Match queries to FAQ entries ```python # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Example: Link headlines to articles ```python # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Example: Find similar job postings ```python # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(right_on, ColumnOrName): raise ValidationError( f"The `right_on` argument must be a `Column` or a string representing a column name, " f"but got `{type(right_on).__name__}` instead." ) if not isinstance(other, DataFrame): raise ValidationError( f"The `other` argument to `sim_join()` must be a DataFrame`, but got `{type(other).__name__}`." ) if not (isinstance(k, int) and k > 0): raise ValidationError( f"The parameter `k` must be a positive integer, but received `{k}`." ) args = get_args(SemanticSimilarityMetric) if similarity_metric not in args: raise ValidationError( f"The `similarity_metric` argument must be one of {args}, but got `{similarity_metric}`." ) def _validate_column(column: ColumnOrName, name: str): if column is None: raise ValidationError(f"The `{name}` argument must not be None.") if not isinstance(column, ColumnOrName): raise ValidationError( f"The `{name}` argument must be a `Column` or a string representing a column name, " f"but got `{type(column).__name__}` instead." ) _validate_column(left_on, "left_on") _validate_column(right_on, "right_on") # Validate request_timeout request_timeout = validate_timeout(request_timeout) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticSimilarityJoin.from_session_state( self._df._logical_plan, other._logical_plan, Column._from_col_or_name(left_on)._logical_expr, Column._from_col_or_name(right_on)._logical_expr, k, similarity_metric=similarity_metric, similarity_score_column=similarity_score_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ### with_cluster_labels ``` with_cluster_labels(by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = 'cluster_label', centroid_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Parameters: - **`by`** (`ColumnOrName`) – Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). - **`num_clusters`** (`int`) – Number of clusters to compute (must be > 0). - **`max_iter`** (`int`, default: `300` ) – Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. - **`num_init`** (`int`, default: `1` ) – Number of independent runs of k-means with different centroid seeds. The best result is selected. - **`label_column`** (`str`, default: `'cluster_label'` ) – Name of the output column for cluster IDs. Default is "cluster_label". - **`centroid_column`** (`Optional[str]`, default: `None` ) – If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame with all original columns plus: - `DataFrame` – - ``: integer cluster assignment (0 to num_clusters - 1) - `DataFrame` – - ``: cluster centroid embedding, if specified Basic clustering ``` # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Filter outliers using centroids ``` # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def with_cluster_labels( self, by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = "cluster_label", centroid_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note: Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Args: by: Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). num_clusters: Number of clusters to compute (must be > 0). max_iter: Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. num_init: Number of independent runs of k-means with different centroid seeds. The best result is selected. label_column: Name of the output column for cluster IDs. Default is "cluster_label". centroid_column: If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame with all original columns plus: - ``: integer cluster assignment (0 to num_clusters - 1) - ``: cluster centroid embedding, if specified Example: Basic clustering ```python # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Example: Filter outliers using centroids ```python # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` """ # Validate request_timeout request_timeout = validate_timeout(request_timeout) # Validate num_clusters if not isinstance(num_clusters, int) or num_clusters <= 0: raise ValidationError("`num_clusters` must be a positive integer.") # Validate max_iter if not isinstance(max_iter, int) or max_iter <= 0: raise ValidationError("`max_iter` must be a positive integer.") # Validate num_init if not isinstance(num_init, int) or num_init <= 0: raise ValidationError("`num_init` must be a positive integer.") # Validate clustering target if not isinstance(by, ColumnOrName): raise ValidationError( f"Invalid cluster by: expected a column name (str) or Column object, got {type(by).__name__}." ) # Validate label_column if not isinstance(label_column, str) or not label_column: raise ValidationError("`label_column` must be a non-empty string.") # Validate centroid_column if provided if centroid_column is not None: if not isinstance(centroid_column, str) or not centroid_column: raise ValidationError("`centroid_column` must be a non-empty string if provided.") # Check that the expression isn't a literal by_expr = Column._from_col_or_name(by)._logical_expr if isinstance(by_expr, LiteralExpr): raise ValidationError( f"Invalid cluster by: Cannot cluster by a literal value: {by_expr}." ) return self._df._from_logical_plan( SemanticCluster.from_session_state( self._df._logical_plan, by_expr, num_clusters=num_clusters, max_iter=max_iter, num_init=num_init, label_column=label_column, centroid_column=centroid_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ## Session The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. Create a session with default configuration ``` session = Session.get_or_create(SessionConfig(app_name="my_app")) ``` Create a session with cloud configuration ``` config = SessionConfig( app_name="my_app", cloud=True, api_key="your_api_key" ) session = Session.get_or_create(config) ``` Methods: - **`create_dataframe`** – Create a DataFrame from a variety of Python-native data formats. - **`get_or_create`** – Gets an existing Session or creates a new one with the configured settings. - **`sql`** – Execute a read-only SQL query against one or more DataFrames using named placeholders. - **`stop`** – Stops the session and closes all connections. - **`table`** – Returns the specified table as a DataFrame. - **`view`** – Returns the specified view as a DataFrame. Attributes: - **`catalog`** (`Catalog`) – Interface for catalog operations on the Session. - **`read`** (`DataFrameReader`) – Returns a DataFrameReader that can be used to read data in as a DataFrame. ### catalog ``` catalog: Catalog ``` Interface for catalog operations on the Session. ### read ``` read: DataFrameReader ``` Returns a DataFrameReader that can be used to read data in as a DataFrame. Returns: - **`DataFrameReader`** ( `DataFrameReader` ) – A reader interface to read data into DataFrame Raises: - `RuntimeError` – If the session has been stopped ### create_dataframe ``` create_dataframe(data: DataLike, schema: Schema | None = None) -> DataFrame ``` Create a DataFrame from a variety of Python-native data formats. Parameters: - **`data`** (`DataLike`) – Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table - **`schema`** (`Schema | None`, default: `None` ) – Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: - `DataFrame` – A new DataFrame instance Raises: - `ValidationError` – If the input format is unsupported or the provided columns do not match the schema. - `PlanError` – If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Create from Polars DataFrame ``` import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from Pandas DataFrame ``` import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from dictionary ``` session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Create from list of dictionaries ``` session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Create from pyarrow Table ``` import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Create with an explicit schema ``` import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` Source code in `src/fenic/api/session/session.py` ``` def create_dataframe( self, data: DataLike, schema: Schema | None = None, ) -> DataFrame: """Create a DataFrame from a variety of Python-native data formats. Args: data: Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table schema: Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: A new DataFrame instance Raises: ValidationError: If the input format is unsupported or the provided columns do not match the schema. PlanError: If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Example: Create from Polars DataFrame ```python import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from Pandas DataFrame ```python import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from dictionary ```python session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Example: Create from list of dictionaries ```python session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Example: Create from pyarrow Table ```python import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Example: Create with an explicit schema ```python import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` """ pl_df, row_field_names = _normalize_data_like_to_polars( data, allow_empty_list=schema is not None, validate_all_rows=schema is not None, ) if schema is None: return DataFrame._from_logical_plan( InMemorySource.from_session_state(pl_df, self._session_state), self._session_state, ) coerced_pl_df = _coerce_to_schema(pl_df, schema, row_field_names=row_field_names) return DataFrame._from_logical_plan( InMemorySource.from_schema(coerced_pl_df, schema), self._session_state, ) ``` ### get_or_create ``` get_or_create(config: SessionConfig) -> Session ``` Gets an existing Session or creates a new one with the configured settings. Returns: - `Session` – A Session instance configured with the provided settings Source code in `src/fenic/api/session/session.py` ``` @classmethod def get_or_create( cls, config: SessionConfig, ) -> Session: """Gets an existing Session or creates a new one with the configured settings. Returns: A Session instance configured with the provided settings """ if config.cloud: from fenic._backends.cloud.manager import CloudSessionManager cloud_session_manager = CloudSessionManager() if not cloud_session_manager.initialized: session_manager_dependencies = ( CloudSessionManager.create_global_session_dependencies() ) cloud_session_manager.configure(session_manager_dependencies) future = asyncio.run_coroutine_threadsafe( cloud_session_manager.get_or_create_session_state(config), cloud_session_manager._asyncio_loop, ) cloud_session_state = future.result() return Session._create_cloud_session(cloud_session_state) local_session_state: LocalSessionState = LocalSessionManager().get_or_create_session_state(config._to_resolved_config()) return Session._create_local_session(local_session_state) ``` ### sql ``` sql(query: str, /, **tables: DataFrame) -> DataFrame ``` Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Parameters: - **`query`** (`str`) – A SQL query string with placeholders like `{df}` - **`**tables`** (`DataFrame`, default: `{}` ) – Keyword arguments mapping placeholder names to DataFrames Returns: - `DataFrame` – A lazy DataFrame representing the result of the SQL query Raises: - `ValidationError` – If a placeholder is used in the query but not passed as a keyword argument Simple join between two DataFrames ``` df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Complex query with multiple DataFrames ``` users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(""" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id """, users=users, orders=orders, products=products) ``` Source code in `src/fenic/api/session/session.py` ``` def sql(self, query: str, /, **tables: DataFrame) -> DataFrame: """Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Args: query: A SQL query string with placeholders like `{df}` **tables: Keyword arguments mapping placeholder names to DataFrames Returns: A lazy DataFrame representing the result of the SQL query Raises: ValidationError: If a placeholder is used in the query but not passed as a keyword argument Example: Simple join between two DataFrames ```python df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Example: Complex query with multiple DataFrames ```python users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(\"\"\" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id \"\"\", users=users, orders=orders, products=products) ``` """ query = query.strip() if not query: raise ValidationError("SQL query must not be empty.") placeholders = set(SQL_PLACEHOLDER_RE.findall(query)) missing = placeholders - tables.keys() if missing: raise ValidationError( f"Missing DataFrames for placeholders in SQL query: {', '.join(sorted(missing))}. " f"Make sure to pass them as keyword arguments, e.g., sql(..., {next(iter(missing))}=df)." ) logical_plans = [] template_names = [] input_session_states = [] for name, table in tables.items(): if name in placeholders: template_names.append(name) logical_plans.append(table._logical_plan) input_session_states.append(table._session_state) DataFrame._ensure_same_session(self._session_state, input_session_states) return DataFrame._from_logical_plan( SQL.from_session_state(logical_plans, template_names, query, self._session_state), self._session_state, ) ``` ### stop ``` stop(skip_usage_summary: bool = False) ``` Stops the session and closes all connections. Parameters: - **`skip_usage_summary`** (`bool`, default: `False` ) – Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. Source code in `src/fenic/api/session/session.py` ``` def stop(self, skip_usage_summary: bool = False): """Stops the session and closes all connections. Args: skip_usage_summary: Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. """ self._session_state.stop(skip_usage_summary=skip_usage_summary) ``` ### table ``` table(table_name: str) -> DataFrame ``` Returns the specified table as a DataFrame. Parameters: - **`table_name`** (`str`) – Name of the table Returns: - `DataFrame` – Table as a DataFrame Raises: - `ValueError` – If the table does not exist Load an existing table ``` df = session.table("my_table") ``` Source code in `src/fenic/api/session/session.py` ``` def table(self, table_name: str) -> DataFrame: """Returns the specified table as a DataFrame. Args: table_name: Name of the table Returns: Table as a DataFrame Raises: ValueError: If the table does not exist Example: Load an existing table ```python df = session.table("my_table") ``` """ if not self._session_state.catalog.does_table_exist(table_name): raise ValueError(f"Table {table_name} does not exist") return DataFrame._from_logical_plan( TableSource.from_session_state(table_name, self._session_state), self._session_state, ) ``` ### view ``` view(view_name: str) -> DataFrame ``` Returns the specified view as a DataFrame. Parameters: - **`view_name`** (`str`) – Name of the view Returns: DataFrame: Dataframe with the given view Source code in `src/fenic/api/session/session.py` ``` def view(self, view_name: str) -> DataFrame: """Returns the specified view as a DataFrame. Args: view_name: Name of the view Returns: DataFrame: Dataframe with the given view """ if not self._session_state.catalog.does_view_exist(view_name): raise CatalogError(f"View {view_name} does not exist") view_plan = self._session_state.catalog.get_view_plan(view_name) validate_view(view_name, view_plan, self._session_state) return DataFrame._from_logical_plan( view_plan, self._session_state, ) ``` ## SessionConfig Bases: `BaseModel` ``` flowchart TD fenic.SessionConfig[SessionConfig] click fenic.SessionConfig href "" "fenic.SessionConfig" ``` Configuration for a user session. This class defines the complete configuration for a user session, including application settings, model configurations, and optional cloud settings. It serves as the central configuration object for all language model operations. Attributes: - **`app_name`** (`str`) – Name of the application using this session. Defaults to "default_app". - **`db_path`** (`Optional[Path]`) – Optional path to a local database file for persistent storage. - **`semantic`** (`Optional[SemanticConfig]`) – Configuration for semantic models (optional). - **`cloud`** (`Optional[CloudConfig]`) – Optional configuration for cloud execution. - **`cache`** (`Optional[CloudConfig]`) – Optional configuration for LLM response caching. Note The semantic configuration is optional. When not provided, only non-semantic operations are available. The cloud configuration is optional and only needed for distributed processing. Example Configuring a basic session with a single language model: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ) } ), ) ``` Configuring a session with multiple models and cloud execution: ``` config = SessionConfig( app_name="production_app", db_path=Path("/path/to/database.db"), semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ), cloud=CloudConfig(size=CloudExecutorSize.MEDIUM), ) ``` Methods: - **`to_json`** – Export the session config to a JSON string. ### to_json ``` to_json() -> str ``` Export the session config to a JSON string. Source code in `src/fenic/api/session/config.py` ``` def to_json(self) -> str: """Export the session config to a JSON string.""" return self.model_dump_json(indent=2) ``` ## StructField A field in a StructType. Fields are nullable. Attributes: - **`name`** (`str`) – The name of the field. - **`data_type`** (`DataType`) – The data type of the field. ## StructType Bases: `DataType` ``` flowchart TD fenic.StructType[StructType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.StructType click fenic.StructType href "" "fenic.StructType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a struct (record) with named fields. Attributes: - **`fields`** – List of field definitions. Create a struct with name and age fields ``` StructType([ StructField("name", StringType), StructField("age", IntegerType), ]) ``` ## SystemTool A tool implemented as a regular Python function with explicit parameters. The function must be a `Callable[..., LogicalPlan]` (a function defined with `async def`). Collection/formatting is handled by the MCP generator wrapper. ## SystemToolConfig ``` SystemToolConfig(table_names: list[str], tool_namespace: Optional[str] = None, max_result_rows: int = 100) ``` Configuration for canonical system tools. fenic can automatically generate a set of canonical tools for operating on one or more fenic tables. - Schema: list columns/types for any or all tables - Profile: column statistics (counts, basic numeric analysis [min, max, mean, etc.], contextual information for text columns [average_length, etc.]) - Read: read a selection of rows from a single table. These rows can be paged over, filtered and can use column projections. - Search Summary: literal or regex search across all text columns in all tables -- returns back dataframe names with result counts. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Search Content: literal or regex search across a single table, specifying one or more text columns to search across -- returns back rows corresponding to the query. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Analyze: Write raw SQL to perform complex analysis on one or more tables. Attributes: - **`table_names`** (`list[str]`) – List of the fenic table names the tools should be able to access. To allow access to all tables, pass `session.catalog.list_tables()` - **`tool_namespace`** (`Optional[str]`) – If provided, will prefix the names of the generated tools with this namespace value. For example, by default the generated tools will be named `read`, `profile`, etc. With multiple fenic MCP servers, these tool names will clash, which can be confusing. In order to disambiguate, the `tool_namespace` is prefixed to the tool name (in snake case), so a `tool_namespace` of `fenic` would create the tools `fenic_read`, `fenic_profile`, etc. - **`max_result_rows`** (`int`) – Maximum number of rows to be returned from Read/Analyze tools. Example: ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) df = session.create_dataframe({ "c1": [1, 2, 3], "c2": [4, 5, 6] }) df.write.save_as_table("table1", mode="overwrite") session.catalog.set_table_description("table1", "Table 1 Description") server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=["table1"], tool_namespace="Auto", max_result_rows=100 )) ``` Example: Allow generated tools to access all tables in the catalog. ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) # Assuming you already have one or more tables saved to the catalog, with descriptions. server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=session.catalog.list_tables() tool_namespace="Auto", max_result_rows=100 )) ``` ## ToolParam Bases: `BaseModel` ``` flowchart TD fenic.ToolParam[ToolParam] click fenic.ToolParam href "" "fenic.ToolParam" ``` A parameter for a parameterized view tool. A parameter is a named value that can be passed to a tool. These are matched to the parameter names of the `tool_param` UnresolvedLiteralExpr expressions captured in the Logical Plan. Attributes: - **`name`** (`str`) – The name of the parameter. - **`description`** (`str`) – The description of the parameter. - **`allowed_values`** (`Optional[List[ToolParameterType]]`) – The allowed values for the parameter. - **`has_default`** (`bool`) – Whether the parameter has a default value. - **`default_value`** (`Optional[ToolParameterType]`) – The default value for the parameter. ### required ``` required: bool ``` Whether the parameter is required. Returns: - `bool` – True if the parameter is required, False otherwise. ## TranscriptType Bases: `_LogicalType` ``` flowchart TD fenic.TranscriptType[TranscriptType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.TranscriptType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.TranscriptType href "" "fenic.TranscriptType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a transcript in a specific format. ## UserDefinedTool A tool that has been bound to a specific Parameterized View. ## approx_count_distinct ``` approx_count_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: - `Column` – A Column expression representing the approximate count-distinct aggregation Note Differs from the pyspark implementation in that the relative standard deviation is not configurable. Approximate distinct count per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Nulls are ignored in approximate distinct counts ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: - `TypeMismatchError` – If column is a StructType or ArrayType column. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def approx_count_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Args: column: Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: A Column expression representing the approximate count-distinct aggregation Note: Differs from the pyspark implementation in that the relative standard deviation is not configurable. Example: Approximate distinct count per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Example: Nulls are ignored in approximate distinct counts ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: TypeMismatchError: If column is a StructType or ArrayType column. """ return Column._from_logical_expr( ApproxCountDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## array_agg ``` array_agg(column: ColumnOrName) -> Column ``` Alias for collect_list(). Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array_agg(column: ColumnOrName) -> Column: """Alias for collect_list().""" return collect_list(column) ``` ## asc ``` asc(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls first. Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc() ``` ## asc_nulls_first ``` asc_nulls_first(column: ColumnOrName) -> Column ``` Alias for asc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_first(column: ColumnOrName) -> Column: """Alias for asc(). Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc_nulls_first() ``` ## asc_nulls_last ``` asc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A Column expression representing the column and the ascending sort order with nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls last. Args: column: The column to apply the ascending ordering to. Returns: A Column expression representing the column and the ascending sort order with nulls last. """ return Column._from_col_or_name(column).asc_nulls_last() ``` ## async_udf ``` async_udf(f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0) ``` A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Parameters: - **`f`** (`Optional[Callable[..., Awaitable[Any]]]`, default: `None` ) – Async function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. - **`max_concurrency`** (`int`, default: `10` ) – Maximum number of concurrent executions (default: 10) - **`timeout_seconds`** (`float`, default: `30` ) – Per-item timeout in seconds (default: 30) - **`num_retries`** (`int`, default: `0` ) – Number of retries for failed items (default: 0) Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) ### Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries `python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() }` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def async_udf( f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0, ): """A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Args: f: Async function to convert to UDF return_type: Expected return type of the UDF. Required parameter. max_concurrency: Maximum number of concurrent executions (default: 10) timeout_seconds: Per-item timeout in seconds (default: 30) num_retries: Number of retries for failed items (default: 0) Example: Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) # Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries ```python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() } ``` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. """ def _create_async_udf(func: Callable[..., Awaitable[Any]]) -> Callable: if not inspect.iscoroutinefunction(func): raise ValidationError( f"@async_udf requires an async function, but found a synchronous " f"function {func.__name__!r} of type {type(func)}" ) @wraps(func) def _async_udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr( AsyncUDFExpr( func, col_exprs, return_type, max_concurrency=max_concurrency, timeout_seconds=timeout_seconds, num_retries=num_retries ) ) return _async_udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for async UDFs") # Support both @async_udf and async_udf(...) syntax if f is None: return _create_async_udf else: return _create_async_udf(f) ``` ## avg ``` avg(column: ColumnOrName) -> Column ``` Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the average of Returns: - `Column` – A Column expression representing the average aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def avg(column: ColumnOrName) -> Column: """Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Args: column: Column or column name to compute the average of Returns: A Column expression representing the average aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## coalesce ``` coalesce(*cols: ColumnOrName) -> Column ``` Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the first non-null value from the input columns. Raises: - `ValidationError` – If no columns are provided. coalesce usage ``` df.select(coalesce("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def coalesce(*cols: ColumnOrName) -> Column: """Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the first non-null value from the input columns. Raises: ValidationError: If no columns are provided. Example: coalesce usage ```python df.select(coalesce("col1", "col2", "col3")) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the coalesce method.") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(CoalesceExpr(exprs)) ``` ## col ``` col(col_name: str) -> Column ``` Creates a Column expression referencing a column in the DataFrame. Parameters: - **`col_name`** (`str`) – Name of the column to reference Returns: - `Column` – A Column expression for the specified column Raises: - `TypeError` – If colName is not a string Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True)) def col(col_name: str) -> Column: """Creates a Column expression referencing a column in the DataFrame. Args: col_name: Name of the column to reference Returns: A Column expression for the specified column Raises: TypeError: If colName is not a string """ return Column._from_column_name(col_name) ``` ## collect_list ``` collect_list(column: ColumnOrName) -> Column ``` Aggregate function: collects all values from the specified column into a list. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to collect values from Returns: - `Column` – A Column expression representing the list aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def collect_list(column: ColumnOrName) -> Column: """Aggregate function: collects all values from the specified column into a list. Args: column: Column or column name to collect values from Returns: A Column expression representing the list aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( ListExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## configure_logging ``` configure_logging(log_level: int = logging.INFO, log_format: str = '%(asctime)s [%(name)s] %(levelname)s: %(message)s', log_stream: Optional[TextIO] = None) -> None ``` Configure logging for the library and root logger in interactive environments. This function ensures that logs from the library's modules appear in output by setting up a default handler on the root logger *only if* one does not already exist. This is especially useful in notebooks, scripts, or REPLs where logging is often unset. It configures the root logger and sets the library's top-level logger to propagate logs to the root. If the root logger has no handlers, this function sets up a default configuration and silences noisy dependencies like 'openai' and 'httpx'. In more complex applications or when integrating with existing logging configurations, you might prefer to manage logging setup externally. In such cases, you may not need to call this function. Source code in `src/fenic/logging.py` ``` def configure_logging( log_level: int = logging.INFO, log_format: str = "%(asctime)s [%(name)s] %(levelname)s: %(message)s", log_stream: Optional[TextIO] = None, ) -> None: """Configure logging for the library and root logger in interactive environments. This function ensures that logs from the library's modules appear in output by setting up a default handler on the root logger *only if* one does not already exist. This is especially useful in notebooks, scripts, or REPLs where logging is often unset. It configures the root logger and sets the library's top-level logger to propagate logs to the root. If the root logger has no handlers, this function sets up a default configuration and silences noisy dependencies like 'openai' and 'httpx'. In more complex applications or when integrating with existing logging configurations, you might prefer to manage logging setup externally. In such cases, you may not need to call this function. """ stream = log_stream or sys.stderr formatter = logging.Formatter(log_format) handler = logging.StreamHandler(stream) handler.setFormatter(formatter) root_logger = logging.getLogger() if not root_logger.hasHandlers(): # Set up root logger only if not already configured root_logger.setLevel(log_level) root_logger.addHandler(handler) # Silence noisy dependencies for noisy_logger_name in NOISY_LOGGER_NAMES: noisy_logger = logging.getLogger(noisy_logger_name) noisy_logger.setLevel(logging.ERROR) # Set the library logger level and enable propagation library_root_name = __name__.split(".")[0] library_logger = logging.getLogger(library_root_name) library_logger.setLevel(log_level) library_logger.propagate = True ``` ## count ``` count(column: ColumnOrName) -> Column ``` Aggregate function: returns the count of non-null values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to count values in Returns: - `Column` – A Column expression representing the count aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count(column: ColumnOrName) -> Column: """Aggregate function: returns the count of non-null values in the specified column. Args: column: Column or column name to count values in Returns: A Column expression representing the count aggregation Raises: TypeError: If column is not a Column or string """ if isinstance(column, str) and column == "*": return Column._from_logical_expr(CountExpr(lit("*")._logical_expr)) return Column._from_logical_expr( CountExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count_distinct ``` count_distinct(*cols: ColumnOrName) -> Column ``` Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – One or more columns or column names to include in the distinct count. Returns: - `Column` – A Column expression representing the count-distinct aggregation over the provided columns. Distinct count per group (single column) ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Distinct count across multiple columns (whole DataFrame) ``` # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Nulls in any input column are ignored for multi-column distinct ``` df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: - `ValidationError` – If no columns are provided. - `TypeMismatchError` – If a column has an unsupported type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count_distinct(*cols: ColumnOrName) -> Column: """Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Args: *cols: One or more columns or column names to include in the distinct count. Returns: A Column expression representing the count-distinct aggregation over the provided columns. Example: Distinct count per group (single column) ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Example: Distinct count across multiple columns (whole DataFrame) ```python # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Example: Nulls in any input column are ignored for multi-column distinct ```python df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: ValidationError: If no columns are provided. TypeMismatchError: If a column has an unsupported type """ if not cols: raise ValidationError("count_distinct requires at least one column") exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(CountDistinctExpr(exprs)) ``` ## create_mcp_server ``` create_mcp_server(session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8) -> FenicMCPServer ``` Create an MCP server from datasets and tools. Parameters: - **`session`** (`Session`) – Fenic session used to execute tools. - **`server_name`** (`str`) – Name of the MCP server. - **`user_defined_tools`** (`Optional[List[UserDefinedTool]]`, default: `None` ) – User defined tools to register with the MCP server. - **`system_tools`** (`Optional[SystemToolConfig]`, default: `None` ) – Configuration for automatically created system tools. - **`concurrency_limit`** (`int`, default: `8` ) – Maximum number of concurrent tool executions. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_mcp_server( session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8, ) -> FenicMCPServer: """Create an MCP server from datasets and tools. Args: session: Fenic session used to execute tools. server_name: Name of the MCP server. user_defined_tools: User defined tools to register with the MCP server. system_tools: Configuration for automatically created system tools. concurrency_limit: Maximum number of concurrent tool executions. """ generated_system_tools = [] user_defined_tools = user_defined_tools or [] if system_tools: generated_system_tools.extend( auto_generate_system_tools_from_tables( system_tools.table_names, session, tool_namespace=system_tools.tool_namespace, max_result_limit=system_tools.max_result_rows ) ) if not (user_defined_tools or system_tools): raise ConfigurationError("No tools provided. Either provide `user_defined_tools` or set `system_tools` to create system tools for catalog tables.") return FenicMCPServer(session._session_state, user_defined_tools, generated_system_tools, server_name, concurrency_limit) ``` ## desc ``` desc(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls first. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc() ``` ## desc_nulls_first ``` desc_nulls_first(column: ColumnOrName) -> Column ``` Alias for desc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_first(column: ColumnOrName) -> Column: """Alias for desc(). Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc_nulls_first() ``` ## desc_nulls_last ``` desc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls last. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls last. """ return Column._from_col_or_name(column).desc_nulls_last() ``` ## empty ``` empty(data_type: DataType) -> Column ``` Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the empty value Returns: - `Column` – A Column expression representing the empty value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with an empty array type ``` # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Creating a column with an empty struct type ``` # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Creating a column with an empty primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` Source code in `src/fenic/api/functions/core.py` ``` def empty(data_type: DataType) -> Column: """Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Args: data_type: The data type of the empty value Returns: A Column expression representing the empty value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with an empty array type ```python # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Example: Creating a column with an empty struct type ```python # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Example: Creating a column with an empty primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` """ if isinstance(data_type, ArrayType): return Column._from_logical_expr(LiteralExpr([], data_type)) elif isinstance(data_type, StructType): return Column._from_logical_expr(LiteralExpr({}, data_type)) return null(data_type) ``` ## first ``` first(column: ColumnOrName) -> Column ``` Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for the first value. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def first(column: ColumnOrName) -> Column: """Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Args: column: Column or column name. Returns: Column expression for the first value. """ return Column._from_logical_expr( FirstExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## flatten ``` flatten(column: ColumnOrName) -> Column ``` Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of arrays. Returns: - `Column` – A Column with flattened arrays (one level deep). Flattening nested arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` One level only ``` # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def flatten(column: ColumnOrName) -> Column: """Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Args: column: Column or column name containing arrays of arrays. Returns: A Column with flattened arrays (one level deep). Example: Flattening nested arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: One level only ```python # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` """ return Column._from_logical_expr( FlattenExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## greatest ``` greatest(*cols: ColumnOrName) -> Column ``` Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the greatest value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. greatest usage ``` df.select(fc.greatest("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def greatest(*cols: ColumnOrName) -> Column: """Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the greatest value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: greatest usage ```python df.select(fc.greatest("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"greatest() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(GreatestExpr(exprs)) ``` ## least ``` least(*cols: ColumnOrName) -> Column ``` Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the least value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. least usage ``` df.select(fc.least("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def least(*cols: ColumnOrName) -> Column: """Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the least value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: least usage ```python df.select(fc.least("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"least() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(LeastExpr(exprs)) ``` ## lit ``` lit(value: Any) -> Column ``` Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Parameters: - **`value`** (`Any`) – The literal value to create a column for Returns: - `Column` – A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred Source code in `src/fenic/api/functions/core.py` ``` def lit(value: Any) -> Column: """Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value: - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Args: value: The literal value to create a column for Returns: A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred """ if value is None: raise ValidationError("Cannot create a literal with value `None`. Use `null(...)` instead.") elif value == []: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(ArrayType(...))` instead.") elif value == {}: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(StructType(...))` instead.") try: inferred_type = infer_dtype_from_pyobj(value) except TypeInferenceError as e: raise ValidationError(f"`lit` failed to infer type for value `{value}`") from e literal_expr = LiteralExpr(value, inferred_type) return Column._from_logical_expr(literal_expr) ``` ## max ``` max(column: ColumnOrName) -> Column ``` Aggregate function: returns the maximum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the maximum of Returns: - `Column` – A Column expression representing the maximum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def max(column: ColumnOrName) -> Column: """Aggregate function: returns the maximum value in the specified column. Args: column: Column or column name to compute the maximum of Returns: A Column expression representing the maximum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MaxExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## mean ``` mean(column: ColumnOrName) -> Column ``` Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the mean of Returns: - `Column` – A Column expression representing the mean aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def mean(column: ColumnOrName) -> Column: """Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Args: column: Column or column name to compute the mean of Returns: A Column expression representing the mean aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## min ``` min(column: ColumnOrName) -> Column ``` Aggregate function: returns the minimum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the minimum of Returns: - `Column` – A Column expression representing the minimum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def min(column: ColumnOrName) -> Column: """Aggregate function: returns the minimum value in the specified column. Args: column: Column or column name to compute the minimum of Returns: A Column expression representing the minimum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MinExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## null ``` null(data_type: DataType) -> Column ``` Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the null value Returns: - `Column` – A Column expression representing the null value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with a null value of a primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Creating a column with a null value of an array/struct type ``` # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` Source code in `src/fenic/api/functions/core.py` ``` def null(data_type: DataType) -> Column: """Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Args: data_type: The data type of the null value Returns: A Column expression representing the null value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with a null value of a primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Example: Creating a column with a null value of an array/struct type ```python # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` """ return Column._from_logical_expr(LiteralExpr(None, data_type)) ``` ## run_mcp_server_asgi ``` run_mcp_server_asgi(server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`transport`** (`Literal['streamable-http', 'sse']`, default: `'streamable-http'` ) – Transport protocol to use (streamable-http, sse). - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional starlette-specific arguments to pass to FastMCP. Notes Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_asgi( server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Args: server: MCP server to run. stateless_http: If True, use stateless HTTP. transport: Transport protocol to use (streamable-http, sse). path: Path to listen on. kwargs: Additional starlette-specific arguments to pass to FastMCP. Notes: Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. """ return server.http_app(stateless_http=stateless_http, transport=transport, path=path, **kwargs) ``` ## run_mcp_server_async ``` run_mcp_server_async(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) async def run_mcp_server_async( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ await server.run_async(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## run_mcp_server_sync ``` run_mcp_server_sync(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_sync( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ server.run(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## stddev ``` stddev(column: ColumnOrName) -> Column ``` Aggregate function: returns the sample standard deviation of the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for sample standard deviation. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def stddev(column: ColumnOrName) -> Column: """Aggregate function: returns the sample standard deviation of the specified column. Args: column: Column or column name. Returns: Column expression for sample standard deviation. """ return Column._from_logical_expr( StdDevExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## struct ``` struct(*args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]) -> Column ``` Creates a new struct column from multiple input columns. Parameters: - **`*args`** (`Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]`, default: `()` ) – Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: - `Column` – A Column expression representing a struct containing the input columns Raises: - `TypeError` – If any argument is not a Column, string, or collection of Columns/strings Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def struct( *args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]] ) -> Column: """Creates a new struct column from multiple input columns. Args: *args: Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: A Column expression representing a struct containing the input columns Raises: TypeError: If any argument is not a Column, string, or collection of Columns/strings """ flattened_args = [] for arg in args: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_columns = [Column._from_col_or_name(c)._logical_expr for c in flattened_args] return Column._from_logical_expr(StructExpr(expr_columns)) ``` ## sum ``` sum(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of all values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of Returns: - `Column` – A Column expression representing the sum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of all values in the specified column. Args: column: Column or column name to compute the sum of Returns: A Column expression representing the sum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( SumExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## sum_distinct ``` sum_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of distinct numeric values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of distinct values Returns: - `Column` – A Column expression representing the sum-distinct aggregation Sum of distinct values per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Nulls are ignored when summing distinct values ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: - `TypeMismatchError` – If column is not a numeric or boolean type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of distinct numeric values in the specified column. Args: column: Column or column name to compute the sum of distinct values Returns: A Column expression representing the sum-distinct aggregation Example: Sum of distinct values per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Example: Nulls are ignored when summing distinct values ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: TypeMismatchError: If column is not a numeric or boolean type """ return Column._from_logical_expr( SumDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## tool_param ``` tool_param(parameter_name: str, data_type: DataType) -> Column ``` Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Parameters: - **`parameter_name`** (`str`) – The name of the parameter to reference. - **`data_type`** (`DataType`) – The expected data type for the parameter value. Returns: - `Column` – A Column wrapping an UnresolvedLiteralExpr for the given parameter. A simple tool with one parameter ```python ### Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) A tool with multiple filters ```python ### Assume we are reading data with an `age` column. df = session.read.csv(users.csv) ### create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def tool_param(parameter_name: str, data_type: DataType) -> Column: """Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes: Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Args: parameter_name: The name of the parameter to reference. data_type: The expected data type for the parameter value. Returns: A Column wrapping an UnresolvedLiteralExpr for the given parameter. Example: A simple tool with one parameter ```python # Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) Example: A tool with multiple filters ```python # Assume we are reading data with an `age` column. df = session.read.csv(users.csv) # create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) """ if isinstance(data_type, _LogicalType): raise ValidationError(f"Cannot use a logical type as a parameter type: {data_type}") return Column._from_logical_expr(UnresolvedLiteralExpr(data_type=data_type, parameter_name=parameter_name)) ``` ## udf ``` udf(f: Optional[Callable] = None, *, return_type: DataType) ``` A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Parameters: - **`f`** (`Optional[Callable]`, default: `None` ) – Python function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. UDF with primitive types ``` # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` UDF with nested types ``` # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def udf(f: Optional[Callable] = None, *, return_type: DataType): """A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning: UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Args: f: Python function to convert to UDF return_type: Expected return type of the UDF. Required parameter. Example: UDF with primitive types ```python # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` Example: UDF with nested types ```python # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` """ def _create_udf(func: Callable) -> Callable: @wraps(func) def _udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(UDFExpr(func, col_exprs, return_type)) return _udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for UDFs") if f is not None: return _create_udf(f) return _create_udf ``` ## when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A when expression that can be chained with more conditions Raises: - `TypeMismatchError` – If the condition is not a boolean Column expression. Example ``` # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def when(condition: Column, value: Column) -> Column: """Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A when expression that can be chained with more conditions Raises: TypeMismatchError: If the condition is not a boolean Column expression. Example: ```python # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null """ return Column._from_logical_expr( WhenExpr(None, condition._logical_expr, value._logical_expr) ) ``` --- # fenic.api Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/ Query module for semantic operations on DataFrames. Classes: - **`AdaptiveTokenEstimationConfig`** – Tunes adaptive output-token reservation for rate limiting. - **`AnthropicLanguageModel`** – Configuration for Anthropic language models. - **`Catalog`** – Entry point for catalog operations. - **`CloudConfig`** – Configuration for cloud-based execution. - **`CohereEmbeddingModel`** – Configuration for Cohere embedding models. - **`Column`** – A column expression in a DataFrame. - **`DataFrame`** – A data collection organized into named columns. - **`DataFrameReader`** – Interface used to load a DataFrame from external storage systems. - **`DataFrameWriter`** – Interface used to write a DataFrame to external storage systems. - **`GoogleDeveloperEmbeddingModel`** – Configuration for Google Developer embedding models. - **`GoogleDeveloperLanguageModel`** – Configuration for Gemini models accessible through Google Developer AI Studio. - **`GoogleVertexEmbeddingModel`** – Configuration for Google Vertex AI embedding models. - **`GoogleVertexLanguageModel`** – Configuration for Google Vertex AI models. - **`GroupedData`** – Methods for aggregations on a grouped DataFrame. - **`LLMResponseCacheConfig`** – Configuration for LLM response caching. - **`Lineage`** – Query interface for tracing data lineage through a query plan. - **`OpenAIEmbeddingModel`** – Configuration for OpenAI embedding models. - **`OpenAILanguageModel`** – Configuration for OpenAI language models. - **`OpenRouterLanguageModel`** – Configuration for OpenRouter language models. - **`SemanticConfig`** – Configuration for semantic language and embedding models. - **`SemanticExtensions`** – A namespace for semantic dataframe operators. - **`Session`** – The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. - **`SessionConfig`** – Configuration for a user session. - **`SystemToolConfig`** – Configuration for canonical system tools. Functions: - **`approx_count_distinct`** – Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. - **`array_agg`** – Alias for collect_list(). - **`asc`** – Mark this column for ascending sort order with nulls first. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`async_udf`** – A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. - **`avg`** – Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. - **`coalesce`** – Returns the first non-null value from the given columns for each row. - **`col`** – Creates a Column expression referencing a column in the DataFrame. - **`collect_list`** – Aggregate function: collects all values from the specified column into a list. - **`count`** – Aggregate function: returns the count of non-null values in the specified column. - **`count_distinct`** – Aggregate function: returns the number of distinct non-null rows across one or more columns. - **`create_mcp_server`** – Create an MCP server from datasets and tools. - **`desc`** – Mark this column for descending sort order with nulls first. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`empty`** – Creates a Column expression representing an empty value of the given type. - **`first`** – Aggregate function: returns the first non-null value in the specified column. - **`flatten`** – Flattens an array of arrays into a single array (one level deep). - **`greatest`** – Returns the greatest value from the given columns for each row. - **`least`** – Returns the least value from the given columns for each row. - **`lit`** – Creates a Column expression representing a literal value. - **`max`** – Aggregate function: returns the maximum value in the specified column. - **`mean`** – Aggregate function: returns the mean (average) of all values in the specified column. - **`min`** – Aggregate function: returns the minimum value in the specified column. - **`null`** – Creates a Column expression representing a null value of the specified data type. - **`run_mcp_server_asgi`** – Run an MCP server as a Starlette ASGI app. - **`run_mcp_server_async`** – Run an MCP server asynchronously. - **`run_mcp_server_sync`** – Run an MCP server synchronously. - **`stddev`** – Aggregate function: returns the sample standard deviation of the specified column. - **`struct`** – Creates a new struct column from multiple input columns. - **`sum`** – Aggregate function: returns the sum of all values in the specified column. - **`sum_distinct`** – Aggregate function: returns the sum of distinct numeric values in the specified column. - **`tool_param`** – Creates an unresolved literal placeholder column with a declared data type. - **`udf`** – A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. - **`when`** – Evaluates a conditional expression (like if-then). ## AdaptiveTokenEstimationConfig Bases: `BaseModel` ``` flowchart TD fenic.api.AdaptiveTokenEstimationConfig[AdaptiveTokenEstimationConfig] click fenic.api.AdaptiveTokenEstimationConfig href "" "fenic.api.AdaptiveTokenEstimationConfig" ``` Tunes adaptive output-token reservation for rate limiting. Output-token reservations are learned from observed usage and clamped to the request's max_completion_tokens ceiling, then corrected after each response (settlement). Enabled by default. Setting `enabled=False` disables adaptive *estimation* β€” reservations fall back to the static worst-case ceiling instead of the learned distribution. Settlement (reconciling the token bucket to actual usage after each response) is **always on** regardless of this flag. It corrects the bucket in both directions β€” refunding the over-reservation (the common case) and debiting further when a request exceeds its reservation β€” and neither direction increases 429 risk: a refund only returns capacity the provider never charged, and a debit only makes the limiter more conservative. ## AnthropicLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.AnthropicLanguageModel[AnthropicLanguageModel] click fenic.api.AnthropicLanguageModel href "" "fenic.api.AnthropicLanguageModel" ``` Configuration for Anthropic language models. This class defines the configuration settings for Anthropic language models, including model selection and separate rate limiting parameters for input and output tokens. Attributes: - **`model_name`** (`AnthropicLanguageModelName`) – The name of the Anthropic model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`input_tpm`** (`int`) – Input tokens per minute limit; must be greater than 0. - **`output_tpm`** (`int`) – Output tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring an Anthropic model with separate input/output rate limits: ``` config = AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100 ) ``` Configuring an Anthropic model with profiles: ``` config = SessionConfig( semantic=SemanticConfig( language_models={ "claude": AnthropicLanguageModel( model_name="claude-sonnet-4-6", rpm=100, input_tpm=100, output_tpm=100, profiles={ "thinking_disabled": AnthropicLanguageModel.Profile(), "fast": AnthropicLanguageModel.Profile(thinking_token_budget=1024), "thorough": AnthropicLanguageModel.Profile(thinking_token_budget=4096) }, default_profile="fast" ) }, default_language_model="claude" ) # Using the default "fast" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias="claude") # Using the "thorough" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="claude", profile="thorough")) ``` Classes: - **`Profile`** – Anthropic-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.AnthropicLanguageModel.Profile[Profile] click fenic.api.AnthropicLanguageModel.Profile href "" "fenic.api.AnthropicLanguageModel.Profile" ``` Anthropic-specific profile configurations. This class defines profile configurations for Anthropic models, allowing different thinking and effort settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – Provide a default thinking budget in tokens. If not provided, thinking will be disabled for the profile. The minimum token budget supported by Anthropic is 1024 tokens. For Claude models that use adaptive thinking, use `effort` instead. - **`effort`** (`Optional[AnthropicReasoningEffortType]`) – Provider-native Anthropic effort level. Supported values vary by model: low, medium, high, xhigh, and max. On adaptive-thinking models the thinking budget shares the request's output token window rather than being reserved on top of it, so very high effort levels can consume part of the visible completion budget. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note If `thinking_token_budget` or adaptive `effort` enables thinking, `temperature` cannot be customized -- any changes to `temperature` will be ignored. Effort-only profiles on non-adaptive models configure Anthropic `output_config` without enabling thinking, so custom `temperature` remains available when the model supports it. Example Configuring a profile with a thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=2048) ``` Configuring a profile with a large thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=8192) ``` Configuring a profile with effort: ``` profile = AnthropicLanguageModel.Profile(effort="xhigh") ``` ## Catalog ``` Catalog(catalog: BaseCatalog) ``` Entry point for catalog operations. Provides methods to manage catalogs, databases, and tables, as well as read-only access to system tables such as `fenic_system.query_metrics`. ##### Catalog and Database Management Example: ```python # Create a catalog session.catalog.create_catalog("my_catalog") # β†’ True ``` # Set active catalog session.catalog.set_current_catalog("my_catalog") # Create a database session.catalog.create_database("my_database") # β†’ True # Set active database session.catalog.set_current_database("my_database") # Create a table session.catalog.create_table( "my_table", Schema([ColumnField("id", IntegerType)]) ) # β†’ True ``` ``` ##### Metrics Table (Local Sessions Only) Query metrics are recorded for each session and stored locally in `fenic_system.query_metrics`. Metrics can be loaded into a DataFrame for analysis. Example ``` # Load all metrics for the current application metrics_df = session.table("fenic_system.query_metrics") # Show the 10 most recent queries in the application recent_queries = session.sql(""" SELECT * FROM {df} ORDER BY CAST(end_ts AS TIMESTAMP) DESC LIMIT 10 """, df=metrics_df) recent_queries.show() # Find query metrics for a specific session with non-zero LM costs specific_session_queries = session.sql(""" SELECT * FROM {df} WHERE session_id = '9e7e256f-fad9-4cd9-844e-399d795aaea0' AND total_lm_cost > 0 ORDER BY CAST(end_ts AS TIMESTAMP) ASC """, df=metrics_df) specific_session_queries.show() # Aggregate total LM costs and requests between a specific time window metrics_window = session.sql(""" SELECT CAST(SUM(total_lm_cost) AS DOUBLE) AS total_lm_cost_in_window, CAST(SUM(total_lm_requests) AS DOUBLE) AS total_lm_requests_in_window FROM {df} WHERE CAST(end_ts AS TIMESTAMP) BETWEEN CAST('2025-08-29 10:00:00' AS TIMESTAMP) AND CAST('2025-08-29 12:00:00' AS TIMESTAMP) """, df=metrics_df) metrics_window.show() ``` Initialize a Catalog instance. Parameters: - **`catalog`** (`BaseCatalog`) – The underlying catalog implementation. Methods: - **`create_catalog`** – Creates a new catalog. - **`create_database`** – Creates a new database. - **`create_table`** – Creates a new table. - **`create_tool`** – Creates a new tool in the current catalog. - **`describe_table`** – Returns the schema of the specified table. - **`describe_tool`** – Returns the tool with the specified name from the current catalog. - **`describe_view`** – Returns the schema and description of the specified view. - **`does_catalog_exist`** – Checks if a catalog with the specified name exists. - **`does_database_exist`** – Checks if a database with the specified name exists. - **`does_table_exist`** – Checks if a table with the specified name exists. - **`does_view_exist`** – Checks if a view with the specified name exists. - **`drop_catalog`** – Drops a catalog. - **`drop_database`** – Drops a database. - **`drop_table`** – Drops the specified table. - **`drop_tool`** – Drops the specified tool from the current catalog. - **`drop_view`** – Drops the specified view. - **`get_current_catalog`** – Returns the name of the current catalog. - **`get_current_database`** – Returns the name of the current database in the current catalog. - **`list_catalogs`** – Returns a list of available catalogs. - **`list_databases`** – Returns a list of databases in the current catalog. - **`list_tables`** – Returns a list of tables stored in the current database. - **`list_tools`** – Lists the tools available in the current catalog. - **`list_views`** – Returns a list of views stored in the current database. - **`set_current_catalog`** – Sets the current catalog. - **`set_current_database`** – Sets the current database. - **`set_table_description`** – Set or unset the description for a table. - **`set_view_description`** – Set the description for a view. Source code in `src/fenic/api/catalog.py` ``` def __init__(self, catalog: BaseCatalog): """Initialize a Catalog instance. Args: catalog: The underlying catalog implementation. """ self.catalog = catalog ``` ### create_catalog ``` create_catalog(catalog_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: - `CatalogAlreadyExistsError` – If the catalog already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the catalog was created successfully, False if the catalog - `bool` – already exists and ignore_if_exists is True. Create a new catalog ``` # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Create an existing catalog with ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Create an existing catalog without ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_catalog(self, catalog_name: str, ignore_if_exists: bool = True) -> bool: """Creates a new catalog. Args: catalog_name (str): Name of the catalog to create. ignore_if_exists (bool): If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: CatalogAlreadyExistsError: If the catalog already exists and ignore_if_exists is False. Returns: bool: True if the catalog was created successfully, False if the catalog already exists and ignore_if_exists is True. Example: Create a new catalog ```python # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Example: Create an existing catalog with ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Example: Create an existing catalog without ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` """ return self.catalog.create_catalog(catalog_name, ignore_if_exists) ``` ### create_database ``` create_database(database_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: - `DatabaseAlreadyExistsError` – If the database already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the database was created successfully, False if the database - `bool` – already exists and ignore_if_exists is True. Create a new database ``` # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Create an existing database with ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Create an existing database without ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_database( self, database_name: str, ignore_if_exists: bool = True ) -> bool: """Creates a new database. Args: database_name (str): Fully qualified or relative database name to create. ignore_if_exists (bool): If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: DatabaseAlreadyExistsError: If the database already exists and ignore_if_exists is False. Returns: bool: True if the database was created successfully, False if the database already exists and ignore_if_exists is True. Example: Create a new database ```python # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Example: Create an existing database with ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Example: Create an existing database without ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` """ return self.catalog.create_database(database_name, ignore_if_exists) ``` ### create_table ``` create_table(table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None) -> bool ``` Creates a new table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to create. - **`schema`** (`Schema`) – Schema of the table to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. - **`description`** (`Optional[str]`, default: `None` ) – Description of the table to create. Defaults to None. Returns: - **`bool`** ( `bool` ) – True if the table was created successfully, False if the table - `bool` – already exists and ignore_if_exists is True. Raises: - `TableAlreadyExistsError` – If the table already exists and ignore_if_exists is False Create a new table ``` # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Create an existing table with ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Create an existing table without ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_table( self, table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None ) -> bool: """Creates a new table. Args: table_name (str): Fully qualified or relative table name to create. schema (Schema): Schema of the table to create. ignore_if_exists (bool): If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. description (Optional[str]): Description of the table to create. Defaults to None. Returns: bool: True if the table was created successfully, False if the table already exists and ignore_if_exists is True. Raises: TableAlreadyExistsError: If the table already exists and ignore_if_exists is False Example: Create a new table ```python # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Example: Create an existing table with ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Example: Create an existing table without ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` """ return self.catalog.create_table(table_name, schema, ignore_if_exists, description) ``` ### create_tool ``` create_tool(tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True) -> bool ``` Creates a new tool in the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool. - **`tool_description`** (`str`) – The description of the tool. - **`tool_query`** (`DataFrame`) – The query to execute when the tool is called. - **`tool_params`** (`Sequence[ToolParam]`) – The parameters of the tool. - **`result_limit`** (`int`, default: `50` ) – The maximum number of rows to return from the tool. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was created successfully, False otherwise. Raises: - `ToolAlreadyExistsError` – If the tool already exists. Examples: ``` # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_tool( self, tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True ) -> bool: """Creates a new tool in the current catalog. Args: tool_name (str): The name of the tool. tool_description (str): The description of the tool. tool_query (DataFrame): The query to execute when the tool is called. tool_params (Sequence[ToolParam]): The parameters of the tool. result_limit (int): The maximum number of rows to return from the tool. ignore_if_exists (bool): If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: bool: True if the tool was created successfully, False otherwise. Raises: ToolAlreadyExistsError: If the tool already exists. Examples: ```python # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` """ return self.catalog.create_tool( tool_name, tool_description, tool_params, tool_query._logical_plan, result_limit, ignore_if_exists, ) ``` ### describe_table ``` describe_table(table_name: str) -> DatasetMetadata ``` Returns the schema of the specified table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: - `TableNotFoundError` – If the table doesn't exist. Describe a table's schema ``` # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_table(self, table_name: str) -> DatasetMetadata: """Returns the schema of the specified table. Args: table_name (str): Fully qualified or relative table name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: TableNotFoundError: If the table doesn't exist. Example: Describe a table's schema ```python # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` """ return self.catalog.describe_table(table_name) ``` ### describe_tool ``` describe_tool(tool_name: str) -> UserDefinedTool ``` Returns the tool with the specified name from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to get. Raises: - `ToolNotFoundError` – If the tool doesn't exist. Returns: - **`Tool`** ( `UserDefinedTool` ) – The tool with the specified name. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_tool(self, tool_name: str) -> UserDefinedTool: """Returns the tool with the specified name from the current catalog. Args: tool_name (str): The name of the tool to get. Raises: ToolNotFoundError: If the tool doesn't exist. Returns: Tool: The tool with the specified name. """ return self.catalog.describe_tool(tool_name) ``` ### describe_view ``` describe_view(view_name: str) -> DatasetMetadata ``` Returns the schema and description of the specified view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: - `TableNotFoundError` – If the view doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_view(self, view_name: str) -> DatasetMetadata: """Returns the schema and description of the specified view. Args: view_name (str): Fully qualified or relative view name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: TableNotFoundError: If the view doesn't exist. """ return self.catalog.describe_view(view_name) ``` ### does_catalog_exist ``` does_catalog_exist(catalog_name: str) -> bool ``` Checks if a catalog with the specified name exists. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to check. Returns: - **`bool`** ( `bool` ) – True if the catalog exists, False otherwise. Check if a catalog exists ``` # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_catalog_exist(self, catalog_name: str) -> bool: """Checks if a catalog with the specified name exists. Args: catalog_name (str): Name of the catalog to check. Returns: bool: True if the catalog exists, False otherwise. Example: Check if a catalog exists ```python # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` """ return self.catalog.does_catalog_exist(catalog_name) ``` ### does_database_exist ``` does_database_exist(database_name: str) -> bool ``` Checks if a database with the specified name exists. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to check. Returns: - **`bool`** ( `bool` ) – True if the database exists, False otherwise. Check if a database exists ``` # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_database_exist(self, database_name: str) -> bool: """Checks if a database with the specified name exists. Args: database_name (str): Fully qualified or relative database name to check. Returns: bool: True if the database exists, False otherwise. Example: Check if a database exists ```python # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` """ return self.catalog.does_database_exist(database_name) ``` ### does_table_exist ``` does_table_exist(table_name: str) -> bool ``` Checks if a table with the specified name exists. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to check. Returns: - **`bool`** ( `bool` ) – True if the table exists, False otherwise. Check if a table exists ``` # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_table_exist(self, table_name: str) -> bool: """Checks if a table with the specified name exists. Args: table_name (str): Fully qualified or relative table name to check. Returns: bool: True if the table exists, False otherwise. Example: Check if a table exists ```python # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` """ return self.catalog.does_table_exist(table_name) ``` ### does_view_exist ``` does_view_exist(view_name: str) -> bool ``` Checks if a view with the specified name exists. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to check. Returns: - **`bool`** ( `bool` ) – True if the view exists, False otherwise. Example > > > session.catalog.does_view_exist('my_view') > > > True. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_view_exist(self, view_name: str) -> bool: """Checks if a view with the specified name exists. Args: view_name (str): Fully qualified or relative view name to check. Returns: bool: True if the view exists, False otherwise. Example: >>> session.catalog.does_view_exist('my_view') True. """ return self.catalog.does_view_exist(view_name) ``` ### drop_catalog ``` drop_catalog(catalog_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops a catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: - `CatalogNotFoundError` – If the catalog does not exist and ignore_if_not_exists is False Returns: - **`bool`** ( `bool` ) – True if the catalog was dropped successfully, False if the catalog - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent catalog ``` # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Drop a non-existent catalog without ignore_if_not_exists ``` # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_catalog( self, catalog_name: str, ignore_if_not_exists: bool = True ) -> bool: """Drops a catalog. Args: catalog_name (str): Name of the catalog to drop. ignore_if_not_exists (bool): If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: CatalogNotFoundError: If the catalog does not exist and ignore_if_not_exists is False Returns: bool: True if the catalog was dropped successfully, False if the catalog didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent catalog ```python # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Example: Drop a non-existent catalog without ignore_if_not_exists ```python # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` """ return self.catalog.drop_catalog(catalog_name, ignore_if_not_exists) ``` ### drop_database ``` drop_database(database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True) -> bool ``` Drops a database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to drop. - **`cascade`** (`bool`, default: `False` ) – If True, drop all tables in the database. Defaults to False. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: - `DatabaseNotFoundError` – If the database does not exist and ignore_if_not_exists is False - `CatalogError` – If the current database is being dropped, if the database is not empty and cascade is False Returns: - **`bool`** ( `bool` ) – True if the database was dropped successfully, False if the database - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent database ``` # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Drop a non-existent database without ignore_if_not_exists ``` # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_database( self, database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True, ) -> bool: """Drops a database. Args: database_name (str): Fully qualified or relative database name to drop. cascade (bool): If True, drop all tables in the database. Defaults to False. ignore_if_not_exists (bool): If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: DatabaseNotFoundError: If the database does not exist and ignore_if_not_exists is False CatalogError: If the current database is being dropped, if the database is not empty and cascade is False Returns: bool: True if the database was dropped successfully, False if the database didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent database ```python # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Example: Drop a non-existent database without ignore_if_not_exists ```python # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` """ return self.catalog.drop_database(database_name, cascade, ignore_if_not_exists) ``` ### drop_table ``` drop_table(table_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified table. By default this method will return False if the table doesn't exist. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the table was dropped successfully, False if the table - `bool` – didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the table doesn't exist and ignore_if_not_exists is False Drop an existing table ``` # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Drop a non-existent table with ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Drop a non-existent table without ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_table(self, table_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified table. By default this method will return False if the table doesn't exist. Args: table_name (str): Fully qualified or relative table name to drop. ignore_if_not_exists (bool): If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: bool: True if the table was dropped successfully, False if the table didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the table doesn't exist and ignore_if_not_exists is False Example: Drop an existing table ```python # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Example: Drop a non-existent table with ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Example: Drop a non-existent table without ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` """ return self.catalog.drop_table(table_name, ignore_if_not_exists) ``` ### drop_tool ``` drop_tool(tool_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified tool from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: - `ToolNotFoundError` – If the tool doesn't exist and ignore_if_not_exists is False Example > > > session.catalog.drop_tool('my_tool') > > > True > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) > > > False > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) #### Raises ToolNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_tool(self, tool_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified tool from the current catalog. Args: tool_name (str): The name of the tool to drop. ignore_if_not_exists (bool): If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: bool: True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: ToolNotFoundError: If the tool doesn't exist and ignore_if_not_exists is False Example: >>> session.catalog.drop_tool('my_tool') True >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) False >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) # Raises ToolNotFoundError. """ return self.catalog.drop_tool(tool_name, ignore_if_not_exists) ``` ### drop_view ``` drop_view(view_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified view. By default this method will return False if the view doesn't exist. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_view(self, view_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified view. By default this method will return False if the view doesn't exist. Args: view_name (str): Fully qualified or relative view name to drop. ignore_if_not_exists (bool, optional): If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: bool: True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. """ return self.catalog.drop_view(view_name, ignore_if_not_exists) ``` ### get_current_catalog ``` get_current_catalog() -> str ``` Returns the name of the current catalog. Returns: - **`str`** ( `str` ) – The name of the current catalog. Get current catalog name ``` # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_catalog(self) -> str: """Returns the name of the current catalog. Returns: str: The name of the current catalog. Example: Get current catalog name ```python # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` """ return self.catalog.get_current_catalog() ``` ### get_current_database ``` get_current_database() -> str ``` Returns the name of the current database in the current catalog. Returns: - **`str`** ( `str` ) – The name of the current database. Get current database name ``` # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_database(self) -> str: """Returns the name of the current database in the current catalog. Returns: str: The name of the current database. Example: Get current database name ```python # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` """ return self.catalog.get_current_database() ``` ### list_catalogs ``` list_catalogs() -> List[str] ``` Returns a list of available catalogs. Returns: - `List[str]` – List[str]: A list of catalog names available in the system. - `List[str]` – Returns an empty list if no catalogs are found. List all catalogs ``` # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_catalogs(self) -> List[str]: """Returns a list of available catalogs. Returns: List[str]: A list of catalog names available in the system. Returns an empty list if no catalogs are found. Example: List all catalogs ```python # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` """ return self.catalog.list_catalogs() ``` ### list_databases ``` list_databases() -> List[str] ``` Returns a list of databases in the current catalog. Returns: - `List[str]` – List[str]: A list of database names in the current catalog. - `List[str]` – Returns an empty list if no databases are found. List all databases ``` # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_databases(self) -> List[str]: """Returns a list of databases in the current catalog. Returns: List[str]: A list of database names in the current catalog. Returns an empty list if no databases are found. Example: List all databases ```python # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` """ return self.catalog.list_databases() ``` ### list_tables ``` list_tables() -> List[str] ``` Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: - `List[str]` – List[str]: A list of table names stored in the database. - `List[str]` – Returns an empty list if no tables are found. List all tables ``` # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_tables(self) -> List[str]: """Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: List[str]: A list of table names stored in the database. Returns an empty list if no tables are found. Example: List all tables ```python # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` """ return self.catalog.list_tables() ``` ### list_tools ``` list_tools() -> List[UserDefinedTool] ``` Lists the tools available in the current catalog. Source code in `src/fenic/api/catalog.py` ``` def list_tools(self) -> List[UserDefinedTool]: """Lists the tools available in the current catalog.""" return self.catalog.list_tools() ``` ### list_views ``` list_views() -> List[str] ``` Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: - `List[str]` – List[str]: A list of view names stored in the database. - `List[str]` – Returns an empty list if no views are found. Example > > > session.catalog.list_views() > > > ['view1', 'view2', 'view3']. Source code in `src/fenic/api/catalog.py` ``` def list_views(self) -> List[str]: """Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: List[str]: A list of view names stored in the database. Returns an empty list if no views are found. Example: >>> session.catalog.list_views() ['view1', 'view2', 'view3']. """ return self.catalog.list_views() ``` ### set_current_catalog ``` set_current_catalog(catalog_name: str) -> None ``` Sets the current catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to set as current. Raises: - `ValueError` – If the specified catalog doesn't exist. Set current catalog ``` # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_catalog(self, catalog_name: str) -> None: """Sets the current catalog. Args: catalog_name (str): Name of the catalog to set as current. Raises: ValueError: If the specified catalog doesn't exist. Example: Set current catalog ```python # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` """ self.catalog.set_current_catalog(catalog_name) ``` ### set_current_database ``` set_current_database(database_name: str) -> None ``` Sets the current database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to set as current. Raises: - `DatabaseNotFoundError` – If the specified database doesn't exist. Set current database ``` # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_database(self, database_name: str) -> None: """Sets the current database. Args: database_name (str): Fully qualified or relative database name to set as current. Raises: DatabaseNotFoundError: If the specified database doesn't exist. Example: Set current database ```python # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` """ self.catalog.set_current_database(database_name) ``` ### set_table_description ``` set_table_description(table_name: str, description: Optional[str] = None) -> None ``` Set or unset the description for a table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to set the description for. - **`description`** (`Optional[str]`, default: `None` ) – The description to set for the table. Raises: - `TableNotFoundError` – If the table doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_table_description(self, table_name: str, description: Optional[str] = None) -> None: """Set or unset the description for a table. Args: table_name: Fully qualified or relative table name to set the description for. description: The description to set for the table. Raises: TableNotFoundError: If the table doesn't exist. """ self.catalog.set_table_description(table_name, description) ``` ### set_view_description ``` set_view_description(view_name: str, description: Optional[str] = None) -> None ``` Set the description for a view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to set the description for. - **`description`** (`str`, default: `None` ) – The description to set for the view. Raises: - `TableNotFoundError` – If the view doesn't exist. - `ValidationError` – If the description is empty. Set a description for a view ```python #### Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_view_description(self, view_name: str, description: Optional[str] = None) -> None: """Set the description for a view. Args: view_name (str): Fully qualified or relative view name to set the description for. description (str): The description to set for the view. Raises: TableNotFoundError: If the view doesn't exist. ValidationError: If the description is empty. Example: Set a description for a view ```python # Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') """ self.catalog.set_view_description(view_name, description) ``` ## CloudConfig Bases: `BaseModel` ``` flowchart TD fenic.api.CloudConfig[CloudConfig] click fenic.api.CloudConfig href "" "fenic.api.CloudConfig" ``` Configuration for cloud-based execution. This class defines settings for running operations in a cloud environment, allowing for scalable and distributed processing of language model operations. Attributes: - **`size`** (`Optional[CloudExecutorSize]`) – Size of the cloud executor instance. If None, the default size will be used. Example Configuring cloud execution with a specific size: ``` config = CloudConfig(size=CloudExecutorSize.MEDIUM) ``` Using default cloud configuration: ``` config = CloudConfig() ``` ## CohereEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.CohereEmbeddingModel[CohereEmbeddingModel] click fenic.api.CohereEmbeddingModel href "" "fenic.api.CohereEmbeddingModel" ``` Configuration for Cohere embedding models. This class defines the configuration settings for Cohere embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`CohereEmbeddingModelName`) – The name of the Cohere model to use. - **`rpm`** (`int`) – Requests per minute limit for the model. - **`tpm`** (`int`) – Tokens per minute limit for the model. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional dictionary of profile configurations. - **`default_profile`** (`Optional[str]`) – Default profile name to use if none specified. Example Configuring a Cohere embedding model with profiles: ``` cohere_config = CohereEmbeddingModel( model_name="embed-v4.0", rpm=100, tpm=50_000, profiles={ "high_dim": CohereEmbeddingModel.Profile( embedding_dimensionality=1536, embedding_task_type="search_document" ), "classification": CohereEmbeddingModel.Profile( embedding_dimensionality=1024, embedding_task_type="classification" ), }, default_profile="high_dim", ) ``` Classes: - **`Profile`** – Profile configurations for Cohere embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.CohereEmbeddingModel.Profile[Profile] click fenic.api.CohereEmbeddingModel.Profile href "" "fenic.api.CohereEmbeddingModel.Profile" ``` Profile configurations for Cohere embedding models. This class defines profile configurations for Cohere embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`input_type`** (`CohereEmbeddingTaskType`) – The type of input text (search_query, search_document, classification, clustering) Example Configuring a profile with custom dimensionality: ``` profile = CohereEmbeddingModel.Profile(output_dimensionality=1536) ``` Configuring a profile with default settings: ``` profile = CohereEmbeddingModel.Profile() ``` ## Column A column expression in a DataFrame. This class represents a column expression that can be used in DataFrame operations. It provides methods for accessing, transforming, and combining column data. Create a column reference ``` # Reference a column by name using col() function col("column_name") ``` Use column in operations ``` # Perform arithmetic operations df.select(col("price") * col("quantity")) ``` Chain column operations ``` # Chain multiple operations df.select(col("name").upper().contains("John")) ``` Methods: - **`alias`** – Create an alias for this column. - **`asc`** – Mark this column for ascending sort order. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`cast`** – Cast the column to a new data type. - **`contains`** – Check if the column contains a substring. - **`contains_any`** – Check if the column contains any of the specified substrings. - **`desc`** – Mark this column for descending sort order. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`ends_with`** – Check if the column ends with a substring. - **`get_item`** – Access an item in a struct or array column. - **`ilike`** – Check if the column matches a SQL LIKE pattern (case-insensitive). - **`is_in`** – Check if the column is in a list of values or a column expression. - **`is_not_null`** – Check if the column contains non-NULL values. - **`is_null`** – Check if the column contains NULL values. - **`like`** – Check if the column matches a SQL LIKE pattern. - **`otherwise`** – Returns a value when no prior conditions are True. - **`rlike`** – Check if the column matches a regular expression pattern. - **`starts_with`** – Check if the column starts with a substring. - **`when`** – Evaluates a condition for each row and returns a value when true. ### alias ``` alias(name: str) -> Column ``` Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Parameters: - **`name`** (`str`) – The alias name to assign Returns: - **`Column`** ( `Column` ) – Column with the specified alias Rename a column ``` # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Name a complex expression ``` # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` Source code in `src/fenic/api/column.py` ``` def alias(self, name: str) -> Column: """Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Args: name (str): The alias name to assign Returns: Column: Column with the specified alias Example: Rename a column ```python # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Example: Name a complex expression ```python # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` """ return Column._from_logical_expr(AliasExpr(self._logical_expr, name)) ``` ### asc ``` asc() -> Column ``` Mark this column for ascending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls first. Sort by age in ascending order ``` # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc(self) -> Column: """Mark this column for ascending sort order. Returns: Column: A sort expression with ascending order and nulls first. Example: Sort by age in ascending order ```python # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` """ return Column._from_logical_expr(SortExpr(self._logical_expr, ascending=True)) ``` ### asc_nulls_first ``` asc_nulls_first() -> Column ``` Alias for asc(). Returns: - **`Column`** ( `Column` ) – A Column expression that provides a column and sort order to the sort function Source code in `src/fenic/api/column.py` ``` def asc_nulls_first(self) -> Column: """Alias for asc(). Returns: Column: A Column expression that provides a column and sort order to the sort function """ return self.asc() ``` ### asc_nulls_last ``` asc_nulls_last() -> Column ``` Mark this column for ascending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls last. Sort by age in ascending order with nulls last ``` # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc_nulls_last(self) -> Column: """Mark this column for ascending sort order with nulls last. Returns: Column: A sort expression with ascending order and nulls last. Example: Sort by age in ascending order with nulls last ```python # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=True, nulls_last=True) ) ``` ### cast ``` cast(data_type: DataType) -> Column ``` Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Parameters: - **`data_type`** (`DataType`) – The target DataType to cast the column to Returns: - **`Column`** ( `Column` ) – A Column representing the casted expression Cast integer to string ``` # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Cast array of integers to array of strings ``` # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Cast struct fields to different types ``` # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: - `TypeError` – If the requested cast operation is not supported Source code in `src/fenic/api/column.py` ``` def cast(self, data_type: DataType) -> Column: """Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Args: data_type (DataType): The target DataType to cast the column to Returns: Column: A Column representing the casted expression Example: Cast integer to string ```python # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Example: Cast array of integers to array of strings ```python # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Example: Cast struct fields to different types ```python # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: TypeError: If the requested cast operation is not supported """ return Column._from_logical_expr(CastExpr(self._logical_expr, data_type)) ``` ### contains ``` contains(other: Union[str, Column]) -> Column ``` Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to search for (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains the substring Find rows where name contains "john" ``` # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Find rows where text contains a dynamic pattern ``` # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` Source code in `src/fenic/api/column.py` ``` def contains(self, other: Union[str, Column]) -> Column: """Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to search for (can be a string or column expression) Returns: Column: A boolean column indicating whether each value contains the substring Example: Find rows where name contains "john" ```python # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Example: Find rows where text contains a dynamic pattern ```python # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ContainsExpr(self._logical_expr, other_expr)) ``` ### contains_any ``` contains_any(others: List[str], case_insensitive: bool = True) -> Column ``` Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Parameters: - **`others`** (`List[str]`) – List of substrings to search for - **`case_insensitive`** (`bool`, default: `True` ) – Whether to perform case-insensitive matching (default: True) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains any substring Find rows where name contains "john" or "jane" (case-insensitive) ``` # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Case-sensitive matching ``` # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` Source code in `src/fenic/api/column.py` ``` def contains_any(self, others: List[str], case_insensitive: bool = True) -> Column: """Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Args: others (List[str]): List of substrings to search for case_insensitive (bool): Whether to perform case-insensitive matching (default: True) Returns: Column: A boolean column indicating whether each value contains any substring Example: Find rows where name contains "john" or "jane" (case-insensitive) ```python # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Example: Case-sensitive matching ```python # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` """ return Column._from_logical_expr( ContainsAnyExpr(self._logical_expr, others, case_insensitive) ) ``` ### desc ``` desc() -> Column ``` Mark this column for descending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order. Sort by age in descending order ``` # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc(self) -> Column: """Mark this column for descending sort order. Returns: Column: A sort expression with descending order. Example: Sort by age in descending order ```python # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False) ) ``` ### desc_nulls_first ``` desc_nulls_first() -> Column ``` Alias for desc(). Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls first. Sort by age in descending order with nulls first ``` df.sort(col("age").desc_nulls_first()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_first(self) -> Column: """Alias for desc(). Returns: Column: A sort expression with descending order and nulls first. Example: Sort by age in descending order with nulls first ```python df.sort(col("age").desc_nulls_first()).show() ``` """ return self.desc() ``` ### desc_nulls_last ``` desc_nulls_last() -> Column ``` Mark this column for descending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls last. Sort by age in descending order with nulls last ``` # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_last(self) -> Column: """Mark this column for descending sort order with nulls last. Returns: Column: A sort expression with descending order and nulls last. Example: Sort by age in descending order with nulls last ```python # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False, nulls_last=True) ) ``` ### ends_with ``` ends_with(other: Union[str, Column]) -> Column ``` Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the end (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value ends with the substring Find rows where email ends with "@gmail.com" ``` df.filter(col("email").ends_with("@gmail.com")) ``` Find rows where text ends with a dynamic pattern ``` df.filter(col("text").ends_with(col("suffix"))) ``` Raises: - `ValueError` – If the substring ends with a regular expression anchor ($) Source code in `src/fenic/api/column.py` ``` def ends_with(self, other: Union[str, Column]) -> Column: """Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the end (can be a string or column expression) Returns: Column: A boolean column indicating whether each value ends with the substring Example: Find rows where email ends with "@gmail.com" ```python df.filter(col("email").ends_with("@gmail.com")) ``` Example: Find rows where text ends with a dynamic pattern ```python df.filter(col("text").ends_with(col("suffix"))) ``` Raises: ValueError: If the substring ends with a regular expression anchor ($) """ if isinstance(other, str): if other.endswith("$"): raise ValidationError("substr should not end with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(EndsWithExpr(self._logical_expr, other_expr)) ``` ### get_item ``` get_item(key: Union[str, int, Column]) -> Column ``` Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Parameters: - **`key`** (`Union[str, int]`) – The index (for arrays) or field name (for structs) to access Returns: - **`Column`** ( `Column` ) – A Column representing the accessed item Access an array element ``` # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Access a struct field ``` # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` Source code in `src/fenic/api/column.py` ``` def get_item(self, key: Union[str, int, Column]) -> Column: """Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Args: key (Union[str, int]): The index (for arrays) or field name (for structs) to access Returns: Column: A Column representing the accessed item Example: Access an array element ```python # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Example: Access a struct field ```python # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` """ if isinstance(key, Column): return Column._from_logical_expr(IndexExpr(self._logical_expr, key._logical_expr)) elif isinstance(key, str): return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, StringType))) else: return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, IntegerType))) ``` ### ilike ``` ilike(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "j" and ends with "n" (case-insensitive) ``` # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Find rows where code matches pattern (case-insensitive) ``` # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` Source code in `src/fenic/api/column.py` ``` def ilike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "j" and ends with "n" (case-insensitive) ```python # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Example: Find rows where code matches pattern (case-insensitive) ```python # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ILikeExpr(self._logical_expr, other_expr)) ``` ### is_in ``` is_in(other: Union[List[Any], ColumnOrName]) -> Column ``` Check if the column is in a list of values or a column expression. Parameters: - **`other`** (`Union[List[Any], ColumnOrName]`) – A list of values or a Column expression Returns: - **`Column`** ( `Column` ) – A Column expression representing whether each element of Column is in the list Check if name is in a list of values ``` # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Check if value is in another column ``` # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` Source code in `src/fenic/api/column.py` ``` def is_in(self, other: Union[List[Any], ColumnOrName]) -> Column: """Check if the column is in a list of values or a column expression. Args: other (Union[List[Any], ColumnOrName]): A list of values or a Column expression Returns: Column: A Column expression representing whether each element of Column is in the list Example: Check if name is in a list of values ```python # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Example: Check if value is in another column ```python # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` """ if isinstance(other, list): try: type_ = infer_dtype_from_pyobj(other) return Column._from_logical_expr(InExpr(self._logical_expr, LiteralExpr(other, type_))) except TypeInferenceError as e: raise ValidationError(f"Cannot apply IN on {other}. List argument to IN must be be a valid Python List literal.") from e else: return Column._from_logical_expr(InExpr(self._logical_expr, other._logical_expr)) ``` ### is_not_null ``` is_not_null() -> Column ``` Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is not NULL Filter rows where a column is not NULL ``` df.filter(col("some_column").is_not_null()) ``` Use in a complex condition ``` df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` Source code in `src/fenic/api/column.py` ``` def is_not_null(self) -> Column: """Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is not NULL Example: Filter rows where a column is not NULL ```python df.filter(col("some_column").is_not_null()) ``` Example: Use in a complex condition ```python df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, False)) ``` ### is_null ``` is_null() -> Column ``` Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is NULL Filter rows where a column is NULL ``` # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Use in a complex condition ``` # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` Source code in `src/fenic/api/column.py` ``` def is_null(self) -> Column: """Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is NULL Example: Filter rows where a column is NULL ```python # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Example: Use in a complex condition ```python # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, True)) ``` ### like ``` like(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "J" and ends with "n" ``` # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Find rows where code matches specific pattern ``` # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` Source code in `src/fenic/api/column.py` ``` def like(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "J" and ends with "n" ```python # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Example: Find rows where code matches specific pattern ```python # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(LikeExpr(self._logical_expr, other_expr)) ``` ### otherwise ``` otherwise(value: Column) -> Column ``` Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Parameters: - **`value`** (`Column`) – Value to return when no prior conditions are True Returns: - **`Column`** ( `Column` ) – The complete conditional expression chain Raises: - `ValidationError` – If called on a non-when expression Example ``` # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain Source code in `src/fenic/api/column.py` ``` def otherwise(self, value: Column) -> Column: """Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Args: value: Value to return when no prior conditions are True Returns: Column: The complete conditional expression chain Raises: ValidationError: If called on a non-when expression Example: ```python # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note: - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain """ return Column._from_logical_expr(OtherwiseExpr(self._logical_expr, value._logical_expr)) ``` ### rlike ``` rlike(other: Union[str, Column]) -> Column ``` Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Parameters: - **`other`** (`Union[str, Column]`) – The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where phone number matches pattern ``` # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Find rows where text contains word boundaries ``` # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` Source code in `src/fenic/api/column.py` ``` def rlike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Args: other (Union[str, Column]): The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where phone number matches pattern ```python # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Example: Find rows where text contains word boundaries ```python # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(RLikeExpr(self._logical_expr, other_expr)) ``` ### starts_with ``` starts_with(other: Union[str, Column]) -> Column ``` Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the start (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value starts with the substring Find rows where name starts with "Mr" ``` # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Find rows where text starts with a dynamic pattern ``` # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: - `ValueError` – If the substring starts with a regular expression anchor (^) Source code in `src/fenic/api/column.py` ``` def starts_with(self, other: Union[str, Column]) -> Column: """Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the start (can be a string or column expression) Returns: Column: A boolean column indicating whether each value starts with the substring Example: Find rows where name starts with "Mr" ```python # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Example: Find rows where text starts with a dynamic pattern ```python # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: ValueError: If the substring starts with a regular expression anchor (^) """ if isinstance(other, str): if other.startswith("^"): raise ValidationError("substr should not start with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(StartsWithExpr(self._logical_expr, other_expr)) ``` ### when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A new when expression with this condition added to the chain Raises: - `ValidationError` – If called on a non-when expression (e.g., regular columns) Example ``` # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. Source code in `src/fenic/api/column.py` ``` def when(self, condition: Column, value: Column) -> Column: """Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A new when expression with this condition added to the chain Raises: ValidationError: If called on a non-when expression (e.g., regular columns) Example: ```python # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. """ return Column._from_logical_expr(WhenExpr(self._logical_expr, condition._logical_expr, value._logical_expr)) ``` ## DataFrame A data collection organized into named columns. The DataFrame class represents a lazily evaluated computation on data. Operations on DataFrame build up a logical query plan that is only executed when an action like show(), to_polars(), to_pandas(), to_arrow(), to_pydict(), to_pylist(), or count() is called. The DataFrame supports method chaining for building complex transformations. Create and transform a DataFrame ``` # Create a DataFrame from a dictionary df = session.create_dataframe({"id": [1, 2, 3], "value": ["a", "b", "c"]}) # Chain transformations result = df.filter(col("id") > 1).select("id", "value") # Show results result.show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 2| b| # | 3| c| # +---+-----+ ``` Methods: - **`agg`** – Aggregate on the entire DataFrame without groups. - **`cache`** – Alias for persist(). Mark DataFrame for caching after first computation. - **`collect`** – Execute the DataFrame computation and return the result as a QueryResult. - **`count`** – Count the number of rows in the DataFrame. - **`distinct`** – Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). - **`drop`** – Remove one or more columns from this DataFrame. - **`drop_duplicates`** – Return a DataFrame with duplicate rows removed. - **`explain`** – Display the logical plan of the DataFrame. - **`explode`** – Create a new row for each element in an array column. - **`explode_outer`** – Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. - **`explode_with_index`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`filter`** – Filters rows using the given condition. - **`group_by`** – Groups the DataFrame using the specified columns. - **`join`** – Joins this DataFrame with another DataFrame. - **`limit`** – Limits the number of rows to the specified number. - **`lineage`** – Create a Lineage object to trace data through transformations. - **`order_by`** – Sort the DataFrame by the specified columns. Alias for sort(). - **`persist`** – Mark this DataFrame to be persisted after first computation. - **`posexplode`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`posexplode_outer`** – Create a new row for each element in an array column with position and value, preserving null/empty arrays. - **`select`** – Projects a set of Column expressions or column names. - **`show`** – Display the DataFrame content in a tabular form. - **`sort`** – Sort the DataFrame by the specified columns. - **`to_arrow`** – Execute the DataFrame computation and return an Apache Arrow Table. - **`to_pandas`** – Execute the DataFrame computation and return a Pandas DataFrame. - **`to_polars`** – Execute the DataFrame computation and return the result as a Polars DataFrame. - **`to_pydict`** – Execute the DataFrame computation and return a dictionary of column arrays. - **`to_pylist`** – Execute the DataFrame computation and return a list of row dictionaries. - **`union`** – Return a new DataFrame containing the union of rows in this and another DataFrame. - **`unnest`** – Unnest the specified struct columns into separate columns. - **`where`** – Filters rows using the given condition (alias for filter()). - **`with_column`** – Add a new column or replace an existing column. - **`with_column_renamed`** – Rename a column. No-op if the column does not exist. - **`with_columns`** – Add multiple new columns or replace existing columns. Attributes: - **`columns`** (`List[str]`) – Get list of column names. - **`schema`** (`Schema`) – Get the schema of this DataFrame. - **`semantic`** (`SemanticExtensions`) – Interface for semantic operations on the DataFrame. - **`write`** (`DataFrameWriter`) – Interface for saving the content of the DataFrame. ### columns ``` columns: List[str] ``` Get list of column names. Returns: - `List[str]` – List[str]: List of all column names in the DataFrame Examples: ``` >>> df.columns ['name', 'age', 'city'] ``` ### schema ``` schema: Schema ``` Get the schema of this DataFrame. Returns: - **`Schema`** ( `Schema` ) – Schema containing field names and data types Examples: ``` >>> df.schema Schema([ ColumnField('name', StringType), ColumnField('age', IntegerType) ]) ``` ### semantic ``` semantic: SemanticExtensions ``` Interface for semantic operations on the DataFrame. ### write ``` write: DataFrameWriter ``` Interface for saving the content of the DataFrame. Returns: - **`DataFrameWriter`** ( `DataFrameWriter` ) – Writer interface to write DataFrame. ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions or dictionary of aggregations. Returns: - **`DataFrame`** ( `DataFrame` ) – Aggregation results. Multiple aggregations ``` # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Dictionary style ``` # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Args: *exprs: Aggregation expressions or dictionary of aggregations. Returns: DataFrame: Aggregation results. Example: Multiple aggregations ```python # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Example: Dictionary style ```python # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` """ return self.group_by().agg(*exprs) ``` ### cache ``` cache() -> DataFrame ``` Alias for persist(). Mark DataFrame for caching after first computation. Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for caching See Also persist(): Full documentation of caching behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def cache(self) -> DataFrame: """Alias for persist(). Mark DataFrame for caching after first computation. Returns: DataFrame: Same DataFrame, but marked for caching See Also: persist(): Full documentation of caching behavior """ return self.persist() ``` ### collect ``` collect(data_type: DataLikeType = 'polars') -> QueryResult ``` Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Parameters: - **`data_type`** (`DataLikeType`, default: `'polars'` ) – The type of data to return Returns: - **`QueryResult`** ( `QueryResult` ) – A QueryResult with materialized data and query metrics Source code in `src/fenic/api/dataframe/dataframe.py` ``` def collect(self, data_type: DataLikeType = "polars") -> QueryResult: """Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Args: data_type: The type of data to return Returns: QueryResult: A QueryResult with materialized data and query metrics """ result: Tuple[pl.DataFrame, QueryMetrics] = self._session_state.execution.collect(self._logical_plan) df, metrics = result logger.info(metrics.get_summary()) if data_type == "polars": return QueryResult(df, metrics) elif data_type == "pandas": return QueryResult(df.to_pandas(use_pyarrow_extension_array=True), metrics) elif data_type == "arrow": return QueryResult(df.to_arrow(), metrics) elif data_type == "pydict": return QueryResult(df.to_dict(as_series=False), metrics) elif data_type == "pylist": return QueryResult(df.to_dicts(), metrics) else: raise ValidationError(f"Invalid data type: {data_type} in collect(). Valid data types are: polars, pandas, arrow, pydict, pylist") ``` ### count ``` count() -> int ``` Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: - **`int`** ( `int` ) – The number of rows in the DataFrame Source code in `src/fenic/api/dataframe/dataframe.py` ``` def count(self) -> int: """Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: int: The number of rows in the DataFrame """ return self._session_state.execution.count(self._logical_plan)[0] ``` ### distinct ``` distinct() -> DataFrame ``` Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Example ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def distinct(self) -> DataFrame: """Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: DataFrame: A new DataFrame with duplicate rows removed. Example: ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ return self.drop_duplicates() ``` ### drop ``` drop(*col_names: str) -> DataFrame ``` Remove one or more columns from this DataFrame. Parameters: - **`*col_names`** (`str`, default: `()` ) – Names of columns to drop. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame without specified columns. Raises: - `ValueError` – If any specified column doesn't exist in the DataFrame. - `ValueError` – If dropping the columns would result in an empty DataFrame. Drop single column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Drop multiple columns ``` # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Error when dropping non-existent column ``` # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop(self, *col_names: str) -> DataFrame: """Remove one or more columns from this DataFrame. Args: *col_names: Names of columns to drop. Returns: DataFrame: New DataFrame without specified columns. Raises: ValueError: If any specified column doesn't exist in the DataFrame. ValueError: If dropping the columns would result in an empty DataFrame. Example: Drop single column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Drop multiple columns ```python # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Example: Error when dropping non-existent column ```python # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` """ if not col_names: return self current_cols = set(self.columns) to_drop = set(col_names) missing = to_drop - current_cols if missing: missing_str = ( f"Column '{next(iter(missing))}'" if len(missing) == 1 else f"Columns {sorted(missing)}" ) raise ValueError(f"{missing_str} not found in DataFrame") remaining_cols = [ col(c)._logical_expr for c in self.columns if c not in to_drop ] if not remaining_cols: raise ValueError("Cannot drop all columns from DataFrame") return self._from_logical_plan( Projection.from_session_state(self._logical_plan, remaining_cols, self._session_state), self._session_state, ) ``` ### drop_duplicates ``` drop_duplicates(subset: Optional[List[str]] = None) -> DataFrame ``` Return a DataFrame with duplicate rows removed. Parameters: - **`subset`** (`Optional[List[str]]`, default: `None` ) – Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Raises: - `ValueError` – If a specified column is not present in the current DataFrame schema. Remove duplicates considering specific columns ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop_duplicates( self, subset: Optional[List[str]] = None, ) -> DataFrame: """Return a DataFrame with duplicate rows removed. Args: subset: Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: DataFrame: A new DataFrame with duplicate rows removed. Raises: ValueError: If a specified column is not present in the current DataFrame schema. Example: Remove duplicates considering specific columns ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ exprs = [] if subset: for c in subset: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( DropDuplicates.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### explain ``` explain() -> None ``` Display the logical plan of the DataFrame. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explain(self) -> None: """Display the logical plan of the DataFrame.""" print(str(self._logical_plan)) ``` ### explode ``` explode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. Raises: - `TypeError` – If column argument is not a string or Column. Explode array column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Using column expression ``` # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Raises: TypeError: If column argument is not a string or Column. Example: Explode array column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Example: Using column expression ```python # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state(self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state), self._session_state, ) ``` ### explode_outer ``` explode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. - `DataFrame` – Rows with null or empty arrays are preserved with null in the exploded column. Explode with outer join behavior ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Rows with null or empty arrays are preserved with null in the exploded column. Example: Explode with outer join behavior ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state, keep_null_and_empty=True ), self._session_state, ) ``` ### explode_with_index ``` explode_with_index(column: ColumnOrName, index_col_name: str = 'pos', value_col_name: str = 'col', keep_null_and_empty: bool = False) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. - **`index_col_name`** (`str`, default: `'pos'` ) – Name for the column containing 0-based array positions (default: "pos"). - **`value_col_name`** (`str`, default: `'col'` ) – Name for the exploded value column (default: "col"). - **`keep_null_and_empty`** (`bool`, default: `False` ) – If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Explode with index ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Custom column names ``` df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_with_index( self, column: ColumnOrName, index_col_name: str = "pos", value_col_name: str = "col", keep_null_and_empty: bool = False, ) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Args: column: Name of array column to explode (as string) or Column expression. index_col_name: Name for the column containing 0-based array positions (default: "pos"). value_col_name: Name for the exploded value column (default: "col"). keep_null_and_empty: If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: DataFrame: New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Example: Explode with index ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Example: Custom column names ```python df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` """ return self._from_logical_plan( ExplodeWithIndex.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, index_col_name, value_col_name, self._session_state, keep_null_and_empty, ), self._session_state, ) ``` ### filter ``` filter(condition: Column) -> DataFrame ``` Filters rows using the given condition. Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame Filter with numeric comparison ``` # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with semantic predicate ``` # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with multiple conditions ``` # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def filter(self, condition: Column) -> DataFrame: """Filters rows using the given condition. Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame Example: Filter with numeric comparison ```python # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with semantic predicate ```python # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with multiple conditions ```python # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` """ return self._from_logical_plan( Filter.from_session_state(self._logical_plan, condition._logical_expr, self._session_state), self._session_state, ) ``` ### group_by ``` group_by(*cols: ColumnOrName) -> GroupedData ``` Groups the DataFrame using the specified columns. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns to group by. Can be column names as strings or Column expressions. Returns: - **`GroupedData`** ( `GroupedData` ) – Object for performing aggregations on the grouped data. Group by single column ``` # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Group by multiple columns ``` # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Group by expression ``` # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def group_by(self, *cols: ColumnOrName) -> GroupedData: """Groups the DataFrame using the specified columns. Args: *cols: Columns to group by. Can be column names as strings or Column expressions. Returns: GroupedData: Object for performing aggregations on the grouped data. Example: Group by single column ```python # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Example: Group by multiple columns ```python # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Example: Group by expression ```python # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` """ return GroupedData(self, list(cols) if cols else None) ``` ### join ``` join(other: DataFrame, on: Union[str, List[str]], *, how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, *, left_on: Union[ColumnOrName, List[ColumnOrName]], right_on: Union[ColumnOrName, List[ColumnOrName]], how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = 'inner') -> DataFrame ``` Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Parameters: - **`other`** (`DataFrame`) – DataFrame to join with. - **`on`** (`Optional[Union[str, List[str]]]`, default: `None` ) – Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins - **`left_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions - **`right_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions - **`how`** (`JoinType`, default: `'inner'` ) – Type of join to perform. Returns: - `DataFrame` – Joined DataFrame. Raises: - `ValidationError` – If cross join is used with an ON clause. - `ValidationError` – If join condition is invalid. - `ValidationError` – If both 'on' and 'left_on'/'right_on' parameters are provided. - `ValidationError` – If only one of 'left_on' or 'right_on' is provided. - `ValidationError` – If 'left_on' and 'right_on' have different lengths Inner join on column name ``` # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Join with expression ``` # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Cross join ``` # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def join( self, other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = "inner", ) -> DataFrame: """Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Args: other: DataFrame to join with. on: Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins left_on: Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions right_on: Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions how: Type of join to perform. Returns: Joined DataFrame. Raises: ValidationError: If cross join is used with an ON clause. ValidationError: If join condition is invalid. ValidationError: If both 'on' and 'left_on'/'right_on' parameters are provided. ValidationError: If only one of 'left_on' or 'right_on' is provided. ValidationError: If 'left_on' and 'right_on' have different lengths Example: Inner join on column name ```python # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Join with expression ```python # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Cross join ```python # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` """ validate_join_parameters(self, on, left_on, right_on, how) # Build join conditions left_conditions, right_conditions = build_join_conditions(on, left_on, right_on) self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( Join.from_session_state( self._logical_plan, other._logical_plan, left_conditions, right_conditions, how, self._session_state), self._session_state, ) ``` ### limit ``` limit(n: int) -> DataFrame ``` Limits the number of rows to the specified number. Parameters: - **`n`** (`int`) – Maximum number of rows to return. Returns: - **`DataFrame`** ( `DataFrame` ) – DataFrame with at most n rows. Raises: - `TypeError` – If n is not an integer. Limit rows ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Limit with other operations ``` # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def limit(self, n: int) -> DataFrame: """Limits the number of rows to the specified number. Args: n: Maximum number of rows to return. Returns: DataFrame: DataFrame with at most n rows. Raises: TypeError: If n is not an integer. Example: Limit rows ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Limit with other operations ```python # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` """ return self._from_logical_plan( Limit.from_session_state(self._logical_plan, n, self._session_state), self._session_state) ``` ### lineage ``` lineage() -> Lineage ``` Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: - **`Lineage`** ( `Lineage` ) – Interface for querying data lineage Example ``` # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also LineageQuery: Full documentation of lineage querying capabilities Source code in `src/fenic/api/dataframe/dataframe.py` ``` def lineage(self) -> Lineage: """Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: Lineage: Interface for querying data lineage Example: ```python # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also: LineageQuery: Full documentation of lineage querying capabilities """ return Lineage(self._session_state.execution.build_lineage(self._logical_plan)) ``` ### order_by ``` order_by(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Alias for sort(). Returns: - **`DataFrame`** ( `DataFrame` ) – sorted Dataframe. See Also sort(): Full documentation of sorting behavior and parameters. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def order_by( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Alias for sort(). Returns: DataFrame: sorted Dataframe. See Also: sort(): Full documentation of sorting behavior and parameters. """ return self.sort(cols, ascending) ``` ### persist ``` persist() -> DataFrame ``` Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for persistence Example ``` # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def persist(self) -> DataFrame: """Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: DataFrame: Same DataFrame, but marked for persistence Example: ```python # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` """ cache_info = CacheInfo(cache_key=f"cache_{uuid.uuid4().hex}") self._logical_plan.set_cache_info(cache_info) return self._from_logical_plan( self._logical_plan, self._session_state) ``` ### posexplode ``` posexplode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. PySpark-style posexplode ``` df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Example: PySpark-style posexplode ```python df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` """ return self.explode_with_index(column) ``` ### posexplode_outer ``` posexplode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. - `DataFrame` – Rows with null or empty arrays are preserved with (null, null). PySpark-style posexplode_outer ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Rows with null or empty arrays are preserved with (null, null). Example: PySpark-style posexplode_outer ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` """ return self.explode_with_index( column, keep_null_and_empty=True ) ``` ### select ``` select(*cols: ColumnOrName) -> DataFrame ``` Projects a set of Column expressions or column names. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with selected columns Select by column names ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Select with expressions ``` # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Mix strings and expressions ``` # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def select(self, *cols: ColumnOrName) -> DataFrame: """Projects a set of Column expressions or column names. Args: *cols: Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: DataFrame: A new DataFrame with selected columns Example: Select by column names ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Example: Select with expressions ```python # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Example: Mix strings and expressions ```python # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` """ exprs = [] if not cols: return self for c in cols: if isinstance(c, str): if c == "*": exprs.extend(col(field)._logical_expr for field in self.columns) else: exprs.append(col(c)._logical_expr) else: exprs.append(c._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### show ``` show(n: int = 10, explain_analyze: bool = False) -> None ``` Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Parameters: - **`n`** (`int`, default: `10` ) – Number of rows to display - **`explain_analyze`** (`bool`, default: `False` ) – Whether to print the explain analyze plan Source code in `src/fenic/api/dataframe/dataframe.py` ``` def show(self, n: int = 10, explain_analyze: bool = False) -> None: """Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Args: n: Number of rows to display explain_analyze: Whether to print the explain analyze plan """ output, metrics = self._session_state.execution.show(self._logical_plan, n) logger.info(metrics.get_summary()) print(output) if explain_analyze: print(metrics.get_execution_plan_details()) ``` ### sort ``` sort(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Parameters: - **`cols`** (`Union[ColumnOrName, List[ColumnOrName], None]`, default: `None` ) – Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. - **`ascending`** (`Optional[Union[bool, List[bool]]]`, default: `None` ) – A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame sorted by the specified columns. Raises: - `ValueError` – - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used - `TypeError` – - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Sort in ascending order ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Sort in descending order ``` # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Sort with boolean ascending parameter ``` # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Multiple columns with different sort orders ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Multiple columns with list of ascending strategies ``` # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def sort( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Args: cols: Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. ascending: A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: DataFrame: A new DataFrame sorted by the specified columns. Raises: ValueError: - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used TypeError: - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Example: Sort in ascending order ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Example: Sort in descending order ```python # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Sort with boolean ascending parameter ```python # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Multiple columns with different sort orders ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Example: Multiple columns with list of ascending strategies ```python # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` """ col_args = cols if cols is None: return self._from_logical_plan( Sort.from_session_state(self._logical_plan, [], self._session_state), self._session_state, ) elif not isinstance(cols, List): col_args = [cols] # parse the ascending arguments bool_ascending = [] using_default_ascending = False if ascending is None: using_default_ascending = True bool_ascending = [True] * len(col_args) elif isinstance(ascending, bool): bool_ascending = [ascending] * len(col_args) elif isinstance(ascending, List): bool_ascending = ascending if len(bool_ascending) != len(cols): raise ValueError( f"the list length of ascending sort strategies must match the specified sort columns" f"Got {len(cols)} column expressions and {len(bool_ascending)} ascending strategies. " ) else: raise TypeError( f"Invalid ascending strategy type: {type(ascending)}. Must be a boolean or list of booleans." ) # create our list of sort expressions, for each column expression # that isn't already provided as a asc()/desc() SortExpr sort_exprs = [] for c, asc_bool in zip(col_args, bool_ascending, strict=True): if isinstance(c, ColumnOrName): c_expr = Column._from_col_or_name(c)._logical_expr else: raise TypeError( f"Invalid column type: {type(c).__name__}. Must be a string or Column Expression." ) if not isinstance(asc_bool, bool): raise TypeError( f"Invalid ascending strategy type: {type(asc_bool).__name__}. Must be a boolean." ) if isinstance(c_expr, SortExpr): if not using_default_ascending: raise TypeError( "Cannot specify both asc()/desc() expressions and boolean ascending strategies." f"Got expression: {c_expr} and ascending argument: {bool_ascending}" ) sort_exprs.append(c_expr) else: sort_exprs.append(SortExpr(c_expr, ascending=asc_bool)) return self._from_logical_plan( Sort.from_session_state(self._logical_plan, sort_exprs, self._session_state), self._session_state, ) ``` ### to_arrow ``` to_arrow() -> pa.Table ``` Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: - `Table` – pa.Table: An Apache Arrow Table containing the computed results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_arrow(self) -> pa.Table: """Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: pa.Table: An Apache Arrow Table containing the computed results """ return self.collect("arrow").data ``` ### to_pandas ``` to_pandas() -> pd.DataFrame ``` Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: - `DataFrame` – pd.DataFrame: A Pandas DataFrame containing the computed results with Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pandas(self) -> pd.DataFrame: """Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: pd.DataFrame: A Pandas DataFrame containing the computed results with """ return self.collect("pandas").data ``` ### to_polars ``` to_polars() -> pl.DataFrame ``` Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: - `DataFrame` – pl.DataFrame: A Polars DataFrame with materialized results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_polars(self) -> pl.DataFrame: """Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: pl.DataFrame: A Polars DataFrame with materialized results """ return self.collect("polars").data ``` ### to_pydict ``` to_pydict() -> Dict[str, List[Any]] ``` Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: - `Dict[str, List[Any]]` – Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pydict(self) -> Dict[str, List[Any]]: """Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column """ return self.collect("pydict").data ``` ### to_pylist ``` to_pylist() -> List[Dict[str, Any]] ``` Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: - `List[Dict[str, Any]]` – List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pylist(self) -> List[Dict[str, Any]]: """Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result """ return self.collect("pylist").data ``` ### union ``` union(other: DataFrame) -> DataFrame ``` Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Parameters: - **`other`** (`DataFrame`) – Another DataFrame with the same schema. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing rows from both DataFrames. Raises: - `ValueError` – If the DataFrames have different schemas. - `TypeError` – If other is not a DataFrame. Union two DataFrames ``` # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Union with duplicates ``` # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def union(self, other: DataFrame) -> DataFrame: """Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Args: other: Another DataFrame with the same schema. Returns: DataFrame: A new DataFrame containing rows from both DataFrames. Raises: ValueError: If the DataFrames have different schemas. TypeError: If other is not a DataFrame. Example: Union two DataFrames ```python # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Example: Union with duplicates ```python # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` """ self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( UnionLogicalPlan.from_session_state([self._logical_plan, other._logical_plan], self._session_state), self._session_state, ) ``` ### unnest ``` unnest(*col_names: str) -> DataFrame ``` Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Parameters: - **`*col_names`** (`str`, default: `()` ) – One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with the specified struct columns expanded. Raises: - `TypeError` – If any argument is not a string or Column. - `ValueError` – If a specified column does not contain struct data. Unnest struct column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Unnest multiple struct columns ``` # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def unnest(self, *col_names: str) -> DataFrame: """Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Args: *col_names: One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: DataFrame: A new DataFrame with the specified struct columns expanded. Raises: TypeError: If any argument is not a string or Column. ValueError: If a specified column does not contain struct data. Example: Unnest struct column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Example: Unnest multiple struct columns ```python # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` """ if not col_names: return self exprs = [] for c in col_names: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( Unnest.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### where ``` where(condition: Column) -> DataFrame ``` Filters rows using the given condition (alias for filter()). Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame See Also filter(): Full documentation of filtering behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def where(self, condition: Column) -> DataFrame: """Filters rows using the given condition (alias for filter()). Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame See Also: filter(): Full documentation of filtering behavior """ return self.filter(condition) ``` ### with_column ``` with_column(col_name: str, col: Union[Any, Column, Series, Series]) -> DataFrame ``` Add a new column or replace an existing column. Parameters: - **`col_name`** (`str`) – Name of the new column - **`col`** (`Union[Any, Column, Series, Series]`) – Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced column Raises: - `ExecutionError` – - If a Series length does not match the DataFrame height - `ValidationError` – - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Add literal column ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Add computed column ``` # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Replace existing column ``` # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Add column with complex expression ``` # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Add column from Polars Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Add column from pandas Series ``` import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column(self, col_name: str, col: Union[Any, Column, pl.Series, pd.Series]) -> DataFrame: """Add a new column or replace an existing column. Args: col_name: Name of the new column col: Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced column Raises: ExecutionError: - If a Series length does not match the DataFrame height ValidationError: - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Example: Add literal column ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Example: Add computed column ```python # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Example: Replace existing column ```python # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Example: Add column with complex expression ```python # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Example: Add column from Polars Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Example: Add column from pandas Series ```python import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` """ exprs = [] # Handle different input types: Column, Series, or literal value if isinstance(col, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col = Column._from_logical_expr(SeriesLiteralExpr(col)) elif not isinstance(col, Column): # Wrap other values as literals col = lit(col) for field in self.columns: if field != col_name: exprs.append(Column._from_column_name(field)._logical_expr) # Add the new column with alias exprs.append(col.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_column_renamed ``` with_column_renamed(col_name: str, new_col_name: str) -> DataFrame ``` Rename a column. No-op if the column does not exist. Parameters: - **`col_name`** (`str`) – Name of the column to rename. - **`new_col_name`** (`str`) – New name for the column. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the column renamed. Rename a column ``` # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Rename multiple columns ``` # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column_renamed(self, col_name: str, new_col_name: str) -> DataFrame: """Rename a column. No-op if the column does not exist. Args: col_name: Name of the column to rename. new_col_name: New name for the column. Returns: DataFrame: New DataFrame with the column renamed. Example: Rename a column ```python # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Example: Rename multiple columns ```python # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` """ exprs = [] renamed = False for field in self.schema.column_fields: name = field.name if name == col_name: exprs.append(col(name).alias(new_col_name)._logical_expr) renamed = True else: exprs.append(col(name)._logical_expr) if not renamed: return self return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_columns ``` with_columns(cols_map: Dict[str, Union[Any, Column, Series, Series]]) -> DataFrame ``` Add multiple new columns or replace existing columns. Parameters: - **`cols_map`** (`Dict[str, Union[Any, Column, Series, Series]]`) – A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced columns Raises: - `ValueError` – - If two columns being created in the same `with_columns` call depend on each other - `ExecutionError` – - If any Series length does not match the DataFrame height - `ValidationError` – - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Add multiple columns ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Replace and add columns ``` # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Complex expressions ``` # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Add columns from Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Mix Series with Column expressions ``` import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Error when adding columns that depend on each other ``` df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_columns(self, cols_map: Dict[str, Union[Any, Column, pl.Series, pd.Series]]) -> DataFrame: """Add multiple new columns or replace existing columns. Args: cols_map: A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced columns Raises: ValueError: - If two columns being created in the same `with_columns` call depend on each other ExecutionError: - If any Series length does not match the DataFrame height ValidationError: - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Example: Add multiple columns ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Example: Replace and add columns ```python # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Example: Complex expressions ```python # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Example: Add columns from Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Example: Mix Series with Column expressions ```python import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Example: Error when adding columns that depend on each other ```python df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` """ if not cols_map: return self exprs = [] new_col_names = set(cols_map.keys()) # Add existing columns that are not being replaced for field in self.columns: if field not in new_col_names: exprs.append(Column._from_column_name(field)._logical_expr) # Add all new columns with aliases for col_name, col_expr in cols_map.items(): # Handle different input types: Column, Series, or literal value if isinstance(col_expr, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col_expr = Column._from_logical_expr(SeriesLiteralExpr(col_expr)) elif not isinstance(col_expr, Column): # Automatically wrap non-Column values (literals) with lit() for convenience # This allows users to pass raw Python values like: {"constant": 100, "status": "active"} # instead of requiring: {"constant": lit(100), "status": lit("active")} col_expr = lit(col_expr) exprs.append(col_expr.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ## DataFrameReader ``` DataFrameReader(session_state: BaseSessionState) ``` Interface used to load a DataFrame from external storage systems. Similar to PySpark's DataFrameReader. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Hugging Face Datasets (hf://) - Format: hf://{repo_type}/{repo_id}/{path_to_file} - Notes: - Supports glob patterns (*,* \*) - Supports dataset revisions and branch aliases (e.g., @refs/convert/parquet, @~parquet) - HF_TOKEN environment variable is required to read private datasets. - Examples: - hf://datasets/datasets-examples/doc-formats-csv-1/data.csv - hf://datasets/cais/mmlu/astronomy/\*.parquet - hf://datasets/datasets-examples/doc-formats-csv-1@~parquet/\**/*.parquet - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Creates a DataFrameReader. Parameters: - **`session_state`** (`BaseSessionState`) – The session state to use for reading Methods: - **`csv`** – Load a DataFrame from one or more CSV files. - **`docs`** – Load a DataFrame with the document contents of a list of paths (markdown or json). - **`parquet`** – Load a DataFrame from one or more Parquet files. - **`pdf_metadata`** – Load a DataFrame with metadata of PDF files in a list of paths. Source code in `src/fenic/api/io/reader.py` ``` def __init__(self, session_state: BaseSessionState): """Creates a DataFrameReader. Args: session_state: The session state to use for reading """ self._options: Dict[str, Any] = {} self._session_state = session_state ``` ### csv ``` csv(paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more CSV files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.csv"), or a list of paths. - **`schema`** (`Optional[Schema]`, default: `None` ) – (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. - **`merge_schemas`** (`bool`, default: `False` ) – Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If both `schema` and `merge_schemas=True` are provided. - `ValidationError` – If any path does not end with `.csv`. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single CSV file ``` df = session.read.csv("file.csv") ``` Read multiple CSV files with schema merging ``` df = session.read.csv("data/*.csv", merge_schemas=True) ``` Read CSV files with explicit schema `python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) )` Source code in `src/fenic/api/io/reader.py` ``` def csv( self, paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more CSV files. Args: paths: A single file path, a glob pattern (e.g., "data/*.csv"), or a list of paths. schema: (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. merge_schemas: Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: ValidationError: If both `schema` and `merge_schemas=True` are provided. ValidationError: If any path does not end with `.csv`. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single CSV file ```python df = session.read.csv("file.csv") ``` Example: Read multiple CSV files with schema merging ```python df = session.read.csv("data/*.csv", merge_schemas=True) ``` Example: Read CSV files with explicit schema ```python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) ) ``` """ if schema is not None and merge_schemas: raise ValidationError( "Cannot specify both 'schema' and 'merge_schemas=True' - these options conflict. " "Choose one approach: " "1) Use 'schema' to enforce a specific schema: csv(paths, schema=your_schema), " "2) Use 'merge_schemas=True' to automatically merge schemas: csv(paths, merge_schemas=True), " "3) Use neither to inherit schema from the first file: csv(paths)" ) if schema is not None: for col_field in schema.column_fields: if not isinstance( col_field.data_type, _PrimitiveType, ): raise ValidationError( f"CSV files only support primitive data types in schema definitions. " f"Column '{col_field.name}' has type {type(col_field.data_type).__name__}, but CSV schemas must use: " f"IntegerType, FloatType, DoubleType, BooleanType, or StringType. " f"Example: Schema([ColumnField(name='id', data_type=IntegerType), ColumnField(name='name', data_type=StringType)])" ) options = { "merge_schemas": merge_schemas, } if schema: options["schema"] = schema return self._read_file( paths, file_format="csv", file_extension=".csv", **options ) ``` ### docs ``` docs(paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal['markdown', 'json'], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with the document contents of a list of paths (markdown or json). Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`content_type`** (`Literal['markdown', 'json']`) – Content type of the files. One of "markdown" or "json". - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.md` or `.json` depending on the content_type. - `UnsupportedFileTypeError` – If the specified content_type is not "markdown" or "json" . Notes - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read all the markdown files in a folder and all its subfolders. ``` df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Read a folder of markdown files excluding some files. ``` df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` Source code in `src/fenic/api/io/reader.py` ``` def docs( self, paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal["markdown", "json"], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with the document contents of a list of paths (markdown or json). Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. content_type: Content type of the files. One of "markdown" or "json". exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.md` or `.json` depending on the content_type. UnsupportedFileTypeError: If the specified content_type is not "markdown" or "json" . Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read all the markdown files in a folder and all its subfolders. ```python df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Example: Read a folder of markdown files excluding some files. ```python df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) if content_type not in ["markdown", "json"]: raise UnsupportedFileTypeError(f"{content_type}, must be 'markdown' or 'json'") logical_node = DocSource.from_session_state( paths=path_str_list, content_type=content_type, exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ### parquet ``` parquet(paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more Parquet files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.parquet"), or a list of paths. - **`merge_schemas`** (`bool`, default: `False` ) – If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - Date and datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If any file does not have a `.parquet` extension. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single Parquet file ``` df = session.read.parquet("file.parquet") ``` Read multiple Parquet files ``` df = session.read.parquet("data/*.parquet") ``` Read Parquet files with schema merging ``` df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` Source code in `src/fenic/api/io/reader.py` ``` def parquet( self, paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more Parquet files. Args: paths: A single file path, a glob pattern (e.g., "data/*.parquet"), or a list of paths. merge_schemas: If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior: - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - Date and datetime columns are cast to strings during ingestion. Raises: ValidationError: If any file does not have a `.parquet` extension. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single Parquet file ```python df = session.read.parquet("file.parquet") ``` Example: Read multiple Parquet files ```python df = session.read.parquet("data/*.parquet") ``` Example: Read Parquet files with schema merging ```python df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` """ options = { "merge_schemas": merge_schemas, } return self._read_file( paths, file_format="parquet", file_extension=".parquet", **options ) ``` ### pdf_metadata ``` pdf_metadata(paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with metadata of PDF files in a list of paths. Note Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.pdf` extension. Notes - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read the metadata of all the PDF files in a folder and all its subfolders. ``` df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Read a metadata of PDFS in a folder, excluding some files. ``` df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` Source code in `src/fenic/api/io/reader.py` ``` def pdf_metadata( self, paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with metadata of PDF files in a list of paths. Note: Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.pdf` extension. Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read the metadata of all the PDF files in a folder and all its subfolders. ```python df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Example: Read a metadata of PDFS in a folder, excluding some files. ```python df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) logical_node = DocSource.from_session_state( paths=path_str_list, content_type="pdf", exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ## DataFrameWriter ``` DataFrameWriter(dataframe: DataFrame) ``` Interface used to write a DataFrame to external storage systems. Similar to PySpark's DataFrameWriter. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Initialize a DataFrameWriter. Parameters: - **`dataframe`** (`DataFrame`) – The DataFrame to write. Methods: - **`csv`** – Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. - **`parquet`** – Saves the content of the DataFrame as a single Parquet file. - **`save_as_table`** – Saves the content of the DataFrame as the specified table. - **`save_as_view`** – Saves the content of the DataFrame as a view. Source code in `src/fenic/api/io/writer.py` ``` def __init__(self, dataframe: DataFrame): """Initialize a DataFrameWriter. Args: dataframe: The DataFrame to write. """ self._dataframe = dataframe ``` ### csv ``` csv(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the CSV file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.csv("output.csv") # Overwrites if exists ``` Save with error mode ``` df.write.csv("output.csv", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.csv("output.csv", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def csv( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Args: file_path: Path to save the CSV file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.csv("output.csv") # Overwrites if exists ``` Example: Save with error mode ```python df.write.csv("output.csv", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.csv("output.csv", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".csv"): raise ValidationError( f"CSV writer requires a '.csv' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="csv", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### parquet ``` parquet(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single Parquet file. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the Parquet file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.parquet("output.parquet") # Overwrites if exists ``` Save with error mode ``` df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def parquet( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single Parquet file. Args: file_path: Path to save the Parquet file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.parquet("output.parquet") # Overwrites if exists ``` Example: Save with error mode ```python df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".parquet"): raise ValidationError( f"Parquet writer requires a '.parquet' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="parquet", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_table ``` save_as_table(table_name: str, mode: Literal['error', 'append', 'overwrite', 'ignore'] = 'error') -> QueryMetrics ``` Saves the content of the DataFrame as the specified table. Parameters: - **`table_name`** (`str`) – Name of the table to save to - **`mode`** (`Literal['error', 'append', 'overwrite', 'ignore']`, default: `'error'` ) – Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with error mode (default) ``` df.write.save_as_table("my_table") # Raises error if table exists ``` Save with append mode ``` df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Save with overwrite mode ``` df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` Source code in `src/fenic/api/io/writer.py` ``` def save_as_table( self, table_name: str, mode: Literal["error", "append", "overwrite", "ignore"] = "error", ) -> QueryMetrics: """Saves the content of the DataFrame as the specified table. Args: table_name: Name of the table to save to mode: Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: QueryMetrics: The query metrics Example: Save with error mode (default) ```python df.write.save_as_table("my_table") # Raises error if table exists ``` Example: Save with append mode ```python df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Example: Save with overwrite mode ```python df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` """ sink_plan = TableSink.from_session_state( child=self._dataframe._logical_plan, table_name=table_name, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_as_table( sink_plan, table_name=table_name, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_view ``` save_as_view(view_name: str, description: str | None = None) -> None ``` Saves the content of the DataFrame as a view. Parameters: - **`view_name`** (`str`) – Name of the view to save to - **`description`** (`str | None`, default: `None` ) – Optional human-readable view description to store in the catalog. Returns: - `None` – None. Source code in `src/fenic/api/io/writer.py` ``` def save_as_view( self, view_name: str, description: str | None = None, ) -> None: """Saves the content of the DataFrame as a view. Args: view_name: Name of the view to save to description: Optional human-readable view description to store in the catalog. Returns: None. """ self._dataframe._session_state.execution.save_as_view( logical_plan=self._dataframe._logical_plan, view_name=view_name, view_description=description ) ``` ## GoogleDeveloperEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleDeveloperEmbeddingModel[GoogleDeveloperEmbeddingModel] click fenic.api.GoogleDeveloperEmbeddingModel href "" "fenic.api.GoogleDeveloperEmbeddingModel" ``` Configuration for Google Developer embedding models. This class defines the configuration settings for Google embedding models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperEmbeddingModelName`) – The name of the Google Developer embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer embedding model with rate limits: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Developer embedding model with profiles: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleDeveloperEmbeddingModelConfig.Profile(), "high_dim": GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleDeveloperEmbeddingModel.Profile[Profile] click fenic.api.GoogleDeveloperEmbeddingModel.Profile href "" "fenic.api.GoogleDeveloperEmbeddingModel.Profile" ``` Profile configurations for Google Developer embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile() ``` ## GoogleDeveloperLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleDeveloperLanguageModel[GoogleDeveloperLanguageModel] click fenic.api.GoogleDeveloperLanguageModel href "" "fenic.api.GoogleDeveloperLanguageModel" ``` Configuration for Gemini models accessible through Google Developer AI Studio. This class defines the configuration settings for Google Gemini models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperLanguageModelName`) – The name of the Google Developer model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer model with rate limits: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Developer model with profiles: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleDeveloperLanguageModel.Profile(), "fast": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=1024 ), "thorough": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleDeveloperLanguageModel.Profile[Profile] click fenic.api.GoogleDeveloperLanguageModel.Profile href "" "fenic.api.GoogleDeveloperLanguageModel.Profile" ``` Profile configurations for Google Developer models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_level="high") ``` ## GoogleVertexEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleVertexEmbeddingModel[GoogleVertexEmbeddingModel] click fenic.api.GoogleVertexEmbeddingModel href "" "fenic.api.GoogleVertexEmbeddingModel" ``` Configuration for Google Vertex AI embedding models. This class defines the configuration settings for Google embedding models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexEmbeddingModelName`) – The name of the Google Vertex embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex embedding model with rate limits: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Vertex embedding model with profiles: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleVertexEmbeddingModel.Profile(), "high_dim": GoogleVertexEmbeddingModel.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleVertexEmbeddingModel.Profile[Profile] click fenic.api.GoogleVertexEmbeddingModel.Profile href "" "fenic.api.GoogleVertexEmbeddingModel.Profile" ``` Profile configurations for Google Vertex embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleVertexEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleVertexEmbeddingModelConfig.Profile() ``` ## GoogleVertexLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleVertexLanguageModel[GoogleVertexLanguageModel] click fenic.api.GoogleVertexLanguageModel href "" "fenic.api.GoogleVertexLanguageModel" ``` Configuration for Google Vertex AI models. This class defines the configuration settings for Google Gemini models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexLanguageModelName`) – The name of the Google Vertex model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex model with rate limits: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Vertex model with profiles: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleVertexLanguageModel.Profile(), "fast": GoogleVertexLanguageModel.Profile(thinking_token_budget=1024), "thorough": GoogleVertexLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.GoogleVertexLanguageModel.Profile[Profile] click fenic.api.GoogleVertexLanguageModel.Profile href "" "fenic.api.GoogleVertexLanguageModel.Profile" ``` Profile configurations for Google Vertex models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same underlying model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleVertexLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleVertexLanguageModel.Profile(thinking_level="high") ``` ## GroupedData ``` GroupedData(df: DataFrame, by: Optional[List[ColumnOrName]] = None) ``` Bases: `BaseGroupedData` ``` flowchart TD fenic.api.GroupedData[GroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData[BaseGroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData --> fenic.api.GroupedData click fenic.api.GroupedData href "" "fenic.api.GroupedData" click fenic.api.dataframe._base_grouped_data.BaseGroupedData href "" "fenic.api.dataframe._base_grouped_data.BaseGroupedData" ``` Methods for aggregations on a grouped DataFrame. Initialize grouped data. Parameters: - **`df`** (`DataFrame`) – The DataFrame to group. - **`by`** (`Optional[List[ColumnOrName]]`, default: `None` ) – Optional list of columns to group by. Methods: - **`agg`** – Compute aggregations on grouped data and return the result as a DataFrame. Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def __init__(self, df: DataFrame, by: Optional[List[ColumnOrName]] = None): """Initialize grouped data. Args: df: The DataFrame to group. by: Optional list of columns to group by. """ super().__init__(df) self._by: List[Column] = [] for c in by or []: if isinstance(c, str): self._by.append(col(c)) elif isinstance(c, Column): # Allow any expression except literals if isinstance(c._logical_expr, LiteralExpr): raise ValueError(f"Cannot group by literal value: {c}") self._by.append(c) else: raise TypeError( f"Group by expressions must be string or Column, got {type(c)}" ) self._by_exprs = [c._logical_expr for c in self._by] ``` ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with one row per group and columns for group keys and aggregated values Raises: - `ValueError` – If arguments are not Column expressions or a dictionary - `ValueError` – If dictionary values are not valid aggregate function names Count employees by department ``` # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Multiple aggregations ``` # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Dictionary style aggregations ``` # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Args: *exprs: Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: DataFrame: A new DataFrame with one row per group and columns for group keys and aggregated values Raises: ValueError: If arguments are not Column expressions or a dictionary ValueError: If dictionary values are not valid aggregate function names Example: Count employees by department ```python # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Example: Multiple aggregations ```python # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Example: Dictionary style aggregations ```python # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` """ self._validate_agg_exprs(*exprs) if len(exprs) == 1 and isinstance(exprs[0], dict): agg_dict = exprs[0] return self.agg(*self._process_agg_dict(agg_dict)) agg_exprs = self._process_agg_exprs(exprs) return self._df._from_logical_plan( Aggregate.from_session_state(self._df._logical_plan, self._by_exprs, agg_exprs, self._df._session_state), self._df._session_state, ) ``` ## LLMResponseCacheConfig Bases: `BaseModel` ``` flowchart TD fenic.api.LLMResponseCacheConfig[LLMResponseCacheConfig] click fenic.api.LLMResponseCacheConfig href "" "fenic.api.LLMResponseCacheConfig" ``` Configuration for LLM response caching. LLM response caching stores the results of language model API calls to reduce costs and improve performance for repeated queries. This is distinct from DataFrame caching (the `.cache()` operator). Attributes: - **`enabled`** – Whether caching is enabled (default: True). - **`backend`** (`CacheBackend`) – Cache backend to use (default: LOCAL). - **`ttl`** (`str`) – Time-to-live duration string (default: "1h"). Format: where unit is s/m/h/d. Examples: "30s", "15m", "2h", "7d". Maximum: 30 days, Minimum: 1 second. - **`max_size_mb`** (`int`) – Maximum cache size in MB before LRU eviction (default: 128MB). - **`namespace`** (`str`) – Cache namespace for isolation (default: "default"). Example Basic configuration within SemanticConfig: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt": OpenAILanguageModel(model_name="gpt-4o-mini", rpm=100, tpm=1000) }, llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="1h", max_size_mb=1000, ) ) ) ``` Custom TTL and larger cache: ``` llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="7d", # 7 days max_size_mb=5000, ) ``` Disabled caching: ``` llm_response_cache=LLMResponseCacheConfig(enabled=False) ``` Methods: - **`ttl_seconds`** – Convert TTL string to seconds. - **`validate_ttl`** – Validate TTL duration string format. ### ttl_seconds ``` ttl_seconds() -> int ``` Convert TTL string to seconds. Returns: - `int` – TTL duration in seconds. Raises: - `ValueError` – If TTL format is invalid. Source code in `src/fenic/api/session/config.py` ``` def ttl_seconds(self) -> int: """Convert TTL string to seconds. Returns: TTL duration in seconds. Raises: ValueError: If TTL format is invalid. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, self.ttl.lower()) if not match: raise ValueError(f"Invalid TTL format: '{self.ttl}'") value, unit = match.groups() value = int(value) multipliers = {"s": 1, "m": 60, "h": 3600, "d": 86400} return value * multipliers[unit] ``` ### validate_ttl ``` validate_ttl(v: str) -> str ``` Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Parameters: - **`v`** (`str`) – TTL duration string to validate. Returns: - `str` – The validated TTL string. Raises: - `ValueError` – If format is invalid or value is out of range. Source code in `src/fenic/api/session/config.py` ``` @field_validator("ttl") @classmethod def validate_ttl(cls, v: str) -> str: """Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Args: v: TTL duration string to validate. Returns: The validated TTL string. Raises: ValueError: If format is invalid or value is out of range. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, v.lower()) if not match: raise ValueError( f"Invalid TTL format: '{v}'. " "Expected: where unit is s/m/h/d. " "Examples: '30m', '2h', '1d'" ) value, unit = match.groups() value = int(value) # Validate ranges if unit == "s" and value < 1: raise ValueError("TTL must be at least 1 second") if unit == "h" and value > 720: # 30 days raise ValueError("TTL cannot exceed 720 hours (30 days)") if unit == "d" and value > 30: raise ValueError("TTL cannot exceed 30 days") return v ``` ## Lineage ``` Lineage(lineage: BaseLineage) ``` Query interface for tracing data lineage through a query plan. This class allows you to navigate through the query plan both forwards and backwards, tracing how specific rows are transformed through each operation. Example ``` # Create a lineage query starting from the root query = LineageQuery(lineage, session.execution) # Or start from a specific source query.start_from_source("my_table") # Trace rows backwards through a transformation result = query.backward(["uuid1", "uuid2"]) # Trace rows forward to see their outputs result = query.forward(["uuid3", "uuid4"]) ``` Initialize a Lineage instance. Parameters: - **`lineage`** (`BaseLineage`) – The underlying lineage implementation. Methods: - **`backwards`** – Trace rows backwards to see which input rows produced them. - **`forwards`** – Trace rows forward to see how they are transformed by the next operation. - **`get_result_df`** – Get the result of the query as a Polars DataFrame. - **`get_source_df`** – Get a query source by name as a Polars DataFrame. - **`get_source_names`** – Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. - **`show`** – Print the operator tree of the query. - **`skip_backwards`** – [Not Implemented] Trace rows backwards through multiple operations at once. - **`skip_forwards`** – [Not Implemented] Trace rows forward through multiple operations at once. - **`start_from_source`** – Set the current position to a specific source in the query plan. Source code in `src/fenic/api/lineage.py` ``` def __init__(self, lineage: BaseLineage): """Initialize a Lineage instance. Args: lineage: The underlying lineage implementation. """ self.lineage = lineage ``` ### backwards ``` backwards(ids: List[str], branch_side: Optional[BranchSide] = None) -> pl.DataFrame ``` Trace rows backwards to see which input rows produced them. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back - **`branch_side`** (`Optional[BranchSide]`, default: `None` ) – For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: - `DataFrame` – DataFrame containing the source rows that produced the specified outputs Raises: - `ValueError` – If invalid ids format or incorrect branch_side specification Example ``` # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def backwards( self, ids: List[str], branch_side: Optional[BranchSide] = None ) -> pl.DataFrame: """Trace rows backwards to see which input rows produced them. Args: ids: List of UUIDs identifying the rows to trace back branch_side: For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: DataFrame containing the source rows that produced the specified outputs Raises: ValueError: If invalid ids format or incorrect branch_side specification Example: ```python # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` """ return self.lineage.backwards(ids, branch_side) ``` ### forwards ``` forwards(row_ids: List[str]) -> pl.DataFrame ``` Trace rows forward to see how they are transformed by the next operation. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the transformed rows in the next operation Raises: - `ValueError` – If at root node or if row_ids format is invalid Example ``` # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def forwards(self, row_ids: List[str]) -> pl.DataFrame: """Trace rows forward to see how they are transformed by the next operation. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the transformed rows in the next operation Raises: ValueError: If at root node or if row_ids format is invalid Example: ```python # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` """ return self.lineage.forwards(row_ids) ``` ### get_result_df ``` get_result_df() -> pl.DataFrame ``` Get the result of the query as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` def get_result_df(self) -> pl.DataFrame: """Get the result of the query as a Polars DataFrame.""" return self.lineage.get_result_df() ``` ### get_source_df ``` get_source_df(source_name: str) -> pl.DataFrame ``` Get a query source by name as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_df(self, source_name: str) -> pl.DataFrame: """Get a query source by name as a Polars DataFrame.""" return self.lineage.get_source_df(source_name) ``` ### get_source_names ``` get_source_names() -> List[str] ``` Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_names(self) -> List[str]: """Get the names of all sources in the query plan. Used to determine where to start the lineage traversal.""" return self.lineage.get_source_names() ``` ### show ``` show() -> None ``` Print the operator tree of the query. Source code in `src/fenic/api/lineage.py` ``` def show(self) -> None: """Print the operator tree of the query.""" print(self.lineage.stringify_graph()) ``` ### skip_backwards ``` skip_backwards(ids: List[str]) -> Dict[str, pl.DataFrame] ``` [Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back Returns: - `Dict[str, DataFrame]` – Dictionary mapping operation names to their source DataFrames Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_backwards(self, ids: List[str]) -> Dict[str, pl.DataFrame]: """[Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: ids: List of UUIDs identifying the rows to trace back Returns: Dictionary mapping operation names to their source DataFrames Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip backwards not yet implemented") ``` ### skip_forwards ``` skip_forwards(row_ids: List[str]) -> pl.DataFrame ``` [Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the final transformed rows Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_forwards(self, row_ids: List[str]) -> pl.DataFrame: """[Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the final transformed rows Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip forwards not yet implemented") ``` ### start_from_source ``` start_from_source(source_name: str) -> None ``` Set the current position to a specific source in the query plan. Parameters: - **`source_name`** (`str`) – Name of the source table to start from Example ``` query.start_from_source("customers") # Now you can trace forward from the customers table ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def start_from_source(self, source_name: str) -> None: """Set the current position to a specific source in the query plan. Args: source_name: Name of the source table to start from Example: ```python query.start_from_source("customers") # Now you can trace forward from the customers table ``` """ self.lineage.start_from_source(source_name) ``` ## OpenAIEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.OpenAIEmbeddingModel[OpenAIEmbeddingModel] click fenic.api.OpenAIEmbeddingModel href "" "fenic.api.OpenAIEmbeddingModel" ``` Configuration for OpenAI embedding models. This class defines the configuration settings for OpenAI embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAIEmbeddingModelName`) – The name of the OpenAI embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. Example Configuring an OpenAI embedding model with rate limits: ``` config = OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) ``` ## OpenAILanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.OpenAILanguageModel[OpenAILanguageModel] click fenic.api.OpenAILanguageModel href "" "fenic.api.OpenAILanguageModel" ``` Configuration for OpenAI language models. This class defines the configuration settings for OpenAI language models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAILanguageModelName`) – The name of the OpenAI model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Note When using an o-series or gpt5 reasoning model without specifying a reasoning effort in a Profile, the `reasoning_effort` will default to `low` (for o-series models) or `minimal` (for gpt5 models). Example Configuring an OpenAI language model with rate limits: ``` config = OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) ``` Configuring an OpenAI model with profiles: ``` config = OpenAILanguageModel( model_name="o4-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile(reasoning_effort="high"), }, default_profile="fast", ) ``` Using a profile in a semantic operation: ``` config = SemanticConfig( language_models={ "o4": OpenAILanguageModel( model_name="o4-mini", rpm=1_000, tpm=1_000_000, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ) }, default_language_model="o4", ) # Will use the default "fast" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias="o4", ) # Will use the "thorough" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="o4", profile="thorough"), ) ``` Classes: - **`Profile`** – OpenAI-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.OpenAILanguageModel.Profile[Profile] click fenic.api.OpenAILanguageModel.Profile href "" "fenic.api.OpenAILanguageModel.Profile" ``` OpenAI-specific profile configurations. This class defines profile configurations for OpenAI models, allowing a user to reference the same underlying model in semantic operations with different settings. Attributes: - **`reasoning_effort`** (`Optional[ReasoningEffort]`) – Provide a reasoning effort. Only for gpt5 and o-series models. Valid values: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'. - For gpt-5.6 models: supports 'xhigh' and 'max' - For gpt-5.5 models: defaults to 'medium', supports 'xhigh' - For gpt-5.4 models: defaults to 'none' (disabled reasoning), supports 'xhigh' - For gpt-5.1 and gpt-5.2 models: defaults to 'none' (disabled reasoning), does NOT support 'minimal' or 'xhigh' - For gpt-5 models: defaults to 'minimal', does NOT support 'none' - For o-series models: defaults to 'low', does NOT support 'none' or 'minimal' - **`verbosity`** (`Optional[Verbosity]`) – Provide a verbosity level. Only for gpt5/gpt5.1 models. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note When using an o-series or gpt5 reasoning model with reasoning enabled, the `temperature` cannot be customized. For gpt-5.1 models with reasoning_effort='none', temperature CAN be customized. Example Configuring a profile with medium reasoning effort: ``` profile = OpenAILanguageModel.Profile(reasoning_effort="medium") ``` ## OpenRouterLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.OpenRouterLanguageModel[OpenRouterLanguageModel] click fenic.api.OpenRouterLanguageModel href "" "fenic.api.OpenRouterLanguageModel" ``` Configuration for OpenRouter language models. This class defines the configuration settings for OpenRouter language models, including model selection and rate limiting parameters. When fetching available models from OpenRouter, results will be filtered to only include models from providers that are not in the user’s ignored providers list and are either in the user’s allowed providers list (if configured) or from any provider (if no allowed providers are specified). Attributes: - **`model_name`** (`str`) – `{family}/{model}` identifier (e.g., `anthropic/claude-3-5-sonnet`). - **`profiles`** (`Optional[dict[str, Profile]]`) – Mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The key in `profiles` to select by default. - **`structured_output_strategy`** (`Optional[StructuredOutputStrategy]`) – The strategy to use for structured output if a model supports both tool calling and structured outputs. - `prefer_tools`: prefer using tools over response format. - `prefer_response_format`: prefer using response format over tools. Requirements - Set `OPENROUTER_API_KEY` in your environment. Example: ``` OpenRouterLanguageModel( model_name="openai/gpt-oss-20b", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="price" # Routes to the cheapest available provider ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="anthropic/claude-sonnet-4", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( only=[ "Anthropic" ] # ensures the request will only be routed to Anthropic and not AWS Bedrock or Google Vertex ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="qwen/qwen3-next-80b-a3b-instruct", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="throughput", # routes to the provider with the highest overall throughput data_collection="deny" # eliminates providers that retain prompt data (would only route to DeepInfra/AtlasCloud, in this example) # Eliminate providers that offer an fp8 quantized version of the model, only allowing bf16. # Note that many providers have an `unknown` quantization, so you may be excluding more providers than you expect. quantizations=["bf16"] ) ) } ) ``` Classes: - **`Profile`** – Profile configurations for OpenRouter language models. - **`Provider`** – Provider routing configuration for OpenRouter language models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.OpenRouterLanguageModel.Profile[Profile] click fenic.api.OpenRouterLanguageModel.Profile href "" "fenic.api.OpenRouterLanguageModel.Profile" ``` Profile configurations for OpenRouter language models. Attributes: - **`models`** (`Optional[list[str]]`) – A list of fallback models to use if the primary model is unavailable. ([OpenRouter Documentation](https://openrouter.ai/docs/features/model-routing#the-models-parameter)). - **`provider`** (`Optional[Provider]`) – Provider routing preferences (include/exclude specific providers, set provider ranking method preference) ([OpenRouter Documentation](https://openrouter.ai/docs/features/provider-routing)). - **`reasoning_effort`** (`Optional[OpenRouterReasoningEffort]`) – OpenRouter reasoning effort configuration (none, minimal, low, medium, high, xhigh, max). If the model does support reasoning, but not `reasoning_effort`, a `reasoning_max_tokens` will be calculated that is roughly equivalent as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#reasoning-effort-level)) - **`reasoning_max_tokens`** (`Optional[int]`) – Supported by Anthropic, Gemini, etc., sets a token budget for reasoning If the model does support reasoning, but not `reasoning_max_tokens`, a `reasoning_effort_ will be automatically calculated based on`reasoning_max_tokens` as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#max-tokens-for-reasoning)) - **`parsing_engine`** (`Optional[ParsingEngine]`) – The parsing engine to use for processing PDF files. By default, the model's native parsing engine will be used. If the model doesn't support PDF processing and the parsing engine is not provided, an error will be raised. Note: 'mistral-ocr' incurs additional costs. ([OpenRouter Documentation](https://openrouter.ai/docs/features/multimodal/pdfs)) ### Provider Bases: `BaseModel` ``` flowchart TD fenic.api.OpenRouterLanguageModel.Provider[Provider] click fenic.api.OpenRouterLanguageModel.Provider href "" "fenic.api.OpenRouterLanguageModel.Provider" ``` Provider routing configuration for OpenRouter language models. [Provider Routing Documentation](https://openrouter.ai/docs/features/provider-routing) Attributes: - **`order`** (`Optional[list[str]]`) – List of providers to try in order (e.g. ['Anthropic', 'Amazon Bedrock']). - **`sort`** (`Optional[ProviderSort]`) – Provider routing preference (e.g. 'price', 'throughput', 'latency'). "price" will route to the cheapest available provider first, progressing through the list of providers in order of price. "throughput" will route to the provider with the highest overall recent throughput, progressing through the list of providers in order of throughput. "latency" will route to the provider with the lowest overall recent latency, progressing through the list of providers in order of latency. - **`quantizations`** (`Optional[list[ModelQuantization]]`) – Allowed quantizations. Note: many providers report `unknown`. - **`data_collection`** (`Optional[DataCollection]`) – Data collection preference. `allow`: allows the use of providers which store prompt data non-transiently and may train on it. `deny`: use only providers which do not collect/store prompt data. - **`only`** (`Optional[list[str]]`) – Only include these providers when performing provider routing. - **`exclude`** (`Optional[list[str]]`) – Exclude these providers when performing provider routing. - **`max_prompt_price`** (`Optional[float]`) – Maximum prompt price ($USD per 1M tokens). - **`max_completion_price`** (`Optional[float]`) – Maximum completion price ($USD per 1M tokens). ## SemanticConfig Bases: `BaseModel` ``` flowchart TD fenic.api.SemanticConfig[SemanticConfig] click fenic.api.SemanticConfig href "" "fenic.api.SemanticConfig" ``` Configuration for semantic language and embedding models. This class defines the configuration for both language models and optional embedding models used in semantic operations. It ensures that all configured models are valid and supported by their respective providers. Attributes: - **`language_models`** (`Optional[dict[str, LanguageModel]]`) – Mapping of model aliases to language model configurations. - **`default_language_model`** (`Optional[str]`) – The alias of the default language model to use for semantic operations. Not required if only one language model is configured. - **`embedding_models`** (`Optional[dict[str, EmbeddingModel]]`) – Optional mapping of model aliases to embedding model configurations. - **`default_embedding_model`** (`Optional[str]`) – The alias of the default embedding model to use for semantic operations. Note The embedding model is optional and only required for operations that need semantic search or embedding capabilities. Example Configuring semantic models with a single language model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) } ) ``` Configuring semantic models with multiple language models and an embedding model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), "gemini": GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ) ``` Configuring models with profiles: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4o-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, profiles={ "fast": AnthropicLanguageModel.Profile(effort="low"), "thorough": AnthropicLanguageModel.Profile(effort="high"), }, default_profile="fast", ), }, default_language_model="gpt4", ) ``` Methods: - **`model_post_init`** – Post initialization hook to set defaults. - **`validate_models`** – Validates that the selected models are supported by the system. ### model_post_init ``` model_post_init(__context) -> None ``` Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. Source code in `src/fenic/api/session/config.py` ``` def model_post_init(self, __context) -> None: """Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. """ if self.language_models: # Set default language model if not set and only one model exists if self.default_language_model is None and len(self.language_models) == 1: self.default_language_model = list(self.language_models.keys())[0] # Auto-create profiles for Google models that support thinking_level for model_config in self.language_models.values(): if isinstance(model_config, (GoogleDeveloperLanguageModel, GoogleVertexLanguageModel)): model_provider = _get_model_provider_for_model_config(model_config) model_params = model_catalog.get_completion_model_parameters( model_provider, model_config.model_name ) if model_params and model_params.supported_thinking_levels and model_config.profiles is None: # Auto-create profiles for each supported thinking level if isinstance(model_config, GoogleDeveloperLanguageModel): model_config.profiles = { level: GoogleDeveloperLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } else: model_config.profiles = { level: GoogleVertexLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } model_config.default_profile = "low" # Set default profile for each model if not set and only one profile exists for model_config in self.language_models.values(): if model_config.profiles is not None: profile_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(profile_names) == 1: model_config.default_profile = profile_names[0] # Set default embedding model if not set and only one model exists if self.embedding_models: if self.default_embedding_model is None and len(self.embedding_models) == 1: self.default_embedding_model = list(self.embedding_models.keys())[0] # Set default profile for each model if not set and only one preset exists for model_config in self.embedding_models.values(): if ( hasattr(model_config, "profiles") and model_config.profiles is not None ): preset_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(preset_names) == 1: model_config.default_profile = preset_names[0] ``` ### validate_models ``` validate_models() -> SemanticConfig ``` Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: - `SemanticConfig` – The validated SemanticConfig instance. Raises: - `ConfigurationError` – If any of the models are not supported. Source code in `src/fenic/api/session/config.py` ``` @model_validator(mode="after") def validate_models(self) -> SemanticConfig: """Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: The validated SemanticConfig instance. Raises: ConfigurationError: If any of the models are not supported. """ # Skip validation if no models configured (embedding-only or empty config) if not self.language_models and not self.embedding_models: return self # Validate language models if provided if self.language_models: available_language_model_aliases = list(self.language_models.keys()) if self.default_language_model is None and len(self.language_models) > 1: raise ConfigurationError( f"default_language_model is not set, and multiple language models are configured. Please specify one of: {available_language_model_aliases} as a default_language_model." ) if ( self.default_language_model is not None and self.default_language_model not in self.language_models ): raise ConfigurationError( f"default_language_model {self.default_language_model} is not in configured map of language models. Available models: {available_language_model_aliases} ." ) for model_alias, language_model in self.language_models.items(): language_model_name = language_model.model_name language_model_provider = _get_model_provider_for_model_config( language_model ) completion_model_params = model_catalog.get_completion_model_parameters( language_model_provider, language_model_name ) if completion_model_params is None: raise ConfigurationError( model_catalog.generate_unsupported_completion_model_error_message( language_model_provider, language_model_name ) ) if language_model.profiles is not None: if not completion_model_params.supports_profiles: raise ConfigurationError( f"Model '{model_alias}' does not support parameter profiles. Please remove the Profile configuration." ) profile_names = list(language_model.profiles.keys()) if ( language_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( language_model.default_profile is not None and language_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {language_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in language_model.profiles.items(): _validate_language_profile( language_model, model_alias, completion_model_params, profile, profile_alias, ) if self.embedding_models is not None: available_embedding_model_aliases = list(self.embedding_models.keys()) if self.default_embedding_model is None and len(self.embedding_models) > 1: raise ConfigurationError( f"default_embedding_model is not set, and multiple embedding models are configured. Please specify one of: {available_embedding_model_aliases} as a default_embedding_model." ) if ( self.default_embedding_model is not None and self.default_embedding_model not in self.embedding_models ): raise ConfigurationError( f"default_embedding_model {self.default_embedding_model} is not in configured map of embedding models. Available models: {available_embedding_model_aliases} ." ) for model_alias, embedding_model in self.embedding_models.items(): embedding_model_provider = _get_model_provider_for_model_config( embedding_model ) embedding_model_name = embedding_model.model_name embedding_model_parameters = ( model_catalog.get_embedding_model_parameters( embedding_model_provider, embedding_model_name ) ) if embedding_model_parameters is None: raise ConfigurationError( model_catalog.generate_unsupported_embedding_model_error_message( embedding_model_provider, embedding_model_name ) ) if hasattr(embedding_model, "profiles") and embedding_model.profiles: profile_names = list(embedding_model.profiles.keys()) if ( embedding_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( embedding_model.default_profile is not None and embedding_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {embedding_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in embedding_model.profiles.items(): _validate_embedding_profile( embedding_model_parameters, model_alias, profile_alias, profile, ) return self ``` ## SemanticExtensions ``` SemanticExtensions(df: DataFrame) ``` A namespace for semantic dataframe operators. Initialize semantic extensions. Parameters: - **`df`** (`DataFrame`) – The DataFrame to extend with semantic operations. Methods: - **`join`** – Performs a semantic join between two DataFrames using a natural language predicate. - **`sim_join`** – Performs a semantic similarity join between two DataFrames using embedding expressions. - **`with_cluster_labels`** – Cluster rows using K-means and add cluster metadata columns. Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def __init__(self, df: DataFrame): """Initialize semantic extensions. Args: df: The DataFrame to extend with semantic operations. """ self._df = df ``` ### join ``` join(other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Parameters: - **`other`** (`DataFrame`) – The DataFrame to join with. - **`predicate`** (`str`) – A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. - **`left_on`** (`Column`) – The column from the left DataFrame (self) to use in the join predicate. - **`right_on`** (`Column`) – The column from the right DataFrame (other) to use in the join predicate. - **`strict`** (`bool`, default: `True` ) – If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[JoinExampleCollection]`, default: `None` ) – Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use. If None, uses the default model. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing matched row pairs with all columns from both DataFrames. Basic semantic join ``` # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Semantic join with examples ``` # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def join( self, other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Args: other: The DataFrame to join with. predicate: A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. left_on: The column from the left DataFrame (self) to use in the join predicate. right_on: The column from the right DataFrame (other) to use in the join predicate. strict: If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. examples: Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) model_alias: Optional alias for the language model to use. If None, uses the default model. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: DataFrame: A new DataFrame containing matched row pairs with all columns from both DataFrames. Example: Basic semantic join ```python # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Example: Semantic join with examples ```python # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(other, DataFrame): raise ValidationError(f"other argument must be a DataFrame, got {type(other)}") if not isinstance(predicate, str): raise ValidationError( f"The `predicate` argument to `semantic.join` must be a string, got {type(predicate)}" ) if not isinstance(left_on, Column): raise ValidationError(f"`left_on` argument must be a Column, got {type(left_on)} instead.") if not isinstance(right_on, Column): raise ValidationError(f"`right_on` argument must be a Column, got {type(right_on)} instead.") if examples is not None and not isinstance(examples, JoinExampleCollection): raise ValidationError(f"`examples` argument must be a JoinExampleCollection, got {type(examples)} instead.") if model_alias is not None and not isinstance(model_alias, (str, ModelAlias)): raise ValidationError(f"`model_alias` argument must be a string or ModelAlias, got {type(model_alias)} instead.") # Validate request_timeout request_timeout = validate_timeout(request_timeout) resolved_model_alias = _resolve_model_alias(model_alias) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticJoin.from_session_state( left=self._df._logical_plan, right=other._logical_plan, left_on=left_on._logical_expr, right_on=right_on._logical_expr, jinja_template=predicate, strict=strict, model_alias=resolved_model_alias, examples=examples, request_timeout=request_timeout, session_state=self._df._session_state, ), self._df._session_state, ) ``` ### sim_join ``` sim_join(other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = 'cosine', similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Parameters: - **`other`** (`DataFrame`) – The right-hand DataFrame to join with. - **`left_on`** (`ColumnOrName`) – Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. - **`right_on`** (`ColumnOrName`) – Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. - **`k`** (`int`, default: `1` ) – Number of most similar matches to return per row. - **`similarity_metric`** (`SemanticSimilarityMetric`, default: `'cosine'` ) – Similarity metric to use: "l2", "cosine", or "dot". - **`similarity_score_column`** (`Optional[str]`, default: `None` ) – If set, adds a column with this name containing similarity scores. If None, the scores are omitted. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. - `DataFrame` – The result includes all columns from both input DataFrames and, when `similarity_score_column` - `DataFrame` – is provided, a similarity score column. Join key columns that already exist in the input - `DataFrame` – DataFrames are preserved as normal user columns. Join keys derived from expressions are - `DataFrame` – temporary execution columns and are not included in the result schema. Raises: - `ValidationError` – If `k` is not positive or if the columns are invalid. - `ValidationError` – If `similarity_metric` is not one of "l2", "cosine", "dot" Match queries to FAQ entries ``` # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Link headlines to articles ``` # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Find similar job postings ``` # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def sim_join( self, other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = "cosine", similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note: Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Args: other: The right-hand DataFrame to join with. left_on: Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. right_on: Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. k: Number of most similar matches to return per row. similarity_metric: Similarity metric to use: "l2", "cosine", or "dot". similarity_score_column: If set, adds a column with this name containing similarity scores. If None, the scores are omitted. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. The result includes all columns from both input DataFrames and, when `similarity_score_column` is provided, a similarity score column. Join key columns that already exist in the input DataFrames are preserved as normal user columns. Join keys derived from expressions are temporary execution columns and are not included in the result schema. Raises: ValidationError: If `k` is not positive or if the columns are invalid. ValidationError: If `similarity_metric` is not one of "l2", "cosine", "dot" Example: Match queries to FAQ entries ```python # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Example: Link headlines to articles ```python # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Example: Find similar job postings ```python # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(right_on, ColumnOrName): raise ValidationError( f"The `right_on` argument must be a `Column` or a string representing a column name, " f"but got `{type(right_on).__name__}` instead." ) if not isinstance(other, DataFrame): raise ValidationError( f"The `other` argument to `sim_join()` must be a DataFrame`, but got `{type(other).__name__}`." ) if not (isinstance(k, int) and k > 0): raise ValidationError( f"The parameter `k` must be a positive integer, but received `{k}`." ) args = get_args(SemanticSimilarityMetric) if similarity_metric not in args: raise ValidationError( f"The `similarity_metric` argument must be one of {args}, but got `{similarity_metric}`." ) def _validate_column(column: ColumnOrName, name: str): if column is None: raise ValidationError(f"The `{name}` argument must not be None.") if not isinstance(column, ColumnOrName): raise ValidationError( f"The `{name}` argument must be a `Column` or a string representing a column name, " f"but got `{type(column).__name__}` instead." ) _validate_column(left_on, "left_on") _validate_column(right_on, "right_on") # Validate request_timeout request_timeout = validate_timeout(request_timeout) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticSimilarityJoin.from_session_state( self._df._logical_plan, other._logical_plan, Column._from_col_or_name(left_on)._logical_expr, Column._from_col_or_name(right_on)._logical_expr, k, similarity_metric=similarity_metric, similarity_score_column=similarity_score_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ### with_cluster_labels ``` with_cluster_labels(by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = 'cluster_label', centroid_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Parameters: - **`by`** (`ColumnOrName`) – Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). - **`num_clusters`** (`int`) – Number of clusters to compute (must be > 0). - **`max_iter`** (`int`, default: `300` ) – Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. - **`num_init`** (`int`, default: `1` ) – Number of independent runs of k-means with different centroid seeds. The best result is selected. - **`label_column`** (`str`, default: `'cluster_label'` ) – Name of the output column for cluster IDs. Default is "cluster_label". - **`centroid_column`** (`Optional[str]`, default: `None` ) – If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame with all original columns plus: - `DataFrame` – - ``: integer cluster assignment (0 to num_clusters - 1) - `DataFrame` – - ``: cluster centroid embedding, if specified Basic clustering ``` # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Filter outliers using centroids ``` # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def with_cluster_labels( self, by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = "cluster_label", centroid_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note: Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Args: by: Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). num_clusters: Number of clusters to compute (must be > 0). max_iter: Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. num_init: Number of independent runs of k-means with different centroid seeds. The best result is selected. label_column: Name of the output column for cluster IDs. Default is "cluster_label". centroid_column: If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame with all original columns plus: - ``: integer cluster assignment (0 to num_clusters - 1) - ``: cluster centroid embedding, if specified Example: Basic clustering ```python # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Example: Filter outliers using centroids ```python # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` """ # Validate request_timeout request_timeout = validate_timeout(request_timeout) # Validate num_clusters if not isinstance(num_clusters, int) or num_clusters <= 0: raise ValidationError("`num_clusters` must be a positive integer.") # Validate max_iter if not isinstance(max_iter, int) or max_iter <= 0: raise ValidationError("`max_iter` must be a positive integer.") # Validate num_init if not isinstance(num_init, int) or num_init <= 0: raise ValidationError("`num_init` must be a positive integer.") # Validate clustering target if not isinstance(by, ColumnOrName): raise ValidationError( f"Invalid cluster by: expected a column name (str) or Column object, got {type(by).__name__}." ) # Validate label_column if not isinstance(label_column, str) or not label_column: raise ValidationError("`label_column` must be a non-empty string.") # Validate centroid_column if provided if centroid_column is not None: if not isinstance(centroid_column, str) or not centroid_column: raise ValidationError("`centroid_column` must be a non-empty string if provided.") # Check that the expression isn't a literal by_expr = Column._from_col_or_name(by)._logical_expr if isinstance(by_expr, LiteralExpr): raise ValidationError( f"Invalid cluster by: Cannot cluster by a literal value: {by_expr}." ) return self._df._from_logical_plan( SemanticCluster.from_session_state( self._df._logical_plan, by_expr, num_clusters=num_clusters, max_iter=max_iter, num_init=num_init, label_column=label_column, centroid_column=centroid_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ## Session The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. Create a session with default configuration ``` session = Session.get_or_create(SessionConfig(app_name="my_app")) ``` Create a session with cloud configuration ``` config = SessionConfig( app_name="my_app", cloud=True, api_key="your_api_key" ) session = Session.get_or_create(config) ``` Methods: - **`create_dataframe`** – Create a DataFrame from a variety of Python-native data formats. - **`get_or_create`** – Gets an existing Session or creates a new one with the configured settings. - **`sql`** – Execute a read-only SQL query against one or more DataFrames using named placeholders. - **`stop`** – Stops the session and closes all connections. - **`table`** – Returns the specified table as a DataFrame. - **`view`** – Returns the specified view as a DataFrame. Attributes: - **`catalog`** (`Catalog`) – Interface for catalog operations on the Session. - **`read`** (`DataFrameReader`) – Returns a DataFrameReader that can be used to read data in as a DataFrame. ### catalog ``` catalog: Catalog ``` Interface for catalog operations on the Session. ### read ``` read: DataFrameReader ``` Returns a DataFrameReader that can be used to read data in as a DataFrame. Returns: - **`DataFrameReader`** ( `DataFrameReader` ) – A reader interface to read data into DataFrame Raises: - `RuntimeError` – If the session has been stopped ### create_dataframe ``` create_dataframe(data: DataLike, schema: Schema | None = None) -> DataFrame ``` Create a DataFrame from a variety of Python-native data formats. Parameters: - **`data`** (`DataLike`) – Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table - **`schema`** (`Schema | None`, default: `None` ) – Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: - `DataFrame` – A new DataFrame instance Raises: - `ValidationError` – If the input format is unsupported or the provided columns do not match the schema. - `PlanError` – If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Create from Polars DataFrame ``` import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from Pandas DataFrame ``` import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from dictionary ``` session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Create from list of dictionaries ``` session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Create from pyarrow Table ``` import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Create with an explicit schema ``` import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` Source code in `src/fenic/api/session/session.py` ``` def create_dataframe( self, data: DataLike, schema: Schema | None = None, ) -> DataFrame: """Create a DataFrame from a variety of Python-native data formats. Args: data: Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table schema: Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: A new DataFrame instance Raises: ValidationError: If the input format is unsupported or the provided columns do not match the schema. PlanError: If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Example: Create from Polars DataFrame ```python import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from Pandas DataFrame ```python import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from dictionary ```python session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Example: Create from list of dictionaries ```python session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Example: Create from pyarrow Table ```python import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Example: Create with an explicit schema ```python import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` """ pl_df, row_field_names = _normalize_data_like_to_polars( data, allow_empty_list=schema is not None, validate_all_rows=schema is not None, ) if schema is None: return DataFrame._from_logical_plan( InMemorySource.from_session_state(pl_df, self._session_state), self._session_state, ) coerced_pl_df = _coerce_to_schema(pl_df, schema, row_field_names=row_field_names) return DataFrame._from_logical_plan( InMemorySource.from_schema(coerced_pl_df, schema), self._session_state, ) ``` ### get_or_create ``` get_or_create(config: SessionConfig) -> Session ``` Gets an existing Session or creates a new one with the configured settings. Returns: - `Session` – A Session instance configured with the provided settings Source code in `src/fenic/api/session/session.py` ``` @classmethod def get_or_create( cls, config: SessionConfig, ) -> Session: """Gets an existing Session or creates a new one with the configured settings. Returns: A Session instance configured with the provided settings """ if config.cloud: from fenic._backends.cloud.manager import CloudSessionManager cloud_session_manager = CloudSessionManager() if not cloud_session_manager.initialized: session_manager_dependencies = ( CloudSessionManager.create_global_session_dependencies() ) cloud_session_manager.configure(session_manager_dependencies) future = asyncio.run_coroutine_threadsafe( cloud_session_manager.get_or_create_session_state(config), cloud_session_manager._asyncio_loop, ) cloud_session_state = future.result() return Session._create_cloud_session(cloud_session_state) local_session_state: LocalSessionState = LocalSessionManager().get_or_create_session_state(config._to_resolved_config()) return Session._create_local_session(local_session_state) ``` ### sql ``` sql(query: str, /, **tables: DataFrame) -> DataFrame ``` Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Parameters: - **`query`** (`str`) – A SQL query string with placeholders like `{df}` - **`**tables`** (`DataFrame`, default: `{}` ) – Keyword arguments mapping placeholder names to DataFrames Returns: - `DataFrame` – A lazy DataFrame representing the result of the SQL query Raises: - `ValidationError` – If a placeholder is used in the query but not passed as a keyword argument Simple join between two DataFrames ``` df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Complex query with multiple DataFrames ``` users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(""" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id """, users=users, orders=orders, products=products) ``` Source code in `src/fenic/api/session/session.py` ``` def sql(self, query: str, /, **tables: DataFrame) -> DataFrame: """Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Args: query: A SQL query string with placeholders like `{df}` **tables: Keyword arguments mapping placeholder names to DataFrames Returns: A lazy DataFrame representing the result of the SQL query Raises: ValidationError: If a placeholder is used in the query but not passed as a keyword argument Example: Simple join between two DataFrames ```python df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Example: Complex query with multiple DataFrames ```python users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(\"\"\" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id \"\"\", users=users, orders=orders, products=products) ``` """ query = query.strip() if not query: raise ValidationError("SQL query must not be empty.") placeholders = set(SQL_PLACEHOLDER_RE.findall(query)) missing = placeholders - tables.keys() if missing: raise ValidationError( f"Missing DataFrames for placeholders in SQL query: {', '.join(sorted(missing))}. " f"Make sure to pass them as keyword arguments, e.g., sql(..., {next(iter(missing))}=df)." ) logical_plans = [] template_names = [] input_session_states = [] for name, table in tables.items(): if name in placeholders: template_names.append(name) logical_plans.append(table._logical_plan) input_session_states.append(table._session_state) DataFrame._ensure_same_session(self._session_state, input_session_states) return DataFrame._from_logical_plan( SQL.from_session_state(logical_plans, template_names, query, self._session_state), self._session_state, ) ``` ### stop ``` stop(skip_usage_summary: bool = False) ``` Stops the session and closes all connections. Parameters: - **`skip_usage_summary`** (`bool`, default: `False` ) – Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. Source code in `src/fenic/api/session/session.py` ``` def stop(self, skip_usage_summary: bool = False): """Stops the session and closes all connections. Args: skip_usage_summary: Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. """ self._session_state.stop(skip_usage_summary=skip_usage_summary) ``` ### table ``` table(table_name: str) -> DataFrame ``` Returns the specified table as a DataFrame. Parameters: - **`table_name`** (`str`) – Name of the table Returns: - `DataFrame` – Table as a DataFrame Raises: - `ValueError` – If the table does not exist Load an existing table ``` df = session.table("my_table") ``` Source code in `src/fenic/api/session/session.py` ``` def table(self, table_name: str) -> DataFrame: """Returns the specified table as a DataFrame. Args: table_name: Name of the table Returns: Table as a DataFrame Raises: ValueError: If the table does not exist Example: Load an existing table ```python df = session.table("my_table") ``` """ if not self._session_state.catalog.does_table_exist(table_name): raise ValueError(f"Table {table_name} does not exist") return DataFrame._from_logical_plan( TableSource.from_session_state(table_name, self._session_state), self._session_state, ) ``` ### view ``` view(view_name: str) -> DataFrame ``` Returns the specified view as a DataFrame. Parameters: - **`view_name`** (`str`) – Name of the view Returns: DataFrame: Dataframe with the given view Source code in `src/fenic/api/session/session.py` ``` def view(self, view_name: str) -> DataFrame: """Returns the specified view as a DataFrame. Args: view_name: Name of the view Returns: DataFrame: Dataframe with the given view """ if not self._session_state.catalog.does_view_exist(view_name): raise CatalogError(f"View {view_name} does not exist") view_plan = self._session_state.catalog.get_view_plan(view_name) validate_view(view_name, view_plan, self._session_state) return DataFrame._from_logical_plan( view_plan, self._session_state, ) ``` ## SessionConfig Bases: `BaseModel` ``` flowchart TD fenic.api.SessionConfig[SessionConfig] click fenic.api.SessionConfig href "" "fenic.api.SessionConfig" ``` Configuration for a user session. This class defines the complete configuration for a user session, including application settings, model configurations, and optional cloud settings. It serves as the central configuration object for all language model operations. Attributes: - **`app_name`** (`str`) – Name of the application using this session. Defaults to "default_app". - **`db_path`** (`Optional[Path]`) – Optional path to a local database file for persistent storage. - **`semantic`** (`Optional[SemanticConfig]`) – Configuration for semantic models (optional). - **`cloud`** (`Optional[CloudConfig]`) – Optional configuration for cloud execution. - **`cache`** (`Optional[CloudConfig]`) – Optional configuration for LLM response caching. Note The semantic configuration is optional. When not provided, only non-semantic operations are available. The cloud configuration is optional and only needed for distributed processing. Example Configuring a basic session with a single language model: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ) } ), ) ``` Configuring a session with multiple models and cloud execution: ``` config = SessionConfig( app_name="production_app", db_path=Path("/path/to/database.db"), semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ), cloud=CloudConfig(size=CloudExecutorSize.MEDIUM), ) ``` Methods: - **`to_json`** – Export the session config to a JSON string. ### to_json ``` to_json() -> str ``` Export the session config to a JSON string. Source code in `src/fenic/api/session/config.py` ``` def to_json(self) -> str: """Export the session config to a JSON string.""" return self.model_dump_json(indent=2) ``` ## SystemToolConfig ``` SystemToolConfig(table_names: list[str], tool_namespace: Optional[str] = None, max_result_rows: int = 100) ``` Configuration for canonical system tools. fenic can automatically generate a set of canonical tools for operating on one or more fenic tables. - Schema: list columns/types for any or all tables - Profile: column statistics (counts, basic numeric analysis [min, max, mean, etc.], contextual information for text columns [average_length, etc.]) - Read: read a selection of rows from a single table. These rows can be paged over, filtered and can use column projections. - Search Summary: literal or regex search across all text columns in all tables -- returns back dataframe names with result counts. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Search Content: literal or regex search across a single table, specifying one or more text columns to search across -- returns back rows corresponding to the query. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Analyze: Write raw SQL to perform complex analysis on one or more tables. Attributes: - **`table_names`** (`list[str]`) – List of the fenic table names the tools should be able to access. To allow access to all tables, pass `session.catalog.list_tables()` - **`tool_namespace`** (`Optional[str]`) – If provided, will prefix the names of the generated tools with this namespace value. For example, by default the generated tools will be named `read`, `profile`, etc. With multiple fenic MCP servers, these tool names will clash, which can be confusing. In order to disambiguate, the `tool_namespace` is prefixed to the tool name (in snake case), so a `tool_namespace` of `fenic` would create the tools `fenic_read`, `fenic_profile`, etc. - **`max_result_rows`** (`int`) – Maximum number of rows to be returned from Read/Analyze tools. Example: ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) df = session.create_dataframe({ "c1": [1, 2, 3], "c2": [4, 5, 6] }) df.write.save_as_table("table1", mode="overwrite") session.catalog.set_table_description("table1", "Table 1 Description") server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=["table1"], tool_namespace="Auto", max_result_rows=100 )) ``` Example: Allow generated tools to access all tables in the catalog. ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) # Assuming you already have one or more tables saved to the catalog, with descriptions. server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=session.catalog.list_tables() tool_namespace="Auto", max_result_rows=100 )) ``` ## approx_count_distinct ``` approx_count_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: - `Column` – A Column expression representing the approximate count-distinct aggregation Note Differs from the pyspark implementation in that the relative standard deviation is not configurable. Approximate distinct count per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Nulls are ignored in approximate distinct counts ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: - `TypeMismatchError` – If column is a StructType or ArrayType column. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def approx_count_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Args: column: Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: A Column expression representing the approximate count-distinct aggregation Note: Differs from the pyspark implementation in that the relative standard deviation is not configurable. Example: Approximate distinct count per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Example: Nulls are ignored in approximate distinct counts ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: TypeMismatchError: If column is a StructType or ArrayType column. """ return Column._from_logical_expr( ApproxCountDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## array_agg ``` array_agg(column: ColumnOrName) -> Column ``` Alias for collect_list(). Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array_agg(column: ColumnOrName) -> Column: """Alias for collect_list().""" return collect_list(column) ``` ## asc ``` asc(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls first. Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc() ``` ## asc_nulls_first ``` asc_nulls_first(column: ColumnOrName) -> Column ``` Alias for asc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_first(column: ColumnOrName) -> Column: """Alias for asc(). Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc_nulls_first() ``` ## asc_nulls_last ``` asc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A Column expression representing the column and the ascending sort order with nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls last. Args: column: The column to apply the ascending ordering to. Returns: A Column expression representing the column and the ascending sort order with nulls last. """ return Column._from_col_or_name(column).asc_nulls_last() ``` ## async_udf ``` async_udf(f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0) ``` A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Parameters: - **`f`** (`Optional[Callable[..., Awaitable[Any]]]`, default: `None` ) – Async function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. - **`max_concurrency`** (`int`, default: `10` ) – Maximum number of concurrent executions (default: 10) - **`timeout_seconds`** (`float`, default: `30` ) – Per-item timeout in seconds (default: 30) - **`num_retries`** (`int`, default: `0` ) – Number of retries for failed items (default: 0) Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) ### Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries `python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() }` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def async_udf( f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0, ): """A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Args: f: Async function to convert to UDF return_type: Expected return type of the UDF. Required parameter. max_concurrency: Maximum number of concurrent executions (default: 10) timeout_seconds: Per-item timeout in seconds (default: 30) num_retries: Number of retries for failed items (default: 0) Example: Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) # Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries ```python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() } ``` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. """ def _create_async_udf(func: Callable[..., Awaitable[Any]]) -> Callable: if not inspect.iscoroutinefunction(func): raise ValidationError( f"@async_udf requires an async function, but found a synchronous " f"function {func.__name__!r} of type {type(func)}" ) @wraps(func) def _async_udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr( AsyncUDFExpr( func, col_exprs, return_type, max_concurrency=max_concurrency, timeout_seconds=timeout_seconds, num_retries=num_retries ) ) return _async_udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for async UDFs") # Support both @async_udf and async_udf(...) syntax if f is None: return _create_async_udf else: return _create_async_udf(f) ``` ## avg ``` avg(column: ColumnOrName) -> Column ``` Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the average of Returns: - `Column` – A Column expression representing the average aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def avg(column: ColumnOrName) -> Column: """Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Args: column: Column or column name to compute the average of Returns: A Column expression representing the average aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## coalesce ``` coalesce(*cols: ColumnOrName) -> Column ``` Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the first non-null value from the input columns. Raises: - `ValidationError` – If no columns are provided. coalesce usage ``` df.select(coalesce("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def coalesce(*cols: ColumnOrName) -> Column: """Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the first non-null value from the input columns. Raises: ValidationError: If no columns are provided. Example: coalesce usage ```python df.select(coalesce("col1", "col2", "col3")) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the coalesce method.") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(CoalesceExpr(exprs)) ``` ## col ``` col(col_name: str) -> Column ``` Creates a Column expression referencing a column in the DataFrame. Parameters: - **`col_name`** (`str`) – Name of the column to reference Returns: - `Column` – A Column expression for the specified column Raises: - `TypeError` – If colName is not a string Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True)) def col(col_name: str) -> Column: """Creates a Column expression referencing a column in the DataFrame. Args: col_name: Name of the column to reference Returns: A Column expression for the specified column Raises: TypeError: If colName is not a string """ return Column._from_column_name(col_name) ``` ## collect_list ``` collect_list(column: ColumnOrName) -> Column ``` Aggregate function: collects all values from the specified column into a list. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to collect values from Returns: - `Column` – A Column expression representing the list aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def collect_list(column: ColumnOrName) -> Column: """Aggregate function: collects all values from the specified column into a list. Args: column: Column or column name to collect values from Returns: A Column expression representing the list aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( ListExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count ``` count(column: ColumnOrName) -> Column ``` Aggregate function: returns the count of non-null values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to count values in Returns: - `Column` – A Column expression representing the count aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count(column: ColumnOrName) -> Column: """Aggregate function: returns the count of non-null values in the specified column. Args: column: Column or column name to count values in Returns: A Column expression representing the count aggregation Raises: TypeError: If column is not a Column or string """ if isinstance(column, str) and column == "*": return Column._from_logical_expr(CountExpr(lit("*")._logical_expr)) return Column._from_logical_expr( CountExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count_distinct ``` count_distinct(*cols: ColumnOrName) -> Column ``` Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – One or more columns or column names to include in the distinct count. Returns: - `Column` – A Column expression representing the count-distinct aggregation over the provided columns. Distinct count per group (single column) ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Distinct count across multiple columns (whole DataFrame) ``` # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Nulls in any input column are ignored for multi-column distinct ``` df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: - `ValidationError` – If no columns are provided. - `TypeMismatchError` – If a column has an unsupported type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count_distinct(*cols: ColumnOrName) -> Column: """Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Args: *cols: One or more columns or column names to include in the distinct count. Returns: A Column expression representing the count-distinct aggregation over the provided columns. Example: Distinct count per group (single column) ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Example: Distinct count across multiple columns (whole DataFrame) ```python # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Example: Nulls in any input column are ignored for multi-column distinct ```python df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: ValidationError: If no columns are provided. TypeMismatchError: If a column has an unsupported type """ if not cols: raise ValidationError("count_distinct requires at least one column") exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(CountDistinctExpr(exprs)) ``` ## create_mcp_server ``` create_mcp_server(session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8) -> FenicMCPServer ``` Create an MCP server from datasets and tools. Parameters: - **`session`** (`Session`) – Fenic session used to execute tools. - **`server_name`** (`str`) – Name of the MCP server. - **`user_defined_tools`** (`Optional[List[UserDefinedTool]]`, default: `None` ) – User defined tools to register with the MCP server. - **`system_tools`** (`Optional[SystemToolConfig]`, default: `None` ) – Configuration for automatically created system tools. - **`concurrency_limit`** (`int`, default: `8` ) – Maximum number of concurrent tool executions. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_mcp_server( session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8, ) -> FenicMCPServer: """Create an MCP server from datasets and tools. Args: session: Fenic session used to execute tools. server_name: Name of the MCP server. user_defined_tools: User defined tools to register with the MCP server. system_tools: Configuration for automatically created system tools. concurrency_limit: Maximum number of concurrent tool executions. """ generated_system_tools = [] user_defined_tools = user_defined_tools or [] if system_tools: generated_system_tools.extend( auto_generate_system_tools_from_tables( system_tools.table_names, session, tool_namespace=system_tools.tool_namespace, max_result_limit=system_tools.max_result_rows ) ) if not (user_defined_tools or system_tools): raise ConfigurationError("No tools provided. Either provide `user_defined_tools` or set `system_tools` to create system tools for catalog tables.") return FenicMCPServer(session._session_state, user_defined_tools, generated_system_tools, server_name, concurrency_limit) ``` ## desc ``` desc(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls first. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc() ``` ## desc_nulls_first ``` desc_nulls_first(column: ColumnOrName) -> Column ``` Alias for desc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_first(column: ColumnOrName) -> Column: """Alias for desc(). Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc_nulls_first() ``` ## desc_nulls_last ``` desc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls last. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls last. """ return Column._from_col_or_name(column).desc_nulls_last() ``` ## empty ``` empty(data_type: DataType) -> Column ``` Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the empty value Returns: - `Column` – A Column expression representing the empty value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with an empty array type ``` # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Creating a column with an empty struct type ``` # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Creating a column with an empty primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` Source code in `src/fenic/api/functions/core.py` ``` def empty(data_type: DataType) -> Column: """Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Args: data_type: The data type of the empty value Returns: A Column expression representing the empty value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with an empty array type ```python # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Example: Creating a column with an empty struct type ```python # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Example: Creating a column with an empty primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` """ if isinstance(data_type, ArrayType): return Column._from_logical_expr(LiteralExpr([], data_type)) elif isinstance(data_type, StructType): return Column._from_logical_expr(LiteralExpr({}, data_type)) return null(data_type) ``` ## first ``` first(column: ColumnOrName) -> Column ``` Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for the first value. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def first(column: ColumnOrName) -> Column: """Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Args: column: Column or column name. Returns: Column expression for the first value. """ return Column._from_logical_expr( FirstExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## flatten ``` flatten(column: ColumnOrName) -> Column ``` Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of arrays. Returns: - `Column` – A Column with flattened arrays (one level deep). Flattening nested arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` One level only ``` # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def flatten(column: ColumnOrName) -> Column: """Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Args: column: Column or column name containing arrays of arrays. Returns: A Column with flattened arrays (one level deep). Example: Flattening nested arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: One level only ```python # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` """ return Column._from_logical_expr( FlattenExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## greatest ``` greatest(*cols: ColumnOrName) -> Column ``` Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the greatest value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. greatest usage ``` df.select(fc.greatest("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def greatest(*cols: ColumnOrName) -> Column: """Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the greatest value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: greatest usage ```python df.select(fc.greatest("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"greatest() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(GreatestExpr(exprs)) ``` ## least ``` least(*cols: ColumnOrName) -> Column ``` Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the least value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. least usage ``` df.select(fc.least("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def least(*cols: ColumnOrName) -> Column: """Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the least value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: least usage ```python df.select(fc.least("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"least() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(LeastExpr(exprs)) ``` ## lit ``` lit(value: Any) -> Column ``` Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Parameters: - **`value`** (`Any`) – The literal value to create a column for Returns: - `Column` – A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred Source code in `src/fenic/api/functions/core.py` ``` def lit(value: Any) -> Column: """Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value: - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Args: value: The literal value to create a column for Returns: A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred """ if value is None: raise ValidationError("Cannot create a literal with value `None`. Use `null(...)` instead.") elif value == []: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(ArrayType(...))` instead.") elif value == {}: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(StructType(...))` instead.") try: inferred_type = infer_dtype_from_pyobj(value) except TypeInferenceError as e: raise ValidationError(f"`lit` failed to infer type for value `{value}`") from e literal_expr = LiteralExpr(value, inferred_type) return Column._from_logical_expr(literal_expr) ``` ## max ``` max(column: ColumnOrName) -> Column ``` Aggregate function: returns the maximum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the maximum of Returns: - `Column` – A Column expression representing the maximum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def max(column: ColumnOrName) -> Column: """Aggregate function: returns the maximum value in the specified column. Args: column: Column or column name to compute the maximum of Returns: A Column expression representing the maximum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MaxExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## mean ``` mean(column: ColumnOrName) -> Column ``` Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the mean of Returns: - `Column` – A Column expression representing the mean aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def mean(column: ColumnOrName) -> Column: """Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Args: column: Column or column name to compute the mean of Returns: A Column expression representing the mean aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## min ``` min(column: ColumnOrName) -> Column ``` Aggregate function: returns the minimum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the minimum of Returns: - `Column` – A Column expression representing the minimum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def min(column: ColumnOrName) -> Column: """Aggregate function: returns the minimum value in the specified column. Args: column: Column or column name to compute the minimum of Returns: A Column expression representing the minimum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MinExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## null ``` null(data_type: DataType) -> Column ``` Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the null value Returns: - `Column` – A Column expression representing the null value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with a null value of a primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Creating a column with a null value of an array/struct type ``` # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` Source code in `src/fenic/api/functions/core.py` ``` def null(data_type: DataType) -> Column: """Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Args: data_type: The data type of the null value Returns: A Column expression representing the null value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with a null value of a primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Example: Creating a column with a null value of an array/struct type ```python # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` """ return Column._from_logical_expr(LiteralExpr(None, data_type)) ``` ## run_mcp_server_asgi ``` run_mcp_server_asgi(server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`transport`** (`Literal['streamable-http', 'sse']`, default: `'streamable-http'` ) – Transport protocol to use (streamable-http, sse). - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional starlette-specific arguments to pass to FastMCP. Notes Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_asgi( server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Args: server: MCP server to run. stateless_http: If True, use stateless HTTP. transport: Transport protocol to use (streamable-http, sse). path: Path to listen on. kwargs: Additional starlette-specific arguments to pass to FastMCP. Notes: Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. """ return server.http_app(stateless_http=stateless_http, transport=transport, path=path, **kwargs) ``` ## run_mcp_server_async ``` run_mcp_server_async(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) async def run_mcp_server_async( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ await server.run_async(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## run_mcp_server_sync ``` run_mcp_server_sync(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_sync( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ server.run(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## stddev ``` stddev(column: ColumnOrName) -> Column ``` Aggregate function: returns the sample standard deviation of the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for sample standard deviation. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def stddev(column: ColumnOrName) -> Column: """Aggregate function: returns the sample standard deviation of the specified column. Args: column: Column or column name. Returns: Column expression for sample standard deviation. """ return Column._from_logical_expr( StdDevExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## struct ``` struct(*args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]) -> Column ``` Creates a new struct column from multiple input columns. Parameters: - **`*args`** (`Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]`, default: `()` ) – Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: - `Column` – A Column expression representing a struct containing the input columns Raises: - `TypeError` – If any argument is not a Column, string, or collection of Columns/strings Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def struct( *args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]] ) -> Column: """Creates a new struct column from multiple input columns. Args: *args: Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: A Column expression representing a struct containing the input columns Raises: TypeError: If any argument is not a Column, string, or collection of Columns/strings """ flattened_args = [] for arg in args: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_columns = [Column._from_col_or_name(c)._logical_expr for c in flattened_args] return Column._from_logical_expr(StructExpr(expr_columns)) ``` ## sum ``` sum(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of all values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of Returns: - `Column` – A Column expression representing the sum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of all values in the specified column. Args: column: Column or column name to compute the sum of Returns: A Column expression representing the sum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( SumExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## sum_distinct ``` sum_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of distinct numeric values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of distinct values Returns: - `Column` – A Column expression representing the sum-distinct aggregation Sum of distinct values per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Nulls are ignored when summing distinct values ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: - `TypeMismatchError` – If column is not a numeric or boolean type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of distinct numeric values in the specified column. Args: column: Column or column name to compute the sum of distinct values Returns: A Column expression representing the sum-distinct aggregation Example: Sum of distinct values per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Example: Nulls are ignored when summing distinct values ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: TypeMismatchError: If column is not a numeric or boolean type """ return Column._from_logical_expr( SumDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## tool_param ``` tool_param(parameter_name: str, data_type: DataType) -> Column ``` Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Parameters: - **`parameter_name`** (`str`) – The name of the parameter to reference. - **`data_type`** (`DataType`) – The expected data type for the parameter value. Returns: - `Column` – A Column wrapping an UnresolvedLiteralExpr for the given parameter. A simple tool with one parameter ```python ### Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) A tool with multiple filters ```python ### Assume we are reading data with an `age` column. df = session.read.csv(users.csv) ### create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def tool_param(parameter_name: str, data_type: DataType) -> Column: """Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes: Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Args: parameter_name: The name of the parameter to reference. data_type: The expected data type for the parameter value. Returns: A Column wrapping an UnresolvedLiteralExpr for the given parameter. Example: A simple tool with one parameter ```python # Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) Example: A tool with multiple filters ```python # Assume we are reading data with an `age` column. df = session.read.csv(users.csv) # create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) """ if isinstance(data_type, _LogicalType): raise ValidationError(f"Cannot use a logical type as a parameter type: {data_type}") return Column._from_logical_expr(UnresolvedLiteralExpr(data_type=data_type, parameter_name=parameter_name)) ``` ## udf ``` udf(f: Optional[Callable] = None, *, return_type: DataType) ``` A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Parameters: - **`f`** (`Optional[Callable]`, default: `None` ) – Python function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. UDF with primitive types ``` # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` UDF with nested types ``` # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def udf(f: Optional[Callable] = None, *, return_type: DataType): """A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning: UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Args: f: Python function to convert to UDF return_type: Expected return type of the UDF. Required parameter. Example: UDF with primitive types ```python # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` Example: UDF with nested types ```python # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` """ def _create_udf(func: Callable) -> Callable: @wraps(func) def _udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(UDFExpr(func, col_exprs, return_type)) return _udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for UDFs") if f is not None: return _create_udf(f) return _create_udf ``` ## when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A when expression that can be chained with more conditions Raises: - `TypeMismatchError` – If the condition is not a boolean Column expression. Example ``` # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def when(condition: Column, value: Column) -> Column: """Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A when expression that can be chained with more conditions Raises: TypeMismatchError: If the condition is not a boolean Column expression. Example: ```python # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null """ return Column._from_logical_expr( WhenExpr(None, condition._logical_expr, value._logical_expr) ) ``` --- # fenic.api.catalog Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/catalog/ Catalog API for managing database objects in Fenic. Classes: - **`Catalog`** – Entry point for catalog operations. ## Catalog ``` Catalog(catalog: BaseCatalog) ``` Entry point for catalog operations. Provides methods to manage catalogs, databases, and tables, as well as read-only access to system tables such as `fenic_system.query_metrics`. ##### Catalog and Database Management Example: ```python # Create a catalog session.catalog.create_catalog("my_catalog") # β†’ True ``` # Set active catalog session.catalog.set_current_catalog("my_catalog") # Create a database session.catalog.create_database("my_database") # β†’ True # Set active database session.catalog.set_current_database("my_database") # Create a table session.catalog.create_table( "my_table", Schema([ColumnField("id", IntegerType)]) ) # β†’ True ``` ``` ##### Metrics Table (Local Sessions Only) Query metrics are recorded for each session and stored locally in `fenic_system.query_metrics`. Metrics can be loaded into a DataFrame for analysis. Example ``` # Load all metrics for the current application metrics_df = session.table("fenic_system.query_metrics") # Show the 10 most recent queries in the application recent_queries = session.sql(""" SELECT * FROM {df} ORDER BY CAST(end_ts AS TIMESTAMP) DESC LIMIT 10 """, df=metrics_df) recent_queries.show() # Find query metrics for a specific session with non-zero LM costs specific_session_queries = session.sql(""" SELECT * FROM {df} WHERE session_id = '9e7e256f-fad9-4cd9-844e-399d795aaea0' AND total_lm_cost > 0 ORDER BY CAST(end_ts AS TIMESTAMP) ASC """, df=metrics_df) specific_session_queries.show() # Aggregate total LM costs and requests between a specific time window metrics_window = session.sql(""" SELECT CAST(SUM(total_lm_cost) AS DOUBLE) AS total_lm_cost_in_window, CAST(SUM(total_lm_requests) AS DOUBLE) AS total_lm_requests_in_window FROM {df} WHERE CAST(end_ts AS TIMESTAMP) BETWEEN CAST('2025-08-29 10:00:00' AS TIMESTAMP) AND CAST('2025-08-29 12:00:00' AS TIMESTAMP) """, df=metrics_df) metrics_window.show() ``` Initialize a Catalog instance. Parameters: - **`catalog`** (`BaseCatalog`) – The underlying catalog implementation. Methods: - **`create_catalog`** – Creates a new catalog. - **`create_database`** – Creates a new database. - **`create_table`** – Creates a new table. - **`create_tool`** – Creates a new tool in the current catalog. - **`describe_table`** – Returns the schema of the specified table. - **`describe_tool`** – Returns the tool with the specified name from the current catalog. - **`describe_view`** – Returns the schema and description of the specified view. - **`does_catalog_exist`** – Checks if a catalog with the specified name exists. - **`does_database_exist`** – Checks if a database with the specified name exists. - **`does_table_exist`** – Checks if a table with the specified name exists. - **`does_view_exist`** – Checks if a view with the specified name exists. - **`drop_catalog`** – Drops a catalog. - **`drop_database`** – Drops a database. - **`drop_table`** – Drops the specified table. - **`drop_tool`** – Drops the specified tool from the current catalog. - **`drop_view`** – Drops the specified view. - **`get_current_catalog`** – Returns the name of the current catalog. - **`get_current_database`** – Returns the name of the current database in the current catalog. - **`list_catalogs`** – Returns a list of available catalogs. - **`list_databases`** – Returns a list of databases in the current catalog. - **`list_tables`** – Returns a list of tables stored in the current database. - **`list_tools`** – Lists the tools available in the current catalog. - **`list_views`** – Returns a list of views stored in the current database. - **`set_current_catalog`** – Sets the current catalog. - **`set_current_database`** – Sets the current database. - **`set_table_description`** – Set or unset the description for a table. - **`set_view_description`** – Set the description for a view. Source code in `src/fenic/api/catalog.py` ``` def __init__(self, catalog: BaseCatalog): """Initialize a Catalog instance. Args: catalog: The underlying catalog implementation. """ self.catalog = catalog ``` ### create_catalog ``` create_catalog(catalog_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: - `CatalogAlreadyExistsError` – If the catalog already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the catalog was created successfully, False if the catalog - `bool` – already exists and ignore_if_exists is True. Create a new catalog ``` # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Create an existing catalog with ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Create an existing catalog without ignore_if_exists ``` # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_catalog(self, catalog_name: str, ignore_if_exists: bool = True) -> bool: """Creates a new catalog. Args: catalog_name (str): Name of the catalog to create. ignore_if_exists (bool): If True, return False when the catalog already exists. If False, raise an error when the catalog already exists. Defaults to True. Raises: CatalogAlreadyExistsError: If the catalog already exists and ignore_if_exists is False. Returns: bool: True if the catalog was created successfully, False if the catalog already exists and ignore_if_exists is True. Example: Create a new catalog ```python # Create a new catalog named 'my_catalog' session.catalog.create_catalog('my_catalog') # Returns: True ``` Example: Create an existing catalog with ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=True session.catalog.create_catalog('my_catalog', ignore_if_exists=True) # Returns: False ``` Example: Create an existing catalog without ignore_if_exists ```python # Try to create an existing catalog with ignore_if_exists=False session.catalog.create_catalog('my_catalog', ignore_if_exists=False) # Raises: CatalogAlreadyExistsError ``` """ return self.catalog.create_catalog(catalog_name, ignore_if_exists) ``` ### create_database ``` create_database(database_name: str, ignore_if_exists: bool = True) -> bool ``` Creates a new database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: - `DatabaseAlreadyExistsError` – If the database already exists and ignore_if_exists is False. Returns: - **`bool`** ( `bool` ) – True if the database was created successfully, False if the database - `bool` – already exists and ignore_if_exists is True. Create a new database ``` # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Create an existing database with ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Create an existing database without ignore_if_exists ``` # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_database( self, database_name: str, ignore_if_exists: bool = True ) -> bool: """Creates a new database. Args: database_name (str): Fully qualified or relative database name to create. ignore_if_exists (bool): If True, return False when the database already exists. If False, raise an error when the database already exists. Defaults to True. Raises: DatabaseAlreadyExistsError: If the database already exists and ignore_if_exists is False. Returns: bool: True if the database was created successfully, False if the database already exists and ignore_if_exists is True. Example: Create a new database ```python # Create a new database named 'my_database' session.catalog.create_database('my_database') # Returns: True ``` Example: Create an existing database with ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=True session.catalog.create_database('my_database', ignore_if_exists=True) # Returns: False ``` Example: Create an existing database without ignore_if_exists ```python # Try to create an existing database with ignore_if_exists=False session.catalog.create_database('my_database', ignore_if_exists=False) # Raises: DatabaseAlreadyExistsError ``` """ return self.catalog.create_database(database_name, ignore_if_exists) ``` ### create_table ``` create_table(table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None) -> bool ``` Creates a new table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to create. - **`schema`** (`Schema`) – Schema of the table to create. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. - **`description`** (`Optional[str]`, default: `None` ) – Description of the table to create. Defaults to None. Returns: - **`bool`** ( `bool` ) – True if the table was created successfully, False if the table - `bool` – already exists and ignore_if_exists is True. Raises: - `TableAlreadyExistsError` – If the table already exists and ignore_if_exists is False Create a new table ``` # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Create an existing table with ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Create an existing table without ignore_if_exists ``` # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def create_table( self, table_name: str, schema: Schema, ignore_if_exists: bool = True, description: Optional[str] = None ) -> bool: """Creates a new table. Args: table_name (str): Fully qualified or relative table name to create. schema (Schema): Schema of the table to create. ignore_if_exists (bool): If True, return False when the table already exists. If False, raise an error when the table already exists. Defaults to True. description (Optional[str]): Description of the table to create. Defaults to None. Returns: bool: True if the table was created successfully, False if the table already exists and ignore_if_exists is True. Raises: TableAlreadyExistsError: If the table already exists and ignore_if_exists is False Example: Create a new table ```python # Create a new table with an integer column session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), description='My table description') # Returns: True ``` Example: Create an existing table with ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=True session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=True, description='My table description') # Returns: False ``` Example: Create an existing table without ignore_if_exists ```python # Try to create an existing table with ignore_if_exists=False session.catalog.create_table('my_table', Schema([ ColumnField('id', IntegerType), ]), ignore_if_exists=False, description='My table description') # Raises: TableAlreadyExistsError ``` """ return self.catalog.create_table(table_name, schema, ignore_if_exists, description) ``` ### create_tool ``` create_tool(tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True) -> bool ``` Creates a new tool in the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool. - **`tool_description`** (`str`) – The description of the tool. - **`tool_query`** (`DataFrame`) – The query to execute when the tool is called. - **`tool_params`** (`Sequence[ToolParam]`) – The parameters of the tool. - **`result_limit`** (`int`, default: `50` ) – The maximum number of rows to return from the tool. - **`ignore_if_exists`** (`bool`, default: `True` ) – If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was created successfully, False otherwise. Raises: - `ToolAlreadyExistsError` – If the tool already exists. Examples: ``` # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_tool( self, tool_name: str, tool_description: str, tool_query: DataFrame, tool_params: List[ToolParam], result_limit: int = 50, ignore_if_exists: bool = True ) -> bool: """Creates a new tool in the current catalog. Args: tool_name (str): The name of the tool. tool_description (str): The description of the tool. tool_query (DataFrame): The query to execute when the tool is called. tool_params (Sequence[ToolParam]): The parameters of the tool. result_limit (int): The maximum number of rows to return from the tool. ignore_if_exists (bool): If True, return False when the tool already exists. If False, raise an error when the tool already exists. Defaults to True. Returns: bool: True if the tool was created successfully, False otherwise. Raises: ToolAlreadyExistsError: If the tool already exists. Examples: ```python # Create a new tool with a single parameter df = session.create_dataframe(...) session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that does something", tool_query=df, result_limit=100, tool_params=[ToolParam(name="param1", description="A parameter", allowed_values=["value1", "value2"])], ) # Returns: True ``` """ return self.catalog.create_tool( tool_name, tool_description, tool_params, tool_query._logical_plan, result_limit, ignore_if_exists, ) ``` ### describe_table ``` describe_table(table_name: str) -> DatasetMetadata ``` Returns the schema of the specified table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: - `TableNotFoundError` – If the table doesn't exist. Describe a table's schema ``` # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_table(self, table_name: str) -> DatasetMetadata: """Returns the schema of the specified table. Args: table_name (str): Fully qualified or relative table name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the table's structure with field names and types. description: A natural language description of the table's contents and uses. Raises: TableNotFoundError: If the table doesn't exist. Example: Describe a table's schema ```python # For a table created with: create_table('t1', Schema([ColumnField('id', IntegerType)]), description='My table description') session.catalog.describe_table('t1') # Returns: DatasetMetadata(schema=Schema([ # ColumnField('id', IntegerType), # ]), description="My table description") ``` """ return self.catalog.describe_table(table_name) ``` ### describe_tool ``` describe_tool(tool_name: str) -> UserDefinedTool ``` Returns the tool with the specified name from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to get. Raises: - `ToolNotFoundError` – If the tool doesn't exist. Returns: - **`Tool`** ( `UserDefinedTool` ) – The tool with the specified name. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_tool(self, tool_name: str) -> UserDefinedTool: """Returns the tool with the specified name from the current catalog. Args: tool_name (str): The name of the tool to get. Raises: ToolNotFoundError: If the tool doesn't exist. Returns: Tool: The tool with the specified name. """ return self.catalog.describe_tool(tool_name) ``` ### describe_view ``` describe_view(view_name: str) -> DatasetMetadata ``` Returns the schema and description of the specified view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to describe. Returns: - **`DatasetMetadata`** ( `DatasetMetadata` ) – An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: - `TableNotFoundError` – If the view doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def describe_view(self, view_name: str) -> DatasetMetadata: """Returns the schema and description of the specified view. Args: view_name (str): Fully qualified or relative view name to describe. Returns: DatasetMetadata: An object containing: schema: A schema object describing the view's structure with field names and types. description: A natural language description of the view's contents and uses. Raises: TableNotFoundError: If the view doesn't exist. """ return self.catalog.describe_view(view_name) ``` ### does_catalog_exist ``` does_catalog_exist(catalog_name: str) -> bool ``` Checks if a catalog with the specified name exists. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to check. Returns: - **`bool`** ( `bool` ) – True if the catalog exists, False otherwise. Check if a catalog exists ``` # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_catalog_exist(self, catalog_name: str) -> bool: """Checks if a catalog with the specified name exists. Args: catalog_name (str): Name of the catalog to check. Returns: bool: True if the catalog exists, False otherwise. Example: Check if a catalog exists ```python # Check if 'my_catalog' exists session.catalog.does_catalog_exist('my_catalog') # Returns: True ``` """ return self.catalog.does_catalog_exist(catalog_name) ``` ### does_database_exist ``` does_database_exist(database_name: str) -> bool ``` Checks if a database with the specified name exists. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to check. Returns: - **`bool`** ( `bool` ) – True if the database exists, False otherwise. Check if a database exists ``` # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_database_exist(self, database_name: str) -> bool: """Checks if a database with the specified name exists. Args: database_name (str): Fully qualified or relative database name to check. Returns: bool: True if the database exists, False otherwise. Example: Check if a database exists ```python # Check if 'my_database' exists session.catalog.does_database_exist('my_database') # Returns: True ``` """ return self.catalog.does_database_exist(database_name) ``` ### does_table_exist ``` does_table_exist(table_name: str) -> bool ``` Checks if a table with the specified name exists. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to check. Returns: - **`bool`** ( `bool` ) – True if the table exists, False otherwise. Check if a table exists ``` # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_table_exist(self, table_name: str) -> bool: """Checks if a table with the specified name exists. Args: table_name (str): Fully qualified or relative table name to check. Returns: bool: True if the table exists, False otherwise. Example: Check if a table exists ```python # Check if 'my_table' exists session.catalog.does_table_exist('my_table') # Returns: True ``` """ return self.catalog.does_table_exist(table_name) ``` ### does_view_exist ``` does_view_exist(view_name: str) -> bool ``` Checks if a view with the specified name exists. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to check. Returns: - **`bool`** ( `bool` ) – True if the view exists, False otherwise. Example > > > session.catalog.does_view_exist('my_view') > > > True. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def does_view_exist(self, view_name: str) -> bool: """Checks if a view with the specified name exists. Args: view_name (str): Fully qualified or relative view name to check. Returns: bool: True if the view exists, False otherwise. Example: >>> session.catalog.does_view_exist('my_view') True. """ return self.catalog.does_view_exist(view_name) ``` ### drop_catalog ``` drop_catalog(catalog_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops a catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: - `CatalogNotFoundError` – If the catalog does not exist and ignore_if_not_exists is False Returns: - **`bool`** ( `bool` ) – True if the catalog was dropped successfully, False if the catalog - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent catalog ``` # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Drop a non-existent catalog without ignore_if_not_exists ``` # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_catalog( self, catalog_name: str, ignore_if_not_exists: bool = True ) -> bool: """Drops a catalog. Args: catalog_name (str): Name of the catalog to drop. ignore_if_not_exists (bool): If True, silently return if the catalog doesn't exist. If False, raise an error if the catalog doesn't exist. Defaults to True. Raises: CatalogNotFoundError: If the catalog does not exist and ignore_if_not_exists is False Returns: bool: True if the catalog was dropped successfully, False if the catalog didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent catalog ```python # Try to drop a non-existent catalog session.catalog.drop_catalog('my_catalog') # Returns: False ``` Example: Drop a non-existent catalog without ignore_if_not_exists ```python # Try to drop a non-existent catalog with ignore_if_not_exists=False session.catalog.drop_catalog('my_catalog', ignore_if_not_exists=False) # Raises: CatalogNotFoundError ``` """ return self.catalog.drop_catalog(catalog_name, ignore_if_not_exists) ``` ### drop_database ``` drop_database(database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True) -> bool ``` Drops a database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to drop. - **`cascade`** (`bool`, default: `False` ) – If True, drop all tables in the database. Defaults to False. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: - `DatabaseNotFoundError` – If the database does not exist and ignore_if_not_exists is False - `CatalogError` – If the current database is being dropped, if the database is not empty and cascade is False Returns: - **`bool`** ( `bool` ) – True if the database was dropped successfully, False if the database - `bool` – didn't exist and ignore_if_not_exists is True. Drop a non-existent database ``` # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Drop a non-existent database without ignore_if_not_exists ``` # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_database( self, database_name: str, cascade: bool = False, ignore_if_not_exists: bool = True, ) -> bool: """Drops a database. Args: database_name (str): Fully qualified or relative database name to drop. cascade (bool): If True, drop all tables in the database. Defaults to False. ignore_if_not_exists (bool): If True, silently return if the database doesn't exist. If False, raise an error if the database doesn't exist. Defaults to True. Raises: DatabaseNotFoundError: If the database does not exist and ignore_if_not_exists is False CatalogError: If the current database is being dropped, if the database is not empty and cascade is False Returns: bool: True if the database was dropped successfully, False if the database didn't exist and ignore_if_not_exists is True. Example: Drop a non-existent database ```python # Try to drop a non-existent database session.catalog.drop_database('my_database') # Returns: False ``` Example: Drop a non-existent database without ignore_if_not_exists ```python # Try to drop a non-existent database with ignore_if_not_exists=False session.catalog.drop_database('my_database', ignore_if_not_exists=False) # Raises: DatabaseNotFoundError ``` """ return self.catalog.drop_database(database_name, cascade, ignore_if_not_exists) ``` ### drop_table ``` drop_table(table_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified table. By default this method will return False if the table doesn't exist. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the table was dropped successfully, False if the table - `bool` – didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the table doesn't exist and ignore_if_not_exists is False Drop an existing table ``` # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Drop a non-existent table with ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Drop a non-existent table without ignore_if_not_exists ``` # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_table(self, table_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified table. By default this method will return False if the table doesn't exist. Args: table_name (str): Fully qualified or relative table name to drop. ignore_if_not_exists (bool): If True, return False when the table doesn't exist. If False, raise an error when the table doesn't exist. Defaults to True. Returns: bool: True if the table was dropped successfully, False if the table didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the table doesn't exist and ignore_if_not_exists is False Example: Drop an existing table ```python # Drop an existing table 't1' session.catalog.drop_table('t1') # Returns: True ``` Example: Drop a non-existent table with ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=True session.catalog.drop_table('t2', ignore_if_not_exists=True) # Returns: False ``` Example: Drop a non-existent table without ignore_if_not_exists ```python # Try to drop a non-existent table with ignore_if_not_exists=False session.catalog.drop_table('t2', ignore_if_not_exists=False) # Raises: TableNotFoundError ``` """ return self.catalog.drop_table(table_name, ignore_if_not_exists) ``` ### drop_tool ``` drop_tool(tool_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified tool from the current catalog. Parameters: - **`tool_name`** (`str`) – The name of the tool to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: - `ToolNotFoundError` – If the tool doesn't exist and ignore_if_not_exists is False Example > > > session.catalog.drop_tool('my_tool') > > > True > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) > > > False > > > session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) #### Raises ToolNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_tool(self, tool_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified tool from the current catalog. Args: tool_name (str): The name of the tool to drop. ignore_if_not_exists (bool): If True, return False when the tool doesn't exist. If False, raise an error when the tool doesn't exist. Defaults to True. Returns: bool: True if the tool was dropped successfully, False if the tool didn't exist and ignore_if_not_exists is True. Raises: ToolNotFoundError: If the tool doesn't exist and ignore_if_not_exists is False Example: >>> session.catalog.drop_tool('my_tool') True >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=True) False >>> session.catalog.drop_tool('my_tool', ignore_if_not_exists=False) # Raises ToolNotFoundError. """ return self.catalog.drop_tool(tool_name, ignore_if_not_exists) ``` ### drop_view ``` drop_view(view_name: str, ignore_if_not_exists: bool = True) -> bool ``` Drops the specified view. By default this method will return False if the view doesn't exist. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to drop. - **`ignore_if_not_exists`** (`bool`, default: `True` ) – If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: - **`bool`** ( `bool` ) – True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: - `TableNotFoundError` – If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def drop_view(self, view_name: str, ignore_if_not_exists: bool = True) -> bool: """Drops the specified view. By default this method will return False if the view doesn't exist. Args: view_name (str): Fully qualified or relative view name to drop. ignore_if_not_exists (bool, optional): If True, return False when the view doesn't exist. If False, raise an error when the view doesn't exist. Defaults to True. Returns: bool: True if the view was dropped successfully, False if the view didn't exist and ignore_if_not_exist is True. Raises: TableNotFoundError: If the view doesn't exist and ignore_if_not_exists is False Example: >>> # For an existing view 'v1' >>> session.catalog.drop_table('v1') True >>> # For a non-existent table 'v2' >>> session.catalog.drop_table('v2', ignore_if_not_exists=True) False >>> session.catalog.drop_table('v2', ignore_if_not_exists=False) # Raises TableNotFoundError. """ return self.catalog.drop_view(view_name, ignore_if_not_exists) ``` ### get_current_catalog ``` get_current_catalog() -> str ``` Returns the name of the current catalog. Returns: - **`str`** ( `str` ) – The name of the current catalog. Get current catalog name ``` # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_catalog(self) -> str: """Returns the name of the current catalog. Returns: str: The name of the current catalog. Example: Get current catalog name ```python # Get the name of the current catalog session.catalog.get_current_catalog() # Returns: 'default' ``` """ return self.catalog.get_current_catalog() ``` ### get_current_database ``` get_current_database() -> str ``` Returns the name of the current database in the current catalog. Returns: - **`str`** ( `str` ) – The name of the current database. Get current database name ``` # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` Source code in `src/fenic/api/catalog.py` ``` def get_current_database(self) -> str: """Returns the name of the current database in the current catalog. Returns: str: The name of the current database. Example: Get current database name ```python # Get the name of the current database session.catalog.get_current_database() # Returns: 'default' ``` """ return self.catalog.get_current_database() ``` ### list_catalogs ``` list_catalogs() -> List[str] ``` Returns a list of available catalogs. Returns: - `List[str]` – List[str]: A list of catalog names available in the system. - `List[str]` – Returns an empty list if no catalogs are found. List all catalogs ``` # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_catalogs(self) -> List[str]: """Returns a list of available catalogs. Returns: List[str]: A list of catalog names available in the system. Returns an empty list if no catalogs are found. Example: List all catalogs ```python # Get all available catalogs session.catalog.list_catalogs() # Returns: ['default', 'my_catalog', 'other_catalog'] ``` """ return self.catalog.list_catalogs() ``` ### list_databases ``` list_databases() -> List[str] ``` Returns a list of databases in the current catalog. Returns: - `List[str]` – List[str]: A list of database names in the current catalog. - `List[str]` – Returns an empty list if no databases are found. List all databases ``` # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_databases(self) -> List[str]: """Returns a list of databases in the current catalog. Returns: List[str]: A list of database names in the current catalog. Returns an empty list if no databases are found. Example: List all databases ```python # Get all databases in the current catalog session.catalog.list_databases() # Returns: ['default', 'my_database', 'other_database'] ``` """ return self.catalog.list_databases() ``` ### list_tables ``` list_tables() -> List[str] ``` Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: - `List[str]` – List[str]: A list of table names stored in the database. - `List[str]` – Returns an empty list if no tables are found. List all tables ``` # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` Source code in `src/fenic/api/catalog.py` ``` def list_tables(self) -> List[str]: """Returns a list of tables stored in the current database. This method queries the current database to retrieve all available table names. Returns: List[str]: A list of table names stored in the database. Returns an empty list if no tables are found. Example: List all tables ```python # Get all tables in the current database session.catalog.list_tables() # Returns: ['table1', 'table2', 'table3'] ``` """ return self.catalog.list_tables() ``` ### list_tools ``` list_tools() -> List[UserDefinedTool] ``` Lists the tools available in the current catalog. Source code in `src/fenic/api/catalog.py` ``` def list_tools(self) -> List[UserDefinedTool]: """Lists the tools available in the current catalog.""" return self.catalog.list_tools() ``` ### list_views ``` list_views() -> List[str] ``` Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: - `List[str]` – List[str]: A list of view names stored in the database. - `List[str]` – Returns an empty list if no views are found. Example > > > session.catalog.list_views() > > > ['view1', 'view2', 'view3']. Source code in `src/fenic/api/catalog.py` ``` def list_views(self) -> List[str]: """Returns a list of views stored in the current database. This method queries the current database to retrieve all available view names. Returns: List[str]: A list of view names stored in the database. Returns an empty list if no views are found. Example: >>> session.catalog.list_views() ['view1', 'view2', 'view3']. """ return self.catalog.list_views() ``` ### set_current_catalog ``` set_current_catalog(catalog_name: str) -> None ``` Sets the current catalog. Parameters: - **`catalog_name`** (`str`) – Name of the catalog to set as current. Raises: - `ValueError` – If the specified catalog doesn't exist. Set current catalog ``` # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_catalog(self, catalog_name: str) -> None: """Sets the current catalog. Args: catalog_name (str): Name of the catalog to set as current. Raises: ValueError: If the specified catalog doesn't exist. Example: Set current catalog ```python # Set 'my_catalog' as the current catalog session.catalog.set_current_catalog('my_catalog') ``` """ self.catalog.set_current_catalog(catalog_name) ``` ### set_current_database ``` set_current_database(database_name: str) -> None ``` Sets the current database. Parameters: - **`database_name`** (`str`) – Fully qualified or relative database name to set as current. Raises: - `DatabaseNotFoundError` – If the specified database doesn't exist. Set current database ``` # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_current_database(self, database_name: str) -> None: """Sets the current database. Args: database_name (str): Fully qualified or relative database name to set as current. Raises: DatabaseNotFoundError: If the specified database doesn't exist. Example: Set current database ```python # Set 'my_database' as the current database session.catalog.set_current_database('my_database') ``` """ self.catalog.set_current_database(database_name) ``` ### set_table_description ``` set_table_description(table_name: str, description: Optional[str] = None) -> None ``` Set or unset the description for a table. Parameters: - **`table_name`** (`str`) – Fully qualified or relative table name to set the description for. - **`description`** (`Optional[str]`, default: `None` ) – The description to set for the table. Raises: - `TableNotFoundError` – If the table doesn't exist. Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_table_description(self, table_name: str, description: Optional[str] = None) -> None: """Set or unset the description for a table. Args: table_name: Fully qualified or relative table name to set the description for. description: The description to set for the table. Raises: TableNotFoundError: If the table doesn't exist. """ self.catalog.set_table_description(table_name, description) ``` ### set_view_description ``` set_view_description(view_name: str, description: Optional[str] = None) -> None ``` Set the description for a view. Parameters: - **`view_name`** (`str`) – Fully qualified or relative view name to set the description for. - **`description`** (`str`, default: `None` ) – The description to set for the view. Raises: - `TableNotFoundError` – If the view doesn't exist. - `ValidationError` – If the description is empty. Set a description for a view ```python #### Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') Source code in `src/fenic/api/catalog.py` ``` @validate_call(config=ConfigDict(strict=True)) def set_view_description(self, view_name: str, description: Optional[str] = None) -> None: """Set the description for a view. Args: view_name (str): Fully qualified or relative view name to set the description for. description (str): The description to set for the view. Raises: TableNotFoundError: If the view doesn't exist. ValidationError: If the description is empty. Example: Set a description for a view ```python # Set a description for a view 'v1' session.catalog.set_view_description('v1', 'My view description') """ self.catalog.set_view_description(view_name, description) ``` --- # fenic.api.column Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/column/ Column API for Fenic DataFrames - represents column expressions and operations. Classes: - **`Column`** – A column expression in a DataFrame. ## Column A column expression in a DataFrame. This class represents a column expression that can be used in DataFrame operations. It provides methods for accessing, transforming, and combining column data. Create a column reference ``` # Reference a column by name using col() function col("column_name") ``` Use column in operations ``` # Perform arithmetic operations df.select(col("price") * col("quantity")) ``` Chain column operations ``` # Chain multiple operations df.select(col("name").upper().contains("John")) ``` Methods: - **`alias`** – Create an alias for this column. - **`asc`** – Mark this column for ascending sort order. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`cast`** – Cast the column to a new data type. - **`contains`** – Check if the column contains a substring. - **`contains_any`** – Check if the column contains any of the specified substrings. - **`desc`** – Mark this column for descending sort order. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`ends_with`** – Check if the column ends with a substring. - **`get_item`** – Access an item in a struct or array column. - **`ilike`** – Check if the column matches a SQL LIKE pattern (case-insensitive). - **`is_in`** – Check if the column is in a list of values or a column expression. - **`is_not_null`** – Check if the column contains non-NULL values. - **`is_null`** – Check if the column contains NULL values. - **`like`** – Check if the column matches a SQL LIKE pattern. - **`otherwise`** – Returns a value when no prior conditions are True. - **`rlike`** – Check if the column matches a regular expression pattern. - **`starts_with`** – Check if the column starts with a substring. - **`when`** – Evaluates a condition for each row and returns a value when true. ### alias ``` alias(name: str) -> Column ``` Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Parameters: - **`name`** (`str`) – The alias name to assign Returns: - **`Column`** ( `Column` ) – Column with the specified alias Rename a column ``` # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Name a complex expression ``` # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` Source code in `src/fenic/api/column.py` ``` def alias(self, name: str) -> Column: """Create an alias for this column. This method assigns a new name to the column expression, which is useful for renaming columns or providing names for complex expressions. Args: name (str): The alias name to assign Returns: Column: Column with the specified alias Example: Rename a column ```python # Rename a column to a new name df.select(col("original_name").alias("new_name")) ``` Example: Name a complex expression ```python # Give a name to a calculated column df.select((col("price") * col("quantity")).alias("total_value")) ``` """ return Column._from_logical_expr(AliasExpr(self._logical_expr, name)) ``` ### asc ``` asc() -> Column ``` Mark this column for ascending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls first. Sort by age in ascending order ``` # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc(self) -> Column: """Mark this column for ascending sort order. Returns: Column: A sort expression with ascending order and nulls first. Example: Sort by age in ascending order ```python # Sort a dataframe by age in ascending order df.sort(col("age").asc()).show() ``` """ return Column._from_logical_expr(SortExpr(self._logical_expr, ascending=True)) ``` ### asc_nulls_first ``` asc_nulls_first() -> Column ``` Alias for asc(). Returns: - **`Column`** ( `Column` ) – A Column expression that provides a column and sort order to the sort function Source code in `src/fenic/api/column.py` ``` def asc_nulls_first(self) -> Column: """Alias for asc(). Returns: Column: A Column expression that provides a column and sort order to the sort function """ return self.asc() ``` ### asc_nulls_last ``` asc_nulls_last() -> Column ``` Mark this column for ascending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with ascending order and nulls last. Sort by age in ascending order with nulls last ``` # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def asc_nulls_last(self) -> Column: """Mark this column for ascending sort order with nulls last. Returns: Column: A sort expression with ascending order and nulls last. Example: Sort by age in ascending order with nulls last ```python # Sort a dataframe by age in ascending order, with nulls appearing last df.sort(col("age").asc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=True, nulls_last=True) ) ``` ### cast ``` cast(data_type: DataType) -> Column ``` Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Parameters: - **`data_type`** (`DataType`) – The target DataType to cast the column to Returns: - **`Column`** ( `Column` ) – A Column representing the casted expression Cast integer to string ``` # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Cast array of integers to array of strings ``` # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Cast struct fields to different types ``` # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: - `TypeError` – If the requested cast operation is not supported Source code in `src/fenic/api/column.py` ``` def cast(self, data_type: DataType) -> Column: """Cast the column to a new data type. This method creates an expression that casts the column to a specified data type. The casting behavior depends on the source and target types: Primitive type casting: - Numeric types (IntegerType, FloatType, DoubleType) can be cast between each other - Numeric types can be cast to/from StringType - DateType and TimestampType can be cast between each other - DateType and TimestampType can be cast to/from numeric types and StringType - BooleanType can be cast to/from numeric types and StringType - StringType cannot be directly cast to BooleanType (will raise TypeError) - BooleanType cannot be cast to DateType or TimestampType (will raise TypeError) - DateType and TimestampType cannot be cast to BooleanType (will raise TypeError) Complex type casting: - ArrayType can only be cast to another ArrayType (with castable element types) - StructType can only be cast to another StructType (with matching/castable fields) - Primitive types cannot be cast to/from complex types Args: data_type (DataType): The target DataType to cast the column to Returns: Column: A Column representing the casted expression Example: Cast integer to string ```python # Convert an integer column to string type df.select(col("int_col").cast(StringType)) ``` Example: Cast array of integers to array of strings ```python # Convert an array of integers to an array of strings df.select(col("int_array").cast(ArrayType(element_type=StringType))) ``` Example: Cast struct fields to different types ```python # Convert struct fields to different types new_type = StructType([ StructField("id", StringType), StructField("value", FloatType) ]) df.select(col("data_struct").cast(new_type)) ``` Raises: TypeError: If the requested cast operation is not supported """ return Column._from_logical_expr(CastExpr(self._logical_expr, data_type)) ``` ### contains ``` contains(other: Union[str, Column]) -> Column ``` Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to search for (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains the substring Find rows where name contains "john" ``` # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Find rows where text contains a dynamic pattern ``` # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` Source code in `src/fenic/api/column.py` ``` def contains(self, other: Union[str, Column]) -> Column: """Check if the column contains a substring. This method creates a boolean expression that checks if each value in the column contains the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to search for (can be a string or column expression) Returns: Column: A boolean column indicating whether each value contains the substring Example: Find rows where name contains "john" ```python # Filter rows where the name column contains "john" df.filter(col("name").contains("john")) ``` Example: Find rows where text contains a dynamic pattern ```python # Filter rows where text contains a value from another column df.filter(col("text").contains(col("pattern"))) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ContainsExpr(self._logical_expr, other_expr)) ``` ### contains_any ``` contains_any(others: List[str], case_insensitive: bool = True) -> Column ``` Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Parameters: - **`others`** (`List[str]`) – List of substrings to search for - **`case_insensitive`** (`bool`, default: `True` ) – Whether to perform case-insensitive matching (default: True) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value contains any substring Find rows where name contains "john" or "jane" (case-insensitive) ``` # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Case-sensitive matching ``` # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` Source code in `src/fenic/api/column.py` ``` def contains_any(self, others: List[str], case_insensitive: bool = True) -> Column: """Check if the column contains any of the specified substrings. This method creates a boolean expression that checks if each value in the column contains any of the specified substrings. The matching can be case-sensitive or case-insensitive. Args: others (List[str]): List of substrings to search for case_insensitive (bool): Whether to perform case-insensitive matching (default: True) Returns: Column: A boolean column indicating whether each value contains any substring Example: Find rows where name contains "john" or "jane" (case-insensitive) ```python # Filter rows where name contains either "john" or "jane" df.filter(col("name").contains_any(["john", "jane"])) ``` Example: Case-sensitive matching ```python # Filter rows with case-sensitive matching df.filter(col("name").contains_any(["John", "Jane"], case_insensitive=False)) ``` """ return Column._from_logical_expr( ContainsAnyExpr(self._logical_expr, others, case_insensitive) ) ``` ### desc ``` desc() -> Column ``` Mark this column for descending sort order. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order. Sort by age in descending order ``` # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc(self) -> Column: """Mark this column for descending sort order. Returns: Column: A sort expression with descending order. Example: Sort by age in descending order ```python # Sort a dataframe by age in descending order df.sort(col("age").desc()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False) ) ``` ### desc_nulls_first ``` desc_nulls_first() -> Column ``` Alias for desc(). Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls first. Sort by age in descending order with nulls first ``` df.sort(col("age").desc_nulls_first()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_first(self) -> Column: """Alias for desc(). Returns: Column: A sort expression with descending order and nulls first. Example: Sort by age in descending order with nulls first ```python df.sort(col("age").desc_nulls_first()).show() ``` """ return self.desc() ``` ### desc_nulls_last ``` desc_nulls_last() -> Column ``` Mark this column for descending sort order with nulls last. Returns: - **`Column`** ( `Column` ) – A sort expression with descending order and nulls last. Sort by age in descending order with nulls last ``` # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` Source code in `src/fenic/api/column.py` ``` def desc_nulls_last(self) -> Column: """Mark this column for descending sort order with nulls last. Returns: Column: A sort expression with descending order and nulls last. Example: Sort by age in descending order with nulls last ```python # Sort a dataframe by age in descending order, with nulls appearing last df.sort(col("age").desc_nulls_last()).show() ``` """ return Column._from_logical_expr( SortExpr(self._logical_expr, ascending=False, nulls_last=True) ) ``` ### ends_with ``` ends_with(other: Union[str, Column]) -> Column ``` Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the end (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value ends with the substring Find rows where email ends with "@gmail.com" ``` df.filter(col("email").ends_with("@gmail.com")) ``` Find rows where text ends with a dynamic pattern ``` df.filter(col("text").ends_with(col("suffix"))) ``` Raises: - `ValueError` – If the substring ends with a regular expression anchor ($) Source code in `src/fenic/api/column.py` ``` def ends_with(self, other: Union[str, Column]) -> Column: """Check if the column ends with a substring. This method creates a boolean expression that checks if each value in the column ends with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the end (can be a string or column expression) Returns: Column: A boolean column indicating whether each value ends with the substring Example: Find rows where email ends with "@gmail.com" ```python df.filter(col("email").ends_with("@gmail.com")) ``` Example: Find rows where text ends with a dynamic pattern ```python df.filter(col("text").ends_with(col("suffix"))) ``` Raises: ValueError: If the substring ends with a regular expression anchor ($) """ if isinstance(other, str): if other.endswith("$"): raise ValidationError("substr should not end with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(EndsWithExpr(self._logical_expr, other_expr)) ``` ### get_item ``` get_item(key: Union[str, int, Column]) -> Column ``` Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Parameters: - **`key`** (`Union[str, int]`) – The index (for arrays) or field name (for structs) to access Returns: - **`Column`** ( `Column` ) – A Column representing the accessed item Access an array element ``` # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Access a struct field ``` # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` Source code in `src/fenic/api/column.py` ``` def get_item(self, key: Union[str, int, Column]) -> Column: """Access an item in a struct or array column. This method allows accessing elements in complex data types: - For array columns, the key should be an integer index or a column expression that evaluates to an integer - For struct columns, the key should be a literal field name Args: key (Union[str, int]): The index (for arrays) or field name (for structs) to access Returns: Column: A Column representing the accessed item Example: Access an array element ```python # Get the first element from an array column df.select(col("array_column").get_item(0)) ``` Example: Access a struct field ```python # Get a field from a struct column df.select(col("struct_column").get_item("field_name")) ``` """ if isinstance(key, Column): return Column._from_logical_expr(IndexExpr(self._logical_expr, key._logical_expr)) elif isinstance(key, str): return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, StringType))) else: return Column._from_logical_expr(IndexExpr(self._logical_expr, LiteralExpr(key, IntegerType))) ``` ### ilike ``` ilike(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "j" and ends with "n" (case-insensitive) ``` # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Find rows where code matches pattern (case-insensitive) ``` # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` Source code in `src/fenic/api/column.py` ``` def ilike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern (case-insensitive). This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern, ignoring case. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "j" and ends with "n" (case-insensitive) ```python # Filter rows where name matches the pattern "j%n" (case-insensitive) df.filter(col("name").ilike("j%n")) ``` Example: Find rows where code matches pattern (case-insensitive) ```python # Filter rows where code matches the pattern "a_b%" (case-insensitive) df.filter(col("code").ilike("a_b%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(ILikeExpr(self._logical_expr, other_expr)) ``` ### is_in ``` is_in(other: Union[List[Any], ColumnOrName]) -> Column ``` Check if the column is in a list of values or a column expression. Parameters: - **`other`** (`Union[List[Any], ColumnOrName]`) – A list of values or a Column expression Returns: - **`Column`** ( `Column` ) – A Column expression representing whether each element of Column is in the list Check if name is in a list of values ``` # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Check if value is in another column ``` # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` Source code in `src/fenic/api/column.py` ``` def is_in(self, other: Union[List[Any], ColumnOrName]) -> Column: """Check if the column is in a list of values or a column expression. Args: other (Union[List[Any], ColumnOrName]): A list of values or a Column expression Returns: Column: A Column expression representing whether each element of Column is in the list Example: Check if name is in a list of values ```python # Filter rows where name is in a list of values df.filter(col("name").is_in(["Alice", "Bob"])) ``` Example: Check if value is in another column ```python # Filter rows where name is in another column df.filter(col("name").is_in(col("other_column"))) ``` """ if isinstance(other, list): try: type_ = infer_dtype_from_pyobj(other) return Column._from_logical_expr(InExpr(self._logical_expr, LiteralExpr(other, type_))) except TypeInferenceError as e: raise ValidationError(f"Cannot apply IN on {other}. List argument to IN must be be a valid Python List literal.") from e else: return Column._from_logical_expr(InExpr(self._logical_expr, other._logical_expr)) ``` ### is_not_null ``` is_not_null() -> Column ``` Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is not NULL Filter rows where a column is not NULL ``` df.filter(col("some_column").is_not_null()) ``` Use in a complex condition ``` df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` Source code in `src/fenic/api/column.py` ``` def is_not_null(self) -> Column: """Check if the column contains non-NULL values. This method creates an expression that evaluates to TRUE when the column value is not NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is not NULL Example: Filter rows where a column is not NULL ```python df.filter(col("some_column").is_not_null()) ``` Example: Use in a complex condition ```python df.filter(col("col1").is_not_null() & (col("col2") <= 50)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, False)) ``` ### is_null ``` is_null() -> Column ``` Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: - **`Column`** ( `Column` ) – A Column representing a boolean expression that is TRUE when this column is NULL Filter rows where a column is NULL ``` # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Use in a complex condition ``` # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` Source code in `src/fenic/api/column.py` ``` def is_null(self) -> Column: """Check if the column contains NULL values. This method creates an expression that evaluates to TRUE when the column value is NULL. Returns: Column: A Column representing a boolean expression that is TRUE when this column is NULL Example: Filter rows where a column is NULL ```python # Filter rows where some_column is NULL df.filter(col("some_column").is_null()) ``` Example: Use in a complex condition ```python # Filter rows where col1 is NULL or col2 is greater than 100 df.filter(col("col1").is_null() | (col("col2") > 100)) ``` """ return Column._from_logical_expr(IsNullExpr(self._logical_expr, True)) ``` ### like ``` like(other: Union[str, Column]) -> Column ``` Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Parameters: - **`other`** (`str`) – The SQL LIKE pattern to match against Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where name starts with "J" and ends with "n" ``` # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Find rows where code matches specific pattern ``` # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` Source code in `src/fenic/api/column.py` ``` def like(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a SQL LIKE pattern. This method creates a boolean expression that checks if each value in the column matches the specified SQL LIKE pattern. The pattern can be a string or a a column expression that resolves to a string. SQL LIKE pattern syntax: - % matches any sequence of characters - _ matches any single character Args: other (str): The SQL LIKE pattern to match against Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where name starts with "J" and ends with "n" ```python # Filter rows where name matches the pattern "J%n" df.filter(col("name").like("J%n")) ``` Example: Find rows where code matches specific pattern ```python # Filter rows where code matches the pattern "A_B%" df.filter(col("code").like("A_B%")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(LikeExpr(self._logical_expr, other_expr)) ``` ### otherwise ``` otherwise(value: Column) -> Column ``` Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Parameters: - **`value`** (`Column`) – Value to return when no prior conditions are True Returns: - **`Column`** ( `Column` ) – The complete conditional expression chain Raises: - `ValidationError` – If called on a non-when expression Example ``` # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain Source code in `src/fenic/api/column.py` ``` def otherwise(self, value: Column) -> Column: """Returns a value when no prior conditions are True. This is the final part of a when-chain, like the 'else' in an if-elif-else. All branches must return the same type. Args: value: Value to return when no prior conditions are True Returns: Column: The complete conditional expression chain Raises: ValidationError: If called on a non-when expression Example: ```python # Create age-based categories df = session.createDataFrame({"age": [8, 25, 67]}) result = df.select( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) .otherwise(fc.lit("senior")) .alias("category") ) result.show() # +--------+ # |category| # +--------+ # | minor| # | adult| # | senior| # +--------+ ``` Note: - If otherwise() is not called, unmatched rows return null - Can only be called once per when-chain - Typically the last method in the chain """ return Column._from_logical_expr(OtherwiseExpr(self._logical_expr, value._logical_expr)) ``` ### rlike ``` rlike(other: Union[str, Column]) -> Column ``` Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Parameters: - **`other`** (`Union[str, Column]`) – The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value matches the pattern Find rows where phone number matches pattern ``` # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Find rows where text contains word boundaries ``` # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` Source code in `src/fenic/api/column.py` ``` def rlike(self, other: Union[str, Column]) -> Column: r"""Check if the column matches a regular expression pattern. This method creates a boolean expression that checks if each value in the column matches the specified regular expression pattern. Args: other (Union[str, Column]): The regular expression pattern to match against. Can be a string or a a column expression that resolves to a string. Returns: Column: A boolean column indicating whether each value matches the pattern Example: Find rows where phone number matches pattern ```python # Filter rows where phone number matches a specific pattern df.filter(col("phone").rlike(r"^\d{3}-\d{3}-\d{4}$")) ``` Example: Find rows where text contains word boundaries ```python # Filter rows where text contains a word with boundaries df.filter(col("text").rlike(r"\bhello\b")) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(RLikeExpr(self._logical_expr, other_expr)) ``` ### starts_with ``` starts_with(other: Union[str, Column]) -> Column ``` Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Parameters: - **`other`** (`Union[str, Column]`) – The substring to check for at the start (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A boolean column indicating whether each value starts with the substring Find rows where name starts with "Mr" ``` # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Find rows where text starts with a dynamic pattern ``` # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: - `ValueError` – If the substring starts with a regular expression anchor (^) Source code in `src/fenic/api/column.py` ``` def starts_with(self, other: Union[str, Column]) -> Column: """Check if the column starts with a substring. This method creates a boolean expression that checks if each value in the column starts with the specified substring. The substring can be either a literal string or a column expression. Args: other (Union[str, Column]): The substring to check for at the start (can be a string or column expression) Returns: Column: A boolean column indicating whether each value starts with the substring Example: Find rows where name starts with "Mr" ```python # Filter rows where name starts with "Mr" df.filter(col("name").starts_with("Mr")) ``` Example: Find rows where text starts with a dynamic pattern ```python # Filter rows where text starts with a value from another column df.filter(col("text").starts_with(col("prefix"))) ``` Raises: ValueError: If the substring starts with a regular expression anchor (^) """ if isinstance(other, str): if other.startswith("^"): raise ValidationError("substr should not start with a regular expression anchor") other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(StartsWithExpr(self._logical_expr, other_expr)) ``` ### when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A new when expression with this condition added to the chain Raises: - `ValidationError` – If called on a non-when expression (e.g., regular columns) Example ``` # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. Source code in `src/fenic/api/column.py` ``` def when(self, condition: Column, value: Column) -> Column: """Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A new when expression with this condition added to the chain Raises: ValidationError: If called on a non-when expression (e.g., regular columns) Example: ```python # Build a multi-condition expression: result = ( fc.when(col("age") < 18, fc.lit("minor")) .when(col("age") < 65, fc.lit("adult")) # Add another condition .when(col("age") >= 65, fc.lit("senior")) .otherwise(fc.lit("unknown")) ) # This evaluates like: # if age < 18: return "minor" # elif age < 65: return "adult" # elif age >= 65: return "senior" # else: return "unknown" # ERROR - cannot call .when() on non-when expressions: # col("age").when(...) # ValidationError! ``` Note: Conditions are evaluated in order. The first True condition wins. """ return Column._from_logical_expr(WhenExpr(self._logical_expr, condition._logical_expr, value._logical_expr)) ``` --- # fenic.api.dataframe Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/dataframe/ DataFrame API for Fenic - provides DataFrame and grouped data operations. Classes: - **`DataFrame`** – A data collection organized into named columns. - **`GroupedData`** – Methods for aggregations on a grouped DataFrame. - **`SemanticExtensions`** – A namespace for semantic dataframe operators. ## DataFrame A data collection organized into named columns. The DataFrame class represents a lazily evaluated computation on data. Operations on DataFrame build up a logical query plan that is only executed when an action like show(), to_polars(), to_pandas(), to_arrow(), to_pydict(), to_pylist(), or count() is called. The DataFrame supports method chaining for building complex transformations. Create and transform a DataFrame ``` # Create a DataFrame from a dictionary df = session.create_dataframe({"id": [1, 2, 3], "value": ["a", "b", "c"]}) # Chain transformations result = df.filter(col("id") > 1).select("id", "value") # Show results result.show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 2| b| # | 3| c| # +---+-----+ ``` Methods: - **`agg`** – Aggregate on the entire DataFrame without groups. - **`cache`** – Alias for persist(). Mark DataFrame for caching after first computation. - **`collect`** – Execute the DataFrame computation and return the result as a QueryResult. - **`count`** – Count the number of rows in the DataFrame. - **`distinct`** – Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). - **`drop`** – Remove one or more columns from this DataFrame. - **`drop_duplicates`** – Return a DataFrame with duplicate rows removed. - **`explain`** – Display the logical plan of the DataFrame. - **`explode`** – Create a new row for each element in an array column. - **`explode_outer`** – Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. - **`explode_with_index`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`filter`** – Filters rows using the given condition. - **`group_by`** – Groups the DataFrame using the specified columns. - **`join`** – Joins this DataFrame with another DataFrame. - **`limit`** – Limits the number of rows to the specified number. - **`lineage`** – Create a Lineage object to trace data through transformations. - **`order_by`** – Sort the DataFrame by the specified columns. Alias for sort(). - **`persist`** – Mark this DataFrame to be persisted after first computation. - **`posexplode`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`posexplode_outer`** – Create a new row for each element in an array column with position and value, preserving null/empty arrays. - **`select`** – Projects a set of Column expressions or column names. - **`show`** – Display the DataFrame content in a tabular form. - **`sort`** – Sort the DataFrame by the specified columns. - **`to_arrow`** – Execute the DataFrame computation and return an Apache Arrow Table. - **`to_pandas`** – Execute the DataFrame computation and return a Pandas DataFrame. - **`to_polars`** – Execute the DataFrame computation and return the result as a Polars DataFrame. - **`to_pydict`** – Execute the DataFrame computation and return a dictionary of column arrays. - **`to_pylist`** – Execute the DataFrame computation and return a list of row dictionaries. - **`union`** – Return a new DataFrame containing the union of rows in this and another DataFrame. - **`unnest`** – Unnest the specified struct columns into separate columns. - **`where`** – Filters rows using the given condition (alias for filter()). - **`with_column`** – Add a new column or replace an existing column. - **`with_column_renamed`** – Rename a column. No-op if the column does not exist. - **`with_columns`** – Add multiple new columns or replace existing columns. Attributes: - **`columns`** (`List[str]`) – Get list of column names. - **`schema`** (`Schema`) – Get the schema of this DataFrame. - **`semantic`** (`SemanticExtensions`) – Interface for semantic operations on the DataFrame. - **`write`** (`DataFrameWriter`) – Interface for saving the content of the DataFrame. ### columns ``` columns: List[str] ``` Get list of column names. Returns: - `List[str]` – List[str]: List of all column names in the DataFrame Examples: ``` >>> df.columns ['name', 'age', 'city'] ``` ### schema ``` schema: Schema ``` Get the schema of this DataFrame. Returns: - **`Schema`** ( `Schema` ) – Schema containing field names and data types Examples: ``` >>> df.schema Schema([ ColumnField('name', StringType), ColumnField('age', IntegerType) ]) ``` ### semantic ``` semantic: SemanticExtensions ``` Interface for semantic operations on the DataFrame. ### write ``` write: DataFrameWriter ``` Interface for saving the content of the DataFrame. Returns: - **`DataFrameWriter`** ( `DataFrameWriter` ) – Writer interface to write DataFrame. ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions or dictionary of aggregations. Returns: - **`DataFrame`** ( `DataFrame` ) – Aggregation results. Multiple aggregations ``` # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Dictionary style ``` # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Args: *exprs: Aggregation expressions or dictionary of aggregations. Returns: DataFrame: Aggregation results. Example: Multiple aggregations ```python # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Example: Dictionary style ```python # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` """ return self.group_by().agg(*exprs) ``` ### cache ``` cache() -> DataFrame ``` Alias for persist(). Mark DataFrame for caching after first computation. Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for caching See Also persist(): Full documentation of caching behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def cache(self) -> DataFrame: """Alias for persist(). Mark DataFrame for caching after first computation. Returns: DataFrame: Same DataFrame, but marked for caching See Also: persist(): Full documentation of caching behavior """ return self.persist() ``` ### collect ``` collect(data_type: DataLikeType = 'polars') -> QueryResult ``` Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Parameters: - **`data_type`** (`DataLikeType`, default: `'polars'` ) – The type of data to return Returns: - **`QueryResult`** ( `QueryResult` ) – A QueryResult with materialized data and query metrics Source code in `src/fenic/api/dataframe/dataframe.py` ``` def collect(self, data_type: DataLikeType = "polars") -> QueryResult: """Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Args: data_type: The type of data to return Returns: QueryResult: A QueryResult with materialized data and query metrics """ result: Tuple[pl.DataFrame, QueryMetrics] = self._session_state.execution.collect(self._logical_plan) df, metrics = result logger.info(metrics.get_summary()) if data_type == "polars": return QueryResult(df, metrics) elif data_type == "pandas": return QueryResult(df.to_pandas(use_pyarrow_extension_array=True), metrics) elif data_type == "arrow": return QueryResult(df.to_arrow(), metrics) elif data_type == "pydict": return QueryResult(df.to_dict(as_series=False), metrics) elif data_type == "pylist": return QueryResult(df.to_dicts(), metrics) else: raise ValidationError(f"Invalid data type: {data_type} in collect(). Valid data types are: polars, pandas, arrow, pydict, pylist") ``` ### count ``` count() -> int ``` Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: - **`int`** ( `int` ) – The number of rows in the DataFrame Source code in `src/fenic/api/dataframe/dataframe.py` ``` def count(self) -> int: """Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: int: The number of rows in the DataFrame """ return self._session_state.execution.count(self._logical_plan)[0] ``` ### distinct ``` distinct() -> DataFrame ``` Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Example ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def distinct(self) -> DataFrame: """Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: DataFrame: A new DataFrame with duplicate rows removed. Example: ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ return self.drop_duplicates() ``` ### drop ``` drop(*col_names: str) -> DataFrame ``` Remove one or more columns from this DataFrame. Parameters: - **`*col_names`** (`str`, default: `()` ) – Names of columns to drop. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame without specified columns. Raises: - `ValueError` – If any specified column doesn't exist in the DataFrame. - `ValueError` – If dropping the columns would result in an empty DataFrame. Drop single column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Drop multiple columns ``` # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Error when dropping non-existent column ``` # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop(self, *col_names: str) -> DataFrame: """Remove one or more columns from this DataFrame. Args: *col_names: Names of columns to drop. Returns: DataFrame: New DataFrame without specified columns. Raises: ValueError: If any specified column doesn't exist in the DataFrame. ValueError: If dropping the columns would result in an empty DataFrame. Example: Drop single column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Drop multiple columns ```python # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Example: Error when dropping non-existent column ```python # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` """ if not col_names: return self current_cols = set(self.columns) to_drop = set(col_names) missing = to_drop - current_cols if missing: missing_str = ( f"Column '{next(iter(missing))}'" if len(missing) == 1 else f"Columns {sorted(missing)}" ) raise ValueError(f"{missing_str} not found in DataFrame") remaining_cols = [ col(c)._logical_expr for c in self.columns if c not in to_drop ] if not remaining_cols: raise ValueError("Cannot drop all columns from DataFrame") return self._from_logical_plan( Projection.from_session_state(self._logical_plan, remaining_cols, self._session_state), self._session_state, ) ``` ### drop_duplicates ``` drop_duplicates(subset: Optional[List[str]] = None) -> DataFrame ``` Return a DataFrame with duplicate rows removed. Parameters: - **`subset`** (`Optional[List[str]]`, default: `None` ) – Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Raises: - `ValueError` – If a specified column is not present in the current DataFrame schema. Remove duplicates considering specific columns ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop_duplicates( self, subset: Optional[List[str]] = None, ) -> DataFrame: """Return a DataFrame with duplicate rows removed. Args: subset: Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: DataFrame: A new DataFrame with duplicate rows removed. Raises: ValueError: If a specified column is not present in the current DataFrame schema. Example: Remove duplicates considering specific columns ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ exprs = [] if subset: for c in subset: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( DropDuplicates.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### explain ``` explain() -> None ``` Display the logical plan of the DataFrame. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explain(self) -> None: """Display the logical plan of the DataFrame.""" print(str(self._logical_plan)) ``` ### explode ``` explode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. Raises: - `TypeError` – If column argument is not a string or Column. Explode array column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Using column expression ``` # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Raises: TypeError: If column argument is not a string or Column. Example: Explode array column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Example: Using column expression ```python # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state(self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state), self._session_state, ) ``` ### explode_outer ``` explode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. - `DataFrame` – Rows with null or empty arrays are preserved with null in the exploded column. Explode with outer join behavior ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Rows with null or empty arrays are preserved with null in the exploded column. Example: Explode with outer join behavior ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state, keep_null_and_empty=True ), self._session_state, ) ``` ### explode_with_index ``` explode_with_index(column: ColumnOrName, index_col_name: str = 'pos', value_col_name: str = 'col', keep_null_and_empty: bool = False) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. - **`index_col_name`** (`str`, default: `'pos'` ) – Name for the column containing 0-based array positions (default: "pos"). - **`value_col_name`** (`str`, default: `'col'` ) – Name for the exploded value column (default: "col"). - **`keep_null_and_empty`** (`bool`, default: `False` ) – If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Explode with index ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Custom column names ``` df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_with_index( self, column: ColumnOrName, index_col_name: str = "pos", value_col_name: str = "col", keep_null_and_empty: bool = False, ) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Args: column: Name of array column to explode (as string) or Column expression. index_col_name: Name for the column containing 0-based array positions (default: "pos"). value_col_name: Name for the exploded value column (default: "col"). keep_null_and_empty: If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: DataFrame: New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Example: Explode with index ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Example: Custom column names ```python df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` """ return self._from_logical_plan( ExplodeWithIndex.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, index_col_name, value_col_name, self._session_state, keep_null_and_empty, ), self._session_state, ) ``` ### filter ``` filter(condition: Column) -> DataFrame ``` Filters rows using the given condition. Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame Filter with numeric comparison ``` # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with semantic predicate ``` # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with multiple conditions ``` # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def filter(self, condition: Column) -> DataFrame: """Filters rows using the given condition. Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame Example: Filter with numeric comparison ```python # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with semantic predicate ```python # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with multiple conditions ```python # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` """ return self._from_logical_plan( Filter.from_session_state(self._logical_plan, condition._logical_expr, self._session_state), self._session_state, ) ``` ### group_by ``` group_by(*cols: ColumnOrName) -> GroupedData ``` Groups the DataFrame using the specified columns. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns to group by. Can be column names as strings or Column expressions. Returns: - **`GroupedData`** ( `GroupedData` ) – Object for performing aggregations on the grouped data. Group by single column ``` # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Group by multiple columns ``` # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Group by expression ``` # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def group_by(self, *cols: ColumnOrName) -> GroupedData: """Groups the DataFrame using the specified columns. Args: *cols: Columns to group by. Can be column names as strings or Column expressions. Returns: GroupedData: Object for performing aggregations on the grouped data. Example: Group by single column ```python # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Example: Group by multiple columns ```python # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Example: Group by expression ```python # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` """ return GroupedData(self, list(cols) if cols else None) ``` ### join ``` join(other: DataFrame, on: Union[str, List[str]], *, how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, *, left_on: Union[ColumnOrName, List[ColumnOrName]], right_on: Union[ColumnOrName, List[ColumnOrName]], how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = 'inner') -> DataFrame ``` Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Parameters: - **`other`** (`DataFrame`) – DataFrame to join with. - **`on`** (`Optional[Union[str, List[str]]]`, default: `None` ) – Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins - **`left_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions - **`right_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions - **`how`** (`JoinType`, default: `'inner'` ) – Type of join to perform. Returns: - `DataFrame` – Joined DataFrame. Raises: - `ValidationError` – If cross join is used with an ON clause. - `ValidationError` – If join condition is invalid. - `ValidationError` – If both 'on' and 'left_on'/'right_on' parameters are provided. - `ValidationError` – If only one of 'left_on' or 'right_on' is provided. - `ValidationError` – If 'left_on' and 'right_on' have different lengths Inner join on column name ``` # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Join with expression ``` # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Cross join ``` # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def join( self, other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = "inner", ) -> DataFrame: """Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Args: other: DataFrame to join with. on: Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins left_on: Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions right_on: Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions how: Type of join to perform. Returns: Joined DataFrame. Raises: ValidationError: If cross join is used with an ON clause. ValidationError: If join condition is invalid. ValidationError: If both 'on' and 'left_on'/'right_on' parameters are provided. ValidationError: If only one of 'left_on' or 'right_on' is provided. ValidationError: If 'left_on' and 'right_on' have different lengths Example: Inner join on column name ```python # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Join with expression ```python # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Cross join ```python # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` """ validate_join_parameters(self, on, left_on, right_on, how) # Build join conditions left_conditions, right_conditions = build_join_conditions(on, left_on, right_on) self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( Join.from_session_state( self._logical_plan, other._logical_plan, left_conditions, right_conditions, how, self._session_state), self._session_state, ) ``` ### limit ``` limit(n: int) -> DataFrame ``` Limits the number of rows to the specified number. Parameters: - **`n`** (`int`) – Maximum number of rows to return. Returns: - **`DataFrame`** ( `DataFrame` ) – DataFrame with at most n rows. Raises: - `TypeError` – If n is not an integer. Limit rows ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Limit with other operations ``` # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def limit(self, n: int) -> DataFrame: """Limits the number of rows to the specified number. Args: n: Maximum number of rows to return. Returns: DataFrame: DataFrame with at most n rows. Raises: TypeError: If n is not an integer. Example: Limit rows ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Limit with other operations ```python # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` """ return self._from_logical_plan( Limit.from_session_state(self._logical_plan, n, self._session_state), self._session_state) ``` ### lineage ``` lineage() -> Lineage ``` Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: - **`Lineage`** ( `Lineage` ) – Interface for querying data lineage Example ``` # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also LineageQuery: Full documentation of lineage querying capabilities Source code in `src/fenic/api/dataframe/dataframe.py` ``` def lineage(self) -> Lineage: """Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: Lineage: Interface for querying data lineage Example: ```python # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also: LineageQuery: Full documentation of lineage querying capabilities """ return Lineage(self._session_state.execution.build_lineage(self._logical_plan)) ``` ### order_by ``` order_by(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Alias for sort(). Returns: - **`DataFrame`** ( `DataFrame` ) – sorted Dataframe. See Also sort(): Full documentation of sorting behavior and parameters. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def order_by( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Alias for sort(). Returns: DataFrame: sorted Dataframe. See Also: sort(): Full documentation of sorting behavior and parameters. """ return self.sort(cols, ascending) ``` ### persist ``` persist() -> DataFrame ``` Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for persistence Example ``` # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def persist(self) -> DataFrame: """Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: DataFrame: Same DataFrame, but marked for persistence Example: ```python # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` """ cache_info = CacheInfo(cache_key=f"cache_{uuid.uuid4().hex}") self._logical_plan.set_cache_info(cache_info) return self._from_logical_plan( self._logical_plan, self._session_state) ``` ### posexplode ``` posexplode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. PySpark-style posexplode ``` df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Example: PySpark-style posexplode ```python df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` """ return self.explode_with_index(column) ``` ### posexplode_outer ``` posexplode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. - `DataFrame` – Rows with null or empty arrays are preserved with (null, null). PySpark-style posexplode_outer ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Rows with null or empty arrays are preserved with (null, null). Example: PySpark-style posexplode_outer ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` """ return self.explode_with_index( column, keep_null_and_empty=True ) ``` ### select ``` select(*cols: ColumnOrName) -> DataFrame ``` Projects a set of Column expressions or column names. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with selected columns Select by column names ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Select with expressions ``` # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Mix strings and expressions ``` # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def select(self, *cols: ColumnOrName) -> DataFrame: """Projects a set of Column expressions or column names. Args: *cols: Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: DataFrame: A new DataFrame with selected columns Example: Select by column names ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Example: Select with expressions ```python # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Example: Mix strings and expressions ```python # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` """ exprs = [] if not cols: return self for c in cols: if isinstance(c, str): if c == "*": exprs.extend(col(field)._logical_expr for field in self.columns) else: exprs.append(col(c)._logical_expr) else: exprs.append(c._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### show ``` show(n: int = 10, explain_analyze: bool = False) -> None ``` Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Parameters: - **`n`** (`int`, default: `10` ) – Number of rows to display - **`explain_analyze`** (`bool`, default: `False` ) – Whether to print the explain analyze plan Source code in `src/fenic/api/dataframe/dataframe.py` ``` def show(self, n: int = 10, explain_analyze: bool = False) -> None: """Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Args: n: Number of rows to display explain_analyze: Whether to print the explain analyze plan """ output, metrics = self._session_state.execution.show(self._logical_plan, n) logger.info(metrics.get_summary()) print(output) if explain_analyze: print(metrics.get_execution_plan_details()) ``` ### sort ``` sort(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Parameters: - **`cols`** (`Union[ColumnOrName, List[ColumnOrName], None]`, default: `None` ) – Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. - **`ascending`** (`Optional[Union[bool, List[bool]]]`, default: `None` ) – A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame sorted by the specified columns. Raises: - `ValueError` – - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used - `TypeError` – - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Sort in ascending order ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Sort in descending order ``` # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Sort with boolean ascending parameter ``` # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Multiple columns with different sort orders ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Multiple columns with list of ascending strategies ``` # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def sort( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Args: cols: Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. ascending: A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: DataFrame: A new DataFrame sorted by the specified columns. Raises: ValueError: - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used TypeError: - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Example: Sort in ascending order ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Example: Sort in descending order ```python # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Sort with boolean ascending parameter ```python # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Multiple columns with different sort orders ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Example: Multiple columns with list of ascending strategies ```python # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` """ col_args = cols if cols is None: return self._from_logical_plan( Sort.from_session_state(self._logical_plan, [], self._session_state), self._session_state, ) elif not isinstance(cols, List): col_args = [cols] # parse the ascending arguments bool_ascending = [] using_default_ascending = False if ascending is None: using_default_ascending = True bool_ascending = [True] * len(col_args) elif isinstance(ascending, bool): bool_ascending = [ascending] * len(col_args) elif isinstance(ascending, List): bool_ascending = ascending if len(bool_ascending) != len(cols): raise ValueError( f"the list length of ascending sort strategies must match the specified sort columns" f"Got {len(cols)} column expressions and {len(bool_ascending)} ascending strategies. " ) else: raise TypeError( f"Invalid ascending strategy type: {type(ascending)}. Must be a boolean or list of booleans." ) # create our list of sort expressions, for each column expression # that isn't already provided as a asc()/desc() SortExpr sort_exprs = [] for c, asc_bool in zip(col_args, bool_ascending, strict=True): if isinstance(c, ColumnOrName): c_expr = Column._from_col_or_name(c)._logical_expr else: raise TypeError( f"Invalid column type: {type(c).__name__}. Must be a string or Column Expression." ) if not isinstance(asc_bool, bool): raise TypeError( f"Invalid ascending strategy type: {type(asc_bool).__name__}. Must be a boolean." ) if isinstance(c_expr, SortExpr): if not using_default_ascending: raise TypeError( "Cannot specify both asc()/desc() expressions and boolean ascending strategies." f"Got expression: {c_expr} and ascending argument: {bool_ascending}" ) sort_exprs.append(c_expr) else: sort_exprs.append(SortExpr(c_expr, ascending=asc_bool)) return self._from_logical_plan( Sort.from_session_state(self._logical_plan, sort_exprs, self._session_state), self._session_state, ) ``` ### to_arrow ``` to_arrow() -> pa.Table ``` Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: - `Table` – pa.Table: An Apache Arrow Table containing the computed results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_arrow(self) -> pa.Table: """Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: pa.Table: An Apache Arrow Table containing the computed results """ return self.collect("arrow").data ``` ### to_pandas ``` to_pandas() -> pd.DataFrame ``` Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: - `DataFrame` – pd.DataFrame: A Pandas DataFrame containing the computed results with Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pandas(self) -> pd.DataFrame: """Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: pd.DataFrame: A Pandas DataFrame containing the computed results with """ return self.collect("pandas").data ``` ### to_polars ``` to_polars() -> pl.DataFrame ``` Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: - `DataFrame` – pl.DataFrame: A Polars DataFrame with materialized results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_polars(self) -> pl.DataFrame: """Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: pl.DataFrame: A Polars DataFrame with materialized results """ return self.collect("polars").data ``` ### to_pydict ``` to_pydict() -> Dict[str, List[Any]] ``` Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: - `Dict[str, List[Any]]` – Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pydict(self) -> Dict[str, List[Any]]: """Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column """ return self.collect("pydict").data ``` ### to_pylist ``` to_pylist() -> List[Dict[str, Any]] ``` Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: - `List[Dict[str, Any]]` – List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pylist(self) -> List[Dict[str, Any]]: """Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result """ return self.collect("pylist").data ``` ### union ``` union(other: DataFrame) -> DataFrame ``` Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Parameters: - **`other`** (`DataFrame`) – Another DataFrame with the same schema. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing rows from both DataFrames. Raises: - `ValueError` – If the DataFrames have different schemas. - `TypeError` – If other is not a DataFrame. Union two DataFrames ``` # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Union with duplicates ``` # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def union(self, other: DataFrame) -> DataFrame: """Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Args: other: Another DataFrame with the same schema. Returns: DataFrame: A new DataFrame containing rows from both DataFrames. Raises: ValueError: If the DataFrames have different schemas. TypeError: If other is not a DataFrame. Example: Union two DataFrames ```python # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Example: Union with duplicates ```python # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` """ self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( UnionLogicalPlan.from_session_state([self._logical_plan, other._logical_plan], self._session_state), self._session_state, ) ``` ### unnest ``` unnest(*col_names: str) -> DataFrame ``` Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Parameters: - **`*col_names`** (`str`, default: `()` ) – One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with the specified struct columns expanded. Raises: - `TypeError` – If any argument is not a string or Column. - `ValueError` – If a specified column does not contain struct data. Unnest struct column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Unnest multiple struct columns ``` # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def unnest(self, *col_names: str) -> DataFrame: """Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Args: *col_names: One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: DataFrame: A new DataFrame with the specified struct columns expanded. Raises: TypeError: If any argument is not a string or Column. ValueError: If a specified column does not contain struct data. Example: Unnest struct column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Example: Unnest multiple struct columns ```python # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` """ if not col_names: return self exprs = [] for c in col_names: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( Unnest.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### where ``` where(condition: Column) -> DataFrame ``` Filters rows using the given condition (alias for filter()). Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame See Also filter(): Full documentation of filtering behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def where(self, condition: Column) -> DataFrame: """Filters rows using the given condition (alias for filter()). Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame See Also: filter(): Full documentation of filtering behavior """ return self.filter(condition) ``` ### with_column ``` with_column(col_name: str, col: Union[Any, Column, Series, Series]) -> DataFrame ``` Add a new column or replace an existing column. Parameters: - **`col_name`** (`str`) – Name of the new column - **`col`** (`Union[Any, Column, Series, Series]`) – Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced column Raises: - `ExecutionError` – - If a Series length does not match the DataFrame height - `ValidationError` – - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Add literal column ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Add computed column ``` # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Replace existing column ``` # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Add column with complex expression ``` # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Add column from Polars Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Add column from pandas Series ``` import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column(self, col_name: str, col: Union[Any, Column, pl.Series, pd.Series]) -> DataFrame: """Add a new column or replace an existing column. Args: col_name: Name of the new column col: Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced column Raises: ExecutionError: - If a Series length does not match the DataFrame height ValidationError: - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Example: Add literal column ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Example: Add computed column ```python # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Example: Replace existing column ```python # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Example: Add column with complex expression ```python # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Example: Add column from Polars Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Example: Add column from pandas Series ```python import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` """ exprs = [] # Handle different input types: Column, Series, or literal value if isinstance(col, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col = Column._from_logical_expr(SeriesLiteralExpr(col)) elif not isinstance(col, Column): # Wrap other values as literals col = lit(col) for field in self.columns: if field != col_name: exprs.append(Column._from_column_name(field)._logical_expr) # Add the new column with alias exprs.append(col.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_column_renamed ``` with_column_renamed(col_name: str, new_col_name: str) -> DataFrame ``` Rename a column. No-op if the column does not exist. Parameters: - **`col_name`** (`str`) – Name of the column to rename. - **`new_col_name`** (`str`) – New name for the column. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the column renamed. Rename a column ``` # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Rename multiple columns ``` # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column_renamed(self, col_name: str, new_col_name: str) -> DataFrame: """Rename a column. No-op if the column does not exist. Args: col_name: Name of the column to rename. new_col_name: New name for the column. Returns: DataFrame: New DataFrame with the column renamed. Example: Rename a column ```python # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Example: Rename multiple columns ```python # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` """ exprs = [] renamed = False for field in self.schema.column_fields: name = field.name if name == col_name: exprs.append(col(name).alias(new_col_name)._logical_expr) renamed = True else: exprs.append(col(name)._logical_expr) if not renamed: return self return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_columns ``` with_columns(cols_map: Dict[str, Union[Any, Column, Series, Series]]) -> DataFrame ``` Add multiple new columns or replace existing columns. Parameters: - **`cols_map`** (`Dict[str, Union[Any, Column, Series, Series]]`) – A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced columns Raises: - `ValueError` – - If two columns being created in the same `with_columns` call depend on each other - `ExecutionError` – - If any Series length does not match the DataFrame height - `ValidationError` – - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Add multiple columns ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Replace and add columns ``` # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Complex expressions ``` # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Add columns from Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Mix Series with Column expressions ``` import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Error when adding columns that depend on each other ``` df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_columns(self, cols_map: Dict[str, Union[Any, Column, pl.Series, pd.Series]]) -> DataFrame: """Add multiple new columns or replace existing columns. Args: cols_map: A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced columns Raises: ValueError: - If two columns being created in the same `with_columns` call depend on each other ExecutionError: - If any Series length does not match the DataFrame height ValidationError: - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Example: Add multiple columns ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Example: Replace and add columns ```python # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Example: Complex expressions ```python # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Example: Add columns from Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Example: Mix Series with Column expressions ```python import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Example: Error when adding columns that depend on each other ```python df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` """ if not cols_map: return self exprs = [] new_col_names = set(cols_map.keys()) # Add existing columns that are not being replaced for field in self.columns: if field not in new_col_names: exprs.append(Column._from_column_name(field)._logical_expr) # Add all new columns with aliases for col_name, col_expr in cols_map.items(): # Handle different input types: Column, Series, or literal value if isinstance(col_expr, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col_expr = Column._from_logical_expr(SeriesLiteralExpr(col_expr)) elif not isinstance(col_expr, Column): # Automatically wrap non-Column values (literals) with lit() for convenience # This allows users to pass raw Python values like: {"constant": 100, "status": "active"} # instead of requiring: {"constant": lit(100), "status": lit("active")} col_expr = lit(col_expr) exprs.append(col_expr.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ## GroupedData ``` GroupedData(df: DataFrame, by: Optional[List[ColumnOrName]] = None) ``` Bases: `BaseGroupedData` ``` flowchart TD fenic.api.dataframe.GroupedData[GroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData[BaseGroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData --> fenic.api.dataframe.GroupedData click fenic.api.dataframe.GroupedData href "" "fenic.api.dataframe.GroupedData" click fenic.api.dataframe._base_grouped_data.BaseGroupedData href "" "fenic.api.dataframe._base_grouped_data.BaseGroupedData" ``` Methods for aggregations on a grouped DataFrame. Initialize grouped data. Parameters: - **`df`** (`DataFrame`) – The DataFrame to group. - **`by`** (`Optional[List[ColumnOrName]]`, default: `None` ) – Optional list of columns to group by. Methods: - **`agg`** – Compute aggregations on grouped data and return the result as a DataFrame. Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def __init__(self, df: DataFrame, by: Optional[List[ColumnOrName]] = None): """Initialize grouped data. Args: df: The DataFrame to group. by: Optional list of columns to group by. """ super().__init__(df) self._by: List[Column] = [] for c in by or []: if isinstance(c, str): self._by.append(col(c)) elif isinstance(c, Column): # Allow any expression except literals if isinstance(c._logical_expr, LiteralExpr): raise ValueError(f"Cannot group by literal value: {c}") self._by.append(c) else: raise TypeError( f"Group by expressions must be string or Column, got {type(c)}" ) self._by_exprs = [c._logical_expr for c in self._by] ``` ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with one row per group and columns for group keys and aggregated values Raises: - `ValueError` – If arguments are not Column expressions or a dictionary - `ValueError` – If dictionary values are not valid aggregate function names Count employees by department ``` # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Multiple aggregations ``` # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Dictionary style aggregations ``` # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Args: *exprs: Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: DataFrame: A new DataFrame with one row per group and columns for group keys and aggregated values Raises: ValueError: If arguments are not Column expressions or a dictionary ValueError: If dictionary values are not valid aggregate function names Example: Count employees by department ```python # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Example: Multiple aggregations ```python # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Example: Dictionary style aggregations ```python # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` """ self._validate_agg_exprs(*exprs) if len(exprs) == 1 and isinstance(exprs[0], dict): agg_dict = exprs[0] return self.agg(*self._process_agg_dict(agg_dict)) agg_exprs = self._process_agg_exprs(exprs) return self._df._from_logical_plan( Aggregate.from_session_state(self._df._logical_plan, self._by_exprs, agg_exprs, self._df._session_state), self._df._session_state, ) ``` ## SemanticExtensions ``` SemanticExtensions(df: DataFrame) ``` A namespace for semantic dataframe operators. Initialize semantic extensions. Parameters: - **`df`** (`DataFrame`) – The DataFrame to extend with semantic operations. Methods: - **`join`** – Performs a semantic join between two DataFrames using a natural language predicate. - **`sim_join`** – Performs a semantic similarity join between two DataFrames using embedding expressions. - **`with_cluster_labels`** – Cluster rows using K-means and add cluster metadata columns. Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def __init__(self, df: DataFrame): """Initialize semantic extensions. Args: df: The DataFrame to extend with semantic operations. """ self._df = df ``` ### join ``` join(other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Parameters: - **`other`** (`DataFrame`) – The DataFrame to join with. - **`predicate`** (`str`) – A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. - **`left_on`** (`Column`) – The column from the left DataFrame (self) to use in the join predicate. - **`right_on`** (`Column`) – The column from the right DataFrame (other) to use in the join predicate. - **`strict`** (`bool`, default: `True` ) – If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[JoinExampleCollection]`, default: `None` ) – Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use. If None, uses the default model. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing matched row pairs with all columns from both DataFrames. Basic semantic join ``` # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Semantic join with examples ``` # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def join( self, other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Args: other: The DataFrame to join with. predicate: A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. left_on: The column from the left DataFrame (self) to use in the join predicate. right_on: The column from the right DataFrame (other) to use in the join predicate. strict: If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. examples: Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) model_alias: Optional alias for the language model to use. If None, uses the default model. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: DataFrame: A new DataFrame containing matched row pairs with all columns from both DataFrames. Example: Basic semantic join ```python # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Example: Semantic join with examples ```python # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(other, DataFrame): raise ValidationError(f"other argument must be a DataFrame, got {type(other)}") if not isinstance(predicate, str): raise ValidationError( f"The `predicate` argument to `semantic.join` must be a string, got {type(predicate)}" ) if not isinstance(left_on, Column): raise ValidationError(f"`left_on` argument must be a Column, got {type(left_on)} instead.") if not isinstance(right_on, Column): raise ValidationError(f"`right_on` argument must be a Column, got {type(right_on)} instead.") if examples is not None and not isinstance(examples, JoinExampleCollection): raise ValidationError(f"`examples` argument must be a JoinExampleCollection, got {type(examples)} instead.") if model_alias is not None and not isinstance(model_alias, (str, ModelAlias)): raise ValidationError(f"`model_alias` argument must be a string or ModelAlias, got {type(model_alias)} instead.") # Validate request_timeout request_timeout = validate_timeout(request_timeout) resolved_model_alias = _resolve_model_alias(model_alias) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticJoin.from_session_state( left=self._df._logical_plan, right=other._logical_plan, left_on=left_on._logical_expr, right_on=right_on._logical_expr, jinja_template=predicate, strict=strict, model_alias=resolved_model_alias, examples=examples, request_timeout=request_timeout, session_state=self._df._session_state, ), self._df._session_state, ) ``` ### sim_join ``` sim_join(other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = 'cosine', similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Parameters: - **`other`** (`DataFrame`) – The right-hand DataFrame to join with. - **`left_on`** (`ColumnOrName`) – Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. - **`right_on`** (`ColumnOrName`) – Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. - **`k`** (`int`, default: `1` ) – Number of most similar matches to return per row. - **`similarity_metric`** (`SemanticSimilarityMetric`, default: `'cosine'` ) – Similarity metric to use: "l2", "cosine", or "dot". - **`similarity_score_column`** (`Optional[str]`, default: `None` ) – If set, adds a column with this name containing similarity scores. If None, the scores are omitted. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. - `DataFrame` – The result includes all columns from both input DataFrames and, when `similarity_score_column` - `DataFrame` – is provided, a similarity score column. Join key columns that already exist in the input - `DataFrame` – DataFrames are preserved as normal user columns. Join keys derived from expressions are - `DataFrame` – temporary execution columns and are not included in the result schema. Raises: - `ValidationError` – If `k` is not positive or if the columns are invalid. - `ValidationError` – If `similarity_metric` is not one of "l2", "cosine", "dot" Match queries to FAQ entries ``` # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Link headlines to articles ``` # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Find similar job postings ``` # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def sim_join( self, other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = "cosine", similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note: Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Args: other: The right-hand DataFrame to join with. left_on: Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. right_on: Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. k: Number of most similar matches to return per row. similarity_metric: Similarity metric to use: "l2", "cosine", or "dot". similarity_score_column: If set, adds a column with this name containing similarity scores. If None, the scores are omitted. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. The result includes all columns from both input DataFrames and, when `similarity_score_column` is provided, a similarity score column. Join key columns that already exist in the input DataFrames are preserved as normal user columns. Join keys derived from expressions are temporary execution columns and are not included in the result schema. Raises: ValidationError: If `k` is not positive or if the columns are invalid. ValidationError: If `similarity_metric` is not one of "l2", "cosine", "dot" Example: Match queries to FAQ entries ```python # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Example: Link headlines to articles ```python # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Example: Find similar job postings ```python # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(right_on, ColumnOrName): raise ValidationError( f"The `right_on` argument must be a `Column` or a string representing a column name, " f"but got `{type(right_on).__name__}` instead." ) if not isinstance(other, DataFrame): raise ValidationError( f"The `other` argument to `sim_join()` must be a DataFrame`, but got `{type(other).__name__}`." ) if not (isinstance(k, int) and k > 0): raise ValidationError( f"The parameter `k` must be a positive integer, but received `{k}`." ) args = get_args(SemanticSimilarityMetric) if similarity_metric not in args: raise ValidationError( f"The `similarity_metric` argument must be one of {args}, but got `{similarity_metric}`." ) def _validate_column(column: ColumnOrName, name: str): if column is None: raise ValidationError(f"The `{name}` argument must not be None.") if not isinstance(column, ColumnOrName): raise ValidationError( f"The `{name}` argument must be a `Column` or a string representing a column name, " f"but got `{type(column).__name__}` instead." ) _validate_column(left_on, "left_on") _validate_column(right_on, "right_on") # Validate request_timeout request_timeout = validate_timeout(request_timeout) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticSimilarityJoin.from_session_state( self._df._logical_plan, other._logical_plan, Column._from_col_or_name(left_on)._logical_expr, Column._from_col_or_name(right_on)._logical_expr, k, similarity_metric=similarity_metric, similarity_score_column=similarity_score_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ### with_cluster_labels ``` with_cluster_labels(by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = 'cluster_label', centroid_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Parameters: - **`by`** (`ColumnOrName`) – Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). - **`num_clusters`** (`int`) – Number of clusters to compute (must be > 0). - **`max_iter`** (`int`, default: `300` ) – Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. - **`num_init`** (`int`, default: `1` ) – Number of independent runs of k-means with different centroid seeds. The best result is selected. - **`label_column`** (`str`, default: `'cluster_label'` ) – Name of the output column for cluster IDs. Default is "cluster_label". - **`centroid_column`** (`Optional[str]`, default: `None` ) – If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame with all original columns plus: - `DataFrame` – - ``: integer cluster assignment (0 to num_clusters - 1) - `DataFrame` – - ``: cluster centroid embedding, if specified Basic clustering ``` # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Filter outliers using centroids ``` # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def with_cluster_labels( self, by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = "cluster_label", centroid_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note: Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Args: by: Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). num_clusters: Number of clusters to compute (must be > 0). max_iter: Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. num_init: Number of independent runs of k-means with different centroid seeds. The best result is selected. label_column: Name of the output column for cluster IDs. Default is "cluster_label". centroid_column: If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame with all original columns plus: - ``: integer cluster assignment (0 to num_clusters - 1) - ``: cluster centroid embedding, if specified Example: Basic clustering ```python # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Example: Filter outliers using centroids ```python # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` """ # Validate request_timeout request_timeout = validate_timeout(request_timeout) # Validate num_clusters if not isinstance(num_clusters, int) or num_clusters <= 0: raise ValidationError("`num_clusters` must be a positive integer.") # Validate max_iter if not isinstance(max_iter, int) or max_iter <= 0: raise ValidationError("`max_iter` must be a positive integer.") # Validate num_init if not isinstance(num_init, int) or num_init <= 0: raise ValidationError("`num_init` must be a positive integer.") # Validate clustering target if not isinstance(by, ColumnOrName): raise ValidationError( f"Invalid cluster by: expected a column name (str) or Column object, got {type(by).__name__}." ) # Validate label_column if not isinstance(label_column, str) or not label_column: raise ValidationError("`label_column` must be a non-empty string.") # Validate centroid_column if provided if centroid_column is not None: if not isinstance(centroid_column, str) or not centroid_column: raise ValidationError("`centroid_column` must be a non-empty string if provided.") # Check that the expression isn't a literal by_expr = Column._from_col_or_name(by)._logical_expr if isinstance(by_expr, LiteralExpr): raise ValidationError( f"Invalid cluster by: Cannot cluster by a literal value: {by_expr}." ) return self._df._from_logical_plan( SemanticCluster.from_session_state( self._df._logical_plan, by_expr, num_clusters=num_clusters, max_iter=max_iter, num_init=num_init, label_column=label_column, centroid_column=centroid_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` --- # fenic.api.dataframe.dataframe Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/dataframe/dataframe/ DataFrame class providing PySpark-inspired API for data manipulation. Classes: - **`DataFrame`** – A data collection organized into named columns. ## DataFrame A data collection organized into named columns. The DataFrame class represents a lazily evaluated computation on data. Operations on DataFrame build up a logical query plan that is only executed when an action like show(), to_polars(), to_pandas(), to_arrow(), to_pydict(), to_pylist(), or count() is called. The DataFrame supports method chaining for building complex transformations. Create and transform a DataFrame ``` # Create a DataFrame from a dictionary df = session.create_dataframe({"id": [1, 2, 3], "value": ["a", "b", "c"]}) # Chain transformations result = df.filter(col("id") > 1).select("id", "value") # Show results result.show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 2| b| # | 3| c| # +---+-----+ ``` Methods: - **`agg`** – Aggregate on the entire DataFrame without groups. - **`cache`** – Alias for persist(). Mark DataFrame for caching after first computation. - **`collect`** – Execute the DataFrame computation and return the result as a QueryResult. - **`count`** – Count the number of rows in the DataFrame. - **`distinct`** – Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). - **`drop`** – Remove one or more columns from this DataFrame. - **`drop_duplicates`** – Return a DataFrame with duplicate rows removed. - **`explain`** – Display the logical plan of the DataFrame. - **`explode`** – Create a new row for each element in an array column. - **`explode_outer`** – Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. - **`explode_with_index`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`filter`** – Filters rows using the given condition. - **`group_by`** – Groups the DataFrame using the specified columns. - **`join`** – Joins this DataFrame with another DataFrame. - **`limit`** – Limits the number of rows to the specified number. - **`lineage`** – Create a Lineage object to trace data through transformations. - **`order_by`** – Sort the DataFrame by the specified columns. Alias for sort(). - **`persist`** – Mark this DataFrame to be persisted after first computation. - **`posexplode`** – Create a new row for each element in an array column, with the element's position in the array and its value. - **`posexplode_outer`** – Create a new row for each element in an array column with position and value, preserving null/empty arrays. - **`select`** – Projects a set of Column expressions or column names. - **`show`** – Display the DataFrame content in a tabular form. - **`sort`** – Sort the DataFrame by the specified columns. - **`to_arrow`** – Execute the DataFrame computation and return an Apache Arrow Table. - **`to_pandas`** – Execute the DataFrame computation and return a Pandas DataFrame. - **`to_polars`** – Execute the DataFrame computation and return the result as a Polars DataFrame. - **`to_pydict`** – Execute the DataFrame computation and return a dictionary of column arrays. - **`to_pylist`** – Execute the DataFrame computation and return a list of row dictionaries. - **`union`** – Return a new DataFrame containing the union of rows in this and another DataFrame. - **`unnest`** – Unnest the specified struct columns into separate columns. - **`where`** – Filters rows using the given condition (alias for filter()). - **`with_column`** – Add a new column or replace an existing column. - **`with_column_renamed`** – Rename a column. No-op if the column does not exist. - **`with_columns`** – Add multiple new columns or replace existing columns. Attributes: - **`columns`** (`List[str]`) – Get list of column names. - **`schema`** (`Schema`) – Get the schema of this DataFrame. - **`semantic`** (`SemanticExtensions`) – Interface for semantic operations on the DataFrame. - **`write`** (`DataFrameWriter`) – Interface for saving the content of the DataFrame. ### columns ``` columns: List[str] ``` Get list of column names. Returns: - `List[str]` – List[str]: List of all column names in the DataFrame Examples: ``` >>> df.columns ['name', 'age', 'city'] ``` ### schema ``` schema: Schema ``` Get the schema of this DataFrame. Returns: - **`Schema`** ( `Schema` ) – Schema containing field names and data types Examples: ``` >>> df.schema Schema([ ColumnField('name', StringType), ColumnField('age', IntegerType) ]) ``` ### semantic ``` semantic: SemanticExtensions ``` Interface for semantic operations on the DataFrame. ### write ``` write: DataFrameWriter ``` Interface for saving the content of the DataFrame. Returns: - **`DataFrameWriter`** ( `DataFrameWriter` ) – Writer interface to write DataFrame. ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions or dictionary of aggregations. Returns: - **`DataFrame`** ( `DataFrame` ) – Aggregation results. Multiple aggregations ``` # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Dictionary style ``` # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Aggregate on the entire DataFrame without groups. This is equivalent to group_by() without any grouping columns. Args: *exprs: Aggregation expressions or dictionary of aggregations. Returns: DataFrame: Aggregation results. Example: Multiple aggregations ```python # Create sample DataFrame df = session.create_dataframe({ "salary": [80000, 70000, 90000, 75000, 85000], "age": [25, 30, 35, 28, 32] }) # Multiple aggregations df.agg( count().alias("total_rows"), avg(col("salary")).alias("avg_salary") ).show() # Output: # +----------+-----------+ # |total_rows|avg_salary| # +----------+-----------+ # | 5| 80000.0| # +----------+-----------+ ``` Example: Dictionary style ```python # Dictionary style df.agg({col("salary"): "avg", col("age"): "max"}).show() # Output: # +-----------+--------+ # |avg(salary)|max(age)| # +-----------+--------+ # | 80000.0| 35| # +-----------+--------+ ``` """ return self.group_by().agg(*exprs) ``` ### cache ``` cache() -> DataFrame ``` Alias for persist(). Mark DataFrame for caching after first computation. Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for caching See Also persist(): Full documentation of caching behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def cache(self) -> DataFrame: """Alias for persist(). Mark DataFrame for caching after first computation. Returns: DataFrame: Same DataFrame, but marked for caching See Also: persist(): Full documentation of caching behavior """ return self.persist() ``` ### collect ``` collect(data_type: DataLikeType = 'polars') -> QueryResult ``` Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Parameters: - **`data_type`** (`DataLikeType`, default: `'polars'` ) – The type of data to return Returns: - **`QueryResult`** ( `QueryResult` ) – A QueryResult with materialized data and query metrics Source code in `src/fenic/api/dataframe/dataframe.py` ``` def collect(self, data_type: DataLikeType = "polars") -> QueryResult: """Execute the DataFrame computation and return the result as a QueryResult. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a QueryResult, which contains both the result data and the query metrics. Args: data_type: The type of data to return Returns: QueryResult: A QueryResult with materialized data and query metrics """ result: Tuple[pl.DataFrame, QueryMetrics] = self._session_state.execution.collect(self._logical_plan) df, metrics = result logger.info(metrics.get_summary()) if data_type == "polars": return QueryResult(df, metrics) elif data_type == "pandas": return QueryResult(df.to_pandas(use_pyarrow_extension_array=True), metrics) elif data_type == "arrow": return QueryResult(df.to_arrow(), metrics) elif data_type == "pydict": return QueryResult(df.to_dict(as_series=False), metrics) elif data_type == "pylist": return QueryResult(df.to_dicts(), metrics) else: raise ValidationError(f"Invalid data type: {data_type} in collect(). Valid data types are: polars, pandas, arrow, pydict, pylist") ``` ### count ``` count() -> int ``` Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: - **`int`** ( `int` ) – The number of rows in the DataFrame Source code in `src/fenic/api/dataframe/dataframe.py` ``` def count(self) -> int: """Count the number of rows in the DataFrame. This is an action that triggers computation of the DataFrame. The output is an integer representing the number of rows. Returns: int: The number of rows in the DataFrame """ return self._session_state.execution.count(self._logical_plan)[0] ``` ### distinct ``` distinct() -> DataFrame ``` Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Example ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def distinct(self) -> DataFrame: """Return a DataFrame with duplicate rows removed. Alias for drop_duplicates(subset=None). Returns: DataFrame: A new DataFrame with duplicate rows removed. Example: ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.distinct().show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ return self.drop_duplicates() ``` ### drop ``` drop(*col_names: str) -> DataFrame ``` Remove one or more columns from this DataFrame. Parameters: - **`*col_names`** (`str`, default: `()` ) – Names of columns to drop. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame without specified columns. Raises: - `ValueError` – If any specified column doesn't exist in the DataFrame. - `ValueError` – If dropping the columns would result in an empty DataFrame. Drop single column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Drop multiple columns ``` # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Error when dropping non-existent column ``` # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop(self, *col_names: str) -> DataFrame: """Remove one or more columns from this DataFrame. Args: *col_names: Names of columns to drop. Returns: DataFrame: New DataFrame without specified columns. Raises: ValueError: If any specified column doesn't exist in the DataFrame. ValueError: If dropping the columns would result in an empty DataFrame. Example: Drop single column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35] }) # Drop single column df.drop("age").show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Drop multiple columns ```python # Drop multiple columns df.drop(col("id"), "age").show() # Output: # +-------+ # | name| # +-------+ # | Alice| # | Bob| # |Charlie| # +-------+ ``` Example: Error when dropping non-existent column ```python # This will raise a ValueError df.drop("non_existent_column") # ValueError: Column 'non_existent_column' not found in DataFrame ``` """ if not col_names: return self current_cols = set(self.columns) to_drop = set(col_names) missing = to_drop - current_cols if missing: missing_str = ( f"Column '{next(iter(missing))}'" if len(missing) == 1 else f"Columns {sorted(missing)}" ) raise ValueError(f"{missing_str} not found in DataFrame") remaining_cols = [ col(c)._logical_expr for c in self.columns if c not in to_drop ] if not remaining_cols: raise ValueError("Cannot drop all columns from DataFrame") return self._from_logical_plan( Projection.from_session_state(self._logical_plan, remaining_cols, self._session_state), self._session_state, ) ``` ### drop_duplicates ``` drop_duplicates(subset: Optional[List[str]] = None) -> DataFrame ``` Return a DataFrame with duplicate rows removed. Parameters: - **`subset`** (`Optional[List[str]]`, default: `None` ) – Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with duplicate rows removed. Raises: - `ValueError` – If a specified column is not present in the current DataFrame schema. Remove duplicates considering specific columns ``` # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def drop_duplicates( self, subset: Optional[List[str]] = None, ) -> DataFrame: """Return a DataFrame with duplicate rows removed. Args: subset: Column names to consider when identifying duplicates. If not provided, all columns are considered. Returns: DataFrame: A new DataFrame with duplicate rows removed. Raises: ValueError: If a specified column is not present in the current DataFrame schema. Example: Remove duplicates considering specific columns ```python # Create sample DataFrame df = session.create_dataframe({ "c1": [1, 2, 3, 1], "c2": ["a", "a", "a", "a"], "c3": ["b", "b", "b", "b"] }) # Remove duplicates considering all columns df.drop_duplicates([col("c1"), col("c2"), col("c3")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ # Remove duplicates considering only c1 df.drop_duplicates([col("c1")]).show() # Output: # +---+---+---+ # | c1| c2| c3| # +---+---+---+ # | 1| a| b| # | 2| a| b| # | 3| a| b| # +---+---+---+ ``` """ exprs = [] if subset: for c in subset: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( DropDuplicates.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### explain ``` explain() -> None ``` Display the logical plan of the DataFrame. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explain(self) -> None: """Display the logical plan of the DataFrame.""" print(str(self._logical_plan)) ``` ### explode ``` explode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. Raises: - `TypeError` – If column argument is not a string or Column. Explode array column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Using column expression ``` # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column. This operation is useful for flattening nested data structures. For each row in the input DataFrame that contains an array/list in the specified column, this method will: 1. Create N new rows, where N is the length of the array 2. Each new row will be identical to the original row, except the array column will contain just a single element from the original array 3. Rows with NULL values or empty arrays in the specified column are filtered out Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Raises: TypeError: If column argument is not a string or Column. Example: Explode array column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4], "tags": [["red", "blue"], ["green"], [], None], "name": ["Alice", "Bob", "Carol", "Dave"] }) # Explode the tags column df.explode("tags").show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` Example: Using column expression ```python # Explode using column expression df.explode(col("tags")).show() # Output: # +---+-----+-----+ # | id| tags| name| # +---+-----+-----+ # | 1| red|Alice| # | 1| blue|Alice| # | 2|green| Bob| # +---+-----+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state(self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state), self._session_state, ) ``` ### explode_outer ``` explode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the array column exploded into multiple rows. - `DataFrame` – Rows with null or empty arrays are preserved with null in the exploded column. Explode with outer join behavior ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, containing the element's position in the array and its value, and preserving null/empty arrays. This operation is similar to explode(), but keeps rows where the array column is null or empty, producing a row with null in the exploded column. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with the array column exploded into multiple rows. Rows with null or empty arrays are preserved with null in the exploded column. Example: Explode with outer join behavior ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.explode_outer("tags").show() # Output: # +---+-----+ # | id| tags| # +---+-----+ # | 1| red| # | 1| blue| # | 2| NULL| # empty array preserved as null # | 3| NULL| # null array preserved as null # +---+-----+ ``` """ return self._from_logical_plan( Explode.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, self._session_state, keep_null_and_empty=True ), self._session_state, ) ``` ### explode_with_index ``` explode_with_index(column: ColumnOrName, index_col_name: str = 'pos', value_col_name: str = 'col', keep_null_and_empty: bool = False) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. - **`index_col_name`** (`str`, default: `'pos'` ) – Name for the column containing 0-based array positions (default: "pos"). - **`value_col_name`** (`str`, default: `'col'` ) – Name for the exploded value column (default: "col"). - **`keep_null_and_empty`** (`bool`, default: `False` ) – If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Explode with index ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Custom column names ``` df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def explode_with_index( self, column: ColumnOrName, index_col_name: str = "pos", value_col_name: str = "col", keep_null_and_empty: bool = False, ) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This operation is similar to explode(), but also adds a column containing the 0-based position of each element within its original array. By default, the position column is named "pos". and the value column is named "col". These columns replace the original column in the output DataFrame. If keep_null_and_empty is True, the position column will be null for rows where the array is null or empty. Args: column: Name of array column to explode (as string) or Column expression. index_col_name: Name for the column containing 0-based array positions (default: "pos"). value_col_name: Name for the exploded value column (default: "col"). keep_null_and_empty: If True, preserves rows where the array is null or empty (default: False). Mimicks the behavior of posexplode (false) vs posexplode_outer (true). Returns: DataFrame: New DataFrame with: - An integer column (named `index_col_name`) containing 0-based positions - The exploded array column (named `value_col_name`) - All other columns from the original DataFrame Example: Explode with index ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], ["green"], []], }) df.explode_with_index("tags").show() # Output: # +-----+---+-----+ # | pos| id| tags| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` Example: Custom column names ```python df.explode_with_index("tags", index_col_name="index", value_name="tag").show() # Output: # +-----+---+-----+ # |index| id| tag| # +-----+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +-----+---+-----+ ``` """ return self._from_logical_plan( ExplodeWithIndex.from_session_state( self._logical_plan, Column._from_col_or_name(column)._logical_expr, index_col_name, value_col_name, self._session_state, keep_null_and_empty, ), self._session_state, ) ``` ### filter ``` filter(condition: Column) -> DataFrame ``` Filters rows using the given condition. Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame Filter with numeric comparison ``` # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with semantic predicate ``` # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Filter with multiple conditions ``` # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def filter(self, condition: Column) -> DataFrame: """Filters rows using the given condition. Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame Example: Filter with numeric comparison ```python # Create a DataFrame df = session.create_dataframe({"age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"]}) # Filter with numeric comparison df.filter(col("age") > 25).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with semantic predicate ```python # Filter with semantic predicate df.filter((col("age") > 25) & semantic.predicate("This {feedback} mentions problems with the user interface or navigation")).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` Example: Filter with multiple conditions ```python # Filter with multiple conditions df.filter((col("age") > 25) & (col("age") <= 35)).show() # Output: # +---+-------+ # |age| name| # +---+-------+ # | 30| Bob| # | 35|Charlie| # +---+-------+ ``` """ return self._from_logical_plan( Filter.from_session_state(self._logical_plan, condition._logical_expr, self._session_state), self._session_state, ) ``` ### group_by ``` group_by(*cols: ColumnOrName) -> GroupedData ``` Groups the DataFrame using the specified columns. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns to group by. Can be column names as strings or Column expressions. Returns: - **`GroupedData`** ( `GroupedData` ) – Object for performing aggregations on the grouped data. Group by single column ``` # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Group by multiple columns ``` # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Group by expression ``` # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def group_by(self, *cols: ColumnOrName) -> GroupedData: """Groups the DataFrame using the specified columns. Args: *cols: Columns to group by. Can be column names as strings or Column expressions. Returns: GroupedData: Object for performing aggregations on the grouped data. Example: Group by single column ```python # Create sample DataFrame df = session.create_dataframe({ "department": ["IT", "HR", "IT", "HR", "IT"], "salary": [80000, 70000, 90000, 75000, 85000] }) # Group by single column df.group_by(col("department")).agg(count("*")).show() # Output: # +----------+-----+ # |department|count| # +----------+-----+ # | IT| 3| # | HR| 2| # +----------+-----+ ``` Example: Group by multiple columns ```python # Group by multiple columns df.group_by(col("department"), col("location")).agg({"salary": "avg"}).show() # Output: # +----------+--------+-----------+ # |department|location|avg(salary)| # +----------+--------+-----------+ # | IT| NYC| 85000.0| # | HR| NYC| 72500.0| # +----------+--------+-----------+ ``` Example: Group by expression ```python # Group by expression df.group_by(lower(col("department")).alias("department")).agg(count("*")).show() # Output: # +---------+-----+ # |department|count| # +----------+-----+ # | it| 3| # | hr| 2| # +---------+-----+ ``` """ return GroupedData(self, list(cols) if cols else None) ``` ### join ``` join(other: DataFrame, on: Union[str, List[str]], *, how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, *, left_on: Union[ColumnOrName, List[ColumnOrName]], right_on: Union[ColumnOrName, List[ColumnOrName]], how: JoinType = 'inner') -> DataFrame ``` ``` join(other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = 'inner') -> DataFrame ``` Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Parameters: - **`other`** (`DataFrame`) – DataFrame to join with. - **`on`** (`Optional[Union[str, List[str]]]`, default: `None` ) – Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins - **`left_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions - **`right_on`** (`Optional[Union[ColumnOrName, List[ColumnOrName]]]`, default: `None` ) – Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions - **`how`** (`JoinType`, default: `'inner'` ) – Type of join to perform. Returns: - `DataFrame` – Joined DataFrame. Raises: - `ValidationError` – If cross join is used with an ON clause. - `ValidationError` – If join condition is invalid. - `ValidationError` – If both 'on' and 'left_on'/'right_on' parameters are provided. - `ValidationError` – If only one of 'left_on' or 'right_on' is provided. - `ValidationError` – If 'left_on' and 'right_on' have different lengths Inner join on column name ``` # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Join with expression ``` # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Cross join ``` # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def join( self, other: DataFrame, on: Optional[Union[str, List[str]]] = None, *, left_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, right_on: Optional[Union[ColumnOrName, List[ColumnOrName]]] = None, how: JoinType = "inner", ) -> DataFrame: """Joins this DataFrame with another DataFrame. The Dataframes must have no duplicate column names between them. This API only supports equi-joins. For non-equi-joins, use session.sql(). Args: other: DataFrame to join with. on: Join condition(s). Can be: - A column name (str) - A list of column names (List[str]) - A Column expression (e.g., col('a')) - A list of Column expressions - `None` for cross joins left_on: Column(s) from the left DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('a'), col('a') + 1) - A list of column names or expressions right_on: Column(s) from the right DataFrame to join on. Can be: - A column name (str) - A Column expression (e.g., col('b'), upper(col('b'))) - A list of column names or expressions how: Type of join to perform. Returns: Joined DataFrame. Raises: ValidationError: If cross join is used with an ON clause. ValidationError: If join condition is invalid. ValidationError: If both 'on' and 'left_on'/'right_on' parameters are provided. ValidationError: If only one of 'left_on' or 'right_on' is provided. ValidationError: If 'left_on' and 'right_on' have different lengths Example: Inner join on column name ```python # Create sample DataFrames df1 = session.create_dataframe({ "id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"] }) df2 = session.create_dataframe({ "id": [1, 2, 4], "age": [25, 30, 35] }) # Join on single column df1.join(df2, on=col("id")).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Join with expression ```python # Join with Column expressions df1.join( df2, left_on=col("id"), right_on=col("id"), ).show() # Output: # +---+-----+---+ # | id| name|age| # +---+-----+---+ # | 1|Alice| 25| # | 2| Bob| 30| # +---+-----+---+ ``` Example: Cross join ```python # Cross join (cartesian product) df1.join(df2, how="cross").show() # Output: # +---+-----+---+---+ # | id| name| id|age| # +---+-----+---+---+ # | 1|Alice| 1| 25| # | 1|Alice| 2| 30| # | 1|Alice| 4| 35| # | 2| Bob| 1| 25| # | 2| Bob| 2| 30| # | 2| Bob| 4| 35| # | 3|Charlie| 1| 25| # | 3|Charlie| 2| 30| # | 3|Charlie| 4| 35| # +---+-----+---+---+ ``` """ validate_join_parameters(self, on, left_on, right_on, how) # Build join conditions left_conditions, right_conditions = build_join_conditions(on, left_on, right_on) self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( Join.from_session_state( self._logical_plan, other._logical_plan, left_conditions, right_conditions, how, self._session_state), self._session_state, ) ``` ### limit ``` limit(n: int) -> DataFrame ``` Limits the number of rows to the specified number. Parameters: - **`n`** (`int`) – Maximum number of rows to return. Returns: - **`DataFrame`** ( `DataFrame` ) – DataFrame with at most n rows. Raises: - `TypeError` – If n is not an integer. Limit rows ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Limit with other operations ``` # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def limit(self, n: int) -> DataFrame: """Limits the number of rows to the specified number. Args: n: Maximum number of rows to return. Returns: DataFrame: DataFrame with at most n rows. Raises: TypeError: If n is not an integer. Example: Limit rows ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2, 3, 4, 5], "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"] }) # Get first 3 rows df.limit(3).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 1| Alice| # | 2| Bob| # | 3|Charlie| # +---+-------+ ``` Example: Limit with other operations ```python # Limit after filtering df.filter(col("id") > 2).limit(2).show() # Output: # +---+-------+ # | id| name| # +---+-------+ # | 3|Charlie| # | 4| Dave| # +---+-------+ ``` """ return self._from_logical_plan( Limit.from_session_state(self._logical_plan, n, self._session_state), self._session_state) ``` ### lineage ``` lineage() -> Lineage ``` Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: - **`Lineage`** ( `Lineage` ) – Interface for querying data lineage Example ``` # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also LineageQuery: Full documentation of lineage querying capabilities Source code in `src/fenic/api/dataframe/dataframe.py` ``` def lineage(self) -> Lineage: """Create a Lineage object to trace data through transformations. The Lineage interface allows you to trace how specific rows are transformed through your DataFrame operations, both forwards and backwards through the computation graph. Returns: Lineage: Interface for querying data lineage Example: ```python # Create lineage query lineage = df.lineage() # Trace specific rows backwards through transformations source_rows = lineage.backward(["result_uuid1", "result_uuid2"]) # Or trace forwards to see outputs result_rows = lineage.forward(["source_uuid1"]) ``` See Also: LineageQuery: Full documentation of lineage querying capabilities """ return Lineage(self._session_state.execution.build_lineage(self._logical_plan)) ``` ### order_by ``` order_by(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Alias for sort(). Returns: - **`DataFrame`** ( `DataFrame` ) – sorted Dataframe. See Also sort(): Full documentation of sorting behavior and parameters. Source code in `src/fenic/api/dataframe/dataframe.py` ``` def order_by( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Alias for sort(). Returns: DataFrame: sorted Dataframe. See Also: sort(): Full documentation of sorting behavior and parameters. """ return self.sort(cols, ascending) ``` ### persist ``` persist() -> DataFrame ``` Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: - **`DataFrame`** ( `DataFrame` ) – Same DataFrame, but marked for persistence Example ``` # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def persist(self) -> DataFrame: """Mark this DataFrame to be persisted after first computation. The persisted DataFrame will be cached after its first computation, avoiding recomputation in subsequent operations. This is useful for: - DataFrames that are created once and reused multiple times in your workflow - DataFrames that are computationally expensive (large joins, aggregations, etc.) Returns: DataFrame: Same DataFrame, but marked for persistence Example: ```python # Cache intermediate results for reuse filtered_df = (df .filter(col("age") > 25) .persist() # Cache these results ) # Both operations will use cached results result1 = filtered_df.group_by("department").count() result2 = filtered_df.select("name", "salary") ``` """ cache_info = CacheInfo(cache_key=f"cache_{uuid.uuid4().hex}") self._logical_plan.set_cache_info(cache_info) return self._from_logical_plan( self._logical_plan, self._session_state) ``` ### posexplode ``` posexplode(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. PySpark-style posexplode ``` df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column, with the element's position in the array and its value. This is a PySpark-compatible alias for explode_with_index. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). These columns replace the original column in the output DataFrame. Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Example: PySpark-style posexplode ```python df = session.create_dataframe({ "id": [1, 2], "tags": [["red", "blue"], ["green"]], }) df.posexplode("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # | 0| 2|green| # +---+---+-----+ ``` """ return self.explode_with_index(column) ``` ### posexplode_outer ``` posexplode_outer(column: ColumnOrName) -> DataFrame ``` Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Parameters: - **`column`** (`ColumnOrName`) – Name of array column to explode (as string) or Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with 'pos' and 'col' columns, plus all other original columns. - `DataFrame` – Rows with null or empty arrays are preserved with (null, null). PySpark-style posexplode_outer ``` df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def posexplode_outer(self, column: ColumnOrName) -> DataFrame: """Create a new row for each element in an array column with position and value, preserving null/empty arrays. This is a PySpark-compatible alias for explode_with_index with keep_null_and_empty=True. Creates two columns: 'pos' (0-based position) and 'col' (the array element value). Rows with null or empty arrays produce (null, null). Args: column: Name of array column to explode (as string) or Column expression. Returns: DataFrame: New DataFrame with 'pos' and 'col' columns, plus all other original columns. Rows with null or empty arrays are preserved with (null, null). Example: PySpark-style posexplode_outer ```python df = session.create_dataframe({ "id": [1, 2, 3], "tags": [["red", "blue"], [], None], }) df.posexplode_outer("tags").show() # Output: # +---+---+-----+ # |pos| id| col| # +---+---+-----+ # | 0| 1| red| # | 1| 1| blue| # |NULL| 2| NULL| # empty array preserved # |NULL| 3| NULL| # null array preserved # +---+---+-----+ ``` """ return self.explode_with_index( column, keep_null_and_empty=True ) ``` ### select ``` select(*cols: ColumnOrName) -> DataFrame ``` Projects a set of Column expressions or column names. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with selected columns Select by column names ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Select with expressions ``` # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Mix strings and expressions ``` # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def select(self, *cols: ColumnOrName) -> DataFrame: """Projects a set of Column expressions or column names. Args: *cols: Column expressions to select. Can be: - String column names (e.g., "id", "name") - Column objects (e.g., col("id"), col("age") + 1) Returns: DataFrame: A new DataFrame with selected columns Example: Select by column names ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Select by column names df.select(col("name"), col("age")).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 25| # | Bob| 30| # +-----+---+ ``` Example: Select with expressions ```python # Select with expressions df.select(col("name"), col("age") + 1).show() # Output: # +-----+-------+ # | name|age + 1| # +-----+-------+ # |Alice| 26| # | Bob| 31| # +-----+-------+ ``` Example: Mix strings and expressions ```python # Mix strings and expressions df.select(col("name"), col("age") * 2).show() # Output: # +-----+-------+ # | name|age * 2| # +-----+-------+ # |Alice| 50| # | Bob| 60| # +-----+-------+ ``` """ exprs = [] if not cols: return self for c in cols: if isinstance(c, str): if c == "*": exprs.extend(col(field)._logical_expr for field in self.columns) else: exprs.append(col(c)._logical_expr) else: exprs.append(c._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### show ``` show(n: int = 10, explain_analyze: bool = False) -> None ``` Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Parameters: - **`n`** (`int`, default: `10` ) – Number of rows to display - **`explain_analyze`** (`bool`, default: `False` ) – Whether to print the explain analyze plan Source code in `src/fenic/api/dataframe/dataframe.py` ``` def show(self, n: int = 10, explain_analyze: bool = False) -> None: """Display the DataFrame content in a tabular form. This is an action that triggers computation of the DataFrame. The output is printed to stdout in a formatted table. Args: n: Number of rows to display explain_analyze: Whether to print the explain analyze plan """ output, metrics = self._session_state.execution.show(self._logical_plan, n) logger.info(metrics.get_summary()) print(output) if explain_analyze: print(metrics.get_execution_plan_details()) ``` ### sort ``` sort(cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None) -> DataFrame ``` Sort the DataFrame by the specified columns. Parameters: - **`cols`** (`Union[ColumnOrName, List[ColumnOrName], None]`, default: `None` ) – Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. - **`ascending`** (`Optional[Union[bool, List[bool]]]`, default: `None` ) – A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame sorted by the specified columns. Raises: - `ValueError` – - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used - `TypeError` – - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Sort in ascending order ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Sort in descending order ``` # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Sort with boolean ascending parameter ``` # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Multiple columns with different sort orders ``` # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Multiple columns with list of ascending strategies ``` # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def sort( self, cols: Union[ColumnOrName, List[ColumnOrName], None] = None, ascending: Optional[Union[bool, List[bool]]] = None, ) -> DataFrame: """Sort the DataFrame by the specified columns. Args: cols: Columns to sort by. This can be: - A single column name (str) - A Column expression (e.g., `col("name")`) - A list of column names or Column expressions - Column expressions may include sorting directives such as `asc("col")`, `desc("col")`, `asc_nulls_last("col")`, etc. - If no columns are provided, the operation is a no-op. ascending: A boolean or list of booleans indicating sort order. - If `True`, sorts in ascending order; if `False`, descending. - If a list is provided, its length must match the number of columns. - Cannot be used if any of the columns use `asc()`/`desc()` expressions. - If not specified and no sort expressions are used, columns will be sorted in ascending order by default. Returns: DataFrame: A new DataFrame sorted by the specified columns. Raises: ValueError: - If `ascending` is provided and its length does not match `cols` - If both `ascending` and column expressions like `asc()`/`desc()` are used TypeError: - If `cols` is not a column name, Column, or list of column names/Columns - If `ascending` is not a boolean or list of booleans Example: Sort in ascending order ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) # Sort by age in ascending order df.sort(asc(col("age"))).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 2|Alice| # | 5| Bob| # +---+-----+ ``` Example: Sort in descending order ```python # Sort by age in descending order df.sort(col("age").desc()).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Sort with boolean ascending parameter ```python # Sort by age in descending order using boolean df.sort(col("age"), ascending=False).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # +---+-----+ ``` Example: Multiple columns with different sort orders ```python # Create sample DataFrame df = session.create_dataframe([(2, "Alice"), (2, "Bob"), (5, "Bob")], schema=["age", "name"]) # Sort by age descending, then name ascending df.sort(desc(col("age")), col("name")).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2|Alice| # | 2| Bob| # +---+-----+ ``` Example: Multiple columns with list of ascending strategies ```python # Sort both columns in descending order df.sort([col("age"), col("name")], ascending=[False, False]).show() # Output: # +---+-----+ # |age| name| # +---+-----+ # | 5| Bob| # | 2| Bob| # | 2|Alice| # +---+-----+ ``` """ col_args = cols if cols is None: return self._from_logical_plan( Sort.from_session_state(self._logical_plan, [], self._session_state), self._session_state, ) elif not isinstance(cols, List): col_args = [cols] # parse the ascending arguments bool_ascending = [] using_default_ascending = False if ascending is None: using_default_ascending = True bool_ascending = [True] * len(col_args) elif isinstance(ascending, bool): bool_ascending = [ascending] * len(col_args) elif isinstance(ascending, List): bool_ascending = ascending if len(bool_ascending) != len(cols): raise ValueError( f"the list length of ascending sort strategies must match the specified sort columns" f"Got {len(cols)} column expressions and {len(bool_ascending)} ascending strategies. " ) else: raise TypeError( f"Invalid ascending strategy type: {type(ascending)}. Must be a boolean or list of booleans." ) # create our list of sort expressions, for each column expression # that isn't already provided as a asc()/desc() SortExpr sort_exprs = [] for c, asc_bool in zip(col_args, bool_ascending, strict=True): if isinstance(c, ColumnOrName): c_expr = Column._from_col_or_name(c)._logical_expr else: raise TypeError( f"Invalid column type: {type(c).__name__}. Must be a string or Column Expression." ) if not isinstance(asc_bool, bool): raise TypeError( f"Invalid ascending strategy type: {type(asc_bool).__name__}. Must be a boolean." ) if isinstance(c_expr, SortExpr): if not using_default_ascending: raise TypeError( "Cannot specify both asc()/desc() expressions and boolean ascending strategies." f"Got expression: {c_expr} and ascending argument: {bool_ascending}" ) sort_exprs.append(c_expr) else: sort_exprs.append(SortExpr(c_expr, ascending=asc_bool)) return self._from_logical_plan( Sort.from_session_state(self._logical_plan, sort_exprs, self._session_state), self._session_state, ) ``` ### to_arrow ``` to_arrow() -> pa.Table ``` Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: - `Table` – pa.Table: An Apache Arrow Table containing the computed results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_arrow(self) -> pa.Table: """Execute the DataFrame computation and return an Apache Arrow Table. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into an Apache Arrow Table with columnar memory layout optimized for analytics and zero-copy data exchange. Returns: pa.Table: An Apache Arrow Table containing the computed results """ return self.collect("arrow").data ``` ### to_pandas ``` to_pandas() -> pd.DataFrame ``` Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: - `DataFrame` – pd.DataFrame: A Pandas DataFrame containing the computed results with Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pandas(self) -> pd.DataFrame: """Execute the DataFrame computation and return a Pandas DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Pandas DataFrame. Returns: pd.DataFrame: A Pandas DataFrame containing the computed results with """ return self.collect("pandas").data ``` ### to_polars ``` to_polars() -> pl.DataFrame ``` Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: - `DataFrame` – pl.DataFrame: A Polars DataFrame with materialized results Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_polars(self) -> pl.DataFrame: """Execute the DataFrame computation and return the result as a Polars DataFrame. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Polars DataFrame. Returns: pl.DataFrame: A Polars DataFrame with materialized results """ return self.collect("polars").data ``` ### to_pydict ``` to_pydict() -> Dict[str, List[Any]] ``` Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: - `Dict[str, List[Any]]` – Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pydict(self) -> Dict[str, List[Any]]: """Execute the DataFrame computation and return a dictionary of column arrays. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python dictionary where each column becomes a list of values. Returns: Dict[str, List[Any]]: A dictionary containing the computed results with: - Keys: Column names as strings - Values: Lists containing all values for each column """ return self.collect("pydict").data ``` ### to_pylist ``` to_pylist() -> List[Dict[str, Any]] ``` Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: - `List[Dict[str, Any]]` – List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result Source code in `src/fenic/api/dataframe/dataframe.py` ``` def to_pylist(self) -> List[Dict[str, Any]]: """Execute the DataFrame computation and return a list of row dictionaries. This is an action that triggers computation of the DataFrame query plan. All transformations and operations are executed, and the results are materialized into a Python list where each element is a dictionary representing one row with column names as keys. Returns: List[Dict[str, Any]]: A list containing the computed results with: - Each element: A dictionary representing one row - Dictionary keys: Column names as strings - Dictionary values: Cell values in Python native types - List length equals number of rows in the result """ return self.collect("pylist").data ``` ### union ``` union(other: DataFrame) -> DataFrame ``` Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Parameters: - **`other`** (`DataFrame`) – Another DataFrame with the same schema. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing rows from both DataFrames. Raises: - `ValueError` – If the DataFrames have different schemas. - `TypeError` – If other is not a DataFrame. Union two DataFrames ``` # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Union with duplicates ``` # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def union(self, other: DataFrame) -> DataFrame: """Return a new DataFrame containing the union of rows in this and another DataFrame. This is equivalent to UNION ALL in SQL. To remove duplicates, use drop_duplicates() after union(). Args: other: Another DataFrame with the same schema. Returns: DataFrame: A new DataFrame containing rows from both DataFrames. Raises: ValueError: If the DataFrames have different schemas. TypeError: If other is not a DataFrame. Example: Union two DataFrames ```python # Create two DataFrames df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [3, 4], "value": ["c", "d"] }) # Union the DataFrames df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # | 4| d| # +---+-----+ ``` Example: Union with duplicates ```python # Create DataFrames with overlapping data df1 = session.create_dataframe({ "id": [1, 2], "value": ["a", "b"] }) df2 = session.create_dataframe({ "id": [2, 3], "value": ["b", "c"] }) # Union with duplicates df1.union(df2).show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 2| b| # | 3| c| # +---+-----+ # Remove duplicates after union df1.union(df2).drop_duplicates().show() # Output: # +---+-----+ # | id|value| # +---+-----+ # | 1| a| # | 2| b| # | 3| c| # +---+-----+ ``` """ self._ensure_same_session(self._session_state, [other._session_state]) return self._from_logical_plan( UnionLogicalPlan.from_session_state([self._logical_plan, other._logical_plan], self._session_state), self._session_state, ) ``` ### unnest ``` unnest(*col_names: str) -> DataFrame ``` Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Parameters: - **`*col_names`** (`str`, default: `()` ) – One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with the specified struct columns expanded. Raises: - `TypeError` – If any argument is not a string or Column. - `ValueError` – If a specified column does not contain struct data. Unnest struct column ``` # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Unnest multiple struct columns ``` # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def unnest(self, *col_names: str) -> DataFrame: """Unnest the specified struct columns into separate columns. This operation flattens nested struct data by expanding each field of a struct into its own top-level column. For each specified column containing a struct: 1. Each field in the struct becomes a separate column. 2. New columns are named after the corresponding struct fields. 3. The new columns are inserted into the DataFrame in place of the original struct column. 4. The overall column order is preserved. Args: *col_names: One or more struct columns to unnest. Each can be a string (column name) or a Column expression. Returns: DataFrame: A new DataFrame with the specified struct columns expanded. Raises: TypeError: If any argument is not a string or Column. ValueError: If a specified column does not contain struct data. Example: Unnest struct column ```python # Create sample DataFrame df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "name": ["Alice", "Bob"] }) # Unnest the tags column df.unnest(col("tags")).show() # Output: # +---+---+----+-----+ # | id| red|blue| name| # +---+---+----+-----+ # | 1| 1| 2|Alice| # | 2| 3|null| Bob| # +---+---+----+-----+ ``` Example: Unnest multiple struct columns ```python # Create sample DataFrame with multiple struct columns df = session.create_dataframe({ "id": [1, 2], "tags": [{"red": 1, "blue": 2}, {"red": 3}], "info": [{"age": 25, "city": "NY"}, {"age": 30, "city": "LA"}], "name": ["Alice", "Bob"] }) # Unnest multiple struct columns df.unnest(col("tags"), col("info")).show() # Output: # +---+---+----+---+----+-----+ # | id| red|blue|age|city| name| # +---+---+----+---+----+-----+ # | 1| 1| 2| 25| NY|Alice| # | 2| 3|null| 30| LA| Bob| # +---+---+----+---+----+-----+ ``` """ if not col_names: return self exprs = [] for c in col_names: if c not in self.columns: raise TypeError(f"Column {c} not found in DataFrame.") exprs.append(col(c)._logical_expr) return self._from_logical_plan( Unnest.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### where ``` where(condition: Column) -> DataFrame ``` Filters rows using the given condition (alias for filter()). Parameters: - **`condition`** (`Column`) – A Column expression that evaluates to a boolean Returns: - **`DataFrame`** ( `DataFrame` ) – Filtered DataFrame See Also filter(): Full documentation of filtering behavior Source code in `src/fenic/api/dataframe/dataframe.py` ``` def where(self, condition: Column) -> DataFrame: """Filters rows using the given condition (alias for filter()). Args: condition: A Column expression that evaluates to a boolean Returns: DataFrame: Filtered DataFrame See Also: filter(): Full documentation of filtering behavior """ return self.filter(condition) ``` ### with_column ``` with_column(col_name: str, col: Union[Any, Column, Series, Series]) -> DataFrame ``` Add a new column or replace an existing column. Parameters: - **`col_name`** (`str`) – Name of the new column - **`col`** (`Union[Any, Column, Series, Series]`) – Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced column Raises: - `ExecutionError` – - If a Series length does not match the DataFrame height - `ValidationError` – - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Add literal column ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Add computed column ``` # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Replace existing column ``` # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Add column with complex expression ``` # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Add column from Polars Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Add column from pandas Series ``` import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column(self, col_name: str, col: Union[Any, Column, pl.Series, pd.Series]) -> DataFrame: """Add a new column or replace an existing column. Args: col_name: Name of the new column col: Column expression, Series, or value to assign to the column: - Column: A Column expression (e.g., `col("age") + 1`) - `pl.Series` or `pd.Series`: A Polars or pandas Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as a literal value (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced column Raises: ExecutionError: - If a Series length does not match the DataFrame height ValidationError: - If the Series contains all null values and no dtype is specified - If the Series has length 0 Notes: - The name of the created column will be the name defined in col_name, even if input is a Series with a different name. Example: Add literal column ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add literal column df.with_column("constant", lit(1)).show() # Output: # +-----+---+--------+ # | name|age|constant| # +-----+---+--------+ # |Alice| 25| 1| # | Bob| 30| 1| # +-----+---+--------+ ``` Example: Add computed column ```python # Add computed column df.with_column("double_age", col("age") * 2).show() # Output: # +-----+---+----------+ # | name|age|double_age| # +-----+---+----------+ # |Alice| 25| 50| # | Bob| 30| 60| # +-----+---+----------+ ``` Example: Replace existing column ```python # Replace existing column df.with_column("age", col("age") + 1).show() # Output: # +-----+---+ # | name|age| # +-----+---+ # |Alice| 26| # | Bob| 31| # +-----+---+ ``` Example: Add column with complex expression ```python # Add column with complex expression df.with_column( "age_category", when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior") ).show() # Output: # +-----+---+------------+ # | name|age|age_category| # +-----+---+------------+ # |Alice| 25| young| # | Bob| 30| middle| # +-----+---+------------+ ``` Example: Add column from Polars Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from Polars Series bonus = pl.Series([100, 200]) df.with_column("bonus", bonus).show() # Output: # +-----+---+-----+ # | name|age|bonus| # +-----+---+-----+ # |Alice| 25| 100| # | Bob| 30| 200| # +-----+---+-----+ ``` Example: Add column from pandas Series ```python import pandas as pd # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add column from pandas Series score = pd.Series([85.5, 92.0]) df.with_column("score", score).show() # Output: # +-----+---+-----+ # | name|age|score| # +-----+---+-----+ # |Alice| 25| 85.5| # | Bob| 30| 92.0| # +-----+---+-----+ ``` """ exprs = [] # Handle different input types: Column, Series, or literal value if isinstance(col, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col = Column._from_logical_expr(SeriesLiteralExpr(col)) elif not isinstance(col, Column): # Wrap other values as literals col = lit(col) for field in self.columns: if field != col_name: exprs.append(Column._from_column_name(field)._logical_expr) # Add the new column with alias exprs.append(col.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_column_renamed ``` with_column_renamed(col_name: str, new_col_name: str) -> DataFrame ``` Rename a column. No-op if the column does not exist. Parameters: - **`col_name`** (`str`) – Name of the column to rename. - **`new_col_name`** (`str`) – New name for the column. Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with the column renamed. Rename a column ``` # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Rename multiple columns ``` # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_column_renamed(self, col_name: str, new_col_name: str) -> DataFrame: """Rename a column. No-op if the column does not exist. Args: col_name: Name of the column to rename. new_col_name: New name for the column. Returns: DataFrame: New DataFrame with the column renamed. Example: Rename a column ```python # Create sample DataFrame df = session.create_dataframe({ "age": [25, 30, 35], "name": ["Alice", "Bob", "Charlie"] }) # Rename a column df.with_column_renamed("age", "age_in_years").show() # Output: # +------------+-------+ # |age_in_years| name| # +------------+-------+ # | 25| Alice| # | 30| Bob| # | 35|Charlie| # +------------+-------+ ``` Example: Rename multiple columns ```python # Rename multiple columns df = (df .with_column_renamed("age", "age_in_years") .with_column_renamed("name", "full_name") ).show() # Output: # +------------+----------+ # |age_in_years|full_name | # +------------+----------+ # | 25| Alice| # | 30| Bob| # | 35| Charlie| # +------------+----------+ ``` """ exprs = [] renamed = False for field in self.schema.column_fields: name = field.name if name == col_name: exprs.append(col(name).alias(new_col_name)._logical_expr) renamed = True else: exprs.append(col(name)._logical_expr) if not renamed: return self return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` ### with_columns ``` with_columns(cols_map: Dict[str, Union[Any, Column, Series, Series]]) -> DataFrame ``` Add multiple new columns or replace existing columns. Parameters: - **`cols_map`** (`Dict[str, Union[Any, Column, Series, Series]]`) – A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: - **`DataFrame`** ( `DataFrame` ) – New DataFrame with added/replaced columns Raises: - `ValueError` – - If two columns being created in the same `with_columns` call depend on each other - `ExecutionError` – - If any Series length does not match the DataFrame height - `ValidationError` – - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Add multiple columns ``` # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Replace and add columns ``` # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Complex expressions ``` # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Add columns from Series ``` import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Mix Series with Column expressions ``` import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Error when adding columns that depend on each other ``` df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` Source code in `src/fenic/api/dataframe/dataframe.py` ``` def with_columns(self, cols_map: Dict[str, Union[Any, Column, pl.Series, pd.Series]]) -> DataFrame: """Add multiple new columns or replace existing columns. Args: cols_map: A dictionary where keys are column names and values are: - Column: Column expressions (e.g., col("age") + 1) - pl.Series or pd.Series: Series with data - **Note: Series length MUST match the DataFrame height** - Any other value: Treated as literal values (broadcast to all rows) Returns: DataFrame: New DataFrame with added/replaced columns Raises: ValueError: - If two columns being created in the same `with_columns` call depend on each other ExecutionError: - If any Series length does not match the DataFrame height ValidationError: - If any Series contains all null values and no dtype is specified - If any Series has length 0 Notes: - All columns are created at once, so new columns cannot depend on each other. - The name of the created column will be the name defined in cols_map, even if input is a Series with a different name. Example: Add multiple columns ```python # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns at once df.with_columns({ "double_age": col("age") * 2, "constant": lit(1), "age_plus_10": col("age") + 10 }).show() # Output: # +-----+---+----------+--------+-----------+ # | name|age|double_age|constant|age_plus_10| # +-----+---+----------+--------+-----------+ # |Alice| 25| 50| 1| 35| # | Bob| 30| 60| 1| 40| # +-----+---+----------+--------+-----------+ ``` Example: Replace and add columns ```python # Replace existing column and add new ones df.with_columns({ "age": col("age") + 1, "is_adult": col("age") >= 18 }).show() # Output: # +-----+---+--------+ # | name|age|is_adult| # +-----+---+--------+ # |Alice| 26| true| # | Bob| 31| true| # +-----+---+--------+ ``` Example: Complex expressions ```python # Add multiple columns with complex expressions df.with_columns({ "age_category": when(col("age") < 30, "young") .when(col("age") < 50, "middle") .otherwise("senior"), "name_length": length(col("name")), "name_upper": upper(col("name")) }).show() # Output: # +-----+---+------------+-----------+----------+ # | name|age|age_category|name_length|name_upper| # +-----+---+------------+-----------+----------+ # |Alice| 25| young| 5| ALICE| # | Bob| 30| middle| 3| BOB| # +-----+---+------------+-----------+----------+ ``` Example: Add columns from Series ```python import polars as pl # Create a DataFrame df = session.create_dataframe({"name": ["Alice", "Bob"], "age": [25, 30]}) # Add multiple columns from Series df.with_columns({ "bonus": pl.Series([100, 200]), "score": pl.Series([85.5, 92.0]) }).show() # Output: # +-----+---+-----+-----+ # | name|age|bonus|score| # +-----+---+-----+-----+ # |Alice| 25| 100| 85.5| # | Bob| 30| 200| 92.0| # +-----+---+-----+-----+ ``` Example: Mix Series with Column expressions ```python import polars as pl # Mix Series with Column expressions df.with_columns({ "bonus": pl.Series([100, 200]), "double_age": col("age") * 2, "constant": 1 }).show() # Output: # +-----+---+-----+----------+--------+ # | name|age|bonus|double_age|constant| # +-----+---+-----+----------+--------+ # |Alice| 25| 100| 50| 1| # | Bob| 30| 200| 60| 1| # +-----+---+-----+----------+--------+ ``` Example: Error when adding columns that depend on each other ```python df.with_columns({ "age_plus_1": col("age") + 1, "age_plus_2": col("age_plus_1") + 1 }) # ValueError: Column 'age_plus_1' not found in schema # Instead, use a single with_column call df = df.with_column( "age_plus_1", col("age") + 1 ).with_column( "age_plus_2", col("age_plus_1") + 1 ) df.show() # Output: # +-----+---+----------+----------+ # | name|age|age_plus_1|age_plus_2| # +-----+---+----------+----------+ # |Alice| 25| 26| 27| # | Bob| 30| 31| 32| # +-----+---+----------+----------+ ``` """ if not cols_map: return self exprs = [] new_col_names = set(cols_map.keys()) # Add existing columns that are not being replaced for field in self.columns: if field not in new_col_names: exprs.append(Column._from_column_name(field)._logical_expr) # Add all new columns with aliases for col_name, col_expr in cols_map.items(): # Handle different input types: Column, Series, or literal value if isinstance(col_expr, (pl.Series, pd.Series)): # Wrap Series in SeriesLiteralExpr and then in Column col_expr = Column._from_logical_expr(SeriesLiteralExpr(col_expr)) elif not isinstance(col_expr, Column): # Automatically wrap non-Column values (literals) with lit() for convenience # This allows users to pass raw Python values like: {"constant": 100, "status": "active"} # instead of requiring: {"constant": lit(100), "status": lit("active")} col_expr = lit(col_expr) exprs.append(col_expr.alias(col_name)._logical_expr) return self._from_logical_plan( Projection.from_session_state(self._logical_plan, exprs, self._session_state), self._session_state, ) ``` --- # fenic.api.dataframe.grouped_data Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/dataframe/grouped_data/ GroupedData class for aggregations on grouped DataFrames. Classes: - **`GroupedData`** – Methods for aggregations on a grouped DataFrame. ## GroupedData ``` GroupedData(df: DataFrame, by: Optional[List[ColumnOrName]] = None) ``` Bases: `BaseGroupedData` ``` flowchart TD fenic.api.dataframe.grouped_data.GroupedData[GroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData[BaseGroupedData] fenic.api.dataframe._base_grouped_data.BaseGroupedData --> fenic.api.dataframe.grouped_data.GroupedData click fenic.api.dataframe.grouped_data.GroupedData href "" "fenic.api.dataframe.grouped_data.GroupedData" click fenic.api.dataframe._base_grouped_data.BaseGroupedData href "" "fenic.api.dataframe._base_grouped_data.BaseGroupedData" ``` Methods for aggregations on a grouped DataFrame. Initialize grouped data. Parameters: - **`df`** (`DataFrame`) – The DataFrame to group. - **`by`** (`Optional[List[ColumnOrName]]`, default: `None` ) – Optional list of columns to group by. Methods: - **`agg`** – Compute aggregations on grouped data and return the result as a DataFrame. Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def __init__(self, df: DataFrame, by: Optional[List[ColumnOrName]] = None): """Initialize grouped data. Args: df: The DataFrame to group. by: Optional list of columns to group by. """ super().__init__(df) self._by: List[Column] = [] for c in by or []: if isinstance(c, str): self._by.append(col(c)) elif isinstance(c, Column): # Allow any expression except literals if isinstance(c._logical_expr, LiteralExpr): raise ValueError(f"Cannot group by literal value: {c}") self._by.append(c) else: raise TypeError( f"Group by expressions must be string or Column, got {type(c)}" ) self._by_exprs = [c._logical_expr for c in self._by] ``` ### agg ``` agg(*exprs: Union[Column, Dict[str, str]]) -> DataFrame ``` Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Parameters: - **`*exprs`** (`Union[Column, Dict[str, str]]`, default: `()` ) – Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame with one row per group and columns for group keys and aggregated values Raises: - `ValueError` – If arguments are not Column expressions or a dictionary - `ValueError` – If dictionary values are not valid aggregate function names Count employees by department ``` # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Multiple aggregations ``` # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Dictionary style aggregations ``` # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` Source code in `src/fenic/api/dataframe/grouped_data.py` ``` def agg(self, *exprs: Union[Column, Dict[str, str]]) -> DataFrame: """Compute aggregations on grouped data and return the result as a DataFrame. This method applies aggregate functions to the grouped data. Args: *exprs: Aggregation expressions. Can be: - Column expressions with aggregate functions (e.g., `count("*")`, `sum("amount")`) - A dictionary mapping column names to aggregate function names (e.g., `{"amount": "sum", "age": "avg"}`) Returns: DataFrame: A new DataFrame with one row per group and columns for group keys and aggregated values Raises: ValueError: If arguments are not Column expressions or a dictionary ValueError: If dictionary values are not valid aggregate function names Example: Count employees by department ```python # Group by department and count employees df.group_by("department").agg(count("*").alias("employee_count")) ``` Example: Multiple aggregations ```python # Multiple aggregations df.group_by("department").agg( count("*").alias("employee_count"), avg("salary").alias("avg_salary"), max("age").alias("max_age") ) ``` Example: Dictionary style aggregations ```python # Dictionary style for simple aggregations df.group_by("department", "location").agg({"salary": "avg", "age": "max"}) ``` """ self._validate_agg_exprs(*exprs) if len(exprs) == 1 and isinstance(exprs[0], dict): agg_dict = exprs[0] return self.agg(*self._process_agg_dict(agg_dict)) agg_exprs = self._process_agg_exprs(exprs) return self._df._from_logical_plan( Aggregate.from_session_state(self._df._logical_plan, self._by_exprs, agg_exprs, self._df._session_state), self._df._session_state, ) ``` --- # fenic.api.dataframe.semantic_extensions Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/dataframe/semantic_extensions/ Semantic extensions for DataFrames providing clustering and semantic join operations. Classes: - **`SemanticExtensions`** – A namespace for semantic dataframe operators. ## SemanticExtensions ``` SemanticExtensions(df: DataFrame) ``` A namespace for semantic dataframe operators. Initialize semantic extensions. Parameters: - **`df`** (`DataFrame`) – The DataFrame to extend with semantic operations. Methods: - **`join`** – Performs a semantic join between two DataFrames using a natural language predicate. - **`sim_join`** – Performs a semantic similarity join between two DataFrames using embedding expressions. - **`with_cluster_labels`** – Cluster rows using K-means and add cluster metadata columns. Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def __init__(self, df: DataFrame): """Initialize semantic extensions. Args: df: The DataFrame to extend with semantic operations. """ self._df = df ``` ### join ``` join(other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Parameters: - **`other`** (`DataFrame`) – The DataFrame to join with. - **`predicate`** (`str`) – A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. - **`left_on`** (`Column`) – The column from the left DataFrame (self) to use in the join predicate. - **`right_on`** (`Column`) – The column from the right DataFrame (other) to use in the join predicate. - **`strict`** (`bool`, default: `True` ) – If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[JoinExampleCollection]`, default: `None` ) – Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use. If None, uses the default model. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`DataFrame`** ( `DataFrame` ) – A new DataFrame containing matched row pairs with all columns from both DataFrames. Basic semantic join ``` # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Semantic join with examples ``` # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent(''' Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def join( self, other: DataFrame, predicate: str, left_on: Column, right_on: Column, strict: bool = True, examples: Optional[JoinExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic join between two DataFrames using a natural language predicate. This method evaluates a boolean predicate for each potential row pair between the two DataFrames, including only those pairs where the predicate evaluates to True. The join process: 1. For each row in the left DataFrame, evaluates the predicate in the jinja template against each row in the right DataFrame 2. Includes row pairs where the predicate returns True 3. Excludes row pairs where the predicate returns False 4. Returns a new DataFrame containing all columns from both DataFrames for the matched pairs The jinja template must use exactly two column placeholders: - One from the left DataFrame: `{{ left_on }}` - One from the right DataFrame: `{{ right_on }}` Args: other: The DataFrame to join with. predicate: A Jinja2 template containing the natural language predicate. Must include placeholders for exactly one column from each DataFrame. The template is evaluated as a boolean - True includes the pair, False excludes it. left_on: The column from the left DataFrame (self) to use in the join predicate. right_on: The column from the right DataFrame (other) to use in the join predicate. strict: If True, when either the left_on or right_on column has a None value for a row pair, that pair is automatically excluded from the join (predicate is not evaluated). If False, None values are rendered according to Jinja2's null rendering behavior. Default is True. examples: Optional JoinExampleCollection containing labeled examples to guide the join. Each example should have: - left: Sample value from the left column - right: Sample value from the right column - output: Boolean indicating whether this pair should be joined (True) or not (False) model_alias: Optional alias for the language model to use. If None, uses the default model. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: DataFrame: A new DataFrame containing matched row pairs with all columns from both DataFrames. Example: Basic semantic join ```python # Match job listings with candidate resumes based on title/skills # Only includes pairs where the predicate evaluates to True df_jobs.semantic.join(df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` Example: Semantic join with examples ```python # Improve join quality with examples examples = JoinExampleCollection() examples.create_example(JoinExample( left="5 years experience building backend services in Python using asyncio, FastAPI, and PostgreSQL", right="Senior Software Engineer - Backend", output=True)) # This pair WILL be included in similar cases examples.create_example(JoinExample( left="5 years experience with growth strategy, private equity due diligence, and M&A", right="Product Manager - Hardware", output=False)) # This pair will NOT be included in similar cases df_jobs.semantic.join( other=df_resumes, predicate=dedent('''\ Job Description: {{left_on}} Candidate Background: {{right_on}} The candidate is qualified for the job.'''), left_on=col("job_description"), right_on=col("work_experience"), examples=examples ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(other, DataFrame): raise ValidationError(f"other argument must be a DataFrame, got {type(other)}") if not isinstance(predicate, str): raise ValidationError( f"The `predicate` argument to `semantic.join` must be a string, got {type(predicate)}" ) if not isinstance(left_on, Column): raise ValidationError(f"`left_on` argument must be a Column, got {type(left_on)} instead.") if not isinstance(right_on, Column): raise ValidationError(f"`right_on` argument must be a Column, got {type(right_on)} instead.") if examples is not None and not isinstance(examples, JoinExampleCollection): raise ValidationError(f"`examples` argument must be a JoinExampleCollection, got {type(examples)} instead.") if model_alias is not None and not isinstance(model_alias, (str, ModelAlias)): raise ValidationError(f"`model_alias` argument must be a string or ModelAlias, got {type(model_alias)} instead.") # Validate request_timeout request_timeout = validate_timeout(request_timeout) resolved_model_alias = _resolve_model_alias(model_alias) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticJoin.from_session_state( left=self._df._logical_plan, right=other._logical_plan, left_on=left_on._logical_expr, right_on=right_on._logical_expr, jinja_template=predicate, strict=strict, model_alias=resolved_model_alias, examples=examples, request_timeout=request_timeout, session_state=self._df._session_state, ), self._df._session_state, ) ``` ### sim_join ``` sim_join(other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = 'cosine', similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Parameters: - **`other`** (`DataFrame`) – The right-hand DataFrame to join with. - **`left_on`** (`ColumnOrName`) – Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. - **`right_on`** (`ColumnOrName`) – Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. - **`k`** (`int`, default: `1` ) – Number of most similar matches to return per row. - **`similarity_metric`** (`SemanticSimilarityMetric`, default: `'cosine'` ) – Similarity metric to use: "l2", "cosine", or "dot". - **`similarity_score_column`** (`Optional[str]`, default: `None` ) – If set, adds a column with this name containing similarity scores. If None, the scores are omitted. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. - `DataFrame` – The result includes all columns from both input DataFrames and, when `similarity_score_column` - `DataFrame` – is provided, a similarity score column. Join key columns that already exist in the input - `DataFrame` – DataFrames are preserved as normal user columns. Join keys derived from expressions are - `DataFrame` – temporary execution columns and are not included in the result schema. Raises: - `ValidationError` – If `k` is not positive or if the columns are invalid. - `ValidationError` – If `similarity_metric` is not one of "l2", "cosine", "dot" Match queries to FAQ entries ``` # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Link headlines to articles ``` # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Find similar job postings ``` # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def sim_join( self, other: DataFrame, left_on: ColumnOrName, right_on: ColumnOrName, k: int = 1, similarity_metric: SemanticSimilarityMetric = "cosine", similarity_score_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Performs a semantic similarity join between two DataFrames using embedding expressions. For each row in the left DataFrame, returns the top `k` most semantically similar rows from the right DataFrame based on the specified similarity metric. Note: Local execution requires the `sim-join` extra: `pip install "fenic[sim-join]"`. Args: other: The right-hand DataFrame to join with. left_on: Expression or column representing embeddings in the left DataFrame. If this is a named column, that column is treated as a user column and is included in the output. If this is an expression, it is treated as a temporary join key and is not included as an output column. right_on: Expression or column representing embeddings in the right DataFrame. Named columns are included in the output; expression-derived join keys are temporary and are not included in the output. k: Number of most similar matches to return per row. similarity_metric: Similarity metric to use: "l2", "cosine", or "dot". similarity_score_column: If set, adds a column with this name containing similarity scores. If None, the scores are omitted. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame containing one row for each of the top-k matches per row in the left DataFrame. The result includes all columns from both input DataFrames and, when `similarity_score_column` is provided, a similarity score column. Join key columns that already exist in the input DataFrames are preserved as normal user columns. Join keys derived from expressions are temporary execution columns and are not included in the result schema. Raises: ValidationError: If `k` is not positive or if the columns are invalid. ValidationError: If `similarity_metric` is not one of "l2", "cosine", "dot" Example: Match queries to FAQ entries ```python # Match customer queries to FAQ entries df_queries.semantic.sim_join( df_faqs, left_on=embeddings(col("query_text")), right_on=embeddings(col("faq_question")), k=1 ) ``` Example: Link headlines to articles ```python # Link news headlines to full articles df_headlines.semantic.sim_join( df_articles, left_on=embeddings(col("headline")), right_on=embeddings(col("content")), k=3, return_similarity_scores=True ) ``` Example: Find similar job postings ```python # Find similar job postings across two sources df_linkedin.semantic.sim_join( df_indeed, left_on=embeddings(col("job_title")), right_on=embeddings(col("job_description")), k=2 ) ``` """ from fenic.api.dataframe.dataframe import DataFrame if not isinstance(right_on, ColumnOrName): raise ValidationError( f"The `right_on` argument must be a `Column` or a string representing a column name, " f"but got `{type(right_on).__name__}` instead." ) if not isinstance(other, DataFrame): raise ValidationError( f"The `other` argument to `sim_join()` must be a DataFrame`, but got `{type(other).__name__}`." ) if not (isinstance(k, int) and k > 0): raise ValidationError( f"The parameter `k` must be a positive integer, but received `{k}`." ) args = get_args(SemanticSimilarityMetric) if similarity_metric not in args: raise ValidationError( f"The `similarity_metric` argument must be one of {args}, but got `{similarity_metric}`." ) def _validate_column(column: ColumnOrName, name: str): if column is None: raise ValidationError(f"The `{name}` argument must not be None.") if not isinstance(column, ColumnOrName): raise ValidationError( f"The `{name}` argument must be a `Column` or a string representing a column name, " f"but got `{type(column).__name__}` instead." ) _validate_column(left_on, "left_on") _validate_column(right_on, "right_on") # Validate request_timeout request_timeout = validate_timeout(request_timeout) DataFrame._ensure_same_session(self._df._session_state, [other._session_state]) return self._df._from_logical_plan( SemanticSimilarityJoin.from_session_state( self._df._logical_plan, other._logical_plan, Column._from_col_or_name(left_on)._logical_expr, Column._from_col_or_name(right_on)._logical_expr, k, similarity_metric=similarity_metric, similarity_score_column=similarity_score_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` ### with_cluster_labels ``` with_cluster_labels(by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = 'cluster_label', centroid_column: Optional[str] = None, request_timeout: Optional[float] = None) -> DataFrame ``` Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Parameters: - **`by`** (`ColumnOrName`) – Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). - **`num_clusters`** (`int`) – Number of clusters to compute (must be > 0). - **`max_iter`** (`int`, default: `300` ) – Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. - **`num_init`** (`int`, default: `1` ) – Number of independent runs of k-means with different centroid seeds. The best result is selected. - **`label_column`** (`str`, default: `'cluster_label'` ) – Name of the output column for cluster IDs. Default is "cluster_label". - **`centroid_column`** (`Optional[str]`, default: `None` ) – If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. - **`request_timeout`** (`Optional[float]`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - `DataFrame` – A DataFrame with all original columns plus: - `DataFrame` – - ``: integer cluster assignment (0 to num_clusters - 1) - `DataFrame` – - ``: cluster centroid embedding, if specified Basic clustering ``` # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Filter outliers using centroids ``` # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` Source code in `src/fenic/api/dataframe/semantic_extensions.py` ``` def with_cluster_labels( self, by: ColumnOrName, num_clusters: int, max_iter: int = 300, num_init: int = 1, label_column: str = "cluster_label", centroid_column: Optional[str] = None, request_timeout: Optional[float] = None, ) -> DataFrame: """Cluster rows using K-means and add cluster metadata columns. This method clusters rows based on the given embedding column or expression using K-means. It adds a new column with cluster assignments, and optionally includes the centroid embedding for each assigned cluster. Note: Local execution requires the `cluster` extra: `pip install "fenic[cluster]"`. Args: by: Column or expression producing embeddings to cluster (e.g., `embed(col("text"))`). num_clusters: Number of clusters to compute (must be > 0). max_iter: Maximum iterations for a single run of the k-means algorithm. The algorithm stops when it either converges or reaches this limit. num_init: Number of independent runs of k-means with different centroid seeds. The best result is selected. label_column: Name of the output column for cluster IDs. Default is "cluster_label". centroid_column: If provided, adds a column with this name containing the centroid embedding for each row's assigned cluster. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: A DataFrame with all original columns plus: - ``: integer cluster assignment (0 to num_clusters - 1) - ``: cluster centroid embedding, if specified Example: Basic clustering ```python # Cluster customer feedback and add cluster metadata clustered_df = df.semantic.with_cluster_labels("feedback_embeddings", num_clusters=5) # Then use regular operations to analyze clusters clustered_df.group_by("cluster_label").agg(count("*"), avg("rating")) ``` Example: Filter outliers using centroids ```python # Cluster and filter out rows far from their centroid clustered_df = df.semantic.with_cluster_labels("embeddings", num_clusters=3, num_init=10, centroid_column="cluster_centroid") clean_df = clustered_df.filter( embedding.compute_similarity("embeddings", "cluster_centroid", metric="cosine") > 0.7 ) ``` """ # Validate request_timeout request_timeout = validate_timeout(request_timeout) # Validate num_clusters if not isinstance(num_clusters, int) or num_clusters <= 0: raise ValidationError("`num_clusters` must be a positive integer.") # Validate max_iter if not isinstance(max_iter, int) or max_iter <= 0: raise ValidationError("`max_iter` must be a positive integer.") # Validate num_init if not isinstance(num_init, int) or num_init <= 0: raise ValidationError("`num_init` must be a positive integer.") # Validate clustering target if not isinstance(by, ColumnOrName): raise ValidationError( f"Invalid cluster by: expected a column name (str) or Column object, got {type(by).__name__}." ) # Validate label_column if not isinstance(label_column, str) or not label_column: raise ValidationError("`label_column` must be a non-empty string.") # Validate centroid_column if provided if centroid_column is not None: if not isinstance(centroid_column, str) or not centroid_column: raise ValidationError("`centroid_column` must be a non-empty string if provided.") # Check that the expression isn't a literal by_expr = Column._from_col_or_name(by)._logical_expr if isinstance(by_expr, LiteralExpr): raise ValidationError( f"Invalid cluster by: Cannot cluster by a literal value: {by_expr}." ) return self._df._from_logical_plan( SemanticCluster.from_session_state( self._df._logical_plan, by_expr, num_clusters=num_clusters, max_iter=max_iter, num_init=num_init, label_column=label_column, centroid_column=centroid_column, session_state=self._df._session_state, request_timeout=request_timeout, ), self._df._session_state, ) ``` --- # fenic.api.functions Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/ Functions for working with DataFrame columns. Note: Array functions are available via fc.arr.\* namespace (e.g., fc.arr.size()). Functions: - **`approx_count_distinct`** – Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. - **`array_agg`** – Alias for collect_list(). - **`asc`** – Mark this column for ascending sort order with nulls first. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`async_udf`** – A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. - **`avg`** – Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. - **`coalesce`** – Returns the first non-null value from the given columns for each row. - **`col`** – Creates a Column expression referencing a column in the DataFrame. - **`collect_list`** – Aggregate function: collects all values from the specified column into a list. - **`count`** – Aggregate function: returns the count of non-null values in the specified column. - **`count_distinct`** – Aggregate function: returns the number of distinct non-null rows across one or more columns. - **`desc`** – Mark this column for descending sort order with nulls first. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`empty`** – Creates a Column expression representing an empty value of the given type. - **`first`** – Aggregate function: returns the first non-null value in the specified column. - **`flatten`** – Flattens an array of arrays into a single array (one level deep). - **`greatest`** – Returns the greatest value from the given columns for each row. - **`least`** – Returns the least value from the given columns for each row. - **`lit`** – Creates a Column expression representing a literal value. - **`max`** – Aggregate function: returns the maximum value in the specified column. - **`mean`** – Aggregate function: returns the mean (average) of all values in the specified column. - **`min`** – Aggregate function: returns the minimum value in the specified column. - **`null`** – Creates a Column expression representing a null value of the specified data type. - **`stddev`** – Aggregate function: returns the sample standard deviation of the specified column. - **`struct`** – Creates a new struct column from multiple input columns. - **`sum`** – Aggregate function: returns the sum of all values in the specified column. - **`sum_distinct`** – Aggregate function: returns the sum of distinct numeric values in the specified column. - **`tool_param`** – Creates an unresolved literal placeholder column with a declared data type. - **`udf`** – A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. - **`when`** – Evaluates a conditional expression (like if-then). ## approx_count_distinct ``` approx_count_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: - `Column` – A Column expression representing the approximate count-distinct aggregation Note Differs from the pyspark implementation in that the relative standard deviation is not configurable. Approximate distinct count per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Nulls are ignored in approximate distinct counts ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: - `TypeMismatchError` – If column is a StructType or ArrayType column. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def approx_count_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Args: column: Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: A Column expression representing the approximate count-distinct aggregation Note: Differs from the pyspark implementation in that the relative standard deviation is not configurable. Example: Approximate distinct count per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Example: Nulls are ignored in approximate distinct counts ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: TypeMismatchError: If column is a StructType or ArrayType column. """ return Column._from_logical_expr( ApproxCountDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## array_agg ``` array_agg(column: ColumnOrName) -> Column ``` Alias for collect_list(). Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array_agg(column: ColumnOrName) -> Column: """Alias for collect_list().""" return collect_list(column) ``` ## asc ``` asc(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls first. Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc() ``` ## asc_nulls_first ``` asc_nulls_first(column: ColumnOrName) -> Column ``` Alias for asc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_first(column: ColumnOrName) -> Column: """Alias for asc(). Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc_nulls_first() ``` ## asc_nulls_last ``` asc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A Column expression representing the column and the ascending sort order with nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls last. Args: column: The column to apply the ascending ordering to. Returns: A Column expression representing the column and the ascending sort order with nulls last. """ return Column._from_col_or_name(column).asc_nulls_last() ``` ## async_udf ``` async_udf(f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0) ``` A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Parameters: - **`f`** (`Optional[Callable[..., Awaitable[Any]]]`, default: `None` ) – Async function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. - **`max_concurrency`** (`int`, default: `10` ) – Maximum number of concurrent executions (default: 10) - **`timeout_seconds`** (`float`, default: `30` ) – Per-item timeout in seconds (default: 30) - **`num_retries`** (`int`, default: `0` ) – Number of retries for failed items (default: 0) Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) ### Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries `python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() }` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def async_udf( f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0, ): """A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Args: f: Async function to convert to UDF return_type: Expected return type of the UDF. Required parameter. max_concurrency: Maximum number of concurrent executions (default: 10) timeout_seconds: Per-item timeout in seconds (default: 30) num_retries: Number of retries for failed items (default: 0) Example: Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) # Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries ```python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() } ``` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. """ def _create_async_udf(func: Callable[..., Awaitable[Any]]) -> Callable: if not inspect.iscoroutinefunction(func): raise ValidationError( f"@async_udf requires an async function, but found a synchronous " f"function {func.__name__!r} of type {type(func)}" ) @wraps(func) def _async_udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr( AsyncUDFExpr( func, col_exprs, return_type, max_concurrency=max_concurrency, timeout_seconds=timeout_seconds, num_retries=num_retries ) ) return _async_udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for async UDFs") # Support both @async_udf and async_udf(...) syntax if f is None: return _create_async_udf else: return _create_async_udf(f) ``` ## avg ``` avg(column: ColumnOrName) -> Column ``` Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the average of Returns: - `Column` – A Column expression representing the average aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def avg(column: ColumnOrName) -> Column: """Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Args: column: Column or column name to compute the average of Returns: A Column expression representing the average aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## coalesce ``` coalesce(*cols: ColumnOrName) -> Column ``` Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the first non-null value from the input columns. Raises: - `ValidationError` – If no columns are provided. coalesce usage ``` df.select(coalesce("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def coalesce(*cols: ColumnOrName) -> Column: """Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the first non-null value from the input columns. Raises: ValidationError: If no columns are provided. Example: coalesce usage ```python df.select(coalesce("col1", "col2", "col3")) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the coalesce method.") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(CoalesceExpr(exprs)) ``` ## col ``` col(col_name: str) -> Column ``` Creates a Column expression referencing a column in the DataFrame. Parameters: - **`col_name`** (`str`) – Name of the column to reference Returns: - `Column` – A Column expression for the specified column Raises: - `TypeError` – If colName is not a string Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True)) def col(col_name: str) -> Column: """Creates a Column expression referencing a column in the DataFrame. Args: col_name: Name of the column to reference Returns: A Column expression for the specified column Raises: TypeError: If colName is not a string """ return Column._from_column_name(col_name) ``` ## collect_list ``` collect_list(column: ColumnOrName) -> Column ``` Aggregate function: collects all values from the specified column into a list. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to collect values from Returns: - `Column` – A Column expression representing the list aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def collect_list(column: ColumnOrName) -> Column: """Aggregate function: collects all values from the specified column into a list. Args: column: Column or column name to collect values from Returns: A Column expression representing the list aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( ListExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count ``` count(column: ColumnOrName) -> Column ``` Aggregate function: returns the count of non-null values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to count values in Returns: - `Column` – A Column expression representing the count aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count(column: ColumnOrName) -> Column: """Aggregate function: returns the count of non-null values in the specified column. Args: column: Column or column name to count values in Returns: A Column expression representing the count aggregation Raises: TypeError: If column is not a Column or string """ if isinstance(column, str) and column == "*": return Column._from_logical_expr(CountExpr(lit("*")._logical_expr)) return Column._from_logical_expr( CountExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count_distinct ``` count_distinct(*cols: ColumnOrName) -> Column ``` Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – One or more columns or column names to include in the distinct count. Returns: - `Column` – A Column expression representing the count-distinct aggregation over the provided columns. Distinct count per group (single column) ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Distinct count across multiple columns (whole DataFrame) ``` # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Nulls in any input column are ignored for multi-column distinct ``` df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: - `ValidationError` – If no columns are provided. - `TypeMismatchError` – If a column has an unsupported type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count_distinct(*cols: ColumnOrName) -> Column: """Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Args: *cols: One or more columns or column names to include in the distinct count. Returns: A Column expression representing the count-distinct aggregation over the provided columns. Example: Distinct count per group (single column) ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Example: Distinct count across multiple columns (whole DataFrame) ```python # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Example: Nulls in any input column are ignored for multi-column distinct ```python df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: ValidationError: If no columns are provided. TypeMismatchError: If a column has an unsupported type """ if not cols: raise ValidationError("count_distinct requires at least one column") exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(CountDistinctExpr(exprs)) ``` ## desc ``` desc(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls first. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc() ``` ## desc_nulls_first ``` desc_nulls_first(column: ColumnOrName) -> Column ``` Alias for desc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_first(column: ColumnOrName) -> Column: """Alias for desc(). Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc_nulls_first() ``` ## desc_nulls_last ``` desc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls last. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls last. """ return Column._from_col_or_name(column).desc_nulls_last() ``` ## empty ``` empty(data_type: DataType) -> Column ``` Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the empty value Returns: - `Column` – A Column expression representing the empty value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with an empty array type ``` # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Creating a column with an empty struct type ``` # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Creating a column with an empty primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` Source code in `src/fenic/api/functions/core.py` ``` def empty(data_type: DataType) -> Column: """Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Args: data_type: The data type of the empty value Returns: A Column expression representing the empty value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with an empty array type ```python # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Example: Creating a column with an empty struct type ```python # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Example: Creating a column with an empty primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` """ if isinstance(data_type, ArrayType): return Column._from_logical_expr(LiteralExpr([], data_type)) elif isinstance(data_type, StructType): return Column._from_logical_expr(LiteralExpr({}, data_type)) return null(data_type) ``` ## first ``` first(column: ColumnOrName) -> Column ``` Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for the first value. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def first(column: ColumnOrName) -> Column: """Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Args: column: Column or column name. Returns: Column expression for the first value. """ return Column._from_logical_expr( FirstExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## flatten ``` flatten(column: ColumnOrName) -> Column ``` Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of arrays. Returns: - `Column` – A Column with flattened arrays (one level deep). Flattening nested arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` One level only ``` # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def flatten(column: ColumnOrName) -> Column: """Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Args: column: Column or column name containing arrays of arrays. Returns: A Column with flattened arrays (one level deep). Example: Flattening nested arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: One level only ```python # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` """ return Column._from_logical_expr( FlattenExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## greatest ``` greatest(*cols: ColumnOrName) -> Column ``` Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the greatest value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. greatest usage ``` df.select(fc.greatest("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def greatest(*cols: ColumnOrName) -> Column: """Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the greatest value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: greatest usage ```python df.select(fc.greatest("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"greatest() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(GreatestExpr(exprs)) ``` ## least ``` least(*cols: ColumnOrName) -> Column ``` Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the least value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. least usage ``` df.select(fc.least("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def least(*cols: ColumnOrName) -> Column: """Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the least value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: least usage ```python df.select(fc.least("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"least() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(LeastExpr(exprs)) ``` ## lit ``` lit(value: Any) -> Column ``` Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Parameters: - **`value`** (`Any`) – The literal value to create a column for Returns: - `Column` – A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred Source code in `src/fenic/api/functions/core.py` ``` def lit(value: Any) -> Column: """Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value: - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Args: value: The literal value to create a column for Returns: A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred """ if value is None: raise ValidationError("Cannot create a literal with value `None`. Use `null(...)` instead.") elif value == []: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(ArrayType(...))` instead.") elif value == {}: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(StructType(...))` instead.") try: inferred_type = infer_dtype_from_pyobj(value) except TypeInferenceError as e: raise ValidationError(f"`lit` failed to infer type for value `{value}`") from e literal_expr = LiteralExpr(value, inferred_type) return Column._from_logical_expr(literal_expr) ``` ## max ``` max(column: ColumnOrName) -> Column ``` Aggregate function: returns the maximum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the maximum of Returns: - `Column` – A Column expression representing the maximum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def max(column: ColumnOrName) -> Column: """Aggregate function: returns the maximum value in the specified column. Args: column: Column or column name to compute the maximum of Returns: A Column expression representing the maximum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MaxExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## mean ``` mean(column: ColumnOrName) -> Column ``` Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the mean of Returns: - `Column` – A Column expression representing the mean aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def mean(column: ColumnOrName) -> Column: """Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Args: column: Column or column name to compute the mean of Returns: A Column expression representing the mean aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## min ``` min(column: ColumnOrName) -> Column ``` Aggregate function: returns the minimum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the minimum of Returns: - `Column` – A Column expression representing the minimum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def min(column: ColumnOrName) -> Column: """Aggregate function: returns the minimum value in the specified column. Args: column: Column or column name to compute the minimum of Returns: A Column expression representing the minimum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MinExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## null ``` null(data_type: DataType) -> Column ``` Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the null value Returns: - `Column` – A Column expression representing the null value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with a null value of a primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Creating a column with a null value of an array/struct type ``` # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` Source code in `src/fenic/api/functions/core.py` ``` def null(data_type: DataType) -> Column: """Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Args: data_type: The data type of the null value Returns: A Column expression representing the null value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with a null value of a primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Example: Creating a column with a null value of an array/struct type ```python # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` """ return Column._from_logical_expr(LiteralExpr(None, data_type)) ``` ## stddev ``` stddev(column: ColumnOrName) -> Column ``` Aggregate function: returns the sample standard deviation of the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for sample standard deviation. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def stddev(column: ColumnOrName) -> Column: """Aggregate function: returns the sample standard deviation of the specified column. Args: column: Column or column name. Returns: Column expression for sample standard deviation. """ return Column._from_logical_expr( StdDevExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## struct ``` struct(*args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]) -> Column ``` Creates a new struct column from multiple input columns. Parameters: - **`*args`** (`Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]`, default: `()` ) – Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: - `Column` – A Column expression representing a struct containing the input columns Raises: - `TypeError` – If any argument is not a Column, string, or collection of Columns/strings Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def struct( *args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]] ) -> Column: """Creates a new struct column from multiple input columns. Args: *args: Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: A Column expression representing a struct containing the input columns Raises: TypeError: If any argument is not a Column, string, or collection of Columns/strings """ flattened_args = [] for arg in args: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_columns = [Column._from_col_or_name(c)._logical_expr for c in flattened_args] return Column._from_logical_expr(StructExpr(expr_columns)) ``` ## sum ``` sum(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of all values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of Returns: - `Column` – A Column expression representing the sum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of all values in the specified column. Args: column: Column or column name to compute the sum of Returns: A Column expression representing the sum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( SumExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## sum_distinct ``` sum_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of distinct numeric values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of distinct values Returns: - `Column` – A Column expression representing the sum-distinct aggregation Sum of distinct values per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Nulls are ignored when summing distinct values ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: - `TypeMismatchError` – If column is not a numeric or boolean type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of distinct numeric values in the specified column. Args: column: Column or column name to compute the sum of distinct values Returns: A Column expression representing the sum-distinct aggregation Example: Sum of distinct values per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Example: Nulls are ignored when summing distinct values ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: TypeMismatchError: If column is not a numeric or boolean type """ return Column._from_logical_expr( SumDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## tool_param ``` tool_param(parameter_name: str, data_type: DataType) -> Column ``` Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Parameters: - **`parameter_name`** (`str`) – The name of the parameter to reference. - **`data_type`** (`DataType`) – The expected data type for the parameter value. Returns: - `Column` – A Column wrapping an UnresolvedLiteralExpr for the given parameter. A simple tool with one parameter ```python ### Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) A tool with multiple filters ```python ### Assume we are reading data with an `age` column. df = session.read.csv(users.csv) ### create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def tool_param(parameter_name: str, data_type: DataType) -> Column: """Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes: Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Args: parameter_name: The name of the parameter to reference. data_type: The expected data type for the parameter value. Returns: A Column wrapping an UnresolvedLiteralExpr for the given parameter. Example: A simple tool with one parameter ```python # Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) Example: A tool with multiple filters ```python # Assume we are reading data with an `age` column. df = session.read.csv(users.csv) # create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) """ if isinstance(data_type, _LogicalType): raise ValidationError(f"Cannot use a logical type as a parameter type: {data_type}") return Column._from_logical_expr(UnresolvedLiteralExpr(data_type=data_type, parameter_name=parameter_name)) ``` ## udf ``` udf(f: Optional[Callable] = None, *, return_type: DataType) ``` A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Parameters: - **`f`** (`Optional[Callable]`, default: `None` ) – Python function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. UDF with primitive types ``` # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` UDF with nested types ``` # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def udf(f: Optional[Callable] = None, *, return_type: DataType): """A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning: UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Args: f: Python function to convert to UDF return_type: Expected return type of the UDF. Required parameter. Example: UDF with primitive types ```python # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` Example: UDF with nested types ```python # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` """ def _create_udf(func: Callable) -> Callable: @wraps(func) def _udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(UDFExpr(func, col_exprs, return_type)) return _udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for UDFs") if f is not None: return _create_udf(f) return _create_udf ``` ## when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A when expression that can be chained with more conditions Raises: - `TypeMismatchError` – If the condition is not a boolean Column expression. Example ``` # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def when(condition: Column, value: Column) -> Column: """Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A when expression that can be chained with more conditions Raises: TypeMismatchError: If the condition is not a boolean Column expression. Example: ```python # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null """ return Column._from_logical_expr( WhenExpr(None, condition._logical_expr, value._logical_expr) ) ``` --- # fenic.api.functions.array Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/array/ Array functions for Fenic DataFrames. This module provides array manipulation functions following PySpark conventions. Functions are available via fc.arr.\* namespace (e.g., fc.arr.size()). Functions: - **`compact`** – Removes null values from an array. - **`contains`** – Checks if array column contains a specific value. - **`distinct`** – Removes duplicate values from an array column. - **`element_at`** – Returns the element at the given index in an array using 1-based indexing (PySpark compatible). - **`except_`** – Returns elements in the first array but not in the second. - **`intersect`** – Returns the intersection of two arrays. - **`max`** – Returns the maximum value in an array. - **`min`** – Returns the minimum value in an array. - **`overlap`** – Checks if two arrays have at least one common element. - **`remove`** – Removes all occurrences of an element from an array. - **`repeat`** – Creates an array containing the element repeated count times. - **`reverse`** – Reverses the elements of an array. - **`size`** – Returns the number of elements in an array column. - **`slice`** – Extracts a subarray from an array using 1-based indexing (PySpark compatible). - **`sort`** – Sorts the array in ascending order. - **`union`** – Returns the union of two arrays without duplicates. ## compact ``` compact(column: ColumnOrName) -> Column ``` Removes null values from an array. Returns a new array with all null values removed. Returns null if the input array itself is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. Returns: - `Column` – A Column with arrays having null values removed. Removing nulls from arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "values": [[1, None, 2, None, 3], ["a", None, "b"], None] }) result = df.select(fc.array.compact("values").alias("compact")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ compact β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3] β”‚ # β”‚ ["a", "b"]β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` All nulls removed ``` df = fc.Session.local().create_dataframe({ "sparse": [[None, None, 1], [None]] }) result = df.select(fc.array.compact("sparse")) # Output: [[1], []] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def compact(column: ColumnOrName) -> Column: """Removes null values from an array. Returns a new array with all null values removed. Returns null if the input array itself is null. Args: column: Column or column name containing arrays. Returns: A Column with arrays having null values removed. Example: Removing nulls from arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "values": [[1, None, 2, None, 3], ["a", None, "b"], None] }) result = df.select(fc.array.compact("values").alias("compact")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ compact β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3] β”‚ # β”‚ ["a", "b"]β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: All nulls removed ```python df = fc.Session.local().create_dataframe({ "sparse": [[None, None, 1], [None]] }) result = df.select(fc.array.compact("sparse")) # Output: [[1], []] ``` """ return Column._from_logical_expr( ArrayCompactExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## contains ``` contains(column: ColumnOrName, value: Union[str, int, float, bool, Column]) -> Column ``` Checks if array column contains a specific value. This function returns True if the array in the specified column contains the given value, and False otherwise. Returns False if the array is None. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing the arrays to check. - **`value`** (`Union[str, int, float, bool, Column]`) – Value to search for in the arrays. Can be: - A literal value (string, number, boolean) - A Column expression Returns: - `Column` – A boolean Column expression (True if value is found, False otherwise). Raises: - `TypeError` – If value type is incompatible with the array element type. - `TypeError` – If the column does not contain array data. Check for values in arrays ``` # Check if 'python' exists in arrays in the 'tags' column df.select(fc.arr.contains("tags", "python")) # Check using a value from another column df.select(fc.arr.contains("tags", fc.col("search_term"))) ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def contains( column: ColumnOrName, value: Union[str, int, float, bool, Column] ) -> Column: """Checks if array column contains a specific value. This function returns True if the array in the specified column contains the given value, and False otherwise. Returns False if the array is None. Args: column: Column or column name containing the arrays to check. value: Value to search for in the arrays. Can be: - A literal value (string, number, boolean) - A Column expression Returns: A boolean Column expression (True if value is found, False otherwise). Raises: TypeError: If value type is incompatible with the array element type. TypeError: If the column does not contain array data. Example: Check for values in arrays ```python # Check if 'python' exists in arrays in the 'tags' column df.select(fc.arr.contains("tags", "python")) # Check using a value from another column df.select(fc.arr.contains("tags", fc.col("search_term"))) ``` """ value_column = None if isinstance(value, Column): value_column = value else: value_column = lit(value) return Column._from_logical_expr( ArrayContainsExpr( Column._from_col_or_name(column)._logical_expr, value_column._logical_expr ) ) ``` ## distinct ``` distinct(column: ColumnOrName) -> Column ``` Removes duplicate values from an array column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. Returns: - `Column` – A new column that is an array of unique values from the input column. Notes - Will attempt to preserve order of first appearances, but order is not guaranteed. Example ``` # create a dataframe with an array column with duplicates df = session.create_dataframe({ "array_col": [[1, 2, 2, 3], [4, 4, 4], None] }) # remove duplicates df.select(fc.arr.distinct("array_col").alias("distinct_array")) # Output: # +--------------------+ # | distinct_array | # +--------------------+ # | [1, 2, 3] | # | [4] | # | None | # +--------------------+ ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def distinct(column: ColumnOrName) -> Column: """Removes duplicate values from an array column. Args: column: Column or column name containing arrays. Returns: A new column that is an array of unique values from the input column. Notes: - Will attempt to preserve order of first appearances, but order is not guaranteed. Example: ```python # create a dataframe with an array column with duplicates df = session.create_dataframe({ "array_col": [[1, 2, 2, 3], [4, 4, 4], None] }) # remove duplicates df.select(fc.arr.distinct("array_col").alias("distinct_array")) # Output: # +--------------------+ # | distinct_array | # +--------------------+ # | [1, 2, 3] | # | [4] | # | None | # +--------------------+ ``` """ return Column._from_logical_expr( ArrayDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## element_at ``` element_at(column: ColumnOrName, index: Union[int, ColumnOrName]) -> Column ``` Returns the element at the given index in an array using 1-based indexing (PySpark compatible). Uses 1-based indexing for compatibility with PySpark. Returns null if the index is out of bounds or if the input array is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. - **`index`** (`Union[int, ColumnOrName]`) – Index of the element (1-based). Positive indices count from the start (1 = first element), negative indices count from the end (-1 = last element). Can be an integer literal or a Column expression. Returns: - `Column` – A Column containing the element at the specified index. Accessing with positive indices ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[10, 20, 30, 40], [100, 200]] }) result = df.select( fc.array.element_at("numbers", 1).alias("first"), fc.array.element_at("numbers", 2).alias("second") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ first β”‚ second β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 10 β”‚ 20 β”‚ # β”‚ 100 β”‚ 200 β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Accessing with negative indices ``` df = fc.Session.local().create_dataframe({ "arr": [["a", "b", "c", "d"], ["x", "y", "z"]] }) result = df.select( fc.array.element_at("arr", -1).alias("last"), fc.array.element_at("arr", -2).alias("second_last") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ last β”‚ second_last β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ "d" β”‚ "c" β”‚ # β”‚ "z" β”‚ "y" β”‚ # β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Dynamic indexing with columns ``` df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3], [10, 20, 30]], "position": [2, 3] }) result = df.select(fc.array.element_at("values", fc.col("position"))) # Output: [2, 30] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def element_at(column: ColumnOrName, index: Union[int, ColumnOrName]) -> Column: """Returns the element at the given index in an array using 1-based indexing (PySpark compatible). Uses 1-based indexing for compatibility with PySpark. Returns null if the index is out of bounds or if the input array is null. Args: column: Column or column name containing arrays. index: Index of the element (1-based). Positive indices count from the start (1 = first element), negative indices count from the end (-1 = last element). Can be an integer literal or a Column expression. Returns: A Column containing the element at the specified index. Example: Accessing with positive indices ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[10, 20, 30, 40], [100, 200]] }) result = df.select( fc.array.element_at("numbers", 1).alias("first"), fc.array.element_at("numbers", 2).alias("second") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ first β”‚ second β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 10 β”‚ 20 β”‚ # β”‚ 100 β”‚ 200 β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Accessing with negative indices ```python df = fc.Session.local().create_dataframe({ "arr": [["a", "b", "c", "d"], ["x", "y", "z"]] }) result = df.select( fc.array.element_at("arr", -1).alias("last"), fc.array.element_at("arr", -2).alias("second_last") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ last β”‚ second_last β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ "d" β”‚ "c" β”‚ # β”‚ "z" β”‚ "y" β”‚ # β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Dynamic indexing with columns ```python df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3], [10, 20, 30]], "position": [2, 3] }) result = df.select(fc.array.element_at("values", fc.col("position"))) # Output: [2, 30] ``` """ index_column = index if isinstance(index, Column) else lit(index) return Column._from_logical_expr( ElementAtExpr( Column._from_col_or_name(column)._logical_expr, index_column._logical_expr ) ) ``` ## except_ ``` except_(col1: ColumnOrName, col2: ColumnOrName) -> Column ``` Returns elements in the first array but not in the second. Returns distinct elements from the first array that are not present in the second array (set difference). Returns null if either input array is null. Parameters: - **`col1`** (`ColumnOrName`) – First array column or column name. - **`col2`** (`ColumnOrName`) – Second array column or column name. Returns: - `Column` – A Column containing distinct elements in col1 but not in col2. Filtering out deprecated tags ``` import fenic as fc df = fc.Session.local().create_dataframe({ "all_tags": [["a", "b", "c", "d"], ["x", "y", "z"]], "deprecated": [["b", "d"], ["y"]] }) result = df.select(fc.array.except_("all_tags", "deprecated").alias("active")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ active β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["a", "c"] β”‚ # β”‚ ["x", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` No common elements ``` df = fc.Session.local().create_dataframe({ "arr1": [[1, 2, 3]], "arr2": [[4, 5, 6]] }) result = df.select(fc.array.except_("arr1", "arr2")) # Output: [[1, 2, 3]] # All elements retained ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def except_(col1: ColumnOrName, col2: ColumnOrName) -> Column: """Returns elements in the first array but not in the second. Returns distinct elements from the first array that are not present in the second array (set difference). Returns null if either input array is null. Args: col1: First array column or column name. col2: Second array column or column name. Returns: A Column containing distinct elements in col1 but not in col2. Example: Filtering out deprecated tags ```python import fenic as fc df = fc.Session.local().create_dataframe({ "all_tags": [["a", "b", "c", "d"], ["x", "y", "z"]], "deprecated": [["b", "d"], ["y"]] }) result = df.select(fc.array.except_("all_tags", "deprecated").alias("active")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ active β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["a", "c"] β”‚ # β”‚ ["x", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: No common elements ```python df = fc.Session.local().create_dataframe({ "arr1": [[1, 2, 3]], "arr2": [[4, 5, 6]] }) result = df.select(fc.array.except_("arr1", "arr2")) # Output: [[1, 2, 3]] # All elements retained ``` """ return Column._from_logical_expr( ArrayExceptExpr( Column._from_col_or_name(col1)._logical_expr, Column._from_col_or_name(col2)._logical_expr ) ) ``` ## intersect ``` intersect(col1: ColumnOrName, col2: ColumnOrName) -> Column ``` Returns the intersection of two arrays. Returns distinct elements that appear in both arrays. The order of elements is not guaranteed. Returns null if either input array is null. Parameters: - **`col1`** (`ColumnOrName`) – First array column or column name. - **`col2`** (`ColumnOrName`) – Second array column or column name. Returns: - `Column` – A Column containing distinct elements present in both arrays. Intersection of arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "arr1": [["a", "b", "c"], ["x", "y", "z"]], "arr2": [["b", "c", "d"], ["y", "z", "w"]] }) result = df.select(fc.array.intersect("arr1", "arr2").alias("common")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ common β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["b", "c"] β”‚ # β”‚ ["y", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` No intersection ``` df = fc.Session.local().create_dataframe({ "arr1": [[1, 2, 3]], "arr2": [[4, 5, 6]] }) result = df.select(fc.array.intersect("arr1", "arr2")) # Output: [[]] # Empty array when no common elements ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def intersect(col1: ColumnOrName, col2: ColumnOrName) -> Column: """Returns the intersection of two arrays. Returns distinct elements that appear in both arrays. The order of elements is not guaranteed. Returns null if either input array is null. Args: col1: First array column or column name. col2: Second array column or column name. Returns: A Column containing distinct elements present in both arrays. Example: Intersection of arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "arr1": [["a", "b", "c"], ["x", "y", "z"]], "arr2": [["b", "c", "d"], ["y", "z", "w"]] }) result = df.select(fc.array.intersect("arr1", "arr2").alias("common")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ common β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["b", "c"] β”‚ # β”‚ ["y", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: No intersection ```python df = fc.Session.local().create_dataframe({ "arr1": [[1, 2, 3]], "arr2": [[4, 5, 6]] }) result = df.select(fc.array.intersect("arr1", "arr2")) # Output: [[]] # Empty array when no common elements ``` """ return Column._from_logical_expr( ArrayIntersectExpr( Column._from_col_or_name(col1)._logical_expr, Column._from_col_or_name(col2)._logical_expr ) ) ``` ## max ``` max(column: ColumnOrName) -> Column ``` Returns the maximum value in an array. Only works on arrays of comparable types (numeric, string, date, boolean). Returns null if the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: - `Column` – A Column containing the maximum value from each array. Returns the element - `Column` – type of the array (e.g., int for array of ints). Raises: - `TypeMismatchError` – If array contains non-comparable element types (e.g., structs). Finding maximum in numeric arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 20], None, []] }) result = df.select(fc.arr.max("numbers").alias("max_value")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ max_value β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 5 β”‚ # β”‚ 20 β”‚ # β”‚ null β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Finding maximum in string arrays ``` df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "zebra"], ["dog", "bat"]] }) result = df.select(fc.array.max("words").alias("max_word")) # Output: ["zebra", "dog"] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def max(column: ColumnOrName) -> Column: """Returns the maximum value in an array. Only works on arrays of comparable types (numeric, string, date, boolean). Returns null if the array is null or empty. Args: column: Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: A Column containing the maximum value from each array. Returns the element type of the array (e.g., int for array of ints). Raises: TypeMismatchError: If array contains non-comparable element types (e.g., structs). Example: Finding maximum in numeric arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 20], None, []] }) result = df.select(fc.arr.max("numbers").alias("max_value")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ max_value β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 5 β”‚ # β”‚ 20 β”‚ # β”‚ null β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Finding maximum in string arrays ```python df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "zebra"], ["dog", "bat"]] }) result = df.select(fc.array.max("words").alias("max_word")) # Output: ["zebra", "dog"] ``` """ return Column._from_logical_expr( ArrayMaxExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## min ``` min(column: ColumnOrName) -> Column ``` Returns the minimum value in an array. Only works on arrays of comparable types (numeric, string, date, boolean). Returns null if the array is null or empty. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: - `Column` – A Column containing the minimum value from each array. Returns the element - `Column` – type of the array (e.g., int for array of ints). Raises: - `TypeMismatchError` – If array contains non-comparable element types (e.g., structs). Finding minimum in numeric arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 20], None, []] }) result = df.select(fc.arr.min("numbers").alias("min_value")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ min_value β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 1 β”‚ # β”‚ 10 β”‚ # β”‚ null β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Finding minimum in string arrays ``` df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "zebra"], ["dog", "bat"]] }) result = df.select(fc.array.min("words").alias("min_word")) # Output: ["apple", "bat"] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def min(column: ColumnOrName) -> Column: """Returns the minimum value in an array. Only works on arrays of comparable types (numeric, string, date, boolean). Returns null if the array is null or empty. Args: column: Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: A Column containing the minimum value from each array. Returns the element type of the array (e.g., int for array of ints). Raises: TypeMismatchError: If array contains non-comparable element types (e.g., structs). Example: Finding minimum in numeric arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 20], None, []] }) result = df.select(fc.arr.min("numbers").alias("min_value")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ min_value β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ 1 β”‚ # β”‚ 10 β”‚ # β”‚ null β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Finding minimum in string arrays ```python df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "zebra"], ["dog", "bat"]] }) result = df.select(fc.array.min("words").alias("min_word")) # Output: ["apple", "bat"] ``` """ return Column._from_logical_expr( ArrayMinExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## overlap ``` overlap(col1: ColumnOrName, col2: ColumnOrName) -> Column ``` Checks if two arrays have at least one common element. Returns true if the two arrays share at least one common element, false if they have no common elements. Returns null if either input array is null. Parameters: - **`col1`** (`ColumnOrName`) – First array column or column name. - **`col2`** (`ColumnOrName`) – Second array column or column name. Returns: - `Column` – A boolean Column (True if arrays have common elements, False otherwise). Detecting overlap ``` import fenic as fc df = fc.Session.local().create_dataframe({ "arr1": [["a", "b", "c"], ["x", "y"], ["p", "q"]], "arr2": [["c", "d", "e"], ["w", "z"], ["q", "r"]] }) result = df.select(fc.array.overlap("arr1", "arr2").alias("has_overlap")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ has_overlap β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ true β”‚ # "c" is common # β”‚ false β”‚ # No common elements # β”‚ true β”‚ # "q" is common # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Using with filtering ``` df = fc.Session.local().create_dataframe({ "user_tags": [["python", "ml"], ["java", "web"], ["python", "web"]], "required": [["python", "data"], ["python", "data"], ["python", "data"]] }) # Filter users with at least one required tag result = df.filter(fc.array.overlap("user_tags", "required")) # Output: Rows with indices 0 and 2 (have "python" tag) ``` Numeric arrays ``` df = fc.Session.local().create_dataframe({ "nums1": [[1, 2, 3], [4, 5, 6]], "nums2": [[3, 4, 5], [7, 8, 9]] }) result = df.select(fc.array.overlap("nums1", "nums2")) # Output: [true, false] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def overlap(col1: ColumnOrName, col2: ColumnOrName) -> Column: """Checks if two arrays have at least one common element. Returns true if the two arrays share at least one common element, false if they have no common elements. Returns null if either input array is null. Args: col1: First array column or column name. col2: Second array column or column name. Returns: A boolean Column (True if arrays have common elements, False otherwise). Example: Detecting overlap ```python import fenic as fc df = fc.Session.local().create_dataframe({ "arr1": [["a", "b", "c"], ["x", "y"], ["p", "q"]], "arr2": [["c", "d", "e"], ["w", "z"], ["q", "r"]] }) result = df.select(fc.array.overlap("arr1", "arr2").alias("has_overlap")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ has_overlap β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ true β”‚ # "c" is common # β”‚ false β”‚ # No common elements # β”‚ true β”‚ # "q" is common # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Using with filtering ```python df = fc.Session.local().create_dataframe({ "user_tags": [["python", "ml"], ["java", "web"], ["python", "web"]], "required": [["python", "data"], ["python", "data"], ["python", "data"]] }) # Filter users with at least one required tag result = df.filter(fc.array.overlap("user_tags", "required")) # Output: Rows with indices 0 and 2 (have "python" tag) ``` Example: Numeric arrays ```python df = fc.Session.local().create_dataframe({ "nums1": [[1, 2, 3], [4, 5, 6]], "nums2": [[3, 4, 5], [7, 8, 9]] }) result = df.select(fc.array.overlap("nums1", "nums2")) # Output: [true, false] ``` """ return Column._from_logical_expr( ArraysOverlapExpr( Column._from_col_or_name(col1)._logical_expr, Column._from_col_or_name(col2)._logical_expr ) ) ``` ## remove ``` remove(column: ColumnOrName, element: Union[str, int, float, bool, Column]) -> Column ``` Removes all occurrences of an element from an array. Returns a new array with all instances of the specified element removed. Returns null if the input array is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. - **`element`** (`Union[str, int, float, bool, Column]`) – Element to remove from the arrays. Can be a literal value or a Column expression. Returns: - `Column` – A Column with arrays having all occurrences of the element removed. Removing literals ``` import fenic as fc df = fc.Session.local().create_dataframe({ "tags": [["a", "b", "a", "c"], ["x", "y", "x"]], "numbers": [[1, 2, 1, 3], [5, 5, 5]] }) result = df.select( fc.array.remove("tags", "a").alias("no_a"), fc.array.remove("numbers", 5).alias("no_five") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ no_a β”‚ no_five β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["b", "c"] β”‚ [1, 2, 1, 3] β”‚ # β”‚ ["x", "y"] β”‚ [] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Removing with column expression ``` df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3], [4, 5, 6]], "to_remove": [2, 5] }) result = df.select(fc.array.remove("values", fc.col("to_remove"))) # Output: [[1, 3], [4, 6]] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def remove(column: ColumnOrName, element: Union[str, int, float, bool, Column]) -> Column: """Removes all occurrences of an element from an array. Returns a new array with all instances of the specified element removed. Returns null if the input array is null. Args: column: Column or column name containing arrays. element: Element to remove from the arrays. Can be a literal value or a Column expression. Returns: A Column with arrays having all occurrences of the element removed. Example: Removing literals ```python import fenic as fc df = fc.Session.local().create_dataframe({ "tags": [["a", "b", "a", "c"], ["x", "y", "x"]], "numbers": [[1, 2, 1, 3], [5, 5, 5]] }) result = df.select( fc.array.remove("tags", "a").alias("no_a"), fc.array.remove("numbers", 5).alias("no_five") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ no_a β”‚ no_five β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["b", "c"] β”‚ [1, 2, 1, 3] β”‚ # β”‚ ["x", "y"] β”‚ [] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Removing with column expression ```python df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3], [4, 5, 6]], "to_remove": [2, 5] }) result = df.select(fc.array.remove("values", fc.col("to_remove"))) # Output: [[1, 3], [4, 6]] ``` """ element_column = element if isinstance(element, Column) else lit(element) return Column._from_logical_expr( ArrayRemoveExpr( Column._from_col_or_name(column)._logical_expr, element_column._logical_expr ) ) ``` ## repeat ``` repeat(col: ColumnOrName, count: Union[int, ColumnOrName]) -> Column ``` Creates an array containing the element repeated count times. Returns a new array where the element is repeated the specified number of times. Returns null if count is null or negative. Parameters: - **`col`** (`ColumnOrName`) – Column, column name, or literal value to repeat. - **`count`** (`Union[int, ColumnOrName]`) – Number of times to repeat the element. Can be an integer literal or a Column expression. Returns: - `Column` – A Column containing an array with the element repeated count times. Repeating literals ``` import fenic as fc df = fc.Session.local().create_dataframe({ "id": [1, 2, 3] }) result = df.select( fc.array.repeat(fc.lit("x"), 3).alias("repeated"), fc.array.repeat(fc.lit(0), 5).alias("zeros") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ repeated β”‚ zeros β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Repeating column values ``` df = fc.Session.local().create_dataframe({ "value": ["a", "b", "c"], "count": [2, 3, 1] }) result = df.select(fc.array.repeat(fc.col("value"), fc.col("count"))) # Output: [["a", "a"], ["b", "b", "b"], ["c"]] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def repeat(col: ColumnOrName, count: Union[int, ColumnOrName]) -> Column: """Creates an array containing the element repeated count times. Returns a new array where the element is repeated the specified number of times. Returns null if count is null or negative. Args: col: Column, column name, or literal value to repeat. count: Number of times to repeat the element. Can be an integer literal or a Column expression. Returns: A Column containing an array with the element repeated count times. Example: Repeating literals ```python import fenic as fc df = fc.Session.local().create_dataframe({ "id": [1, 2, 3] }) result = df.select( fc.array.repeat(fc.lit("x"), 3).alias("repeated"), fc.array.repeat(fc.lit(0), 5).alias("zeros") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ repeated β”‚ zeros β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β”‚ ["x", "x", "x"] β”‚ [0, 0, 0, 0, 0] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Repeating column values ```python df = fc.Session.local().create_dataframe({ "value": ["a", "b", "c"], "count": [2, 3, 1] }) result = df.select(fc.array.repeat(fc.col("value"), fc.col("count"))) # Output: [["a", "a"], ["b", "b", "b"], ["c"]] ``` """ count_column = count if isinstance(count, Column) else lit(count) return Column._from_logical_expr( ArrayRepeatExpr( Column._from_col_or_name(col)._logical_expr, count_column._logical_expr ) ) ``` ## reverse ``` reverse(column: ColumnOrName) -> Column ``` Reverses the elements of an array. Returns a new array with elements in reverse order. Returns null if the input array is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. Returns: - `Column` – A Column with reversed arrays. Reversing arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[1, 2, 3, 4], [10, 20]], "words": [["a", "b", "c"], ["x", "y"]] }) result = df.select( fc.array.reverse("numbers").alias("reversed_nums"), fc.array.reverse("words").alias("reversed_words") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ reversed_nums β”‚ reversed_words β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [4, 3, 2, 1] β”‚ ["c", "b", "a"] β”‚ # β”‚ [20, 10] β”‚ ["y", "x"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def reverse(column: ColumnOrName) -> Column: """Reverses the elements of an array. Returns a new array with elements in reverse order. Returns null if the input array is null. Args: column: Column or column name containing arrays. Returns: A Column with reversed arrays. Example: Reversing arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[1, 2, 3, 4], [10, 20]], "words": [["a", "b", "c"], ["x", "y"]] }) result = df.select( fc.array.reverse("numbers").alias("reversed_nums"), fc.array.reverse("words").alias("reversed_words") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ reversed_nums β”‚ reversed_words β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [4, 3, 2, 1] β”‚ ["c", "b", "a"] β”‚ # β”‚ [20, 10] β”‚ ["y", "x"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` """ return Column._from_logical_expr( ArrayReverseExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## size ``` size(column: ColumnOrName) -> Column ``` Returns the number of elements in an array column. This function computes the length of arrays stored in the specified column. Returns None for None arrays. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays whose length to compute. Returns: - `Column` – A Column expression representing the array length. Raises: - `TypeError` – If the column does not contain array data. Get array sizes ``` # Get the size of arrays in 'tags' column df.select(fc.arr.size("tags")) # Use with column reference df.select(fc.arr.size(fc.col("tags"))) ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def size(column: ColumnOrName) -> Column: """Returns the number of elements in an array column. This function computes the length of arrays stored in the specified column. Returns None for None arrays. Args: column: Column or column name containing arrays whose length to compute. Returns: A Column expression representing the array length. Raises: TypeError: If the column does not contain array data. Example: Get array sizes ```python # Get the size of arrays in 'tags' column df.select(fc.arr.size("tags")) # Use with column reference df.select(fc.arr.size(fc.col("tags"))) ``` """ return Column._from_logical_expr( ArrayLengthExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## slice ``` slice(column: ColumnOrName, start: Union[int, ColumnOrName], length: Union[int, ColumnOrName]) -> Column ``` Extracts a subarray from an array using 1-based indexing (PySpark compatible). Extracts a contiguous subarray starting from the given position. Uses 1-based indexing for compatibility with PySpark. Returns null if the input array is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays. - **`start`** (`Union[int, ColumnOrName]`) – Starting position (1-based index). Positive indices count from the start (1 = first element), negative indices count from the end (-1 = last element). - **`length`** (`Union[int, ColumnOrName]`) – Number of elements to extract. Must be positive. Returns: - `Column` – A Column with subarrays extracted. Extracting from the start ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[1, 2, 3, 4, 5], [10, 20, 30]] }) result = df.select( fc.array.slice("numbers", 1, 3).alias("first_three"), fc.array.slice("numbers", 2, 2).alias("middle_two") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ first_three β”‚ middle_two β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3] β”‚ [2, 3] β”‚ # β”‚ [10, 20, 30] β”‚ [20, 30] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Using negative indices ``` df = fc.Session.local().create_dataframe({ "arr": [[1, 2, 3, 4, 5]] }) # Extract last 3 elements: start at -3, take 3 result = df.select(fc.array.slice("arr", -3, 3)) # Output: [[3, 4, 5]] ``` Dynamic slicing with columns ``` df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3, 4, 5], [10, 20, 30]], "start_idx": [2, 1], "num_elements": [2, 2] }) result = df.select( fc.array.slice("values", fc.col("start_idx"), fc.col("num_elements")) ) # Output: [[2, 3], [10, 20]] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def slice(column: ColumnOrName, start: Union[int, ColumnOrName], length: Union[int, ColumnOrName]) -> Column: """Extracts a subarray from an array using 1-based indexing (PySpark compatible). Extracts a contiguous subarray starting from the given position. Uses 1-based indexing for compatibility with PySpark. Returns null if the input array is null. Args: column: Column or column name containing arrays. start: Starting position (1-based index). Positive indices count from the start (1 = first element), negative indices count from the end (-1 = last element). length: Number of elements to extract. Must be positive. Returns: A Column with subarrays extracted. Example: Extracting from the start ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[1, 2, 3, 4, 5], [10, 20, 30]] }) result = df.select( fc.array.slice("numbers", 1, 3).alias("first_three"), fc.array.slice("numbers", 2, 2).alias("middle_two") ) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ first_three β”‚ middle_two β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3] β”‚ [2, 3] β”‚ # β”‚ [10, 20, 30] β”‚ [20, 30] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Using negative indices ```python df = fc.Session.local().create_dataframe({ "arr": [[1, 2, 3, 4, 5]] }) # Extract last 3 elements: start at -3, take 3 result = df.select(fc.array.slice("arr", -3, 3)) # Output: [[3, 4, 5]] ``` Example: Dynamic slicing with columns ```python df = fc.Session.local().create_dataframe({ "values": [[1, 2, 3, 4, 5], [10, 20, 30]], "start_idx": [2, 1], "num_elements": [2, 2] }) result = df.select( fc.array.slice("values", fc.col("start_idx"), fc.col("num_elements")) ) # Output: [[2, 3], [10, 20]] ``` """ start_column = start if isinstance(start, Column) else lit(start) length_column = length if isinstance(length, Column) else lit(length) return Column._from_logical_expr( ArraySliceExpr( Column._from_col_or_name(column)._logical_expr, start_column._logical_expr, length_column._logical_expr ) ) ``` ## sort ``` sort(column: ColumnOrName) -> Column ``` Sorts the array in ascending order. Only works on arrays of comparable types (numeric, string, date, boolean). Null values are placed at the end of the array. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: - `Column` – A Column with sorted arrays in ascending order. Returns null if the input - `Column` – array is null. Raises: - `TypeMismatchError` – If array contains non-comparable element types (e.g., structs). Note Unlike PySpark's array_sort, this does not support a custom comparator function. For custom sorting logic on complex types, consider using other transformations. Sorting numeric arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 30, 20], None] }) result = df.select(fc.array.sort("numbers").alias("sorted")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ sorted β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 5] β”‚ # β”‚ [10, 20, 30] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Sorting string arrays ``` df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "bat"], ["zebra", "apple"]] }) result = df.select(fc.array.sort("words").alias("sorted")) # Output: [["apple", "bat", "cat"], ["apple", "zebra"]] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sort(column: ColumnOrName) -> Column: """Sorts the array in ascending order. Only works on arrays of comparable types (numeric, string, date, boolean). Null values are placed at the end of the array. Args: column: Column or column name containing arrays of comparable types (numeric, string, date, boolean). Does not work on arrays of structs. Returns: A Column with sorted arrays in ascending order. Returns null if the input array is null. Raises: TypeMismatchError: If array contains non-comparable element types (e.g., structs). Note: Unlike PySpark's array_sort, this does not support a custom comparator function. For custom sorting logic on complex types, consider using other transformations. Example: Sorting numeric arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "numbers": [[3, 1, 5, 2], [10, 30, 20], None] }) result = df.select(fc.array.sort("numbers").alias("sorted")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ sorted β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 5] β”‚ # β”‚ [10, 20, 30] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Sorting string arrays ```python df = fc.Session.local().create_dataframe({ "words": [["cat", "apple", "bat"], ["zebra", "apple"]] }) result = df.select(fc.array.sort("words").alias("sorted")) # Output: [["apple", "bat", "cat"], ["apple", "zebra"]] ``` """ return Column._from_logical_expr( ArraySortExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## union ``` union(col1: ColumnOrName, col2: ColumnOrName) -> Column ``` Returns the union of two arrays without duplicates. Returns all distinct elements from both arrays. The order of elements is not guaranteed. Returns null if either input array is null. Parameters: - **`col1`** (`ColumnOrName`) – First array column or column name. - **`col2`** (`ColumnOrName`) – Second array column or column name. Returns: - `Column` – A Column containing the distinct union of both arrays. Union of tag arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "tags1": [["a", "b", "c"], ["x", "y"]], "tags2": [["b", "c", "d"], ["y", "z"]] }) result = df.select(fc.array.union("tags1", "tags2").alias("all_tags")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ all_tags β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["a", "b", "c", "d"] β”‚ # β”‚ ["x", "y", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Union with numeric arrays ``` df = fc.Session.local().create_dataframe({ "nums1": [[1, 2, 3], [5, 6]], "nums2": [[2, 3, 4], [6, 7]] }) result = df.select(fc.array.union("nums1", "nums2")) # Output: [[1, 2, 3, 4], [5, 6, 7]] ``` Source code in `src/fenic/api/functions/array.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def union(col1: ColumnOrName, col2: ColumnOrName) -> Column: """Returns the union of two arrays without duplicates. Returns all distinct elements from both arrays. The order of elements is not guaranteed. Returns null if either input array is null. Args: col1: First array column or column name. col2: Second array column or column name. Returns: A Column containing the distinct union of both arrays. Example: Union of tag arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "tags1": [["a", "b", "c"], ["x", "y"]], "tags2": [["b", "c", "d"], ["y", "z"]] }) result = df.select(fc.array.union("tags1", "tags2").alias("all_tags")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ all_tags β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ ["a", "b", "c", "d"] β”‚ # β”‚ ["x", "y", "z"] β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: Union with numeric arrays ```python df = fc.Session.local().create_dataframe({ "nums1": [[1, 2, 3], [5, 6]], "nums2": [[2, 3, 4], [6, 7]] }) result = df.select(fc.array.union("nums1", "nums2")) # Output: [[1, 2, 3, 4], [5, 6, 7]] ``` """ return Column._from_logical_expr( ArrayUnionExpr( Column._from_col_or_name(col1)._logical_expr, Column._from_col_or_name(col2)._logical_expr ) ) ``` --- # fenic.api.functions.builtin Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/builtin/ Built-in functions for Fenic DataFrames. Functions: - **`approx_count_distinct`** – Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. - **`array`** – Creates a new array column from multiple input columns. - **`array_agg`** – Alias for collect_list(). - **`asc`** – Mark this column for ascending sort order with nulls first. - **`asc_nulls_first`** – Alias for asc(). - **`asc_nulls_last`** – Mark this column for ascending sort order with nulls last. - **`async_udf`** – A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. - **`avg`** – Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. - **`coalesce`** – Returns the first non-null value from the given columns for each row. - **`collect_list`** – Aggregate function: collects all values from the specified column into a list. - **`count`** – Aggregate function: returns the count of non-null values in the specified column. - **`count_distinct`** – Aggregate function: returns the number of distinct non-null rows across one or more columns. - **`desc`** – Mark this column for descending sort order with nulls first. - **`desc_nulls_first`** – Alias for desc(). - **`desc_nulls_last`** – Mark this column for descending sort order with nulls last. - **`first`** – Aggregate function: returns the first non-null value in the specified column. - **`flatten`** – Flattens an array of arrays into a single array (one level deep). - **`greatest`** – Returns the greatest value from the given columns for each row. - **`least`** – Returns the least value from the given columns for each row. - **`max`** – Aggregate function: returns the maximum value in the specified column. - **`mean`** – Aggregate function: returns the mean (average) of all values in the specified column. - **`min`** – Aggregate function: returns the minimum value in the specified column. - **`stddev`** – Aggregate function: returns the sample standard deviation of the specified column. - **`struct`** – Creates a new struct column from multiple input columns. - **`sum`** – Aggregate function: returns the sum of all values in the specified column. - **`sum_distinct`** – Aggregate function: returns the sum of distinct numeric values in the specified column. - **`udf`** – A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. - **`when`** – Evaluates a conditional expression (like if-then). ## approx_count_distinct ``` approx_count_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: - `Column` – A Column expression representing the approximate count-distinct aggregation Note Differs from the pyspark implementation in that the relative standard deviation is not configurable. Approximate distinct count per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Nulls are ignored in approximate distinct counts ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: - `TypeMismatchError` – If column is a StructType or ArrayType column. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def approx_count_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns an approximate count (HyperLogLog++) of distinct non-null values. Args: column: Column or column name to approximately count distinct values in. Cannot be a StructType column. Returns: A Column expression representing the approximate count-distinct aggregation Note: Differs from the pyspark implementation in that the relative standard deviation is not configurable. Example: Approximate distinct count per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b", "b"], "v": [1, None, 1, 2, 3], }) df.group_by(fc.col("k")).agg( fc.approx_count_distinct(fc.col("v")).alias("approx_unique_v") ).show() # Output: # +---+------------------+ # | k | approx_unique_v | # +---+------------------+ # | a | 1 | # | b | 3 | # +---+------------------+ ``` Example: Nulls are ignored in approximate distinct counts ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.approx_count_distinct(fc.col("v")).alias("acd")).show() # Output: # +---+-----+ # | k | acd | # +---+-----+ # | x | 1 | # +---+-----+ ``` Raises: TypeMismatchError: If column is a StructType or ArrayType column. """ return Column._from_logical_expr( ApproxCountDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## array ``` array(*args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]) -> Column ``` Creates a new array column from multiple input columns. Parameters: - **`*args`** (`Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]`, default: `()` ) – Columns or column names to combine into an array. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: - `Column` – A Column expression representing an array containing values from the input columns Raises: - `TypeError` – If any argument is not a Column, string, or collection of Columns/strings Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array( *args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]] ) -> Column: """Creates a new array column from multiple input columns. Args: *args: Columns or column names to combine into an array. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: A Column expression representing an array containing values from the input columns Raises: TypeError: If any argument is not a Column, string, or collection of Columns/strings """ flattened_args = [] for arg in args: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_columns = [Column._from_col_or_name(c)._logical_expr for c in flattened_args] return Column._from_logical_expr(ArrayExpr(expr_columns)) ``` ## array_agg ``` array_agg(column: ColumnOrName) -> Column ``` Alias for collect_list(). Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array_agg(column: ColumnOrName) -> Column: """Alias for collect_list().""" return collect_list(column) ``` ## asc ``` asc(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls first. Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc() ``` ## asc_nulls_first ``` asc_nulls_first(column: ColumnOrName) -> Column ``` Alias for asc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A sort expression with ascending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_first(column: ColumnOrName) -> Column: """Alias for asc(). Args: column: The column to apply the ascending ordering to. Returns: A sort expression with ascending order and nulls first. """ return Column._from_col_or_name(column).asc_nulls_first() ``` ## asc_nulls_last ``` asc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for ascending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the ascending ordering to. Returns: - `Column` – A Column expression representing the column and the ascending sort order with nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def asc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for ascending sort order with nulls last. Args: column: The column to apply the ascending ordering to. Returns: A Column expression representing the column and the ascending sort order with nulls last. """ return Column._from_col_or_name(column).asc_nulls_last() ``` ## async_udf ``` async_udf(f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0) ``` A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Parameters: - **`f`** (`Optional[Callable[..., Awaitable[Any]]]`, default: `None` ) – Async function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. - **`max_concurrency`** (`int`, default: `10` ) – Maximum number of concurrent executions (default: 10) - **`timeout_seconds`** (`float`, default: `30` ) – Per-item timeout in seconds (default: 30) - **`num_retries`** (`int`, default: `0` ) – Number of retries for failed items (default: 0) Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) ### Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries `python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() }` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def async_udf( f: Optional[Callable[..., Awaitable[Any]]] = None, *, return_type: DataType, max_concurrency: int = 10, timeout_seconds: float = 30, num_retries: int = 0, ): """A decorator for creating async user-defined functions (UDFs) with configurable concurrency and retries. Async UDFs allow IO-bound operations (API calls, database queries, MCP tool calls) to be executed concurrently while maintaining DataFrame semantics. Args: f: Async function to convert to UDF return_type: Expected return type of the UDF. Required parameter. max_concurrency: Maximum number of concurrent executions (default: 10) timeout_seconds: Per-item timeout in seconds (default: 30) num_retries: Number of retries for failed items (default: 0) Example: Basic async UDF ```python @async_udf(return_type=IntegerType) async def slow_add(x: int, y: int) -> int: await asyncio.sleep(1) return x + y df = df.select(slow_add(fc.col("x"), fc.col("y")).alias("slow_sum")) # Or async def slow_add_fn(x: int, y: int) -> int: await asyncio.sleep(1) return x + y slow_add = async_udf( slow_add_fn, return_type=IntegerType ) ``` Example: API call with custom concurrency and retries ```python @async_udf( return_type=StructType([ StructField("status", IntegerType), StructField("data", StringType) ]), max_concurrency=20, timeout_seconds=5, num_retries=2 ) async def fetch_data(id: str) -> dict: async with aiohttp.ClientSession() as session: async with session.get(f"https://api.example.com/{id}") as resp: return { "status": resp.status, "data": await resp.text() } ``` Note: - Individual failures return None instead of raising exceptions - Async UDFs should not block or do CPU-intensive work, as they will block execution of other instances of the function call. """ def _create_async_udf(func: Callable[..., Awaitable[Any]]) -> Callable: if not inspect.iscoroutinefunction(func): raise ValidationError( f"@async_udf requires an async function, but found a synchronous " f"function {func.__name__!r} of type {type(func)}" ) @wraps(func) def _async_udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr( AsyncUDFExpr( func, col_exprs, return_type, max_concurrency=max_concurrency, timeout_seconds=timeout_seconds, num_retries=num_retries ) ) return _async_udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for async UDFs") # Support both @async_udf and async_udf(...) syntax if f is None: return _create_async_udf else: return _create_async_udf(f) ``` ## avg ``` avg(column: ColumnOrName) -> Column ``` Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the average of Returns: - `Column` – A Column expression representing the average aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def avg(column: ColumnOrName) -> Column: """Aggregate function: returns the average (mean) of all values in the specified column. Applies to numeric and embedding types. Args: column: Column or column name to compute the average of Returns: A Column expression representing the average aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## coalesce ``` coalesce(*cols: ColumnOrName) -> Column ``` Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the first non-null value from the input columns. Raises: - `ValidationError` – If no columns are provided. coalesce usage ``` df.select(coalesce("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def coalesce(*cols: ColumnOrName) -> Column: """Returns the first non-null value from the given columns for each row. This function mimics the behavior of SQL's COALESCE function. It evaluates the input columns in order and returns the first non-null value encountered. If all values are null, returns null. Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the first non-null value from the input columns. Raises: ValidationError: If no columns are provided. Example: coalesce usage ```python df.select(coalesce("col1", "col2", "col3")) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the coalesce method.") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(CoalesceExpr(exprs)) ``` ## collect_list ``` collect_list(column: ColumnOrName) -> Column ``` Aggregate function: collects all values from the specified column into a list. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to collect values from Returns: - `Column` – A Column expression representing the list aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def collect_list(column: ColumnOrName) -> Column: """Aggregate function: collects all values from the specified column into a list. Args: column: Column or column name to collect values from Returns: A Column expression representing the list aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( ListExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count ``` count(column: ColumnOrName) -> Column ``` Aggregate function: returns the count of non-null values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to count values in Returns: - `Column` – A Column expression representing the count aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count(column: ColumnOrName) -> Column: """Aggregate function: returns the count of non-null values in the specified column. Args: column: Column or column name to count values in Returns: A Column expression representing the count aggregation Raises: TypeError: If column is not a Column or string """ if isinstance(column, str) and column == "*": return Column._from_logical_expr(CountExpr(lit("*")._logical_expr)) return Column._from_logical_expr( CountExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## count_distinct ``` count_distinct(*cols: ColumnOrName) -> Column ``` Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – One or more columns or column names to include in the distinct count. Returns: - `Column` – A Column expression representing the count-distinct aggregation over the provided columns. Distinct count per group (single column) ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Distinct count across multiple columns (whole DataFrame) ``` # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Nulls in any input column are ignored for multi-column distinct ``` df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: - `ValidationError` – If no columns are provided. - `TypeMismatchError` – If a column has an unsupported type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count_distinct(*cols: ColumnOrName) -> Column: """Aggregate function: returns the number of distinct non-null rows across one or more columns. Behavior: Any row where one or more inputs is null is ignored. Args: *cols: One or more columns or column names to include in the distinct count. Returns: A Column expression representing the count-distinct aggregation over the provided columns. Example: Distinct count per group (single column) ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) df.group_by(fc.col("k")).agg( fc.count_distinct(fc.col("v")).alias("num_unique_v") ).show() # Output: # +---+--------------+ # | k | num_unique_v | # +---+--------------+ # | a | 1 | # | b | 1 | # +---+--------------+ ``` Example: Distinct count across multiple columns (whole DataFrame) ```python # Sample input df = session.create_dataframe({ "a": [1, 1, 2, 2, None], "b": ["x", "x", "y", "y", "z"], }) df.agg( fc.count_distinct(fc.col("a"), fc.col("b")).alias("num_unique_pairs") ).show() # Output: # +------------------+ # | num_unique_pairs | # +------------------+ # | 2 | # +------------------+ ``` Example: Nulls in any input column are ignored for multi-column distinct ```python df = session.create_dataframe({"a": [1, 1, None], "b": [1, 2, 1]}) df.agg(fc.count_distinct(fc.col("a"), fc.col("b")).alias("cd")).show() # Output: # +----+ # | cd | # +----+ # | 2 | # +----+ ``` Raises: ValidationError: If no columns are provided. TypeMismatchError: If a column has an unsupported type """ if not cols: raise ValidationError("count_distinct requires at least one column") exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(CountDistinctExpr(exprs)) ``` ## desc ``` desc(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls first. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls first. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc() ``` ## desc_nulls_first ``` desc_nulls_first(column: ColumnOrName) -> Column ``` Alias for desc(). Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls first. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_first(column: ColumnOrName) -> Column: """Alias for desc(). Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls first. """ return Column._from_col_or_name(column).desc_nulls_first() ``` ## desc_nulls_last ``` desc_nulls_last(column: ColumnOrName) -> Column ``` Mark this column for descending sort order with nulls last. Parameters: - **`column`** (`ColumnOrName`) – The column to apply the descending ordering to. Returns: - `Column` – A sort expression with descending order and nulls last. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def desc_nulls_last(column: ColumnOrName) -> Column: """Mark this column for descending sort order with nulls last. Args: column: The column to apply the descending ordering to. Returns: A sort expression with descending order and nulls last. """ return Column._from_col_or_name(column).desc_nulls_last() ``` ## first ``` first(column: ColumnOrName) -> Column ``` Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for the first value. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def first(column: ColumnOrName) -> Column: """Aggregate function: returns the first non-null value in the specified column. Typically used in aggregations to select the first observed value per group. Args: column: Column or column name. Returns: Column expression for the first value. """ return Column._from_logical_expr( FirstExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## flatten ``` flatten(column: ColumnOrName) -> Column ``` Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing arrays of arrays. Returns: - `Column` – A Column with flattened arrays (one level deep). Flattening nested arrays ``` import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` One level only ``` # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def flatten(column: ColumnOrName) -> Column: """Flattens an array of arrays into a single array (one level deep). Flattens nested arrays by concatenating all inner arrays into a single array. Only flattens one level of nesting. Returns null if the input is null. Args: column: Column or column name containing arrays of arrays. Returns: A Column with flattened arrays (one level deep). Example: Flattening nested arrays ```python import fenic as fc df = fc.Session.local().create_dataframe({ "nested": [[[1, 2], [3, 4]], [[5], [6, 7, 8]], None] }) result = df.select(fc.flatten("nested").alias("flat")) # Output: # β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” # β”‚ flat β”‚ # β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ # β”‚ [1, 2, 3, 4] β”‚ # β”‚ [5, 6, 7, 8] β”‚ # β”‚ null β”‚ # β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Example: One level only ```python # Deeply nested arrays - only flattens one level df = fc.Session.local().create_dataframe({ "deep": [[[[1]], [[2]]], [[[3]]]] }) result = df.select(fc.flatten("deep")) # Output: [[[1], [2]], [[3]]] # Still nested after one level ``` """ return Column._from_logical_expr( FlattenExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## greatest ``` greatest(*cols: ColumnOrName) -> Column ``` Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the greatest value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. greatest usage ``` df.select(fc.greatest("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def greatest(*cols: ColumnOrName) -> Column: """Returns the greatest value from the given columns for each row. This function mimics the behavior of SQL's GREATEST function. It evaluates the input columns in order and returns the greatest value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the greatest value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: greatest usage ```python df.select(fc.greatest("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"greatest() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(GreatestExpr(exprs)) ``` ## least ``` least(*cols: ColumnOrName) -> Column ``` Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: - `Column` – A Column expression containing the least value from the input columns. Raises: - `ValidationError` – If fewer than two columns are provided. least usage ``` df.select(fc.least("col1", "col2", "col3")) ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def least(*cols: ColumnOrName) -> Column: """Returns the least value from the given columns for each row. This function mimics the behavior of SQL's LEAST function. It evaluates the input columns in order and returns the least value encountered. If all values are null, returns null. All arguments must be of the same primitive type (e.g., StringType, BooleanType, FloatType, IntegerType, etc). Args: *cols: Column expressions or column names to evaluate. Each argument should be a single column expression or column name string. Returns: A Column expression containing the least value from the input columns. Raises: ValidationError: If fewer than two columns are provided. Example: least usage ```python df.select(fc.least("col1", "col2", "col3")) ``` """ if len(cols) < 2: raise ValidationError(f"least() requires at least 2 columns, got {len(cols)}") exprs = [ Column._from_col_or_name(c)._logical_expr for c in cols ] return Column._from_logical_expr(LeastExpr(exprs)) ``` ## max ``` max(column: ColumnOrName) -> Column ``` Aggregate function: returns the maximum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the maximum of Returns: - `Column` – A Column expression representing the maximum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def max(column: ColumnOrName) -> Column: """Aggregate function: returns the maximum value in the specified column. Args: column: Column or column name to compute the maximum of Returns: A Column expression representing the maximum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MaxExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## mean ``` mean(column: ColumnOrName) -> Column ``` Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the mean of Returns: - `Column` – A Column expression representing the mean aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def mean(column: ColumnOrName) -> Column: """Aggregate function: returns the mean (average) of all values in the specified column. Alias for avg(). Args: column: Column or column name to compute the mean of Returns: A Column expression representing the mean aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( AvgExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## min ``` min(column: ColumnOrName) -> Column ``` Aggregate function: returns the minimum value in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the minimum of Returns: - `Column` – A Column expression representing the minimum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def min(column: ColumnOrName) -> Column: """Aggregate function: returns the minimum value in the specified column. Args: column: Column or column name to compute the minimum of Returns: A Column expression representing the minimum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( MinExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## stddev ``` stddev(column: ColumnOrName) -> Column ``` Aggregate function: returns the sample standard deviation of the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name. Returns: - `Column` – Column expression for sample standard deviation. Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def stddev(column: ColumnOrName) -> Column: """Aggregate function: returns the sample standard deviation of the specified column. Args: column: Column or column name. Returns: Column expression for sample standard deviation. """ return Column._from_logical_expr( StdDevExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## struct ``` struct(*args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]) -> Column ``` Creates a new struct column from multiple input columns. Parameters: - **`*args`** (`Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]]`, default: `()` ) – Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: - `Column` – A Column expression representing a struct containing the input columns Raises: - `TypeError` – If any argument is not a Column, string, or collection of Columns/strings Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def struct( *args: Union[ColumnOrName, List[ColumnOrName], Tuple[ColumnOrName, ...]] ) -> Column: """Creates a new struct column from multiple input columns. Args: *args: Columns or column names to combine into a struct. Can be: - Individual arguments - Lists of columns/column names - Tuples of columns/column names Returns: A Column expression representing a struct containing the input columns Raises: TypeError: If any argument is not a Column, string, or collection of Columns/strings """ flattened_args = [] for arg in args: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_columns = [Column._from_col_or_name(c)._logical_expr for c in flattened_args] return Column._from_logical_expr(StructExpr(expr_columns)) ``` ## sum ``` sum(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of all values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of Returns: - `Column` – A Column expression representing the sum aggregation Raises: - `TypeError` – If column is not a Column or string Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of all values in the specified column. Args: column: Column or column name to compute the sum of Returns: A Column expression representing the sum aggregation Raises: TypeError: If column is not a Column or string """ return Column._from_logical_expr( SumExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## sum_distinct ``` sum_distinct(column: ColumnOrName) -> Column ``` Aggregate function: returns the sum of distinct numeric values in the specified column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name to compute the sum of distinct values Returns: - `Column` – A Column expression representing the sum-distinct aggregation Sum of distinct values per group ``` # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Nulls are ignored when summing distinct values ``` df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: - `TypeMismatchError` – If column is not a numeric or boolean type Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def sum_distinct(column: ColumnOrName) -> Column: """Aggregate function: returns the sum of distinct numeric values in the specified column. Args: column: Column or column name to compute the sum of distinct values Returns: A Column expression representing the sum-distinct aggregation Example: Sum of distinct values per group ```python # Sample input df = session.create_dataframe({ "k": ["a", "a", "b", "b"], "v": [1, None, 2, 2], }) # Sum distinct values of column `v` within each group `k` df.group_by(fc.col("k")).agg( fc.sum_distinct(fc.col("v")).alias("sum_distinct_v") ).show() # Output: # +---+----------------+ # | k | sum_distinct_v | # +---+----------------+ # | a | 1 | # | b | 2 | # +---+----------------+ ``` Example: Nulls are ignored when summing distinct values ```python df = session.create_dataframe({"k": ["x", "x"], "v": [None, 3]}) df.group_by(fc.col("k")).agg(fc.sum_distinct(fc.col("v")).alias("sd")).show() # Output: # +---+----+ # | k | sd | # +---+----+ # | x | 3 | # +---+----+ ``` Raises: TypeMismatchError: If column is not a numeric or boolean type """ return Column._from_logical_expr( SumDistinctExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## udf ``` udf(f: Optional[Callable] = None, *, return_type: DataType) ``` A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Parameters: - **`f`** (`Optional[Callable]`, default: `None` ) – Python function to convert to UDF - **`return_type`** (`DataType`) – Expected return type of the UDF. Required parameter. UDF with primitive types ``` # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` UDF with nested types ``` # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def udf(f: Optional[Callable] = None, *, return_type: DataType): """A decorator or function for creating user-defined functions (UDFs) that can be applied to DataFrame rows. Warning: UDFs cannot be serialized and are not supported in cloud execution. User-defined functions contain arbitrary Python code that cannot be transmitted to remote workers. For cloud compatibility, use built-in fenic functions instead. When applied, UDFs will: - Access `StructType` columns as Python dictionaries (`dict[str, Any]`). - Access `ArrayType` columns as Python lists (`list[Any]`). - Access primitive types (e.g., `int`, `float`, `str`) as their respective Python types. Args: f: Python function to convert to UDF return_type: Expected return type of the UDF. Required parameter. Example: UDF with primitive types ```python # UDF with primitive types @udf(return_type=IntegerType) def add_one(x: int): return x + 1 # Or add_one = udf(lambda x: x + 1, return_type=IntegerType) ``` Example: UDF with nested types ```python # UDF with nested types @udf(return_type=StructType([StructField("value1", IntegerType), StructField("value2", IntegerType)])) def example_udf(x: dict[str, int], y: list[int]): return { "value1": x["value1"] + x["value2"] + y[0], "value2": x["value1"] + x["value2"] + y[1], } ``` """ def _create_udf(func: Callable) -> Callable: @wraps(func) def _udf_wrapper(*cols: ColumnOrName) -> Column: col_exprs = [Column._from_col_or_name(c)._logical_expr for c in cols] return Column._from_logical_expr(UDFExpr(func, col_exprs, return_type)) return _udf_wrapper if _is_logical_type(return_type): raise NotImplementedError(f"return_type {return_type} is not supported for UDFs") if f is not None: return _create_udf(f) return _create_udf ``` ## when ``` when(condition: Column, value: Column) -> Column ``` Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Parameters: - **`condition`** (`Column`) – Boolean expression to test - **`value`** (`Column`) – Value to return when condition is True Returns: - **`Column`** ( `Column` ) – A when expression that can be chained with more conditions Raises: - `TypeMismatchError` – If the condition is not a boolean Column expression. Example ``` # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null Source code in `src/fenic/api/functions/builtin.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def when(condition: Column, value: Column) -> Column: """Evaluates a conditional expression (like if-then). Evaluates a condition for each row and returns a value when true. Can be chained with more .when() calls or finished with .otherwise(). All branches must return the same type. Args: condition: Boolean expression to test value: Value to return when condition is True Returns: Column: A when expression that can be chained with more conditions Raises: TypeMismatchError: If the condition is not a boolean Column expression. Example: ```python # Simple if-then (returns null when false) df.select(fc.when(col("age") >= 18, fc.lit("adult"))) # If-then-else df.select( fc.when(col("age") >= 18, fc.lit("adult")).otherwise(fc.lit("minor")) ) # Multiple conditions (if-elif-else) df.select( when(col("score") >= 90, "A") .when(col("score") >= 80, "B") .when(col("score") >= 70, "C") .otherwise("F") ) ``` Note: Without .otherwise(), unmatched rows return null """ return Column._from_logical_expr( WhenExpr(None, condition._logical_expr, value._logical_expr) ) ``` --- # fenic.api.functions.core Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/core/ Core functions for Fenic DataFrames. Functions: - **`col`** – Creates a Column expression referencing a column in the DataFrame. - **`empty`** – Creates a Column expression representing an empty value of the given type. - **`lit`** – Creates a Column expression representing a literal value. - **`null`** – Creates a Column expression representing a null value of the specified data type. - **`tool_param`** – Creates an unresolved literal placeholder column with a declared data type. ## col ``` col(col_name: str) -> Column ``` Creates a Column expression referencing a column in the DataFrame. Parameters: - **`col_name`** (`str`) – Name of the column to reference Returns: - `Column` – A Column expression for the specified column Raises: - `TypeError` – If colName is not a string Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True)) def col(col_name: str) -> Column: """Creates a Column expression referencing a column in the DataFrame. Args: col_name: Name of the column to reference Returns: A Column expression for the specified column Raises: TypeError: If colName is not a string """ return Column._from_column_name(col_name) ``` ## empty ``` empty(data_type: DataType) -> Column ``` Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the empty value Returns: - `Column` – A Column expression representing the empty value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with an empty array type ``` # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Creating a column with an empty struct type ``` # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Creating a column with an empty primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` Source code in `src/fenic/api/functions/core.py` ``` def empty(data_type: DataType) -> Column: """Creates a Column expression representing an empty value of the given type. - If the data type is `ArrayType(...)`, the empty value will be an empty array. - If the data type is `StructType(...)`, the empty value will be an instance of the struct type with all fields set to `None`. - For all other data types, the empty value is None (equivalent to calling `null(data_type)`) This function is useful for creating columns with empty values of a particular type. Args: data_type: The data type of the empty value Returns: A Column expression representing the empty value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with an empty array type ```python # The newly created `b` column will have a value of `[]` for all rows df.select(fc.col("a"), fc.empty(fc.ArrayType(fc.IntegerType)).alias("b")) ``` Example: Creating a column with an empty struct type ```python # The newly created `b` column will have a value of `{b: None}` for all rows df.select(fc.col("a"), fc.empty(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("b")) ``` Example: Creating a column with an empty primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.empty(fc.IntegerType).alias("b")) ``` """ if isinstance(data_type, ArrayType): return Column._from_logical_expr(LiteralExpr([], data_type)) elif isinstance(data_type, StructType): return Column._from_logical_expr(LiteralExpr({}, data_type)) return null(data_type) ``` ## lit ``` lit(value: Any) -> Column ``` Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Parameters: - **`value`** (`Any`) – The literal value to create a column for Returns: - `Column` – A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred Source code in `src/fenic/api/functions/core.py` ``` def lit(value: Any) -> Column: """Creates a Column expression representing a literal value. Column Data Type must be inferrable from the value: - Cannot be used to create a columm with the literal value `None`. Use `null(data_type)` instead. - Cannot be used to create a columm with the literal value `[]`. Use `empty(ArrayType(...))` instead. - Cannot be used to create a columm with the literal value `{}`. Use `empty(StructType(...))` instead. Args: value: The literal value to create a column for Returns: A Column expression representing the literal value Raises: ValidationError: If the type of the value cannot be inferred """ if value is None: raise ValidationError("Cannot create a literal with value `None`. Use `null(...)` instead.") elif value == []: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(ArrayType(...))` instead.") elif value == {}: raise ValidationError(f"Cannot create a literal with empty value `{value}` Use `empty(StructType(...))` instead.") try: inferred_type = infer_dtype_from_pyobj(value) except TypeInferenceError as e: raise ValidationError(f"`lit` failed to infer type for value `{value}`") from e literal_expr = LiteralExpr(value, inferred_type) return Column._from_logical_expr(literal_expr) ``` ## null ``` null(data_type: DataType) -> Column ``` Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Parameters: - **`data_type`** (`DataType`) – The data type of the null value Returns: - `Column` – A Column expression representing the null value Raises: - `ValidationError` – If the data type is not a valid data type Creating a column with a null value of a primitive type ``` # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Creating a column with a null value of an array/struct type ``` # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` Source code in `src/fenic/api/functions/core.py` ``` def null(data_type: DataType) -> Column: """Creates a Column expression representing a null value of the specified data type. Regardless of the data type, the column will contain a null (None) value. This function is useful for creating columns with null values of a particular type. Args: data_type: The data type of the null value Returns: A Column expression representing the null value Raises: ValidationError: If the data type is not a valid data type Example: Creating a column with a null value of a primitive type ```python # The newly created `b` column will have a value of `None` for all rows df.select(fc.col("a"), fc.null(fc.IntegerType).alias("b")) ``` Example: Creating a column with a null value of an array/struct type ```python # The newly created `b` and `c` columns will have a value of `None` for all rows df.select( fc.col("a"), fc.null(fc.ArrayType(fc.IntegerType)).alias("b"), fc.null(fc.StructType([fc.StructField("b", fc.IntegerType)])).alias("c"), ) ``` """ return Column._from_logical_expr(LiteralExpr(None, data_type)) ``` ## tool_param ``` tool_param(parameter_name: str, data_type: DataType) -> Column ``` Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Parameters: - **`parameter_name`** (`str`) – The name of the parameter to reference. - **`data_type`** (`DataType`) – The expected data type for the parameter value. Returns: - `Column` – A Column wrapping an UnresolvedLiteralExpr for the given parameter. A simple tool with one parameter ```python ### Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) A tool with multiple filters ```python ### Assume we are reading data with an `age` column. df = session.read.csv(users.csv) ### create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) Source code in `src/fenic/api/functions/core.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def tool_param(parameter_name: str, data_type: DataType) -> Column: """Creates an unresolved literal placeholder column with a declared data type. A placeholder argument for a DataFrame, representing a literal value to be provided at execution time. If no value is supplied, it defaults to null. Enables parameterized views and macros over fenic DataFrames. Notes: Supports only Primitive/Object/ArrayLike Types (StringType, IntegerType, FloatType, DoubleType, BooleanType, StructType, ArrayType) Args: parameter_name: The name of the parameter to reference. data_type: The expected data type for the parameter value. Returns: A Column wrapping an UnresolvedLiteralExpr for the given parameter. Example: A simple tool with one parameter ```python # Assume we are reading data with a `name` column. df = session.read.csv(data.csv) parameterized_df = df.filter(fc.col("name").contains(fc.tool_param('query', StringType))) ... session.catalog.create_tool( tool_name="my_tool", tool_description="A tool that searches the name field", tool_query=parameterized_df, result_limit=100, tool_params=[ToolParam(name="query", description="The name should contain the following value")] ) Example: A tool with multiple filters ```python # Assume we are reading data with an `age` column. df = session.read.csv(users.csv) # create multiple filters that evaluate to true if a param is not passed. optional_min = fc.coalesce(fc.col("age") >= tool_param("min_age", IntegerType), fc.lit(True)) optional_max = fc.coalesce(fc.col("age") <= tool_param("max_age", IntegerType), fc.lit(True)) core_filter = df.filter(optional_min & optional_max) session.catalog.create_tool( "users_filter", "Filter users by age", core_filter, tool_params=[ ToolParam(name="min_age", description="Minimum age", has_default=True, default_value=None), ToolParam(name="max_age", description="Maximum age", has_default=True, default_value=None), ] ) """ if isinstance(data_type, _LogicalType): raise ValidationError(f"Cannot use a logical type as a parameter type: {data_type}") return Column._from_logical_expr(UnresolvedLiteralExpr(data_type=data_type, parameter_name=parameter_name)) ``` --- # fenic.api.functions.dt Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/dt/ Date and time functions. Functions: - **`current_date`** – Get the current date. - **`current_timestamp`** – Get the current date and time. - **`date_add`** – Adds the number of days to the date/timestamp column. - **`date_format`** – Formats a date/timestamp column to a given format. - **`date_sub`** – Subtracts the number of days from the date/timestamp column. - **`date_trunc`** – Truncate a date to a given unit. - **`datediff`** – Calculates the number of days between two date/timestamp columns. - **`day`** – Extract the day from a day column. - **`from_utc_timestamp`** – Accepts a Column with [TimestampType] (UTC). For each row, converts the timestamp value to the provided `tz` timezone, then renders that timestamp as UTC without changing the timestamp value. In other words, this function shifts the timestamp by the timezone offset `out = t+offset(t+tz)`. - **`hour`** – Extract the hour from a day column. - **`millisecond`** – Extract the hour from a millisecond column. - **`minute`** – Extract the minute from a day column. - **`month`** – Extract the month from a month column. - **`now`** – Get the current date and time. - **`second`** – Extract the hour from a second column. - **`timestamp_add`** – Adds the quantity of the given unit to the timestamp column. - **`timestamp_diff`** – Calculates the difference between two timestamp columns. - **`to_date`** – Transform a string into a DateType. - **`to_timestamp`** – Transform a string into a TimestampType. - **`to_utc_timestamp`** – Accepts a Column with [TimestampType] (UTC), interprets each value as wall-clock time in the specified timezone `tz`, and converts it to a timestamp in UTC. - **`year`** – Extract the year from a date column. ## current_date ``` current_date() -> Column ``` Get the current date. Returns: - `Column` – A Column object with the current date. - `Column` – The type of the column is DateType. Example ``` df.select(dt.current_date().alias("cur_date")).to_pydict() # Output: {'cur_date': [datetime.date(2025, 9, 26)]} ``` Source code in `src/fenic/api/functions/dt.py` ``` def current_date() -> Column: """Get the current date. Returns: A Column object with the current date. The type of the column is DateType. Example: ```python df.select(dt.current_date().alias("cur_date")).to_pydict() # Output: {'cur_date': [datetime.date(2025, 9, 26)]} ``` """ return Column._from_logical_expr(NowExpr(as_date=True)) ``` ## current_timestamp ``` current_timestamp() -> Column ``` Get the current date and time. Returns: - `Column` – A Column object with the current date and time. - `Column` – The type of the column is TimestampType in UTC timezone. Example ``` df.select(dt.current_timestamp().alias("cur_ts")).to_pydict() # Output: {'cur_ts': [datetime.datetime(2025, 9, 26, 10, 0)]} ``` Source code in `src/fenic/api/functions/dt.py` ``` def current_timestamp() -> Column: """Get the current date and time. Returns: A Column object with the current date and time. The type of the column is TimestampType in UTC timezone. Example: ```python df.select(dt.current_timestamp().alias("cur_ts")).to_pydict() # Output: {'cur_ts': [datetime.datetime(2025, 9, 26, 10, 0)]} ``` """ return Column._from_logical_expr(NowExpr()) ``` ## date_add ``` date_add(column: ColumnOrName, days: Union[int, ColumnOrName]) -> Column ``` Adds the number of days to the date/timestamp column. Parameters: - **`column`** (`ColumnOrName`) – The column to add the days to. - **`days`** (`Union[int, ColumnOrName]`) – The number of days to add to the date/timestamp column. If the days is negative, the days will be subtracted. Returns: - `Column` – A Column object with the date/timestamp column with the days added. Raises: - `TypeError` – If column type is not a DateType or TimestampType, or if days is not an IntegerType. Example ``` # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_add(col("date"), 1).alias("date_add")).to_pydict() # Output: {'date_add': ['2025-01-02', '2025-02-02', '2025-03-02']} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def date_add(column: ColumnOrName, days: Union[int, ColumnOrName]) -> Column: """Adds the number of days to the date/timestamp column. Args: column: The column to add the days to. days: The number of days to add to the date/timestamp column. If the days is negative, the days will be subtracted. Returns: A Column object with the date/timestamp column with the days added. Raises: TypeError: If column type is not a DateType or TimestampType, or if days is not an IntegerType. Example: ```python # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_add(col("date"), 1).alias("date_add")).to_pydict() # Output: {'date_add': ['2025-01-02', '2025-02-02', '2025-03-02']} ``` """ if isinstance(days, int): days_expr = LiteralExpr(days, IntegerType) else: days_expr = Column._from_col_or_name(days)._logical_expr return Column._from_logical_expr(DateAddExpr(Column._from_col_or_name(column)._logical_expr, days_expr)) ``` ## date_format ``` date_format(column: ColumnOrName, format: str) -> Column ``` Formats a date/timestamp column to a given format. Parameters: - **`column`** (`ColumnOrName`) – The column to format. - **`format`** (`str`) – The format to format the column to. Returns: - `Column` – A Column object with the date/timestamp column formatted into a string. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Notes - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example ``` # ts: "2025-01-01 10:00:00", "2025-02-01 11:00:00", "2025-03-01 15:00:00"] df.select(dt.date_format(col("date"), "MM-dd-yyyy hh:mm:ss a").alias("date")).to_pydict() # Output: {'date': ['01-01-2025 10:00:00 AM', '02-01-2025 11:00:00 AM', '03-01-2025 03:00:00 PM']} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def date_format(column: ColumnOrName, format: str) -> Column: """Formats a date/timestamp column to a given format. Args: column: The column to format. format: The format to format the column to. Returns: A Column object with the date/timestamp column formatted into a string. Raises: TypeError: If column type is not a DateType or TimestampType. Notes: - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example: ```python # ts: "2025-01-01 10:00:00", "2025-02-01 11:00:00", "2025-03-01 15:00:00"] df.select(dt.date_format(col("date"), "MM-dd-yyyy hh:mm:ss a").alias("date")).to_pydict() # Output: {'date': ['01-01-2025 10:00:00 AM', '02-01-2025 11:00:00 AM', '03-01-2025 03:00:00 PM']} ``` """ return Column._from_logical_expr(DateFormatExpr(Column._from_col_or_name(column)._logical_expr, format)) ``` ## date_sub ``` date_sub(column: ColumnOrName, days: Union[int, ColumnOrName]) -> Column ``` Subtracts the number of days from the date/timestamp column. Parameters: - **`column`** (`ColumnOrName`) – The column to subtract the days from. - **`days`** (`Union[int, ColumnOrName]`) – The amount of days to subtract. If the days is negative, the days will be added. Returns: - `Column` – A Column object with the date/timestamp column with the days substracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType, or if days is not an IntegerType. Example ``` # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_sub(col("date"), 1).alias("date_sub")).to_pydict() # Output: {'date_sub': ['2024-12-31', '2025-01-31', '2025-02-28']} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def date_sub(column: ColumnOrName, days: Union[int, ColumnOrName]) -> Column: """Subtracts the number of days from the date/timestamp column. Args: column: The column to subtract the days from. days: The amount of days to subtract. If the days is negative, the days will be added. Returns: A Column object with the date/timestamp column with the days substracted. Raises: TypeError: If column type is not a DateType or TimestampType, or if days is not an IntegerType. Example: ```python # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_sub(col("date"), 1).alias("date_sub")).to_pydict() # Output: {'date_sub': ['2024-12-31', '2025-01-31', '2025-02-28']} ``` """ if isinstance(days, int): days_expr = LiteralExpr(days, IntegerType) else: days_expr = Column._from_col_or_name(days)._logical_expr return Column._from_logical_expr(DateAddExpr(Column._from_col_or_name(column)._logical_expr, days_expr, sub=True)) ``` ## date_trunc ``` date_trunc(column: ColumnOrName, unit: DateTimeUnit) -> Column ``` Truncate a date to a given unit. Parameters: - **`column`** (`ColumnOrName`) – The column to truncate. - **`unit`** (`DateTimeUnit`) – The unit to truncate to. Returns: - `Column` – A Column object with the date truncated. Raises: - `TypeError` – If column type is not a DateType or TimestampType. - `ValueError` – If unit is not supported, must be one of the supported ones. Notes The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example ``` # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_trunc(col("date"), "year").alias("date_trunc")).to_pydict() # Output: {'date_trunc': ['2025-01-01', '2025-01-01', '2025-01-01']} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def date_trunc(column: ColumnOrName, unit: DateTimeUnit) -> Column: """Truncate a date to a given unit. Args: column: The column to truncate. unit: The unit to truncate to. Returns: A Column object with the date truncated. Raises: TypeError: If column type is not a DateType or TimestampType. ValueError: If unit is not supported, must be one of the supported ones. Notes: The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example: ```python # dates: "2025-01-01", "2025-02-01", "2025-03-01"] df.select(dt.date_trunc(col("date"), "year").alias("date_trunc")).to_pydict() # Output: {'date_trunc': ['2025-01-01', '2025-01-01', '2025-01-01']} ``` """ return Column._from_logical_expr(DateTruncExpr(Column._from_col_or_name(column)._logical_expr, unit)) ``` ## datediff ``` datediff(end: ColumnOrName, start: ColumnOrName) -> Column ``` Calculates the number of days between two date/timestamp columns. Parameters: - **`end`** (`ColumnOrName`) – To date column to work on. - **`start`** (`ColumnOrName`) – From date column to work on. Returns: - `Column` – A Column object with the difference in days between the two date/timestamp columns. Example ``` # end: "2025-01-01", "2025-02-02", "2025-03-06"] # start: "2025-01-02", "2025-02-01", "2025-03-02"] df.select(dt.datediff(col("end"), col("start")).alias("date_diff")).to_pydict() # Output: {'date_diff': [-1, 1, 4]} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def datediff(end: ColumnOrName, start: ColumnOrName) -> Column: """Calculates the number of days between two date/timestamp columns. Args: end: To date column to work on. start: From date column to work on. Returns: A Column object with the difference in days between the two date/timestamp columns. Example: ```python # end: "2025-01-01", "2025-02-02", "2025-03-06"] # start: "2025-01-02", "2025-02-01", "2025-03-02"] df.select(dt.datediff(col("end"), col("start")).alias("date_diff")).to_pydict() # Output: {'date_diff': [-1, 1, 4]} ``` """ return Column._from_logical_expr( DateDiffExpr( Column._from_col_or_name(end)._logical_expr, Column._from_col_or_name(start)._logical_expr)) ``` ## day ``` day(column: ColumnOrName) -> Column ``` Extract the day from a day column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the day from. Returns: - `Column` – A Column object with the day extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Example ``` # dates: "2025-01-01", "2025-01-02", "2025-01-03"] df.select(dt.day(col("date"))).to_pydict() # Output: [{'day': 1}, {'day': 2}, {'day': 3}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def day(column: ColumnOrName) -> Column: """Extract the day from a day column. Args: column: The column to extract the day from. Returns: A Column object with the day extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Example: ```python # dates: "2025-01-01", "2025-01-02", "2025-01-03"] df.select(dt.day(col("date"))).to_pydict() # Output: [{'day': 1}, {'day': 2}, {'day': 3}] ``` """ return Column._from_logical_expr(DayExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## from_utc_timestamp ``` from_utc_timestamp(column: ColumnOrName, tz: str) -> Column ``` Accepts a Column with [TimestampType] (UTC). For each row, converts the timestamp value to the provided `tz` timezone, then renders that timestamp as UTC without changing the timestamp value. In other words, this function shifts the timestamp by the timezone offset `out = t+offset(t+tz)`. Parameters: - **`column`** (`ColumnOrName`) – The column containing the timestamp. - **`tz`** (`str`) – A timezone that the input will be converted from. Returns: - `Column` – A Column object with timestamp expressed in UTC. Raises: - `TypeError` – If column type is not a TimestampType. Notes - In fenic, the TimestampType data type is always in UTC, which is not timezone-agnostic. - Spark also represents all timestamps as not timezone-agnostic, except Spark uses a timestamp type with the local session timezone. - Similarly to Spark from_utc_timestamp function, this function will take a UTC timestamp and convert it to the requested timezone, then represent it as a timestamp in the session (UTC) timezone. - nulls are preserved. - Use when applying `tz` to timestamps in UTC and the resulting wall‑clock timestamp is required (though the result is still stored with a UTC timezone). - Use to_utc_timestamp for the inverse (treats naive/local values in tz, then converts to UTC). - see Spark documentation for more details: https://spark.apache.org/docs/4.0.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.from_utc_timestamp.html# Example ``` df.select("timestamp", dt.to_timestamp(col("timestamp"), "yyyy-MM-dd HH:mm:ss").alias("utc_time")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ timestamp ┆ utc_time β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-15 10:30:00 ┆ 2025-01-15 10:30:00 UTC β”‚ #β”‚ 2025-01-16 14:00:00 ┆ 2025-01-16 14:00:00 UTC β”‚ #β”‚ 2025-01-17 18:45:00 ┆ 2025-01-17 18:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ # df.select(dt.from_utc_timestamp(col("utc_time"), "America/Los_Angeles").alias("la_time_in_utc")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ la_time_in_utc β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 02:30:00 UTC β”‚ #β”‚ 2025-01-16 06:00:00 UTC β”‚ #β”‚ 2025-01-17 10:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def from_utc_timestamp(column: ColumnOrName, tz: str) -> Column: """Accepts a Column with [TimestampType] (UTC). For each row, converts the timestamp value to the provided `tz` timezone, then renders that timestamp as UTC without changing the timestamp value. In other words, this function shifts the timestamp by the timezone offset `out = t+offset(t+tz)`. Args: column: The column containing the timestamp. tz: A timezone that the input will be converted from. Returns: A Column object with timestamp expressed in UTC. Raises: TypeError: If column type is not a TimestampType. Notes: - In fenic, the TimestampType data type is always in UTC, which is not timezone-agnostic. - Spark also represents all timestamps as not timezone-agnostic, except Spark uses a timestamp type with the local session timezone. - Similarly to Spark from_utc_timestamp function, this function will take a UTC timestamp and convert it to the requested timezone, then represent it as a timestamp in the session (UTC) timezone. - nulls are preserved. - Use when applying `tz` to timestamps in UTC and the resulting wall‑clock timestamp is required (though the result is still stored with a UTC timezone). - Use to_utc_timestamp for the inverse (treats naive/local values in tz, then converts to UTC). - see Spark documentation for more details: https://spark.apache.org/docs/4.0.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.from_utc_timestamp.html# Example: ```python df.select("timestamp", dt.to_timestamp(col("timestamp"), "yyyy-MM-dd HH:mm:ss").alias("utc_time")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ timestamp ┆ utc_time β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-15 10:30:00 ┆ 2025-01-15 10:30:00 UTC β”‚ #β”‚ 2025-01-16 14:00:00 ┆ 2025-01-16 14:00:00 UTC β”‚ #β”‚ 2025-01-17 18:45:00 ┆ 2025-01-17 18:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ # df.select(dt.from_utc_timestamp(col("utc_time"), "America/Los_Angeles").alias("la_time_in_utc")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ la_time_in_utc β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 02:30:00 UTC β”‚ #β”‚ 2025-01-16 06:00:00 UTC β”‚ #β”‚ 2025-01-17 10:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` """ return Column._from_logical_expr( FromUTCTimestampExpr( Column._from_col_or_name(column)._logical_expr, tz)) ``` ## hour ``` hour(column: ColumnOrName) -> Column ``` Extract the hour from a day column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the hour from. Returns: - `Column` – A Column object with the hour extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Notes This will return 0 for DateType columns. Example ``` # ts: "2025-01-01 10:00:00", "2025-01-02 11:00:00", "2025-01-03 12:00:00"] df.select(dt.hour(col("ts"))).to_pydict() # Output: [{'hour': 10}, {'hour': 11}, {'hour': 12}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def hour(column: ColumnOrName) -> Column: """Extract the hour from a day column. Args: column: The column to extract the hour from. Returns: A Column object with the hour extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Notes: This will return 0 for DateType columns. Example: ```python # ts: "2025-01-01 10:00:00", "2025-01-02 11:00:00", "2025-01-03 12:00:00"] df.select(dt.hour(col("ts"))).to_pydict() # Output: [{'hour': 10}, {'hour': 11}, {'hour': 12}] ``` """ return Column._from_logical_expr(HourExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## millisecond ``` millisecond(column: ColumnOrName) -> Column ``` Extract the hour from a millisecond column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the millisecond from. Returns: - `Column` – A Column object with the millisecond extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Notes This will return 0 for DateType columns. Example ``` # ts: "2025-01-01 10:10:01.123", "2025-01-02 11:11:02.234", "2025-01-03 12:12:03.345"] df.select(dt.millisecond(col("ts"))).to_pydict() # Output: [{'millisecond': 123}, {'millisecond': 234}, {'millisecond': 345}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def millisecond(column: ColumnOrName) -> Column: """Extract the hour from a millisecond column. Args: column: The column to extract the millisecond from. Returns: A Column object with the millisecond extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Notes: This will return 0 for DateType columns. Example: ```python # ts: "2025-01-01 10:10:01.123", "2025-01-02 11:11:02.234", "2025-01-03 12:12:03.345"] df.select(dt.millisecond(col("ts"))).to_pydict() # Output: [{'millisecond': 123}, {'millisecond': 234}, {'millisecond': 345}] ``` """ return Column._from_logical_expr(MilliSecondExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## minute ``` minute(column: ColumnOrName) -> Column ``` Extract the minute from a day column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the minute from. Returns: - `Column` – A Column object with the minute extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Notes This will return 0 for DateType columns. Example ``` # ts: "2025-01-01 10:10:00", "2025-01-02 11:11:00", "2025-01-03 12:12:00"] df.select(dt.minute(col("ts"))).to_pydict() # Output: [{'minute': 10}, {'minute': 11}, {'minute': 12}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def minute(column: ColumnOrName) -> Column: """Extract the minute from a day column. Args: column: The column to extract the minute from. Returns: A Column object with the minute extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Notes: This will return 0 for DateType columns. Example: ```python # ts: "2025-01-01 10:10:00", "2025-01-02 11:11:00", "2025-01-03 12:12:00"] df.select(dt.minute(col("ts"))).to_pydict() # Output: [{'minute': 10}, {'minute': 11}, {'minute': 12}] ``` """ return Column._from_logical_expr(MinuteExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## month ``` month(column: ColumnOrName) -> Column ``` Extract the month from a month column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the month from. Returns: - `Column` – A Column object with the month extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Example ``` # dates: "2025-01-01", "2025-01-02", "2024-12-03"] df.select(dt.month(col("date"))).to_pydict() # Output: [{'month': 1}, {'month': 1}, {'month': 12}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def month(column: ColumnOrName) -> Column: """Extract the month from a month column. Args: column: The column to extract the month from. Returns: A Column object with the month extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Example: ```python # dates: "2025-01-01", "2025-01-02", "2024-12-03"] df.select(dt.month(col("date"))).to_pydict() # Output: [{'month': 1}, {'month': 1}, {'month': 12}] ``` """ return Column._from_logical_expr(MonthExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## now ``` now() -> Column ``` Get the current date and time. Returns: - `Column` – A Column object with the current date and time. - `Column` – The type of the column is TimestampType. Example ``` df.select(dt.now()).to_pydict() # Output: [{'date': ''}] ``` Source code in `src/fenic/api/functions/dt.py` ``` def now() -> Column: """Get the current date and time. Returns: A Column object with the current date and time. The type of the column is TimestampType. Example: ```python df.select(dt.now()).to_pydict() # Output: [{'date': ''}] ``` """ return Column._from_logical_expr(NowExpr()) ``` ## second ``` second(column: ColumnOrName) -> Column ``` Extract the hour from a second column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the second from. Returns: - `Column` – A Column object with the second extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Notes This will return 0 for DateType columns. Example ``` # ts: "2025-01-01 10:10:01", "2025-01-02 11:11:02", "2025-01-03 12:12:03"] df.select(dt.second(col("ts"))).to_pydict() # Output: [{'second': 1}, {'second': 2}, {'second': 3}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def second(column: ColumnOrName) -> Column: """Extract the hour from a second column. Args: column: The column to extract the second from. Returns: A Column object with the second extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Notes: This will return 0 for DateType columns. Example: ```python # ts: "2025-01-01 10:10:01", "2025-01-02 11:11:02", "2025-01-03 12:12:03"] df.select(dt.second(col("ts"))).to_pydict() # Output: [{'second': 1}, {'second': 2}, {'second': 3}] ``` """ return Column._from_logical_expr(SecondExpr(Column._from_col_or_name(column)._logical_expr)) ``` ## timestamp_add ``` timestamp_add(column: ColumnOrName, quantity: Union[int, ColumnOrName], unit: DateTimeUnit) -> Column ``` Adds the quantity of the given unit to the timestamp column. Parameters: - **`column`** (`ColumnOrName`) – The column to add the quantity to. - **`quantity`** (`Union[int, ColumnOrName]`) – The quantity to add. If the quantity is negative, the quantity will be subtracted. - **`unit`** (`DateTimeUnit`) – The unit of the quantity. Returns: - `Column` – A Column object with the timestamp column with the quantity added. Raises: - `TypeError` – If column type is not a TimestampType, or if quantity is not an IntegerType. - `ValueError` – If unit is not supported, must be one of the supported ones. Notes The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example ``` # ts: "2025-01-01 10:00:00", "2025-02-01 11:00:00", "2025-03-01 12:00:00"] df.select(dt.timestamp_add(col("ts"), 1, "day").alias("ts_add")).to_pydict() # Output: {'ts_add': ['2025-01-02 10:00:00', '2025-02-02 11:00:00', '2025-03-02 12:00:00']} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def timestamp_add(column: ColumnOrName, quantity: Union[int, ColumnOrName], unit: DateTimeUnit) -> Column: """Adds the quantity of the given unit to the timestamp column. Args: column: The column to add the quantity to. quantity: The quantity to add. If the quantity is negative, the quantity will be subtracted. unit: The unit of the quantity. Returns: A Column object with the timestamp column with the quantity added. Raises: TypeError: If column type is not a TimestampType, or if quantity is not an IntegerType. ValueError: If unit is not supported, must be one of the supported ones. Notes: The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example: ```python # ts: "2025-01-01 10:00:00", "2025-02-01 11:00:00", "2025-03-01 12:00:00"] df.select(dt.timestamp_add(col("ts"), 1, "day").alias("ts_add")).to_pydict() # Output: {'ts_add': ['2025-01-02 10:00:00', '2025-02-02 11:00:00', '2025-03-02 12:00:00']} ``` """ if isinstance(quantity, int): quantity_expr = LiteralExpr(quantity, IntegerType) else: quantity_expr = Column._from_col_or_name(quantity)._logical_expr return Column._from_logical_expr(TimestampAddExpr(Column._from_col_or_name(column)._logical_expr, quantity_expr, unit)) ``` ## timestamp_diff ``` timestamp_diff(start: ColumnOrName, end: ColumnOrName, unit: DateTimeUnit) -> Column ``` Calculates the difference between two timestamp columns. Parameters: - **`start`** (`ColumnOrName`) – The first column to calculate the difference from. - **`end`** (`ColumnOrName`) – The second column to calculate the difference from. - **`unit`** (`DateTimeUnit`) – The unit of the difference. Returns: - `Column` – A Column object with the difference in the given unit between the two timestamp columns. Raises: - `ValueError` – If unit is not supported, must be one of the supported ones. Notes The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example ``` # start: "2025-01-01 10:00:00", "2025-02-02 11:00:00", "2025-03-06 12:00:00"] # end: "2025-01-02 10:00:00", "2025-02-01 11:00:00", "2025-03-01 12:00:00"] df.select(dt.timestamp_diff(col("start"), col("end"), "day").alias("ts_diff")).to_pydict() # Output: {'ts_diff': [-1, 1, 5]} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def timestamp_diff(start: ColumnOrName, end: ColumnOrName, unit: DateTimeUnit) -> Column: """Calculates the difference between two timestamp columns. Args: start: The first column to calculate the difference from. end: The second column to calculate the difference from. unit: The unit of the difference. Returns: A Column object with the difference in the given unit between the two timestamp columns. Raises: ValueError: If unit is not supported, must be one of the supported ones. Notes: The supported units are: "year", "month", "day", "hour", "minute", "second", "millisecond". Example: ```python # start: "2025-01-01 10:00:00", "2025-02-02 11:00:00", "2025-03-06 12:00:00"] # end: "2025-01-02 10:00:00", "2025-02-01 11:00:00", "2025-03-01 12:00:00"] df.select(dt.timestamp_diff(col("start"), col("end"), "day").alias("ts_diff")).to_pydict() # Output: {'ts_diff': [-1, 1, 5]} ``` """ return Column._from_logical_expr( TimestampDiffExpr( Column._from_col_or_name(start)._logical_expr, Column._from_col_or_name(end)._logical_expr, unit)) ``` ## to_date ``` to_date(column: ColumnOrName, format: Optional[str] = None) -> Column ``` Transform a string into a DateType. Parameters: - **`column`** (`ColumnOrName`) – The column to transform into a DateType. - **`format`** (`Optional[str]`, default: `None` ) – The format of the date string. Returns: - `Column` – A Column object with the DateType transformed. Raises: - `TypeError` – If column type is not a StringType. Notes - If format is not provided, the default format is "YYYY-MM-DD". - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example ``` # date_str: "11-01-2025", "12-02-2025", "01-03-2025"] df.select(to_date(col("date_str"), format="MM-dd-yyyy").alias("date")).to_pydict() # Output: {'date': [datetime.datetime(2025, 11, 1, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 12, 2, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 1, 3, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC'))]} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def to_date(column: ColumnOrName, format: Optional[str] = None) -> Column: """Transform a string into a DateType. Args: column: The column to transform into a DateType. format: The format of the date string. Returns: A Column object with the DateType transformed. Raises: TypeError: If column type is not a StringType. Notes: - If format is not provided, the default format is "YYYY-MM-DD". - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example: ```python # date_str: "11-01-2025", "12-02-2025", "01-03-2025"] df.select(to_date(col("date_str"), format="MM-dd-yyyy").alias("date")).to_pydict() # Output: {'date': [datetime.datetime(2025, 11, 1, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 12, 2, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 1, 3, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC'))]} ``` """ return Column._from_logical_expr(ToDateExpr(Column._from_col_or_name(column)._logical_expr, format)) ``` ## to_timestamp ``` to_timestamp(column: ColumnOrName, format: Optional[str] = None) -> Column ``` Transform a string into a TimestampType. Parameters: - **`column`** (`ColumnOrName`) – The column to transform into a TimestampType. - **`format`** (`Optional[str]`, default: `None` ) – The format of the timestamp string. Returns: - `Column` – A Column object with the `TimestampType` type, with a UTC timezone. If the provided `format` contains a timezone specifier, the result timestamp value will be converted from the `format` timezone to UTC. Raises: - `TypeError` – If column type is not a StringType. Notes - If format is not provided, the default format is ISO 8601 with milliseconds. - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example ``` # date_str: ["11-01-2025 10:00:00", "12-02-2025 11:00:00", "01-03-2025 12:00:00"] df.select(dt.to_date(col("date_str"), format="MM-dd-yyyy HH:mm:ss").alias("timestamp")).to_pydict() # Output: {'timestamp': [datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC'))]} ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def to_timestamp(column: ColumnOrName, format: Optional[str] = None) -> Column: """Transform a string into a TimestampType. Args: column: The column to transform into a TimestampType. format: The format of the timestamp string. Returns: A Column object with the `TimestampType` type, with a UTC timezone. If the provided `format` contains a timezone specifier, the result timestamp value will be converted from the `format` timezone to UTC. Raises: TypeError: If column type is not a StringType. Notes: - If format is not provided, the default format is ISO 8601 with milliseconds. - The accepted formats should follow this pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html Example: ```python # date_str: ["11-01-2025 10:00:00", "12-02-2025 11:00:00", "01-03-2025 12:00:00"] df.select(dt.to_date(col("date_str"), format="MM-dd-yyyy HH:mm:ss").alias("timestamp")).to_pydict() # Output: {'timestamp': [datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC')), datetime.datetime(2025, 11, 1, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='UTC'))]} ``` """ return Column._from_logical_expr(ToTimestampExpr(Column._from_col_or_name(column)._logical_expr, format)) ``` ## to_utc_timestamp ``` to_utc_timestamp(column: ColumnOrName, tz: str) -> Column ``` Accepts a Column with [TimestampType] (UTC), interprets each value as wall-clock time in the specified timezone `tz`, and converts it to a timestamp in UTC. Parameters: - **`column`** (`ColumnOrName`) – The column containing the timestamp. Will be treated as timezone-agnostic. - **`tz`** (`str`) – A timezone that the input should be converted to. Returns: - `Column` – A Column object with timestamp expressed in UTC. Raises: - `TypeError` – If column type is not a TimestampType. Notes - In fenic, the TimestampType data type is always in UTC, which is not timezone-agnostic. - Spark also represents all timestamps as not timezone-agnostic, except Spark uses a timestamp type with the local session timezone. - Similarly to Spark to_utc_timestamp function, this function treats the input timestamp as timezone-agnostic, sets it to the requested timezone (without changing the timestamp), then converts the timestamp to UTC. - nulls are preserved. - Use this when data contains local/wall‑clock timestamps from `tz` (ignoring the UTC timezone in the type), and timestamp values converted to UTC are required. - For the inverse operation (UTC β†’ local wall‑clock, then re‑expressed in UTC), see from_utc_timestamp. - see Spark documentation for more details: https://spark.apache.org/docs/4.0.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.to_utc_timestamp.html Example ``` df.select("timestamp", dt.to_timestamp(col("timestamp"), "yyyy-MM-dd HH:mm:ss").alias("la_time")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ timestamp ┆ la_time β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-15 10:30:00 ┆ 2025-01-15 10:30:00 UTC β”‚ #β”‚ 2025-01-16 14:00:00 ┆ 2025-01-16 14:00:00 UTC β”‚ #β”‚ 2025-01-17 18:45:00 ┆ 2025-01-17 18:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ # df.select(dt.to_utc_timestamp(col("la_time"), "America/Los_Angeles").alias("la_time_to_utc")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ la_time_to_utc β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 18:30:00 UTC β”‚ #β”‚ 2025-01-16 22:00:00 UTC β”‚ #β”‚ 2025-01-18 02:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def to_utc_timestamp(column: ColumnOrName, tz: str) -> Column: """Accepts a Column with [TimestampType] (UTC), interprets each value as wall-clock time in the specified timezone `tz`, and converts it to a timestamp in UTC. Args: column: The column containing the timestamp. Will be treated as timezone-agnostic. tz: A timezone that the input should be converted to. Returns: A Column object with timestamp expressed in UTC. Raises: TypeError: If column type is not a TimestampType. Notes: - In fenic, the TimestampType data type is always in UTC, which is not timezone-agnostic. - Spark also represents all timestamps as not timezone-agnostic, except Spark uses a timestamp type with the local session timezone. - Similarly to Spark to_utc_timestamp function, this function treats the input timestamp as timezone-agnostic, sets it to the requested timezone (without changing the timestamp), then converts the timestamp to UTC. - nulls are preserved. - Use this when data contains local/wall‑clock timestamps from `tz` (ignoring the UTC timezone in the type), and timestamp values converted to UTC are required. - For the inverse operation (UTC β†’ local wall‑clock, then re‑expressed in UTC), see from_utc_timestamp. - see Spark documentation for more details: https://spark.apache.org/docs/4.0.1/api/python/reference/pyspark.sql/api/pyspark.sql.functions.to_utc_timestamp.html Example: ```python df.select("timestamp", dt.to_timestamp(col("timestamp"), "yyyy-MM-dd HH:mm:ss").alias("la_time")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ timestamp ┆ la_time β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ═════════════════════════║ #β”‚ 2025-01-15 10:30:00 ┆ 2025-01-15 10:30:00 UTC β”‚ #β”‚ 2025-01-16 14:00:00 ┆ 2025-01-16 14:00:00 UTC β”‚ #β”‚ 2025-01-17 18:45:00 ┆ 2025-01-17 18:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ # df.select(dt.to_utc_timestamp(col("la_time"), "America/Los_Angeles").alias("la_time_to_utc")).show() # Output: #β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” #β”‚ la_time_to_utc β”‚ #β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•‘ #β”‚ 2025-01-15 18:30:00 UTC β”‚ #β”‚ 2025-01-16 22:00:00 UTC β”‚ #β”‚ 2025-01-18 02:45:00 UTC β”‚ #β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` """ return Column._from_logical_expr( ToUTCTimestampExpr( Column._from_col_or_name(column)._logical_expr, tz)) ``` ## year ``` year(column: ColumnOrName) -> Column ``` Extract the year from a date column. Parameters: - **`column`** (`ColumnOrName`) – The column to extract the year from. Returns: - `Column` – A Column object with the year extracted. Raises: - `TypeError` – If column type is not a DateType or TimestampType. Example ``` # dates: "2025-01-01", "2025-01-02", "2025-01-03"] df.select(dt.year(col("date"))).to_pydict() # Output: [{'year': 2025}, {'year': 2025}, {'year': 2025}] ``` Source code in `src/fenic/api/functions/dt.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def year(column: ColumnOrName) -> Column: """Extract the year from a date column. Args: column: The column to extract the year from. Returns: A Column object with the year extracted. Raises: TypeError: If column type is not a DateType or TimestampType. Example: ```python # dates: "2025-01-01", "2025-01-02", "2025-01-03"] df.select(dt.year(col("date"))).to_pydict() # Output: [{'year': 2025}, {'year': 2025}, {'year': 2025}] ``` """ return Column._from_logical_expr(YearExpr(Column._from_col_or_name(column)._logical_expr)) ``` --- # fenic.api.functions.embedding Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/embedding/ Embedding functions. Functions: - **`compute_similarity`** – Compute similarity between embedding vectors using specified metric. - **`normalize`** – Normalize embedding vectors to unit length. ## compute_similarity ``` compute_similarity(column: ColumnOrName, other: Union[ColumnOrName, List[float], ndarray], metric: SemanticSimilarityMetric = 'cosine') -> Column ``` Compute similarity between embedding vectors using specified metric. Parameters: - **`column`** (`ColumnOrName`) – Column containing embedding vectors. - **`other`** (`Union[ColumnOrName, List[float], ndarray]`) – Either: - Another column containing embedding vectors for pairwise similarity - A query vector (list of floats or numpy array) for similarity with each embedding - **`metric`** (`SemanticSimilarityMetric`, default: `'cosine'` ) – The similarity metric to use. Options: - `cosine`: Cosine similarity (range: -1 to 1, higher is more similar) - `dot`: Dot product similarity (raw inner product) - `l2`: L2 (Euclidean) distance (lower is more similar) Returns: - **`Column`** ( `Column` ) – A column of float values representing similarity scores. Raises: - `ValidationError` – If query vector contains NaN values or has invalid dimensions. Notes - Cosine similarity normalizes vectors internally, so pre-normalization is not required - Dot product does not normalize, useful when vectors are already normalized - L2 distance measures the straight-line distance between vectors - When using two columns, dimensions must match between embeddings Compute dot product with a query vector ``` # Compute dot product with a query vector query = [0.1, 0.2, 0.3] df.select( embedding.compute_similarity(col("embeddings"), query).alias("similarity") ) ``` Compute cosine similarity with a query vector ``` query = [0.6, ... 0.8] # Already normalized df.select( embedding.compute_similarity(col("embeddings"), query, metric="cosine").alias("cosine_sim") ) ``` Compute pairwise dot products between columns ``` # Compute L2 distance between two columns of embeddings df.select( embedding.compute_similarity(col("embeddings1"), col("embeddings2"), metric="l2").alias("distance") ) ``` Using numpy array as query vector ``` # Use numpy array as query vector import numpy as np query = np.array([0.1, 0.2, 0.3]) df.select(embedding.compute_similarity("embeddings", query)) ``` Source code in `src/fenic/api/functions/embedding.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def compute_similarity( column: ColumnOrName, other: Union[ColumnOrName, List[float], np.ndarray], metric: SemanticSimilarityMetric = "cosine" ) -> Column: """Compute similarity between embedding vectors using specified metric. Args: column: Column containing embedding vectors. other: Either: - Another column containing embedding vectors for pairwise similarity - A query vector (list of floats or numpy array) for similarity with each embedding metric: The similarity metric to use. Options: - `cosine`: Cosine similarity (range: -1 to 1, higher is more similar) - `dot`: Dot product similarity (raw inner product) - `l2`: L2 (Euclidean) distance (lower is more similar) Returns: Column: A column of float values representing similarity scores. Raises: ValidationError: If query vector contains NaN values or has invalid dimensions. Notes: - Cosine similarity normalizes vectors internally, so pre-normalization is not required - Dot product does not normalize, useful when vectors are already normalized - L2 distance measures the straight-line distance between vectors - When using two columns, dimensions must match between embeddings Example: Compute dot product with a query vector ```python # Compute dot product with a query vector query = [0.1, 0.2, 0.3] df.select( embedding.compute_similarity(col("embeddings"), query).alias("similarity") ) ``` Example: Compute cosine similarity with a query vector ```python query = [0.6, ... 0.8] # Already normalized df.select( embedding.compute_similarity(col("embeddings"), query, metric="cosine").alias("cosine_sim") ) ``` Example: Compute pairwise dot products between columns ```python # Compute L2 distance between two columns of embeddings df.select( embedding.compute_similarity(col("embeddings1"), col("embeddings2"), metric="l2").alias("distance") ) ``` Example: Using numpy array as query vector ```python # Use numpy array as query vector import numpy as np query = np.array([0.1, 0.2, 0.3]) df.select(embedding.compute_similarity("embeddings", query)) ``` """ column_expr = Column._from_col_or_name(column)._logical_expr # Check if other is a column if isinstance(other, ColumnOrName): other_expr = Column._from_col_or_name(other)._logical_expr return Column._from_logical_expr( EmbeddingSimilarityExpr(column_expr, other_expr, metric) ) # Otherwise it's a query vector if isinstance(other, list): query_array = np.array(other, dtype=np.float32) else: query_array = other.astype(np.float32) # Check for NaNs if np.any(np.isnan(query_array)): raise ValidationError("Query vector cannot contain NaN values") return Column._from_logical_expr( EmbeddingSimilarityExpr(column_expr, query_array, metric) ) ``` ## normalize ``` normalize(column: ColumnOrName) -> Column ``` Normalize embedding vectors to unit length. Parameters: - **`column`** (`ColumnOrName`) – Column containing embedding vectors. Returns: - **`Column`** ( `Column` ) – A column of normalized embedding vectors with the same embedding type. Notes - Normalizes each embedding vector to have unit length (L2 norm = 1) - Preserves the original embedding model in the type - Null values are preserved as null - Zero vectors become NaN after normalization Normalize embeddings for dot product similarity ``` # Normalize embeddings for dot product similarity comparisons df.select( embedding.normalize(col("embeddings")).alias("unit_embeddings") ) ``` Compare normalized embeddings using dot product ``` # Compare normalized embeddings using dot product (equivalent to cosine similarity) normalized_df = df.select(embedding.normalize(col("embeddings")).alias("norm_emb")) query = [0.6, 0.8] # Already normalized normalized_df.select( embedding.compute_similarity(col("norm_emb"), query, metric="dot").alias("dot_product_sim") ) ``` Source code in `src/fenic/api/functions/embedding.py` ``` def normalize(column: ColumnOrName) -> Column: """Normalize embedding vectors to unit length. Args: column: Column containing embedding vectors. Returns: Column: A column of normalized embedding vectors with the same embedding type. Notes: - Normalizes each embedding vector to have unit length (L2 norm = 1) - Preserves the original embedding model in the type - Null values are preserved as null - Zero vectors become NaN after normalization Example: Normalize embeddings for dot product similarity ```python # Normalize embeddings for dot product similarity comparisons df.select( embedding.normalize(col("embeddings")).alias("unit_embeddings") ) ``` Example: Compare normalized embeddings using dot product ```python # Compare normalized embeddings using dot product (equivalent to cosine similarity) normalized_df = df.select(embedding.normalize(col("embeddings")).alias("norm_emb")) query = [0.6, 0.8] # Already normalized normalized_df.select( embedding.compute_similarity(col("norm_emb"), query, metric="dot").alias("dot_product_sim") ) ``` """ column_expr = Column._from_col_or_name(column)._logical_expr return Column._from_logical_expr(EmbeddingNormalizeExpr(column_expr)) ``` --- # fenic.api.functions.json Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/json/ JSON functions. Functions: - **`contains`** – Check if a JSON value contains the specified value using recursive deep search. - **`get_type`** – Get the JSON type of each value. - **`jq`** – Applies a JQ query to a column containing JSON-formatted strings. ## contains ``` contains(column: ColumnOrName, value: str) -> Column ``` Check if a JSON value contains the specified value using recursive deep search. Parameters: - **`column`** (`ColumnOrName`) – Input column of type `JsonType`. - **`value`** (`str`) – Valid JSON string to search for. Returns: - **`Column`** ( `Column` ) – A column of booleans indicating whether the JSON contains the value. Matching Rules - **Objects**: Uses partial matching - `{"role": "admin"}` matches `{"role": "admin", "level": 5}` - **Arrays**: Uses exact matching - `[1, 2]` only matches exactly `[1, 2]`, not `[1, 2, 3]` - **Primitives**: Uses exact matching - `42` matches `42` but not `"42"` - **Search is recursive**: Searches at all nesting levels throughout the JSON structure - **Type-aware**: Distinguishes between `42` (number) and `"42"` (string) Find objects with partial structure match ``` # Find objects with partial structure match (at any nesting level) df.select(json.contains(col("json_data"), '{"name": "Alice"}')) # Matches: {"name": "Alice", "age": 30} and {"user": {"name": "Alice"}} ``` Find exact array match ``` # Find exact array match (at any nesting level) df.select(json.contains(col("json_data"), '["read", "write"]')) # Matches: {"permissions": ["read", "write"]} but not ["read", "write", "admin"] ``` Find exact primitive values ``` # Find exact primitive values (at any nesting level) df.select(json.contains(col("json_data"), '"admin"')) # Matches: {"role": "admin"} and ["admin", "user"] but not {"role": "administrator"} ``` Type distinction matters ``` # Type distinction matters df.select(json.contains(col("json_data"), '42')) # number 42 df.select(json.contains(col("json_data"), '"42"')) # string "42" ``` Raises: - `ValidationError` – If `value` is not valid JSON. Source code in `src/fenic/api/functions/json.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def contains(column: ColumnOrName, value: str) -> Column: """Check if a JSON value contains the specified value using recursive deep search. Args: column (ColumnOrName): Input column of type `JsonType`. value (str): Valid JSON string to search for. Returns: Column: A column of booleans indicating whether the JSON contains the value. Matching Rules: - **Objects**: Uses partial matching - `{"role": "admin"}` matches `{"role": "admin", "level": 5}` - **Arrays**: Uses exact matching - `[1, 2]` only matches exactly `[1, 2]`, not `[1, 2, 3]` - **Primitives**: Uses exact matching - `42` matches `42` but not `"42"` - **Search is recursive**: Searches at all nesting levels throughout the JSON structure - **Type-aware**: Distinguishes between `42` (number) and `"42"` (string) Example: Find objects with partial structure match ```python # Find objects with partial structure match (at any nesting level) df.select(json.contains(col("json_data"), '{"name": "Alice"}')) # Matches: {"name": "Alice", "age": 30} and {"user": {"name": "Alice"}} ``` Example: Find exact array match ```python # Find exact array match (at any nesting level) df.select(json.contains(col("json_data"), '["read", "write"]')) # Matches: {"permissions": ["read", "write"]} but not ["read", "write", "admin"] ``` Example: Find exact primitive values ```python # Find exact primitive values (at any nesting level) df.select(json.contains(col("json_data"), '"admin"')) # Matches: {"role": "admin"} and ["admin", "user"] but not {"role": "administrator"} ``` Example: Type distinction matters ```python # Type distinction matters df.select(json.contains(col("json_data"), '42')) # number 42 df.select(json.contains(col("json_data"), '"42"')) # string "42" ``` Raises: ValidationError: If `value` is not valid JSON. """ return Column._from_logical_expr( JsonContainsExpr(Column._from_col_or_name(column)._logical_expr, value) ) ``` ## get_type ``` get_type(column: ColumnOrName) -> Column ``` Get the JSON type of each value. Parameters: - **`column`** (`ColumnOrName`) – Input column of type `JsonType`. Returns: - **`Column`** ( `Column` ) – A column of strings indicating the JSON type ("string", "number", "boolean", "array", "object", "null"). Get JSON types ``` df.select(json.get_type(col("json_data"))) ``` Filter by type ``` # Filter by type df.filter(json.get_type(col("data")) == "array") ``` Source code in `src/fenic/api/functions/json.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def get_type(column: ColumnOrName) -> Column: """Get the JSON type of each value. Args: column (ColumnOrName): Input column of type `JsonType`. Returns: Column: A column of strings indicating the JSON type ("string", "number", "boolean", "array", "object", "null"). Example: Get JSON types ```python df.select(json.get_type(col("json_data"))) ``` Example: Filter by type ```python # Filter by type df.filter(json.get_type(col("data")) == "array") ``` """ return Column._from_logical_expr( JsonTypeExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## jq ``` jq(column: ColumnOrName, query: str) -> Column ``` Applies a JQ query to a column containing JSON-formatted strings. Parameters: - **`column`** (`ColumnOrName`) – Input column of type `JsonType`. - **`query`** (`str`) – A [JQ](https://jqlang.org/) expression used to extract or transform values. Returns: - **`Column`** ( `Column` ) – A column containing the result of applying the JQ query to each row's JSON input. Notes - The input column *must* be of type `JsonType`. Use `cast(JsonType)` if needed to ensure correct typing. - This function supports extracting nested fields, transforming arrays/objects, and other standard JQ operations. Extract nested field ``` # Extract the "user.name" field from a JSON column df.select(json.jq(col("json_col"), ".user.name")) ``` Cast to JsonType before querying ``` df.select(json.jq(col("raw_json").cast(JsonType), ".event.type")) ``` Work with arrays ``` # Work with arrays using JQ functions df.select(json.jq(col("json_array"), "map(.id)")) ``` Source code in `src/fenic/api/functions/json.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def jq(column: ColumnOrName, query: str) -> Column: """Applies a JQ query to a column containing JSON-formatted strings. Args: column (ColumnOrName): Input column of type `JsonType`. query (str): A [JQ](https://jqlang.org/) expression used to extract or transform values. Returns: Column: A column containing the result of applying the JQ query to each row's JSON input. Notes: - The input column *must* be of type `JsonType`. Use `cast(JsonType)` if needed to ensure correct typing. - This function supports extracting nested fields, transforming arrays/objects, and other standard JQ operations. Example: Extract nested field ```python # Extract the "user.name" field from a JSON column df.select(json.jq(col("json_col"), ".user.name")) ``` Example: Cast to JsonType before querying ```python df.select(json.jq(col("raw_json").cast(JsonType), ".event.type")) ``` Example: Work with arrays ```python # Work with arrays using JQ functions df.select(json.jq(col("json_array"), "map(.id)")) ``` """ return Column._from_logical_expr( JqExpr(Column._from_col_or_name(column)._logical_expr, query) ) ``` --- # fenic.api.functions.markdown Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/markdown/ Markdown functions. Functions: - **`extract_header_chunks`** – Splits markdown documents into logical chunks based on heading hierarchy. - **`generate_toc`** – Generates a table of contents from markdown headings. - **`get_code_blocks`** – Extracts all code blocks from a column of Markdown-formatted strings. - **`to_json`** – Converts a column of Markdown-formatted strings into a hierarchical JSON representation. ## extract_header_chunks ``` extract_header_chunks(column: ColumnOrName, header_level: int) -> Column ``` Splits markdown documents into logical chunks based on heading hierarchy. Parameters: - **`column`** (`ColumnOrName`) – Input column containing Markdown strings. - **`header_level`** (`int`) – Heading level to split on (1-6). Creates a new chunk at every heading of this level, including all nested content and subsections. Returns: - **`Column`** ( `Column` ) – A column of arrays containing chunk objects with the following structure: ``` ArrayType(StructType([ StructField("heading", StringType), # Heading text (clean, no markdown) StructField("level", IntegerType), # Heading level (1-6) StructField("content", StringType), # All content under this heading (clean text) StructField("parent_heading", StringType), # Parent heading text (or null) StructField("full_path", StringType), # Full breadcrumb path ])) ``` Notes: - **Context-preserving**: Each chunk contains all content and subsections under the heading - **Hierarchical awareness**: Includes parent heading context for better LLM understanding - **Clean text output**: Strips markdown formatting for direct LLM consumption Chunking Behavior With `header_level=2`, this markdown: ``` # Introduction Overview text ## Getting Started Setup instructions ### Prerequisites Python 3.8+ required ## API Reference Function documentation ``` Produces 2 chunks: 1. `Getting Started` chunk (includes `Prerequisites` subsection) 2. `API Reference` chunk Split articles into top-level sections ``` df.select(markdown.extract_header_chunks(col("articles"), header_level=1)) ``` Split documentation into feature sections ``` df.select(markdown.extract_header_chunks(col("docs"), header_level=2)) ``` Create fine-grained chunks for detailed analysis ``` df.select(markdown.extract_header_chunks(col("content"), header_level=3)) ``` Explode chunks into individual rows for processing ``` chunks_df = df.select( markdown.extract_header_chunks(col("markdown"), header_level=2).alias("chunks") ).explode("chunks") chunks_df.select( col("chunks").heading, col("chunks").content, col("chunks").full_path ) ``` Source code in `src/fenic/api/functions/markdown.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def extract_header_chunks(column: ColumnOrName, header_level: int) -> Column: """Splits markdown documents into logical chunks based on heading hierarchy. Args: column (ColumnOrName): Input column containing Markdown strings. header_level (int): Heading level to split on (1-6). Creates a new chunk at every heading of this level, including all nested content and subsections. Returns: Column: A column of arrays containing chunk objects with the following structure: ```python ArrayType(StructType([ StructField("heading", StringType), # Heading text (clean, no markdown) StructField("level", IntegerType), # Heading level (1-6) StructField("content", StringType), # All content under this heading (clean text) StructField("parent_heading", StringType), # Parent heading text (or null) StructField("full_path", StringType), # Full breadcrumb path ])) ``` Notes: - **Context-preserving**: Each chunk contains all content and subsections under the heading - **Hierarchical awareness**: Includes parent heading context for better LLM understanding - **Clean text output**: Strips markdown formatting for direct LLM consumption Chunking Behavior: With `header_level=2`, this markdown: ```markdown # Introduction Overview text ## Getting Started Setup instructions ### Prerequisites Python 3.8+ required ## API Reference Function documentation ``` Produces 2 chunks: 1. `Getting Started` chunk (includes `Prerequisites` subsection) 2. `API Reference` chunk Example: Split articles into top-level sections ```python df.select(markdown.extract_header_chunks(col("articles"), header_level=1)) ``` Example: Split documentation into feature sections ```python df.select(markdown.extract_header_chunks(col("docs"), header_level=2)) ``` Example: Create fine-grained chunks for detailed analysis ```python df.select(markdown.extract_header_chunks(col("content"), header_level=3)) ``` Example: Explode chunks into individual rows for processing ```python chunks_df = df.select( markdown.extract_header_chunks(col("markdown"), header_level=2).alias("chunks") ).explode("chunks") chunks_df.select( col("chunks").heading, col("chunks").content, col("chunks").full_path ) ``` """ if header_level < 1 or header_level > 6: raise ValidationError(f"split_level must be between 1 and 6 (inclusive), but got {header_level}. Use 1 for # headings, 2 for ## headings, etc.") return Column._from_logical_expr( MdExtractHeaderChunks(Column._from_col_or_name(column)._logical_expr, header_level) ) ``` ## generate_toc ``` generate_toc(column: ColumnOrName, max_level: Optional[int] = None) -> Column ``` Generates a table of contents from markdown headings. Parameters: - **`column`** (`ColumnOrName`) – Input column containing Markdown strings. - **`max_level`** (`Optional[int]`, default: `None` ) – Maximum heading level to include in the TOC (1-6). Defaults to 6 (all levels). Returns: - **`Column`** ( `Column` ) – A column of Markdown-formatted table of contents strings. Notes - The TOC is generated using markdown heading syntax (# ## ### etc.) - Each heading in the source document becomes a line in the TOC - The heading level is preserved in the output - This creates a valid markdown document that can be rendered or processed further Generate a complete TOC ``` df.select(markdown.generate_toc(col("documentation"))) ``` Generate a simplified TOC with only top 2 levels ``` df.select(markdown.generate_toc(col("documentation"), max_level=2)) ``` Add TOC as a new column ``` df = df.with_column("toc", markdown.generate_toc(col("content"), max_level=3)) ``` Source code in `src/fenic/api/functions/markdown.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def generate_toc(column: ColumnOrName, max_level: Optional[int] = None) -> Column: """Generates a table of contents from markdown headings. Args: column (ColumnOrName): Input column containing Markdown strings. max_level (Optional[int]): Maximum heading level to include in the TOC (1-6). Defaults to 6 (all levels). Returns: Column: A column of Markdown-formatted table of contents strings. Notes: - The TOC is generated using markdown heading syntax (# ## ### etc.) - Each heading in the source document becomes a line in the TOC - The heading level is preserved in the output - This creates a valid markdown document that can be rendered or processed further Example: Generate a complete TOC ```python df.select(markdown.generate_toc(col("documentation"))) ``` Example: Generate a simplified TOC with only top 2 levels ```python df.select(markdown.generate_toc(col("documentation"), max_level=2)) ``` Example: Add TOC as a new column ```python df = df.with_column("toc", markdown.generate_toc(col("content"), max_level=3)) ``` """ if max_level and (max_level < 1 or max_level > 6): raise ValidationError(f"max_level must be between 1 and 6 (inclusive), but got {max_level}. Use 1 for # headings, 2 for ## headings, etc.") return Column._from_logical_expr( MdGenerateTocExpr(Column._from_col_or_name(column)._logical_expr, max_level) ) ``` ## get_code_blocks ``` get_code_blocks(column: ColumnOrName, language_filter: Optional[str] = None) -> Column ``` Extracts all code blocks from a column of Markdown-formatted strings. Parameters: - **`column`** (`ColumnOrName`) – Input column containing Markdown strings. - **`language_filter`** (`Optional[str]`, default: `None` ) – Optional language filter to extract only code blocks with a specific language. By default, all code blocks are extracted. Returns: - **`Column`** ( `Column` ) – A column of code blocks. The output column type is: ArrayType(StructType([ StructField("language", StringType), StructField("code", StringType), ])) Notes - Code blocks are parsed from fenced Markdown blocks (e.g., triple backticks ```). - Language identifiers are optional and may be null if not provided in the original Markdown. - Indented code blocks without fences are not currently supported. - This function is useful for extracting embedded logic, configuration, or examples from documentation or notebooks. Extract all code blocks ``` df.select(markdown.get_code_blocks(col("markdown_text"))) ``` Explode code blocks into individual rows ``` # Explode the list of code blocks into individual rows df = df.explode(df.with_column("blocks", markdown.get_code_blocks(col("md")))) df = df.select(col("blocks")["language"], col("blocks")["code"]) ``` Source code in `src/fenic/api/functions/markdown.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def get_code_blocks(column: ColumnOrName, language_filter: Optional[str] = None) -> Column: """Extracts all code blocks from a column of Markdown-formatted strings. Args: column (ColumnOrName): Input column containing Markdown strings. language_filter (Optional[str]): Optional language filter to extract only code blocks with a specific language. By default, all code blocks are extracted. Returns: Column: A column of code blocks. The output column type is: ArrayType(StructType([ StructField("language", StringType), StructField("code", StringType), ])) Notes: - Code blocks are parsed from fenced Markdown blocks (e.g., triple backticks ```). - Language identifiers are optional and may be null if not provided in the original Markdown. - Indented code blocks without fences are not currently supported. - This function is useful for extracting embedded logic, configuration, or examples from documentation or notebooks. Example: Extract all code blocks ```python df.select(markdown.get_code_blocks(col("markdown_text"))) ``` Example: Explode code blocks into individual rows ```python # Explode the list of code blocks into individual rows df = df.explode(df.with_column("blocks", markdown.get_code_blocks(col("md")))) df = df.select(col("blocks")["language"], col("blocks")["code"]) ``` """ return Column._from_logical_expr( MdGetCodeBlocksExpr(Column._from_col_or_name(column)._logical_expr, language_filter) ) ``` ## to_json ``` to_json(column: ColumnOrName) -> Column ``` Converts a column of Markdown-formatted strings into a hierarchical JSON representation. Parameters: - **`column`** (`ColumnOrName`) – Input column containing Markdown strings. Returns: - **`Column`** ( `Column` ) – A column of JSON-formatted strings representing the structured document tree. Notes - This function parses Markdown into a structured JSON format optimized for document chunking, semantic analysis, and `jq` queries. - The output conforms to a custom schema that organizes content into nested sections based on heading levels. This makes it more expressive than flat ASTs like `mdast`. - The full JSON schema is available at: docs.fenic.ai/topics/markdown-json Supported Markdown Features - Headings with nested hierarchy (e.g., h2 β†’ h3 β†’ h4) - Paragraphs with inline formatting (bold, italics, links, code, etc.) - Lists (ordered, unordered, task lists) - Tables with header alignment and inline content - Code blocks with language info - Blockquotes, horizontal rules, and inline/flow HTML Convert markdown to JSON ``` df.select(markdown.to_json(col("markdown_text"))) ``` Extract all level-2 headings with jq ``` # Combine with jq to extract all level-2 headings df.select(json.jq(markdown.to_json(col("md")), ".. | select(.type == 'heading' and .level == 2)")) ``` Source code in `src/fenic/api/functions/markdown.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def to_json(column: ColumnOrName) -> Column: """Converts a column of Markdown-formatted strings into a hierarchical JSON representation. Args: column (ColumnOrName): Input column containing Markdown strings. Returns: Column: A column of JSON-formatted strings representing the structured document tree. Notes: - This function parses Markdown into a structured JSON format optimized for document chunking, semantic analysis, and `jq` queries. - The output conforms to a custom schema that organizes content into nested sections based on heading levels. This makes it more expressive than flat ASTs like `mdast`. - The full JSON schema is available at: docs.fenic.ai/topics/markdown-json Supported Markdown Features: - Headings with nested hierarchy (e.g., h2 β†’ h3 β†’ h4) - Paragraphs with inline formatting (bold, italics, links, code, etc.) - Lists (ordered, unordered, task lists) - Tables with header alignment and inline content - Code blocks with language info - Blockquotes, horizontal rules, and inline/flow HTML Example: Convert markdown to JSON ```python df.select(markdown.to_json(col("markdown_text"))) ``` Example: Extract all level-2 headings with jq ```python # Combine with jq to extract all level-2 headings df.select(json.jq(markdown.to_json(col("md")), ".. | select(.type == 'heading' and .level == 2)")) ``` """ return Column._from_logical_expr( MdToJsonExpr(Column._from_col_or_name(column)._logical_expr) ) ``` --- # fenic.api.functions.semantic Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/semantic/ Semantic functions for Fenic DataFrames - LLM-based operations. Functions: - **`analyze_sentiment`** – Analyzes the sentiment of a string column. Returns one of 'positive', 'negative', or 'neutral'. - **`classify`** – Classifies a string column into one of the provided classes. - **`embed`** – Generate embeddings for the specified string column. - **`extract`** – Extracts structured information from unstructured text using a provided Pydantic model schema. - **`map`** – Applies a generation prompt to one or more columns, enabling rich summarization and generation tasks. - **`parse_pdf`** – Parses a column of PDF paths into markdown. - **`predicate`** – Applies a boolean predicate to one or more columns, typically used for filtering. - **`reduce`** – Aggregate function: reduces a set of strings in a column to a single string using a natural language instruction. - **`summarize`** – Summarizes strings from a column. ## analyze_sentiment ``` analyze_sentiment(column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, request_timeout: TimeoutParam = None) -> Column ``` Analyzes the sentiment of a string column. Returns one of 'positive', 'negative', or 'neutral'. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing text for sentiment analysis. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use for the mapping. If None, will use the language model configured as the default. - **`temperature`** (`float`, default: `0` ) – Optional temperature parameter for the language model. If None, will use the default temperature (0.0). - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`Column`** ( `Column` ) – Expression containing sentiment results ('positive', 'negative', or 'neutral'). Raises: - `ValueError` – If column is invalid or cannot be resolved. Analyzing the sentiment of a user comment ``` semantic.analyze_sentiment(col('user_comment')) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def analyze_sentiment( column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, request_timeout: TimeoutParam = None, ) -> Column: """Analyzes the sentiment of a string column. Returns one of 'positive', 'negative', or 'neutral'. Args: column: Column or column name containing text for sentiment analysis. model_alias: Optional alias for the language model to use for the mapping. If None, will use the language model configured as the default. temperature: Optional temperature parameter for the language model. If None, will use the default temperature (0.0). request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: Column: Expression containing sentiment results ('positive', 'negative', or 'neutral'). Raises: ValueError: If column is invalid or cannot be resolved. Example: Analyzing the sentiment of a user comment ```python semantic.analyze_sentiment(col('user_comment')) ``` """ resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( AnalyzeSentimentExpr( Column._from_col_or_name(column)._logical_expr, model_alias=resolved_model_alias, temperature=temperature, request_timeout=request_timeout, ) ) ``` ## classify ``` classify(column: ColumnOrName, classes: Union[List[str], List[ClassDefinition]], examples: Optional[ClassifyExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, request_timeout: TimeoutParam = None) -> Column ``` Classifies a string column into one of the provided classes. This is useful for tagging incoming documents with predefined categories. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing text to classify. - **`classes`** (`Union[List[str], List[ClassDefinition]]`) – List of class labels or ClassDefinition objects defining the available classes. Use ClassDefinition objects to provide descriptions for the classes. - **`examples`** (`Optional[ClassifyExampleCollection]`, default: `None` ) – Optional collection of example classifications to guide the model. Examples should be created using ClassifyExampleCollection.create_example(), with instruction variables mapped to their expected classifications. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use for the mapping. If None, will use the language model configured as the default. - **`temperature`** (`float`, default: `0` ) – Optional temperature parameter for the language model. If None, will use the default temperature (0.0). - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`Column`** ( `Column` ) – Expression containing the classification results. Raises: - `ValueError` – If column is invalid or classes is empty or has duplicate labels. Categorizing incoming support requests ``` # Categorize incoming support requests semantic.classify("message", ["Account Access", "Billing Issue", "Technical Problem"]) ``` Categorizing incoming support requests using ClassDefinition objects ``` # Categorize incoming support requests semantic.classify("message", [ ClassDefinition(label="Account Access", description="General questions, feature requests, or non-technical assistance"), ClassDefinition(label="Billing Issue", description="Questions about charges, payments, subscriptions, or account billing"), ClassDefinition(label="Technical Problem", description="Problems with product functionality, bugs, or technical difficulties") ]) ``` Categorizing incoming support requests with ClassDefinition objects and examples ``` examples = ClassifyExampleCollection() class_definitions = [ ClassDefinition(label="Account Access", description="General questions, feature requests, or non-technical assistance"), ClassDefinition(label="Billing Issue", description="Questions about charges, payments, subscriptions, or account billing"), ClassDefinition(label="Technical Problem", description="Problems with product functionality, bugs, or technical difficulties") ] examples.create_example(ClassifyExample( input="I can't reset my password or access my account.", output="Account Access")) examples.create_example(ClassifyExample( input="You charged me twice for the same month.", output="Billing Issue")) semantic.classify("message", class_definitions, examples) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def classify( column: ColumnOrName, classes: Union[List[str], List[ClassDefinition]], examples: Optional[ClassifyExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, request_timeout: TimeoutParam = None, ) -> Column: """Classifies a string column into one of the provided classes. This is useful for tagging incoming documents with predefined categories. Args: column: Column or column name containing text to classify. classes: List of class labels or ClassDefinition objects defining the available classes. Use ClassDefinition objects to provide descriptions for the classes. examples: Optional collection of example classifications to guide the model. Examples should be created using ClassifyExampleCollection.create_example(), with instruction variables mapped to their expected classifications. model_alias: Optional alias for the language model to use for the mapping. If None, will use the language model configured as the default. temperature: Optional temperature parameter for the language model. If None, will use the default temperature (0.0). request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: Column: Expression containing the classification results. Raises: ValueError: If column is invalid or classes is empty or has duplicate labels. Example: Categorizing incoming support requests ```python # Categorize incoming support requests semantic.classify("message", ["Account Access", "Billing Issue", "Technical Problem"]) ``` Example: Categorizing incoming support requests using ClassDefinition objects ```python # Categorize incoming support requests semantic.classify("message", [ ClassDefinition(label="Account Access", description="General questions, feature requests, or non-technical assistance"), ClassDefinition(label="Billing Issue", description="Questions about charges, payments, subscriptions, or account billing"), ClassDefinition(label="Technical Problem", description="Problems with product functionality, bugs, or technical difficulties") ]) ``` Example: Categorizing incoming support requests with ClassDefinition objects and examples ```python examples = ClassifyExampleCollection() class_definitions = [ ClassDefinition(label="Account Access", description="General questions, feature requests, or non-technical assistance"), ClassDefinition(label="Billing Issue", description="Questions about charges, payments, subscriptions, or account billing"), ClassDefinition(label="Technical Problem", description="Problems with product functionality, bugs, or technical difficulties") ] examples.create_example(ClassifyExample( input="I can't reset my password or access my account.", output="Account Access")) examples.create_example(ClassifyExample( input="You charged me twice for the same month.", output="Billing Issue")) semantic.classify("message", class_definitions, examples) ``` """ if len(classes) < 2: raise ValidationError( "The `classes` list must contain at least two ClassDefinition objects. " "You provided only one. Classification requires at least two possible labels." ) # Validate unique labels if isinstance(classes[0], ClassDefinition): classes = [ResolvedClassDefinition(label=class_def.label, description=class_def.description) for class_def in classes] else: classes = [ResolvedClassDefinition(label=class_def, description=None) for class_def in classes] labels = [class_def.label for class_def in classes] duplicates = {label for label in labels if labels.count(label) > 1} if duplicates: raise ValidationError( f"Class labels must be unique. The following duplicate label(s) were found: {sorted(duplicates)}" ) resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( SemanticClassifyExpr( Column._from_col_or_name(column)._logical_expr, classes, examples=examples, model_alias=resolved_model_alias, temperature=temperature, request_timeout=request_timeout, ) ) ``` ## embed ``` embed(column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None) -> Column ``` Generate embeddings for the specified string column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing the values to generate embeddings for. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the embedding model to use for the mapping. If None, will use the embedding model configured as the default. Returns: - `Column` – A Column expression that represents the embeddings for each value in the input column Raises: - `TypeError` – If the input column is not a string column. Generate embeddings for a text column ``` df.select(semantic.embed(col("text_column")).alias("text_embeddings")) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def embed( column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None, ) -> Column: """Generate embeddings for the specified string column. Args: column: Column or column name containing the values to generate embeddings for. model_alias: Optional alias for the embedding model to use for the mapping. If None, will use the embedding model configured as the default. Returns: A Column expression that represents the embeddings for each value in the input column Raises: TypeError: If the input column is not a string column. Example: Generate embeddings for a text column ```python df.select(semantic.embed(col("text_column")).alias("text_embeddings")) ``` """ resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( EmbeddingsExpr(Column._from_col_or_name(column)._logical_expr, model_alias=resolved_model_alias) ) ``` ## extract ``` extract(column: ColumnOrName, response_format: type[BaseModel], max_output_tokens: int = 1024, temperature: float = 0.0, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: TimeoutParam = None) -> Column ``` Extracts structured information from unstructured text using a provided Pydantic model schema. This function applies an instruction-driven extraction process to text columns, returning structured data based on the fields and descriptions provided. Useful for pulling out key entities, facts, or labels from documents. The schema must be a valid Pydantic model type with supported field types. These include: - Primitive types: `str`, `int`, `float`, `bool` - Optional fields: `Optional[T]` where `T` is a supported type - Lists: `List[T]` where `T` is a supported type - Literals: `Literal[...`] (for enum-like constraints) - Nested Pydantic models (recursive schemas are supported, but must be JSON-serializable and acyclic) Unsupported types (e.g., unions, custom classes, runtime circular references, or complex generics) will raise errors at runtime. Parameters: - **`column`** (`ColumnOrName`) – Column containing text to extract from. - **`response_format`** (`type[BaseModel]`) – A Pydantic model type that defines the output structure with descriptions for each field. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use for the extraction. If None, will use the language model configured as the default. - **`temperature`** (`float`, default: `0.0` ) – Optional temperature parameter for the language model. If None, will use the default temperature (0.0). - **`max_output_tokens`** (`int`, default: `1024` ) – Optional parameter to constrain the model to generate at most this many tokens. If None, fenic will calculate the expected max tokens, based on the model's context length and other operator-specific parameters. - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`Column`** ( `Column` ) – A new column with structured values (a struct) based on the provided schema. Extracting knowledge graph triples and named entities from text ``` class Triple(BaseModel): subject: str = Field(description="The subject of the triple") predicate: str = Field(description="The predicate or relation") object: str = Field(description="The object of the triple") class KGResult(BaseModel): triples: List[Triple] = Field(description="List of extracted knowledge graph triples") entities: list[str] = Field(description="Flat list of all detected named entities") df.select(semantic.extract("blurb", KGResult)) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def extract( column: ColumnOrName, response_format: type[BaseModel], max_output_tokens: int = 1024, temperature: float = 0.0, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: TimeoutParam = None, ) -> Column: """Extracts structured information from unstructured text using a provided Pydantic model schema. This function applies an instruction-driven extraction process to text columns, returning structured data based on the fields and descriptions provided. Useful for pulling out key entities, facts, or labels from documents. The schema must be a valid Pydantic model type with supported field types. These include: - Primitive types: `str`, `int`, `float`, `bool` - Optional fields: `Optional[T]` where `T` is a supported type - Lists: `List[T]` where `T` is a supported type - Literals: `Literal[...`] (for enum-like constraints) - Nested Pydantic models (recursive schemas are supported, but must be JSON-serializable and acyclic) Unsupported types (e.g., unions, custom classes, runtime circular references, or complex generics) will raise errors at runtime. Args: column: Column containing text to extract from. response_format: A Pydantic model type that defines the output structure with descriptions for each field. model_alias: Optional alias for the language model to use for the extraction. If None, will use the language model configured as the default. temperature: Optional temperature parameter for the language model. If None, will use the default temperature (0.0). max_output_tokens: Optional parameter to constrain the model to generate at most this many tokens. If None, fenic will calculate the expected max tokens, based on the model's context length and other operator-specific parameters. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: Column: A new column with structured values (a struct) based on the provided schema. Example: Extracting knowledge graph triples and named entities from text ```python class Triple(BaseModel): subject: str = Field(description="The subject of the triple") predicate: str = Field(description="The predicate or relation") object: str = Field(description="The object of the triple") class KGResult(BaseModel): triples: List[Triple] = Field(description="List of extracted knowledge graph triples") entities: list[str] = Field(description="Flat list of all detected named entities") df.select(semantic.extract("blurb", KGResult)) ``` """ try: validate_output_format(response_format) except OutputFormatValidationError as e: raise ValidationError(f"Invalid response format: {str(e)}") from None resolved_model_alias = _resolve_model_alias(model_alias) resolved_response_format = ResolvedResponseFormat.from_pydantic_model(response_format) return Column._from_logical_expr( SemanticExtractExpr( Column._from_col_or_name(column)._logical_expr, max_tokens=max_output_tokens, temperature=temperature, response_format=resolved_response_format, model_alias=resolved_model_alias, request_timeout=request_timeout, ) ) ``` ## map ``` map(prompt: str, /, *, strict: bool = True, examples: Optional[MapExampleCollection] = None, response_format: Optional[type[BaseModel]] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0.0, max_output_tokens: int = 512, request_timeout: TimeoutParam = None, **columns: Column) -> Column ``` Applies a generation prompt to one or more columns, enabling rich summarization and generation tasks. Parameters: - **`prompt`** (`str`) – A Jinja2 template for the generation prompt. References column values using {{ column_name }} syntax. Each placeholder is replaced with the corresponding value from the current row during execution. - **`strict`** (`bool`, default: `True` ) – If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[MapExampleCollection]`, default: `None` ) – Optional few-shot examples to guide the model's output format and style. - **`response_format`** (`Optional[type[BaseModel]]`, default: `None` ) – Optional Pydantic model to enforce structured output. Must include descriptions for each field. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional language model alias. If None, uses the default model. - **`temperature`** (`float`, default: `0.0` ) – Language model temperature (default: 0.0). - **`max_output_tokens`** (`int`, default: `512` ) – Maximum tokens to generate (default: 512). - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). - **`**columns`** (`Column`, default: `{}` ) – Named column arguments that correspond to template variables. Keys must match the variable names used in the template. Returns: - **`Column`** ( `Column` ) – A column expression representing the semantic mapping operation. Mapping without examples ``` fc.semantic.map( "Write a compelling one-line description for {{ name }}: {{ details }}", name=fc.col("name"), details=fc.col("details") ) ``` Mapping with few-shot examples ``` examples = MapExampleCollection() examples.create_example(MapExample( input={"name": "GlowMate", "details": "A rechargeable bedside lamp with adjustable color temperatures, touch controls, and a sleek minimalist design."}, output="The modern touch-controlled lamp for better sleep and style." )) examples.create_example(MapExample( input={"name": "AquaPure", "details": "A compact water filter that attaches to your faucet, removes over 99% of contaminants, and improves taste instantly."}, output="Clean, great-tasting water straight from your tap." )) fc.semantic.map( "Write a compelling one-line description for {{ name }}: {{ details }}", name=fc.col("name"), details=fc.col("details"), examples=examples ) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(arbitrary_types_allowed=True, strict=True)) def map( prompt: str, /, *, strict: bool = True, examples: Optional[MapExampleCollection] = None, response_format: Optional[type[BaseModel]] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0.0, max_output_tokens: int = 512, request_timeout: TimeoutParam = None, **columns: Column, ) -> Column: """Applies a generation prompt to one or more columns, enabling rich summarization and generation tasks. Args: prompt: A Jinja2 template for the generation prompt. References column values using {{ column_name }} syntax. Each placeholder is replaced with the corresponding value from the current row during execution. strict: If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. examples: Optional few-shot examples to guide the model's output format and style. response_format: Optional Pydantic model to enforce structured output. Must include descriptions for each field. model_alias: Optional language model alias. If None, uses the default model. temperature: Language model temperature (default: 0.0). max_output_tokens: Maximum tokens to generate (default: 512). request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). **columns: Named column arguments that correspond to template variables. Keys must match the variable names used in the template. Returns: Column: A column expression representing the semantic mapping operation. Example: Mapping without examples ```python fc.semantic.map( "Write a compelling one-line description for {{ name }}: {{ details }}", name=fc.col("name"), details=fc.col("details") ) ``` Example: Mapping with few-shot examples ```python examples = MapExampleCollection() examples.create_example(MapExample( input={"name": "GlowMate", "details": "A rechargeable bedside lamp with adjustable color temperatures, touch controls, and a sleek minimalist design."}, output="The modern touch-controlled lamp for better sleep and style." )) examples.create_example(MapExample( input={"name": "AquaPure", "details": "A compact water filter that attaches to your faucet, removes over 99% of contaminants, and improves taste instantly."}, output="Clean, great-tasting water straight from your tap." )) fc.semantic.map( "Write a compelling one-line description for {{ name }}: {{ details }}", name=fc.col("name"), details=fc.col("details"), examples=examples ) ``` """ if not prompt: raise ValidationError("The `prompt` argument to `semantic.map` cannot be empty.") if not columns: raise ValidationError("`semantic.map` requires at least one named column argument (e.g. `text=col('text')`).") if response_format: try: validate_output_format(response_format) except OutputFormatValidationError as e: raise ValidationError(f"Invalid response format: {str(e)}") from None exprs: List[Union[ColumnExpr, AliasExpr]] = [] for var_name, column in columns.items(): if isinstance(column._logical_expr, ColumnExpr) and column._logical_expr.name == var_name: exprs.append(column._logical_expr) else: exprs.append(column.alias(var_name)._logical_expr) resolved_model_alias = _resolve_model_alias(model_alias) if response_format: resolved_response_format = ResolvedResponseFormat.from_pydantic_model(response_format) else: resolved_response_format = None return Column._from_logical_expr( SemanticMapExpr( prompt, strict=strict, exprs=exprs, max_tokens=max_output_tokens, temperature=temperature, model_alias=resolved_model_alias, response_format=resolved_response_format, examples=examples, request_timeout=request_timeout, ) ) ``` ## parse_pdf ``` parse_pdf(column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None, page_separator: Optional[str] = None, describe_images: bool = False, max_output_tokens: Optional[int] = None, request_timeout: TimeoutParam = None) -> Column ``` Parses a column of PDF paths into markdown. Note Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Returns: - **`Dataframe`** ( `Column` ) – a dataframe with markdown strings for each PDF file. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing the PDF to parse. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use for the parsing. If None, will use the language model configured as the default. - **`page_separator`** (`Optional[str]`, default: `None` ) – Optional page separator to use for the parsing. If the separator includes the {page} placeholder, the model will replace it with the current page number. - **`describe_images`** (`bool`, default: `False` ) – Flag to describe images in the PDF. If True, the prompt will ask the model to include a description of the image in the markdown output. If False, the prompt asks the model to ignore images that aren't tables or charts. - **`max_output_tokens`** (`Optional[int]`, default: `None` ) – Optional maximum number of output tokens per ~3 pages of PDF (does not include reasoning tokens). If None, don't constrain the model's output. - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Note For Gemini models, this function uses the google file API, uploading PDF files to Google's file store and deleting them after each request. Raises: - `ExecutionError` – If paths in the column are not valid PDF files. Parse PDF paths in a column into markdown ``` semantic.parse_pdf(col("pdf_path")).show() ``` Parsing PDFs into markdown with a page separator and describing images larger than 50 pixels ``` pdf_markdown = semantic.parse_pdf(col("pdf_path"), page_separator="--- PAGE {page} ---", describe_images=True) pdf_markdown.select(col("markdown_content")).show() ``` Grabbing PDFs from a glob pattern using pdf_metadata and parsing the PDF content ``` pdf_metadata = local_session.read.pdf_metadata("data/docs/**/*.pdf") pdf_markdown = pdf_metadata.select(semantic.parse_pdf(col("file_path"), page_separator="--- PAGE BREAK ---") pdf_markdown.select(col("markdown_content")).show() ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def parse_pdf( column: ColumnOrName, model_alias: Optional[Union[str, ModelAlias]] = None, page_separator: Optional[str] = None, describe_images: bool = False, # for images that aren't tables max_output_tokens: Optional[int] = None, request_timeout: TimeoutParam = None, ) -> Column: r"""Parses a column of PDF paths into markdown. Note: Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Returns: Dataframe: a dataframe with markdown strings for each PDF file. Args: column: Column or column name containing the PDF to parse. model_alias: Optional alias for the language model to use for the parsing. If None, will use the language model configured as the default. page_separator: Optional page separator to use for the parsing. If the separator includes the {page} placeholder, the model will replace it with the current page number. describe_images: Flag to describe images in the PDF. If True, the prompt will ask the model to include a description of the image in the markdown output. If False, the prompt asks the model to ignore images that aren't tables or charts. max_output_tokens: Optional maximum number of output tokens per ~3 pages of PDF (does not include reasoning tokens). If None, don't constrain the model's output. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Note: For Gemini models, this function uses the google file API, uploading PDF files to Google's file store and deleting them after each request. Raises: ExecutionError: If paths in the column are not valid PDF files. Example: Parse PDF paths in a column into markdown ```python semantic.parse_pdf(col("pdf_path")).show() ``` Example: Parsing PDFs into markdown with a page separator and describing images larger than 50 pixels ```python pdf_markdown = semantic.parse_pdf(col("pdf_path"), page_separator="--- PAGE {page} ---", describe_images=True) pdf_markdown.select(col("markdown_content")).show() ``` Example: Grabbing PDFs from a glob pattern using pdf_metadata and parsing the PDF content ```python pdf_metadata = local_session.read.pdf_metadata("data/docs/**/*.pdf") pdf_markdown = pdf_metadata.select(semantic.parse_pdf(col("file_path"), page_separator="--- PAGE BREAK ---") pdf_markdown.select(col("markdown_content")).show() ``` """ resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( SemanticParsePDFExpr( Column._from_col_or_name(column)._logical_expr, model_alias=resolved_model_alias, page_separator=page_separator, describe_images=describe_images, max_output_tokens=max_output_tokens, request_timeout=request_timeout, ) ) ``` ## predicate ``` predicate(predicate: str, /, *, strict: bool = True, examples: Optional[PredicateExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0.0, request_timeout: TimeoutParam = None, **columns: Column) -> Column ``` Applies a boolean predicate to one or more columns, typically used for filtering. Parameters: - **`predicate`** (`str`) – A Jinja2 template containing a yes/no question or boolean claim. Should reference column values using {{ column_name }} syntax. The model will evaluate this condition for each row and return True or False. - **`strict`** (`bool`, default: `True` ) – If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. - **`examples`** (`Optional[PredicateExampleCollection]`, default: `None` ) – Optional few-shot examples showing how to evaluate the predicate. Helps ensure consistent True/False decisions. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional language model alias. If None, uses the default model. - **`temperature`** (`float`, default: `0.0` ) – Language model temperature (default: 0.0). - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). - **`**columns`** (`Column`, default: `{}` ) – Named column arguments that correspond to template variables. Keys must match the variable names used in the template. Returns: - **`Column`** ( `Column` ) – A boolean column expression. Filtering product descriptions ``` wireless_products = df.filter( fc.semantic.predicate( dedent('''\ Product: {{ description }} Is this product wireless or battery-powered?'''), description=fc.col("product_description") ) ) ``` Filtering support tickets ``` df = df.with_column( "is_urgent", fc.semantic.predicate( dedent('''\ Subject: {{ subject }} Body: {{ body }} This ticket indicates an urgent issue.'''), subject=fc.col("ticket_subject"), body=fc.col("ticket_body") ) ) ``` Filtering with examples ``` examples = PredicateExampleCollection() examples.create_example(PredicateExample( input={"ticket": "I was charged twice for my subscription and need help."}, output=True )) examples.create_example(PredicateExample( input={"ticket": "How do I reset my password?"}, output=False )) fc.semantic.predicate( dedent('''\ Ticket: {{ ticket }} This ticket is about billing.'''), ticket=fc.col("ticket_text"), examples=examples ) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(arbitrary_types_allowed=True, strict=True)) def predicate( predicate: str, /, *, strict: bool = True, examples: Optional[PredicateExampleCollection] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0.0, request_timeout: TimeoutParam = None, **columns: Column, ) -> Column: r"""Applies a boolean predicate to one or more columns, typically used for filtering. Args: predicate: A Jinja2 template containing a yes/no question or boolean claim. Should reference column values using {{ column_name }} syntax. The model will evaluate this condition for each row and return True or False. strict: If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. examples: Optional few-shot examples showing how to evaluate the predicate. Helps ensure consistent True/False decisions. model_alias: Optional language model alias. If None, uses the default model. temperature: Language model temperature (default: 0.0). request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). **columns: Named column arguments that correspond to template variables. Keys must match the variable names used in the template. Returns: Column: A boolean column expression. Example: Filtering product descriptions ```python wireless_products = df.filter( fc.semantic.predicate( dedent('''\ Product: {{ description }} Is this product wireless or battery-powered?'''), description=fc.col("product_description") ) ) ``` Example: Filtering support tickets ```python df = df.with_column( "is_urgent", fc.semantic.predicate( dedent('''\ Subject: {{ subject }} Body: {{ body }} This ticket indicates an urgent issue.'''), subject=fc.col("ticket_subject"), body=fc.col("ticket_body") ) ) ``` Example: Filtering with examples ```python examples = PredicateExampleCollection() examples.create_example(PredicateExample( input={"ticket": "I was charged twice for my subscription and need help."}, output=True )) examples.create_example(PredicateExample( input={"ticket": "How do I reset my password?"}, output=False )) fc.semantic.predicate( dedent('''\ Ticket: {{ ticket }} This ticket is about billing.'''), ticket=fc.col("ticket_text"), examples=examples ) ``` """ if not predicate: raise ValidationError("The `predicate` argument to `semantic.predicate` cannot be empty.") if not columns: raise ValidationError("`semantic.predicate` requires at least one named column argument (e.g. `text=col('text')`).") exprs: List[Union[ColumnExpr, AliasExpr]] = [] for var_name, column in columns.items(): if isinstance(column._logical_expr, ColumnExpr) and column._logical_expr.name == var_name: exprs.append(column._logical_expr) else: exprs.append(column.alias(var_name)._logical_expr) resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( SemanticPredExpr( predicate, strict=strict, exprs=exprs, temperature=temperature, model_alias=resolved_model_alias, examples=examples, request_timeout=request_timeout, ) ) ``` ## reduce ``` reduce(prompt: str, column: ColumnOrName, *, group_context: Optional[Dict[str, Column]] = None, order_by: List[ColumnOrName] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, max_output_tokens: int = 512, request_timeout: TimeoutParam = None) -> Column ``` Aggregate function: reduces a set of strings in a column to a single string using a natural language instruction. Parameters: - **`prompt`** (`str`) – A string containing the semantic.reduce prompt. The instruction can optionally include Jinja2 template variables (e.g., {{variable}}) that reference columns from the group_context parameter. These will be replaced with actual values from the first row of each group during execution. - **`column`** (`ColumnOrName`) – The column containing documents/strings to reduce. - **`group_context`** (`Optional[Dict[str, Column]]`, default: `None` ) – Optional dictionary mapping variable names to columns. These columns provide context for each group and can be referenced in the instruction template. - **`order_by`** (`List[ColumnOrName]`, default: `None` ) – Optional list of columns to sort grouped documents by before reduction. Documents are processed in ascending order by default if no sort function is provided. Use a sort function (e.g., col("date").desc()/fc.desc("date")) for descending order. The order_by columns help preserve the temporal/logical sequence of the documents (e.g chunks in a document, speaker turns in a meeting transcript) for more coherent summaries. - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use. If None, uses the default model. - **`temperature`** (`float`, default: `0` ) – Temperature parameter for the language model (default: 0.0). - **`max_output_tokens`** (`int`, default: `512` ) – Maximum tokens the model can generate (default: 512). - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`Column`** ( `Column` ) – A column expression representing the semantic reduction operation. Simple reduction ``` # Simple reduction df.group_by("category").agg( semantic.reduce("Summarize the documents", col("document_text")) ) ``` With group context ``` df.group_by("department", "region").agg( semantic.reduce( "Summarize these {{department}} reports from {{region}}", col("document_text"), group_context={ "department": col("department"), "region": col("region") } ) ) ``` With sorting ``` df.group_by("category").agg( semantic.reduce( "Summarize the documents", col("document_text"), order_by=col("date") ) ) ``` Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def reduce( prompt: str, column: ColumnOrName, *, group_context: Optional[Dict[str, Column]] = None, order_by: List[ColumnOrName] = None, model_alias: Optional[Union[str, ModelAlias]] = None, temperature: float = 0, max_output_tokens: int = 512, request_timeout: TimeoutParam = None, ) -> Column: """Aggregate function: reduces a set of strings in a column to a single string using a natural language instruction. Args: prompt: A string containing the semantic.reduce prompt. The instruction can optionally include Jinja2 template variables (e.g., {{variable}}) that reference columns from the group_context parameter. These will be replaced with actual values from the first row of each group during execution. column: The column containing documents/strings to reduce. group_context: Optional dictionary mapping variable names to columns. These columns provide context for each group and can be referenced in the instruction template. order_by: Optional list of columns to sort grouped documents by before reduction. Documents are processed in ascending order by default if no sort function is provided. Use a sort function (e.g., col("date").desc()/fc.desc("date")) for descending order. The order_by columns help preserve the temporal/logical sequence of the documents (e.g chunks in a document, speaker turns in a meeting transcript) for more coherent summaries. model_alias: Optional alias for the language model to use. If None, uses the default model. temperature: Temperature parameter for the language model (default: 0.0). max_output_tokens: Maximum tokens the model can generate (default: 512). request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: Column: A column expression representing the semantic reduction operation. Example: Simple reduction ```python # Simple reduction df.group_by("category").agg( semantic.reduce("Summarize the documents", col("document_text")) ) ``` Example: With group context ```python df.group_by("department", "region").agg( semantic.reduce( "Summarize these {{department}} reports from {{region}}", col("document_text"), group_context={ "department": col("department"), "region": col("region") } ) ) ``` Example: With sorting ```python df.group_by("category").agg( semantic.reduce( "Summarize the documents", col("document_text"), order_by=col("date") ) ) ``` """ if not prompt: raise ValidationError("The `prompt` argument to `semantic.reduce` cannot be empty.") group_context_exprs: List[Union[ColumnExpr, AliasExpr]] = [] if group_context: for var_name, col in group_context.items(): if isinstance(col, ColumnExpr) and col.expr.name == var_name: group_context_exprs.append(col._logical_expr) else: group_context_exprs.append(col.alias(var_name)._logical_expr) order_by_exprs = [] if order_by: for col in order_by: order_by_exprs.append(Column._from_col_or_name(col)._logical_expr) resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( SemanticReduceExpr( prompt, input_expr=Column._from_col_or_name(column)._logical_expr, max_tokens=max_output_tokens, temperature=temperature, group_context_exprs=group_context_exprs, model_alias=resolved_model_alias, order_by_exprs=order_by_exprs, request_timeout=request_timeout, ) ) ``` ## summarize ``` summarize(column: ColumnOrName, format: Union[KeyPoints, Paragraph, None] = None, temperature: float = 0, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: TimeoutParam = None) -> Column ``` Summarizes strings from a column. Parameters: - **`column`** (`ColumnOrName`) – Column or column name containing text for summarization - **`format`** (`Union[KeyPoints, Paragraph, None]`, default: `None` ) – Format of the summary to generate. Can be either KeyPoints or Paragraph. If None, will default to Paragraph with a maximum of 120 words. - **`temperature`** (`float`, default: `0` ) – Optional temperature parameter for the language model. If None, will use the default temperature (0.0). - **`model_alias`** (`Optional[Union[str, ModelAlias]]`, default: `None` ) – Optional alias for the language model to use for the summarization. If None, will use the language model configured as the default. - **`request_timeout`** (`TimeoutParam`, default: `None` ) – Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: - **`Column`** ( `Column` ) – Expression containing the summarized string Raises: ValueError: If column is invalid or cannot be resolved. Example > > > semantic.summarize(col('user_comment')). Source code in `src/fenic/api/functions/semantic.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def summarize( column: ColumnOrName, format: Union[KeyPoints, Paragraph, None] = None, temperature: float = 0, model_alias: Optional[Union[str, ModelAlias]] = None, request_timeout: TimeoutParam = None, ) -> Column: """Summarizes strings from a column. Args: column: Column or column name containing text for summarization format: Format of the summary to generate. Can be either KeyPoints or Paragraph. If None, will default to Paragraph with a maximum of 120 words. temperature: Optional temperature parameter for the language model. If None, will use the default temperature (0.0). model_alias: Optional alias for the language model to use for the summarization. If None, will use the language model configured as the default. request_timeout: Optional timeout in seconds for a single LLM request. If None, uses the default timeout (120 seconds). Returns: Column: Expression containing the summarized string Raises: ValueError: If column is invalid or cannot be resolved. Example: >>> semantic.summarize(col('user_comment')). """ if format is None: format = Paragraph() resolved_model_alias = _resolve_model_alias(model_alias) return Column._from_logical_expr( SemanticSummarizeExpr(Column._from_col_or_name(column)._logical_expr, format, temperature, model_alias=resolved_model_alias, request_timeout=request_timeout) ) ``` --- # fenic.api.functions.text Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/functions/text/ Text manipulation functions for Fenic DataFrames. Functions: - **`array_join`** – Joins an array of strings into a single string with a delimiter. - **`btrim`** – Remove specified characters from both sides of strings in a column. - **`byte_length`** – Calculate the byte length of each string in the column. - **`character_chunk`** – Chunks a string column into chunks of a specified size (in characters) with an optional overlap. - **`compute_fuzzy_ratio`** – Compute the similarity between two strings using a fuzzy string matching algorithm. - **`compute_fuzzy_token_set_ratio`** – Compute fuzzy similarity using token set comparison. - **`compute_fuzzy_token_sort_ratio`** – Compute fuzzy similarity after sorting tokens in each string. - **`concat`** – Concatenates multiple columns or strings into a single string. - **`concat_ws`** – Concatenates multiple columns or strings into a single string with a separator. - **`count_tokens`** – Returns the number of tokens in a string using OpenAI's cl100k_base encoding (tiktoken). - **`extract`** – Extracts structured data from text using template-based pattern matching. - **`jinja`** – Render a Jinja template using values from the specified columns. - **`length`** – Calculate the character length of each string in the column. - **`lower`** – Convert all characters in a string column to lowercase. - **`ltrim`** – Remove whitespace from the start of strings in a column. - **`parse_transcript`** – Parses a transcript from text to a structured format with unified schema. - **`recursive_character_chunk`** – Chunks a string column into chunks of a specified size (in characters) with an optional overlap. - **`recursive_token_chunk`** – Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. - **`recursive_word_chunk`** – Chunks a string column into chunks of a specified size (in words) with an optional overlap. - **`regexp_count`** – Count the number of times a regex pattern is matched in a string. - **`regexp_extract`** – Extract a specific regex group from a string. - **`regexp_extract_all`** – Extract all strings matching a regex pattern, optionally from a specific group. - **`regexp_instr`** – Find the 1-based position of the first regex match in a string. - **`regexp_replace`** – Replace all occurrences of a pattern with a new string, treating pattern as a regular expression. - **`regexp_substr`** – Extract the first substring matching a regex pattern. - **`replace`** – Replace all occurrences of a pattern with a new string, treating pattern as a literal string. - **`rtrim`** – Remove whitespace from the end of strings in a column. - **`split`** – Split a string column into an array using a regular expression pattern. - **`split_part`** – Split a string and return a specific part using 1-based indexing. - **`title_case`** – Convert the first character of each word in a string column to uppercase. - **`token_chunk`** – Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. - **`trim`** – Remove whitespace from both sides of strings in a column. - **`upper`** – Convert all characters in a string column to uppercase. - **`word_chunk`** – Chunks a string column into chunks of a specified size (in words) with an optional overlap. ## array_join ``` array_join(column: ColumnOrName, delimiter: str) -> Column ``` Joins an array of strings into a single string with a delimiter. Parameters: - **`column`** (`ColumnOrName`) – The column to join - **`delimiter`** (`str`) – The delimiter to use Returns: Column: A column containing the joined strings Join array with comma ``` # Join array elements with comma df.select(text.array_join(col("array_column"), ",")) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def array_join(column: ColumnOrName, delimiter: str) -> Column: """Joins an array of strings into a single string with a delimiter. Args: column: The column to join delimiter: The delimiter to use Returns: Column: A column containing the joined strings Example: Join array with comma ```python # Join array elements with comma df.select(text.array_join(col("array_column"), ",")) ``` """ return Column._from_logical_expr( ArrayJoinExpr(Column._from_col_or_name(column)._logical_expr, delimiter) ) ``` ## btrim ``` btrim(col: ColumnOrName, trim: Optional[Union[Column, str]]) -> Column ``` Remove specified characters from both sides of strings in a column. This function removes all occurrences of the specified characters from both the beginning and end of each string in the column. If trim is a column expression, the characters to remove are determined dynamically from the values in that column. Parameters: - **`col`** (`ColumnOrName`) – The input string column or column name to trim - **`trim`** (`Optional[Union[Column, str]]`) – The characters to remove from both sides (Default: whitespace) Can be a string or column expression. Returns: - **`Column`** ( `Column` ) – A column containing the trimmed strings Remove brackets from both sides ``` # Remove brackets from both sides of text df.select(text.btrim(col("text"), "[]")) ``` Remove characters specified in a column ``` # Remove characters specified in a column df.select(text.btrim(col("text"), col("chars"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def btrim(col: ColumnOrName, trim: Optional[Union[Column, str]]) -> Column: """Remove specified characters from both sides of strings in a column. This function removes all occurrences of the specified characters from both the beginning and end of each string in the column. If trim is a column expression, the characters to remove are determined dynamically from the values in that column. Args: col: The input string column or column name to trim trim: The characters to remove from both sides (Default: whitespace) Can be a string or column expression. Returns: Column: A column containing the trimmed strings Example: Remove brackets from both sides ```python # Remove brackets from both sides of text df.select(text.btrim(col("text"), "[]")) ``` Example: Remove characters specified in a column ```python # Remove characters specified in a column df.select(text.btrim(col("text"), col("chars"))) ``` """ if trim is None: trim_expr = None elif isinstance(trim, Column): trim_expr = trim._logical_expr else: trim_expr = lit(trim)._logical_expr return Column._from_logical_expr( StripCharsExpr(Column._from_col_or_name(col)._logical_expr, trim_expr, "both") ) ``` ## byte_length ``` byte_length(column: ColumnOrName) -> Column ``` Calculate the byte length of each string in the column. Parameters: - **`column`** (`ColumnOrName`) – The input string column to calculate byte lengths for Returns: - **`Column`** ( `Column` ) – A column containing the byte length of each string Get byte lengths ``` # Get the byte length of each string in the name column df.select(text.byte_length(col("name"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def byte_length(column: ColumnOrName) -> Column: """Calculate the byte length of each string in the column. Args: column: The input string column to calculate byte lengths for Returns: Column: A column containing the byte length of each string Example: Get byte lengths ```python # Get the byte length of each string in the name column df.select(text.byte_length(col("name"))) ``` """ return Column._from_logical_expr( ByteLengthExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## character_chunk ``` character_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0) -> Column ``` Chunks a string column into chunks of a specified size (in characters) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in characters - **`chunk_overlap_percentage`** (`int`, default: `0` ) – The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Create character chunks ``` # Create chunks of 100 characters with 20% overlap df.select(text.character_chunk(col("text"), 100, 20)) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def character_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0 ) -> Column: """Chunks a string column into chunks of a specified size (in characters) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in characters chunk_overlap_percentage: The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: Column: A column containing the chunks as an array of strings Example: Create character chunks ```python # Create chunks of 100 characters with 20% overlap df.select(text.character_chunk(col("text"), 100, 20)) ``` """ return Column._from_logical_expr( TextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=TextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.CHARACTER, ) ) ) ``` ## compute_fuzzy_ratio ``` compute_fuzzy_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = 'indel') -> Column ``` Compute the similarity between two strings using a fuzzy string matching algorithm. This function computes a fuzzy similarity score between two string columns (or a string column and a literal string) for each row. It supports multiple well-known string similarity metrics, including Levenshtein, Damerau-Levenshtein, Jaro, Jaro-Winkler, and Hamming. The returned score is a similarity percentage between 0 and 100, where: - 100 indicates the strings are identical - 0 indicates maximum dissimilarity (as defined by the method) Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.ratio Parameters: - **`column`** (`ColumnOrName`) – A string column or column name. This is the left-hand side of the comparison. - **`other`** (`Union[Column, str]`) – A second string column or literal string. This is the right-hand side of the comparison. - **`method`** (`FuzzySimilarityMethod`, default: `'indel'` ) – A string indicating which similarity method to use. Must be one of: - `"indel"`: Indel distance β€” counts only insertions and deletions (no substitutions); based on the Longest Common Subsequence. - `"levenshtein"`: Levenshtein distance (edit distance) - `"damerau_levenshtein"`: Damerau-Levenshtein distance (includes transpositions) - `"jaro"`: Jaro similarity, accounts for transpositions and proximity - `"jaro_winkler"`: Jaro-Winkler similarity, gives higher scores for common prefixes - `"hamming"`: Hamming distance. Counts differing positions between two equal-length strings, padding shorter string if needed. Returns: - **`Column`** ( `Column` ) – A double column with similarity scores in the range [0, 100]. Compare two columns ``` result = df.select( compute_fuzzy_ratio(col("a"), col("b"), method="levenshtein").alias("sim") ) ``` Compare a column to a literal string ``` result = df.select( compute_fuzzy_ratio(col("a"), "world", method="jaro").alias("sim_to_world") ) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def compute_fuzzy_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = "indel") -> Column: """Compute the similarity between two strings using a fuzzy string matching algorithm. This function computes a fuzzy similarity score between two string columns (or a string column and a literal string) for each row. It supports multiple well-known string similarity metrics, including Levenshtein, Damerau-Levenshtein, Jaro, Jaro-Winkler, and Hamming. The returned score is a similarity percentage between 0 and 100, where: - 100 indicates the strings are identical - 0 indicates maximum dissimilarity (as defined by the method) Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.ratio Args: column: A string column or column name. This is the left-hand side of the comparison. other: A second string column or literal string. This is the right-hand side of the comparison. method: A string indicating which similarity method to use. Must be one of: - `"indel"`: Indel distance β€” counts only insertions and deletions (no substitutions); based on the Longest Common Subsequence. - `"levenshtein"`: Levenshtein distance (edit distance) - `"damerau_levenshtein"`: Damerau-Levenshtein distance (includes transpositions) - `"jaro"`: Jaro similarity, accounts for transpositions and proximity - `"jaro_winkler"`: Jaro-Winkler similarity, gives higher scores for common prefixes - `"hamming"`: Hamming distance. Counts differing positions between two equal-length strings, padding shorter string if needed. Returns: Column: A double column with similarity scores in the range [0, 100]. Example: Compare two columns ```python result = df.select( compute_fuzzy_ratio(col("a"), col("b"), method="levenshtein").alias("sim") ) ``` Example: Compare a column to a literal string ```python result = df.select( compute_fuzzy_ratio(col("a"), "world", method="jaro").alias("sim_to_world") ) ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(FuzzyRatioExpr(Column._from_col_or_name(column)._logical_expr, other_expr, method)) ``` ## compute_fuzzy_token_set_ratio ``` compute_fuzzy_token_set_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = 'indel') -> Column ``` Compute fuzzy similarity using token set comparison. Tokenizes strings by whitespace, creates sets of unique tokens, then compares three combinations: diff1 vs diff2, intersection vs left set, and intersection vs right set. Returns the maximum similarity score. Useful for comparing strings where both word order and duplicates don't matter. Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.token_set_ratio Parameters: - **`column`** (`ColumnOrName`) – First string column to compare - **`other`** (`Union[Column, str]`) – Second string column or literal string to compare against - **`method`** (`FuzzySimilarityMethod`, default: `'indel'` ) – Similarity algorithm to use for comparison Returns: - `Column` – Double column with similarity scores between 0 and 100 Example ``` # df.select(compute_fuzzy_token_set_ratio(col("city"), "city of new york", "indel")) # "new york city new" β†’ unique tokens: {"city", "new", "york"} # "city of new york" β†’ unique tokens: {"city", "new", "of", "york"} # intersection: {"city", "new", "york"} # diff1: {} (empty) # diff2: {"of"} # Compares: diff1 vs diff2, intersection vs set1, intersection vs set2 # Returns max similarity score = 100 ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def compute_fuzzy_token_set_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = "indel") -> Column: """Compute fuzzy similarity using token set comparison. Tokenizes strings by whitespace, creates sets of unique tokens, then compares three combinations: diff1 vs diff2, intersection vs left set, and intersection vs right set. Returns the maximum similarity score. Useful for comparing strings where both word order and duplicates don't matter. Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.token_set_ratio Args: column: First string column to compare other: Second string column or literal string to compare against method: Similarity algorithm to use for comparison Returns: Double column with similarity scores between 0 and 100 Example: ```python # df.select(compute_fuzzy_token_set_ratio(col("city"), "city of new york", "indel")) # "new york city new" β†’ unique tokens: {"city", "new", "york"} # "city of new york" β†’ unique tokens: {"city", "new", "of", "york"} # intersection: {"city", "new", "york"} # diff1: {} (empty) # diff2: {"of"} # Compares: diff1 vs diff2, intersection vs set1, intersection vs set2 # Returns max similarity score = 100 ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(FuzzyTokenSetRatioExpr(Column._from_col_or_name(column)._logical_expr, other_expr, method)) ``` ## compute_fuzzy_token_sort_ratio ``` compute_fuzzy_token_sort_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = 'indel') -> Column ``` Compute fuzzy similarity after sorting tokens in each string. Tokenizes strings by whitespace, sorts tokens alphabetically, concatenates them back into a string, then applies the specified similarity metric. Useful for comparing strings where word order doesn't matter. Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.token_sort_ratio Parameters: - **`column`** (`ColumnOrName`) – First string column to compare - **`other`** (`Union[Column, str]`) – Second string column or literal string to compare against - **`method`** (`FuzzySimilarityMethod`, default: `'indel'` ) – Similarity algorithm to use after token sorting Returns: - `Column` – Double column with similarity scores between 0 and 100 Example ``` # df.select(compute_fuzzy_token_sort_ratio(col("city"), "city new york", "levenshtein")) # "new york city" β†’ ["new", "york", "city"] β†’ sorted β†’ ["city", "new", "york"] β†’ "city new york" # "city new york" β†’ ["city", "new", "york"] β†’ sorted β†’ ["city", "new", "york"] β†’ "city new york" # levenshtein similarity("city new york", "city new york") = 100 ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def compute_fuzzy_token_sort_ratio(column: ColumnOrName, other: Union[Column, str], method: FuzzySimilarityMethod = "indel") -> Column: """Compute fuzzy similarity after sorting tokens in each string. Tokenizes strings by whitespace, sorts tokens alphabetically, concatenates them back into a string, then applies the specified similarity metric. Useful for comparing strings where word order doesn't matter. Based on https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html#rapidfuzz.fuzz.token_sort_ratio Args: column: First string column to compare other: Second string column or literal string to compare against method: Similarity algorithm to use after token sorting Returns: Double column with similarity scores between 0 and 100 Example: ```python # df.select(compute_fuzzy_token_sort_ratio(col("city"), "city new york", "levenshtein")) # "new york city" β†’ ["new", "york", "city"] β†’ sorted β†’ ["city", "new", "york"] β†’ "city new york" # "city new york" β†’ ["city", "new", "york"] β†’ sorted β†’ ["city", "new", "york"] β†’ "city new york" # levenshtein similarity("city new york", "city new york") = 100 ``` """ if isinstance(other, str): other_expr = LiteralExpr(other, StringType) else: other_expr = other._logical_expr return Column._from_logical_expr(FuzzyTokenSortRatioExpr(Column._from_col_or_name(column)._logical_expr, other_expr, method)) ``` ## concat ``` concat(*cols: ColumnOrName) -> Column ``` Concatenates multiple columns or strings into a single string. Parameters: - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns or strings to concatenate Returns: - **`Column`** ( `Column` ) – A column containing the concatenated strings Concatenate columns ``` # Concatenate two columns with a space in between df.select(text.concat(col("col1"), lit(" "), col("col2"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def concat(*cols: ColumnOrName) -> Column: """Concatenates multiple columns or strings into a single string. Args: *cols: Columns or strings to concatenate Returns: Column: A column containing the concatenated strings Example: Concatenate columns ```python # Concatenate two columns with a space in between df.select(text.concat(col("col1"), lit(" "), col("col2"))) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the concat method.") flattened_args = [] for arg in cols: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) flattened_exprs = [ Column._from_col_or_name(c)._logical_expr for c in flattened_args ] return Column._from_logical_expr(ConcatExpr(flattened_exprs)) ``` ## concat_ws ``` concat_ws(separator: str, *cols: ColumnOrName) -> Column ``` Concatenates multiple columns or strings into a single string with a separator. Parameters: - **`separator`** (`str`) – The separator to use - **`*cols`** (`ColumnOrName`, default: `()` ) – Columns or strings to concatenate Returns: - **`Column`** ( `Column` ) – A column containing the concatenated strings Concatenate with comma separator ``` # Concatenate columns with comma separator df.select(text.concat_ws(",", col("col1"), col("col2"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def concat_ws(separator: str, *cols: ColumnOrName) -> Column: """Concatenates multiple columns or strings into a single string with a separator. Args: separator: The separator to use *cols: Columns or strings to concatenate Returns: Column: A column containing the concatenated strings Example: Concatenate with comma separator ```python # Concatenate columns with comma separator df.select(text.concat_ws(",", col("col1"), col("col2"))) ``` """ if not cols: raise ValidationError("No columns were provided. Please specify at least one column to use with the concat_ws method.") flattened_args = [] for arg in cols: if isinstance(arg, (list, tuple)): flattened_args.extend(arg) else: flattened_args.append(arg) expr_args = [] for arg in flattened_args: expr_args.append(Column._from_col_or_name(arg)._logical_expr) expr_args.append(lit(separator)._logical_expr) expr_args.pop() return Column._from_logical_expr(ConcatExpr(expr_args)) ``` ## count_tokens ``` count_tokens(column: ColumnOrName) -> Column ``` Returns the number of tokens in a string using OpenAI's cl100k_base encoding (tiktoken). Parameters: - **`column`** (`ColumnOrName`) – The input string column. Returns: - **`Column`** ( `Column` ) – A column with the token counts for each input string. Count tokens in text ``` # Count tokens in a text column df.select(text.count_tokens(col("text"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def count_tokens( column: ColumnOrName, ) -> Column: r"""Returns the number of tokens in a string using OpenAI's cl100k_base encoding (tiktoken). Args: column: The input string column. Returns: Column: A column with the token counts for each input string. Example: Count tokens in text ```python # Count tokens in a text column df.select(text.count_tokens(col("text"))) ``` """ return Column._from_logical_expr( CountTokensExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## extract ``` extract(column: ColumnOrName, template: str) -> Column ``` Extracts structured data from text using template-based pattern matching. Matches each string in the input column against a template pattern with named placeholders. Each placeholder can specify a format rule to handle different data types within the text. Parameters: - **`column`** (`ColumnOrName`) – Input text column to extract from - **`template`** (`str`) – Template string with placeholders as `${field_name}` or `${field_name:format}` Available formats: none, csv, json, quoted Returns: - **`Column`** ( `Column` ) – Struct column with fields corresponding to template placeholders. All fields are strings except JSON fields which preserve their parsed type. Template Syntax - `${field_name}` - Extract field as plain text - `${field_name:csv}` - Parse as CSV field (handles quoted values) - `${field_name:json}` - Parse as JSON and preserve type - `${field_name:quoted}` - Extract quoted string (removes outer quotes) - `$` - Literal dollar sign Raises: - `ValidationError` – If template syntax is invalid Basic extraction ``` text.extract(col("log"), "${date} ${level} ${message}") # Input: "2024-01-15 ERROR Connection failed" # Output: {date: "2024-01-15", level: "ERROR", message: "Connection failed"} ``` Mixed format extraction ``` text.extract(col("data"), 'Name: ${name:csv}, Price: ${price}, Tags: ${tags:json}') # Input: 'Name: "Smith, John", Price: 99.99, Tags: ["a", "b"]' # Output: {name: "Smith, John", price: "99.99", tags: ["a", "b"]} ``` Quoted field handling ``` text.extract(col("record"), 'Title: ${title:quoted}, Author: ${author}') # Input: 'Title: "To Kill a Mockingbird", Author: Harper Lee' # Output: {title: "To Kill a Mockingbird", author: "Harper Lee"} ``` Note If a string doesn't match the template pattern, all extracted fields will be null. Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def extract(column: ColumnOrName, template: str) -> Column: """Extracts structured data from text using template-based pattern matching. Matches each string in the input column against a template pattern with named placeholders. Each placeholder can specify a format rule to handle different data types within the text. Args: column: Input text column to extract from template: Template string with placeholders as ``${field_name}`` or ``${field_name:format}`` Available formats: none, csv, json, quoted Returns: Column: Struct column with fields corresponding to template placeholders. All fields are strings except JSON fields which preserve their parsed type. Template Syntax: - ``${field_name}`` - Extract field as plain text - ``${field_name:csv}`` - Parse as CSV field (handles quoted values) - ``${field_name:json}`` - Parse as JSON and preserve type - ``${field_name:quoted}`` - Extract quoted string (removes outer quotes) - ``$`` - Literal dollar sign Raises: ValidationError: If template syntax is invalid Example: Basic extraction ```python text.extract(col("log"), "${date} ${level} ${message}") # Input: "2024-01-15 ERROR Connection failed" # Output: {date: "2024-01-15", level: "ERROR", message: "Connection failed"} ``` Example: Mixed format extraction ```python text.extract(col("data"), 'Name: ${name:csv}, Price: ${price}, Tags: ${tags:json}') # Input: 'Name: "Smith, John", Price: 99.99, Tags: ["a", "b"]' # Output: {name: "Smith, John", price: "99.99", tags: ["a", "b"]} ``` Example: Quoted field handling ```python text.extract(col("record"), 'Title: ${title:quoted}, Author: ${author}') # Input: 'Title: "To Kill a Mockingbird", Author: Harper Lee' # Output: {title: "To Kill a Mockingbird", author: "Harper Lee"} ``` Note: If a string doesn't match the template pattern, all extracted fields will be null. """ return Column._from_logical_expr( TextractExpr(Column._from_col_or_name(column)._logical_expr, template) ) ``` ## jinja ``` jinja(jinja_template: str, /, strict: bool = True, **columns: Column) -> Column ``` Render a Jinja template using values from the specified columns. This function evaluates a Jinja2 template string for each row, using the provided columns as template variables. Only a subset of Jinja2 features is supported. Parameters: - **`jinja_template`** (`str`) – A Jinja2 template string to render for each row. Variables are referenced using double braces: {{ variable_name }} - **`strict`** (`bool`, default: `True` ) – If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. - **`**columns`** (`Column`, default: `{}` ) – Keyword arguments mapping variable names to columns. Each keyword becomes a variable in the template context. Returns: - **`Column`** ( `Column` ) – A string column containing the rendered template for each row Supported Features - Variable substitution: {{ variable }} - Struct/object field access: {{ user.name }} - Array indexing with literals: {{ items[0] }}, {{ data["key"] }} - For loops: {% for item in items %}...{% endfor %} - If/elif/else conditionals: {% if condition %}...{% endif %} - Loop variables: {{ loop.index }}, {{ loop.first }}, etc. - Constants: {{ "literal string" }}, {{ 42 }} Not Supported (use column expressions instead): - **Filters**: {{ name|upper }} β†’ Use upper_name=fc.upper(col("name")) - **Function calls**: {{ len(items) }} β†’ Use item_count=fc.array_size(col("items")) - **Operators**: {% if price > 100 %} β†’ Use is_expensive=(col("price") > 100) - **Arithmetic**: {{ price \* quantity }} β†’ Use total=col("price") \* col("quantity") - **Dynamic indexing**: {{ items[i] }} β†’ Use item=(fc.col("items").get_item(col("index"))) - **Variable assignment**: {% set x = 5 %} β†’ Pre-compute as column expression - **Macros, includes, extends**: Not supported LLM prompt formatting with conditional context and examples ``` # Format prompts with user query, conditional context, and examples prompt_template = ''' Answer the user's question. {% if context %} Context: {{ context }} {% endif %} {% if examples %} Few-shot examples: {% for ex in examples %} Q: {{ ex.question }} A: {{ ex.answer }} {% endfor %} {% endif %} Question: {{ query }} Please provide a {{ style }} response.''' # Generate prompts with varying context based on query type result = df.select( text.jinja( prompt_template, # Direct columns query=col("user_question"), context=col("retrieved_context"), # Can be null for some rows # Column expression for conditional logic style=fc.when(col("query_type") == "technical", "detailed and technical") .when(col("query_type") == "casual", "conversational") .otherwise("clear and concise"), # Array of examples (struct array) examples=col("few_shot_examples") # Array of {question, answer} structs ).alias("llm_prompt") ) ``` Notes - Template syntax is validated at query planning time - Complex operations can use column expressions - Arrays can only be iterated with {% for %} or accessed with literal indices - Structs can only use literal field names - Null values are rendered according to Jinja2's null rendering behavior Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def jinja( jinja_template: str, /, strict: bool = True, **columns: Column ) -> Column: """Render a Jinja template using values from the specified columns. This function evaluates a Jinja2 template string for each row, using the provided columns as template variables. Only a subset of Jinja2 features is supported. Args: jinja_template: A Jinja2 template string to render for each row. Variables are referenced using double braces: {{ variable_name }} strict: If True, when any of the provided columns has a None value for a row, the entire row's output will be None (template is not rendered). If False, None values are handled using Jinja2's null rendering behavior. Default is True. **columns: Keyword arguments mapping variable names to columns. Each keyword becomes a variable in the template context. Returns: Column: A string column containing the rendered template for each row Supported Features: - Variable substitution: {{ variable }} - Struct/object field access: {{ user.name }} - Array indexing with literals: {{ items[0] }}, {{ data["key"] }} - For loops: {% for item in items %}...{% endfor %} - If/elif/else conditionals: {% if condition %}...{% endif %} - Loop variables: {{ loop.index }}, {{ loop.first }}, etc. - Constants: {{ "literal string" }}, {{ 42 }} Not Supported (use column expressions instead): - **Filters**: {{ name|upper }} β†’ Use upper_name=fc.upper(col("name")) - **Function calls**: {{ len(items) }} β†’ Use item_count=fc.array_size(col("items")) - **Operators**: {% if price > 100 %} β†’ Use is_expensive=(col("price") > 100) - **Arithmetic**: {{ price * quantity }} β†’ Use total=col("price") * col("quantity") - **Dynamic indexing**: {{ items[i] }} β†’ Use item=(fc.col("items").get_item(col("index"))) - **Variable assignment**: {% set x = 5 %} β†’ Pre-compute as column expression - **Macros, includes, extends**: Not supported Example: LLM prompt formatting with conditional context and examples ```python # Format prompts with user query, conditional context, and examples prompt_template = ''' Answer the user's question. {% if context %} Context: {{ context }} {% endif %} {% if examples %} Few-shot examples: {% for ex in examples %} Q: {{ ex.question }} A: {{ ex.answer }} {% endfor %} {% endif %} Question: {{ query }} Please provide a {{ style }} response.''' # Generate prompts with varying context based on query type result = df.select( text.jinja( prompt_template, # Direct columns query=col("user_question"), context=col("retrieved_context"), # Can be null for some rows # Column expression for conditional logic style=fc.when(col("query_type") == "technical", "detailed and technical") .when(col("query_type") == "casual", "conversational") .otherwise("clear and concise"), # Array of examples (struct array) examples=col("few_shot_examples") # Array of {question, answer} structs ).alias("llm_prompt") ) ``` Notes: - Template syntax is validated at query planning time - Complex operations can use column expressions - Arrays can only be iterated with {% for %} or accessed with literal indices - Structs can only use literal field names - Null values are rendered according to Jinja2's null rendering behavior """ # Convert keyword arguments to column expressions with proper names column_exprs: List[LogicalExpr] = [] for var_name, column in columns.items(): if isinstance(column._logical_expr, ColumnExpr) and column._logical_expr.name == var_name: column_exprs.append(column._logical_expr) else: column_exprs.append(column.alias(var_name)._logical_expr) return Column._from_logical_expr( JinjaExpr(column_exprs, jinja_template, strict) ) ``` ## length ``` length(column: ColumnOrName) -> Column ``` Calculate the character length of each string in the column. Parameters: - **`column`** (`ColumnOrName`) – The input string column to calculate lengths for Returns: - **`Column`** ( `Column` ) – A column containing the length of each string in characters Get string lengths ``` # Get the length of each string in the name column df.select(text.length(col("name"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def length(column: ColumnOrName) -> Column: """Calculate the character length of each string in the column. Args: column: The input string column to calculate lengths for Returns: Column: A column containing the length of each string in characters Example: Get string lengths ```python # Get the length of each string in the name column df.select(text.length(col("name"))) ``` """ return Column._from_logical_expr( StrLengthExpr(Column._from_col_or_name(column)._logical_expr) ) ``` ## lower ``` lower(column: ColumnOrName) -> Column ``` Convert all characters in a string column to lowercase. Parameters: - **`column`** (`ColumnOrName`) – The input string column to convert to lowercase Returns: - **`Column`** ( `Column` ) – A column containing the lowercase strings Convert text to lowercase ``` # Convert all text in the name column to lowercase df.select(text.lower(col("name"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def lower(column: ColumnOrName) -> Column: """Convert all characters in a string column to lowercase. Args: column: The input string column to convert to lowercase Returns: Column: A column containing the lowercase strings Example: Convert text to lowercase ```python # Convert all text in the name column to lowercase df.select(text.lower(col("name"))) ``` """ return Column._from_logical_expr( StringCasingExpr(Column._from_col_or_name(column)._logical_expr, "lower") ) ``` ## ltrim ``` ltrim(col: ColumnOrName) -> Column ``` Remove whitespace from the start of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from the beginning of each string in the column. Parameters: - **`col`** (`ColumnOrName`) – The input string column or column name to trim Returns: - **`Column`** ( `Column` ) – A column containing the left-trimmed strings Remove leading whitespace ``` # Remove whitespace from the start of text df.select(text.ltrim(col("text"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def ltrim(col: ColumnOrName) -> Column: """Remove whitespace from the start of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from the beginning of each string in the column. Args: col: The input string column or column name to trim Returns: Column: A column containing the left-trimmed strings Example: Remove leading whitespace ```python # Remove whitespace from the start of text df.select(text.ltrim(col("text"))) ``` """ return Column._from_logical_expr( StripCharsExpr(Column._from_col_or_name(col)._logical_expr, None, "left") ) ``` ## parse_transcript ``` parse_transcript(column: ColumnOrName, format: TranscriptFormatType) -> Column ``` Parses a transcript from text to a structured format with unified schema. Converts transcript text in various formats (srt, webvtt, generic) to a standardized structure with fields: index, speaker, start_time, end_time, duration, content, format. All timestamps are returned as floating-point seconds from the start. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name containing transcript text - **`format`** (`TranscriptFormatType`) – The format of the transcript ("srt", "webvtt", or "generic") Returns: - **`Column`** ( `Column` ) – A column containing an array of structured transcript entries with unified schema: - index: Optional[int] - Entry index (1-based) - speaker: Optional[str] - Speaker name (for generic format) - start_time: float - Start time in seconds - end_time: Optional[float] - End time in seconds - duration: Optional[float] - Duration in seconds - content: str - Transcript content/text - format: str - Original format ("srt", "webvtt", or "generic") Examples: ``` >>> # Parse SRT format transcript >>> df.select(text.parse_transcript(col("transcript"), "srt")) >>> # Parse generic conversation transcript >>> df.select(text.parse_transcript(col("transcript"), "generic")) >>> # Parse WebVTT format transcript >>> df.select(text.parse_transcript(col("transcript"), "webvtt")) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def parse_transcript(column: ColumnOrName, format: TranscriptFormatType) -> Column: """Parses a transcript from text to a structured format with unified schema. Converts transcript text in various formats (srt, webvtt, generic) to a standardized structure with fields: index, speaker, start_time, end_time, duration, content, format. All timestamps are returned as floating-point seconds from the start. Args: column: The input string column or column name containing transcript text format: The format of the transcript ("srt", "webvtt", or "generic") Returns: Column: A column containing an array of structured transcript entries with unified schema: - index: Optional[int] - Entry index (1-based) - speaker: Optional[str] - Speaker name (for generic format) - start_time: float - Start time in seconds - end_time: Optional[float] - End time in seconds - duration: Optional[float] - Duration in seconds - content: str - Transcript content/text - format: str - Original format ("srt", "webvtt", or "generic") Examples: >>> # Parse SRT format transcript >>> df.select(text.parse_transcript(col("transcript"), "srt")) >>> # Parse generic conversation transcript >>> df.select(text.parse_transcript(col("transcript"), "generic")) >>> # Parse WebVTT format transcript >>> df.select(text.parse_transcript(col("transcript"), "webvtt")) """ return Column._from_logical_expr( TsParseExpr(Column._from_col_or_name(column)._logical_expr, format) ) ``` ## recursive_character_chunk ``` recursive_character_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None) -> Column ``` Chunks a string column into chunks of a specified size (in characters) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in characters - **`chunk_overlap_percentage`** (`int`) – The overlap between each chunk as a percentage of the chunk size - **`chunking_character_set_custom_characters`** (`Optional`, default: `None` ) – List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Default character chunking ``` # Create chunks of at most 100 characters with 20% overlap df.select( text.recursive_character_chunk(col("text"), 100, 20).alias("chunks") ) ``` Custom character chunking ``` # Create chunks with custom split characters df.select( text.recursive_character_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def recursive_character_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None, ) -> Column: r"""Chunks a string column into chunks of a specified size (in characters) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in characters chunk_overlap_percentage: The overlap between each chunk as a percentage of the chunk size chunking_character_set_custom_characters (Optional): List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: Column: A column containing the chunks as an array of strings Example: Default character chunking ```python # Create chunks of at most 100 characters with 20% overlap df.select( text.recursive_character_chunk(col("text"), 100, 20).alias("chunks") ) ``` Example: Custom character chunking ```python # Create chunks with custom split characters df.select( text.recursive_character_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` """ if chunking_character_set_custom_characters is None: chunking_character_set_name = ChunkCharacterSet.ASCII else: chunking_character_set_name = ChunkCharacterSet.CUSTOM return Column._from_logical_expr( RecursiveTextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=RecursiveTextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.CHARACTER, chunking_character_set_name=chunking_character_set_name, chunking_character_set_custom_characters=chunking_character_set_custom_characters, ) ) ) ``` ## recursive_token_chunk ``` recursive_token_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None) -> Column ``` Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in tokens - **`chunk_overlap_percentage`** (`int`) – The overlap between each chunk as a percentage of the chunk size - **`chunking_character_set_custom_characters`** (`Optional`, default: `None` ) – List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Default token chunking ``` # Create chunks of at most 100 tokens with 20% overlap df.select( text.recursive_token_chunk(col("text"), 100, 20).alias("chunks") ) ``` Custom token chunking ``` # Create chunks with custom split characters df.select( text.recursive_token_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def recursive_token_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None, ) -> Column: r"""Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in tokens chunk_overlap_percentage: The overlap between each chunk as a percentage of the chunk size chunking_character_set_custom_characters (Optional): List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: Column: A column containing the chunks as an array of strings Example: Default token chunking ```python # Create chunks of at most 100 tokens with 20% overlap df.select( text.recursive_token_chunk(col("text"), 100, 20).alias("chunks") ) ``` Example: Custom token chunking ```python # Create chunks with custom split characters df.select( text.recursive_token_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` """ if chunking_character_set_custom_characters is None: chunking_character_set_name = ChunkCharacterSet.ASCII else: chunking_character_set_name = ChunkCharacterSet.CUSTOM return Column._from_logical_expr( RecursiveTextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=RecursiveTextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.TOKEN, chunking_character_set_name=chunking_character_set_name, chunking_character_set_custom_characters=chunking_character_set_custom_characters, ) ) ) ``` ## recursive_word_chunk ``` recursive_word_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None) -> Column ``` Chunks a string column into chunks of a specified size (in words) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in words - **`chunk_overlap_percentage`** (`int`) – The overlap between each chunk as a percentage of the chunk size - **`chunking_character_set_custom_characters`** (`Optional`, default: `None` ) – List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Default word chunking ``` # Create chunks of at most 100 words with 20% overlap df.select( text.recursive_word_chunk(col("text"), 100, 20).alias("chunks") ) ``` Custom word chunking ``` # Create chunks with custom split characters df.select( text.recursive_word_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def recursive_word_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int, chunking_character_set_custom_characters: Optional[list[str]] = None, ) -> Column: r"""Chunks a string column into chunks of a specified size (in words) with an optional overlap. The chunking is performed recursively, attempting to preserve the underlying structure of the text by splitting on natural boundaries (paragraph breaks, sentence breaks, etc.) to maintain context. By default, these characters are ['\n\n', '\n', '.', ';', ':', ' ', '-', ''], but this can be customized. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in words chunk_overlap_percentage: The overlap between each chunk as a percentage of the chunk size chunking_character_set_custom_characters (Optional): List of alternative characters to split on. Note that the characters should be ordered from coarsest to finest desired granularity -- earlier characters in the list should result in fewer overall splits than later characters. Returns: Column: A column containing the chunks as an array of strings Example: Default word chunking ```python # Create chunks of at most 100 words with 20% overlap df.select( text.recursive_word_chunk(col("text"), 100, 20).alias("chunks") ) ``` Example: Custom word chunking ```python # Create chunks with custom split characters df.select( text.recursive_word_chunk( col("text"), 100, 20, ['\n\n', '\n', '.', ' ', ''] ).alias("chunks") ) ``` """ if chunking_character_set_custom_characters is None: chunking_character_set_name = ChunkCharacterSet.ASCII else: chunking_character_set_name = ChunkCharacterSet.CUSTOM return Column._from_logical_expr( RecursiveTextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=RecursiveTextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.WORD, chunking_character_set_name=chunking_character_set_name, chunking_character_set_custom_characters=chunking_character_set_custom_characters, ) ) ) ``` ## regexp_count ``` regexp_count(src: ColumnOrName, pattern: Union[Column, str]) -> Column ``` Count the number of times a regex pattern is matched in a string. Returns the count of matches for each string in the column. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name - **`pattern`** (`Union[Column, str]`) – The regex pattern to count (can be a string literal or column expression) Returns: - **`Column`** ( `Column` ) – An integer column containing the count of pattern matches Count digits ``` import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123", "456def789", "no digits"] }) result = df.select(fc.text.regexp_count("text", r"\d")) # Output: [3, 6, 0] ``` Count words ``` df = fc.Session.local().create_dataframe({ "text": ["hello world", "one two three"] }) result = df.select(fc.text.regexp_count("text", r"\w+")) # Output: [2, 3] ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_count(src: ColumnOrName, pattern: Union[Column, str]) -> Column: r"""Count the number of times a regex pattern is matched in a string. Returns the count of matches for each string in the column. Args: src: The input string column or column name pattern: The regex pattern to count (can be a string literal or column expression) Returns: Column: An integer column containing the count of pattern matches Example: Count digits ```python import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123", "456def789", "no digits"] }) result = df.select(fc.text.regexp_count("text", r"\d")) # Output: [3, 6, 0] ``` Example: Count words ```python df = fc.Session.local().create_dataframe({ "text": ["hello world", "one two three"] }) result = df.select(fc.text.regexp_count("text", r"\w+")) # Output: [2, 3] ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr return Column._from_logical_expr( RegexpCountExpr(Column._from_col_or_name(src)._logical_expr, pattern_expr) ) ``` ## regexp_extract ``` regexp_extract(src: ColumnOrName, pattern: Union[Column, str], idx: int) -> Column ``` Extract a specific regex group from a string. Extracts a capture group matched by the regex pattern. Group 0 is the entire match, group 1+ are capture groups. If the pattern/group has multiple matches within the same string, the result will be the first match. Use `regexp_extract_all` to extract all matches. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name - **`pattern`** (`Union[Column, str]`) – The regex pattern with capture groups (can be a string literal or column expression) - **`idx`** (`int`) – The group index to extract (0 = entire match, 1+ = capture groups) Returns: - **`Column`** ( `Column` ) – A string column containing the extracted group. If no match is found, or the specified group did not match, the result will be an empty string. Extract email username ``` import fenic as fc df = fc.Session.local().create_dataframe({ "email": ["user@domain.com", "admin@example.org"] }) result = df.select(fc.text.regexp_extract("email", r"([^@]+)@", 1)) # Output: ["user", "admin"] ``` Extract phone area code ``` df = fc.Session.local().create_dataframe({ "phone": ["(555) 123-4567", "(123) 456-7890"] }) result = df.select(fc.text.regexp_extract("phone", r"\((\d{3})\)", 1)) # Output: ["555", "123"] ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_extract( src: ColumnOrName, pattern: Union[Column, str], idx: int ) -> Column: r"""Extract a specific regex group from a string. Extracts a capture group matched by the regex pattern. Group 0 is the entire match, group 1+ are capture groups. If the pattern/group has multiple matches within the same string, the result will be the first match. Use `regexp_extract_all` to extract all matches. Args: src: The input string column or column name pattern: The regex pattern with capture groups (can be a string literal or column expression) idx: The group index to extract (0 = entire match, 1+ = capture groups) Returns: Column: A string column containing the extracted group. If no match is found, or the specified group did not match, the result will be an empty string. Example: Extract email username ```python import fenic as fc df = fc.Session.local().create_dataframe({ "email": ["user@domain.com", "admin@example.org"] }) result = df.select(fc.text.regexp_extract("email", r"([^@]+)@", 1)) # Output: ["user", "admin"] ``` Example: Extract phone area code ```python df = fc.Session.local().create_dataframe({ "phone": ["(555) 123-4567", "(123) 456-7890"] }) result = df.select(fc.text.regexp_extract("phone", r"\((\d{3})\)", 1)) # Output: ["555", "123"] ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr return Column._from_logical_expr( RegexpExtractExpr( Column._from_col_or_name(src)._logical_expr, pattern_expr, idx ) ) ``` ## regexp_extract_all ``` regexp_extract_all(src: ColumnOrName, pattern: Union[Column, str], idx: Union[Column, int] = 0) -> Column ``` Extract all strings matching a regex pattern, optionally from a specific group. Returns an array of all matches. Group 0 is the entire match, group 1+ are capture groups. If the pattern/group has multiple matches within the same string, the result will be an array of all matches. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name - **`pattern`** (`Union[Column, str]`) – The regex pattern with optional capture groups - **`idx`** (`Union[Column, int]`, default: `0` ) – The group index to extract (default: 0 for entire match, 1+ for capture groups) Returns: - **`Column`** ( `Column` ) – An array column containing all matches Extract all digits ``` import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123def456", "no digits", "789"] }) result = df.select(fc.text.regexp_extract_all("text", r"\d+")) # Output: [["123", "456"], [], ["789"]] ``` Extract all hashtags ``` df = fc.Session.local().create_dataframe({ "post": ["Love #coding and #python", "Just #relaxing"] }) result = df.select(fc.text.regexp_extract_all("post", r"#(\w+)", 1)) # Output: [["coding", "python"], ["relaxing"]] ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_extract_all( src: ColumnOrName, pattern: Union[Column, str], idx: Union[Column, int] = 0 ) -> Column: r"""Extract all strings matching a regex pattern, optionally from a specific group. Returns an array of all matches. Group 0 is the entire match, group 1+ are capture groups. If the pattern/group has multiple matches within the same string, the result will be an array of all matches. Args: src: The input string column or column name pattern: The regex pattern with optional capture groups idx: The group index to extract (default: 0 for entire match, 1+ for capture groups) Returns: Column: An array column containing all matches Example: Extract all digits ```python import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123def456", "no digits", "789"] }) result = df.select(fc.text.regexp_extract_all("text", r"\d+")) # Output: [["123", "456"], [], ["789"]] ``` Example: Extract all hashtags ```python df = fc.Session.local().create_dataframe({ "post": ["Love #coding and #python", "Just #relaxing"] }) result = df.select(fc.text.regexp_extract_all("post", r"#(\w+)", 1)) # Output: [["coding", "python"], ["relaxing"]] ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr if isinstance(idx, Column): idx_expr = idx._logical_expr else: idx_expr = lit(idx)._logical_expr return Column._from_logical_expr( RegexpExtractAllExpr( Column._from_col_or_name(src)._logical_expr, pattern_expr, idx_expr ) ) ``` ## regexp_instr ``` regexp_instr(src: ColumnOrName, pattern: Union[Column, str], idx: Union[Column, int] = 0) -> Column ``` Find the 1-based position of the first regex match in a string. Returns the position (1-based) of the first substring matching the pattern. Returns 0 if no match is found. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name - **`pattern`** (`Union[Column, str]`) – The regex pattern to search for - **`idx`** (`Union[Column, int]`, default: `0` ) – The group index to locate (default: 0 for entire match, 1+ for capture groups) Returns: - **`Column`** ( `Column` ) – An integer column with 1-based position (0 if no match) Find position of first digit ``` import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123", "no digits", "456xyz"] }) result = df.select(fc.text.regexp_instr("text", r"\d")) # Output: [4, 0, 1] # 1-based positions ``` Find position of email ``` df = fc.Session.local().create_dataframe({ "text": ["Contact: user@domain.com", "No email here"] }) result = df.select(fc.text.regexp_instr("text", r"[^@]+@[^@]+")) # Output: [10, 0] ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_instr( src: ColumnOrName, pattern: Union[Column, str], idx: Union[Column, int] = 0 ) -> Column: r"""Find the 1-based position of the first regex match in a string. Returns the position (1-based) of the first substring matching the pattern. Returns 0 if no match is found. Args: src: The input string column or column name pattern: The regex pattern to search for idx: The group index to locate (default: 0 for entire match, 1+ for capture groups) Returns: Column: An integer column with 1-based position (0 if no match) Example: Find position of first digit ```python import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["abc123", "no digits", "456xyz"] }) result = df.select(fc.text.regexp_instr("text", r"\d")) # Output: [4, 0, 1] # 1-based positions ``` Example: Find position of email ```python df = fc.Session.local().create_dataframe({ "text": ["Contact: user@domain.com", "No email here"] }) result = df.select(fc.text.regexp_instr("text", r"[^@]+@[^@]+")) # Output: [10, 0] ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr if isinstance(idx, Column): idx_expr = idx._logical_expr else: idx_expr = lit(idx)._logical_expr return Column._from_logical_expr( RegexpInstrExpr( Column._from_col_or_name(src)._logical_expr, pattern_expr, idx_expr ) ) ``` ## regexp_replace ``` regexp_replace(src: ColumnOrName, pattern: Union[Column, str], replacement: Union[Column, str]) -> Column ``` Replace all occurrences of a pattern with a new string, treating pattern as a regular expression. This method creates a new string column with all occurrences of the specified pattern replaced with a new string. The pattern is treated as a regular expression. If either pattern or replacement is a column expression, the operation is performed dynamically using the values from those columns. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name to perform replacements on - **`pattern`** (`Union[Column, str]`) – The regular expression pattern to search for (can be a string or column expression) - **`replacement`** (`Union[Column, str]`) – The string to replace with (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A column containing the strings with replacements applied Replace digits with dashes ``` # Replace all digits with dashes df.select(text.regexp_replace(col("text"), r"\d+", "--")) ``` Dynamic replacement using column values ``` # Replace using patterns from columns df.select(text.regexp_replace(col("text"), col("pattern"), col("replacement"))) ``` Complex pattern replacement ``` # Replace email addresses with [REDACTED] df.select(text.regexp_replace(col("text"), r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", "[REDACTED]")) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_replace( src: ColumnOrName, pattern: Union[Column, str], replacement: Union[Column, str], ) -> Column: r"""Replace all occurrences of a pattern with a new string, treating pattern as a regular expression. This method creates a new string column with all occurrences of the specified pattern replaced with a new string. The pattern is treated as a regular expression. If either pattern or replacement is a column expression, the operation is performed dynamically using the values from those columns. Args: src: The input string column or column name to perform replacements on pattern: The regular expression pattern to search for (can be a string or column expression) replacement: The string to replace with (can be a string or column expression) Returns: Column: A column containing the strings with replacements applied Example: Replace digits with dashes ```python # Replace all digits with dashes df.select(text.regexp_replace(col("text"), r"\d+", "--")) ``` Example: Dynamic replacement using column values ```python # Replace using patterns from columns df.select(text.regexp_replace(col("text"), col("pattern"), col("replacement"))) ``` Example: Complex pattern replacement ```python # Replace email addresses with [REDACTED] df.select(text.regexp_replace(col("text"), r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", "[REDACTED]")) ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr if isinstance(replacement, Column): replacement_expr = replacement._logical_expr else: replacement_expr = lit(replacement)._logical_expr return Column._from_logical_expr( ReplaceExpr( Column._from_col_or_name(src)._logical_expr, pattern_expr, replacement_expr, False, ) ) ``` ## regexp_substr ``` regexp_substr(src: ColumnOrName, pattern: Union[Column, str]) -> Column ``` Extract the first substring matching a regex pattern. Returns the first substring that matches the pattern. Returns null if no match is found. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name - **`pattern`** (`Union[Column, str]`) – The regex pattern to search for Returns: - **`Column`** ( `Column` ) – A string column containing the first match (null if no match) Extract first number ``` import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["Price: $123.45", "No price", "Cost: $67.89"] }) result = df.select(fc.text.regexp_substr("text", r"\d+\.\d+")) # Output: ["123.45", null, "67.89"] ``` Extract first URL ``` df = fc.Session.local().create_dataframe({ "text": ["Visit https://example.com for info", "No URL here"] }) result = df.select(fc.text.regexp_substr("text", r"https?://[^\s]+")) # Output: ["https://example.com", null] ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def regexp_substr(src: ColumnOrName, pattern: Union[Column, str]) -> Column: r"""Extract the first substring matching a regex pattern. Returns the first substring that matches the pattern. Returns null if no match is found. Args: src: The input string column or column name pattern: The regex pattern to search for Returns: Column: A string column containing the first match (null if no match) Example: Extract first number ```python import fenic as fc df = fc.Session.local().create_dataframe({ "text": ["Price: $123.45", "No price", "Cost: $67.89"] }) result = df.select(fc.text.regexp_substr("text", r"\d+\.\d+")) # Output: ["123.45", null, "67.89"] ``` Example: Extract first URL ```python df = fc.Session.local().create_dataframe({ "text": ["Visit https://example.com for info", "No URL here"] }) result = df.select(fc.text.regexp_substr("text", r"https?://[^\s]+")) # Output: ["https://example.com", null] ``` """ if isinstance(pattern, Column): pattern_expr = pattern._logical_expr else: pattern_expr = lit(pattern)._logical_expr return Column._from_logical_expr( RegexpSubstrExpr(Column._from_col_or_name(src)._logical_expr, pattern_expr) ) ``` ## replace ``` replace(src: ColumnOrName, search: Union[Column, str], replace: Union[Column, str]) -> Column ``` Replace all occurrences of a pattern with a new string, treating pattern as a literal string. This method creates a new string column with all occurrences of the specified pattern replaced with a new string. The pattern is treated as a literal string, not a regular expression. If either search or replace is a column expression, the operation is performed dynamically using the values from those columns. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name to perform replacements on - **`search`** (`Union[Column, str]`) – The pattern to search for (can be a string or column expression) - **`replace`** (`Union[Column, str]`) – The string to replace with (can be a string or column expression) Returns: - **`Column`** ( `Column` ) – A column containing the strings with replacements applied Replace with literal string ``` # Replace all occurrences of "foo" in the "name" column with "bar" df.select(text.replace(col("name"), "foo", "bar")) ``` Replace using column values ``` # Replace all occurrences of the value in the "search" column with the value in the "replace" column, for each row in the "text" column df.select(text.replace(col("text"), col("search"), col("replace"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def replace( src: ColumnOrName, search: Union[Column, str], replace: Union[Column, str] ) -> Column: """Replace all occurrences of a pattern with a new string, treating pattern as a literal string. This method creates a new string column with all occurrences of the specified pattern replaced with a new string. The pattern is treated as a literal string, not a regular expression. If either search or replace is a column expression, the operation is performed dynamically using the values from those columns. Args: src: The input string column or column name to perform replacements on search: The pattern to search for (can be a string or column expression) replace: The string to replace with (can be a string or column expression) Returns: Column: A column containing the strings with replacements applied Example: Replace with literal string ```python # Replace all occurrences of "foo" in the "name" column with "bar" df.select(text.replace(col("name"), "foo", "bar")) ``` Example: Replace using column values ```python # Replace all occurrences of the value in the "search" column with the value in the "replace" column, for each row in the "text" column df.select(text.replace(col("text"), col("search"), col("replace"))) ``` """ if isinstance(search, Column): search_expr = search._logical_expr else: search_expr = lit(search)._logical_expr if isinstance(replace, Column): replace_expr = replace._logical_expr else: replace_expr = lit(replace)._logical_expr return Column._from_logical_expr( ReplaceExpr( Column._from_col_or_name(src)._logical_expr, search_expr, replace_expr, True ) ) ``` ## rtrim ``` rtrim(col: ColumnOrName) -> Column ``` Remove whitespace from the end of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from the end of each string in the column. Parameters: - **`col`** (`ColumnOrName`) – The input string column or column name to trim Returns: - **`Column`** ( `Column` ) – A column containing the right-trimmed strings Remove trailing whitespace ``` # Remove whitespace from the end of text df.select(text.rtrim(col("text"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def rtrim(col: ColumnOrName) -> Column: """Remove whitespace from the end of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from the end of each string in the column. Args: col: The input string column or column name to trim Returns: Column: A column containing the right-trimmed strings Example: Remove trailing whitespace ```python # Remove whitespace from the end of text df.select(text.rtrim(col("text"))) ``` """ return Column._from_logical_expr( StripCharsExpr(Column._from_col_or_name(col)._logical_expr, None, "right") ) ``` ## split ``` split(src: ColumnOrName, pattern: str, limit: int = -1) -> Column ``` Split a string column into an array using a regular expression pattern. This method creates an array column by splitting each value in the input string column at matches of the specified regular expression pattern. Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name to split - **`pattern`** (`str`) – The regular expression pattern to split on - **`limit`** (`int`, default: `-1` ) – Maximum number of splits to perform (Default: -1 for unlimited). If > 0, returns at most limit+1 elements, with remainder in last element. Returns: - **`Column`** ( `Column` ) – A column containing arrays of substrings Split on whitespace ``` # Split on whitespace df.select(text.split(col("text"), r"\s+")) ``` Split with limit ``` # Split on whitespace, max 2 splits df.select(text.split(col("text"), r"\s+", limit=2)) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def split(src: ColumnOrName, pattern: str, limit: int = -1) -> Column: r"""Split a string column into an array using a regular expression pattern. This method creates an array column by splitting each value in the input string column at matches of the specified regular expression pattern. Args: src: The input string column or column name to split pattern: The regular expression pattern to split on limit: Maximum number of splits to perform (Default: -1 for unlimited). If > 0, returns at most limit+1 elements, with remainder in last element. Returns: Column: A column containing arrays of substrings Example: Split on whitespace ```python # Split on whitespace df.select(text.split(col("text"), r"\s+")) ``` Example: Split with limit ```python # Split on whitespace, max 2 splits df.select(text.split(col("text"), r"\s+", limit=2)) ``` """ return Column._from_logical_expr( RegexpSplitExpr(Column._from_col_or_name(src)._logical_expr, pattern, limit) ) ``` ## split_part ``` split_part(src: ColumnOrName, delimiter: Union[Column, str], part_number: Union[Column, int]) -> Column ``` Split a string and return a specific part using 1-based indexing. Splits each string by a delimiter and returns the specified part. If the delimiter is a column expression, the split operation is performed dynamically using the delimiter values from that column. Behavior: - If any input is null, returns null - If part_number is out of range of split parts, returns empty string - If part_number is 0, throws an error - If part_number is negative, counts from the end of the split parts - If the delimiter is an empty string, the string is not split Parameters: - **`src`** (`ColumnOrName`) – The input string column or column name to split - **`delimiter`** (`Union[Column, str]`) – The delimiter to split on (can be a string or column expression) - **`part_number`** (`Union[Column, int]`) – Which part to return (1-based integer index or column expression) Returns: - **`Column`** ( `Column` ) – A column containing the specified part from each split string Get second part of comma-separated values ``` # Get second part of comma-separated values df.select(text.split_part(col("text"), ",", 2)) ``` Get last part using negative index ``` # Get last part using negative index df.select(text.split_part(col("text"), ",", -1)) ``` Use dynamic delimiter from column ``` # Use dynamic delimiter from column df.select(text.split_part(col("text"), col("delimiter"), 1)) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def split_part( src: ColumnOrName, delimiter: Union[Column, str], part_number: Union[Column, int] ) -> Column: """Split a string and return a specific part using 1-based indexing. Splits each string by a delimiter and returns the specified part. If the delimiter is a column expression, the split operation is performed dynamically using the delimiter values from that column. Behavior: - If any input is null, returns null - If part_number is out of range of split parts, returns empty string - If part_number is 0, throws an error - If part_number is negative, counts from the end of the split parts - If the delimiter is an empty string, the string is not split Args: src: The input string column or column name to split delimiter: The delimiter to split on (can be a string or column expression) part_number: Which part to return (1-based integer index or column expression) Returns: Column: A column containing the specified part from each split string Example: Get second part of comma-separated values ```python # Get second part of comma-separated values df.select(text.split_part(col("text"), ",", 2)) ``` Example: Get last part using negative index ```python # Get last part using negative index df.select(text.split_part(col("text"), ",", -1)) ``` Example: Use dynamic delimiter from column ```python # Use dynamic delimiter from column df.select(text.split_part(col("text"), col("delimiter"), 1)) ``` """ if isinstance(part_number, int) and part_number == 0: raise ValidationError( f"`split_part` expects a non-zero integer for the part_number, but got {part_number}." ) if isinstance(part_number, Column): part_number_expr = part_number._logical_expr else: part_number_expr = lit(part_number)._logical_expr if isinstance(delimiter, Column): delimiter_expr = delimiter._logical_expr else: delimiter_expr = lit(delimiter)._logical_expr return Column._from_logical_expr( SplitPartExpr( Column._from_col_or_name(src)._logical_expr, delimiter_expr, part_number_expr ) ) ``` ## title_case ``` title_case(column: ColumnOrName) -> Column ``` Convert the first character of each word in a string column to uppercase. Parameters: - **`column`** (`ColumnOrName`) – The input string column to convert to title case Returns: - **`Column`** ( `Column` ) – A column containing the title case strings Convert text to title case ``` # Convert text in the name column to title case df.select(text.title_case(col("name"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def title_case(column: ColumnOrName) -> Column: """Convert the first character of each word in a string column to uppercase. Args: column: The input string column to convert to title case Returns: Column: A column containing the title case strings Example: Convert text to title case ```python # Convert text in the name column to title case df.select(text.title_case(col("name"))) ``` """ return Column._from_logical_expr( StringCasingExpr(Column._from_col_or_name(column)._logical_expr, "title") ) ``` ## token_chunk ``` token_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0) -> Column ``` Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in tokens - **`chunk_overlap_percentage`** (`int`, default: `0` ) – The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Create token chunks ``` # Create chunks of 100 tokens with 20% overlap df.select(text.token_chunk(col("text"), 100, 20)) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def token_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0 ) -> Column: """Chunks a string column into chunks of a specified size (in tokens) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in tokens chunk_overlap_percentage: The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: Column: A column containing the chunks as an array of strings Example: Create token chunks ```python # Create chunks of 100 tokens with 20% overlap df.select(text.token_chunk(col("text"), 100, 20)) ``` """ return Column._from_logical_expr( TextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=TextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.TOKEN, ) ) ) ``` ## trim ``` trim(column: ColumnOrName) -> Column ``` Remove whitespace from both sides of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from both the beginning and end of each string in the column. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to trim Returns: - **`Column`** ( `Column` ) – A column containing the trimmed strings Remove whitespace from both sides ``` # Remove whitespace from both sides of text df.select(text.trim(col("text"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def trim(column: ColumnOrName) -> Column: """Remove whitespace from both sides of strings in a column. This function removes all whitespace characters (spaces, tabs, newlines) from both the beginning and end of each string in the column. Args: column: The input string column or column name to trim Returns: Column: A column containing the trimmed strings Example: Remove whitespace from both sides ```python # Remove whitespace from both sides of text df.select(text.trim(col("text"))) ``` """ return Column._from_logical_expr( StripCharsExpr(Column._from_col_or_name(column)._logical_expr, None, "both") ) ``` ## upper ``` upper(column: ColumnOrName) -> Column ``` Convert all characters in a string column to uppercase. Parameters: - **`column`** (`ColumnOrName`) – The input string column to convert to uppercase Returns: - **`Column`** ( `Column` ) – A column containing the uppercase strings Convert text to uppercase ``` # Convert all text in the name column to uppercase df.select(text.upper(col("name"))) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def upper(column: ColumnOrName) -> Column: """Convert all characters in a string column to uppercase. Args: column: The input string column to convert to uppercase Returns: Column: A column containing the uppercase strings Example: Convert text to uppercase ```python # Convert all text in the name column to uppercase df.select(text.upper(col("name"))) ``` """ return Column._from_logical_expr( StringCasingExpr(Column._from_col_or_name(column)._logical_expr, "upper") ) ``` ## word_chunk ``` word_chunk(column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0) -> Column ``` Chunks a string column into chunks of a specified size (in words) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Parameters: - **`column`** (`ColumnOrName`) – The input string column or column name to chunk - **`chunk_size`** (`int`) – The size of each chunk in words - **`chunk_overlap_percentage`** (`int`, default: `0` ) – The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: - **`Column`** ( `Column` ) – A column containing the chunks as an array of strings Create word chunks ``` # Create chunks of 100 words with 20% overlap df.select(text.word_chunk(col("text"), 100, 20)) ``` Source code in `src/fenic/api/functions/text.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def word_chunk( column: ColumnOrName, chunk_size: int, chunk_overlap_percentage: int = 0 ) -> Column: """Chunks a string column into chunks of a specified size (in words) with an optional overlap. The chunking is done by applying a simple sliding window across the text to create chunks of equal size. This approach does not attempt to preserve the underlying structure of the text. Args: column: The input string column or column name to chunk chunk_size: The size of each chunk in words chunk_overlap_percentage: The overlap between chunks as a percentage of the chunk size (Default: 0) Returns: Column: A column containing the chunks as an array of strings Example: Create word chunks ```python # Create chunks of 100 words with 20% overlap df.select(text.word_chunk(col("text"), 100, 20)) ``` """ return Column._from_logical_expr( TextChunkExpr( Column._from_col_or_name(column)._logical_expr, chunking_configuration=TextChunkExprConfiguration( desired_chunk_size=chunk_size, chunk_overlap_percentage=chunk_overlap_percentage, chunk_length_function_name=ChunkLengthFunction.WORD, ) ) ) ``` --- # fenic.api.io Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/io/ IO module for reading and writing DataFrames to external storage. Classes: - **`DataFrameReader`** – Interface used to load a DataFrame from external storage systems. - **`DataFrameWriter`** – Interface used to write a DataFrame to external storage systems. ## DataFrameReader ``` DataFrameReader(session_state: BaseSessionState) ``` Interface used to load a DataFrame from external storage systems. Similar to PySpark's DataFrameReader. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Hugging Face Datasets (hf://) - Format: hf://{repo_type}/{repo_id}/{path_to_file} - Notes: - Supports glob patterns (*,* \*) - Supports dataset revisions and branch aliases (e.g., @refs/convert/parquet, @~parquet) - HF_TOKEN environment variable is required to read private datasets. - Examples: - hf://datasets/datasets-examples/doc-formats-csv-1/data.csv - hf://datasets/cais/mmlu/astronomy/\*.parquet - hf://datasets/datasets-examples/doc-formats-csv-1@~parquet/\**/*.parquet - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Creates a DataFrameReader. Parameters: - **`session_state`** (`BaseSessionState`) – The session state to use for reading Methods: - **`csv`** – Load a DataFrame from one or more CSV files. - **`docs`** – Load a DataFrame with the document contents of a list of paths (markdown or json). - **`parquet`** – Load a DataFrame from one or more Parquet files. - **`pdf_metadata`** – Load a DataFrame with metadata of PDF files in a list of paths. Source code in `src/fenic/api/io/reader.py` ``` def __init__(self, session_state: BaseSessionState): """Creates a DataFrameReader. Args: session_state: The session state to use for reading """ self._options: Dict[str, Any] = {} self._session_state = session_state ``` ### csv ``` csv(paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more CSV files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.csv"), or a list of paths. - **`schema`** (`Optional[Schema]`, default: `None` ) – (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. - **`merge_schemas`** (`bool`, default: `False` ) – Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If both `schema` and `merge_schemas=True` are provided. - `ValidationError` – If any path does not end with `.csv`. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single CSV file ``` df = session.read.csv("file.csv") ``` Read multiple CSV files with schema merging ``` df = session.read.csv("data/*.csv", merge_schemas=True) ``` Read CSV files with explicit schema `python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) )` Source code in `src/fenic/api/io/reader.py` ``` def csv( self, paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more CSV files. Args: paths: A single file path, a glob pattern (e.g., "data/*.csv"), or a list of paths. schema: (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. merge_schemas: Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: ValidationError: If both `schema` and `merge_schemas=True` are provided. ValidationError: If any path does not end with `.csv`. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single CSV file ```python df = session.read.csv("file.csv") ``` Example: Read multiple CSV files with schema merging ```python df = session.read.csv("data/*.csv", merge_schemas=True) ``` Example: Read CSV files with explicit schema ```python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) ) ``` """ if schema is not None and merge_schemas: raise ValidationError( "Cannot specify both 'schema' and 'merge_schemas=True' - these options conflict. " "Choose one approach: " "1) Use 'schema' to enforce a specific schema: csv(paths, schema=your_schema), " "2) Use 'merge_schemas=True' to automatically merge schemas: csv(paths, merge_schemas=True), " "3) Use neither to inherit schema from the first file: csv(paths)" ) if schema is not None: for col_field in schema.column_fields: if not isinstance( col_field.data_type, _PrimitiveType, ): raise ValidationError( f"CSV files only support primitive data types in schema definitions. " f"Column '{col_field.name}' has type {type(col_field.data_type).__name__}, but CSV schemas must use: " f"IntegerType, FloatType, DoubleType, BooleanType, or StringType. " f"Example: Schema([ColumnField(name='id', data_type=IntegerType), ColumnField(name='name', data_type=StringType)])" ) options = { "merge_schemas": merge_schemas, } if schema: options["schema"] = schema return self._read_file( paths, file_format="csv", file_extension=".csv", **options ) ``` ### docs ``` docs(paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal['markdown', 'json'], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with the document contents of a list of paths (markdown or json). Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`content_type`** (`Literal['markdown', 'json']`) – Content type of the files. One of "markdown" or "json". - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.md` or `.json` depending on the content_type. - `UnsupportedFileTypeError` – If the specified content_type is not "markdown" or "json" . Notes - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read all the markdown files in a folder and all its subfolders. ``` df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Read a folder of markdown files excluding some files. ``` df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` Source code in `src/fenic/api/io/reader.py` ``` def docs( self, paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal["markdown", "json"], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with the document contents of a list of paths (markdown or json). Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. content_type: Content type of the files. One of "markdown" or "json". exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.md` or `.json` depending on the content_type. UnsupportedFileTypeError: If the specified content_type is not "markdown" or "json" . Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read all the markdown files in a folder and all its subfolders. ```python df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Example: Read a folder of markdown files excluding some files. ```python df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) if content_type not in ["markdown", "json"]: raise UnsupportedFileTypeError(f"{content_type}, must be 'markdown' or 'json'") logical_node = DocSource.from_session_state( paths=path_str_list, content_type=content_type, exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ### parquet ``` parquet(paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more Parquet files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.parquet"), or a list of paths. - **`merge_schemas`** (`bool`, default: `False` ) – If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - Date and datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If any file does not have a `.parquet` extension. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single Parquet file ``` df = session.read.parquet("file.parquet") ``` Read multiple Parquet files ``` df = session.read.parquet("data/*.parquet") ``` Read Parquet files with schema merging ``` df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` Source code in `src/fenic/api/io/reader.py` ``` def parquet( self, paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more Parquet files. Args: paths: A single file path, a glob pattern (e.g., "data/*.parquet"), or a list of paths. merge_schemas: If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior: - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - Date and datetime columns are cast to strings during ingestion. Raises: ValidationError: If any file does not have a `.parquet` extension. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single Parquet file ```python df = session.read.parquet("file.parquet") ``` Example: Read multiple Parquet files ```python df = session.read.parquet("data/*.parquet") ``` Example: Read Parquet files with schema merging ```python df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` """ options = { "merge_schemas": merge_schemas, } return self._read_file( paths, file_format="parquet", file_extension=".parquet", **options ) ``` ### pdf_metadata ``` pdf_metadata(paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with metadata of PDF files in a list of paths. Note Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.pdf` extension. Notes - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read the metadata of all the PDF files in a folder and all its subfolders. ``` df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Read a metadata of PDFS in a folder, excluding some files. ``` df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` Source code in `src/fenic/api/io/reader.py` ``` def pdf_metadata( self, paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with metadata of PDF files in a list of paths. Note: Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.pdf` extension. Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read the metadata of all the PDF files in a folder and all its subfolders. ```python df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Example: Read a metadata of PDFS in a folder, excluding some files. ```python df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) logical_node = DocSource.from_session_state( paths=path_str_list, content_type="pdf", exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ## DataFrameWriter ``` DataFrameWriter(dataframe: DataFrame) ``` Interface used to write a DataFrame to external storage systems. Similar to PySpark's DataFrameWriter. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Initialize a DataFrameWriter. Parameters: - **`dataframe`** (`DataFrame`) – The DataFrame to write. Methods: - **`csv`** – Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. - **`parquet`** – Saves the content of the DataFrame as a single Parquet file. - **`save_as_table`** – Saves the content of the DataFrame as the specified table. - **`save_as_view`** – Saves the content of the DataFrame as a view. Source code in `src/fenic/api/io/writer.py` ``` def __init__(self, dataframe: DataFrame): """Initialize a DataFrameWriter. Args: dataframe: The DataFrame to write. """ self._dataframe = dataframe ``` ### csv ``` csv(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the CSV file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.csv("output.csv") # Overwrites if exists ``` Save with error mode ``` df.write.csv("output.csv", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.csv("output.csv", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def csv( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Args: file_path: Path to save the CSV file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.csv("output.csv") # Overwrites if exists ``` Example: Save with error mode ```python df.write.csv("output.csv", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.csv("output.csv", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".csv"): raise ValidationError( f"CSV writer requires a '.csv' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="csv", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### parquet ``` parquet(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single Parquet file. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the Parquet file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.parquet("output.parquet") # Overwrites if exists ``` Save with error mode ``` df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def parquet( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single Parquet file. Args: file_path: Path to save the Parquet file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.parquet("output.parquet") # Overwrites if exists ``` Example: Save with error mode ```python df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".parquet"): raise ValidationError( f"Parquet writer requires a '.parquet' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="parquet", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_table ``` save_as_table(table_name: str, mode: Literal['error', 'append', 'overwrite', 'ignore'] = 'error') -> QueryMetrics ``` Saves the content of the DataFrame as the specified table. Parameters: - **`table_name`** (`str`) – Name of the table to save to - **`mode`** (`Literal['error', 'append', 'overwrite', 'ignore']`, default: `'error'` ) – Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with error mode (default) ``` df.write.save_as_table("my_table") # Raises error if table exists ``` Save with append mode ``` df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Save with overwrite mode ``` df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` Source code in `src/fenic/api/io/writer.py` ``` def save_as_table( self, table_name: str, mode: Literal["error", "append", "overwrite", "ignore"] = "error", ) -> QueryMetrics: """Saves the content of the DataFrame as the specified table. Args: table_name: Name of the table to save to mode: Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: QueryMetrics: The query metrics Example: Save with error mode (default) ```python df.write.save_as_table("my_table") # Raises error if table exists ``` Example: Save with append mode ```python df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Example: Save with overwrite mode ```python df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` """ sink_plan = TableSink.from_session_state( child=self._dataframe._logical_plan, table_name=table_name, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_as_table( sink_plan, table_name=table_name, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_view ``` save_as_view(view_name: str, description: str | None = None) -> None ``` Saves the content of the DataFrame as a view. Parameters: - **`view_name`** (`str`) – Name of the view to save to - **`description`** (`str | None`, default: `None` ) – Optional human-readable view description to store in the catalog. Returns: - `None` – None. Source code in `src/fenic/api/io/writer.py` ``` def save_as_view( self, view_name: str, description: str | None = None, ) -> None: """Saves the content of the DataFrame as a view. Args: view_name: Name of the view to save to description: Optional human-readable view description to store in the catalog. Returns: None. """ self._dataframe._session_state.execution.save_as_view( logical_plan=self._dataframe._logical_plan, view_name=view_name, view_description=description ) ``` --- # fenic.api.io.reader Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/io/reader/ Reader interface for loading DataFrames from external storage systems. Classes: - **`DataFrameReader`** – Interface used to load a DataFrame from external storage systems. ## DataFrameReader ``` DataFrameReader(session_state: BaseSessionState) ``` Interface used to load a DataFrame from external storage systems. Similar to PySpark's DataFrameReader. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Hugging Face Datasets (hf://) - Format: hf://{repo_type}/{repo_id}/{path_to_file} - Notes: - Supports glob patterns (*,* \*) - Supports dataset revisions and branch aliases (e.g., @refs/convert/parquet, @~parquet) - HF_TOKEN environment variable is required to read private datasets. - Examples: - hf://datasets/datasets-examples/doc-formats-csv-1/data.csv - hf://datasets/cais/mmlu/astronomy/\*.parquet - hf://datasets/datasets-examples/doc-formats-csv-1@~parquet/\**/*.parquet - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Creates a DataFrameReader. Parameters: - **`session_state`** (`BaseSessionState`) – The session state to use for reading Methods: - **`csv`** – Load a DataFrame from one or more CSV files. - **`docs`** – Load a DataFrame with the document contents of a list of paths (markdown or json). - **`parquet`** – Load a DataFrame from one or more Parquet files. - **`pdf_metadata`** – Load a DataFrame with metadata of PDF files in a list of paths. Source code in `src/fenic/api/io/reader.py` ``` def __init__(self, session_state: BaseSessionState): """Creates a DataFrameReader. Args: session_state: The session state to use for reading """ self._options: Dict[str, Any] = {} self._session_state = session_state ``` ### csv ``` csv(paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more CSV files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.csv"), or a list of paths. - **`schema`** (`Optional[Schema]`, default: `None` ) – (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. - **`merge_schemas`** (`bool`, default: `False` ) – Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If both `schema` and `merge_schemas=True` are provided. - `ValidationError` – If any path does not end with `.csv`. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single CSV file ``` df = session.read.csv("file.csv") ``` Read multiple CSV files with schema merging ``` df = session.read.csv("data/*.csv", merge_schemas=True) ``` Read CSV files with explicit schema `python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) )` Source code in `src/fenic/api/io/reader.py` ``` def csv( self, paths: Union[str, Path, list[Union[str, Path]]], schema: Optional[Schema] = None, merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more CSV files. Args: paths: A single file path, a glob pattern (e.g., "data/*.csv"), or a list of paths. schema: (optional) A complete schema definition of column names and their types. Only primitive types are supported. - For e.g.: - Schema([ColumnField(name="id", data_type=IntegerType), ColumnField(name="name", data_type=StringType)]) - If provided, all files must match this schema exactlyβ€”all column names must be present, and values must be convertible to the specified types. Partial schemas are not allowed. merge_schemas: Whether to merge schemas across all files. - If True: Column names are unified across files. Missing columns are filled with nulls. Column types are inferred and widened as needed. - If False (default): Only accepts columns from the first file. Column types from the first file are inferred and applied across all files. If subsequent files do not have the same column name and order as the first file, an error is raised. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - The first row in each file is assumed to be a header row. - Delimiters (e.g., comma, tab) are automatically inferred. - You may specify either `schema` or `merge_schemas=True`, but not both. - Any date/datetime columns are cast to strings during ingestion. Raises: ValidationError: If both `schema` and `merge_schemas=True` are provided. ValidationError: If any path does not end with `.csv`. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single CSV file ```python df = session.read.csv("file.csv") ``` Example: Read multiple CSV files with schema merging ```python df = session.read.csv("data/*.csv", merge_schemas=True) ``` Example: Read CSV files with explicit schema ```python df = session.read.csv( ["a.csv", "b.csv"], schema=Schema([ ColumnField(name="id", data_type=IntegerType), ColumnField(name="value", data_type=FloatType) ]) ) ``` """ if schema is not None and merge_schemas: raise ValidationError( "Cannot specify both 'schema' and 'merge_schemas=True' - these options conflict. " "Choose one approach: " "1) Use 'schema' to enforce a specific schema: csv(paths, schema=your_schema), " "2) Use 'merge_schemas=True' to automatically merge schemas: csv(paths, merge_schemas=True), " "3) Use neither to inherit schema from the first file: csv(paths)" ) if schema is not None: for col_field in schema.column_fields: if not isinstance( col_field.data_type, _PrimitiveType, ): raise ValidationError( f"CSV files only support primitive data types in schema definitions. " f"Column '{col_field.name}' has type {type(col_field.data_type).__name__}, but CSV schemas must use: " f"IntegerType, FloatType, DoubleType, BooleanType, or StringType. " f"Example: Schema([ColumnField(name='id', data_type=IntegerType), ColumnField(name='name', data_type=StringType)])" ) options = { "merge_schemas": merge_schemas, } if schema: options["schema"] = schema return self._read_file( paths, file_format="csv", file_extension=".csv", **options ) ``` ### docs ``` docs(paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal['markdown', 'json'], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with the document contents of a list of paths (markdown or json). Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`content_type`** (`Literal['markdown', 'json']`) – Content type of the files. One of "markdown" or "json". - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.md` or `.json` depending on the content_type. - `UnsupportedFileTypeError` – If the specified content_type is not "markdown" or "json" . Notes - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read all the markdown files in a folder and all its subfolders. ``` df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Read a folder of markdown files excluding some files. ``` df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` Source code in `src/fenic/api/io/reader.py` ``` def docs( self, paths: Union[str, Path, list[Union[str, Path]]], content_type: Literal["markdown", "json"], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with the document contents of a list of paths (markdown or json). Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. content_type: Content type of the files. One of "markdown" or "json". exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with all the documents found in the paths. The content of each document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.md` or `.json` depending on the content_type. UnsupportedFileTypeError: If the specified content_type is not "markdown" or "json" . Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The dataframe has the following columns: - file_path: The path to the file. - error: The error message if the file failed to be loaded. - content: The content of the file casted to the content_type. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.md` will load all markdown files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read all the markdown files in a folder and all its subfolders. ```python df = session.read.docs("data/docs/**/*.md", content_type="markdown", recursive=True) ``` Example: Read a folder of markdown files excluding some files. ```python df = session.read.docs("data/docs/*.md", content_type="markdown", exclude=r"\.bak.md$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) if content_type not in ["markdown", "json"]: raise UnsupportedFileTypeError(f"{content_type}, must be 'markdown' or 'json'") logical_node = DocSource.from_session_state( paths=path_str_list, content_type=content_type, exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` ### parquet ``` parquet(paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False) -> DataFrame ``` Load a DataFrame from one or more Parquet files. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – A single file path, a glob pattern (e.g., "data/\*.parquet"), or a list of paths. - **`merge_schemas`** (`bool`, default: `False` ) – If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes - Date and datetime columns are cast to strings during ingestion. Raises: - `ValidationError` – If any file does not have a `.parquet` extension. - `PlanError` – If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Read a single Parquet file ``` df = session.read.parquet("file.parquet") ``` Read multiple Parquet files ``` df = session.read.parquet("data/*.parquet") ``` Read Parquet files with schema merging ``` df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` Source code in `src/fenic/api/io/reader.py` ``` def parquet( self, paths: Union[str, Path, list[Union[str, Path]]], merge_schemas: bool = False, ) -> DataFrame: """Load a DataFrame from one or more Parquet files. Args: paths: A single file path, a glob pattern (e.g., "data/*.parquet"), or a list of paths. merge_schemas: If True, infers and merges schemas across all files. Missing columns are filled with nulls, and differing types are widened to a common supertype. Behavior: - If `merge_schemas=False` (default), all files must match the schema of the first file exactly. Subsequent files must contain all columns from the first file with compatible data types. If any column is missing or has incompatible types, an error is raised. - If `merge_schemas=True`, column names are unified across all files, and data types are automatically widened to accommodate all values. - The "first file" is defined as: - The first file in lexicographic order (for glob patterns), or - The first file in the provided list (for lists of paths). Notes: - Date and datetime columns are cast to strings during ingestion. Raises: ValidationError: If any file does not have a `.parquet` extension. PlanError: If schemas cannot be merged or if there's a schema mismatch when merge_schemas=False. Example: Read a single Parquet file ```python df = session.read.parquet("file.parquet") ``` Example: Read multiple Parquet files ```python df = session.read.parquet("data/*.parquet") ``` Example: Read Parquet files with schema merging ```python df = session.read.parquet(["a.parquet", "b.parquet"], merge_schemas=True) ``` """ options = { "merge_schemas": merge_schemas, } return self._read_file( paths, file_format="parquet", file_extension=".parquet", **options ) ``` ### pdf_metadata ``` pdf_metadata(paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False) -> DataFrame ``` Load a DataFrame with metadata of PDF files in a list of paths. Note Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Parameters: - **`paths`** (`Union[str, Path, list[Union[str, Path]]]`) – Glob pattern (or list of glob patterns) to the folder(s) to load. - **`exclude`** (`Optional[str]`, default: `None` ) – A regex pattern to exclude files. If it is not provided no files will be excluded. - **`recursive`** (`bool`, default: `False` ) – Whether to recursively load files from the folder. Returns: - **`DataFrame`** ( `DataFrame` ) – A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: - `ValidationError` – If any file does not have a `.pdf` extension. Notes - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '\**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then \*\* behaves like a single '*' pattern. Read the metadata of all the PDF files in a folder and all its subfolders. ``` df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Read a metadata of PDFS in a folder, excluding some files. ``` df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` Source code in `src/fenic/api/io/reader.py` ``` def pdf_metadata( self, paths: Union[str, Path, list[Union[str, Path]]], exclude: Optional[str] = None, recursive: bool = False, ) -> DataFrame: r"""Load a DataFrame with metadata of PDF files in a list of paths. Note: Local execution requires the `pdf` extra: `pip install "fenic[pdf]"`. Args: paths: Glob pattern (or list of glob patterns) to the folder(s) to load. exclude: A regex pattern to exclude files. If it is not provided no files will be excluded. recursive: Whether to recursively load files from the folder. Returns: DataFrame: A dataframe with the metadata of all the PDF files found in the paths. the metadata from a single PDF document is a row in the dataframe. Raises: ValidationError: If any file does not have a `.pdf` extension. Notes: - Each row in the dataframe corresponds to a file in the list of paths. - The metadata columns are: - file_path: The path to the document. - error: The error message if the file failed to be loaded. - size: Size of the PDF file in bytes. - title: Title of the PDF document. - author: Author of the PDF document. - creation_date: Creation date of the PDF. - mod_date: Modification date of the PDF. - page_count: Number of pages in the PDF. - has_forms: Whether the PDF contains form fields, or fields that accept user input. - has_signature_fields: Whether the PDF contains signature fields. - image_count: Number of images in the PDF. - is_encrypted: Whether the PDF is encrypted. - Recursive loading is supported in conjunction with the '**' glob pattern, e.g. `data/**/*.pdf` will load all PDF files in the `data` folder and all subfolders when recursive is set to True. Without recursive = True, then ** behaves like a single '*' pattern. Example: Read the metadata of all the PDF files in a folder and all its subfolders. ```python df = session.read.pdf_metadata("data/docs/**/*.pdf", recursive=True) ``` Example: Read a metadata of PDFS in a folder, excluding some files. ```python df = session.read.pdf_metadata("data/docs/*.pdf", exclude=r"\.backup.pdf$") ``` """ path_str_list = validate_paths_and_return_list_of_strings(paths) logical_node = DocSource.from_session_state( paths=path_str_list, content_type="pdf", exclude=exclude, recursive=recursive, session_state=self._session_state, ) from fenic.api.dataframe import DataFrame return DataFrame._from_logical_plan(logical_node, self._session_state) ``` --- # fenic.api.io.writer Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/io/writer/ Writer interface for saving DataFrames to external storage systems. Classes: - **`DataFrameWriter`** – Interface used to write a DataFrame to external storage systems. ## DataFrameWriter ``` DataFrameWriter(dataframe: DataFrame) ``` Interface used to write a DataFrame to external storage systems. Similar to PySpark's DataFrameWriter. Supported External Storage Schemes: - Amazon S3 (s3://) - Format: s3://{bucket_name}/{path_to_file} ``` - Notes: - Uses boto3 to aquire AWS credentials. - Examples: - s3://my-bucket/data.csv - s3://my-bucket/data/*.parquet ``` - Local Files (file:// or implicit) - Format: file://{absolute_or_relative_path} - Notes: - Paths without a scheme (e.g., ./data.csv or /tmp/data.parquet) are treated as local files - Examples: - file:///home/user/data.csv - ./data/\*.parquet Initialize a DataFrameWriter. Parameters: - **`dataframe`** (`DataFrame`) – The DataFrame to write. Methods: - **`csv`** – Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. - **`parquet`** – Saves the content of the DataFrame as a single Parquet file. - **`save_as_table`** – Saves the content of the DataFrame as the specified table. - **`save_as_view`** – Saves the content of the DataFrame as a view. Source code in `src/fenic/api/io/writer.py` ``` def __init__(self, dataframe: DataFrame): """Initialize a DataFrameWriter. Args: dataframe: The DataFrame to write. """ self._dataframe = dataframe ``` ### csv ``` csv(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the CSV file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.csv("output.csv") # Overwrites if exists ``` Save with error mode ``` df.write.csv("output.csv", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.csv("output.csv", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def csv( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single CSV file with comma as the delimiter and headers in the first row. Args: file_path: Path to save the CSV file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.csv("output.csv") # Overwrites if exists ``` Example: Save with error mode ```python df.write.csv("output.csv", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.csv("output.csv", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".csv"): raise ValidationError( f"CSV writer requires a '.csv' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="csv", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### parquet ``` parquet(file_path: Union[str, Path], mode: Literal['error', 'overwrite', 'ignore'] = 'overwrite') -> QueryMetrics ``` Saves the content of the DataFrame as a single Parquet file. Parameters: - **`file_path`** (`Union[str, Path]`) – Path to save the Parquet file to - **`mode`** (`Literal['error', 'overwrite', 'ignore']`, default: `'overwrite'` ) – Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with overwrite mode (default) ``` df.write.parquet("output.parquet") # Overwrites if exists ``` Save with error mode ``` df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Save with ignore mode ``` df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` Source code in `src/fenic/api/io/writer.py` ``` def parquet( self, file_path: Union[str, Path], mode: Literal["error", "overwrite", "ignore"] = "overwrite", ) -> QueryMetrics: """Saves the content of the DataFrame as a single Parquet file. Args: file_path: Path to save the Parquet file to mode: Write mode. Default is "overwrite". - error: Raises an error if file exists - overwrite: Overwrites the file if it exists - ignore: Silently ignores operation if file exists Returns: QueryMetrics: The query metrics Example: Save with overwrite mode (default) ```python df.write.parquet("output.parquet") # Overwrites if exists ``` Example: Save with error mode ```python df.write.parquet("output.parquet", mode="error") # Raises error if exists ``` Example: Save with ignore mode ```python df.write.parquet("output.parquet", mode="ignore") # Skips if exists ``` """ file_path = str(file_path) if not file_path.endswith(".parquet"): raise ValidationError( f"Parquet writer requires a '.parquet' file extension. " f"Your path '{file_path}' is missing the extension." ) sink_plan = FileSink.from_session_state( child=self._dataframe._logical_plan, sink_type="parquet", path=file_path, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_to_file( sink_plan, file_path=file_path, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_table ``` save_as_table(table_name: str, mode: Literal['error', 'append', 'overwrite', 'ignore'] = 'error') -> QueryMetrics ``` Saves the content of the DataFrame as the specified table. Parameters: - **`table_name`** (`str`) – Name of the table to save to - **`mode`** (`Literal['error', 'append', 'overwrite', 'ignore']`, default: `'error'` ) – Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: - **`QueryMetrics`** ( `QueryMetrics` ) – The query metrics Save with error mode (default) ``` df.write.save_as_table("my_table") # Raises error if table exists ``` Save with append mode ``` df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Save with overwrite mode ``` df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` Source code in `src/fenic/api/io/writer.py` ``` def save_as_table( self, table_name: str, mode: Literal["error", "append", "overwrite", "ignore"] = "error", ) -> QueryMetrics: """Saves the content of the DataFrame as the specified table. Args: table_name: Name of the table to save to mode: Write mode. Default is "error". - error: Raises an error if table exists - append: Appends data to table if it exists - overwrite: Overwrites existing table - ignore: Silently ignores operation if table exists Returns: QueryMetrics: The query metrics Example: Save with error mode (default) ```python df.write.save_as_table("my_table") # Raises error if table exists ``` Example: Save with append mode ```python df.write.save_as_table("my_table", mode="append") # Adds to existing table ``` Example: Save with overwrite mode ```python df.write.save_as_table("my_table", mode="overwrite") # Replaces existing table ``` """ sink_plan = TableSink.from_session_state( child=self._dataframe._logical_plan, table_name=table_name, mode=mode, session_state=self._dataframe._session_state, ) metrics = self._dataframe._session_state.execution.save_as_table( sink_plan, table_name=table_name, mode=mode ) logger.info(metrics.get_summary()) return metrics ``` ### save_as_view ``` save_as_view(view_name: str, description: str | None = None) -> None ``` Saves the content of the DataFrame as a view. Parameters: - **`view_name`** (`str`) – Name of the view to save to - **`description`** (`str | None`, default: `None` ) – Optional human-readable view description to store in the catalog. Returns: - `None` – None. Source code in `src/fenic/api/io/writer.py` ``` def save_as_view( self, view_name: str, description: str | None = None, ) -> None: """Saves the content of the DataFrame as a view. Args: view_name: Name of the view to save to description: Optional human-readable view description to store in the catalog. Returns: None. """ self._dataframe._session_state.execution.save_as_view( logical_plan=self._dataframe._logical_plan, view_name=view_name, view_description=description ) ``` --- # fenic.api.lineage Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/lineage/ Query interface for tracing data lineage through a query plan. Classes: - **`Lineage`** – Query interface for tracing data lineage through a query plan. ## Lineage ``` Lineage(lineage: BaseLineage) ``` Query interface for tracing data lineage through a query plan. This class allows you to navigate through the query plan both forwards and backwards, tracing how specific rows are transformed through each operation. Example ``` # Create a lineage query starting from the root query = LineageQuery(lineage, session.execution) # Or start from a specific source query.start_from_source("my_table") # Trace rows backwards through a transformation result = query.backward(["uuid1", "uuid2"]) # Trace rows forward to see their outputs result = query.forward(["uuid3", "uuid4"]) ``` Initialize a Lineage instance. Parameters: - **`lineage`** (`BaseLineage`) – The underlying lineage implementation. Methods: - **`backwards`** – Trace rows backwards to see which input rows produced them. - **`forwards`** – Trace rows forward to see how they are transformed by the next operation. - **`get_result_df`** – Get the result of the query as a Polars DataFrame. - **`get_source_df`** – Get a query source by name as a Polars DataFrame. - **`get_source_names`** – Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. - **`show`** – Print the operator tree of the query. - **`skip_backwards`** – [Not Implemented] Trace rows backwards through multiple operations at once. - **`skip_forwards`** – [Not Implemented] Trace rows forward through multiple operations at once. - **`start_from_source`** – Set the current position to a specific source in the query plan. Source code in `src/fenic/api/lineage.py` ``` def __init__(self, lineage: BaseLineage): """Initialize a Lineage instance. Args: lineage: The underlying lineage implementation. """ self.lineage = lineage ``` ### backwards ``` backwards(ids: List[str], branch_side: Optional[BranchSide] = None) -> pl.DataFrame ``` Trace rows backwards to see which input rows produced them. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back - **`branch_side`** (`Optional[BranchSide]`, default: `None` ) – For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: - `DataFrame` – DataFrame containing the source rows that produced the specified outputs Raises: - `ValueError` – If invalid ids format or incorrect branch_side specification Example ``` # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def backwards( self, ids: List[str], branch_side: Optional[BranchSide] = None ) -> pl.DataFrame: """Trace rows backwards to see which input rows produced them. Args: ids: List of UUIDs identifying the rows to trace back branch_side: For operators with multiple inputs (like joins), specify which input to trace ("left" or "right"). Not needed for single-input operations. Returns: DataFrame containing the source rows that produced the specified outputs Raises: ValueError: If invalid ids format or incorrect branch_side specification Example: ```python # Simple backward trace source_rows = query.backward(["result_uuid1"]) # Trace back through a join left_rows = query.backward(["join_uuid1"], branch_side="left") right_rows = query.backward(["join_uuid1"], branch_side="right") ``` """ return self.lineage.backwards(ids, branch_side) ``` ### forwards ``` forwards(row_ids: List[str]) -> pl.DataFrame ``` Trace rows forward to see how they are transformed by the next operation. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the transformed rows in the next operation Raises: - `ValueError` – If at root node or if row_ids format is invalid Example ``` # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def forwards(self, row_ids: List[str]) -> pl.DataFrame: """Trace rows forward to see how they are transformed by the next operation. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the transformed rows in the next operation Raises: ValueError: If at root node or if row_ids format is invalid Example: ```python # Trace how specific customer rows are transformed transformed = query.forward(["customer_uuid1", "customer_uuid2"]) ``` """ return self.lineage.forwards(row_ids) ``` ### get_result_df ``` get_result_df() -> pl.DataFrame ``` Get the result of the query as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` def get_result_df(self) -> pl.DataFrame: """Get the result of the query as a Polars DataFrame.""" return self.lineage.get_result_df() ``` ### get_source_df ``` get_source_df(source_name: str) -> pl.DataFrame ``` Get a query source by name as a Polars DataFrame. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_df(self, source_name: str) -> pl.DataFrame: """Get a query source by name as a Polars DataFrame.""" return self.lineage.get_source_df(source_name) ``` ### get_source_names ``` get_source_names() -> List[str] ``` Get the names of all sources in the query plan. Used to determine where to start the lineage traversal. Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def get_source_names(self) -> List[str]: """Get the names of all sources in the query plan. Used to determine where to start the lineage traversal.""" return self.lineage.get_source_names() ``` ### show ``` show() -> None ``` Print the operator tree of the query. Source code in `src/fenic/api/lineage.py` ``` def show(self) -> None: """Print the operator tree of the query.""" print(self.lineage.stringify_graph()) ``` ### skip_backwards ``` skip_backwards(ids: List[str]) -> Dict[str, pl.DataFrame] ``` [Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`ids`** (`List[str]`) – List of UUIDs identifying the rows to trace back Returns: - `Dict[str, DataFrame]` – Dictionary mapping operation names to their source DataFrames Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_backwards(self, ids: List[str]) -> Dict[str, pl.DataFrame]: """[Not Implemented] Trace rows backwards through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: ids: List of UUIDs identifying the rows to trace back Returns: Dictionary mapping operation names to their source DataFrames Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip backwards not yet implemented") ``` ### skip_forwards ``` skip_forwards(row_ids: List[str]) -> pl.DataFrame ``` [Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Parameters: - **`row_ids`** (`List[str]`) – List of UUIDs identifying the rows to trace Returns: - `DataFrame` – DataFrame containing the final transformed rows Raises: - `NotImplementedError` – This method is not yet implemented Source code in `src/fenic/api/lineage.py` ``` def skip_forwards(self, row_ids: List[str]) -> pl.DataFrame: """[Not Implemented] Trace rows forward through multiple operations at once. This method will allow efficient tracing through multiple operations without intermediate results. Args: row_ids: List of UUIDs identifying the rows to trace Returns: DataFrame containing the final transformed rows Raises: NotImplementedError: This method is not yet implemented """ raise NotImplementedError("Skip forwards not yet implemented") ``` ### start_from_source ``` start_from_source(source_name: str) -> None ``` Set the current position to a specific source in the query plan. Parameters: - **`source_name`** (`str`) – Name of the source table to start from Example ``` query.start_from_source("customers") # Now you can trace forward from the customers table ``` Source code in `src/fenic/api/lineage.py` ``` @validate_call(config=ConfigDict(strict=True)) def start_from_source(self, source_name: str) -> None: """Set the current position to a specific source in the query plan. Args: source_name: Name of the source table to start from Example: ```python query.start_from_source("customers") # Now you can trace forward from the customers table ``` """ self.lineage.start_from_source(source_name) ``` --- # fenic.api.mcp Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/mcp/ MCP Tool Creation/Server Management API. Classes: - **`SystemToolConfig`** – Configuration for canonical system tools. Functions: - **`create_mcp_server`** – Create an MCP server from datasets and tools. - **`run_mcp_server_asgi`** – Run an MCP server as a Starlette ASGI app. - **`run_mcp_server_async`** – Run an MCP server asynchronously. - **`run_mcp_server_sync`** – Run an MCP server synchronously. ## SystemToolConfig ``` SystemToolConfig(table_names: list[str], tool_namespace: Optional[str] = None, max_result_rows: int = 100) ``` Configuration for canonical system tools. fenic can automatically generate a set of canonical tools for operating on one or more fenic tables. - Schema: list columns/types for any or all tables - Profile: column statistics (counts, basic numeric analysis [min, max, mean, etc.], contextual information for text columns [average_length, etc.]) - Read: read a selection of rows from a single table. These rows can be paged over, filtered and can use column projections. - Search Summary: literal or regex search across all text columns in all tables -- returns back dataframe names with result counts. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Search Content: literal or regex search across a single table, specifying one or more text columns to search across -- returns back rows corresponding to the query. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Analyze: Write raw SQL to perform complex analysis on one or more tables. Attributes: - **`table_names`** (`list[str]`) – List of the fenic table names the tools should be able to access. To allow access to all tables, pass `session.catalog.list_tables()` - **`tool_namespace`** (`Optional[str]`) – If provided, will prefix the names of the generated tools with this namespace value. For example, by default the generated tools will be named `read`, `profile`, etc. With multiple fenic MCP servers, these tool names will clash, which can be confusing. In order to disambiguate, the `tool_namespace` is prefixed to the tool name (in snake case), so a `tool_namespace` of `fenic` would create the tools `fenic_read`, `fenic_profile`, etc. - **`max_result_rows`** (`int`) – Maximum number of rows to be returned from Read/Analyze tools. Example: ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) df = session.create_dataframe({ "c1": [1, 2, 3], "c2": [4, 5, 6] }) df.write.save_as_table("table1", mode="overwrite") session.catalog.set_table_description("table1", "Table 1 Description") server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=["table1"], tool_namespace="Auto", max_result_rows=100 )) ``` Example: Allow generated tools to access all tables in the catalog. ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) # Assuming you already have one or more tables saved to the catalog, with descriptions. server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=session.catalog.list_tables() tool_namespace="Auto", max_result_rows=100 )) ``` ## create_mcp_server ``` create_mcp_server(session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8) -> FenicMCPServer ``` Create an MCP server from datasets and tools. Parameters: - **`session`** (`Session`) – Fenic session used to execute tools. - **`server_name`** (`str`) – Name of the MCP server. - **`user_defined_tools`** (`Optional[List[UserDefinedTool]]`, default: `None` ) – User defined tools to register with the MCP server. - **`system_tools`** (`Optional[SystemToolConfig]`, default: `None` ) – Configuration for automatically created system tools. - **`concurrency_limit`** (`int`, default: `8` ) – Maximum number of concurrent tool executions. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_mcp_server( session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8, ) -> FenicMCPServer: """Create an MCP server from datasets and tools. Args: session: Fenic session used to execute tools. server_name: Name of the MCP server. user_defined_tools: User defined tools to register with the MCP server. system_tools: Configuration for automatically created system tools. concurrency_limit: Maximum number of concurrent tool executions. """ generated_system_tools = [] user_defined_tools = user_defined_tools or [] if system_tools: generated_system_tools.extend( auto_generate_system_tools_from_tables( system_tools.table_names, session, tool_namespace=system_tools.tool_namespace, max_result_limit=system_tools.max_result_rows ) ) if not (user_defined_tools or system_tools): raise ConfigurationError("No tools provided. Either provide `user_defined_tools` or set `system_tools` to create system tools for catalog tables.") return FenicMCPServer(session._session_state, user_defined_tools, generated_system_tools, server_name, concurrency_limit) ``` ## run_mcp_server_asgi ``` run_mcp_server_asgi(server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`transport`** (`Literal['streamable-http', 'sse']`, default: `'streamable-http'` ) – Transport protocol to use (streamable-http, sse). - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional starlette-specific arguments to pass to FastMCP. Notes Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_asgi( server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Args: server: MCP server to run. stateless_http: If True, use stateless HTTP. transport: Transport protocol to use (streamable-http, sse). path: Path to listen on. kwargs: Additional starlette-specific arguments to pass to FastMCP. Notes: Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. """ return server.http_app(stateless_http=stateless_http, transport=transport, path=path, **kwargs) ``` ## run_mcp_server_async ``` run_mcp_server_async(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) async def run_mcp_server_async( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ await server.run_async(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## run_mcp_server_sync ``` run_mcp_server_sync(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_sync( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ server.run(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` --- # fenic.api.mcp.server Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/mcp/server/ Create MCP servers using Fenic DataFrames. This module exposes helpers to: - Build a Fenic-backed MCP server from datasets and tools - Run the server synchronously or asynchronously Functions: - **`create_mcp_server`** – Create an MCP server from datasets and tools. - **`run_mcp_server_asgi`** – Run an MCP server as a Starlette ASGI app. - **`run_mcp_server_async`** – Run an MCP server asynchronously. - **`run_mcp_server_sync`** – Run an MCP server synchronously. ## create_mcp_server ``` create_mcp_server(session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8) -> FenicMCPServer ``` Create an MCP server from datasets and tools. Parameters: - **`session`** (`Session`) – Fenic session used to execute tools. - **`server_name`** (`str`) – Name of the MCP server. - **`user_defined_tools`** (`Optional[List[UserDefinedTool]]`, default: `None` ) – User defined tools to register with the MCP server. - **`system_tools`** (`Optional[SystemToolConfig]`, default: `None` ) – Configuration for automatically created system tools. - **`concurrency_limit`** (`int`, default: `8` ) – Maximum number of concurrent tool executions. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def create_mcp_server( session: Session, server_name: str, *, user_defined_tools: Optional[List[UserDefinedTool]] = None, system_tools: Optional[SystemToolConfig] = None, concurrency_limit: int = 8, ) -> FenicMCPServer: """Create an MCP server from datasets and tools. Args: session: Fenic session used to execute tools. server_name: Name of the MCP server. user_defined_tools: User defined tools to register with the MCP server. system_tools: Configuration for automatically created system tools. concurrency_limit: Maximum number of concurrent tool executions. """ generated_system_tools = [] user_defined_tools = user_defined_tools or [] if system_tools: generated_system_tools.extend( auto_generate_system_tools_from_tables( system_tools.table_names, session, tool_namespace=system_tools.tool_namespace, max_result_limit=system_tools.max_result_rows ) ) if not (user_defined_tools or system_tools): raise ConfigurationError("No tools provided. Either provide `user_defined_tools` or set `system_tools` to create system tools for catalog tables.") return FenicMCPServer(session._session_state, user_defined_tools, generated_system_tools, server_name, concurrency_limit) ``` ## run_mcp_server_asgi ``` run_mcp_server_asgi(server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`transport`** (`Literal['streamable-http', 'sse']`, default: `'streamable-http'` ) – Transport protocol to use (streamable-http, sse). - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional starlette-specific arguments to pass to FastMCP. Notes Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_asgi( server: FenicMCPServer, *, stateless_http: bool = True, transport: Literal['streamable-http', 'sse'] = 'streamable-http', path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server as a Starlette ASGI app. Returns a Starlette ASGI app that can be integrated into any ASGI server. This is useful for running the MCP server in a production environment, or running the MCP server as part of a larger application. Args: server: MCP server to run. stateless_http: If True, use stateless HTTP. transport: Transport protocol to use (streamable-http, sse). path: Path to listen on. kwargs: Additional starlette-specific arguments to pass to FastMCP. Notes: Additional keyword arguments: - `middleware`: A list of Starlette `ASGIMiddleware` middleware to apply to the app. """ return server.http_app(stateless_http=stateless_http, transport=transport, path=path, **kwargs) ``` ## run_mcp_server_async ``` run_mcp_server_async(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) async def run_mcp_server_async( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server asynchronously. Use this when calling from asynchronous code. This does not create a new event loop. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ await server.run_async(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` ## run_mcp_server_sync ``` run_mcp_server_sync(server: FenicMCPServer, *, transport: MCPTransport = 'http', stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = '/mcp', **kwargs) ``` Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Parameters: - **`server`** (`FenicMCPServer`) – MCP server to run. - **`transport`** (`MCPTransport`, default: `'http'` ) – Transport protocol (http, stdio). - **`stateless_http`** (`bool`, default: `True` ) – If True, use stateless HTTP. - **`port`** (`Optional[int]`, default: `None` ) – Port to listen on. - **`host`** (`Optional[str]`, default: `None` ) – Host to listen on. - **`path`** (`Optional[str]`, default: `'/mcp'` ) – Path to listen on. - **`kwargs`** – Additional transport-specific arguments to pass to FastMCP. Source code in `src/fenic/api/mcp/server.py` ``` @validate_call(config=ConfigDict(strict=True, arbitrary_types_allowed=True)) def run_mcp_server_sync( server: FenicMCPServer, *, transport: MCPTransport = "http", stateless_http: bool = True, port: Optional[int] = None, host: Optional[str] = None, path: Optional[str] = "/mcp", **kwargs, ): """Run an MCP server synchronously. Use this when calling from synchronous code. This creates a new event loop and runs the server in it. Args: server: MCP server to run. transport: Transport protocol (http, stdio). stateless_http: If True, use stateless HTTP. port: Port to listen on. host: Host to listen on. path: Path to listen on. kwargs: Additional transport-specific arguments to pass to FastMCP. """ server.run(transport=transport, stateless_http=stateless_http, port=port, host=host, path=path, **kwargs) ``` --- # fenic.api.mcp.tools Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/mcp/tools/ API-layer generators for automatic MCP tools from DataFrames. These helpers generate System Tool Definitions for: - Schema: dataset column names and types - Profile: per-column statistics (counts, numeric summaries, simple string summaries) - Analyze: DuckDB SQL across one or more datasets. All generated tools return LogicalPlan objects. The MCP server wrapper handles execution and result formatting. Classes: - **`SystemToolConfig`** – Configuration for canonical system tools. ## SystemToolConfig ``` SystemToolConfig(table_names: list[str], tool_namespace: Optional[str] = None, max_result_rows: int = 100) ``` Configuration for canonical system tools. fenic can automatically generate a set of canonical tools for operating on one or more fenic tables. - Schema: list columns/types for any or all tables - Profile: column statistics (counts, basic numeric analysis [min, max, mean, etc.], contextual information for text columns [average_length, etc.]) - Read: read a selection of rows from a single table. These rows can be paged over, filtered and can use column projections. - Search Summary: literal or regex search across all text columns in all tables -- returns back dataframe names with result counts. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Search Content: literal or regex search across a single table, specifying one or more text columns to search across -- returns back rows corresponding to the query. Use `search_mode="literal"` for plain substring search or `search_mode="regex"` for regular expressions. - Analyze: Write raw SQL to perform complex analysis on one or more tables. Attributes: - **`table_names`** (`list[str]`) – List of the fenic table names the tools should be able to access. To allow access to all tables, pass `session.catalog.list_tables()` - **`tool_namespace`** (`Optional[str]`) – If provided, will prefix the names of the generated tools with this namespace value. For example, by default the generated tools will be named `read`, `profile`, etc. With multiple fenic MCP servers, these tool names will clash, which can be confusing. In order to disambiguate, the `tool_namespace` is prefixed to the tool name (in snake case), so a `tool_namespace` of `fenic` would create the tools `fenic_read`, `fenic_profile`, etc. - **`max_result_rows`** (`int`) – Maximum number of rows to be returned from Read/Analyze tools. Example: ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) df = session.create_dataframe({ "c1": [1, 2, 3], "c2": [4, 5, 6] }) df.write.save_as_table("table1", mode="overwrite") session.catalog.set_table_description("table1", "Table 1 Description") server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=["table1"], tool_namespace="Auto", max_result_rows=100 )) ``` Example: Allow generated tools to access all tables in the catalog. ``` from fenic import SystemToolConfig from fenic.api.mcp.tools import SystemToolConfig from fenic.api.mcp.server import create_mcp_server from fenic.api.session.session import Session session = Session.get_or_create(...) # Assuming you already have one or more tables saved to the catalog, with descriptions. server = create_mcp_server(session, "Test Server", system_tools=SystemToolConfig( table_names=session.catalog.list_tables() tool_namespace="Auto", max_result_rows=100 )) ``` --- # fenic.api.session Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/session/ Session module for managing query execution context and state. Classes: - **`AdaptiveTokenEstimationConfig`** – Tunes adaptive output-token reservation for rate limiting. - **`AnthropicLanguageModel`** – Configuration for Anthropic language models. - **`CloudConfig`** – Configuration for cloud-based execution. - **`CloudExecutorSize`** – Enum defining available cloud executor sizes. - **`CohereEmbeddingModel`** – Configuration for Cohere embedding models. - **`GoogleDeveloperEmbeddingModel`** – Configuration for Google Developer embedding models. - **`GoogleDeveloperLanguageModel`** – Configuration for Gemini models accessible through Google Developer AI Studio. - **`GoogleVertexEmbeddingModel`** – Configuration for Google Vertex AI embedding models. - **`GoogleVertexLanguageModel`** – Configuration for Google Vertex AI models. - **`LLMResponseCacheConfig`** – Configuration for LLM response caching. - **`OpenAIEmbeddingModel`** – Configuration for OpenAI embedding models. - **`OpenAILanguageModel`** – Configuration for OpenAI language models. - **`OpenRouterLanguageModel`** – Configuration for OpenRouter language models. - **`SemanticConfig`** – Configuration for semantic language and embedding models. - **`Session`** – The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. - **`SessionConfig`** – Configuration for a user session. ## AdaptiveTokenEstimationConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.AdaptiveTokenEstimationConfig[AdaptiveTokenEstimationConfig] click fenic.api.session.AdaptiveTokenEstimationConfig href "" "fenic.api.session.AdaptiveTokenEstimationConfig" ``` Tunes adaptive output-token reservation for rate limiting. Output-token reservations are learned from observed usage and clamped to the request's max_completion_tokens ceiling, then corrected after each response (settlement). Enabled by default. Setting `enabled=False` disables adaptive *estimation* β€” reservations fall back to the static worst-case ceiling instead of the learned distribution. Settlement (reconciling the token bucket to actual usage after each response) is **always on** regardless of this flag. It corrects the bucket in both directions β€” refunding the over-reservation (the common case) and debiting further when a request exceeds its reservation β€” and neither direction increases 429 risk: a refund only returns capacity the provider never charged, and a debit only makes the limiter more conservative. ## AnthropicLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.AnthropicLanguageModel[AnthropicLanguageModel] click fenic.api.session.AnthropicLanguageModel href "" "fenic.api.session.AnthropicLanguageModel" ``` Configuration for Anthropic language models. This class defines the configuration settings for Anthropic language models, including model selection and separate rate limiting parameters for input and output tokens. Attributes: - **`model_name`** (`AnthropicLanguageModelName`) – The name of the Anthropic model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`input_tpm`** (`int`) – Input tokens per minute limit; must be greater than 0. - **`output_tpm`** (`int`) – Output tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring an Anthropic model with separate input/output rate limits: ``` config = AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100 ) ``` Configuring an Anthropic model with profiles: ``` config = SessionConfig( semantic=SemanticConfig( language_models={ "claude": AnthropicLanguageModel( model_name="claude-sonnet-4-6", rpm=100, input_tpm=100, output_tpm=100, profiles={ "thinking_disabled": AnthropicLanguageModel.Profile(), "fast": AnthropicLanguageModel.Profile(thinking_token_budget=1024), "thorough": AnthropicLanguageModel.Profile(thinking_token_budget=4096) }, default_profile="fast" ) }, default_language_model="claude" ) # Using the default "fast" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias="claude") # Using the "thorough" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="claude", profile="thorough")) ``` Classes: - **`Profile`** – Anthropic-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.AnthropicLanguageModel.Profile[Profile] click fenic.api.session.AnthropicLanguageModel.Profile href "" "fenic.api.session.AnthropicLanguageModel.Profile" ``` Anthropic-specific profile configurations. This class defines profile configurations for Anthropic models, allowing different thinking and effort settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – Provide a default thinking budget in tokens. If not provided, thinking will be disabled for the profile. The minimum token budget supported by Anthropic is 1024 tokens. For Claude models that use adaptive thinking, use `effort` instead. - **`effort`** (`Optional[AnthropicReasoningEffortType]`) – Provider-native Anthropic effort level. Supported values vary by model: low, medium, high, xhigh, and max. On adaptive-thinking models the thinking budget shares the request's output token window rather than being reserved on top of it, so very high effort levels can consume part of the visible completion budget. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note If `thinking_token_budget` or adaptive `effort` enables thinking, `temperature` cannot be customized -- any changes to `temperature` will be ignored. Effort-only profiles on non-adaptive models configure Anthropic `output_config` without enabling thinking, so custom `temperature` remains available when the model supports it. Example Configuring a profile with a thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=2048) ``` Configuring a profile with a large thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=8192) ``` Configuring a profile with effort: ``` profile = AnthropicLanguageModel.Profile(effort="xhigh") ``` ## CloudConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.CloudConfig[CloudConfig] click fenic.api.session.CloudConfig href "" "fenic.api.session.CloudConfig" ``` Configuration for cloud-based execution. This class defines settings for running operations in a cloud environment, allowing for scalable and distributed processing of language model operations. Attributes: - **`size`** (`Optional[CloudExecutorSize]`) – Size of the cloud executor instance. If None, the default size will be used. Example Configuring cloud execution with a specific size: ``` config = CloudConfig(size=CloudExecutorSize.MEDIUM) ``` Using default cloud configuration: ``` config = CloudConfig() ``` ## CloudExecutorSize Bases: `str`, `Enum` ``` flowchart TD fenic.api.session.CloudExecutorSize[CloudExecutorSize] click fenic.api.session.CloudExecutorSize href "" "fenic.api.session.CloudExecutorSize" ``` Enum defining available cloud executor sizes. This enum represents the different size options available for cloud-based execution environments. Attributes: - **`SMALL`** – Small instance size. - **`MEDIUM`** – Medium instance size. - **`LARGE`** – Large instance size. - **`XLARGE`** – Extra large instance size. ## CohereEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.CohereEmbeddingModel[CohereEmbeddingModel] click fenic.api.session.CohereEmbeddingModel href "" "fenic.api.session.CohereEmbeddingModel" ``` Configuration for Cohere embedding models. This class defines the configuration settings for Cohere embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`CohereEmbeddingModelName`) – The name of the Cohere model to use. - **`rpm`** (`int`) – Requests per minute limit for the model. - **`tpm`** (`int`) – Tokens per minute limit for the model. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional dictionary of profile configurations. - **`default_profile`** (`Optional[str]`) – Default profile name to use if none specified. Example Configuring a Cohere embedding model with profiles: ``` cohere_config = CohereEmbeddingModel( model_name="embed-v4.0", rpm=100, tpm=50_000, profiles={ "high_dim": CohereEmbeddingModel.Profile( embedding_dimensionality=1536, embedding_task_type="search_document" ), "classification": CohereEmbeddingModel.Profile( embedding_dimensionality=1024, embedding_task_type="classification" ), }, default_profile="high_dim", ) ``` Classes: - **`Profile`** – Profile configurations for Cohere embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.CohereEmbeddingModel.Profile[Profile] click fenic.api.session.CohereEmbeddingModel.Profile href "" "fenic.api.session.CohereEmbeddingModel.Profile" ``` Profile configurations for Cohere embedding models. This class defines profile configurations for Cohere embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`input_type`** (`CohereEmbeddingTaskType`) – The type of input text (search_query, search_document, classification, clustering) Example Configuring a profile with custom dimensionality: ``` profile = CohereEmbeddingModel.Profile(output_dimensionality=1536) ``` Configuring a profile with default settings: ``` profile = CohereEmbeddingModel.Profile() ``` ## GoogleDeveloperEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleDeveloperEmbeddingModel[GoogleDeveloperEmbeddingModel] click fenic.api.session.GoogleDeveloperEmbeddingModel href "" "fenic.api.session.GoogleDeveloperEmbeddingModel" ``` Configuration for Google Developer embedding models. This class defines the configuration settings for Google embedding models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperEmbeddingModelName`) – The name of the Google Developer embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer embedding model with rate limits: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Developer embedding model with profiles: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleDeveloperEmbeddingModelConfig.Profile(), "high_dim": GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleDeveloperEmbeddingModel.Profile[Profile] click fenic.api.session.GoogleDeveloperEmbeddingModel.Profile href "" "fenic.api.session.GoogleDeveloperEmbeddingModel.Profile" ``` Profile configurations for Google Developer embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile() ``` ## GoogleDeveloperLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleDeveloperLanguageModel[GoogleDeveloperLanguageModel] click fenic.api.session.GoogleDeveloperLanguageModel href "" "fenic.api.session.GoogleDeveloperLanguageModel" ``` Configuration for Gemini models accessible through Google Developer AI Studio. This class defines the configuration settings for Google Gemini models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperLanguageModelName`) – The name of the Google Developer model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer model with rate limits: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Developer model with profiles: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleDeveloperLanguageModel.Profile(), "fast": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=1024 ), "thorough": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleDeveloperLanguageModel.Profile[Profile] click fenic.api.session.GoogleDeveloperLanguageModel.Profile href "" "fenic.api.session.GoogleDeveloperLanguageModel.Profile" ``` Profile configurations for Google Developer models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_level="high") ``` ## GoogleVertexEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleVertexEmbeddingModel[GoogleVertexEmbeddingModel] click fenic.api.session.GoogleVertexEmbeddingModel href "" "fenic.api.session.GoogleVertexEmbeddingModel" ``` Configuration for Google Vertex AI embedding models. This class defines the configuration settings for Google embedding models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexEmbeddingModelName`) – The name of the Google Vertex embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex embedding model with rate limits: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Vertex embedding model with profiles: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleVertexEmbeddingModel.Profile(), "high_dim": GoogleVertexEmbeddingModel.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleVertexEmbeddingModel.Profile[Profile] click fenic.api.session.GoogleVertexEmbeddingModel.Profile href "" "fenic.api.session.GoogleVertexEmbeddingModel.Profile" ``` Profile configurations for Google Vertex embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleVertexEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleVertexEmbeddingModelConfig.Profile() ``` ## GoogleVertexLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleVertexLanguageModel[GoogleVertexLanguageModel] click fenic.api.session.GoogleVertexLanguageModel href "" "fenic.api.session.GoogleVertexLanguageModel" ``` Configuration for Google Vertex AI models. This class defines the configuration settings for Google Gemini models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexLanguageModelName`) – The name of the Google Vertex model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex model with rate limits: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Vertex model with profiles: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleVertexLanguageModel.Profile(), "fast": GoogleVertexLanguageModel.Profile(thinking_token_budget=1024), "thorough": GoogleVertexLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.GoogleVertexLanguageModel.Profile[Profile] click fenic.api.session.GoogleVertexLanguageModel.Profile href "" "fenic.api.session.GoogleVertexLanguageModel.Profile" ``` Profile configurations for Google Vertex models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same underlying model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleVertexLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleVertexLanguageModel.Profile(thinking_level="high") ``` ## LLMResponseCacheConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.LLMResponseCacheConfig[LLMResponseCacheConfig] click fenic.api.session.LLMResponseCacheConfig href "" "fenic.api.session.LLMResponseCacheConfig" ``` Configuration for LLM response caching. LLM response caching stores the results of language model API calls to reduce costs and improve performance for repeated queries. This is distinct from DataFrame caching (the `.cache()` operator). Attributes: - **`enabled`** – Whether caching is enabled (default: True). - **`backend`** (`CacheBackend`) – Cache backend to use (default: LOCAL). - **`ttl`** (`str`) – Time-to-live duration string (default: "1h"). Format: where unit is s/m/h/d. Examples: "30s", "15m", "2h", "7d". Maximum: 30 days, Minimum: 1 second. - **`max_size_mb`** (`int`) – Maximum cache size in MB before LRU eviction (default: 128MB). - **`namespace`** (`str`) – Cache namespace for isolation (default: "default"). Example Basic configuration within SemanticConfig: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt": OpenAILanguageModel(model_name="gpt-4o-mini", rpm=100, tpm=1000) }, llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="1h", max_size_mb=1000, ) ) ) ``` Custom TTL and larger cache: ``` llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="7d", # 7 days max_size_mb=5000, ) ``` Disabled caching: ``` llm_response_cache=LLMResponseCacheConfig(enabled=False) ``` Methods: - **`ttl_seconds`** – Convert TTL string to seconds. - **`validate_ttl`** – Validate TTL duration string format. ### ttl_seconds ``` ttl_seconds() -> int ``` Convert TTL string to seconds. Returns: - `int` – TTL duration in seconds. Raises: - `ValueError` – If TTL format is invalid. Source code in `src/fenic/api/session/config.py` ``` def ttl_seconds(self) -> int: """Convert TTL string to seconds. Returns: TTL duration in seconds. Raises: ValueError: If TTL format is invalid. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, self.ttl.lower()) if not match: raise ValueError(f"Invalid TTL format: '{self.ttl}'") value, unit = match.groups() value = int(value) multipliers = {"s": 1, "m": 60, "h": 3600, "d": 86400} return value * multipliers[unit] ``` ### validate_ttl ``` validate_ttl(v: str) -> str ``` Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Parameters: - **`v`** (`str`) – TTL duration string to validate. Returns: - `str` – The validated TTL string. Raises: - `ValueError` – If format is invalid or value is out of range. Source code in `src/fenic/api/session/config.py` ``` @field_validator("ttl") @classmethod def validate_ttl(cls, v: str) -> str: """Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Args: v: TTL duration string to validate. Returns: The validated TTL string. Raises: ValueError: If format is invalid or value is out of range. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, v.lower()) if not match: raise ValueError( f"Invalid TTL format: '{v}'. " "Expected: where unit is s/m/h/d. " "Examples: '30m', '2h', '1d'" ) value, unit = match.groups() value = int(value) # Validate ranges if unit == "s" and value < 1: raise ValueError("TTL must be at least 1 second") if unit == "h" and value > 720: # 30 days raise ValueError("TTL cannot exceed 720 hours (30 days)") if unit == "d" and value > 30: raise ValueError("TTL cannot exceed 30 days") return v ``` ## OpenAIEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenAIEmbeddingModel[OpenAIEmbeddingModel] click fenic.api.session.OpenAIEmbeddingModel href "" "fenic.api.session.OpenAIEmbeddingModel" ``` Configuration for OpenAI embedding models. This class defines the configuration settings for OpenAI embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAIEmbeddingModelName`) – The name of the OpenAI embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. Example Configuring an OpenAI embedding model with rate limits: ``` config = OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) ``` ## OpenAILanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenAILanguageModel[OpenAILanguageModel] click fenic.api.session.OpenAILanguageModel href "" "fenic.api.session.OpenAILanguageModel" ``` Configuration for OpenAI language models. This class defines the configuration settings for OpenAI language models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAILanguageModelName`) – The name of the OpenAI model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Note When using an o-series or gpt5 reasoning model without specifying a reasoning effort in a Profile, the `reasoning_effort` will default to `low` (for o-series models) or `minimal` (for gpt5 models). Example Configuring an OpenAI language model with rate limits: ``` config = OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) ``` Configuring an OpenAI model with profiles: ``` config = OpenAILanguageModel( model_name="o4-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile(reasoning_effort="high"), }, default_profile="fast", ) ``` Using a profile in a semantic operation: ``` config = SemanticConfig( language_models={ "o4": OpenAILanguageModel( model_name="o4-mini", rpm=1_000, tpm=1_000_000, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ) }, default_language_model="o4", ) # Will use the default "fast" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias="o4", ) # Will use the "thorough" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="o4", profile="thorough"), ) ``` Classes: - **`Profile`** – OpenAI-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenAILanguageModel.Profile[Profile] click fenic.api.session.OpenAILanguageModel.Profile href "" "fenic.api.session.OpenAILanguageModel.Profile" ``` OpenAI-specific profile configurations. This class defines profile configurations for OpenAI models, allowing a user to reference the same underlying model in semantic operations with different settings. Attributes: - **`reasoning_effort`** (`Optional[ReasoningEffort]`) – Provide a reasoning effort. Only for gpt5 and o-series models. Valid values: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'. - For gpt-5.6 models: supports 'xhigh' and 'max' - For gpt-5.5 models: defaults to 'medium', supports 'xhigh' - For gpt-5.4 models: defaults to 'none' (disabled reasoning), supports 'xhigh' - For gpt-5.1 and gpt-5.2 models: defaults to 'none' (disabled reasoning), does NOT support 'minimal' or 'xhigh' - For gpt-5 models: defaults to 'minimal', does NOT support 'none' - For o-series models: defaults to 'low', does NOT support 'none' or 'minimal' - **`verbosity`** (`Optional[Verbosity]`) – Provide a verbosity level. Only for gpt5/gpt5.1 models. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note When using an o-series or gpt5 reasoning model with reasoning enabled, the `temperature` cannot be customized. For gpt-5.1 models with reasoning_effort='none', temperature CAN be customized. Example Configuring a profile with medium reasoning effort: ``` profile = OpenAILanguageModel.Profile(reasoning_effort="medium") ``` ## OpenRouterLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenRouterLanguageModel[OpenRouterLanguageModel] click fenic.api.session.OpenRouterLanguageModel href "" "fenic.api.session.OpenRouterLanguageModel" ``` Configuration for OpenRouter language models. This class defines the configuration settings for OpenRouter language models, including model selection and rate limiting parameters. When fetching available models from OpenRouter, results will be filtered to only include models from providers that are not in the user’s ignored providers list and are either in the user’s allowed providers list (if configured) or from any provider (if no allowed providers are specified). Attributes: - **`model_name`** (`str`) – `{family}/{model}` identifier (e.g., `anthropic/claude-3-5-sonnet`). - **`profiles`** (`Optional[dict[str, Profile]]`) – Mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The key in `profiles` to select by default. - **`structured_output_strategy`** (`Optional[StructuredOutputStrategy]`) – The strategy to use for structured output if a model supports both tool calling and structured outputs. - `prefer_tools`: prefer using tools over response format. - `prefer_response_format`: prefer using response format over tools. Requirements - Set `OPENROUTER_API_KEY` in your environment. Example: ``` OpenRouterLanguageModel( model_name="openai/gpt-oss-20b", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="price" # Routes to the cheapest available provider ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="anthropic/claude-sonnet-4", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( only=[ "Anthropic" ] # ensures the request will only be routed to Anthropic and not AWS Bedrock or Google Vertex ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="qwen/qwen3-next-80b-a3b-instruct", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="throughput", # routes to the provider with the highest overall throughput data_collection="deny" # eliminates providers that retain prompt data (would only route to DeepInfra/AtlasCloud, in this example) # Eliminate providers that offer an fp8 quantized version of the model, only allowing bf16. # Note that many providers have an `unknown` quantization, so you may be excluding more providers than you expect. quantizations=["bf16"] ) ) } ) ``` Classes: - **`Profile`** – Profile configurations for OpenRouter language models. - **`Provider`** – Provider routing configuration for OpenRouter language models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenRouterLanguageModel.Profile[Profile] click fenic.api.session.OpenRouterLanguageModel.Profile href "" "fenic.api.session.OpenRouterLanguageModel.Profile" ``` Profile configurations for OpenRouter language models. Attributes: - **`models`** (`Optional[list[str]]`) – A list of fallback models to use if the primary model is unavailable. ([OpenRouter Documentation](https://openrouter.ai/docs/features/model-routing#the-models-parameter)). - **`provider`** (`Optional[Provider]`) – Provider routing preferences (include/exclude specific providers, set provider ranking method preference) ([OpenRouter Documentation](https://openrouter.ai/docs/features/provider-routing)). - **`reasoning_effort`** (`Optional[OpenRouterReasoningEffort]`) – OpenRouter reasoning effort configuration (none, minimal, low, medium, high, xhigh, max). If the model does support reasoning, but not `reasoning_effort`, a `reasoning_max_tokens` will be calculated that is roughly equivalent as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#reasoning-effort-level)) - **`reasoning_max_tokens`** (`Optional[int]`) – Supported by Anthropic, Gemini, etc., sets a token budget for reasoning If the model does support reasoning, but not `reasoning_max_tokens`, a `reasoning_effort_ will be automatically calculated based on`reasoning_max_tokens` as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#max-tokens-for-reasoning)) - **`parsing_engine`** (`Optional[ParsingEngine]`) – The parsing engine to use for processing PDF files. By default, the model's native parsing engine will be used. If the model doesn't support PDF processing and the parsing engine is not provided, an error will be raised. Note: 'mistral-ocr' incurs additional costs. ([OpenRouter Documentation](https://openrouter.ai/docs/features/multimodal/pdfs)) ### Provider Bases: `BaseModel` ``` flowchart TD fenic.api.session.OpenRouterLanguageModel.Provider[Provider] click fenic.api.session.OpenRouterLanguageModel.Provider href "" "fenic.api.session.OpenRouterLanguageModel.Provider" ``` Provider routing configuration for OpenRouter language models. [Provider Routing Documentation](https://openrouter.ai/docs/features/provider-routing) Attributes: - **`order`** (`Optional[list[str]]`) – List of providers to try in order (e.g. ['Anthropic', 'Amazon Bedrock']). - **`sort`** (`Optional[ProviderSort]`) – Provider routing preference (e.g. 'price', 'throughput', 'latency'). "price" will route to the cheapest available provider first, progressing through the list of providers in order of price. "throughput" will route to the provider with the highest overall recent throughput, progressing through the list of providers in order of throughput. "latency" will route to the provider with the lowest overall recent latency, progressing through the list of providers in order of latency. - **`quantizations`** (`Optional[list[ModelQuantization]]`) – Allowed quantizations. Note: many providers report `unknown`. - **`data_collection`** (`Optional[DataCollection]`) – Data collection preference. `allow`: allows the use of providers which store prompt data non-transiently and may train on it. `deny`: use only providers which do not collect/store prompt data. - **`only`** (`Optional[list[str]]`) – Only include these providers when performing provider routing. - **`exclude`** (`Optional[list[str]]`) – Exclude these providers when performing provider routing. - **`max_prompt_price`** (`Optional[float]`) – Maximum prompt price ($USD per 1M tokens). - **`max_completion_price`** (`Optional[float]`) – Maximum completion price ($USD per 1M tokens). ## SemanticConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.SemanticConfig[SemanticConfig] click fenic.api.session.SemanticConfig href "" "fenic.api.session.SemanticConfig" ``` Configuration for semantic language and embedding models. This class defines the configuration for both language models and optional embedding models used in semantic operations. It ensures that all configured models are valid and supported by their respective providers. Attributes: - **`language_models`** (`Optional[dict[str, LanguageModel]]`) – Mapping of model aliases to language model configurations. - **`default_language_model`** (`Optional[str]`) – The alias of the default language model to use for semantic operations. Not required if only one language model is configured. - **`embedding_models`** (`Optional[dict[str, EmbeddingModel]]`) – Optional mapping of model aliases to embedding model configurations. - **`default_embedding_model`** (`Optional[str]`) – The alias of the default embedding model to use for semantic operations. Note The embedding model is optional and only required for operations that need semantic search or embedding capabilities. Example Configuring semantic models with a single language model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) } ) ``` Configuring semantic models with multiple language models and an embedding model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), "gemini": GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ) ``` Configuring models with profiles: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4o-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, profiles={ "fast": AnthropicLanguageModel.Profile(effort="low"), "thorough": AnthropicLanguageModel.Profile(effort="high"), }, default_profile="fast", ), }, default_language_model="gpt4", ) ``` Methods: - **`model_post_init`** – Post initialization hook to set defaults. - **`validate_models`** – Validates that the selected models are supported by the system. ### model_post_init ``` model_post_init(__context) -> None ``` Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. Source code in `src/fenic/api/session/config.py` ``` def model_post_init(self, __context) -> None: """Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. """ if self.language_models: # Set default language model if not set and only one model exists if self.default_language_model is None and len(self.language_models) == 1: self.default_language_model = list(self.language_models.keys())[0] # Auto-create profiles for Google models that support thinking_level for model_config in self.language_models.values(): if isinstance(model_config, (GoogleDeveloperLanguageModel, GoogleVertexLanguageModel)): model_provider = _get_model_provider_for_model_config(model_config) model_params = model_catalog.get_completion_model_parameters( model_provider, model_config.model_name ) if model_params and model_params.supported_thinking_levels and model_config.profiles is None: # Auto-create profiles for each supported thinking level if isinstance(model_config, GoogleDeveloperLanguageModel): model_config.profiles = { level: GoogleDeveloperLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } else: model_config.profiles = { level: GoogleVertexLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } model_config.default_profile = "low" # Set default profile for each model if not set and only one profile exists for model_config in self.language_models.values(): if model_config.profiles is not None: profile_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(profile_names) == 1: model_config.default_profile = profile_names[0] # Set default embedding model if not set and only one model exists if self.embedding_models: if self.default_embedding_model is None and len(self.embedding_models) == 1: self.default_embedding_model = list(self.embedding_models.keys())[0] # Set default profile for each model if not set and only one preset exists for model_config in self.embedding_models.values(): if ( hasattr(model_config, "profiles") and model_config.profiles is not None ): preset_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(preset_names) == 1: model_config.default_profile = preset_names[0] ``` ### validate_models ``` validate_models() -> SemanticConfig ``` Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: - `SemanticConfig` – The validated SemanticConfig instance. Raises: - `ConfigurationError` – If any of the models are not supported. Source code in `src/fenic/api/session/config.py` ``` @model_validator(mode="after") def validate_models(self) -> SemanticConfig: """Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: The validated SemanticConfig instance. Raises: ConfigurationError: If any of the models are not supported. """ # Skip validation if no models configured (embedding-only or empty config) if not self.language_models and not self.embedding_models: return self # Validate language models if provided if self.language_models: available_language_model_aliases = list(self.language_models.keys()) if self.default_language_model is None and len(self.language_models) > 1: raise ConfigurationError( f"default_language_model is not set, and multiple language models are configured. Please specify one of: {available_language_model_aliases} as a default_language_model." ) if ( self.default_language_model is not None and self.default_language_model not in self.language_models ): raise ConfigurationError( f"default_language_model {self.default_language_model} is not in configured map of language models. Available models: {available_language_model_aliases} ." ) for model_alias, language_model in self.language_models.items(): language_model_name = language_model.model_name language_model_provider = _get_model_provider_for_model_config( language_model ) completion_model_params = model_catalog.get_completion_model_parameters( language_model_provider, language_model_name ) if completion_model_params is None: raise ConfigurationError( model_catalog.generate_unsupported_completion_model_error_message( language_model_provider, language_model_name ) ) if language_model.profiles is not None: if not completion_model_params.supports_profiles: raise ConfigurationError( f"Model '{model_alias}' does not support parameter profiles. Please remove the Profile configuration." ) profile_names = list(language_model.profiles.keys()) if ( language_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( language_model.default_profile is not None and language_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {language_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in language_model.profiles.items(): _validate_language_profile( language_model, model_alias, completion_model_params, profile, profile_alias, ) if self.embedding_models is not None: available_embedding_model_aliases = list(self.embedding_models.keys()) if self.default_embedding_model is None and len(self.embedding_models) > 1: raise ConfigurationError( f"default_embedding_model is not set, and multiple embedding models are configured. Please specify one of: {available_embedding_model_aliases} as a default_embedding_model." ) if ( self.default_embedding_model is not None and self.default_embedding_model not in self.embedding_models ): raise ConfigurationError( f"default_embedding_model {self.default_embedding_model} is not in configured map of embedding models. Available models: {available_embedding_model_aliases} ." ) for model_alias, embedding_model in self.embedding_models.items(): embedding_model_provider = _get_model_provider_for_model_config( embedding_model ) embedding_model_name = embedding_model.model_name embedding_model_parameters = ( model_catalog.get_embedding_model_parameters( embedding_model_provider, embedding_model_name ) ) if embedding_model_parameters is None: raise ConfigurationError( model_catalog.generate_unsupported_embedding_model_error_message( embedding_model_provider, embedding_model_name ) ) if hasattr(embedding_model, "profiles") and embedding_model.profiles: profile_names = list(embedding_model.profiles.keys()) if ( embedding_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( embedding_model.default_profile is not None and embedding_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {embedding_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in embedding_model.profiles.items(): _validate_embedding_profile( embedding_model_parameters, model_alias, profile_alias, profile, ) return self ``` ## Session The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. Create a session with default configuration ``` session = Session.get_or_create(SessionConfig(app_name="my_app")) ``` Create a session with cloud configuration ``` config = SessionConfig( app_name="my_app", cloud=True, api_key="your_api_key" ) session = Session.get_or_create(config) ``` Methods: - **`create_dataframe`** – Create a DataFrame from a variety of Python-native data formats. - **`get_or_create`** – Gets an existing Session or creates a new one with the configured settings. - **`sql`** – Execute a read-only SQL query against one or more DataFrames using named placeholders. - **`stop`** – Stops the session and closes all connections. - **`table`** – Returns the specified table as a DataFrame. - **`view`** – Returns the specified view as a DataFrame. Attributes: - **`catalog`** (`Catalog`) – Interface for catalog operations on the Session. - **`read`** (`DataFrameReader`) – Returns a DataFrameReader that can be used to read data in as a DataFrame. ### catalog ``` catalog: Catalog ``` Interface for catalog operations on the Session. ### read ``` read: DataFrameReader ``` Returns a DataFrameReader that can be used to read data in as a DataFrame. Returns: - **`DataFrameReader`** ( `DataFrameReader` ) – A reader interface to read data into DataFrame Raises: - `RuntimeError` – If the session has been stopped ### create_dataframe ``` create_dataframe(data: DataLike, schema: Schema | None = None) -> DataFrame ``` Create a DataFrame from a variety of Python-native data formats. Parameters: - **`data`** (`DataLike`) – Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table - **`schema`** (`Schema | None`, default: `None` ) – Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: - `DataFrame` – A new DataFrame instance Raises: - `ValidationError` – If the input format is unsupported or the provided columns do not match the schema. - `PlanError` – If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Create from Polars DataFrame ``` import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from Pandas DataFrame ``` import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from dictionary ``` session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Create from list of dictionaries ``` session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Create from pyarrow Table ``` import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Create with an explicit schema ``` import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` Source code in `src/fenic/api/session/session.py` ``` def create_dataframe( self, data: DataLike, schema: Schema | None = None, ) -> DataFrame: """Create a DataFrame from a variety of Python-native data formats. Args: data: Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table schema: Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: A new DataFrame instance Raises: ValidationError: If the input format is unsupported or the provided columns do not match the schema. PlanError: If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Example: Create from Polars DataFrame ```python import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from Pandas DataFrame ```python import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from dictionary ```python session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Example: Create from list of dictionaries ```python session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Example: Create from pyarrow Table ```python import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Example: Create with an explicit schema ```python import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` """ pl_df, row_field_names = _normalize_data_like_to_polars( data, allow_empty_list=schema is not None, validate_all_rows=schema is not None, ) if schema is None: return DataFrame._from_logical_plan( InMemorySource.from_session_state(pl_df, self._session_state), self._session_state, ) coerced_pl_df = _coerce_to_schema(pl_df, schema, row_field_names=row_field_names) return DataFrame._from_logical_plan( InMemorySource.from_schema(coerced_pl_df, schema), self._session_state, ) ``` ### get_or_create ``` get_or_create(config: SessionConfig) -> Session ``` Gets an existing Session or creates a new one with the configured settings. Returns: - `Session` – A Session instance configured with the provided settings Source code in `src/fenic/api/session/session.py` ``` @classmethod def get_or_create( cls, config: SessionConfig, ) -> Session: """Gets an existing Session or creates a new one with the configured settings. Returns: A Session instance configured with the provided settings """ if config.cloud: from fenic._backends.cloud.manager import CloudSessionManager cloud_session_manager = CloudSessionManager() if not cloud_session_manager.initialized: session_manager_dependencies = ( CloudSessionManager.create_global_session_dependencies() ) cloud_session_manager.configure(session_manager_dependencies) future = asyncio.run_coroutine_threadsafe( cloud_session_manager.get_or_create_session_state(config), cloud_session_manager._asyncio_loop, ) cloud_session_state = future.result() return Session._create_cloud_session(cloud_session_state) local_session_state: LocalSessionState = LocalSessionManager().get_or_create_session_state(config._to_resolved_config()) return Session._create_local_session(local_session_state) ``` ### sql ``` sql(query: str, /, **tables: DataFrame) -> DataFrame ``` Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Parameters: - **`query`** (`str`) – A SQL query string with placeholders like `{df}` - **`**tables`** (`DataFrame`, default: `{}` ) – Keyword arguments mapping placeholder names to DataFrames Returns: - `DataFrame` – A lazy DataFrame representing the result of the SQL query Raises: - `ValidationError` – If a placeholder is used in the query but not passed as a keyword argument Simple join between two DataFrames ``` df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Complex query with multiple DataFrames ``` users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(""" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id """, users=users, orders=orders, products=products) ``` Source code in `src/fenic/api/session/session.py` ``` def sql(self, query: str, /, **tables: DataFrame) -> DataFrame: """Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Args: query: A SQL query string with placeholders like `{df}` **tables: Keyword arguments mapping placeholder names to DataFrames Returns: A lazy DataFrame representing the result of the SQL query Raises: ValidationError: If a placeholder is used in the query but not passed as a keyword argument Example: Simple join between two DataFrames ```python df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Example: Complex query with multiple DataFrames ```python users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(\"\"\" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id \"\"\", users=users, orders=orders, products=products) ``` """ query = query.strip() if not query: raise ValidationError("SQL query must not be empty.") placeholders = set(SQL_PLACEHOLDER_RE.findall(query)) missing = placeholders - tables.keys() if missing: raise ValidationError( f"Missing DataFrames for placeholders in SQL query: {', '.join(sorted(missing))}. " f"Make sure to pass them as keyword arguments, e.g., sql(..., {next(iter(missing))}=df)." ) logical_plans = [] template_names = [] input_session_states = [] for name, table in tables.items(): if name in placeholders: template_names.append(name) logical_plans.append(table._logical_plan) input_session_states.append(table._session_state) DataFrame._ensure_same_session(self._session_state, input_session_states) return DataFrame._from_logical_plan( SQL.from_session_state(logical_plans, template_names, query, self._session_state), self._session_state, ) ``` ### stop ``` stop(skip_usage_summary: bool = False) ``` Stops the session and closes all connections. Parameters: - **`skip_usage_summary`** (`bool`, default: `False` ) – Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. Source code in `src/fenic/api/session/session.py` ``` def stop(self, skip_usage_summary: bool = False): """Stops the session and closes all connections. Args: skip_usage_summary: Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. """ self._session_state.stop(skip_usage_summary=skip_usage_summary) ``` ### table ``` table(table_name: str) -> DataFrame ``` Returns the specified table as a DataFrame. Parameters: - **`table_name`** (`str`) – Name of the table Returns: - `DataFrame` – Table as a DataFrame Raises: - `ValueError` – If the table does not exist Load an existing table ``` df = session.table("my_table") ``` Source code in `src/fenic/api/session/session.py` ``` def table(self, table_name: str) -> DataFrame: """Returns the specified table as a DataFrame. Args: table_name: Name of the table Returns: Table as a DataFrame Raises: ValueError: If the table does not exist Example: Load an existing table ```python df = session.table("my_table") ``` """ if not self._session_state.catalog.does_table_exist(table_name): raise ValueError(f"Table {table_name} does not exist") return DataFrame._from_logical_plan( TableSource.from_session_state(table_name, self._session_state), self._session_state, ) ``` ### view ``` view(view_name: str) -> DataFrame ``` Returns the specified view as a DataFrame. Parameters: - **`view_name`** (`str`) – Name of the view Returns: DataFrame: Dataframe with the given view Source code in `src/fenic/api/session/session.py` ``` def view(self, view_name: str) -> DataFrame: """Returns the specified view as a DataFrame. Args: view_name: Name of the view Returns: DataFrame: Dataframe with the given view """ if not self._session_state.catalog.does_view_exist(view_name): raise CatalogError(f"View {view_name} does not exist") view_plan = self._session_state.catalog.get_view_plan(view_name) validate_view(view_name, view_plan, self._session_state) return DataFrame._from_logical_plan( view_plan, self._session_state, ) ``` ## SessionConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.SessionConfig[SessionConfig] click fenic.api.session.SessionConfig href "" "fenic.api.session.SessionConfig" ``` Configuration for a user session. This class defines the complete configuration for a user session, including application settings, model configurations, and optional cloud settings. It serves as the central configuration object for all language model operations. Attributes: - **`app_name`** (`str`) – Name of the application using this session. Defaults to "default_app". - **`db_path`** (`Optional[Path]`) – Optional path to a local database file for persistent storage. - **`semantic`** (`Optional[SemanticConfig]`) – Configuration for semantic models (optional). - **`cloud`** (`Optional[CloudConfig]`) – Optional configuration for cloud execution. - **`cache`** (`Optional[CloudConfig]`) – Optional configuration for LLM response caching. Note The semantic configuration is optional. When not provided, only non-semantic operations are available. The cloud configuration is optional and only needed for distributed processing. Example Configuring a basic session with a single language model: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ) } ), ) ``` Configuring a session with multiple models and cloud execution: ``` config = SessionConfig( app_name="production_app", db_path=Path("/path/to/database.db"), semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ), cloud=CloudConfig(size=CloudExecutorSize.MEDIUM), ) ``` Methods: - **`to_json`** – Export the session config to a JSON string. ### to_json ``` to_json() -> str ``` Export the session config to a JSON string. Source code in `src/fenic/api/session/config.py` ``` def to_json(self) -> str: """Export the session config to a JSON string.""" return self.model_dump_json(indent=2) ``` --- # fenic.api.session.config Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/session/config/ Session configuration classes for Fenic. Classes: - **`AdaptiveTokenEstimationConfig`** – Tunes adaptive output-token reservation for rate limiting. - **`AnthropicLanguageModel`** – Configuration for Anthropic language models. - **`CloudConfig`** – Configuration for cloud-based execution. - **`CloudExecutorSize`** – Enum defining available cloud executor sizes. - **`CohereEmbeddingModel`** – Configuration for Cohere embedding models. - **`GoogleDeveloperEmbeddingModel`** – Configuration for Google Developer embedding models. - **`GoogleDeveloperLanguageModel`** – Configuration for Gemini models accessible through Google Developer AI Studio. - **`GoogleVertexEmbeddingModel`** – Configuration for Google Vertex AI embedding models. - **`GoogleVertexLanguageModel`** – Configuration for Google Vertex AI models. - **`LLMResponseCacheConfig`** – Configuration for LLM response caching. - **`OpenAIEmbeddingModel`** – Configuration for OpenAI embedding models. - **`OpenAILanguageModel`** – Configuration for OpenAI language models. - **`OpenRouterLanguageModel`** – Configuration for OpenRouter language models. - **`SemanticConfig`** – Configuration for semantic language and embedding models. - **`SessionConfig`** – Configuration for a user session. ## AdaptiveTokenEstimationConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.AdaptiveTokenEstimationConfig[AdaptiveTokenEstimationConfig] click fenic.api.session.config.AdaptiveTokenEstimationConfig href "" "fenic.api.session.config.AdaptiveTokenEstimationConfig" ``` Tunes adaptive output-token reservation for rate limiting. Output-token reservations are learned from observed usage and clamped to the request's max_completion_tokens ceiling, then corrected after each response (settlement). Enabled by default. Setting `enabled=False` disables adaptive *estimation* β€” reservations fall back to the static worst-case ceiling instead of the learned distribution. Settlement (reconciling the token bucket to actual usage after each response) is **always on** regardless of this flag. It corrects the bucket in both directions β€” refunding the over-reservation (the common case) and debiting further when a request exceeds its reservation β€” and neither direction increases 429 risk: a refund only returns capacity the provider never charged, and a debit only makes the limiter more conservative. ## AnthropicLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.AnthropicLanguageModel[AnthropicLanguageModel] click fenic.api.session.config.AnthropicLanguageModel href "" "fenic.api.session.config.AnthropicLanguageModel" ``` Configuration for Anthropic language models. This class defines the configuration settings for Anthropic language models, including model selection and separate rate limiting parameters for input and output tokens. Attributes: - **`model_name`** (`AnthropicLanguageModelName`) – The name of the Anthropic model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`input_tpm`** (`int`) – Input tokens per minute limit; must be greater than 0. - **`output_tpm`** (`int`) – Output tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring an Anthropic model with separate input/output rate limits: ``` config = AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100 ) ``` Configuring an Anthropic model with profiles: ``` config = SessionConfig( semantic=SemanticConfig( language_models={ "claude": AnthropicLanguageModel( model_name="claude-sonnet-4-6", rpm=100, input_tpm=100, output_tpm=100, profiles={ "thinking_disabled": AnthropicLanguageModel.Profile(), "fast": AnthropicLanguageModel.Profile(thinking_token_budget=1024), "thorough": AnthropicLanguageModel.Profile(thinking_token_budget=4096) }, default_profile="fast" ) }, default_language_model="claude" ) # Using the default "fast" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias="claude") # Using the "thorough" profile for the "claude" model semantic.map(instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="claude", profile="thorough")) ``` Classes: - **`Profile`** – Anthropic-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.AnthropicLanguageModel.Profile[Profile] click fenic.api.session.config.AnthropicLanguageModel.Profile href "" "fenic.api.session.config.AnthropicLanguageModel.Profile" ``` Anthropic-specific profile configurations. This class defines profile configurations for Anthropic models, allowing different thinking and effort settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – Provide a default thinking budget in tokens. If not provided, thinking will be disabled for the profile. The minimum token budget supported by Anthropic is 1024 tokens. For Claude models that use adaptive thinking, use `effort` instead. - **`effort`** (`Optional[AnthropicReasoningEffortType]`) – Provider-native Anthropic effort level. Supported values vary by model: low, medium, high, xhigh, and max. On adaptive-thinking models the thinking budget shares the request's output token window rather than being reserved on top of it, so very high effort levels can consume part of the visible completion budget. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note If `thinking_token_budget` or adaptive `effort` enables thinking, `temperature` cannot be customized -- any changes to `temperature` will be ignored. Effort-only profiles on non-adaptive models configure Anthropic `output_config` without enabling thinking, so custom `temperature` remains available when the model supports it. Example Configuring a profile with a thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=2048) ``` Configuring a profile with a large thinking budget: ``` profile = AnthropicLanguageModel.Profile(thinking_token_budget=8192) ``` Configuring a profile with effort: ``` profile = AnthropicLanguageModel.Profile(effort="xhigh") ``` ## CloudConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.CloudConfig[CloudConfig] click fenic.api.session.config.CloudConfig href "" "fenic.api.session.config.CloudConfig" ``` Configuration for cloud-based execution. This class defines settings for running operations in a cloud environment, allowing for scalable and distributed processing of language model operations. Attributes: - **`size`** (`Optional[CloudExecutorSize]`) – Size of the cloud executor instance. If None, the default size will be used. Example Configuring cloud execution with a specific size: ``` config = CloudConfig(size=CloudExecutorSize.MEDIUM) ``` Using default cloud configuration: ``` config = CloudConfig() ``` ## CloudExecutorSize Bases: `str`, `Enum` ``` flowchart TD fenic.api.session.config.CloudExecutorSize[CloudExecutorSize] click fenic.api.session.config.CloudExecutorSize href "" "fenic.api.session.config.CloudExecutorSize" ``` Enum defining available cloud executor sizes. This enum represents the different size options available for cloud-based execution environments. Attributes: - **`SMALL`** – Small instance size. - **`MEDIUM`** – Medium instance size. - **`LARGE`** – Large instance size. - **`XLARGE`** – Extra large instance size. ## CohereEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.CohereEmbeddingModel[CohereEmbeddingModel] click fenic.api.session.config.CohereEmbeddingModel href "" "fenic.api.session.config.CohereEmbeddingModel" ``` Configuration for Cohere embedding models. This class defines the configuration settings for Cohere embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`CohereEmbeddingModelName`) – The name of the Cohere model to use. - **`rpm`** (`int`) – Requests per minute limit for the model. - **`tpm`** (`int`) – Tokens per minute limit for the model. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional dictionary of profile configurations. - **`default_profile`** (`Optional[str]`) – Default profile name to use if none specified. Example Configuring a Cohere embedding model with profiles: ``` cohere_config = CohereEmbeddingModel( model_name="embed-v4.0", rpm=100, tpm=50_000, profiles={ "high_dim": CohereEmbeddingModel.Profile( embedding_dimensionality=1536, embedding_task_type="search_document" ), "classification": CohereEmbeddingModel.Profile( embedding_dimensionality=1024, embedding_task_type="classification" ), }, default_profile="high_dim", ) ``` Classes: - **`Profile`** – Profile configurations for Cohere embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.CohereEmbeddingModel.Profile[Profile] click fenic.api.session.config.CohereEmbeddingModel.Profile href "" "fenic.api.session.config.CohereEmbeddingModel.Profile" ``` Profile configurations for Cohere embedding models. This class defines profile configurations for Cohere embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`input_type`** (`CohereEmbeddingTaskType`) – The type of input text (search_query, search_document, classification, clustering) Example Configuring a profile with custom dimensionality: ``` profile = CohereEmbeddingModel.Profile(output_dimensionality=1536) ``` Configuring a profile with default settings: ``` profile = CohereEmbeddingModel.Profile() ``` ## GoogleDeveloperEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleDeveloperEmbeddingModel[GoogleDeveloperEmbeddingModel] click fenic.api.session.config.GoogleDeveloperEmbeddingModel href "" "fenic.api.session.config.GoogleDeveloperEmbeddingModel" ``` Configuration for Google Developer embedding models. This class defines the configuration settings for Google embedding models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperEmbeddingModelName`) – The name of the Google Developer embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer embedding model with rate limits: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Developer embedding model with profiles: ``` config = GoogleDeveloperEmbeddingModelConfig( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleDeveloperEmbeddingModelConfig.Profile(), "high_dim": GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleDeveloperEmbeddingModel.Profile[Profile] click fenic.api.session.config.GoogleDeveloperEmbeddingModel.Profile href "" "fenic.api.session.config.GoogleDeveloperEmbeddingModel.Profile" ``` Profile configurations for Google Developer embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleDeveloperEmbeddingModelConfig.Profile() ``` ## GoogleDeveloperLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleDeveloperLanguageModel[GoogleDeveloperLanguageModel] click fenic.api.session.config.GoogleDeveloperLanguageModel href "" "fenic.api.session.config.GoogleDeveloperLanguageModel" ``` Configuration for Gemini models accessible through Google Developer AI Studio. This class defines the configuration settings for Google Gemini models available in Google Developer AI Studio, including model selection and rate limiting parameters. These models are accessible using a GOOGLE_API_KEY environment variable. Attributes: - **`model_name`** (`GoogleDeveloperLanguageModelName`) – The name of the Google Developer model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Developer model with rate limits: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Developer model with profiles: ``` config = GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleDeveloperLanguageModel.Profile(), "fast": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=1024 ), "thorough": GoogleDeveloperLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Developer models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleDeveloperLanguageModel.Profile[Profile] click fenic.api.session.config.GoogleDeveloperLanguageModel.Profile href "" "fenic.api.session.config.GoogleDeveloperLanguageModel.Profile" ``` Profile configurations for Google Developer models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleDeveloperLanguageModel.Profile(thinking_level="high") ``` ## GoogleVertexEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleVertexEmbeddingModel[GoogleVertexEmbeddingModel] click fenic.api.session.config.GoogleVertexEmbeddingModel href "" "fenic.api.session.config.GoogleVertexEmbeddingModel" ``` Configuration for Google Vertex AI embedding models. This class defines the configuration settings for Google embedding models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexEmbeddingModelName`) – The name of the Google Vertex embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex embedding model with rate limits: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000 ) ``` Configuring a Google Vertex embedding model with profiles: ``` embedding_model = GoogleVertexEmbeddingModel( model_name="gemini-embedding-001", rpm=100, tpm=1000, profiles={ "default": GoogleVertexEmbeddingModel.Profile(), "high_dim": GoogleVertexEmbeddingModel.Profile( output_dimensionality=3072 ), }, default_profile="default", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex embedding models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleVertexEmbeddingModel.Profile[Profile] click fenic.api.session.config.GoogleVertexEmbeddingModel.Profile href "" "fenic.api.session.config.GoogleVertexEmbeddingModel.Profile" ``` Profile configurations for Google Vertex embedding models. This class defines profile configurations for Google embedding models, allowing different output dimensionality and task type settings to be applied to the same model. Attributes: - **`output_dimensionality`** (`Optional[int]`) – The dimensionality of the embedding created by this model. If not provided, the model will use its default dimensionality. - **`task_type`** (`GoogleEmbeddingTaskType`) – The type of task for the embedding model. Example Configuring a profile with custom dimensionality: ``` profile = GoogleVertexEmbeddingModelConfig.Profile( output_dimensionality=3072 ) ``` Configuring a profile with default settings: ``` profile = GoogleVertexEmbeddingModelConfig.Profile() ``` ## GoogleVertexLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleVertexLanguageModel[GoogleVertexLanguageModel] click fenic.api.session.config.GoogleVertexLanguageModel href "" "fenic.api.session.config.GoogleVertexLanguageModel" ``` Configuration for Google Vertex AI models. This class defines the configuration settings for Google Gemini models available in Google Vertex AI, including model selection and rate limiting parameters. These models are accessible using Google Cloud credentials. Attributes: - **`model_name`** (`GoogleVertexLanguageModelName`) – The name of the Google Vertex model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Example Configuring a Google Vertex model with rate limits: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ) ``` Configuring a reasoning Google Vertex model with profiles: ``` config = GoogleVertexLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000, profiles={ "thinking_disabled": GoogleVertexLanguageModel.Profile(), "fast": GoogleVertexLanguageModel.Profile(thinking_token_budget=1024), "thorough": GoogleVertexLanguageModel.Profile( thinking_token_budget=8192 ), }, default_profile="fast", ) ``` Classes: - **`Profile`** – Profile configurations for Google Vertex models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.GoogleVertexLanguageModel.Profile[Profile] click fenic.api.session.config.GoogleVertexLanguageModel.Profile href "" "fenic.api.session.config.GoogleVertexLanguageModel.Profile" ``` Profile configurations for Google Vertex models. This class defines profile configurations for Google Gemini models, allowing different thinking/reasoning settings to be applied to the same underlying model. Attributes: - **`thinking_token_budget`** (`Optional[int]`) – If configuring a reasoning model, provide a thinking budget in tokens. If not provided, or if set to 0, thinking will be disabled for the profile (not supported on gemini-2.5-pro). To have the model automatically determine a thinking budget based on the complexity of the prompt, set this to -1. Note that Gemini models take this as a suggestion -- and not a hard limit. It is very possible for the model to generate far more thinking tokens than the suggested budget, and for the model to generate reasoning tokens even if thinking is disabled. Note: For gemini-3 models, use thinking_level instead. - **`thinking_level`** (`Optional[ThinkingLevelType]`) – For gemini-3+ models, set the thinking level to high, medium, low, or minimal. This parameter is mutually exclusive with thinking_token_budget. - **`media_resolution`** (`Optional[MediaResolutionType]`) – For gemini-3+ models, set the media resolution for PDF processing. Can be "low", "medium", or "high". Affects token cost per page. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Example Configuring a profile with a fixed thinking budget (gemini-2.5 and earlier): ``` profile = GoogleVertexLanguageModel.Profile(thinking_token_budget=4096) ``` Configuring a profile with thinking level (gemini-3+): ``` profile = GoogleVertexLanguageModel.Profile(thinking_level="high") ``` ## LLMResponseCacheConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.LLMResponseCacheConfig[LLMResponseCacheConfig] click fenic.api.session.config.LLMResponseCacheConfig href "" "fenic.api.session.config.LLMResponseCacheConfig" ``` Configuration for LLM response caching. LLM response caching stores the results of language model API calls to reduce costs and improve performance for repeated queries. This is distinct from DataFrame caching (the `.cache()` operator). Attributes: - **`enabled`** – Whether caching is enabled (default: True). - **`backend`** (`CacheBackend`) – Cache backend to use (default: LOCAL). - **`ttl`** (`str`) – Time-to-live duration string (default: "1h"). Format: where unit is s/m/h/d. Examples: "30s", "15m", "2h", "7d". Maximum: 30 days, Minimum: 1 second. - **`max_size_mb`** (`int`) – Maximum cache size in MB before LRU eviction (default: 128MB). - **`namespace`** (`str`) – Cache namespace for isolation (default: "default"). Example Basic configuration within SemanticConfig: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt": OpenAILanguageModel(model_name="gpt-4o-mini", rpm=100, tpm=1000) }, llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="1h", max_size_mb=1000, ) ) ) ``` Custom TTL and larger cache: ``` llm_response_cache=LLMResponseCacheConfig( enabled=True, ttl="7d", # 7 days max_size_mb=5000, ) ``` Disabled caching: ``` llm_response_cache=LLMResponseCacheConfig(enabled=False) ``` Methods: - **`ttl_seconds`** – Convert TTL string to seconds. - **`validate_ttl`** – Validate TTL duration string format. ### ttl_seconds ``` ttl_seconds() -> int ``` Convert TTL string to seconds. Returns: - `int` – TTL duration in seconds. Raises: - `ValueError` – If TTL format is invalid. Source code in `src/fenic/api/session/config.py` ``` def ttl_seconds(self) -> int: """Convert TTL string to seconds. Returns: TTL duration in seconds. Raises: ValueError: If TTL format is invalid. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, self.ttl.lower()) if not match: raise ValueError(f"Invalid TTL format: '{self.ttl}'") value, unit = match.groups() value = int(value) multipliers = {"s": 1, "m": 60, "h": 3600, "d": 86400} return value * multipliers[unit] ``` ### validate_ttl ``` validate_ttl(v: str) -> str ``` Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Parameters: - **`v`** (`str`) – TTL duration string to validate. Returns: - `str` – The validated TTL string. Raises: - `ValueError` – If format is invalid or value is out of range. Source code in `src/fenic/api/session/config.py` ``` @field_validator("ttl") @classmethod def validate_ttl(cls, v: str) -> str: """Validate TTL duration string format. Format: where unit is s/m/h/d Examples: "30s", "15m", "2h", "7d" Args: v: TTL duration string to validate. Returns: The validated TTL string. Raises: ValueError: If format is invalid or value is out of range. """ pattern = r"^(\d+)([smhd])$" match = re.match(pattern, v.lower()) if not match: raise ValueError( f"Invalid TTL format: '{v}'. " "Expected: where unit is s/m/h/d. " "Examples: '30m', '2h', '1d'" ) value, unit = match.groups() value = int(value) # Validate ranges if unit == "s" and value < 1: raise ValueError("TTL must be at least 1 second") if unit == "h" and value > 720: # 30 days raise ValueError("TTL cannot exceed 720 hours (30 days)") if unit == "d" and value > 30: raise ValueError("TTL cannot exceed 30 days") return v ``` ## OpenAIEmbeddingModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenAIEmbeddingModel[OpenAIEmbeddingModel] click fenic.api.session.config.OpenAIEmbeddingModel href "" "fenic.api.session.config.OpenAIEmbeddingModel" ``` Configuration for OpenAI embedding models. This class defines the configuration settings for OpenAI embedding models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAIEmbeddingModelName`) – The name of the OpenAI embedding model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. Example Configuring an OpenAI embedding model with rate limits: ``` config = OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) ``` ## OpenAILanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenAILanguageModel[OpenAILanguageModel] click fenic.api.session.config.OpenAILanguageModel href "" "fenic.api.session.config.OpenAILanguageModel" ``` Configuration for OpenAI language models. This class defines the configuration settings for OpenAI language models, including model selection and rate limiting parameters. Attributes: - **`model_name`** (`OpenAILanguageModelName`) – The name of the OpenAI model to use. - **`rpm`** (`int`) – Requests per minute limit; must be greater than 0. - **`tpm`** (`int`) – Tokens per minute limit; must be greater than 0. - **`profiles`** (`Optional[dict[str, Profile]]`) – Optional mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The name of the default profile to use if profiles are configured. Note When using an o-series or gpt5 reasoning model without specifying a reasoning effort in a Profile, the `reasoning_effort` will default to `low` (for o-series models) or `minimal` (for gpt5 models). Example Configuring an OpenAI language model with rate limits: ``` config = OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) ``` Configuring an OpenAI model with profiles: ``` config = OpenAILanguageModel( model_name="o4-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile(reasoning_effort="high"), }, default_profile="fast", ) ``` Using a profile in a semantic operation: ``` config = SemanticConfig( language_models={ "o4": OpenAILanguageModel( model_name="o4-mini", rpm=1_000, tpm=1_000_000, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ) }, default_language_model="o4", ) # Will use the default "fast" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias="o4", ) # Will use the "thorough" profile for the "o4" model semantic.map( instruction="Construct a formal proof of the {hypothesis}.", model_alias=ModelAlias(name="o4", profile="thorough"), ) ``` Classes: - **`Profile`** – OpenAI-specific profile configurations. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenAILanguageModel.Profile[Profile] click fenic.api.session.config.OpenAILanguageModel.Profile href "" "fenic.api.session.config.OpenAILanguageModel.Profile" ``` OpenAI-specific profile configurations. This class defines profile configurations for OpenAI models, allowing a user to reference the same underlying model in semantic operations with different settings. Attributes: - **`reasoning_effort`** (`Optional[ReasoningEffort]`) – Provide a reasoning effort. Only for gpt5 and o-series models. Valid values: 'none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max'. - For gpt-5.6 models: supports 'xhigh' and 'max' - For gpt-5.5 models: defaults to 'medium', supports 'xhigh' - For gpt-5.4 models: defaults to 'none' (disabled reasoning), supports 'xhigh' - For gpt-5.1 and gpt-5.2 models: defaults to 'none' (disabled reasoning), does NOT support 'minimal' or 'xhigh' - For gpt-5 models: defaults to 'minimal', does NOT support 'none' - For o-series models: defaults to 'low', does NOT support 'none' or 'minimal' - **`verbosity`** (`Optional[Verbosity]`) – Provide a verbosity level. Only for gpt5/gpt5.1 models. Raises: - `ConfigurationError` – If a profile is set with parameters that are not supported by the model. Note When using an o-series or gpt5 reasoning model with reasoning enabled, the `temperature` cannot be customized. For gpt-5.1 models with reasoning_effort='none', temperature CAN be customized. Example Configuring a profile with medium reasoning effort: ``` profile = OpenAILanguageModel.Profile(reasoning_effort="medium") ``` ## OpenRouterLanguageModel Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenRouterLanguageModel[OpenRouterLanguageModel] click fenic.api.session.config.OpenRouterLanguageModel href "" "fenic.api.session.config.OpenRouterLanguageModel" ``` Configuration for OpenRouter language models. This class defines the configuration settings for OpenRouter language models, including model selection and rate limiting parameters. When fetching available models from OpenRouter, results will be filtered to only include models from providers that are not in the user’s ignored providers list and are either in the user’s allowed providers list (if configured) or from any provider (if no allowed providers are specified). Attributes: - **`model_name`** (`str`) – `{family}/{model}` identifier (e.g., `anthropic/claude-3-5-sonnet`). - **`profiles`** (`Optional[dict[str, Profile]]`) – Mapping of profile names to profile configurations. - **`default_profile`** (`Optional[str]`) – The key in `profiles` to select by default. - **`structured_output_strategy`** (`Optional[StructuredOutputStrategy]`) – The strategy to use for structured output if a model supports both tool calling and structured outputs. - `prefer_tools`: prefer using tools over response format. - `prefer_response_format`: prefer using response format over tools. Requirements - Set `OPENROUTER_API_KEY` in your environment. Example: ``` OpenRouterLanguageModel( model_name="openai/gpt-oss-20b", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="price" # Routes to the cheapest available provider ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="anthropic/claude-sonnet-4", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( only=[ "Anthropic" ] # ensures the request will only be routed to Anthropic and not AWS Bedrock or Google Vertex ) ) }, ) ``` Example: ``` OpenRouterLanguageModel( model_name="qwen/qwen3-next-80b-a3b-instruct", profiles={ "default": OpenRouterLanguageModel.Profile( provider=OpenRouterLanguageModel.Provider( sort="throughput", # routes to the provider with the highest overall throughput data_collection="deny" # eliminates providers that retain prompt data (would only route to DeepInfra/AtlasCloud, in this example) # Eliminate providers that offer an fp8 quantized version of the model, only allowing bf16. # Note that many providers have an `unknown` quantization, so you may be excluding more providers than you expect. quantizations=["bf16"] ) ) } ) ``` Classes: - **`Profile`** – Profile configurations for OpenRouter language models. - **`Provider`** – Provider routing configuration for OpenRouter language models. ### Profile Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenRouterLanguageModel.Profile[Profile] click fenic.api.session.config.OpenRouterLanguageModel.Profile href "" "fenic.api.session.config.OpenRouterLanguageModel.Profile" ``` Profile configurations for OpenRouter language models. Attributes: - **`models`** (`Optional[list[str]]`) – A list of fallback models to use if the primary model is unavailable. ([OpenRouter Documentation](https://openrouter.ai/docs/features/model-routing#the-models-parameter)). - **`provider`** (`Optional[Provider]`) – Provider routing preferences (include/exclude specific providers, set provider ranking method preference) ([OpenRouter Documentation](https://openrouter.ai/docs/features/provider-routing)). - **`reasoning_effort`** (`Optional[OpenRouterReasoningEffort]`) – OpenRouter reasoning effort configuration (none, minimal, low, medium, high, xhigh, max). If the model does support reasoning, but not `reasoning_effort`, a `reasoning_max_tokens` will be calculated that is roughly equivalent as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#reasoning-effort-level)) - **`reasoning_max_tokens`** (`Optional[int]`) – Supported by Anthropic, Gemini, etc., sets a token budget for reasoning If the model does support reasoning, but not `reasoning_max_tokens`, a `reasoning_effort_ will be automatically calculated based on`reasoning_max_tokens` as a percentage of the model's maximum output size ([OpenRouter Documentation](https://openrouter.ai/docs/use-cases/reasoning-tokens#max-tokens-for-reasoning)) - **`parsing_engine`** (`Optional[ParsingEngine]`) – The parsing engine to use for processing PDF files. By default, the model's native parsing engine will be used. If the model doesn't support PDF processing and the parsing engine is not provided, an error will be raised. Note: 'mistral-ocr' incurs additional costs. ([OpenRouter Documentation](https://openrouter.ai/docs/features/multimodal/pdfs)) ### Provider Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.OpenRouterLanguageModel.Provider[Provider] click fenic.api.session.config.OpenRouterLanguageModel.Provider href "" "fenic.api.session.config.OpenRouterLanguageModel.Provider" ``` Provider routing configuration for OpenRouter language models. [Provider Routing Documentation](https://openrouter.ai/docs/features/provider-routing) Attributes: - **`order`** (`Optional[list[str]]`) – List of providers to try in order (e.g. ['Anthropic', 'Amazon Bedrock']). - **`sort`** (`Optional[ProviderSort]`) – Provider routing preference (e.g. 'price', 'throughput', 'latency'). "price" will route to the cheapest available provider first, progressing through the list of providers in order of price. "throughput" will route to the provider with the highest overall recent throughput, progressing through the list of providers in order of throughput. "latency" will route to the provider with the lowest overall recent latency, progressing through the list of providers in order of latency. - **`quantizations`** (`Optional[list[ModelQuantization]]`) – Allowed quantizations. Note: many providers report `unknown`. - **`data_collection`** (`Optional[DataCollection]`) – Data collection preference. `allow`: allows the use of providers which store prompt data non-transiently and may train on it. `deny`: use only providers which do not collect/store prompt data. - **`only`** (`Optional[list[str]]`) – Only include these providers when performing provider routing. - **`exclude`** (`Optional[list[str]]`) – Exclude these providers when performing provider routing. - **`max_prompt_price`** (`Optional[float]`) – Maximum prompt price ($USD per 1M tokens). - **`max_completion_price`** (`Optional[float]`) – Maximum completion price ($USD per 1M tokens). ## SemanticConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.SemanticConfig[SemanticConfig] click fenic.api.session.config.SemanticConfig href "" "fenic.api.session.config.SemanticConfig" ``` Configuration for semantic language and embedding models. This class defines the configuration for both language models and optional embedding models used in semantic operations. It ensures that all configured models are valid and supported by their respective providers. Attributes: - **`language_models`** (`Optional[dict[str, LanguageModel]]`) – Mapping of model aliases to language model configurations. - **`default_language_model`** (`Optional[str]`) – The alias of the default language model to use for semantic operations. Not required if only one language model is configured. - **`embedding_models`** (`Optional[dict[str, EmbeddingModel]]`) – Optional mapping of model aliases to embedding model configurations. - **`default_embedding_model`** (`Optional[str]`) – The alias of the default embedding model to use for semantic operations. Note The embedding model is optional and only required for operations that need semantic search or embedding capabilities. Example Configuring semantic models with a single language model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel(model_name="gpt-4.1-nano", rpm=100, tpm=100) } ) ``` Configuring semantic models with multiple language models and an embedding model: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), "gemini": GoogleDeveloperLanguageModel( model_name="gemini-2.5-flash", rpm=100, tpm=1000 ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ) ``` Configuring models with profiles: ``` config = SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4o-mini", rpm=100, tpm=100, profiles={ "fast": OpenAILanguageModel.Profile(reasoning_effort="low"), "thorough": OpenAILanguageModel.Profile( reasoning_effort="high" ), }, default_profile="fast", ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, profiles={ "fast": AnthropicLanguageModel.Profile(effort="low"), "thorough": AnthropicLanguageModel.Profile(effort="high"), }, default_profile="fast", ), }, default_language_model="gpt4", ) ``` Methods: - **`model_post_init`** – Post initialization hook to set defaults. - **`validate_models`** – Validates that the selected models are supported by the system. ### model_post_init ``` model_post_init(__context) -> None ``` Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. Source code in `src/fenic/api/session/config.py` ``` def model_post_init(self, __context) -> None: """Post initialization hook to set defaults. This hook runs after the model is initialized and validated. It sets the default language and embedding models if they are not set and there is only one model available. For Google models that support thinking_level, it auto-creates "low" and "high" profiles if no profiles are configured. """ if self.language_models: # Set default language model if not set and only one model exists if self.default_language_model is None and len(self.language_models) == 1: self.default_language_model = list(self.language_models.keys())[0] # Auto-create profiles for Google models that support thinking_level for model_config in self.language_models.values(): if isinstance(model_config, (GoogleDeveloperLanguageModel, GoogleVertexLanguageModel)): model_provider = _get_model_provider_for_model_config(model_config) model_params = model_catalog.get_completion_model_parameters( model_provider, model_config.model_name ) if model_params and model_params.supported_thinking_levels and model_config.profiles is None: # Auto-create profiles for each supported thinking level if isinstance(model_config, GoogleDeveloperLanguageModel): model_config.profiles = { level: GoogleDeveloperLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } else: model_config.profiles = { level: GoogleVertexLanguageModel.Profile(thinking_level=level) for level in model_params.supported_thinking_levels } model_config.default_profile = "low" # Set default profile for each model if not set and only one profile exists for model_config in self.language_models.values(): if model_config.profiles is not None: profile_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(profile_names) == 1: model_config.default_profile = profile_names[0] # Set default embedding model if not set and only one model exists if self.embedding_models: if self.default_embedding_model is None and len(self.embedding_models) == 1: self.default_embedding_model = list(self.embedding_models.keys())[0] # Set default profile for each model if not set and only one preset exists for model_config in self.embedding_models.values(): if ( hasattr(model_config, "profiles") and model_config.profiles is not None ): preset_names = list(model_config.profiles.keys()) if model_config.default_profile is None and len(preset_names) == 1: model_config.default_profile = preset_names[0] ``` ### validate_models ``` validate_models() -> SemanticConfig ``` Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: - `SemanticConfig` – The validated SemanticConfig instance. Raises: - `ConfigurationError` – If any of the models are not supported. Source code in `src/fenic/api/session/config.py` ``` @model_validator(mode="after") def validate_models(self) -> SemanticConfig: """Validates that the selected models are supported by the system. This validator checks that both the language model and embedding model (if provided) are valid and supported by their respective providers. Returns: The validated SemanticConfig instance. Raises: ConfigurationError: If any of the models are not supported. """ # Skip validation if no models configured (embedding-only or empty config) if not self.language_models and not self.embedding_models: return self # Validate language models if provided if self.language_models: available_language_model_aliases = list(self.language_models.keys()) if self.default_language_model is None and len(self.language_models) > 1: raise ConfigurationError( f"default_language_model is not set, and multiple language models are configured. Please specify one of: {available_language_model_aliases} as a default_language_model." ) if ( self.default_language_model is not None and self.default_language_model not in self.language_models ): raise ConfigurationError( f"default_language_model {self.default_language_model} is not in configured map of language models. Available models: {available_language_model_aliases} ." ) for model_alias, language_model in self.language_models.items(): language_model_name = language_model.model_name language_model_provider = _get_model_provider_for_model_config( language_model ) completion_model_params = model_catalog.get_completion_model_parameters( language_model_provider, language_model_name ) if completion_model_params is None: raise ConfigurationError( model_catalog.generate_unsupported_completion_model_error_message( language_model_provider, language_model_name ) ) if language_model.profiles is not None: if not completion_model_params.supports_profiles: raise ConfigurationError( f"Model '{model_alias}' does not support parameter profiles. Please remove the Profile configuration." ) profile_names = list(language_model.profiles.keys()) if ( language_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( language_model.default_profile is not None and language_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {language_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in language_model.profiles.items(): _validate_language_profile( language_model, model_alias, completion_model_params, profile, profile_alias, ) if self.embedding_models is not None: available_embedding_model_aliases = list(self.embedding_models.keys()) if self.default_embedding_model is None and len(self.embedding_models) > 1: raise ConfigurationError( f"default_embedding_model is not set, and multiple embedding models are configured. Please specify one of: {available_embedding_model_aliases} as a default_embedding_model." ) if ( self.default_embedding_model is not None and self.default_embedding_model not in self.embedding_models ): raise ConfigurationError( f"default_embedding_model {self.default_embedding_model} is not in configured map of embedding models. Available models: {available_embedding_model_aliases} ." ) for model_alias, embedding_model in self.embedding_models.items(): embedding_model_provider = _get_model_provider_for_model_config( embedding_model ) embedding_model_name = embedding_model.model_name embedding_model_parameters = ( model_catalog.get_embedding_model_parameters( embedding_model_provider, embedding_model_name ) ) if embedding_model_parameters is None: raise ConfigurationError( model_catalog.generate_unsupported_embedding_model_error_message( embedding_model_provider, embedding_model_name ) ) if hasattr(embedding_model, "profiles") and embedding_model.profiles: profile_names = list(embedding_model.profiles.keys()) if ( embedding_model.default_profile is None and len(profile_names) > 0 ): raise ConfigurationError( f"default_profile is not set for model {model_alias}, but multiple profiles are configured. Please specify one of: {profile_names} as a default_profile." ) if ( embedding_model.default_profile is not None and embedding_model.default_profile not in profile_names ): raise ConfigurationError( f"default_profile {embedding_model.default_profile} is not in configured profiles for model {model_alias}. Available profiles: {profile_names}" ) for profile_alias, profile in embedding_model.profiles.items(): _validate_embedding_profile( embedding_model_parameters, model_alias, profile_alias, profile, ) return self ``` ## SessionConfig Bases: `BaseModel` ``` flowchart TD fenic.api.session.config.SessionConfig[SessionConfig] click fenic.api.session.config.SessionConfig href "" "fenic.api.session.config.SessionConfig" ``` Configuration for a user session. This class defines the complete configuration for a user session, including application settings, model configurations, and optional cloud settings. It serves as the central configuration object for all language model operations. Attributes: - **`app_name`** (`str`) – Name of the application using this session. Defaults to "default_app". - **`db_path`** (`Optional[Path]`) – Optional path to a local database file for persistent storage. - **`semantic`** (`Optional[SemanticConfig]`) – Configuration for semantic models (optional). - **`cloud`** (`Optional[CloudConfig]`) – Optional configuration for cloud execution. - **`cache`** (`Optional[CloudConfig]`) – Optional configuration for LLM response caching. Note The semantic configuration is optional. When not provided, only non-semantic operations are available. The cloud configuration is optional and only needed for distributed processing. Example Configuring a basic session with a single language model: ``` config = SessionConfig( app_name="my_app", semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ) } ), ) ``` Configuring a session with multiple models and cloud execution: ``` config = SessionConfig( app_name="production_app", db_path=Path("/path/to/database.db"), semantic=SemanticConfig( language_models={ "gpt4": OpenAILanguageModel( model_name="gpt-4.1-nano", rpm=100, tpm=100 ), "claude": AnthropicLanguageModel( model_name="claude-haiku-4-5", rpm=100, input_tpm=100, output_tpm=100, ), }, default_language_model="gpt4", embedding_models={ "openai_embeddings": OpenAIEmbeddingModel( model_name="text-embedding-3-small", rpm=100, tpm=100 ) }, default_embedding_model="openai_embeddings", ), cloud=CloudConfig(size=CloudExecutorSize.MEDIUM), ) ``` Methods: - **`to_json`** – Export the session config to a JSON string. ### to_json ``` to_json() -> str ``` Export the session config to a JSON string. Source code in `src/fenic/api/session/config.py` ``` def to_json(self) -> str: """Export the session config to a JSON string.""" return self.model_dump_json(indent=2) ``` --- # fenic.api.session.session Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/api/session/session/ Main session class for interacting with the DataFrame API. Classes: - **`Session`** – The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. ## Session The entry point to programming with the DataFrame API. Similar to PySpark's SparkSession. Create a session with default configuration ``` session = Session.get_or_create(SessionConfig(app_name="my_app")) ``` Create a session with cloud configuration ``` config = SessionConfig( app_name="my_app", cloud=True, api_key="your_api_key" ) session = Session.get_or_create(config) ``` Methods: - **`create_dataframe`** – Create a DataFrame from a variety of Python-native data formats. - **`get_or_create`** – Gets an existing Session or creates a new one with the configured settings. - **`sql`** – Execute a read-only SQL query against one or more DataFrames using named placeholders. - **`stop`** – Stops the session and closes all connections. - **`table`** – Returns the specified table as a DataFrame. - **`view`** – Returns the specified view as a DataFrame. Attributes: - **`catalog`** (`Catalog`) – Interface for catalog operations on the Session. - **`read`** (`DataFrameReader`) – Returns a DataFrameReader that can be used to read data in as a DataFrame. ### catalog ``` catalog: Catalog ``` Interface for catalog operations on the Session. ### read ``` read: DataFrameReader ``` Returns a DataFrameReader that can be used to read data in as a DataFrame. Returns: - **`DataFrameReader`** ( `DataFrameReader` ) – A reader interface to read data into DataFrame Raises: - `RuntimeError` – If the session has been stopped ### create_dataframe ``` create_dataframe(data: DataLike, schema: Schema | None = None) -> DataFrame ``` Create a DataFrame from a variety of Python-native data formats. Parameters: - **`data`** (`DataLike`) – Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table - **`schema`** (`Schema | None`, default: `None` ) – Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: - `DataFrame` – A new DataFrame instance Raises: - `ValidationError` – If the input format is unsupported or the provided columns do not match the schema. - `PlanError` – If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Create from Polars DataFrame ``` import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from Pandas DataFrame ``` import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Create from dictionary ``` session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Create from list of dictionaries ``` session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Create from pyarrow Table ``` import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Create with an explicit schema ``` import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` Source code in `src/fenic/api/session/session.py` ``` def create_dataframe( self, data: DataLike, schema: Schema | None = None, ) -> DataFrame: """Create a DataFrame from a variety of Python-native data formats. Args: data: Input data. Must be one of: - Polars DataFrame - Pandas DataFrame - dict of column_name -> list of values - list of dicts (each dict representing a row) - pyarrow Table schema: Optional complete top-level fenic schema. When provided, field names are authoritative, result columns are ordered to match the schema, values are physically coerced to the schema's Polars representation, and the logical DataFrame schema is preserved exactly. Use this for logical string-backed types such as JSON and Markdown, and for preserving fixed-size embedding arrays through local and cloud execution. Returns: A new DataFrame instance Raises: ValidationError: If the input format is unsupported or the provided columns do not match the schema. PlanError: If the input data cannot be coerced to the provided schema, or the schema is invalid for plan construction. Example: Create from Polars DataFrame ```python import polars as pl df = pl.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from Pandas DataFrame ```python import pandas as pd df = pd.DataFrame({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(df) ``` Example: Create from dictionary ```python session.create_dataframe({"col1": [1, 2], "col2": ["a", "b"]}) ``` Example: Create from list of dictionaries ```python session.create_dataframe([ {"col1": 1, "col2": "a"}, {"col1": 2, "col2": "b"} ]) ``` Example: Create from pyarrow Table ```python import pyarrow as pa table = pa.Table.from_pydict({"col1": [1, 2], "col2": ["a", "b"]}) session.create_dataframe(table) ``` Example: Create with an explicit schema ```python import fenic as fc schema = fc.Schema([ fc.ColumnField("age", fc.IntegerType), fc.ColumnField("name", fc.StringType), ]) session.create_dataframe({"name": ["Alice"], "age": ["42"]}, schema=schema) ``` """ pl_df, row_field_names = _normalize_data_like_to_polars( data, allow_empty_list=schema is not None, validate_all_rows=schema is not None, ) if schema is None: return DataFrame._from_logical_plan( InMemorySource.from_session_state(pl_df, self._session_state), self._session_state, ) coerced_pl_df = _coerce_to_schema(pl_df, schema, row_field_names=row_field_names) return DataFrame._from_logical_plan( InMemorySource.from_schema(coerced_pl_df, schema), self._session_state, ) ``` ### get_or_create ``` get_or_create(config: SessionConfig) -> Session ``` Gets an existing Session or creates a new one with the configured settings. Returns: - `Session` – A Session instance configured with the provided settings Source code in `src/fenic/api/session/session.py` ``` @classmethod def get_or_create( cls, config: SessionConfig, ) -> Session: """Gets an existing Session or creates a new one with the configured settings. Returns: A Session instance configured with the provided settings """ if config.cloud: from fenic._backends.cloud.manager import CloudSessionManager cloud_session_manager = CloudSessionManager() if not cloud_session_manager.initialized: session_manager_dependencies = ( CloudSessionManager.create_global_session_dependencies() ) cloud_session_manager.configure(session_manager_dependencies) future = asyncio.run_coroutine_threadsafe( cloud_session_manager.get_or_create_session_state(config), cloud_session_manager._asyncio_loop, ) cloud_session_state = future.result() return Session._create_cloud_session(cloud_session_state) local_session_state: LocalSessionState = LocalSessionManager().get_or_create_session_state(config._to_resolved_config()) return Session._create_local_session(local_session_state) ``` ### sql ``` sql(query: str, /, **tables: DataFrame) -> DataFrame ``` Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Parameters: - **`query`** (`str`) – A SQL query string with placeholders like `{df}` - **`**tables`** (`DataFrame`, default: `{}` ) – Keyword arguments mapping placeholder names to DataFrames Returns: - `DataFrame` – A lazy DataFrame representing the result of the SQL query Raises: - `ValidationError` – If a placeholder is used in the query but not passed as a keyword argument Simple join between two DataFrames ``` df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Complex query with multiple DataFrames ``` users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(""" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id """, users=users, orders=orders, products=products) ``` Source code in `src/fenic/api/session/session.py` ``` def sql(self, query: str, /, **tables: DataFrame) -> DataFrame: """Execute a read-only SQL query against one or more DataFrames using named placeholders. This allows you to execute ad hoc SQL queries using familiar syntax when it's more convenient than the DataFrame API. Placeholders in the SQL string (e.g. `{df}`) should correspond to keyword arguments (e.g. `df=my_dataframe`). For supported SQL syntax and functions, refer to the DuckDB SQL documentation: https://duckdb.org/docs/sql/introduction. Args: query: A SQL query string with placeholders like `{df}` **tables: Keyword arguments mapping placeholder names to DataFrames Returns: A lazy DataFrame representing the result of the SQL query Raises: ValidationError: If a placeholder is used in the query but not passed as a keyword argument Example: Simple join between two DataFrames ```python df1 = session.create_dataframe({"id": [1, 2]}) df2 = session.create_dataframe({"id": [2, 3]}) result = session.sql( "SELECT * FROM {df1} JOIN {df2} USING (id)", df1=df1, df2=df2 ) ``` Example: Complex query with multiple DataFrames ```python users = session.create_dataframe({"user_id": [1, 2], "name": ["Alice", "Bob"]}) orders = session.create_dataframe({"order_id": [1, 2], "user_id": [1, 2]}) products = session.create_dataframe({"product_id": [1, 2], "name": ["Widget", "Gadget"]}) result = session.sql(\"\"\" SELECT u.name, p.name as product FROM {users} u JOIN {orders} o ON u.user_id = o.user_id JOIN {products} p ON o.product_id = p.product_id \"\"\", users=users, orders=orders, products=products) ``` """ query = query.strip() if not query: raise ValidationError("SQL query must not be empty.") placeholders = set(SQL_PLACEHOLDER_RE.findall(query)) missing = placeholders - tables.keys() if missing: raise ValidationError( f"Missing DataFrames for placeholders in SQL query: {', '.join(sorted(missing))}. " f"Make sure to pass them as keyword arguments, e.g., sql(..., {next(iter(missing))}=df)." ) logical_plans = [] template_names = [] input_session_states = [] for name, table in tables.items(): if name in placeholders: template_names.append(name) logical_plans.append(table._logical_plan) input_session_states.append(table._session_state) DataFrame._ensure_same_session(self._session_state, input_session_states) return DataFrame._from_logical_plan( SQL.from_session_state(logical_plans, template_names, query, self._session_state), self._session_state, ) ``` ### stop ``` stop(skip_usage_summary: bool = False) ``` Stops the session and closes all connections. Parameters: - **`skip_usage_summary`** (`bool`, default: `False` ) – Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. Source code in `src/fenic/api/session/session.py` ``` def stop(self, skip_usage_summary: bool = False): """Stops the session and closes all connections. Args: skip_usage_summary: Whether to skip printing the usage summary. Unless `skip_usage_summary` is set, a summary of your session's metrics will print once you stop your session. """ self._session_state.stop(skip_usage_summary=skip_usage_summary) ``` ### table ``` table(table_name: str) -> DataFrame ``` Returns the specified table as a DataFrame. Parameters: - **`table_name`** (`str`) – Name of the table Returns: - `DataFrame` – Table as a DataFrame Raises: - `ValueError` – If the table does not exist Load an existing table ``` df = session.table("my_table") ``` Source code in `src/fenic/api/session/session.py` ``` def table(self, table_name: str) -> DataFrame: """Returns the specified table as a DataFrame. Args: table_name: Name of the table Returns: Table as a DataFrame Raises: ValueError: If the table does not exist Example: Load an existing table ```python df = session.table("my_table") ``` """ if not self._session_state.catalog.does_table_exist(table_name): raise ValueError(f"Table {table_name} does not exist") return DataFrame._from_logical_plan( TableSource.from_session_state(table_name, self._session_state), self._session_state, ) ``` ### view ``` view(view_name: str) -> DataFrame ``` Returns the specified view as a DataFrame. Parameters: - **`view_name`** (`str`) – Name of the view Returns: DataFrame: Dataframe with the given view Source code in `src/fenic/api/session/session.py` ``` def view(self, view_name: str) -> DataFrame: """Returns the specified view as a DataFrame. Args: view_name: Name of the view Returns: DataFrame: Dataframe with the given view """ if not self._session_state.catalog.does_view_exist(view_name): raise CatalogError(f"View {view_name} does not exist") view_plan = self._session_state.catalog.get_view_plan(view_name) validate_view(view_name, view_plan, self._session_state) return DataFrame._from_logical_plan( view_plan, self._session_state, ) ``` --- # fenic.core Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/ Core module for Fenic. Classes: - **`ArrayType`** – A type representing a homogeneous variable-length array (list) of elements. - **`BoundToolParam`** – A bound tool parameter. - **`ClassDefinition`** – Definition of a classification class with optional description. - **`ClassifyExample`** – A single semantic example for classification operations. - **`ClassifyExampleCollection`** – Collection of text-to-category examples for classification operations. - **`ColumnField`** – Represents a typed column in a DataFrame schema. - **`DataType`** – Base class for all data types. - **`DatasetMetadata`** – Metadata for a dataset (table or view). - **`DocumentPathType`** – Represents a string containing a a document's local (file system) or remote (URL) path. - **`EmbeddingType`** – A type representing a fixed-length embedding vector. - **`JoinExample`** – A single semantic example for semantic join operations. - **`JoinExampleCollection`** – Collection of comparison examples for semantic join operations. - **`KeyPoints`** – Summary as a concise bulleted list. - **`LMMetrics`** – Tracks language model usage metrics including token counts and costs. - **`MapExample`** – A single semantic example for semantic mapping operations. - **`MapExampleCollection`** – Collection of input-output examples for semantic map operations. - **`OperatorMetrics`** – Metrics for a single operator in the query execution plan. - **`Paragraph`** – Summary as a cohesive narrative. - **`PredicateExample`** – A single semantic example for semantic predicate operations. - **`PredicateExampleCollection`** – Collection of input-to-boolean examples for predicate operations. - **`QueryMetrics`** – Comprehensive metrics for an executed query. - **`QueryResult`** – Container for query execution results and associated metadata. - **`RMMetrics`** – Tracks embedding model usage metrics including token counts and costs. - **`Schema`** – Represents the schema of a DataFrame. - **`StructField`** – A field in a StructType. Fields are nullable. - **`StructType`** – A type representing a struct (record) with named fields. - **`SystemTool`** – A tool implemented as a regular Python function with explicit parameters. - **`ToolParam`** – A parameter for a parameterized view tool. - **`TranscriptType`** – Represents a string containing a transcript in a specific format. - **`UserDefinedTool`** – A tool that has been bound to a specific Parameterized View. Attributes: - **`BooleanType`** – Represents a boolean value. (True/False) - **`BranchSide`** – Type alias representing the side of a branch in a lineage graph. - **`DataCollection`** – Type alias representing provider data collection policies. - **`DataLike`** – Union type representing any supported data format for both input and output operations. - **`DataLikeType`** – String literal type for specifying data output formats. - **`DateType`** – Represents a date value. - **`DoubleType`** – Represents a 64-bit floating-point number. - **`FloatType`** – Represents a 32-bit floating-point number. - **`FuzzySimilarityMethod`** – Type alias representing the supported fuzzy string similarity algorithms. - **`HtmlType`** – Represents a string containing raw HTML markup. - **`IntegerType`** – Represents a signed integer value. - **`JsonType`** – Represents a string containing JSON data. - **`MarkdownType`** – Represents a string containing Markdown-formatted text. - **`ModelQuantization`** – Type alias representing supported quantization formats for provider models. - **`ProviderSort`** – Type alias representing provider sorting strategies used by OpenRouter routing. - **`SemanticSimilarityMetric`** – Type alias representing supported semantic similarity metrics. - **`StringType`** – Represents a UTF-8 encoded string value. - **`StructuredOutputStrategy`** – Type alias representing the strategy to use when a model supports both - **`TimestampType`** – Represents a timestamp value. ## BooleanType ``` BooleanType = _BooleanType() ``` Represents a boolean value. (True/False) ## BranchSide ``` BranchSide = Literal['left', 'right'] ``` Type alias representing the side of a branch in a lineage graph. Valid values: - "left": The left branch of a join. - "right": The right branch of a join. ## DataCollection ``` DataCollection = Literal['allow', 'deny'] ``` Type alias representing provider data collection policies. Valid values: - "allow": Permit providers that may retain or train on prompts non-transiently. - "deny": Restrict to providers that do not collect/store user data. ## DataLike ``` DataLike = Union[pl.DataFrame, pd.DataFrame, Dict[str, List[Any]], List[Dict[str, Any]], pa.Table] ``` Union type representing any supported data format for both input and output operations. This type encompasses all possible data structures that can be: 1. Used as input when creating DataFrames 2. Returned as output from query results Supported formats - pl.DataFrame: Native Polars DataFrame with efficient columnar storage - pd.DataFrame: Pandas DataFrame, optionally with PyArrow extension arrays - Dict[str, List[Any]]: Column-oriented dictionary where: - Keys are column names (str) - Values are lists containing all values for that column - List[Dict[str, Any]]: Row-oriented list where: - Each element is a dictionary representing one row - Dictionary keys are column names, values are cell values - pa.Table: Apache Arrow Table with columnar memory layout Usage - Input: Used in create_dataframe() to accept data in various formats - Output: Used in QueryResult.data to return results in requested format The specific type returned depends on the DataLikeType format specified when collecting query results. ## DataLikeType ``` DataLikeType = Literal['polars', 'pandas', 'pydict', 'pylist', 'arrow'] ``` String literal type for specifying data output formats. Valid values - "polars": Native Polars DataFrame format - "pandas": Pandas DataFrame with PyArrow extension arrays - "pydict": Python dictionary with column names as keys, lists as values - "pylist": Python list of dictionaries, each representing one row - "arrow": Apache Arrow Table format Used as input parameter for methods that can return data in multiple formats. ## DateType ``` DateType = _DateType() ``` Represents a date value. ## DoubleType ``` DoubleType = _DoubleType() ``` Represents a 64-bit floating-point number. ## FloatType ``` FloatType = _FloatType() ``` Represents a 32-bit floating-point number. ## FuzzySimilarityMethod ``` FuzzySimilarityMethod = Literal['indel', 'levenshtein', 'damerau_levenshtein', 'jaro_winkler', 'jaro', 'hamming'] ``` Type alias representing the supported fuzzy string similarity algorithms. These algorithms quantify the similarity or difference between two strings using various distance or similarity metrics: - "indel": Computes the Indel (Insertion-Deletion) distance, which counts only insertions and deletions needed to transform one string into another, excluding substitutions. This is equivalent to the Longest Common Subsequence (LCS) problem. Useful when character substitutions should not be considered as valid operations (e.g., DNA sequence alignment where only insertions/deletions occur). - "levenshtein": Computes the Levenshtein distance, which is the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. Suitable for general-purpose fuzzy matching where transpositions do not matter. - "damerau_levenshtein": An extension of Levenshtein distance that also accounts for transpositions of adjacent characters (e.g., "ab" β†’ "ba"). This metric is more accurate for real-world typos and keyboard errors. - "jaro": Measures similarity based on the number and order of common characters between two strings. It is particularly effective for short strings such as names. Returns a normalized score between 0 (no similarity) and 1 (exact match). - "jaro_winkler": A variant of the Jaro distance that gives more weight to common prefixes. Designed to improve accuracy on strings with shared beginnings (e.g., first names, surnames). - "hamming": Measures the number of differing characters between two strings of equal length. Only valid when both strings are the same length. It does not support insertions or deletionsβ€”only substitutions. Choose the method based on the type of expected variation (e.g., typos, transpositions, or structural changes). ## HtmlType ``` HtmlType = _HtmlType() ``` Represents a string containing raw HTML markup. ## IntegerType ``` IntegerType = _IntegerType() ``` Represents a signed integer value. ## JsonType ``` JsonType = _JsonType() ``` Represents a string containing JSON data. ## MarkdownType ``` MarkdownType = _MarkdownType() ``` Represents a string containing Markdown-formatted text. ## ModelQuantization ``` ModelQuantization = Literal['int4', 'int8', 'fp4', 'fp6', 'fp8', 'fp16', 'bf16', 'fp32', 'unknown'] ``` Type alias representing supported quantization formats for provider models. Common values: - "int4", "int8": Integer quantization for smaller, faster models. - "fp4", "fp6", "fp8": Low-precision floating point formats. - "fp16", "bf16": Half-precision formats commonly used on GPUs/TPUs. - "fp32": Full precision floating point. - "unknown": Provider did not specify a quantization. ## ProviderSort ``` ProviderSort = Literal['price', 'throughput', 'latency'] ``` Type alias representing provider sorting strategies used by OpenRouter routing. Valid values: - "price": Prefer providers with the lowest recent price. - "throughput": Prefer providers with the highest recent throughput. - "latency": Prefer providers with the lowest recent latency. ## SemanticSimilarityMetric ``` SemanticSimilarityMetric = Literal['cosine', 'l2', 'dot'] ``` Type alias representing supported semantic similarity metrics. Valid values: - "cosine": Cosine similarity, measures the cosine of the angle between two vectors. - "l2": Euclidean (L2) distance, measures the straight-line distance between two vectors. - "dot": Dot product similarity, the raw inner product of two vectors. These metrics are commonly used for comparing embedding vectors in semantic search and other similarity-based applications. ## StringType ``` StringType = _StringType() ``` Represents a UTF-8 encoded string value. ## StructuredOutputStrategy ``` StructuredOutputStrategy = Literal['prefer_tools', 'prefer_response_format'] ``` Type alias representing the strategy to use when a model supports both tool-calling and response-format-based structured outputs. Valid values: - "prefer_tools": Prefer tool/function calling with a JSON schema. - "prefer_response_format": Prefer response_format structured outputs. ## TimestampType ``` TimestampType = _TimestampType() ``` Represents a timestamp value. ## ArrayType Bases: `DataType` ``` flowchart TD fenic.core.ArrayType[ArrayType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.ArrayType click fenic.core.ArrayType href "" "fenic.core.ArrayType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a homogeneous variable-length array (list) of elements. Attributes: - **`element_type`** (`DataType`) – The data type of each element in the array. Create an array of strings ``` ArrayType(StringType) ArrayType(element_type=StringType) ``` ## BoundToolParam A bound tool parameter. A bound tool parameter is a parameter that has been bound to a specific, typed, `tool_param` usage within a Dataframe. ## ClassDefinition Bases: `BaseModel` ``` flowchart TD fenic.core.ClassDefinition[ClassDefinition] click fenic.core.ClassDefinition href "" "fenic.core.ClassDefinition" ``` Definition of a classification class with optional description. Used to define the available classes for semantic classification operations. The description helps the LLM understand what each class represents. ## ClassifyExample Bases: `BaseModel` ``` flowchart TD fenic.core.ClassifyExample[ClassifyExample] click fenic.core.ClassifyExample href "" "fenic.core.ClassifyExample" ``` A single semantic example for classification operations. Classify examples demonstrate the classification of an input string into a specific category string, used in a semantic.classify operation. ## ClassifyExampleCollection ``` ClassifyExampleCollection(examples: List[ExampleType] = None) ``` Bases: `BaseExampleCollection[ClassifyExample]` ``` flowchart TD fenic.core.ClassifyExampleCollection[ClassifyExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.ClassifyExampleCollection click fenic.core.ClassifyExampleCollection href "" "fenic.core.ClassifyExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of text-to-category examples for classification operations. Stores examples showing which category each input text should be assigned to. Each example contains an input string and its corresponding category label. Methods: - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[ExampleType] = None): """Initialize a collection of semantic examples. Args: examples: Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note: The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. """ self.examples: List[ExampleType] = [] if examples: for example in examples: self.create_example(example) ``` ### from_polars ``` from_polars(df: DataFrame) -> ClassifyExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> ClassifyExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column.""" collection = cls() if EXAMPLE_INPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_INPUT_KEY}' column" ) if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_INPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_INPUT_KEY}' column" ) if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) example = ClassifyExample( input=row[EXAMPLE_INPUT_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## ColumnField Represents a typed column in a DataFrame schema. A ColumnField defines the structure of a single column by specifying its name and data type. This is used as a building block for DataFrame schemas. Attributes: - **`name`** (`str`) – The name of the column. - **`data_type`** (`DataType`) – The data type of the column, as a DataType instance. ## DataType Bases: `ABC` ``` flowchart TD fenic.core.DataType[DataType] click fenic.core.DataType href "" "fenic.core.DataType" ``` Base class for all data types. You won't instantiate this class directly. Instead, use one of the concrete types like `StringType`, `ArrayType`, or `StructType`. Used for casting, type validation, and schema inference in the DataFrame API. ## DatasetMetadata Metadata for a dataset (table or view). Attributes: - **`schema`** (`Schema`) – The schema of the dataset. - **`description`** (`Optional[str]`) – The natural language description of the dataset's contents. ## DocumentPathType Bases: `_LogicalType` ``` flowchart TD fenic.core.DocumentPathType[DocumentPathType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.DocumentPathType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.DocumentPathType href "" "fenic.core.DocumentPathType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a a document's local (file system) or remote (URL) path. ## EmbeddingType Bases: `_LogicalType` ``` flowchart TD fenic.core.EmbeddingType[EmbeddingType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.EmbeddingType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.EmbeddingType href "" "fenic.core.EmbeddingType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a fixed-length embedding vector. Attributes: - **`dimensions`** (`int`) – The number of dimensions in the embedding vector. - **`embedding_model`** (`str`) – Name of the model used to generate the embedding. Create an embedding type for text-embedding-3-small ``` EmbeddingType(384, embedding_model="text-embedding-3-small") ``` ## JoinExample Bases: `BaseModel` ``` flowchart TD fenic.core.JoinExample[JoinExample] click fenic.core.JoinExample href "" "fenic.core.JoinExample" ``` A single semantic example for semantic join operations. Join examples demonstrate the evaluation of two input variables across different datasets against a specific condition, used in a semantic.join operation. ## JoinExampleCollection ``` JoinExampleCollection(examples: List[JoinExample] = None) ``` Bases: `BaseExampleCollection[JoinExample]` ``` flowchart TD fenic.core.JoinExampleCollection[JoinExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.JoinExampleCollection click fenic.core.JoinExampleCollection href "" "fenic.core.JoinExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of comparison examples for semantic join operations. Stores examples showing which pairs of values should be considered matches for joining data. Each example contains a left value, right value, and boolean output indicating whether they match. Initialize a collection of semantic join examples. Parameters: - **`examples`** (`List[JoinExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[JoinExample] = None): """Initialize a collection of semantic join examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: JoinExample) -> JoinExampleCollection ``` Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Parameters: - **`example`** (`JoinExample`) – The JoinExample to add. Returns: - `JoinExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: JoinExample) -> JoinExampleCollection: """Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Args: example: The JoinExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, JoinExample): raise InvalidExampleCollectionError( f"Expected example of type {JoinExample.__name__}, got {type(example).__name__}" ) # Convert to dict format for validation example_dict = { LEFT_ON_KEY: example.left_on, RIGHT_ON_KEY: example.right_on } example_num = len(self.examples) + 1 self._type_validator.process_example(example_dict, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> JoinExampleCollection ``` Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> JoinExampleCollection: """Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns.""" collection = cls() required_columns = [ LEFT_ON_KEY, RIGHT_ON_KEY, EXAMPLE_OUTPUT_KEY, ] for col in required_columns: if col not in df.columns: raise InvalidExampleCollectionError( f"Join Examples DataFrame missing required '{col}' column" ) for row in df.iter_rows(named=True): for col in required_columns: if row[col] is None: raise InvalidExampleCollectionError( f"Join Examples DataFrame contains null values in '{col}' column" ) example = JoinExample( left_on=row[LEFT_ON_KEY], right_on=row[RIGHT_ON_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## KeyPoints Bases: `BaseModel` ``` flowchart TD fenic.core.KeyPoints[KeyPoints] click fenic.core.KeyPoints href "" "fenic.core.KeyPoints" ``` Summary as a concise bulleted list. Each bullet should capture a distinct and essential idea, with a maximum number of points specified. Attributes: - **`max_points`** (`int`) – The maximum number of key points to include in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of key points. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of key points. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of key points.""" return self.max_points * 75 ``` ## LMMetrics ``` LMMetrics(num_uncached_input_tokens: int = 0, num_cached_input_tokens: int = 0, num_output_tokens: int = 0, cost: float = 0.0, num_requests: int = 0, num_reserved_output_tokens: int = 0) ``` Tracks language model usage metrics including token counts and costs. Attributes: - **`num_uncached_input_tokens`** (`int`) – Number of uncached tokens in the prompt/input. - **`num_cached_input_tokens`** (`int`) – Number of cached tokens in the prompt/input. - **`num_output_tokens`** (`int`) – Number of tokens in the completion/output (actual usage). - **`cost`** (`float`) – Total cost in USD for the LM API call. - **`num_requests`** (`int`) – Total number of LM API requests made. - **`num_reserved_output_tokens`** (`int`) – Output tokens debited from the TPM bucket at reservation time. Compare against num_output_tokens to measure reservation efficiency (actual / reserved β†’ 1 is tight). ## MapExample Bases: `BaseModel` ``` flowchart TD fenic.core.MapExample[MapExample] click fenic.core.MapExample href "" "fenic.core.MapExample" ``` A single semantic example for semantic mapping operations. Map examples demonstrate the transformation of input variables to a specific output string or structured model used in a semantic.map operation. ## MapExampleCollection ``` MapExampleCollection(examples: List[MapExample] = None) ``` Bases: `BaseExampleCollection[MapExample]` ``` flowchart TD fenic.core.MapExampleCollection[MapExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.MapExampleCollection click fenic.core.MapExampleCollection href "" "fenic.core.MapExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-output examples for semantic map operations. Stores examples that demonstrate how input data should be transformed into output text or structured data. Each example shows the expected output for a given set of input fields. Initialize a collection of semantic map examples. Parameters: - **`examples`** (`List[MapExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with output and input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[MapExample] = None): """Initialize a collection of semantic map examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: MapExample) -> MapExampleCollection ``` Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Parameters: - **`example`** (`MapExample`) – The MapExample to add. Returns: - `MapExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: MapExample) -> MapExampleCollection: """Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Args: example: The MapExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, MapExample): raise InvalidExampleCollectionError( f"Expected example of type {MapExample.__name__}, got {type(example).__name__}" ) # Validate output type consistency self._validate_single_example_output_type(example) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> MapExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> MapExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column.""" collection = cls() if EXAMPLE_OUTPUT_KEY not in df.columns: raise ValueError( f"Map Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise ValueError( "Map Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): input_dict = {col: row[col] for col in input_cols} example = MapExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## OperatorMetrics ``` OperatorMetrics(operator_id: str, num_output_rows: int = 0, execution_time_ms: float = 0.0, lm_metrics: LMMetrics = LMMetrics(), rm_metrics: RMMetrics = RMMetrics()) ``` Metrics for a single operator in the query execution plan. Attributes: - **`operator_id`** (`str`) – Unique identifier for the operator - **`num_output_rows`** (`int`) – Number of rows output by this operator - **`execution_time_ms`** (`float`) – Execution time in milliseconds - **`lm_metrics`** (`LMMetrics`) – Language model usage metrics for this operator ## Paragraph Bases: `BaseModel` ``` flowchart TD fenic.core.Paragraph[Paragraph] click fenic.core.Paragraph href "" "fenic.core.Paragraph" ``` Summary as a cohesive narrative. The summary should flow naturally and not exceed a specified maximum word count. Attributes: - **`max_words`** (`int`) – The maximum number of words allowed in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of words. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of words. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of words.""" return int(self.max_words * 1.5) ``` ## PredicateExample Bases: `BaseModel` ``` flowchart TD fenic.core.PredicateExample[PredicateExample] click fenic.core.PredicateExample href "" "fenic.core.PredicateExample" ``` A single semantic example for semantic predicate operations. Predicate examples demonstrate the evaluation of input variables against a specific condition, used in a semantic.predicate operation. ## PredicateExampleCollection ``` PredicateExampleCollection(examples: List[PredicateExample] = None) ``` Bases: `BaseExampleCollection[PredicateExample]` ``` flowchart TD fenic.core.PredicateExampleCollection[PredicateExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.PredicateExampleCollection click fenic.core.PredicateExampleCollection href "" "fenic.core.PredicateExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-to-boolean examples for predicate operations. Stores examples showing which inputs should evaluate to True or False based on some condition. Each example contains input fields and a boolean output indicating whether the condition holds. Initialize a collection of semantic predicate examples. Parameters: - **`examples`** (`List[PredicateExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[PredicateExample] = None): """Initialize a collection of semantic predicate examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: PredicateExample) -> PredicateExampleCollection ``` Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Parameters: - **`example`** (`PredicateExample`) – The PredicateExample to add. Returns: - `PredicateExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: PredicateExample) -> PredicateExampleCollection: """Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Args: example: The PredicateExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, PredicateExample): raise InvalidExampleCollectionError( f"Expected example of type {PredicateExample.__name__}, got {type(example).__name__}" ) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> PredicateExampleCollection ``` Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> PredicateExampleCollection: """Create collection from a Polars DataFrame.""" collection = cls() # Validate output column exists if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise InvalidExampleCollectionError( "Predicate Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) input_dict = {col: row[col] for col in input_cols if row[col] is not None} example = PredicateExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## QueryMetrics ``` QueryMetrics(execution_id: str, session_id: str, execution_time_ms: float = 0.0, num_output_rows: int = 0, total_lm_metrics: LMMetrics = LMMetrics(), total_rm_metrics: RMMetrics = RMMetrics(), end_ts: datetime = datetime.now(), _operator_metrics: Dict[str, OperatorMetrics] = dict(), _plan_repr: PhysicalPlanRepr = (lambda: PhysicalPlanRepr(operator_id='empty'))()) ``` Comprehensive metrics for an executed query. Includes overall statistics and detailed metrics for each operator in the execution plan. Attributes: - **`execution_id`** (`str`) – Unique identifier for this query execution - **`session_id`** (`str`) – Identifier for the session this query belongs to - **`execution_time_ms`** (`float`) – Total query execution time in milliseconds - **`num_output_rows`** (`int`) – Total number of rows returned by the query - **`total_lm_metrics`** (`LMMetrics`) – Aggregated language model metrics across all operators - **`end_ts`** (`datetime`) – Timestamp when query execution completed Methods: - **`get_execution_plan_details`** – Generate a formatted execution plan with detailed metrics. - **`get_summary`** – Summarize the query metrics in a single line. - **`to_dict`** – Convert QueryMetrics to a dictionary for table storage. ### start_ts ``` start_ts: datetime ``` Calculate start timestamp from end timestamp and execution time. ### get_execution_plan_details ``` get_execution_plan_details() -> str ``` Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: - **`str`** ( `str` ) – A formatted string showing the execution plan with metrics. Source code in `src/fenic/core/metrics.py` ``` def get_execution_plan_details(self) -> str: """Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: str: A formatted string showing the execution plan with metrics. """ def _format_node(node: PhysicalPlanRepr, indent: int = 1) -> str: op = self._operator_metrics[node.operator_id] indent_str = " " * indent details = [ f"{indent_str}{op.operator_id}", f"{indent_str} Output Rows: {op.num_output_rows:,}", f"{indent_str} Execution Time: {op.execution_time_ms:.2f}ms", ] if op.lm_metrics.cost > 0: details.extend( [ f"{indent_str} Language Model Usage: {op.lm_metrics.num_uncached_input_tokens:,} input tokens, {op.lm_metrics.num_cached_input_tokens:,} cached input tokens, {op.lm_metrics.num_output_tokens:,} output tokens ({op.lm_metrics.num_reserved_output_tokens:,} reserved)", f"{indent_str} Language Model Cost: ${op.lm_metrics.cost:.6f}", ] ) if op.rm_metrics.cost > 0: details.extend( [ f"{indent_str} Embedding Model Usage: {op.rm_metrics.num_input_tokens:,} input tokens", f"{indent_str} Embedding Model Cost: ${op.rm_metrics.cost:.6f}", ] ) return ( "\n".join(details) + "\n" + "".join(_format_node(child, indent + 1) for child in node.children) ) return f"Execution Plan\n{_format_node(self._plan_repr)}" ``` ### get_summary ``` get_summary() -> str ``` Summarize the query metrics in a single line. Returns: - **`str`** ( `str` ) – A concise summary of execution time, row count, and LM cost. Source code in `src/fenic/core/metrics.py` ``` def get_summary(self) -> str: """Summarize the query metrics in a single line. Returns: str: A concise summary of execution time, row count, and LM cost. """ return ( f"Query executed in {self.execution_time_ms:.2f}ms, " f"returned {self.num_output_rows:,} rows, " f"language model cost: ${self.total_lm_metrics.cost:.6f}, " f"embedding model cost: ${self.total_rm_metrics.cost:.6f}" ) ``` ### to_dict ``` to_dict() -> Dict[str, Any] ``` Convert QueryMetrics to a dictionary for table storage. Returns: - `Dict[str, Any]` – Dict containing all metrics fields suitable for database storage. Source code in `src/fenic/core/metrics.py` ``` def to_dict(self) -> Dict[str, Any]: """Convert QueryMetrics to a dictionary for table storage. Returns: Dict containing all metrics fields suitable for database storage. """ return { "execution_id": self.execution_id, "session_id": self.session_id, "execution_time_ms": self.execution_time_ms, "num_output_rows": self.num_output_rows, "start_ts": self.start_ts, "end_ts": self.end_ts, "total_lm_cost": self.total_lm_metrics.cost, "total_lm_uncached_input_tokens": self.total_lm_metrics.num_uncached_input_tokens, "total_lm_cached_input_tokens": self.total_lm_metrics.num_cached_input_tokens, "total_lm_output_tokens": self.total_lm_metrics.num_output_tokens, "total_lm_requests": self.total_lm_metrics.num_requests, "total_rm_cost": self.total_rm_metrics.cost, "total_rm_input_tokens": self.total_rm_metrics.num_input_tokens, "total_rm_requests": self.total_rm_metrics.num_requests, } ``` ## QueryResult ``` QueryResult(data: DataLike, metrics: QueryMetrics) ``` Container for query execution results and associated metadata. This dataclass bundles together the materialized data from a query execution along with metrics about the execution process. It provides a unified interface for accessing both the computed results and performance information. Attributes: - **`data`** (`DataLike`) – The materialized query results in the requested format. Can be any of the supported data types (Polars/Pandas DataFrame, Arrow Table, or Python dict/list structures). - **`metrics`** (`QueryMetrics`) – Execution metadata including timing information, memory usage, rows processed, and other performance metrics collected during query execution. Access query results and metrics ``` # Execute query and get results with metrics result = df.filter(col("age") > 25).collect("pandas") pandas_df = result.data # Access the Pandas DataFrame print(result.metrics.execution_time) # Access execution metrics print(result.metrics.rows_processed) # Access row count ``` Work with different data formats ``` # Get results in different formats polars_result = df.collect("polars") arrow_result = df.collect("arrow") dict_result = df.collect("pydict") # All contain the same data, different formats print(type(polars_result.data)) # print(type(arrow_result.data)) # print(type(dict_result.data)) # ``` Note The actual type of the `data` attribute depends on the format requested during collection. Use type checking or isinstance() if you need to handle the data differently based on its format. ## RMMetrics ``` RMMetrics(num_input_tokens: int = 0, num_requests: int = 0, cost: float = 0.0) ``` Tracks embedding model usage metrics including token counts and costs. Attributes: - **`num_input_tokens`** (`int`) – Number of tokens to embed - **`cost`** (`float`) – Total cost in USD to embed the tokens ## Schema Represents the schema of a DataFrame. A Schema defines the structure of a DataFrame by specifying an ordered collection of column fields. Each column field defines the name and data type of a column in the DataFrame. Attributes: - **`column_fields`** (`List[ColumnField]`) – An ordered list of ColumnField objects that define the structure of the DataFrame. Methods: - **`column_names`** – Get a list of all column names in the schema. ### column_names ``` column_names() -> List[str] ``` Get a list of all column names in the schema. Returns: - `List[str]` – A list of strings containing the names of all columns in the schema. Source code in `src/fenic/core/types/schema.py` ``` def column_names(self) -> List[str]: """Get a list of all column names in the schema. Returns: A list of strings containing the names of all columns in the schema. """ return [field.name for field in self.column_fields] ``` ## StructField A field in a StructType. Fields are nullable. Attributes: - **`name`** (`str`) – The name of the field. - **`data_type`** (`DataType`) – The data type of the field. ## StructType Bases: `DataType` ``` flowchart TD fenic.core.StructType[StructType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.StructType click fenic.core.StructType href "" "fenic.core.StructType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a struct (record) with named fields. Attributes: - **`fields`** – List of field definitions. Create a struct with name and age fields ``` StructType([ StructField("name", StringType), StructField("age", IntegerType), ]) ``` ## SystemTool A tool implemented as a regular Python function with explicit parameters. The function must be a `Callable[..., LogicalPlan]` (a function defined with `async def`). Collection/formatting is handled by the MCP generator wrapper. ## ToolParam Bases: `BaseModel` ``` flowchart TD fenic.core.ToolParam[ToolParam] click fenic.core.ToolParam href "" "fenic.core.ToolParam" ``` A parameter for a parameterized view tool. A parameter is a named value that can be passed to a tool. These are matched to the parameter names of the `tool_param` UnresolvedLiteralExpr expressions captured in the Logical Plan. Attributes: - **`name`** (`str`) – The name of the parameter. - **`description`** (`str`) – The description of the parameter. - **`allowed_values`** (`Optional[List[ToolParameterType]]`) – The allowed values for the parameter. - **`has_default`** (`bool`) – Whether the parameter has a default value. - **`default_value`** (`Optional[ToolParameterType]`) – The default value for the parameter. ### required ``` required: bool ``` Whether the parameter is required. Returns: - `bool` – True if the parameter is required, False otherwise. ## TranscriptType Bases: `_LogicalType` ``` flowchart TD fenic.core.TranscriptType[TranscriptType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.TranscriptType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.TranscriptType href "" "fenic.core.TranscriptType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a transcript in a specific format. ## UserDefinedTool A tool that has been bound to a specific Parameterized View. --- # fenic.core.error Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/error/ Fenic error hierarchy. Classes: - **`CatalogAlreadyExistsError`** – Catalog already exists. - **`CatalogError`** – Catalog and table management errors. - **`CatalogNotFoundError`** – Catalog doesn't exist. - **`CloudExecutionError`** – Errors during physical plan execution in a cloud session. - **`CloudSessionError`** – Cloud session lifecycle errors. - **`ColumnNotFoundError`** – Column doesn't exist. - **`ConfigurationError`** – Errors during session configuration or initialization. - **`DatabaseAlreadyExistsError`** – Database already exists. - **`DatabaseNotFoundError`** – Database doesn't exist. - **`ExecutionError`** – Errors during physical plan execution. - **`FenicError`** – Base exception for all fenic errors. - **`FileLoaderError`** – File loader error. - **`InternalError`** – Internal invariant violations. - **`InvalidExampleCollectionError`** – Exception raised when a semantic example collection is invalid. - **`LineageError`** – Errors during lineage traversal. - **`PlanError`** – Errors during logical plan construction and validation. - **`SessionError`** – Session lifecycle errors. - **`TableAlreadyExistsError`** – Table already exists. - **`TableNotFoundError`** – Table doesn't exist. - **`ToolAlreadyExistsError`** – Tool already exists. - **`ToolNotFoundError`** – Tool doesn't exist. - **`TypeMismatchError`** – Type validation errors. - **`UnsupportedFileTypeError`** – Unsupported file type error. - **`ValidationError`** – Invalid usage of public APIs or incorrect arguments. ## CatalogAlreadyExistsError ``` CatalogAlreadyExistsError(catalog_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.CatalogAlreadyExistsError[CatalogAlreadyExistsError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.CatalogAlreadyExistsError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.CatalogAlreadyExistsError href "" "fenic.core.error.CatalogAlreadyExistsError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Catalog already exists. Initialize a catalog already exists error. Parameters: - **`catalog_name`** (`str`) – The name of the catalog that already exists. Source code in `src/fenic/core/error.py` ``` def __init__(self, catalog_name: str): """Initialize a catalog already exists error. Args: catalog_name: The name of the catalog that already exists. """ super().__init__(f"Catalog '{catalog_name}' already exists") ``` ## CatalogError Bases: `FenicError` ``` flowchart TD fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Catalog and table management errors. ## CatalogNotFoundError ``` CatalogNotFoundError(catalog_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.CatalogNotFoundError[CatalogNotFoundError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.CatalogNotFoundError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.CatalogNotFoundError href "" "fenic.core.error.CatalogNotFoundError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Catalog doesn't exist. Initialize a catalog not found error. Parameters: - **`catalog_name`** (`str`) – The name of the catalog that was not found. Source code in `src/fenic/core/error.py` ``` def __init__(self, catalog_name: str): """Initialize a catalog not found error. Args: catalog_name: The name of the catalog that was not found. """ super().__init__(f"Catalog '{catalog_name}' does not exist") ``` ## CloudExecutionError ``` CloudExecutionError(error_message: str) ``` Bases: `ExecutionError` ``` flowchart TD fenic.core.error.CloudExecutionError[CloudExecutionError] fenic.core.error.ExecutionError[ExecutionError] fenic.core.error.FenicError[FenicError] fenic.core.error.ExecutionError --> fenic.core.error.CloudExecutionError fenic.core.error.FenicError --> fenic.core.error.ExecutionError click fenic.core.error.CloudExecutionError href "" "fenic.core.error.CloudExecutionError" click fenic.core.error.ExecutionError href "" "fenic.core.error.ExecutionError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Errors during physical plan execution in a cloud session. Initialize a cloud execution error. Parameters: - **`error_message`** (`str`) – The error message describing what went wrong. Source code in `src/fenic/core/error.py` ``` def __init__(self, error_message: str): """Initialize a cloud execution error. Args: error_message: The error message describing what went wrong. """ super().__init__( f"{error_message}. " "Please file a ticket with Typedef support." ) ``` ## CloudSessionError ``` CloudSessionError(error_message: str) ``` Bases: `SessionError` ``` flowchart TD fenic.core.error.CloudSessionError[CloudSessionError] fenic.core.error.SessionError[SessionError] fenic.core.error.ConfigurationError[ConfigurationError] fenic.core.error.FenicError[FenicError] fenic.core.error.SessionError --> fenic.core.error.CloudSessionError fenic.core.error.ConfigurationError --> fenic.core.error.SessionError fenic.core.error.FenicError --> fenic.core.error.ConfigurationError click fenic.core.error.CloudSessionError href "" "fenic.core.error.CloudSessionError" click fenic.core.error.SessionError href "" "fenic.core.error.SessionError" click fenic.core.error.ConfigurationError href "" "fenic.core.error.ConfigurationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Cloud session lifecycle errors. Initialize a cloud session error. Parameters: - **`error_message`** (`str`) – The error message describing what went wrong. Source code in `src/fenic/core/error.py` ``` def __init__(self, error_message: str): """Initialize a cloud session error. Args: error_message: The error message describing what went wrong. """ super().__init__( f"{error_message}. " "Please file a ticket with Typedef support." ) ``` ## ColumnNotFoundError ``` ColumnNotFoundError(column_name: str, available_columns: List[str]) ``` Bases: `PlanError` ``` flowchart TD fenic.core.error.ColumnNotFoundError[ColumnNotFoundError] fenic.core.error.PlanError[PlanError] fenic.core.error.FenicError[FenicError] fenic.core.error.PlanError --> fenic.core.error.ColumnNotFoundError fenic.core.error.FenicError --> fenic.core.error.PlanError click fenic.core.error.ColumnNotFoundError href "" "fenic.core.error.ColumnNotFoundError" click fenic.core.error.PlanError href "" "fenic.core.error.PlanError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Column doesn't exist. Initialize a column not found error. Parameters: - **`column_name`** (`str`) – The name of the column that was not found. - **`available_columns`** (`List[str]`) – List of column names that are available. Source code in `src/fenic/core/error.py` ``` def __init__(self, column_name: str, available_columns: List[str]): """Initialize a column not found error. Args: column_name: The name of the column that was not found. available_columns: List of column names that are available. """ super().__init__( f"Column '{column_name}' not found. " f"Available columns: {', '.join(sorted(available_columns))}" ) ``` ## ConfigurationError Bases: `FenicError` ``` flowchart TD fenic.core.error.ConfigurationError[ConfigurationError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.ConfigurationError click fenic.core.error.ConfigurationError href "" "fenic.core.error.ConfigurationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Errors during session configuration or initialization. ## DatabaseAlreadyExistsError ``` DatabaseAlreadyExistsError(database_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.DatabaseAlreadyExistsError[DatabaseAlreadyExistsError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.DatabaseAlreadyExistsError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.DatabaseAlreadyExistsError href "" "fenic.core.error.DatabaseAlreadyExistsError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Database already exists. Initialize a database already exists error. Parameters: - **`database_name`** (`str`) – The name of the database that already exists. Source code in `src/fenic/core/error.py` ``` def __init__(self, database_name: str): """Initialize a database already exists error. Args: database_name: The name of the database that already exists. """ super().__init__(f"Database '{database_name}' already exists") ``` ## DatabaseNotFoundError ``` DatabaseNotFoundError(database_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.DatabaseNotFoundError[DatabaseNotFoundError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.DatabaseNotFoundError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.DatabaseNotFoundError href "" "fenic.core.error.DatabaseNotFoundError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Database doesn't exist. Initialize a database not found error. Parameters: - **`database_name`** (`str`) – The name of the database that was not found. Source code in `src/fenic/core/error.py` ``` def __init__(self, database_name: str): """Initialize a database not found error. Args: database_name: The name of the database that was not found. """ super().__init__(f"Database '{database_name}' does not exist") ``` ## ExecutionError Bases: `FenicError` ``` flowchart TD fenic.core.error.ExecutionError[ExecutionError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.ExecutionError click fenic.core.error.ExecutionError href "" "fenic.core.error.ExecutionError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Errors during physical plan execution. ## FenicError Bases: `Exception` ``` flowchart TD fenic.core.error.FenicError[FenicError] click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Base exception for all fenic errors. ## FileLoaderError ``` FileLoaderError(exception: Exception) ``` Bases: `FenicError` ``` flowchart TD fenic.core.error.FileLoaderError[FileLoaderError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.FileLoaderError click fenic.core.error.FileLoaderError href "" "fenic.core.error.FileLoaderError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` File loader error. Initialize a file loader error. Parameters: - **`exception`** (`Exception`) – The exception that was raised. Source code in `src/fenic/core/error.py` ``` def __init__(self, exception: Exception): """Initialize a file loader error. Args: exception: The exception that was raised. """ super().__init__(f"File loader error: {exception}") ``` ## InternalError Bases: `FenicError` ``` flowchart TD fenic.core.error.InternalError[InternalError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.InternalError click fenic.core.error.InternalError href "" "fenic.core.error.InternalError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Internal invariant violations. ## InvalidExampleCollectionError Bases: `ValidationError` ``` flowchart TD fenic.core.error.InvalidExampleCollectionError[InvalidExampleCollectionError] fenic.core.error.ValidationError[ValidationError] fenic.core.error.FenicError[FenicError] fenic.core.error.ValidationError --> fenic.core.error.InvalidExampleCollectionError fenic.core.error.FenicError --> fenic.core.error.ValidationError click fenic.core.error.InvalidExampleCollectionError href "" "fenic.core.error.InvalidExampleCollectionError" click fenic.core.error.ValidationError href "" "fenic.core.error.ValidationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Exception raised when a semantic example collection is invalid. ## LineageError Bases: `FenicError` ``` flowchart TD fenic.core.error.LineageError[LineageError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.LineageError click fenic.core.error.LineageError href "" "fenic.core.error.LineageError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Errors during lineage traversal. ## PlanError Bases: `FenicError` ``` flowchart TD fenic.core.error.PlanError[PlanError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.PlanError click fenic.core.error.PlanError href "" "fenic.core.error.PlanError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Errors during logical plan construction and validation. ## SessionError Bases: `ConfigurationError` ``` flowchart TD fenic.core.error.SessionError[SessionError] fenic.core.error.ConfigurationError[ConfigurationError] fenic.core.error.FenicError[FenicError] fenic.core.error.ConfigurationError --> fenic.core.error.SessionError fenic.core.error.FenicError --> fenic.core.error.ConfigurationError click fenic.core.error.SessionError href "" "fenic.core.error.SessionError" click fenic.core.error.ConfigurationError href "" "fenic.core.error.ConfigurationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Session lifecycle errors. ## TableAlreadyExistsError ``` TableAlreadyExistsError(table_name: str, database: Optional[str] = None) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.TableAlreadyExistsError[TableAlreadyExistsError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.TableAlreadyExistsError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.TableAlreadyExistsError href "" "fenic.core.error.TableAlreadyExistsError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Table already exists. Initialize a table already exists error. Parameters: - **`table_name`** (`str`) – The name of the table that already exists. - **`database`** (`Optional[str]`, default: `None` ) – Optional name of the database containing the table. Source code in `src/fenic/core/error.py` ``` def __init__(self, table_name: str, database: Optional[str] = None): """Initialize a table already exists error. Args: table_name: The name of the table that already exists. database: Optional name of the database containing the table. """ if database: table_ref = f"{database}.{table_name}" else: table_ref = table_name super().__init__( f"Table '{table_ref}' already exists. " f"Use mode='overwrite' to replace the existing table." ) ``` ## TableNotFoundError ``` TableNotFoundError(table_name: str, database: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.TableNotFoundError[TableNotFoundError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.TableNotFoundError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.TableNotFoundError href "" "fenic.core.error.TableNotFoundError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Table doesn't exist. Initialize a table not found error. Parameters: - **`table_name`** (`str`) – The name of the table that was not found. - **`database`** (`str`) – The name of the database containing the table. Source code in `src/fenic/core/error.py` ``` def __init__(self, table_name: str, database: str): """Initialize a table not found error. Args: table_name: The name of the table that was not found. database: The name of the database containing the table. """ self.table_name = table_name self.database = database super().__init__(f"Table '{database}.{table_name}' does not exist") ``` ## ToolAlreadyExistsError ``` ToolAlreadyExistsError(tool_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.ToolAlreadyExistsError[ToolAlreadyExistsError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.ToolAlreadyExistsError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.ToolAlreadyExistsError href "" "fenic.core.error.ToolAlreadyExistsError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Tool already exists. Initialize a tool already exists error. Parameters: - **`tool_name`** (`str`) – The name of the tool that already exists. Source code in `src/fenic/core/error.py` ``` def __init__(self, tool_name: str): """Initialize a tool already exists error. Args: tool_name: The name of the tool that already exists. """ super().__init__(f"Tool '{tool_name}' already exists") ``` ## ToolNotFoundError ``` ToolNotFoundError(tool_name: str) ``` Bases: `CatalogError` ``` flowchart TD fenic.core.error.ToolNotFoundError[ToolNotFoundError] fenic.core.error.CatalogError[CatalogError] fenic.core.error.FenicError[FenicError] fenic.core.error.CatalogError --> fenic.core.error.ToolNotFoundError fenic.core.error.FenicError --> fenic.core.error.CatalogError click fenic.core.error.ToolNotFoundError href "" "fenic.core.error.ToolNotFoundError" click fenic.core.error.CatalogError href "" "fenic.core.error.CatalogError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Tool doesn't exist. Initialize a tool not found error. Parameters: - **`tool_name`** (`str`) – The name of the tool that was not found. Source code in `src/fenic/core/error.py` ``` def __init__(self, tool_name: str): """Initialize a tool not found error. Args: tool_name: The name of the tool that was not found. """ super().__init__(f"Tool '{tool_name}' does not exist") ``` ## TypeMismatchError ``` TypeMismatchError(expected: Union[DataType, List[DataType]], actual: DataType, context: str) ``` Bases: `PlanError` ``` flowchart TD fenic.core.error.TypeMismatchError[TypeMismatchError] fenic.core.error.PlanError[PlanError] fenic.core.error.FenicError[FenicError] fenic.core.error.PlanError --> fenic.core.error.TypeMismatchError fenic.core.error.FenicError --> fenic.core.error.PlanError click fenic.core.error.TypeMismatchError href "" "fenic.core.error.TypeMismatchError" click fenic.core.error.PlanError href "" "fenic.core.error.PlanError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Type validation errors. Initialize a type mismatch error. Parameters: - **`expected`** (`Union[DataType, List[DataType]]`) – The expected data type. - **`actual`** (`DataType`) – The actual data type that was found. - **`context`** (`str`) – Additional context about where the type mismatch occurred. Methods: - **`from_message`** – Create a TypeMismatchError from a message string. Source code in `src/fenic/core/error.py` ``` def __init__(self, expected: Union[DataType, List[DataType]], actual: DataType, context: str): """Initialize a type mismatch error. Args: expected: The expected data type. actual: The actual data type that was found. context: Additional context about where the type mismatch occurred. """ super().__init__(f"{context}: expected {expected}, got {actual}") ``` ### from_message ``` from_message(msg: str) -> TypeMismatchError ``` Create a TypeMismatchError from a message string. Parameters: - **`msg`** (`str`) – The error message. Returns: - `TypeMismatchError` – A new TypeMismatchError instance with the given message. Source code in `src/fenic/core/error.py` ``` @classmethod def from_message(cls, msg: str) -> TypeMismatchError: """Create a TypeMismatchError from a message string. Args: msg: The error message. Returns: A new TypeMismatchError instance with the given message. """ instance = cls.__new__(cls) # Bypass __init__ super(TypeMismatchError, instance).__init__(msg) return instance ``` ## UnsupportedFileTypeError ``` UnsupportedFileTypeError(file_type: DataType) ``` Bases: `FileLoaderError` ``` flowchart TD fenic.core.error.UnsupportedFileTypeError[UnsupportedFileTypeError] fenic.core.error.FileLoaderError[FileLoaderError] fenic.core.error.FenicError[FenicError] fenic.core.error.FileLoaderError --> fenic.core.error.UnsupportedFileTypeError fenic.core.error.FenicError --> fenic.core.error.FileLoaderError click fenic.core.error.UnsupportedFileTypeError href "" "fenic.core.error.UnsupportedFileTypeError" click fenic.core.error.FileLoaderError href "" "fenic.core.error.FileLoaderError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Unsupported file type error. Initialize a unsupported file type error. Parameters: - **`file_type`** (`DataType`) – The unsupported file type. Source code in `src/fenic/core/error.py` ``` def __init__(self, file_type: DataType): """Initialize a unsupported file type error. Args: file_type: The unsupported file type. """ super().__init__(f"Unsupported file type for: {file_type}") ``` ## ValidationError Bases: `FenicError` ``` flowchart TD fenic.core.error.ValidationError[ValidationError] fenic.core.error.FenicError[FenicError] fenic.core.error.FenicError --> fenic.core.error.ValidationError click fenic.core.error.ValidationError href "" "fenic.core.error.ValidationError" click fenic.core.error.FenicError href "" "fenic.core.error.FenicError" ``` Invalid usage of public APIs or incorrect arguments. --- # fenic.core.mcp Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/mcp/ MCP Tool Generation/FastMCP Server Management. Classes: - **`BoundToolParam`** – A bound tool parameter. - **`SystemTool`** – A tool implemented as a regular Python function with explicit parameters. - **`ToolParam`** – A parameter for a parameterized view tool. - **`UserDefinedTool`** – A tool that has been bound to a specific Parameterized View. ## BoundToolParam A bound tool parameter. A bound tool parameter is a parameter that has been bound to a specific, typed, `tool_param` usage within a Dataframe. ## SystemTool A tool implemented as a regular Python function with explicit parameters. The function must be a `Callable[..., LogicalPlan]` (a function defined with `async def`). Collection/formatting is handled by the MCP generator wrapper. ## ToolParam Bases: `BaseModel` ``` flowchart TD fenic.core.mcp.ToolParam[ToolParam] click fenic.core.mcp.ToolParam href "" "fenic.core.mcp.ToolParam" ``` A parameter for a parameterized view tool. A parameter is a named value that can be passed to a tool. These are matched to the parameter names of the `tool_param` UnresolvedLiteralExpr expressions captured in the Logical Plan. Attributes: - **`name`** (`str`) – The name of the parameter. - **`description`** (`str`) – The description of the parameter. - **`allowed_values`** (`Optional[List[ToolParameterType]]`) – The allowed values for the parameter. - **`has_default`** (`bool`) – Whether the parameter has a default value. - **`default_value`** (`Optional[ToolParameterType]`) – The default value for the parameter. ### required ``` required: bool ``` Whether the parameter is required. Returns: - `bool` – True if the parameter is required, False otherwise. ## UserDefinedTool A tool that has been bound to a specific Parameterized View. --- # fenic.core.mcp.types Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/mcp/types/ Exported Types related to Parameterized View/MCP Tool Generation. Classes: - **`BoundToolParam`** – A bound tool parameter. - **`SystemTool`** – A tool implemented as a regular Python function with explicit parameters. - **`ToolParam`** – A parameter for a parameterized view tool. - **`UserDefinedTool`** – A tool that has been bound to a specific Parameterized View. ## BoundToolParam A bound tool parameter. A bound tool parameter is a parameter that has been bound to a specific, typed, `tool_param` usage within a Dataframe. ## SystemTool A tool implemented as a regular Python function with explicit parameters. The function must be a `Callable[..., LogicalPlan]` (a function defined with `async def`). Collection/formatting is handled by the MCP generator wrapper. ## ToolParam Bases: `BaseModel` ``` flowchart TD fenic.core.mcp.types.ToolParam[ToolParam] click fenic.core.mcp.types.ToolParam href "" "fenic.core.mcp.types.ToolParam" ``` A parameter for a parameterized view tool. A parameter is a named value that can be passed to a tool. These are matched to the parameter names of the `tool_param` UnresolvedLiteralExpr expressions captured in the Logical Plan. Attributes: - **`name`** (`str`) – The name of the parameter. - **`description`** (`str`) – The description of the parameter. - **`allowed_values`** (`Optional[List[ToolParameterType]]`) – The allowed values for the parameter. - **`has_default`** (`bool`) – Whether the parameter has a default value. - **`default_value`** (`Optional[ToolParameterType]`) – The default value for the parameter. ### required ``` required: bool ``` Whether the parameter is required. Returns: - `bool` – True if the parameter is required, False otherwise. ## UserDefinedTool A tool that has been bound to a specific Parameterized View. --- # fenic.core.metrics Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/metrics/ Metrics tracking for query execution and model usage. This module defines classes for tracking various metrics during query execution, including language model usage, embedding model usage, operator performance, and overall query statistics. Classes: - **`LMMetrics`** – Tracks language model usage metrics including token counts and costs. - **`OperatorMetrics`** – Metrics for a single operator in the query execution plan. - **`PhysicalPlanRepr`** – Tree node representing the physical execution plan, used for pretty printing execution plan. - **`QueryMetrics`** – Comprehensive metrics for an executed query. - **`RMMetrics`** – Tracks embedding model usage metrics including token counts and costs. ## LMMetrics ``` LMMetrics(num_uncached_input_tokens: int = 0, num_cached_input_tokens: int = 0, num_output_tokens: int = 0, cost: float = 0.0, num_requests: int = 0, num_reserved_output_tokens: int = 0) ``` Tracks language model usage metrics including token counts and costs. Attributes: - **`num_uncached_input_tokens`** (`int`) – Number of uncached tokens in the prompt/input. - **`num_cached_input_tokens`** (`int`) – Number of cached tokens in the prompt/input. - **`num_output_tokens`** (`int`) – Number of tokens in the completion/output (actual usage). - **`cost`** (`float`) – Total cost in USD for the LM API call. - **`num_requests`** (`int`) – Total number of LM API requests made. - **`num_reserved_output_tokens`** (`int`) – Output tokens debited from the TPM bucket at reservation time. Compare against num_output_tokens to measure reservation efficiency (actual / reserved β†’ 1 is tight). ## OperatorMetrics ``` OperatorMetrics(operator_id: str, num_output_rows: int = 0, execution_time_ms: float = 0.0, lm_metrics: LMMetrics = LMMetrics(), rm_metrics: RMMetrics = RMMetrics()) ``` Metrics for a single operator in the query execution plan. Attributes: - **`operator_id`** (`str`) – Unique identifier for the operator - **`num_output_rows`** (`int`) – Number of rows output by this operator - **`execution_time_ms`** (`float`) – Execution time in milliseconds - **`lm_metrics`** (`LMMetrics`) – Language model usage metrics for this operator ## PhysicalPlanRepr ``` PhysicalPlanRepr(operator_id: str, children: List[PhysicalPlanRepr] = list()) ``` Tree node representing the physical execution plan, used for pretty printing execution plan. ## QueryMetrics ``` QueryMetrics(execution_id: str, session_id: str, execution_time_ms: float = 0.0, num_output_rows: int = 0, total_lm_metrics: LMMetrics = LMMetrics(), total_rm_metrics: RMMetrics = RMMetrics(), end_ts: datetime = datetime.now(), _operator_metrics: Dict[str, OperatorMetrics] = dict(), _plan_repr: PhysicalPlanRepr = (lambda: PhysicalPlanRepr(operator_id='empty'))()) ``` Comprehensive metrics for an executed query. Includes overall statistics and detailed metrics for each operator in the execution plan. Attributes: - **`execution_id`** (`str`) – Unique identifier for this query execution - **`session_id`** (`str`) – Identifier for the session this query belongs to - **`execution_time_ms`** (`float`) – Total query execution time in milliseconds - **`num_output_rows`** (`int`) – Total number of rows returned by the query - **`total_lm_metrics`** (`LMMetrics`) – Aggregated language model metrics across all operators - **`end_ts`** (`datetime`) – Timestamp when query execution completed Methods: - **`get_execution_plan_details`** – Generate a formatted execution plan with detailed metrics. - **`get_summary`** – Summarize the query metrics in a single line. - **`to_dict`** – Convert QueryMetrics to a dictionary for table storage. ### start_ts ``` start_ts: datetime ``` Calculate start timestamp from end timestamp and execution time. ### get_execution_plan_details ``` get_execution_plan_details() -> str ``` Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: - **`str`** ( `str` ) – A formatted string showing the execution plan with metrics. Source code in `src/fenic/core/metrics.py` ``` def get_execution_plan_details(self) -> str: """Generate a formatted execution plan with detailed metrics. Produces a hierarchical representation of the query execution plan, including performance metrics and language model usage for each operator. Returns: str: A formatted string showing the execution plan with metrics. """ def _format_node(node: PhysicalPlanRepr, indent: int = 1) -> str: op = self._operator_metrics[node.operator_id] indent_str = " " * indent details = [ f"{indent_str}{op.operator_id}", f"{indent_str} Output Rows: {op.num_output_rows:,}", f"{indent_str} Execution Time: {op.execution_time_ms:.2f}ms", ] if op.lm_metrics.cost > 0: details.extend( [ f"{indent_str} Language Model Usage: {op.lm_metrics.num_uncached_input_tokens:,} input tokens, {op.lm_metrics.num_cached_input_tokens:,} cached input tokens, {op.lm_metrics.num_output_tokens:,} output tokens ({op.lm_metrics.num_reserved_output_tokens:,} reserved)", f"{indent_str} Language Model Cost: ${op.lm_metrics.cost:.6f}", ] ) if op.rm_metrics.cost > 0: details.extend( [ f"{indent_str} Embedding Model Usage: {op.rm_metrics.num_input_tokens:,} input tokens", f"{indent_str} Embedding Model Cost: ${op.rm_metrics.cost:.6f}", ] ) return ( "\n".join(details) + "\n" + "".join(_format_node(child, indent + 1) for child in node.children) ) return f"Execution Plan\n{_format_node(self._plan_repr)}" ``` ### get_summary ``` get_summary() -> str ``` Summarize the query metrics in a single line. Returns: - **`str`** ( `str` ) – A concise summary of execution time, row count, and LM cost. Source code in `src/fenic/core/metrics.py` ``` def get_summary(self) -> str: """Summarize the query metrics in a single line. Returns: str: A concise summary of execution time, row count, and LM cost. """ return ( f"Query executed in {self.execution_time_ms:.2f}ms, " f"returned {self.num_output_rows:,} rows, " f"language model cost: ${self.total_lm_metrics.cost:.6f}, " f"embedding model cost: ${self.total_rm_metrics.cost:.6f}" ) ``` ### to_dict ``` to_dict() -> Dict[str, Any] ``` Convert QueryMetrics to a dictionary for table storage. Returns: - `Dict[str, Any]` – Dict containing all metrics fields suitable for database storage. Source code in `src/fenic/core/metrics.py` ``` def to_dict(self) -> Dict[str, Any]: """Convert QueryMetrics to a dictionary for table storage. Returns: Dict containing all metrics fields suitable for database storage. """ return { "execution_id": self.execution_id, "session_id": self.session_id, "execution_time_ms": self.execution_time_ms, "num_output_rows": self.num_output_rows, "start_ts": self.start_ts, "end_ts": self.end_ts, "total_lm_cost": self.total_lm_metrics.cost, "total_lm_uncached_input_tokens": self.total_lm_metrics.num_uncached_input_tokens, "total_lm_cached_input_tokens": self.total_lm_metrics.num_cached_input_tokens, "total_lm_output_tokens": self.total_lm_metrics.num_output_tokens, "total_lm_requests": self.total_lm_metrics.num_requests, "total_rm_cost": self.total_rm_metrics.cost, "total_rm_input_tokens": self.total_rm_metrics.num_input_tokens, "total_rm_requests": self.total_rm_metrics.num_requests, } ``` ## RMMetrics ``` RMMetrics(num_input_tokens: int = 0, num_requests: int = 0, cost: float = 0.0) ``` Tracks embedding model usage metrics including token counts and costs. Attributes: - **`num_input_tokens`** (`int`) – Number of tokens to embed - **`cost`** (`float`) – Total cost in USD to embed the tokens --- # fenic.core.types Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/ Schema module for defining and manipulating DataFrame schemas. Classes: - **`ArrayType`** – A type representing a homogeneous variable-length array (list) of elements. - **`ClassDefinition`** – Definition of a classification class with optional description. - **`ClassifyExample`** – A single semantic example for classification operations. - **`ClassifyExampleCollection`** – Collection of text-to-category examples for classification operations. - **`ColumnField`** – Represents a typed column in a DataFrame schema. - **`DataType`** – Base class for all data types. - **`DatasetMetadata`** – Metadata for a dataset (table or view). - **`DocumentPathType`** – Represents a string containing a a document's local (file system) or remote (URL) path. - **`EmbeddingType`** – A type representing a fixed-length embedding vector. - **`JoinExample`** – A single semantic example for semantic join operations. - **`JoinExampleCollection`** – Collection of comparison examples for semantic join operations. - **`KeyPoints`** – Summary as a concise bulleted list. - **`MapExample`** – A single semantic example for semantic mapping operations. - **`MapExampleCollection`** – Collection of input-output examples for semantic map operations. - **`Paragraph`** – Summary as a cohesive narrative. - **`PredicateExample`** – A single semantic example for semantic predicate operations. - **`PredicateExampleCollection`** – Collection of input-to-boolean examples for predicate operations. - **`QueryResult`** – Container for query execution results and associated metadata. - **`Schema`** – Represents the schema of a DataFrame. - **`StructField`** – A field in a StructType. Fields are nullable. - **`StructType`** – A type representing a struct (record) with named fields. - **`TranscriptType`** – Represents a string containing a transcript in a specific format. Attributes: - **`BooleanType`** – Represents a boolean value. (True/False) - **`BranchSide`** – Type alias representing the side of a branch in a lineage graph. - **`DataCollection`** – Type alias representing provider data collection policies. - **`DataLike`** – Union type representing any supported data format for both input and output operations. - **`DataLikeType`** – String literal type for specifying data output formats. - **`DateType`** – Represents a date value. - **`DoubleType`** – Represents a 64-bit floating-point number. - **`FloatType`** – Represents a 32-bit floating-point number. - **`FuzzySimilarityMethod`** – Type alias representing the supported fuzzy string similarity algorithms. - **`HtmlType`** – Represents a string containing raw HTML markup. - **`IntegerType`** – Represents a signed integer value. - **`JsonType`** – Represents a string containing JSON data. - **`MarkdownType`** – Represents a string containing Markdown-formatted text. - **`ModelQuantization`** – Type alias representing supported quantization formats for provider models. - **`ProviderSort`** – Type alias representing provider sorting strategies used by OpenRouter routing. - **`SemanticSimilarityMetric`** – Type alias representing supported semantic similarity metrics. - **`StringType`** – Represents a UTF-8 encoded string value. - **`StructuredOutputStrategy`** – Type alias representing the strategy to use when a model supports both - **`TimestampType`** – Represents a timestamp value. ## BooleanType ``` BooleanType = _BooleanType() ``` Represents a boolean value. (True/False) ## BranchSide ``` BranchSide = Literal['left', 'right'] ``` Type alias representing the side of a branch in a lineage graph. Valid values: - "left": The left branch of a join. - "right": The right branch of a join. ## DataCollection ``` DataCollection = Literal['allow', 'deny'] ``` Type alias representing provider data collection policies. Valid values: - "allow": Permit providers that may retain or train on prompts non-transiently. - "deny": Restrict to providers that do not collect/store user data. ## DataLike ``` DataLike = Union[pl.DataFrame, pd.DataFrame, Dict[str, List[Any]], List[Dict[str, Any]], pa.Table] ``` Union type representing any supported data format for both input and output operations. This type encompasses all possible data structures that can be: 1. Used as input when creating DataFrames 2. Returned as output from query results Supported formats - pl.DataFrame: Native Polars DataFrame with efficient columnar storage - pd.DataFrame: Pandas DataFrame, optionally with PyArrow extension arrays - Dict[str, List[Any]]: Column-oriented dictionary where: - Keys are column names (str) - Values are lists containing all values for that column - List[Dict[str, Any]]: Row-oriented list where: - Each element is a dictionary representing one row - Dictionary keys are column names, values are cell values - pa.Table: Apache Arrow Table with columnar memory layout Usage - Input: Used in create_dataframe() to accept data in various formats - Output: Used in QueryResult.data to return results in requested format The specific type returned depends on the DataLikeType format specified when collecting query results. ## DataLikeType ``` DataLikeType = Literal['polars', 'pandas', 'pydict', 'pylist', 'arrow'] ``` String literal type for specifying data output formats. Valid values - "polars": Native Polars DataFrame format - "pandas": Pandas DataFrame with PyArrow extension arrays - "pydict": Python dictionary with column names as keys, lists as values - "pylist": Python list of dictionaries, each representing one row - "arrow": Apache Arrow Table format Used as input parameter for methods that can return data in multiple formats. ## DateType ``` DateType = _DateType() ``` Represents a date value. ## DoubleType ``` DoubleType = _DoubleType() ``` Represents a 64-bit floating-point number. ## FloatType ``` FloatType = _FloatType() ``` Represents a 32-bit floating-point number. ## FuzzySimilarityMethod ``` FuzzySimilarityMethod = Literal['indel', 'levenshtein', 'damerau_levenshtein', 'jaro_winkler', 'jaro', 'hamming'] ``` Type alias representing the supported fuzzy string similarity algorithms. These algorithms quantify the similarity or difference between two strings using various distance or similarity metrics: - "indel": Computes the Indel (Insertion-Deletion) distance, which counts only insertions and deletions needed to transform one string into another, excluding substitutions. This is equivalent to the Longest Common Subsequence (LCS) problem. Useful when character substitutions should not be considered as valid operations (e.g., DNA sequence alignment where only insertions/deletions occur). - "levenshtein": Computes the Levenshtein distance, which is the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. Suitable for general-purpose fuzzy matching where transpositions do not matter. - "damerau_levenshtein": An extension of Levenshtein distance that also accounts for transpositions of adjacent characters (e.g., "ab" β†’ "ba"). This metric is more accurate for real-world typos and keyboard errors. - "jaro": Measures similarity based on the number and order of common characters between two strings. It is particularly effective for short strings such as names. Returns a normalized score between 0 (no similarity) and 1 (exact match). - "jaro_winkler": A variant of the Jaro distance that gives more weight to common prefixes. Designed to improve accuracy on strings with shared beginnings (e.g., first names, surnames). - "hamming": Measures the number of differing characters between two strings of equal length. Only valid when both strings are the same length. It does not support insertions or deletionsβ€”only substitutions. Choose the method based on the type of expected variation (e.g., typos, transpositions, or structural changes). ## HtmlType ``` HtmlType = _HtmlType() ``` Represents a string containing raw HTML markup. ## IntegerType ``` IntegerType = _IntegerType() ``` Represents a signed integer value. ## JsonType ``` JsonType = _JsonType() ``` Represents a string containing JSON data. ## MarkdownType ``` MarkdownType = _MarkdownType() ``` Represents a string containing Markdown-formatted text. ## ModelQuantization ``` ModelQuantization = Literal['int4', 'int8', 'fp4', 'fp6', 'fp8', 'fp16', 'bf16', 'fp32', 'unknown'] ``` Type alias representing supported quantization formats for provider models. Common values: - "int4", "int8": Integer quantization for smaller, faster models. - "fp4", "fp6", "fp8": Low-precision floating point formats. - "fp16", "bf16": Half-precision formats commonly used on GPUs/TPUs. - "fp32": Full precision floating point. - "unknown": Provider did not specify a quantization. ## ProviderSort ``` ProviderSort = Literal['price', 'throughput', 'latency'] ``` Type alias representing provider sorting strategies used by OpenRouter routing. Valid values: - "price": Prefer providers with the lowest recent price. - "throughput": Prefer providers with the highest recent throughput. - "latency": Prefer providers with the lowest recent latency. ## SemanticSimilarityMetric ``` SemanticSimilarityMetric = Literal['cosine', 'l2', 'dot'] ``` Type alias representing supported semantic similarity metrics. Valid values: - "cosine": Cosine similarity, measures the cosine of the angle between two vectors. - "l2": Euclidean (L2) distance, measures the straight-line distance between two vectors. - "dot": Dot product similarity, the raw inner product of two vectors. These metrics are commonly used for comparing embedding vectors in semantic search and other similarity-based applications. ## StringType ``` StringType = _StringType() ``` Represents a UTF-8 encoded string value. ## StructuredOutputStrategy ``` StructuredOutputStrategy = Literal['prefer_tools', 'prefer_response_format'] ``` Type alias representing the strategy to use when a model supports both tool-calling and response-format-based structured outputs. Valid values: - "prefer_tools": Prefer tool/function calling with a JSON schema. - "prefer_response_format": Prefer response_format structured outputs. ## TimestampType ``` TimestampType = _TimestampType() ``` Represents a timestamp value. ## ArrayType Bases: `DataType` ``` flowchart TD fenic.core.types.ArrayType[ArrayType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.types.ArrayType click fenic.core.types.ArrayType href "" "fenic.core.types.ArrayType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a homogeneous variable-length array (list) of elements. Attributes: - **`element_type`** (`DataType`) – The data type of each element in the array. Create an array of strings ``` ArrayType(StringType) ArrayType(element_type=StringType) ``` ## ClassDefinition Bases: `BaseModel` ``` flowchart TD fenic.core.types.ClassDefinition[ClassDefinition] click fenic.core.types.ClassDefinition href "" "fenic.core.types.ClassDefinition" ``` Definition of a classification class with optional description. Used to define the available classes for semantic classification operations. The description helps the LLM understand what each class represents. ## ClassifyExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.ClassifyExample[ClassifyExample] click fenic.core.types.ClassifyExample href "" "fenic.core.types.ClassifyExample" ``` A single semantic example for classification operations. Classify examples demonstrate the classification of an input string into a specific category string, used in a semantic.classify operation. ## ClassifyExampleCollection ``` ClassifyExampleCollection(examples: List[ExampleType] = None) ``` Bases: `BaseExampleCollection[ClassifyExample]` ``` flowchart TD fenic.core.types.ClassifyExampleCollection[ClassifyExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.ClassifyExampleCollection click fenic.core.types.ClassifyExampleCollection href "" "fenic.core.types.ClassifyExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of text-to-category examples for classification operations. Stores examples showing which category each input text should be assigned to. Each example contains an input string and its corresponding category label. Methods: - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[ExampleType] = None): """Initialize a collection of semantic examples. Args: examples: Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note: The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. """ self.examples: List[ExampleType] = [] if examples: for example in examples: self.create_example(example) ``` ### from_polars ``` from_polars(df: DataFrame) -> ClassifyExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> ClassifyExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column.""" collection = cls() if EXAMPLE_INPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_INPUT_KEY}' column" ) if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_INPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_INPUT_KEY}' column" ) if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) example = ClassifyExample( input=row[EXAMPLE_INPUT_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## ColumnField Represents a typed column in a DataFrame schema. A ColumnField defines the structure of a single column by specifying its name and data type. This is used as a building block for DataFrame schemas. Attributes: - **`name`** (`str`) – The name of the column. - **`data_type`** (`DataType`) – The data type of the column, as a DataType instance. ## DataType Bases: `ABC` ``` flowchart TD fenic.core.types.DataType[DataType] click fenic.core.types.DataType href "" "fenic.core.types.DataType" ``` Base class for all data types. You won't instantiate this class directly. Instead, use one of the concrete types like `StringType`, `ArrayType`, or `StructType`. Used for casting, type validation, and schema inference in the DataFrame API. ## DatasetMetadata Metadata for a dataset (table or view). Attributes: - **`schema`** (`Schema`) – The schema of the dataset. - **`description`** (`Optional[str]`) – The natural language description of the dataset's contents. ## DocumentPathType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.DocumentPathType[DocumentPathType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.DocumentPathType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.DocumentPathType href "" "fenic.core.types.DocumentPathType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a a document's local (file system) or remote (URL) path. ## EmbeddingType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.EmbeddingType[EmbeddingType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.EmbeddingType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.EmbeddingType href "" "fenic.core.types.EmbeddingType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a fixed-length embedding vector. Attributes: - **`dimensions`** (`int`) – The number of dimensions in the embedding vector. - **`embedding_model`** (`str`) – Name of the model used to generate the embedding. Create an embedding type for text-embedding-3-small ``` EmbeddingType(384, embedding_model="text-embedding-3-small") ``` ## JoinExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.JoinExample[JoinExample] click fenic.core.types.JoinExample href "" "fenic.core.types.JoinExample" ``` A single semantic example for semantic join operations. Join examples demonstrate the evaluation of two input variables across different datasets against a specific condition, used in a semantic.join operation. ## JoinExampleCollection ``` JoinExampleCollection(examples: List[JoinExample] = None) ``` Bases: `BaseExampleCollection[JoinExample]` ``` flowchart TD fenic.core.types.JoinExampleCollection[JoinExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.JoinExampleCollection click fenic.core.types.JoinExampleCollection href "" "fenic.core.types.JoinExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of comparison examples for semantic join operations. Stores examples showing which pairs of values should be considered matches for joining data. Each example contains a left value, right value, and boolean output indicating whether they match. Initialize a collection of semantic join examples. Parameters: - **`examples`** (`List[JoinExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[JoinExample] = None): """Initialize a collection of semantic join examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: JoinExample) -> JoinExampleCollection ``` Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Parameters: - **`example`** (`JoinExample`) – The JoinExample to add. Returns: - `JoinExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: JoinExample) -> JoinExampleCollection: """Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Args: example: The JoinExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, JoinExample): raise InvalidExampleCollectionError( f"Expected example of type {JoinExample.__name__}, got {type(example).__name__}" ) # Convert to dict format for validation example_dict = { LEFT_ON_KEY: example.left_on, RIGHT_ON_KEY: example.right_on } example_num = len(self.examples) + 1 self._type_validator.process_example(example_dict, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> JoinExampleCollection ``` Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> JoinExampleCollection: """Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns.""" collection = cls() required_columns = [ LEFT_ON_KEY, RIGHT_ON_KEY, EXAMPLE_OUTPUT_KEY, ] for col in required_columns: if col not in df.columns: raise InvalidExampleCollectionError( f"Join Examples DataFrame missing required '{col}' column" ) for row in df.iter_rows(named=True): for col in required_columns: if row[col] is None: raise InvalidExampleCollectionError( f"Join Examples DataFrame contains null values in '{col}' column" ) example = JoinExample( left_on=row[LEFT_ON_KEY], right_on=row[RIGHT_ON_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## KeyPoints Bases: `BaseModel` ``` flowchart TD fenic.core.types.KeyPoints[KeyPoints] click fenic.core.types.KeyPoints href "" "fenic.core.types.KeyPoints" ``` Summary as a concise bulleted list. Each bullet should capture a distinct and essential idea, with a maximum number of points specified. Attributes: - **`max_points`** (`int`) – The maximum number of key points to include in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of key points. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of key points. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of key points.""" return self.max_points * 75 ``` ## MapExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.MapExample[MapExample] click fenic.core.types.MapExample href "" "fenic.core.types.MapExample" ``` A single semantic example for semantic mapping operations. Map examples demonstrate the transformation of input variables to a specific output string or structured model used in a semantic.map operation. ## MapExampleCollection ``` MapExampleCollection(examples: List[MapExample] = None) ``` Bases: `BaseExampleCollection[MapExample]` ``` flowchart TD fenic.core.types.MapExampleCollection[MapExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.MapExampleCollection click fenic.core.types.MapExampleCollection href "" "fenic.core.types.MapExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-output examples for semantic map operations. Stores examples that demonstrate how input data should be transformed into output text or structured data. Each example shows the expected output for a given set of input fields. Initialize a collection of semantic map examples. Parameters: - **`examples`** (`List[MapExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with output and input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[MapExample] = None): """Initialize a collection of semantic map examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: MapExample) -> MapExampleCollection ``` Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Parameters: - **`example`** (`MapExample`) – The MapExample to add. Returns: - `MapExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: MapExample) -> MapExampleCollection: """Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Args: example: The MapExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, MapExample): raise InvalidExampleCollectionError( f"Expected example of type {MapExample.__name__}, got {type(example).__name__}" ) # Validate output type consistency self._validate_single_example_output_type(example) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> MapExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> MapExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column.""" collection = cls() if EXAMPLE_OUTPUT_KEY not in df.columns: raise ValueError( f"Map Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise ValueError( "Map Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): input_dict = {col: row[col] for col in input_cols} example = MapExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## Paragraph Bases: `BaseModel` ``` flowchart TD fenic.core.types.Paragraph[Paragraph] click fenic.core.types.Paragraph href "" "fenic.core.types.Paragraph" ``` Summary as a cohesive narrative. The summary should flow naturally and not exceed a specified maximum word count. Attributes: - **`max_words`** (`int`) – The maximum number of words allowed in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of words. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of words. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of words.""" return int(self.max_words * 1.5) ``` ## PredicateExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.PredicateExample[PredicateExample] click fenic.core.types.PredicateExample href "" "fenic.core.types.PredicateExample" ``` A single semantic example for semantic predicate operations. Predicate examples demonstrate the evaluation of input variables against a specific condition, used in a semantic.predicate operation. ## PredicateExampleCollection ``` PredicateExampleCollection(examples: List[PredicateExample] = None) ``` Bases: `BaseExampleCollection[PredicateExample]` ``` flowchart TD fenic.core.types.PredicateExampleCollection[PredicateExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.PredicateExampleCollection click fenic.core.types.PredicateExampleCollection href "" "fenic.core.types.PredicateExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-to-boolean examples for predicate operations. Stores examples showing which inputs should evaluate to True or False based on some condition. Each example contains input fields and a boolean output indicating whether the condition holds. Initialize a collection of semantic predicate examples. Parameters: - **`examples`** (`List[PredicateExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[PredicateExample] = None): """Initialize a collection of semantic predicate examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: PredicateExample) -> PredicateExampleCollection ``` Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Parameters: - **`example`** (`PredicateExample`) – The PredicateExample to add. Returns: - `PredicateExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: PredicateExample) -> PredicateExampleCollection: """Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Args: example: The PredicateExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, PredicateExample): raise InvalidExampleCollectionError( f"Expected example of type {PredicateExample.__name__}, got {type(example).__name__}" ) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> PredicateExampleCollection ``` Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> PredicateExampleCollection: """Create collection from a Polars DataFrame.""" collection = cls() # Validate output column exists if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise InvalidExampleCollectionError( "Predicate Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) input_dict = {col: row[col] for col in input_cols if row[col] is not None} example = PredicateExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## QueryResult ``` QueryResult(data: DataLike, metrics: QueryMetrics) ``` Container for query execution results and associated metadata. This dataclass bundles together the materialized data from a query execution along with metrics about the execution process. It provides a unified interface for accessing both the computed results and performance information. Attributes: - **`data`** (`DataLike`) – The materialized query results in the requested format. Can be any of the supported data types (Polars/Pandas DataFrame, Arrow Table, or Python dict/list structures). - **`metrics`** (`QueryMetrics`) – Execution metadata including timing information, memory usage, rows processed, and other performance metrics collected during query execution. Access query results and metrics ``` # Execute query and get results with metrics result = df.filter(col("age") > 25).collect("pandas") pandas_df = result.data # Access the Pandas DataFrame print(result.metrics.execution_time) # Access execution metrics print(result.metrics.rows_processed) # Access row count ``` Work with different data formats ``` # Get results in different formats polars_result = df.collect("polars") arrow_result = df.collect("arrow") dict_result = df.collect("pydict") # All contain the same data, different formats print(type(polars_result.data)) # print(type(arrow_result.data)) # print(type(dict_result.data)) # ``` Note The actual type of the `data` attribute depends on the format requested during collection. Use type checking or isinstance() if you need to handle the data differently based on its format. ## Schema Represents the schema of a DataFrame. A Schema defines the structure of a DataFrame by specifying an ordered collection of column fields. Each column field defines the name and data type of a column in the DataFrame. Attributes: - **`column_fields`** (`List[ColumnField]`) – An ordered list of ColumnField objects that define the structure of the DataFrame. Methods: - **`column_names`** – Get a list of all column names in the schema. ### column_names ``` column_names() -> List[str] ``` Get a list of all column names in the schema. Returns: - `List[str]` – A list of strings containing the names of all columns in the schema. Source code in `src/fenic/core/types/schema.py` ``` def column_names(self) -> List[str]: """Get a list of all column names in the schema. Returns: A list of strings containing the names of all columns in the schema. """ return [field.name for field in self.column_fields] ``` ## StructField A field in a StructType. Fields are nullable. Attributes: - **`name`** (`str`) – The name of the field. - **`data_type`** (`DataType`) – The data type of the field. ## StructType Bases: `DataType` ``` flowchart TD fenic.core.types.StructType[StructType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.types.StructType click fenic.core.types.StructType href "" "fenic.core.types.StructType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a struct (record) with named fields. Attributes: - **`fields`** – List of field definitions. Create a struct with name and age fields ``` StructType([ StructField("name", StringType), StructField("age", IntegerType), ]) ``` ## TranscriptType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.TranscriptType[TranscriptType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.TranscriptType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.TranscriptType href "" "fenic.core.types.TranscriptType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a transcript in a specific format. --- # fenic.core.types.classify Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/classify/ Definition of a ClassDefinition class with optional description for semantic.classify. Classes: - **`ClassDefinition`** – Definition of a classification class with optional description. ## ClassDefinition Bases: `BaseModel` ``` flowchart TD fenic.core.types.classify.ClassDefinition[ClassDefinition] click fenic.core.types.classify.ClassDefinition href "" "fenic.core.types.classify.ClassDefinition" ``` Definition of a classification class with optional description. Used to define the available classes for semantic classification operations. The description helps the LLM understand what each class represents. --- # fenic.core.types.datatypes Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/datatypes/ Core data type definitions for the DataFrame API. This module defines the type system used throughout the DataFrame API. It includes: - Base classes for all data types - Primitive types (string, integer, float, etc.) - Composite types (arrays, structs) - Specialized types (embeddings, markdown, etc.) Classes: - **`ArrayType`** – A type representing a homogeneous variable-length array (list) of elements. - **`DataType`** – Base class for all data types. - **`DocumentPathType`** – Represents a string containing a a document's local (file system) or remote (URL) path. - **`EmbeddingType`** – A type representing a fixed-length embedding vector. - **`StructField`** – A field in a StructType. Fields are nullable. - **`StructType`** – A type representing a struct (record) with named fields. - **`TranscriptType`** – Represents a string containing a transcript in a specific format. Attributes: - **`BooleanType`** – Represents a boolean value. (True/False) - **`DateType`** – Represents a date value. - **`DoubleType`** – Represents a 64-bit floating-point number. - **`FloatType`** – Represents a 32-bit floating-point number. - **`HtmlType`** – Represents a string containing raw HTML markup. - **`IntegerType`** – Represents a signed integer value. - **`JsonType`** – Represents a string containing JSON data. - **`MarkdownType`** – Represents a string containing Markdown-formatted text. - **`StringType`** – Represents a UTF-8 encoded string value. - **`TimestampType`** – Represents a timestamp value. ## BooleanType ``` BooleanType = _BooleanType() ``` Represents a boolean value. (True/False) ## DateType ``` DateType = _DateType() ``` Represents a date value. ## DoubleType ``` DoubleType = _DoubleType() ``` Represents a 64-bit floating-point number. ## FloatType ``` FloatType = _FloatType() ``` Represents a 32-bit floating-point number. ## HtmlType ``` HtmlType = _HtmlType() ``` Represents a string containing raw HTML markup. ## IntegerType ``` IntegerType = _IntegerType() ``` Represents a signed integer value. ## JsonType ``` JsonType = _JsonType() ``` Represents a string containing JSON data. ## MarkdownType ``` MarkdownType = _MarkdownType() ``` Represents a string containing Markdown-formatted text. ## StringType ``` StringType = _StringType() ``` Represents a UTF-8 encoded string value. ## TimestampType ``` TimestampType = _TimestampType() ``` Represents a timestamp value. ## ArrayType Bases: `DataType` ``` flowchart TD fenic.core.types.datatypes.ArrayType[ArrayType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes.ArrayType click fenic.core.types.datatypes.ArrayType href "" "fenic.core.types.datatypes.ArrayType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a homogeneous variable-length array (list) of elements. Attributes: - **`element_type`** (`DataType`) – The data type of each element in the array. Create an array of strings ``` ArrayType(StringType) ArrayType(element_type=StringType) ``` ## DataType Bases: `ABC` ``` flowchart TD fenic.core.types.datatypes.DataType[DataType] click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Base class for all data types. You won't instantiate this class directly. Instead, use one of the concrete types like `StringType`, `ArrayType`, or `StructType`. Used for casting, type validation, and schema inference in the DataFrame API. ## DocumentPathType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.datatypes.DocumentPathType[DocumentPathType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.datatypes.DocumentPathType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.datatypes.DocumentPathType href "" "fenic.core.types.datatypes.DocumentPathType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a a document's local (file system) or remote (URL) path. ## EmbeddingType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.datatypes.EmbeddingType[EmbeddingType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.datatypes.EmbeddingType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.datatypes.EmbeddingType href "" "fenic.core.types.datatypes.EmbeddingType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a fixed-length embedding vector. Attributes: - **`dimensions`** (`int`) – The number of dimensions in the embedding vector. - **`embedding_model`** (`str`) – Name of the model used to generate the embedding. Create an embedding type for text-embedding-3-small ``` EmbeddingType(384, embedding_model="text-embedding-3-small") ``` ## StructField A field in a StructType. Fields are nullable. Attributes: - **`name`** (`str`) – The name of the field. - **`data_type`** (`DataType`) – The data type of the field. ## StructType Bases: `DataType` ``` flowchart TD fenic.core.types.datatypes.StructType[StructType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes.StructType click fenic.core.types.datatypes.StructType href "" "fenic.core.types.datatypes.StructType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` A type representing a struct (record) with named fields. Attributes: - **`fields`** – List of field definitions. Create a struct with name and age fields ``` StructType([ StructField("name", StringType), StructField("age", IntegerType), ]) ``` ## TranscriptType Bases: `_LogicalType` ``` flowchart TD fenic.core.types.datatypes.TranscriptType[TranscriptType] fenic.core.types.datatypes._LogicalType[_LogicalType] fenic.core.types.datatypes.DataType[DataType] fenic.core.types.datatypes._LogicalType --> fenic.core.types.datatypes.TranscriptType fenic.core.types.datatypes.DataType --> fenic.core.types.datatypes._LogicalType click fenic.core.types.datatypes.TranscriptType href "" "fenic.core.types.datatypes.TranscriptType" click fenic.core.types.datatypes._LogicalType href "" "fenic.core.types.datatypes._LogicalType" click fenic.core.types.datatypes.DataType href "" "fenic.core.types.datatypes.DataType" ``` Represents a string containing a transcript in a specific format. --- # fenic.core.types.enums Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/enums/ Enums used in the DataFrame API. Classes: - **`CacheBackend`** – Cache backend implementations. Attributes: - **`BranchSide`** – Type alias representing the side of a branch in a lineage graph. - **`DateTimeUnit`** – Type alias representing the supported date part or time unit to be used in date functions. - **`DocContentType`** – Type alias representing the type of document content. - **`FuzzySimilarityMethod`** – Type alias representing the supported fuzzy string similarity algorithms. - **`JoinType`** – Type alias representing supported join types. - **`SemanticSimilarityMetric`** – Type alias representing supported semantic similarity metrics. - **`StringCasingType`** – Type alias representing the type of string casing. - **`StripCharsSide`** – Type alias representing the side of a string to strip characters from. - **`TranscriptFormatType`** – Type alias representing supported transcript formats. ## BranchSide ``` BranchSide = Literal['left', 'right'] ``` Type alias representing the side of a branch in a lineage graph. Valid values: - "left": The left branch of a join. - "right": The right branch of a join. ## DateTimeUnit ``` DateTimeUnit = Literal['year', 'month', 'day', 'hour', 'minute', 'second', 'millisecond'] ``` Type alias representing the supported date part or time unit to be used in date functions. Valid values: - "year": Year. - "month": Month. - "day": Day. - "hour": Hour. - "minute": Minute. - "second": Second. - "millisecond": Millisecond. ## DocContentType ``` DocContentType = Literal['markdown', 'json', 'pdf'] ``` Type alias representing the type of document content. Valid values: - "markdown": Markdown format. - "json": JSON format. - "pdf": PDF format. ## FuzzySimilarityMethod ``` FuzzySimilarityMethod = Literal['indel', 'levenshtein', 'damerau_levenshtein', 'jaro_winkler', 'jaro', 'hamming'] ``` Type alias representing the supported fuzzy string similarity algorithms. These algorithms quantify the similarity or difference between two strings using various distance or similarity metrics: - "indel": Computes the Indel (Insertion-Deletion) distance, which counts only insertions and deletions needed to transform one string into another, excluding substitutions. This is equivalent to the Longest Common Subsequence (LCS) problem. Useful when character substitutions should not be considered as valid operations (e.g., DNA sequence alignment where only insertions/deletions occur). - "levenshtein": Computes the Levenshtein distance, which is the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. Suitable for general-purpose fuzzy matching where transpositions do not matter. - "damerau_levenshtein": An extension of Levenshtein distance that also accounts for transpositions of adjacent characters (e.g., "ab" β†’ "ba"). This metric is more accurate for real-world typos and keyboard errors. - "jaro": Measures similarity based on the number and order of common characters between two strings. It is particularly effective for short strings such as names. Returns a normalized score between 0 (no similarity) and 1 (exact match). - "jaro_winkler": A variant of the Jaro distance that gives more weight to common prefixes. Designed to improve accuracy on strings with shared beginnings (e.g., first names, surnames). - "hamming": Measures the number of differing characters between two strings of equal length. Only valid when both strings are the same length. It does not support insertions or deletionsβ€”only substitutions. Choose the method based on the type of expected variation (e.g., typos, transpositions, or structural changes). ## JoinType ``` JoinType = Literal['inner', 'full', 'left', 'right', 'cross'] ``` Type alias representing supported join types. Valid values: - "inner": Inner join, returns only rows that have matching values in both tables. - "outer": Outer join, returns all rows from both tables, filling missing values with nulls. - "left": Left join, returns all rows from the left table and matching rows from the right table. - "right": Right join, returns all rows from the right table and matching rows from the left table. - "cross": Cross join, returns the Cartesian product of the two tables. ## SemanticSimilarityMetric ``` SemanticSimilarityMetric = Literal['cosine', 'l2', 'dot'] ``` Type alias representing supported semantic similarity metrics. Valid values: - "cosine": Cosine similarity, measures the cosine of the angle between two vectors. - "l2": Euclidean (L2) distance, measures the straight-line distance between two vectors. - "dot": Dot product similarity, the raw inner product of two vectors. These metrics are commonly used for comparing embedding vectors in semantic search and other similarity-based applications. ## StringCasingType ``` StringCasingType = Literal['lower', 'upper', 'title'] ``` Type alias representing the type of string casing. Valid values: - "lower": Convert to lowercase. - "upper": Convert to uppercase. - "title": Convert to title case. ## StripCharsSide ``` StripCharsSide = Literal['left', 'right', 'both'] ``` Type alias representing the side of a string to strip characters from. Valid values: - "left": Strip characters from the left side. - "right": Strip characters from the right side. - "both": Strip characters from both sides. ## TranscriptFormatType ``` TranscriptFormatType = Literal['srt', 'generic', 'webvtt'] ``` Type alias representing supported transcript formats. Valid values: - "srt": SubRip Subtitle format with indexed entries and timestamp ranges - "generic": Conversation transcript format with speaker names and timestamps - "webvtt": Web Video Text Tracks format with speaker names and timestamps All formats are parsed into a unified schema with fields: index, speaker, start_time, end_time, duration, content, format. ## CacheBackend Bases: `str`, `Enum` ``` flowchart TD fenic.core.types.enums.CacheBackend[CacheBackend] click fenic.core.types.enums.CacheBackend href "" "fenic.core.types.enums.CacheBackend" ``` Cache backend implementations. Attributes: - **`LOCAL`** – Local persistent cache backend. --- # fenic.core.types.provider_routing Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/provider_routing/ Provider routing shared types and constants. Defines the allowed provider sorting strategies for OpenRouter provider routing. Attributes: - **`DataCollection`** – Type alias representing provider data collection policies. - **`ModelQuantization`** – Type alias representing supported quantization formats for provider models. - **`ProviderSort`** – Type alias representing provider sorting strategies used by OpenRouter routing. - **`StructuredOutputStrategy`** – Type alias representing the strategy to use when a model supports both ## DataCollection ``` DataCollection = Literal['allow', 'deny'] ``` Type alias representing provider data collection policies. Valid values: - "allow": Permit providers that may retain or train on prompts non-transiently. - "deny": Restrict to providers that do not collect/store user data. ## ModelQuantization ``` ModelQuantization = Literal['int4', 'int8', 'fp4', 'fp6', 'fp8', 'fp16', 'bf16', 'fp32', 'unknown'] ``` Type alias representing supported quantization formats for provider models. Common values: - "int4", "int8": Integer quantization for smaller, faster models. - "fp4", "fp6", "fp8": Low-precision floating point formats. - "fp16", "bf16": Half-precision formats commonly used on GPUs/TPUs. - "fp32": Full precision floating point. - "unknown": Provider did not specify a quantization. ## ProviderSort ``` ProviderSort = Literal['price', 'throughput', 'latency'] ``` Type alias representing provider sorting strategies used by OpenRouter routing. Valid values: - "price": Prefer providers with the lowest recent price. - "throughput": Prefer providers with the highest recent throughput. - "latency": Prefer providers with the lowest recent latency. ## StructuredOutputStrategy ``` StructuredOutputStrategy = Literal['prefer_tools', 'prefer_response_format'] ``` Type alias representing the strategy to use when a model supports both tool-calling and response-format-based structured outputs. Valid values: - "prefer_tools": Prefer tool/function calling with a JSON schema. - "prefer_response_format": Prefer response_format structured outputs. --- # fenic.core.types.query_result Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/query_result/ QueryResult class and related types. Classes: - **`QueryResult`** – Container for query execution results and associated metadata. Attributes: - **`DataLike`** – Union type representing any supported data format for both input and output operations. - **`DataLikeType`** – String literal type for specifying data output formats. ## DataLike ``` DataLike = Union[pl.DataFrame, pd.DataFrame, Dict[str, List[Any]], List[Dict[str, Any]], pa.Table] ``` Union type representing any supported data format for both input and output operations. This type encompasses all possible data structures that can be: 1. Used as input when creating DataFrames 2. Returned as output from query results Supported formats - pl.DataFrame: Native Polars DataFrame with efficient columnar storage - pd.DataFrame: Pandas DataFrame, optionally with PyArrow extension arrays - Dict[str, List[Any]]: Column-oriented dictionary where: - Keys are column names (str) - Values are lists containing all values for that column - List[Dict[str, Any]]: Row-oriented list where: - Each element is a dictionary representing one row - Dictionary keys are column names, values are cell values - pa.Table: Apache Arrow Table with columnar memory layout Usage - Input: Used in create_dataframe() to accept data in various formats - Output: Used in QueryResult.data to return results in requested format The specific type returned depends on the DataLikeType format specified when collecting query results. ## DataLikeType ``` DataLikeType = Literal['polars', 'pandas', 'pydict', 'pylist', 'arrow'] ``` String literal type for specifying data output formats. Valid values - "polars": Native Polars DataFrame format - "pandas": Pandas DataFrame with PyArrow extension arrays - "pydict": Python dictionary with column names as keys, lists as values - "pylist": Python list of dictionaries, each representing one row - "arrow": Apache Arrow Table format Used as input parameter for methods that can return data in multiple formats. ## QueryResult ``` QueryResult(data: DataLike, metrics: QueryMetrics) ``` Container for query execution results and associated metadata. This dataclass bundles together the materialized data from a query execution along with metrics about the execution process. It provides a unified interface for accessing both the computed results and performance information. Attributes: - **`data`** (`DataLike`) – The materialized query results in the requested format. Can be any of the supported data types (Polars/Pandas DataFrame, Arrow Table, or Python dict/list structures). - **`metrics`** (`QueryMetrics`) – Execution metadata including timing information, memory usage, rows processed, and other performance metrics collected during query execution. Access query results and metrics ``` # Execute query and get results with metrics result = df.filter(col("age") > 25).collect("pandas") pandas_df = result.data # Access the Pandas DataFrame print(result.metrics.execution_time) # Access execution metrics print(result.metrics.rows_processed) # Access row count ``` Work with different data formats ``` # Get results in different formats polars_result = df.collect("polars") arrow_result = df.collect("arrow") dict_result = df.collect("pydict") # All contain the same data, different formats print(type(polars_result.data)) # print(type(arrow_result.data)) # print(type(dict_result.data)) # ``` Note The actual type of the `data` attribute depends on the format requested during collection. Use type checking or isinstance() if you need to handle the data differently based on its format. --- # fenic.core.types.schema Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/schema/ Schema definitions for DataFrame structures. This module provides classes for defining and working with DataFrame schemas. It includes ColumnField for individual column definitions and Schema for complete DataFrame structure definitions. Classes: - **`ColumnField`** – Represents a typed column in a DataFrame schema. - **`DatasetMetadata`** – Metadata for a dataset (table or view). - **`Schema`** – Represents the schema of a DataFrame. ## ColumnField Represents a typed column in a DataFrame schema. A ColumnField defines the structure of a single column by specifying its name and data type. This is used as a building block for DataFrame schemas. Attributes: - **`name`** (`str`) – The name of the column. - **`data_type`** (`DataType`) – The data type of the column, as a DataType instance. ## DatasetMetadata Metadata for a dataset (table or view). Attributes: - **`schema`** (`Schema`) – The schema of the dataset. - **`description`** (`Optional[str]`) – The natural language description of the dataset's contents. ## Schema Represents the schema of a DataFrame. A Schema defines the structure of a DataFrame by specifying an ordered collection of column fields. Each column field defines the name and data type of a column in the DataFrame. Attributes: - **`column_fields`** (`List[ColumnField]`) – An ordered list of ColumnField objects that define the structure of the DataFrame. Methods: - **`column_names`** – Get a list of all column names in the schema. ### column_names ``` column_names() -> List[str] ``` Get a list of all column names in the schema. Returns: - `List[str]` – A list of strings containing the names of all columns in the schema. Source code in `src/fenic/core/types/schema.py` ``` def column_names(self) -> List[str]: """Get a list of all column names in the schema. Returns: A list of strings containing the names of all columns in the schema. """ return [field.name for field in self.column_fields] ``` --- # fenic.core.types.semantic Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/semantic/ Types used to configure model selection for semantic functions. Classes: - **`ModelAlias`** – A combination of a model name and a required profile for that model. ## ModelAlias Bases: `BaseModel` ``` flowchart TD fenic.core.types.semantic.ModelAlias[ModelAlias] click fenic.core.types.semantic.ModelAlias href "" "fenic.core.types.semantic.ModelAlias" ``` A combination of a model name and a required profile for that model. Model aliases are used to select a specific model to use in a semantic operation. Both the model name and profile must be specified when creating a ModelAlias. Attributes: - **`name`** (`str`) – The name of the model. - **`profile`** (`str`) – The name of a profile configuration to use for the model. Example ``` model_alias = ModelAlias(name="o4-mini", profile="low") ``` --- # fenic.core.types.semantic_examples Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/semantic_examples/ Module for handling semantic examples in query processing. This module provides classes and utilities for building, managing, and validating semantic examples used in query processing. Classes: - **`BaseExampleCollection`** – Abstract base class for all semantic example collections. - **`ClassifyExample`** – A single semantic example for classification operations. - **`ClassifyExampleCollection`** – Collection of text-to-category examples for classification operations. - **`JoinExample`** – A single semantic example for semantic join operations. - **`JoinExampleCollection`** – Collection of comparison examples for semantic join operations. - **`MapExample`** – A single semantic example for semantic mapping operations. - **`MapExampleCollection`** – Collection of input-output examples for semantic map operations. - **`PredicateExample`** – A single semantic example for semantic predicate operations. - **`PredicateExampleCollection`** – Collection of input-to-boolean examples for predicate operations. ## BaseExampleCollection ``` BaseExampleCollection(examples: List[ExampleType] = None) ``` Bases: `ABC`, `Generic[ExampleType]` ``` flowchart TD fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Abstract base class for all semantic example collections. Semantic examples demonstrate the expected input-output relationship for a given task, helping guide language models to produce consistent and accurate responses. Each example consists of inputs and the corresponding expected output. These examples are particularly valuable for: - Demonstrating the expected reasoning pattern - Showing correct output formats - Handling edge cases through demonstration - Improving model performance without changing the underlying model Initialize a collection of semantic examples. Parameters: - **`examples`** (`List[ExampleType]`, default: `None` ) – Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection. - **`from_pandas`** – Create a collection from a Pandas DataFrame. - **`from_polars`** – Create a collection from a Polars DataFrame. - **`to_pandas`** – Convert the collection to a Pandas DataFrame. - **`to_polars`** – Convert the collection to a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[ExampleType] = None): """Initialize a collection of semantic examples. Args: examples: Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note: The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. """ self.examples: List[ExampleType] = [] if examples: for example in examples: self.create_example(example) ``` ### create_example ``` create_example(example: ExampleType) -> BaseExampleCollection ``` Create an example in the collection. example: The semantic example to add. Must be an instance of the collection's example_class. Returns: - `BaseExampleCollection` – Self for method chaining. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: ExampleType) -> BaseExampleCollection: """Create an example in the collection. Args: example: The semantic example to add. Must be an instance of the collection's example_class. Returns: Self for method chaining. """ if not isinstance(example, self.example_class): raise InvalidExampleCollectionError( f"Expected example of type {self.example_class.__name__}, got {type(example).__name__}" ) self.examples.append(example) return self ``` ### from_pandas ``` from_pandas(df: DataFrame) -> BaseExampleCollection ``` Create a collection from a Pandas DataFrame. Parameters: - **`df`** (`DataFrame`) – The Pandas DataFrame containing example data. The specific column structure requirements depend on the concrete collection type. Returns: - `BaseExampleCollection` – A new example collection populated with examples from the DataFrame. Raises: - `InvalidExampleCollectionError` – If the DataFrame's structure doesn't match the expected format for this collection type. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_pandas(cls, df: pd.DataFrame) -> BaseExampleCollection: """Create a collection from a Pandas DataFrame. Args: df: The Pandas DataFrame containing example data. The specific column structure requirements depend on the concrete collection type. Returns: A new example collection populated with examples from the DataFrame. Raises: InvalidExampleCollectionError: If the DataFrame's structure doesn't match the expected format for this collection type. """ polars_df = pl.from_pandas(data=df) return cls.from_polars(polars_df) ``` ### from_polars ``` from_polars(df: DataFrame) -> BaseExampleCollection ``` Create a collection from a Polars DataFrame. Parameters: - **`df`** (`DataFrame`) – The Polars DataFrame containing example data. The specific column structure requirements depend on the concrete collection type. Returns: - `BaseExampleCollection` – A new example collection populated with examples from the DataFrame. Raises: - `InvalidExampleCollectionError` – If the DataFrame's structure doesn't match the expected format for this collection type. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod @abstractmethod def from_polars(cls, df: pl.DataFrame) -> BaseExampleCollection: """Create a collection from a Polars DataFrame. Args: df: The Polars DataFrame containing example data. The specific column structure requirements depend on the concrete collection type. Returns: A new example collection populated with examples from the DataFrame. Raises: InvalidExampleCollectionError: If the DataFrame's structure doesn't match the expected format for this collection type. """ pass ``` ### to_pandas ``` to_pandas() -> pd.DataFrame ``` Convert the collection to a Pandas DataFrame. Returns: - `DataFrame` – A Pandas DataFrame representing the collection's examples. - `DataFrame` – Returns an empty DataFrame if the collection contains no examples. Source code in `src/fenic/core/types/semantic_examples.py` ``` def to_pandas(self) -> pd.DataFrame: """Convert the collection to a Pandas DataFrame. Returns: A Pandas DataFrame representing the collection's examples. Returns an empty DataFrame if the collection contains no examples. """ rows = self._as_df_input() return pd.DataFrame(rows) ``` ### to_polars ``` to_polars() -> pl.DataFrame ``` Convert the collection to a Polars DataFrame. Returns: - `DataFrame` – A Polars DataFrame representing the collection's examples. - `DataFrame` – Returns an empty DataFrame if the collection contains no examples. Source code in `src/fenic/core/types/semantic_examples.py` ``` def to_polars(self) -> pl.DataFrame: """Convert the collection to a Polars DataFrame. Returns: A Polars DataFrame representing the collection's examples. Returns an empty DataFrame if the collection contains no examples. """ rows = self._as_df_input() return pl.DataFrame(rows) ``` ## ClassifyExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.semantic_examples.ClassifyExample[ClassifyExample] click fenic.core.types.semantic_examples.ClassifyExample href "" "fenic.core.types.semantic_examples.ClassifyExample" ``` A single semantic example for classification operations. Classify examples demonstrate the classification of an input string into a specific category string, used in a semantic.classify operation. ## ClassifyExampleCollection ``` ClassifyExampleCollection(examples: List[ExampleType] = None) ``` Bases: `BaseExampleCollection[ClassifyExample]` ``` flowchart TD fenic.core.types.semantic_examples.ClassifyExampleCollection[ClassifyExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.semantic_examples.ClassifyExampleCollection click fenic.core.types.semantic_examples.ClassifyExampleCollection href "" "fenic.core.types.semantic_examples.ClassifyExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of text-to-category examples for classification operations. Stores examples showing which category each input text should be assigned to. Each example contains an input string and its corresponding category label. Methods: - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[ExampleType] = None): """Initialize a collection of semantic examples. Args: examples: Optional list of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Note: The examples list is initialized as empty if no examples are provided. Each example in the provided list will be processed through create_example() to ensure proper formatting and validation. """ self.examples: List[ExampleType] = [] if examples: for example in examples: self.create_example(example) ``` ### from_polars ``` from_polars(df: DataFrame) -> ClassifyExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> ClassifyExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and an 'input' column.""" collection = cls() if EXAMPLE_INPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_INPUT_KEY}' column" ) if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Classify Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_INPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_INPUT_KEY}' column" ) if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Classify Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) example = ClassifyExample( input=row[EXAMPLE_INPUT_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## JoinExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.semantic_examples.JoinExample[JoinExample] click fenic.core.types.semantic_examples.JoinExample href "" "fenic.core.types.semantic_examples.JoinExample" ``` A single semantic example for semantic join operations. Join examples demonstrate the evaluation of two input variables across different datasets against a specific condition, used in a semantic.join operation. ## JoinExampleCollection ``` JoinExampleCollection(examples: List[JoinExample] = None) ``` Bases: `BaseExampleCollection[JoinExample]` ``` flowchart TD fenic.core.types.semantic_examples.JoinExampleCollection[JoinExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.semantic_examples.JoinExampleCollection click fenic.core.types.semantic_examples.JoinExampleCollection href "" "fenic.core.types.semantic_examples.JoinExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of comparison examples for semantic join operations. Stores examples showing which pairs of values should be considered matches for joining data. Each example contains a left value, right value, and boolean output indicating whether they match. Initialize a collection of semantic join examples. Parameters: - **`examples`** (`List[JoinExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[JoinExample] = None): """Initialize a collection of semantic join examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: JoinExample) -> JoinExampleCollection ``` Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Parameters: - **`example`** (`JoinExample`) – The JoinExample to add. Returns: - `JoinExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: JoinExample) -> JoinExampleCollection: """Create an example in the collection with type validation. Validates that left_on and right_on values have consistent types across examples. The first example establishes the types and cannot have None values. Subsequent examples must have matching types but can have None values. Args: example: The JoinExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, JoinExample): raise InvalidExampleCollectionError( f"Expected example of type {JoinExample.__name__}, got {type(example).__name__}" ) # Convert to dict format for validation example_dict = { LEFT_ON_KEY: example.left_on, RIGHT_ON_KEY: example.right_on } example_num = len(self.examples) + 1 self._type_validator.process_example(example_dict, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> JoinExampleCollection ``` Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> JoinExampleCollection: """Create collection from a Polars DataFrame. Must have 'left_on', 'right_on', and 'output' columns.""" collection = cls() required_columns = [ LEFT_ON_KEY, RIGHT_ON_KEY, EXAMPLE_OUTPUT_KEY, ] for col in required_columns: if col not in df.columns: raise InvalidExampleCollectionError( f"Join Examples DataFrame missing required '{col}' column" ) for row in df.iter_rows(named=True): for col in required_columns: if row[col] is None: raise InvalidExampleCollectionError( f"Join Examples DataFrame contains null values in '{col}' column" ) example = JoinExample( left_on=row[LEFT_ON_KEY], right_on=row[RIGHT_ON_KEY], output=row[EXAMPLE_OUTPUT_KEY], ) collection.create_example(example) return collection ``` ## MapExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.semantic_examples.MapExample[MapExample] click fenic.core.types.semantic_examples.MapExample href "" "fenic.core.types.semantic_examples.MapExample" ``` A single semantic example for semantic mapping operations. Map examples demonstrate the transformation of input variables to a specific output string or structured model used in a semantic.map operation. ## MapExampleCollection ``` MapExampleCollection(examples: List[MapExample] = None) ``` Bases: `BaseExampleCollection[MapExample]` ``` flowchart TD fenic.core.types.semantic_examples.MapExampleCollection[MapExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.semantic_examples.MapExampleCollection click fenic.core.types.semantic_examples.MapExampleCollection href "" "fenic.core.types.semantic_examples.MapExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-output examples for semantic map operations. Stores examples that demonstrate how input data should be transformed into output text or structured data. Each example shows the expected output for a given set of input fields. Initialize a collection of semantic map examples. Parameters: - **`examples`** (`List[MapExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with output and input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[MapExample] = None): """Initialize a collection of semantic map examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: MapExample) -> MapExampleCollection ``` Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Parameters: - **`example`** (`MapExample`) – The MapExample to add. Returns: - `MapExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: MapExample) -> MapExampleCollection: """Create an example in the collection with output and input type validation. Ensures all examples in the collection have consistent output types (either all strings or all BaseModel instances) and validates that input fields have consistent types across examples. For input validation: - The first example establishes the schema and cannot have None values - Subsequent examples must have the same fields but can have None values - Non-None values must match the established type for each field Args: example: The MapExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example output type doesn't match the existing examples in the collection, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, MapExample): raise InvalidExampleCollectionError( f"Expected example of type {MapExample.__name__}, got {type(example).__name__}" ) # Validate output type consistency self._validate_single_example_output_type(example) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> MapExampleCollection ``` Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> MapExampleCollection: """Create collection from a Polars DataFrame. Must have an 'output' column and at least one input column.""" collection = cls() if EXAMPLE_OUTPUT_KEY not in df.columns: raise ValueError( f"Map Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise ValueError( "Map Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): input_dict = {col: row[col] for col in input_cols} example = MapExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` ## PredicateExample Bases: `BaseModel` ``` flowchart TD fenic.core.types.semantic_examples.PredicateExample[PredicateExample] click fenic.core.types.semantic_examples.PredicateExample href "" "fenic.core.types.semantic_examples.PredicateExample" ``` A single semantic example for semantic predicate operations. Predicate examples demonstrate the evaluation of input variables against a specific condition, used in a semantic.predicate operation. ## PredicateExampleCollection ``` PredicateExampleCollection(examples: List[PredicateExample] = None) ``` Bases: `BaseExampleCollection[PredicateExample]` ``` flowchart TD fenic.core.types.semantic_examples.PredicateExampleCollection[PredicateExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection[BaseExampleCollection] fenic.core.types.semantic_examples.BaseExampleCollection --> fenic.core.types.semantic_examples.PredicateExampleCollection click fenic.core.types.semantic_examples.PredicateExampleCollection href "" "fenic.core.types.semantic_examples.PredicateExampleCollection" click fenic.core.types.semantic_examples.BaseExampleCollection href "" "fenic.core.types.semantic_examples.BaseExampleCollection" ``` Collection of input-to-boolean examples for predicate operations. Stores examples showing which inputs should evaluate to True or False based on some condition. Each example contains input fields and a boolean output indicating whether the condition holds. Initialize a collection of semantic predicate examples. Parameters: - **`examples`** (`List[PredicateExample]`, default: `None` ) – List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. Methods: - **`create_example`** – Create an example in the collection with input type validation. - **`from_polars`** – Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` def __init__(self, examples: List[PredicateExample] = None): """Initialize a collection of semantic predicate examples. Args: examples: List of examples to add to the collection. Each example will be processed through create_example() to ensure proper formatting and validation. """ self._type_validator = _ExampleTypeValidator() super().__init__(examples) ``` ### create_example ``` create_example(example: PredicateExample) -> PredicateExampleCollection ``` Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Parameters: - **`example`** (`PredicateExample`) – The PredicateExample to add. Returns: - `PredicateExampleCollection` – Self for method chaining. Raises: - `InvalidExampleCollectionError` – If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. Source code in `src/fenic/core/types/semantic_examples.py` ``` def create_example(self, example: PredicateExample) -> PredicateExampleCollection: """Create an example in the collection with input type validation. Validates that input fields have consistent types across examples. The first example establishes the schema and cannot have None values. Subsequent examples must have the same fields but can have None values. Args: example: The PredicateExample to add. Returns: Self for method chaining. Raises: InvalidExampleCollectionError: If the example type is wrong, if the first example contains None values, or if subsequent examples have type mismatches. """ if not isinstance(example, PredicateExample): raise InvalidExampleCollectionError( f"Expected example of type {PredicateExample.__name__}, got {type(example).__name__}" ) # Validate input types example_num = len(self.examples) + 1 self._type_validator.process_example(example.input, example_num) self.examples.append(example) return self ``` ### from_polars ``` from_polars(df: DataFrame) -> PredicateExampleCollection ``` Create collection from a Polars DataFrame. Source code in `src/fenic/core/types/semantic_examples.py` ``` @classmethod def from_polars(cls, df: pl.DataFrame) -> PredicateExampleCollection: """Create collection from a Polars DataFrame.""" collection = cls() # Validate output column exists if EXAMPLE_OUTPUT_KEY not in df.columns: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame missing required '{EXAMPLE_OUTPUT_KEY}' column" ) input_cols = [col for col in df.columns if col != EXAMPLE_OUTPUT_KEY] if not input_cols: raise InvalidExampleCollectionError( "Predicate Examples DataFrame must have at least one input column" ) for row in df.iter_rows(named=True): if row[EXAMPLE_OUTPUT_KEY] is None: raise InvalidExampleCollectionError( f"Predicate Examples DataFrame contains null values in '{EXAMPLE_OUTPUT_KEY}' column" ) input_dict = {col: row[col] for col in input_cols if row[col] is not None} example = PredicateExample(input=input_dict, output=row[EXAMPLE_OUTPUT_KEY]) collection.create_example(example) return collection ``` --- # fenic.core.types.summarize Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/core/types/summarize/ Summary structures for different formats of summaries. Classes: - **`KeyPoints`** – Summary as a concise bulleted list. - **`Paragraph`** – Summary as a cohesive narrative. ## KeyPoints Bases: `BaseModel` ``` flowchart TD fenic.core.types.summarize.KeyPoints[KeyPoints] click fenic.core.types.summarize.KeyPoints href "" "fenic.core.types.summarize.KeyPoints" ``` Summary as a concise bulleted list. Each bullet should capture a distinct and essential idea, with a maximum number of points specified. Attributes: - **`max_points`** (`int`) – The maximum number of key points to include in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of key points. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of key points. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of key points.""" return self.max_points * 75 ``` ## Paragraph Bases: `BaseModel` ``` flowchart TD fenic.core.types.summarize.Paragraph[Paragraph] click fenic.core.types.summarize.Paragraph href "" "fenic.core.types.summarize.Paragraph" ``` Summary as a cohesive narrative. The summary should flow naturally and not exceed a specified maximum word count. Attributes: - **`max_words`** (`int`) – The maximum number of words allowed in the summary. Methods: - **`max_tokens`** – Calculate the maximum number of tokens for the summary based on the number of words. ### max_tokens ``` max_tokens() -> int ``` Calculate the maximum number of tokens for the summary based on the number of words. Source code in `src/fenic/core/types/summarize.py` ``` def max_tokens(self) -> int: """Calculate the maximum number of tokens for the summary based on the number of words.""" return int(self.max_words * 1.5) ``` --- # fenic.logging Canonical HTML: https://docs.fenic.ai/latest/reference/fenic/logging/ Logging configuration utilities for Fenic. Functions: - **`configure_logging`** – Configure logging for the library and root logger in interactive environments. ## configure_logging ``` configure_logging(log_level: int = logging.INFO, log_format: str = '%(asctime)s [%(name)s] %(levelname)s: %(message)s', log_stream: Optional[TextIO] = None) -> None ``` Configure logging for the library and root logger in interactive environments. This function ensures that logs from the library's modules appear in output by setting up a default handler on the root logger *only if* one does not already exist. This is especially useful in notebooks, scripts, or REPLs where logging is often unset. It configures the root logger and sets the library's top-level logger to propagate logs to the root. If the root logger has no handlers, this function sets up a default configuration and silences noisy dependencies like 'openai' and 'httpx'. In more complex applications or when integrating with existing logging configurations, you might prefer to manage logging setup externally. In such cases, you may not need to call this function. Source code in `src/fenic/logging.py` ``` def configure_logging( log_level: int = logging.INFO, log_format: str = "%(asctime)s [%(name)s] %(levelname)s: %(message)s", log_stream: Optional[TextIO] = None, ) -> None: """Configure logging for the library and root logger in interactive environments. This function ensures that logs from the library's modules appear in output by setting up a default handler on the root logger *only if* one does not already exist. This is especially useful in notebooks, scripts, or REPLs where logging is often unset. It configures the root logger and sets the library's top-level logger to propagate logs to the root. If the root logger has no handlers, this function sets up a default configuration and silences noisy dependencies like 'openai' and 'httpx'. In more complex applications or when integrating with existing logging configurations, you might prefer to manage logging setup externally. In such cases, you may not need to call this function. """ stream = log_stream or sys.stderr formatter = logging.Formatter(log_format) handler = logging.StreamHandler(stream) handler.setFormatter(formatter) root_logger = logging.getLogger() if not root_logger.hasHandlers(): # Set up root logger only if not already configured root_logger.setLevel(log_level) root_logger.addHandler(handler) # Silence noisy dependencies for noisy_logger_name in NOISY_LOGGER_NAMES: noisy_logger = logging.getLogger(noisy_logger_name) noisy_logger.setLevel(logging.ERROR) # Set the library logger level and enable propagation library_root_name = __name__.split(".")[0] library_logger = logging.getLogger(library_root_name) library_logger.setLevel(log_level) library_logger.propagate = True ```