sdf run sql

Use the SQL mode in the CLI, to be able to run SQL queries on SDF states. For a given dataflow, we will have in context for SQL all the dataframe states, which are essentially the states with an arrow-row value.

For states that are scoped to a window, we will have access to the last flushed state. For states that are not window aware we will have access to the global state.

In order to enter the SQL mode, type sql in the SDF interactive shell created through sdf run or sdf deploy. In the SQL mode we could perform any sql command supported by the polars engine.

We can use the .help command to see the available options.

sql >> .help
shape: (3, 2)
┌─────────┬─────────────────────────────────┐
│ command ┆ description                     │
│ ---     ┆ ---                             │
│ str     ┆ str                             │
╞═════════╪═════════════════════════════════╡
│ .exit   ┆ Exit this program               │
│ .quit   ┆ Exit this program               │
│ .help   ┆ Display this help.              │
└─────────┴─────────────────────────────────┘

Examples:

Run command

Navigate to the directory with dataflow.yaml file, and run the command:

$ sdf run

Enter the SQL mode

Using the sql command:

>> sql
SDF SQL version sdf-beta5
Type .help for help.

Show tables in context

sql >> show tables
shape: (1, 1)
┌────────────────┐
│ name           │
│ ---            │
│ str            │
╞════════════════╡
│ count_per_word │
└────────────────┘

Perform a query

sql >> select * from count_per_word;
shape: (0, 2)
┌──────┬─────────────┐
│ _key ┆ occurrences │
│ ---  ┆ ---         │
│ str  ┆ u32         │
╞══════╪═════════════╡
│ abc  │  10         |
└──────┴─────────────┘

Exit the SQL mode

sql >> .exit