SQL Console on Datasets: 5 Proven DuckDB Query Tips

The SQL console on Hugging Face datasets lets you run DuckDB SQL on a dataset straight from its page, with nothing to install. Hugging Face announced it on 17 September 2024 in the blog post Introducing the SQL Console on Datasets, so it is not a 2026 launch as the first version of this article said. This rewrite checks the old claims against the official docs and runs the same queries with DuckDB 2.0.0 on the command line, which is what the console's "Copy to DuckDB CLI" button is for. The numbers below, including how many bytes each query really downloads, come from that run on 7 November 2026.

Lab note: the browser console itself was not driven by a script here. Everything in a code block was run with the DuckDB 2.0.0 Python package in a python:3.12-slim container against public datasets on the Hub. Statements about the browser UI come from the SQL Console documentation and the launch blog post.

Contents

• What the SQL console on datasets does
• The old DuckDB command fails
• Recreate the SQL console tables in DuckDB
• 3 query patterns from the docs, tested
• Histogram of a column in the SQL console
• Regex filter
• Leakage between splits
• How much data does a SQL console query download?
• SQL console export, other formats and private data
• Tips before you rely on the SQL console
• Conclusion

What the SQL console on datasets does

According to the docs, the SQL console is powered by DuckDB WASM and runs entirely in the browser; you open it from the Data Studio of a dataset. From it you can:

• run DuckDB SQL queries on the dataset,
• share the results with a link,
• download the results as a Parquet or CSV file,
• embed the results in your own page with an iframe,
• query the dataset with natural language.

The launch post explains how the tables appear: the console reads the dataset's Parquet files and "creates views for you to query based on your dataset splits and configs". If a dataset is stored in another format, the first 5 GB is auto-converted to Parquet. It also lists two limits worth knowing before you trust the old "only limited by your RAM" line: the browser engine has a memory limit of about 3 GB, and DuckDB WASM does not have full feature parity with the DuckDB CLI.

The docs also describe an API to create, update and delete saved queries ("embeds") with POST, PATCH and DELETE on /api/datasets/{namespace}/{repo}/sql-console/embed, authenticated with a bearer token.

The old DuckDB command fails

The first version of this post said the console generates a CLI command like this one. Run as written, it does not work:

D SELECT * FROM 'hf://datasets/glue/mrpc/train/*.parquet' LIMIT 5
HTTPException: HTTP Error: HTTP GET error listing 'https://huggingface.co/api/datasets/glue/mrpc/tree/main/train' (HTTP 401 Unauthorized)

Two things are wrong. The GLUE dataset lives under the nyu-mll organisation, and its files are named <config>/<split>-00000-of-00001.parquet, not <config>/train/*.parquet. Note the error: the Hub answers 401 for a repository that does not exist, the same code you get for a private dataset without a token, so a typo looks like an authentication problem. With the real path the query works:

D SELECT * FROM 'hf://datasets/nyu-mll/glue/mrpc/train-*.parquet' LIMIT 3

It returned the columns sentence1, sentence2, label and idx in 2.62 seconds. The DuckDB page of the Hub docs gives the general form as hf://datasets/{my-username}/{my-dataset}/{path_to_file}, with glob patterns such as **/*.parquet allowed in the file part.

Recreate the SQL console tables in DuckDB

In the browser SQL console the split views already exist. In the CLI you create them yourself, which also keeps later queries short. Here are views for the CoLA config of GLUE, named after the splits:

D CREATE VIEW train AS FROM 'hf://datasets/nyu-mll/glue/cola/train-*.parquet'
D CREATE VIEW validation AS FROM 'hf://datasets/nyu-mll/glue/cola/validation-*.parquet'
D DESCRIBE train
┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │  null   │   key   │ default │  extra  │
│   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ sentence    │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ label       │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ idx         │ INTEGER     │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

The old post's SQL console example (short sentences as a sign of bad data) runs unchanged on these views. On CoLA it shows that short rows are not always junk: this dataset is about grammar, and many short sentences are valid examples with label 1.

D SELECT idx, label, sentence FROM train WHERE LENGTH(sentence) < 10 LIMIT 20
┌───────┬───────┬───────────┐
│  idx  │ label │ sentence  │
│ int32 │ int64 │  varchar  │
├───────┼───────┼───────────┤
│  1227 │     1 │ I yawned. │
│  2198 │     1 │ I shaved. │
│  3521 │     1 │ Wash me!  │
│  3573 │     1 │ He left.  │
│  3727 │     1 │ It rains. │
│  5853 │     1 │ He goes.  │

D SELECT count(*) AS rows, count(*) FILTER (LENGTH(sentence) < 15) AS under_15_chars FROM train
┌───────┬────────────────┐
│ rows  │ under_15_chars │
│ int64 │     int64      │
├───────┼────────────────┤
│  8551 │            191 │
└───────┴────────────────┘

3 query patterns from the docs, tested

Histogram of a column in the SQL console

The docs show from histogram(train, Rating) for a chess dataset. On the CoLA labels it gives the class balance in one line:

D FROM histogram(train, label)
┌────────────┬────────┬──────────────────────────────────────────────────────────────────────────────────┐
│    bin     │ count  │                                       bar                                        │
│  varchar   │ uint64 │                                     varchar                                      │
├────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ x <= 0     │   2528 │ █████████████████████████████████▌                                               │
│ 0 < x <= 1 │   6023 │ ████████████████████████████████████████████████████████████████████████████████ │
└────────────┴────────┴──────────────────────────────────────────────────────────────────────────────────┘

Regex filter

The docs use regexp_matches() to find answers that contain code blocks. The same function finds two-word sentences that end in a past-tense verb:

D SELECT count(*) FROM train WHERE regexp_matches(sentence, '^[A-Z][a-z]+ [a-z]+ed\.$')
┌──────────────┐
│ count_star() │
├──────────────┤
│           45 │
└──────────────┘

Leakage between splits

The docs include a leakage detection query that compares the train and test splits. A simpler version you can paste into the SQL console finds rows that appear in both train and validation, and a join shows two of them even carry different labels:

D SELECT count(*) AS shared_sentences FROM (SELECT sentence FROM train INTERSECT SELECT sentence FROM validation)
┌──────────────────┐
│ shared_sentences │
├──────────────────┤
│               16 │
└──────────────────┘

D SELECT t.label AS train_label, v.label AS val_label, v.sentence FROM train t JOIN validation v USING (sentence) WHERE t.label <> v.label LIMIT 5
┌─────────────┬───────────┬────────────────────────────┐
│ train_label │ val_label │          sentence          │
├─────────────┼───────────┼────────────────────────────┤
│           0 │         1 │ I shaved myself.           │
│           1 │         0 │ Any man didn't eat dinner. │
└─────────────┴───────────┴────────────────────────────┘

How much data does a SQL console query download?

The old post claimed the engine "doesn't read the whole file" and fetches only the needed chunks. That is true, but how much it saves depends on the query. I turned on DuckDB's HTTP log and summed the byte ranges of every request for three queries on the MNLI train file, which is 52,224,361 bytes on the Hub:

Range bytes fetched per query, from DuckDB's HTTP log.

Query	GET requests	Bytes fetched	Share of file
count(*)	4	266,601	0.5%
GROUP BY label	395	1,748,992	3.3%
WHERE premise ILIKE '%linux%'	395	34,263,040	65.6%

A row count only needs the Parquet footer. A small integer column is cheap. A text search must read the whole text column, which is most of the file. In the browser that data has to fit in the ~3 GB limit, so on large datasets filter on small columns first and add LIMIT, as the launch post advises.

SQL console export, other formats and private data

The SQL console download button saves results as CSV or Parquet. The CLI equivalent is COPY:

D COPY (SELECT idx, label, sentence FROM train WHERE LENGTH(sentence) < 15) TO '/tmp/short.csv' (HEADER)
┌───────┐
│ Count │
├───────┤
│   191 │
└───────┘

$ head -4 /tmp/short.csv
idx,label,sentence
403,1,He has left.
405,1,Bill saw him.
489,1,John tried.

For a dataset that is not stored as Parquet, DuckDB can read the original file or the auto-converted copy on the @~parquet branch (the refs/convert/parquet revision). Both work on the docs' example dataset:

D SELECT count(*) FROM 'hf://datasets/jamescalam/world-cities-geo/*.jsonl'
│         9083 │

D SELECT city, country FROM 'hf://datasets/jamescalam/world-cities-geo@~parquet/default/train/0000.parquet' LIMIT 3
│ Kabul          │ Afghanistan │
│ Kandahar       │ Afghanistan │
│ Mazar-e Sharif │ Afghanistan │

On private data the old FAQ answered a simple "yes". The launch post only promises shareable links for public datasets. For the CLI, the DuckDB authentication page says to store your token as a secret, either CREATE SECRET hf_token (TYPE HUGGINGFACE, TOKEN 'your_hf_token'); or with PROVIDER credential_chain, which reads ~/.cache/huggingface/token after hf auth login. Private datasets were not tested in this lab.

Tips before you rely on the SQL console

• Copy the exact path from the dataset's file list; a wrong path returns 401, not 404.
• Use DESCRIBE first, in the CLI or the SQL console; column names differ per config (MRPC has sentence1/sentence2, CoLA has sentence).
• Expect text filters to download most of the file; count and group on small columns where you can.
• Move to the CLI for joins across datasets, big scans or private data. If you are setting up Python for this, see how to install Python 3.13 on Linux and Linux AI development.

Conclusion

The SQL console on datasets is a fast way to look inside a Hugging Face dataset before you download it: views per split, DuckDB SQL, histograms, regex and leakage checks, and CSV or Parquet export. The engine really does fetch only the parts of a Parquet file a query needs, but a text search can still pull two thirds of the file, and the browser has a ~3 GB memory limit. When a query outgrows the SQL console, the same SQL runs in the DuckDB CLI with the correct hf://datasets/<org>/<name>/... path.