Skip to content

dev: add make notebook#2528

Merged
kevinjqliu merged 10 commits intoapache:mainfrom
kevinjqliu:kevinjqliu/make-notebook
Jan 20, 2026
Merged

dev: add make notebook#2528
kevinjqliu merged 10 commits intoapache:mainfrom
kevinjqliu:kevinjqliu/make-notebook

Conversation

@kevinjqliu
Copy link
Copy Markdown
Contributor

@kevinjqliu kevinjqliu commented Sep 26, 2025

Rationale for this change

Add 2 new Make command (make notebook) to spin up a jupyter notebook; make notebook-infra spins up jupyter notebook along with integration test infrastructure.

Pyiceberg Example Notebook

Pyiceberg example notebook (notebooks/pyiceberg_example.ipynb) is based on the https://py.iceberg.apache.org/#getting-started-with-pyiceberg page and doesn't require additional test infra.

Spark Example Notebook

Spark integration example notebook (notebooks/spark_integration_example.ipynb) is based on https://iceberg.apache.org/docs/nightly/spark-getting-started/ and requires integration test infrastructure (Spark, IRC, S3)

With spark connect (#2491) and our testing setup, we can quickly spin up a local env with make test-integration-exec which includes:

  • spark
  • iceberg rest catalog
  • hive metastore
  • minio

In the jupyter notebook, connect to spark easily

from pyspark.sql import SparkSession

# Create SparkSession against the remote Spark Connect server
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.sql("SHOW CATALOGS").show()

Are these changes tested?

Yes, run both make notebook and make notebook-infra locally and run the example notebooks

Are there any user-facing changes?

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants