Skip to content

Setting up DataForge with Snowflake Custom Notebook

Starting in DataForge 11.1, you can run custom ingestion, parsing, and post-output notebooks from shared Snowflake Workspaces.

Create and publish the notebook

  1. In Snowsight, create and test an .ipynb notebook in a shared Workspace. Ask your Snowflake administrator to grant the DataForge execution role read access to the Workspace.
  2. Add a requirements.txt at the Workspace root. Pin the DataForge SDK to the version for your DataForge release and add any other dependencies. For example, a DataForge 11.1.0 production environment uses dataforge-sdk[psycopg2]==11.1.0.
  3. Open each edited file in Snowsight and select Publish. Publishing is per file, so publish the notebook and any edits to requirements.txt or supporting files before starting a DataForge run. See Collaborate in a shared workspace.

Configure DataForge

Enter the full Snowflake Workspace URI in Notebook, then save your settings. Use versions/last to point to the published files.

snow://workspace/MY_DB.MY_SCHEMA.MY_WORKSPACE/versions/last/notebooks/import_customers.ipynb

Use the Workspace database, schema, and object name, then the path to the file inside the Workspace. Quote case-sensitive Snowflake identifiers or identifiers with spaces. For example, use "Team Workspace" as the Workspace component. The Snowflake Workspace SQL guide describes the URI format.

To run a notebook for In DataForge
Custom ingestion Set the source connection type to Custom, enter the notebook URI, then use Pull Now.
Custom parsing On a file source, set the parser to Custom and enter the notebook URI. Pull a file or reset parsing for an existing input.
Custom post-output In Output Settings, select Custom Notebook under Post Output Commands and enter the notebook URI. Run the output.

Use the open icon beside Notebook to open the file directly in Snowsight.

Publish updates

Select Publish in Snowsight when your edits are ready. Keep the same URI in DataForge. DataForge creates a temporary Code Bundle from the latest published files for each run. Draft edits take effect after publication on the next run, and a running process keeps the snapshot it started with.

Set Notebook Execution Timeout if the run needs a time limit.

The environment uses Snowflake's system CPU pool and the process's query warehouse. DataForge manages the temporary Code Bundle in the DataForge database’s PUBLIC schema. Administrators can use the bundle and Snowflake query identifiers in the process log to investigate failures.

If Core loses contact with a run, it retains the bundle. An administrator can remove it after confirming in Snowflake that its query has ended.