Setting up DataForge with Snowflake Custom Notebook¶
Starting in DataForge 11.1, you can run custom ingestion, parsing, and post-output notebooks from shared Snowflake Workspaces.
Create and publish the notebook¶
- In Snowsight, create and test an
.ipynbnotebook in a shared Workspace. Ask your Snowflake administrator to grant the DataForge execution role read access to the Workspace. - Add a
requirements.txtat the Workspace root. Pin the DataForge SDK to the version for your DataForge release and add any other dependencies. For example, a DataForge 11.1.0 production environment usesdataforge-sdk[psycopg2]==11.1.0. - Open each edited file in Snowsight and select Publish. Publishing is per file, so publish the notebook and any edits to
requirements.txtor supporting files before starting a DataForge run. See Collaborate in a shared workspace.
Configure DataForge¶
Enter the full Snowflake Workspace URI in Notebook, then save your settings. Use versions/last to point to the published files.
Use the Workspace database, schema, and object name, then the path to the file inside the Workspace. Quote case-sensitive Snowflake identifiers or identifiers with spaces. For example, use "Team Workspace" as the Workspace component. The Snowflake Workspace SQL guide describes the URI format.
| To run a notebook for | In DataForge |
|---|---|
| Custom ingestion | Set the source connection type to Custom, enter the notebook URI, then use Pull Now. |
| Custom parsing | On a file source, set the parser to Custom and enter the notebook URI. Pull a file or reset parsing for an existing input. |
| Custom post-output | In Output Settings, select Custom Notebook under Post Output Commands and enter the notebook URI. Run the output. |
Use the open icon beside Notebook to open the file directly in Snowsight.
Publish updates¶
Select Publish in Snowsight when your edits are ready. Keep the same URI in DataForge. DataForge creates a temporary Code Bundle from the latest published files for each run. Draft edits take effect after publication on the next run, and a running process keeps the snapshot it started with.
Set Notebook Execution Timeout if the run needs a time limit.
The environment uses Snowflake's system CPU pool and the process's query warehouse. DataForge manages the temporary Code Bundle in the DataForge database’s PUBLIC schema. Administrators can use the bundle and Snowflake query identifiers in the process log to investigate failures.
If Core loses contact with a run, it retains the bundle. An administrator can remove it after confirming in Snowflake that its query has ended.