Pins in Databricks

If you haven’t heard of pins, it’s an amazing tool available for R and Python that lets you publish data, models, and other objects for easy sharing across projects and with your colleagues. You “pin” an object to a board, like Dropbox, Posit Connect, Amazon S3, or others, to track changes, re-run analyses on historical data, and undo mistakes. Read more about pins here.

At a recent Data Science Hangout with Chris Engelhardt, we heard great feedback from pins users and excitement about the possibility of using Databricks as a board. Databricks Unity Catalog offers a governance layer to manage access to data assets. R users could connect to their Unity Catalog using packages such as odbc or sparklyr, then prepare, analyze, and model their data. The resulting outputs could be saved as a pin, stored within Databricks in its secure and efficient data environment, and made accessible for versioning, collaboration, reporting, and further analysis without having to move or resave the data elsewhere.

Since that conversation, we have merged support for board_databricks() into the pins R package! Now, you can access and store pins directly in Databricks Volumes from your R script.

According to Databricks + pins user jmbarbone, some key advantages of this setup include:

• Your storage location is closer to where your data is processed, so you’re not transferring files out of Databricks
• Organizations can control access to pinned objects using Databricks’ data governance features
• Databricks Volumes can handle unstructured data, which we detail below
• Databricks Volumes can hold multiple files and folders in a directory-like structure

Let’s explore how this integration opens new possibilities for Databricks users.

library(pins)
library(haven)

# Connect to an existing Databricks Volume
# DATABRICKS_HOST is configured in .Renviron
board <-  board_databricks("/Volumes/workshops/information-schema/volumes")

# Read the GSS data
gss2012 <- read_sav("https://study.sagepub.com/sites/default/files/gss2012.sav")

# Process the GSS data here

# Write the GSS data to a pin
board |>
  pin_write(gss2012)