memshare: Shared memory multithreading in R via C++17, with zero-copy access through ALTREP data structures.

Overview

memshare enables multicore computation in R without redundant memory copies. Large vectors, matrices, or lists are stored once in shared memory and exposed to R processes as ALTREP views. This allows workers in a PSOCK cluster to operate on the same physical data while avoiding serialization overhead.

Key features:

• Shared memory allocation via C++17 back-end (shm_open on Unix, MapViewOfFile on Windows).
• ALTREP wrappers so R sees shared objects as native vectors or matrices.
• High-level parallels to parallel::parApply and parallel::parLapply:
  • memApply() — apply a function row/column-wise over a matrix in shared memory.
  • memLapply() — apply a function element-wise over a list in shared memory.
• Automatic cleanup of shared objects when all views are released or when the package is unloaded.
• Tested on Linux, macOS, and Windows.

Installation

From CRAN:

install.packages("memshare")

From GitHub (development version):

remotes::install_github("mthrun/memshare")

System requirements: R ≥ 4.0 and a C++17 compiler.

Quick start

Example 1: Parallel correlations with a matrix

library(memshare)
library(parallel)

set.seed(1)
n = 10000
p = 2000

X = matrix(rnorm(n * p), n, p)
y = rnorm(n)

res = memApply(
  X = X, MARGIN = 2,
  FUN = function(v, y) cor(v, y),
  VARS = list(y = y)
)
str(res)

Example 2: List operations

library(memshare)
library(parallel)

list_length = 1000
matrix_dim = 100

ListV = lapply(
     1:list_length,
     function(i) matrix(rnorm(matrix_dim * matrix_dim),
     nrow = matrix_dim, ncol = matrix_dim))

y = rnorm(matrix_dim)

namespace = "ID123"
res = memshare::memLapply(ListV, function(el, y) {
   el %*% y
}, NAMESPACE=namespace, VARS=list(y=y), MAX.CORES = 1)

Each element el of ListV is multiplied by y in parallel. The list resides once in shared memory.

Concepts

• Pages: memory regions owned by the current R session that loaded the package.

• Views: ALTREP wrappers exposing shared memory variables (read/write capable).

• Namespaces: string identifiers defining a shared memory context across sessions.

When the package is detached, all handles and associated shared memory pages are released, unless another R process still holds references.

Shared-memory lifecycle

memshare exposes explicit lifecycle functions so you can control when data is placed in shared memory and when it is freed.

registerVariables(namespace, variableList)

Allocate shared memory and copy R objects (matrices or vectors, or lists for memLapply) into it. - namespace: character(1). Identifier of the shared memory context shared across processes. - variableList: a named list of objects to register. Names become the keys under which you can later retrieve views.

Example

library(memshare)

ns <- "my_namespace"
X  <- matrix(rnorm(1e4), 100, 100)
y  <- rnorm(100)

registerVariables(ns, list(X = X, y = y))
# Now X and y live once in shared memory and can be accessed from other R processes

releaseVariables(namespace, variableNames)

Delete variables from the shared memory space. Shared regions are only removed when no active views remain. - namespace: character(1) used above. - variableNames: character vector of variable names to free.

Example

# After all workers have released their views:
releaseVariables(ns, c("X", "y"))

Working with views: retrieveViews() and releaseViews()

To avoid duplication, workers attach to shared memory by views: - retrieveViews(namespace, c("X","y")) returns ALTREP-backed objects that behave like ordinary R matrices/vectors. - Always call releaseViews(namespace, ...) when finished so that the backing memory can be reclaimed.

Example (worker-side)

vlist <- retrieveViews(ns, c("X","y"))
# use vlist$X and vlist$y
releaseViews(ns, c("X","y"))

Tip: memApply() and memLapply() manage views for you automatically, but the low-level API above is useful for custom workflows.

Manual

The full manual for users or developers is available here: Package documentation

References