Rethinking packages & functions preloading in webR 0.2.2

This post is the fifth one of a series of post about webR:

• Using webR in an Express JS REST API
• The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS
• Preloading your R packages in webR in an Express JS API
• Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR
• Rethinking webR package & functions preloading with webR 0.2.2
• webrcli & spidyr: A starter pack for building NodeJS projects with webR inside

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

When I wrote my blogpost about Preloading your R packages in webR in an Express JS API, I mentioned that there was no native way to preload things in the webR filesystem — meaning that you had to reinstall all the R packages whenever the app was launched (and reported it in a Github issue). This also meant that I couldn’t easily take my own functions and run them in the webR environment, as described in Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR. These needs made me write the webrtools NodeJS package, to do just that:

• Download the R packages compiled for webR, to a local folder
• Bundle them in the webR lib (by reading the folder tree and reproducing it in webR filesystem)
• Load your local package to access its functions in webR

The last webR version now exposes Emscripten’s FS.mount, so this makes the process easier, as there is now no need to read the file tree and recreate it: the folder from the server can be mounted into webR filesystem.

That implied some rework (now integrated to webrtools) :

• Use webR.FS.mkdir to create a local lib: the module can’t mount the package library into the libPath as is, because it would overwrite the already bundled library (in other words, if you mount into a folder that is not empty, the content from the server overwrites the content from webR).
• Use webR.FS.mount to mount the local directory into the newly created library
• Ensure that this new library is in the libPath()

Here is the new code for loadPackages, bundled into webrtools:

async function loadPackages(webR, dirPath, libName = "webr_packages") {
  // Create a custom lib so that we don't have to worry about
  // overwriting any packages that are already installed.
  await webR.FS.mkdir(`/usr/lib/R/${libName}`)
  // Mount the custom lib
  await webR.FS.mount("NODEFS", { root: dirPath }, `/usr/lib/R/${libName}`);
  // Add the custom lib to the R search path
  await webR.evalR(`.libPaths(c('/usr/lib/R/${libName}', .libPaths()))`);
}

I’ve also decided to deprecate the loadFolder function, as it is now native with webR.FS.mkdir + webR.FS.mount.

So here is a rewrite of the app from Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR.

const app = require('express')()
const path = require('path');
const { loadPackages } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await globalThis.webR.FS.mkdir("/home/rfuns")

  await globalThis.webR.FS.mount(
    "NODEFS",
    {
      root: path.join(__dirname, 'rfuns')
    },
    "/home/rfuns"
  )

  console.log("📦 Packages written to webR 📦");

  // see https://github.com/r-wasm/webr/issues/292
  await globalThis.webR.evalR("options(expressions=1000)")
  await globalThis.webR.evalR("pkgload::load_all('/home/rfuns')");

  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })

})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'unique_species()'
  );
  try {
    let js_res = await result.toJs()
    res.send(js_res.values)
  } finally {
    webR.destroy(result);
  }

})

app.get('/:n', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'star_wars_by_species(n)',
    { env: { n: req.params.n } }
  );
  try {
    const result_js = await result.toJs();
    res.send(result_js)
  } finally {
    webR.destroy(result);
  }
});

Full code is at ColinFay/webr-examples/.

Further exploration to be done: webR now bundles a way to package a file system in a file, which can then be downloaded and mounted into the runtime, as described here. This might come handy for our current structure, but I’ll have to explore it a bit more.