Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

In this blogpost, you’ll find some updates about some recent changes made in webrcli & spidyr.

What webrcli & spidy wants to achieve

In a nutshell, the goal of webrcli & spidyr is to provide an interface for bringing your R functions inside a NodeJS runtime.

Think of it as something that wants to achieve the following interation:

• An R dev writes, builds and shares a package called {rfuns}, which contains a function called hello_world()

• A NodeJS dev takes this package, can call it with rfuns.hello_world(), access and manipulate the results with JavaScript.

An important note

With webR & the tools described below, there is no R session running while the NodeJS app is running. spidyr imports an R function converted to a native JS. rfuns.hello_world does not call an R session running somewhere else.

That also means that if you build a container containing the app, you don’t need to install R in it. It’s a NodeJS app.

Below are some updates recently made to both webrcli & spidyr.

Making the template code smaller

When drafting the first code of webrcli & spidyr, I focused on providing a minimal template for an index.js loading your R functions, contained in a subdirectory inside your node project.

This worked, but might have been a bit complex to grasp. Provided you don’t change the default names of the folders (in that case you’ll need to modify the default parameters), the new default index.js look like this:

const {
  initSpidyr,
  mountLocalPackage
} = require('spidyr');

(async () => {

  await initSpidyr()

  const rfuns = await mountLocalPackage("./rfuns");

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  console.log("✅ Everything is ready!");
})();

This minimal template makes it easier to read and understand what the code does. Note that you’ll get this template if you run webrcli init myproject.

Let’s decompose what this code does :

• await initSpidyr() will create an object called spidyr_webR stored as globalThis.spidyr_webR, init() it (as in webr.init()), and load the packages contained in the ./webr_packages folder inside your project dir, the one containing the packages installed with webrcli install.

• const rfuns = await mountLocalPackage("./rfuns"); will take the R package contained in ./rfuns, load it into the webR instance (spidyr_webR), recursively get all the exported functions from your package, and store them inside the rfuns object.

• const hw = await rfuns.hello_world() will call the hello_world() function from {rfuns}. The values are then console.loged.

new Library() is now library()

Calling a package is now const spongebob = library("spongebob") instead of the previous approach which was const spongebob = new Library("spongebob");await spongebob.load(globalThis.webR);

Better installation from package.json & DESCRIPTION

The previous version of webrcli did not stored nor re-installed the R package installed in the project, meaning that if you git cloned a project, you had to guess what R packages to re-install.

Now, a project initiated with webrcli will store all the packages installed with webrcli install inside the package.json file. Then, when running the npm install command, the R packages will also be redownloaded.

You can also installed a series of packages based on a DESCRIPTION file with webrcli installFromDesc.

Other tools

• shareEnv allows to copy one, several or all env var from node to the webR instance.

Workflow summary

• webrcli init myproject will create a project skeleton at ./myproject

• webrcli install spongebob will download the version of {spongebob} compiled for webR, and add it to ./webr_packages.

• You can then modify you R code with your own functions.

• Call your R function from rfuns.xyz() in Node

• Run with npm start

Please do give it a shot

I think I’ve been the only user for now 😅, so please do try these and let me know what you think.

Future works

Here are what I plan on working in the upcoming weeks:

• webrcli init should allow for specifying a webr version from npm issue 17

• initSpidyr should allow to pass params to webR.init() issue 12

• Allow to install from r-universe, something like webrcli installFromRUniverse issue 16. Probably in the future it will be merged to webrcli install and we’ll have a way to guess what the user wants based on the input.

• A small converter from R named list to JS objects Issue 3

• More documentation, and probably documenting the conversion of an existing {shiny} app to a spidyr based one.

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Again, a new project?

Most of the previous posts have been about poking around with webR inside NodeJS, but nothing stable on how to build an app.

That’s something I’m trying to change. Based on the experience gathered building apps in R (in {shiny} with {golem}) and in JavaScript, I’ve tried to come up with a solution for a starter kit for building what I think will be the perfect skeleton for webR/NodeJS project.

The idea is to have one project with inside:

• One R package
• One JS app
• A tool that brings the R package into the NodeJS app and allow to use the R functions inside JavaScript

This skeleton needed a toolkit to perform the following task:

1️⃣ From a cli point of view:

• init the project with a specific skeleton
• allow to install packages from the webr CRAN to a local webr library, in the spirit of previous post.
• allow to install the deps from a DESCRIPTION file

2️⃣ From a NodeJS point of view:

• Load the packages downloaded in a local webr library
• Create an interface to load and manipulate the functions from a package downloaded from CRAN, and use them in JS.
• Same but with a local package folder

For the function manipulation mechanism, I wanted something that would allow to think in terms of pkg::fun() but in JavaScript, which would translate as pkg.fun() in your Node app. Something I had implemented in another old NodeJS module called hordes, which I can now safely deprecate in favor of the following tools.

The reasoning being: the R dev writes a standalone package that exports an xyz function, then the web team can load this package, and launch xyz() without writing any R code.

webrcli & spidyr

So, here comes webrcli and spidyr.

❗️ Please note that these tools are work in progress and has been used very few times and only for example apps, will need a lot of bug fixes and documentation, so if ever you plan on using it be indulgent, and report any bug or feature request ❗️

Project init

These packages are made to be used together, and here is an example of how to use them:

# Global installing webrcli
npm install -g webrcli
# Init a webrcli project
cd /tmp
webrcli init webrspongebob

👉 Initializing project ----
(This may take some time, please be patient)
👉 Copying template ----
👉 Installing {pkgload} ----
✅ {pkgload} downloaded and extracted ----

✅ {cli} downloaded and extracted ----

✅ {crayon} downloaded and extracted ----

✅ {desc} downloaded and extracted ----

✅ {fs} downloaded and extracted ----

✅ {glue} downloaded and extracted ----

✅ {pkgbuild} downloaded and extracted ----

✅ {rlang} downloaded and extracted ----

✅ {rprojroot} downloaded and extracted ----

✅ {withr} downloaded and extracted ----

✅ {R6} downloaded and extracted ----

✅ {callr} downloaded and extracted ----

✅ {processx} downloaded and extracted ----

✅ {ps} downloaded and extracted ----

The project is now created, let’s move into it.

cd ./webrspongebob
tree -L 1

.
├── index.js
├── node_modules
├── package-lock.json
├── package.json
├── rfuns
└── webr_packages

4 directories, 3 files

Here is how it’s organized:

• index.js is the main file for the app, node_modules the standard folder for the node deps
• package-lock.json / package.json are standard Node metadata files
• rfuns contains the R package that will be added to the NodeJS app
• webr_packages contains the R dependencies

We can launch the app with node index.js and it will output:

node index.js
👉 Loading WebR ----
👉 Loading R packages ----
ℹ Loading rfuns
[ 'Hello, world!' ]
✅ Everything is ready

Let’s dive a bit inside the index.js:

const path = require('path');
const { WebR } = require('webr');
const { loadPackages, LibraryFromLocalFolder } = require('spidyr');

const rfuns = new LibraryFromLocalFolder("rfuns");


(async () => {

  console.log("👉 Loading WebR ----");
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("👉 Loading R packages ----");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await rfuns.mountAndLoad(
    globalThis.webR,
    path.join(__dirname, 'rfuns')
  );

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  console.log("✅ Everything is ready!");

})();

Here are the bits that are specific to a spidyr project:

const rfuns = new LibraryFromLocalFolder("rfuns");

This function will take a local folder containing an R package, and load the functions from this package into the rfuns object. Here, for example, our R package contains one R function, hello_world(), it will then be available in NodeJS as rfuns.hello_world() once the mountAndLoad function is called.

await loadPackages(
  globalThis.webR,
  path.join(__dirname, 'webr_packages')
)

await rfuns.mountAndLoad(
  globalThis.webR,
  path.join(__dirname, 'rfuns')
);

The first bit loads the webr_packages folder, containing all the R dependencies, then the second bit mount the local folder into the webR file system and load the functions.

Finally, const hw = await rfuns.hello_world() calls the function from the R package, and its value is console.loged just after that.

With a CRAN package

And now, how do I load a CRAN package? For example, let’s say I want to use {spongebob} in my app?

First, let’s install {spongebob} via webrcli :

webrcli install spongebob

✅ {spongebob} downloaded and extracted ----

Then, let’s update our index.js:

const path = require('path');
const { WebR } = require('webr');
const { loadPackages, LibraryFromLocalFolder, Library } = require('spidyr');

const rfuns = new LibraryFromLocalFolder("rfuns");
const spongebob = new Library("spongebob");

(async () => {

  console.log("👉 Loading WebR ----");
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("👉 Loading R packages ----");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await rfuns.mountAndLoad(
    globalThis.webR,
    path.join(__dirname, 'rfuns')
  );

  await spongebob.load(
    globalThis.webR
  );

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  const said = await spongebob.tospongebob("hello from spongebob")

  console.log(said.values)

  console.log("✅ Everything is ready!");;

})();

Here:

• const spongebob = new Library("spongebob"); will create a lib, ready to mount a package
• await spongebob.load(globalThis.webR) will load all the functions from the {spongebob} package
• const said = await spongebob.tospongebob("hello from spongebob") will run the spongebob::tospongebob() R function

Then :

node index.js

👉 Loading WebR ----
👉 Loading R packages ----
ℹ Loading rfuns
[ 'Hello, world!' ]
[ 'helLo fROm spONgEbOb' ]
✅ Everything is ready!

Some random notes

The webrspongebob example is available at https://github.com/ColinFay/webrspongebob

About the current implementation

• Right now I’m not really sure how this handles the infix function like %>% for example. That being said, this might not be a function you’d want to use in the current way of building things.

• The exported functions are read via getNamespaceExports("pkg") and getExportedValue("pkg", "fun"), if ever this is not the perfect way to do this in base R but I’ll be happy to find some other ways to do this.

• webrcli and spidyr both encapsulate code run in webR, especially webrcli which has an R script which is sourced and run in a webR instance.

Future

Please do try these tools. I would be very happy to have your feedback on the philosophy, and on the general workflow.

If ever you find a bug or have an idea, feel free to open issues at :

• https://github.com/ColinFay/webrcli

• https://github.com/ColinFay/spidyr

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

When I wrote my blogpost about Preloading your R packages in webR in an Express JS API, I mentioned that there was no native way to preload things in the webR filesystem — meaning that you had to reinstall all the R packages whenever the app was launched (and reported it in a Github issue). This also meant that I couldn’t easily take my own functions and run them in the webR environment, as described in Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR. These needs made me write the webrtools NodeJS package, to do just that:

• Download the R packages compiled for webR, to a local folder
• Bundle them in the webR lib (by reading the folder tree and reproducing it in webR filesystem)
• Load your local package to access its functions in webR

The last webR version now exposes Emscripten’s FS.mount, so this makes the process easier, as there is now no need to read the file tree and recreate it: the folder from the server can be mounted into webR filesystem.

That implied some rework (now integrated to webrtools) :

• Use webR.FS.mkdir to create a local lib: the module can’t mount the package library into the libPath as is, because it would overwrite the already bundled library (in other words, if you mount into a folder that is not empty, the content from the server overwrites the content from webR).
• Use webR.FS.mount to mount the local directory into the newly created library
• Ensure that this new library is in the libPath()

Here is the new code for loadPackages, bundled into webrtools:

async function loadPackages(webR, dirPath, libName = "webr_packages") {
  // Create a custom lib so that we don't have to worry about
  // overwriting any packages that are already installed.
  await webR.FS.mkdir(`/usr/lib/R/${libName}`)
  // Mount the custom lib
  await webR.FS.mount("NODEFS", { root: dirPath }, `/usr/lib/R/${libName}`);
  // Add the custom lib to the R search path
  await webR.evalR(`.libPaths(c('/usr/lib/R/${libName}', .libPaths()))`);
}

I’ve also decided to deprecate the loadFolder function, as it is now native with webR.FS.mkdir + webR.FS.mount.

So here is a rewrite of the app from Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR.

const app = require('express')()
const path = require('path');
const { loadPackages } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await globalThis.webR.FS.mkdir("/home/rfuns")

  await globalThis.webR.FS.mount(
    "NODEFS",
    {
      root: path.join(__dirname, 'rfuns')
    },
    "/home/rfuns"
  )

  console.log("📦 Packages written to webR 📦");

  // see https://github.com/r-wasm/webr/issues/292
  await globalThis.webR.evalR("options(expressions=1000)")
  await globalThis.webR.evalR("pkgload::load_all('/home/rfuns')");

  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })

})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'unique_species()'
  );
  try {
    let js_res = await result.toJs()
    res.send(js_res.values)
  } finally {
    webR.destroy(result);
  }

})

app.get('/:n', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'star_wars_by_species(n)',
    { env: { n: req.params.n } }
  );
  try {
    const result_js = await result.toJs();
    res.send(result_js)
  } finally {
    webR.destroy(result);
  }
});

Full code is at ColinFay/webr-examples/.

Further exploration to be done: webR now bundles a way to package a file system in a file, which can then be downloaded and mounted into the runtime, as described here. This might come handy for our current structure, but I’ll have to explore it a bit more.

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Note 2: webR and the tools I’ve been writing are moving fast and if you’re reading this from the future, some of the things here might be obsolete.

The problem

Ok, so now that we have our webR / NodeJS machinery up and running, let’s try something more interesting: use our own R functions inside webR.

How can I do that?

There are at least three ways I could think of:

1️⃣ Writing a function inside the JS code to define a function, something like :

await globalThis.webR.evalR("my_fun <- function(x){...}");

But that doesn’t check what I would expect from something I’ll use in prod and I’m pretty sure you don’t need me to detail why 😅

• ❌ Well organized
• ❌ Documented
• ❌ Tested
• ❌ Safely installable

2️⃣ Simply create an R script and source it. Something like:

const fs = require('fs');
const path = require('path');
const script = path.join(__dirname, 'script.R')
const data = fs.readFileSync(script);
await globalThis.webR.FS.writeFile(
  "/home/web_user/script.R",
  data
);
await globalThis.webR.evalR("source('/home/web_user/script.R')");

That’s a bit better, we can at least organize our code in a script and it will be:

• ✅ Well organized (more or less)
• ✅ Documented (more or less)
• ❌ Tested
• ❌ Safely installable

3️⃣ I bet you saw me coming, the best way let’s put stuff into an R package, so that we can check all the boxes.

• ✅ Well organized
• ✅ Documented
• ✅ Tested
• ✅ Safely installable

Jeroen has written a Docker image to compile an R package to WASM, but I was looking for something that wouldn’t involve compiling via a docker container every time I make a change on my R package (even if that does sound appealing, I’m pretty sure this wouldn’t make for a seamless workflow).

So here is what I’m thinking should be a well structured NodeJS / WebR app:

• Putting all the web stuff inside the NodeJS app, because, well, NodeJS is really good at doing that.
• Putting all the “business logic”, data-crunching, modeling stuff (and everything R is really good at) into an R package.
• load webR, write my R package to the webR file system, and pkgload::load_all() it into webR.

That way, I can enjoy the best of both worlds:

• NodeJS is really good at doing web related things, and there are plenty of ways to test and deploy the code.
• And same goes for the R package: if you’re reading my blog I’m pretty sure I don’t need to convince you of why packages are the perfect tool for sharing production code.

The how

Let’s start by creating our project:

mkdir webr-preload-funs
cd webr-preload-funs
npm init -y
touch index.js
npm install express webr
R -e "usethis::create_package('rfuns', rstudio = FALSE)"

Let’s now create a simple function :

> usethis::use_r("sw")

#' @title Star Wars by Species
#' @description Return a tibble of Star Wars characters by species
#' @import dplyr
#' @export
#' @param species character
#' @return tibble
#' @examples
#' star_wars_by_species("Human")
#' star_wars_by_species("Droid")
#' star_wars_by_species("Wookiee")
#' star_wars_by_species("Rodian")
star_wars_by_species <- function(species){
  dplyr::starwars |>
    filter(species == )
}

We can now add {dplyr} and {pkgload} to our DESCRIPTION (we’ll need {pkgload} to load_all() the package).

usethis::use_package("dplyr")
usethis::use_package("pkgload")
devtools::document()

Now that we have a package skeleton, we’ll have to upload it to webR.

As described in the previous post, I’ve started a webrtools NodeJS module, which will contains function to play with webR. Before this post, it had one function, loadPackages, that was used to build a webR dependency library (see Preloading your R packages in webR in an Express JS API for more info).

We’ll need to add two features :

• Install deps from DESCRIPTION (not just a package name), so a wrapper around the Rscript ./node_modules/webrtools/r/install.R dplyr from before
• Copy the package folder in NodeJS, so a more generic version of loadPackages, that can load any folder to the webR filesystem.

First, in R, we’ll need to read the DESCRIPTION and build the lib:

download_packs_and_deps_from_desc <- function (
  description,
  path_to_installation = "./webr_packages"
)
{
    if (!file.exists(description)) {
        stop("DESCRIPTION file not found")
    }
    deps <- desc::desc_get_deps(description)
    for (pak in deps$package) {
        webrtools::download_packs_and_deps(pak, path_to_installation = path_to_installation)
    }
}

Note: the code of webrtools::download_packs_and_deps() is a wrapper around the R code described in Preloading your R packages in webR in an Express JS API

And in Node, we’ll rework our loadPackages and split it into two functions — one to load into any folder, and one to load into the package library.

Note that this function is bundled in the webrtools npm module.

async function loadFolder(webR, dirPath, outputdir = "/usr/lib/R/library") {
  // function from webrtools
  const files = getDirectoryTree(
    dirPath
  )
  for await (const file of files) {
    if (file.type === 'directory') {
      await globalThis.webR.FS.mkdir(
        `${outputdir}/${file.path}`,
      );
    } else {
      const data = fs.readFileSync(`${dirPath}/${file.path}`);
      await globalThis.webR.FS.writeFile(
        `${outputdir}/${file.path}`,
        data
      );
    }
  }
}

async function loadPackages(webR, dirPath) {
  await loadFolder(webR, dirPath, outputdir = "/usr/lib/R/library");
}

The end app

We now have everything we need!

npm i webrtools@0.0.2
Rscript ./node_modules/webrtools/r/install_from_desc.R $(pwd)/rfuns/DESCRIPTION

And now, to our index.js

const app = require('express')()
const path = require('path');
const { loadPackages, loadFolder } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await loadFolder(
    globalThis.webR,
    path.join(__dirname, 'rfuns'),
    "/home/web_user"
  )

  console.log("📦 Packages written to webR 📦");

  // see https://github.com/r-wasm/webr/issues/292
  await globalThis.webR.evalR("options(expressions=1000)")
  await globalThis.webR.evalR("pkgload::load_all('/home/web_user')");

  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })

})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'unique(dplyr::starwars$species)'
  );
  let js_res = await result.toJs()
  res.send(js_res.values)
})


app.get('/:n', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'star_wars_by_species(n)',
    { env: { n: req.params.n } }
    );
  try {
    const result_js = await result.toJs();
    res.send(result_js)
  } finally {
    webR.destroy(result);
  }
});

Let’s now try from another terminal:

curl http://localhost:3000

["Human","Droid","Wookiee","Rodian","Hutt","Yoda's species","Trandoshan","Mon Calamari","Ewok","Sullustan","Neimodian","Gungan",null,"Toydarian","Dug","Zabrak","Twi'lek","Vulptereen","Xexto","Toong","Cerean","Nautolan","Tholothian","Iktotchi","Quermian","Kel Dor","Chagrian","Geonosian","Mirialan","Clawdite","Besalisk","Kaminoan","Aleena","Skakoan","Muun","Togruta","Kaleesh","Pau'an"]

curl http://localhost:3000/Rodian

{"type":"list","names":["name","height","mass","hair_color","skin_color","eye_color","birth_year","sex","gender","homeworld","species","films","vehicles","starships"],"values":[{"type":"character","names":null,"values":["Greedo"]},{"type":"integer","names":null,"values":[173]},{"type":"double","names":null,"values":[74]},{"type":"character","names":null,"values":[null]},{"type":"character","names":null,"values":["green"]},{"type":"character","names":null,"values":["black"]},{"type":"double","names":null,"values":[44]},{"type":"character","names":null,"values":["male"]},{"type":"character","names":null,"values":["masculine"]},{"type":"character","names":null,"values":["Rodia"]},{"type":"character","names":null,"values":["Rodian"]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":["A New Hope"]}]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":[]}]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":[]}]}]}

Yeay 🎉 .

You can find the code here, and see it live at srv.colinfay.me/webr-preload-funs/.

You can also try it with

docker run -it -p 3000:3000 colinfay/webr-preload-funs

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Note 2 (or Disclaimer): this is kind of a long post that get into the weeds of package installation and library structure

Note 3: webR and the tools I’ve been writing are moving fast and if you’re reading this from the future, some of the things here might be obsolete.

The problem

In the two last posts, I’ve been deploying two REST APIs that use functions from the base distribution of R:

• tools::toTitleCase() in express-webr-hello-world
• cut() & faithful for express-webr-old-faithful

One of the strengths of the R ecosystem is its extensive list of contributed packages: a wide collection of code written by R nerds from all over the world. It would be unfortunate to use webR to create a web applications or API without taking advantage of this extensive library of packages: R wouldn’t be R without its packages.

Yet, bringing packages to webR is not that simple: it’s a distribution of R compiled for WASM, and you can’t just install a package from any CRAN repo.

Thankfully we don’t need to scratch our head too much anymore: the webR team has precompiled more than 18000 (at the time of writing this post) packages that we can install by simply calling webr::install() inside the webR instance.

That’s awesome, but there’s an issue with that: you can only install it from the webR JavaScript object As far as I know, there is no way to download them beforehand and upload them to the webR filesystem, as I reported in this issue. (Note: at the time of writing these lines, there has been some discussion on the GH issue, but there is at the moment no native, efficient way to do what I want to do 😅 — there might be one though if you’re reading this post in the future).

That means that every time the API is launched, webR has to download and reinstall the packages, as shown by this simple example:

const app = require('express')()
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  await globalThis.webR.installPackages(['attempt'])
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();


app.get('/', async (req, res) => {
  res.send("hello")
});

If I launch the app twice, the packages are downloaded twice 👇

$ node index.js
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

$ node index.js
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

This approach is definitely suboptimal: imagine having to reinstall packages every time you source() an R script, instead of benefiting from pre-installed packages in your lib. That’s kind of what is happening right now in this webR example.

Sure, it works, but it means that:

• I’m downloading the (same) packages every time the app is launched, which feels weird
• If I have a lot of packages, it would take a lot of time every time
• I can’t deploy this on a service that has no access to internet
• I can’t build my app just as any other language does: by relying on pre-downloaded /pre-installed modules
• I can’t benefit from caching the package installation in a Docker layer

So, faced with something I needed but didn’t find, I behave like any other software engineer: I built my own solution to pre-load R packages in webR, without having to re-download them every time. The rest of this blogpost explains this process.

Preloading R packages in my ExpressJS/webR API

Building the R lib

If you look at the internal of the folder on your computer that contains your packages (which is at .libPaths()), you’ll see that it’s a series of folders with a bunch of files. Whenever you load R and call library(), it looks there to load the code.

Inside these folders (you can have several lib), you’ll find unzipped archives, downloaded from a repository (or installed from git/developed locally). These packages, at installation time, possibly performed some build and compilation, depending on the package and the OS you’re on. I won’t go into the details of this here, as the packages for webR have been pre-compiled, meaning that you simply have to download the tar file, and unarchive it at the correct place. In other words, the content of this archive is exactly what you’ll find in the library folder, and there is nothing to compile/install again.

Keeping this in mind, the first thing to do then is to create a folder that will contain all the folders and files necessary to load a package. In other words, if I want to use {dplyr}, I need to find a way to download the pre-compiled version of {dplyr} and all its dependencies from https://repo.r-wasm.org/, and untar them into the folder I wanna use inside my project.

R has a weird way of managing packages, and if ever you want to hear me monologue, catch me at a conference and ask me to talk about this for 30 minutes (just kidding, it will probably last one hour). I won’t go into details about how it works and why it’s weird, but here are some issues I’ve been facing for my current task:

• By default R will not reinstall the dependencies if they are already installed on the machine. Which is great, but not in my case (I want the destination folder to contain the entire dependency tree). In other words, if I do install.packages("dplyr", lib="webr_packages"), R will only download the packages that are not already on my computer.

• There is no (sane) way to avoid the default library. So basically unless you’re ready to write hacky code that will override the library environments and stuff, R will always look into the default lib. There is an old issue on the {withr} repo if you want to look into this, but basically the idea is that avoiding default lib is quite a challenge.

• That means that if I try to do something as simple as install.packages("dplyr", lib="webr_packages", repos = "https://repo.r-wasm.org/"), it will only install {dplyr}, given that all the other dependencies already exist on my machine :

> setwd(tempdir())
> dir.create("webr_packages")
> install.packages("dplyr", lib="webr_packages", repos = "https://repo.r-wasm.org/")
# [...TRUNCATED]
* DONE (dplyr)
> list.files("webr_packages/")
[1] "dplyr"

I’ve tried the following:

• Hacking .lib.loc as suggest by Andrie here, but it comes with a series of challenges like not having access to other packages and having to reinstall them and load them to the webR FileSystem which is not what I wanted.

• Using {renv} but it feels a bit overkill for what I’m trying to do: download a bunch of zip files from the internet and unzip them in the correct folder. Also, {renv} failed to parse the dep tree in a NodeJS project for some reasons.

So I’ve ended up with my own R script, inspired by the internal of install.packages and webr::install (Read the source, Luke). I’ve first explored a version with {pak} and {pkgdepends}, but ended up using a base solution so that I don’t have to install anything in the Docker container.

#!/usr/bin/env Rscript

# This script only works if you pass it at least one arg
if (is.na(commandArgs(trailingOnly = TRUE)[1])) {
  stop("This script requires at least one argument")
} else {
  pk_to_install <- commandArgs(trailingOnly = TRUE)[1]
}

# If no second value, defaulting to webr_packages
if (is.na(commandArgs(trailingOnly = TRUE)[2])) {
  path_to_installation <- "./webr_packages"
} else {
  path_to_installation <- commandArgs(trailingOnly = TRUE)[2]
}

# Build the repos url
repos <- sprintf(
  "https://repo.r-wasm.org/bin/emscripten/contrib/%s.%s",
  # Get major R version
  R.version$major,
  substr(R.version$minor, 1, 1)
)

deps <- unique(
  unlist(
    use.names = FALSE,
    tools::package_dependencies(
      recursive = TRUE,
      pk_to_install
    )
  )
)

# Now the package list
pkg_deps <- c(
  pk_to_install,
  deps
)

# I don't want to re-download things so I'm listing all the already
# downloaded folders
already_installed <- list.files(
  path_to_installation
)

# Getting the list of available package
info <- utils::available.packages(contriburl = repos)

# Now we can install
for (pkg in pkg_deps){
  # Don't redownload things
  if (pkg %in% already_installed) {
    message("Package ", pkg, " already installed")
    next
  }
  # Not available: either it's not on the repo or it's included
  # in the base distribution of R
  if (!(pkg %in% info[, "Package"])) {
    message("Package {", pkg, "} not found in repo (unavailable or is base package)")
    next
  }
  message("Installing ", pkg)
  # Name of the targz, should be on the form of pkg_version.this.that.tgz
  targz_file <- paste0(pkg, "_", info[info[, "Package"] == pkg, "Version"], ".tgz")
  # Location of file on the repo
  url <- paste0(repos, "/", targz_file)
  # Download the file and untar the archive on the local lib
  tmp <- tempfile(fileext = ".tgz")
  download.file(url, destfile = tmp)
  untar(tmp, exdir = path_to_installation)
  unlink(tmp)
}

message("Done")

And now, let’s try this:

mkdir webr_packages
script tools/installR.r attempt
✔ Loading metadata database ... done
Installing attempt
trying URL 'https://repo.r-wasm.org/bin/emscripten/contrib/4.3/attempt_0.3.1.tgz'
Content type 'application/x-tar' length 93561 bytes (91 KB)
==================================================
downloaded 91 KB

Installing rlang
trying URL 'https://repo.r-wasm.org/bin/emscripten/contrib/4.3/rlang_1.1.1.tgz'
Content type 'application/x-tar' length 1062482 bytes (1.0 MB)
==================================================
downloaded 1.0 MB

Done

Rscript -e "fs::dir_tree('webr_packages')"
webr_packages
├── attempt
│   ├── DESCRIPTION
│   ├── INDEX
│   ├── LICENSE
│   ├── Meta
│   │   ├── Rd.rds
│   │   ├── features.rds
│   │   ├── hsearch.rds
│   │   ├── links.rds
│   │   ├── nsInfo.rds
│   │   ├── package.rds
│   │   └── vignette.rds
│   ├── NAMESPACE
│   ├── NEWS.md
│   ├── R
│   │   ├── attempt
│   │   ├── attempt.rdb
│   │   └── attempt.rdx
│   ├── doc
│   │   ├── a_intro_attempt.R
│   │   ├── a_intro_attempt.Rmd
│   │   ├── a_intro_attempt.html
│   │   ├── b_try_catch.R
│   │   ├── b_try_catch.Rmd
│   │   ├── b_try_catch.html
│   │   ├── c_adverbs.R
│   │   ├── c_adverbs.Rmd
│   │   ├── c_adverbs.html
│   │   ├── d_if.R
│   │   ├── d_if.Rmd
│   │   ├── d_if.html
│   │   ├── e_conditions.R
│   │   ├── e_conditions.Rmd
│   │   ├── e_conditions.html
│   │   └── index.html
│   ├── help
│   │   ├── AnIndex
│   │   ├── aliases.rds
│   │   ├── attempt.rdb
│   │   ├── attempt.rdx
│   │   └── paths.rds
│   └── html
│       ├── 00Index.html
│       └── R.css
└── rlang
    ├── DESCRIPTION
    ├── INDEX
    ├── LICENSE
    ├── Meta
    │   ├── Rd.rds
    │   ├── features.rds
    │   ├── hsearch.rds
    │   ├── links.rds
    │   ├── nsInfo.rds
    │   └── package.rds
    ├── NAMESPACE
    ├── NEWS.md
    ├── R
    │   ├── rlang
    │   ├── rlang.rdb
    │   └── rlang.rdx
    ├── backtrace-ver
    ├── help
    │   ├── AnIndex
    │   ├── aliases.rds
    │   ├── figures
    │   │   ├── lifecycle-archived.svg
    │   │   ├── lifecycle-defunct.svg
    │   │   ├── lifecycle-deprecated.svg
    │   │   ├── lifecycle-experimental.svg
    │   │   ├── lifecycle-maturing.svg
    │   │   ├── lifecycle-questioning.svg
    │   │   ├── lifecycle-retired.svg
    │   │   ├── lifecycle-soft-deprecated.svg
    │   │   ├── lifecycle-stable.svg
    │   │   ├── lifecycle-superseded.svg
    │   │   └── logo.png
    │   ├── paths.rds
    │   ├── rlang.rdb
    │   └── rlang.rdx
    ├── html
    │   ├── 00Index.html
    │   └── R.css
    └── libs
        └── rlang.so

Woot woot, seems to work 🎉.

Moving everything to the webR lib when the API is launched

And now, challenge number two: rebuilding this library in the webR filesystem.

webR has an interface to WASM filesystem (aka Emscripten Virtual File System), exposed via FS (doc is here). This interface allows to create a directory and write a file, which are the two things we want to do here:

• Create the package folders/subfolders
• Copy the files from the local machine (contained in a ./webr_packages folder) into the webR filesystem when launching the Node API.

Creating the directory tree

Creating a directory tree is not an easy task — it implies recursive function, keeping track of parents, and other weird things. R has list.files(recursive = TRUE) and we should be VERY happy about it: I looked on the internet and apparently there is no native way to do that in Node, you have to do it by hand.

As any sane modern JavaScript developer would do, I asked chatGPT to do it for me. It took me several iterations before getting the exact answer I was looking for but here it is.

Note that the JS functions below are bundled in the webrtools npm module.

const fs = require('fs');
const path = require('path');

// comments are mine

// Recursive function to get files, thank you chatGPT
function getDirectoryTree(dirPath, currentPath = '') {
  // Get a list of all the files/folders in the currently inspected folder
  const items = fs.readdirSync(path.join(dirPath, currentPath));
  // output
  const tree = [];

  // Looping on all the files/folders
  for (const item of items) {
    // Inspecting the item
    const itemPath = path.join(dirPath, currentPath, item);
    const isDirectory = fs.statSync(itemPath).isDirectory();
    const relativePath = path.join(currentPath, item);

    if (isDirectory) {
      // If it's a directory, we'll go inside and repeat the
      // current mechanism
      const subtree = getDirectoryTree(dirPath, relativePath);
      // pushing the current item, and its subtree
      tree.push({ path: relativePath, type: 'directory' });
      tree.push(...subtree);
    } else {
      tree.push({ path: relativePath, type: 'file' });
    }
  }
  // Getting the tree
  return tree;
}

Here is what it looks like on our ./webr_packages folder from before:

> getDirectoryTree("webr_packages")
[
  { path: 'attempt', type: 'directory' },
  { path: 'attempt/DESCRIPTION', type: 'file' },
  { path: 'attempt/INDEX', type: 'file' },
  { path: 'attempt/LICENSE', type: 'file' },
  { path: 'attempt/Meta', type: 'directory' },
  { path: 'attempt/Meta/Rd.rds', type: 'file' },
  { path: 'attempt/Meta/features.rds', type: 'file' },
  { path: 'attempt/Meta/hsearch.rds', type: 'file' },
  { path: 'attempt/Meta/links.rds', type: 'file' },
  { path: 'attempt/Meta/nsInfo.rds', type: 'file' },
  { path: 'attempt/Meta/package.rds', type: 'file' },
  { path: 'attempt/Meta/vignette.rds', type: 'file' },
  { path: 'attempt/NAMESPACE', type: 'file' },
  { path: 'attempt/NEWS.md', type: 'file' },
  { path: 'attempt/R', type: 'directory' },
  { path: 'attempt/R/attempt', type: 'file' },
  { path: 'attempt/R/attempt.rdb', type: 'file' },
  { path: 'attempt/R/attempt.rdx', type: 'file' },
  { path: 'attempt/doc', type: 'directory' },
  { path: 'attempt/doc/a_intro_attempt.R', type: 'file' },
  { path: 'attempt/doc/a_intro_attempt.Rmd', type: 'file' },
  { path: 'attempt/doc/a_intro_attempt.html', type: 'file' },
  { path: 'attempt/doc/b_try_catch.R', type: 'file' },
  { path: 'attempt/doc/b_try_catch.Rmd', type: 'file' },
  { path: 'attempt/doc/b_try_catch.html', type: 'file' },
  { path: 'attempt/doc/c_adverbs.R', type: 'file' },
  { path: 'attempt/doc/c_adverbs.Rmd', type: 'file' },
  { path: 'attempt/doc/c_adverbs.html', type: 'file' },
  { path: 'attempt/doc/d_if.R', type: 'file' },
  { path: 'attempt/doc/d_if.Rmd', type: 'file' },
  { path: 'attempt/doc/d_if.html', type: 'file' },
  { path: 'attempt/doc/e_conditions.R', type: 'file' },
  { path: 'attempt/doc/e_conditions.Rmd', type: 'file' },
  { path: 'attempt/doc/e_conditions.html', type: 'file' },
  { path: 'attempt/doc/index.html', type: 'file' },
  { path: 'attempt/help', type: 'directory' },
  { path: 'attempt/help/AnIndex', type: 'file' },
  { path: 'attempt/help/aliases.rds', type: 'file' },
  { path: 'attempt/help/attempt.rdb', type: 'file' },
  { path: 'attempt/help/attempt.rdx', type: 'file' },
  { path: 'attempt/help/paths.rds', type: 'file' },
  { path: 'attempt/html', type: 'directory' },
  { path: 'attempt/html/00Index.html', type: 'file' },
  { path: 'attempt/html/R.css', type: 'file' },
  { path: 'rlang', type: 'directory' },
  { path: 'rlang/DESCRIPTION', type: 'file' },
  { path: 'rlang/INDEX', type: 'file' },
  { path: 'rlang/LICENSE', type: 'file' },
  { path: 'rlang/Meta', type: 'directory' },
  { path: 'rlang/Meta/Rd.rds', type: 'file' },
  { path: 'rlang/Meta/features.rds', type: 'file' },
  { path: 'rlang/Meta/hsearch.rds', type: 'file' },
  { path: 'rlang/Meta/links.rds', type: 'file' },
  { path: 'rlang/Meta/nsInfo.rds', type: 'file' },
  { path: 'rlang/Meta/package.rds', type: 'file' },
  { path: 'rlang/NAMESPACE', type: 'file' },
  { path: 'rlang/NEWS.md', type: 'file' },
  { path: 'rlang/R', type: 'directory' },
  { path: 'rlang/R/rlang', type: 'file' },
  { path: 'rlang/R/rlang.rdb', type: 'file' },
  { path: 'rlang/R/rlang.rdx', type: 'file' },
  { path: 'rlang/backtrace-ver', type: 'file' },
  { path: 'rlang/help', type: 'directory' },
  { path: 'rlang/help/AnIndex', type: 'file' },
  { path: 'rlang/help/aliases.rds', type: 'file' },
  { path: 'rlang/help/figures', type: 'directory' },
  { path: 'rlang/help/figures/lifecycle-archived.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-defunct.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-deprecated.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-experimental.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-maturing.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-questioning.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-retired.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-soft-deprecated.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-stable.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-superseded.svg', type: 'file' },
  { path: 'rlang/help/figures/logo.png', type: 'file' },
  { path: 'rlang/help/paths.rds', type: 'file' },
  { path: 'rlang/help/rlang.rdb', type: 'file' },
  { path: 'rlang/help/rlang.rdx', type: 'file' },
  { path: 'rlang/html', type: 'directory' },
  { path: 'rlang/html/00Index.html', type: 'file' },
  { path: 'rlang/html/R.css', type: 'file' },
  { path: 'rlang/libs', type: 'directory' },
  { path: 'rlang/libs/rlang.so', type: 'file' }
]

Note: I need all the folders and subfolders, for example rlang/libs and rlang/libs/rlang.so, because I haven’t found a way to do a recursive mkdir in webR, and had to create all children level by level.

🚀 Yeay 🚀

Now I have a JavaScript object I can use to move my ./webr_packages library to a webR instance library.

Let’s write a function for that (no need for chatGPT here 😅):

async function loadPackages(webR, dirPath){
  // Our function from before
  const files = getDirectoryTree(
    dirPath
  )
  for await (const file of files) {
    // If the entry is a directory, we'll
    // call mkdir
    if (file.type === 'directory') {
      await globalThis.webR.FS.mkdir(
        `/usr/lib/R/library/${file.path}`,
      );
    } else {
    // case 2, the entry is a file, then we can use fs.readFileSync
    // That will return a ArrayBuffer that can be directly passed
    // to webR.FS.writeFile
      const data = fs.readFileSync(`webr_packages/${file.path}`);
      await globalThis.webR.FS.writeFile(
        `/usr/lib/R/library/${file.path}`,
        data
      );
    }
  }
}

Now we have our whole process:

• downloading the precompiled packages
• listing the dir tree in Node
• bringing everything into the webR filesystem

🚀 Yeay (bis) 🚀

To make things simple, I have put these into a node package, at npmjs.com/package/webrtools.

Bringing everything together

Workflow for building the app:

cd webr-example
mkdir webr-preloadr
cd webr-preloadr
npm init -y
npm install express webr webrtools
Rscript ./node_modules/webrtools/r/install.R dplyr
touch index.js

Then :

const app = require('express')()
const path = require('path');
const { loadPackages } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  console.log("📦 Packages written to webR 📦");

  const res = await globalThis.webR.FS.lookupPath("/usr/lib/R/library");
  console.log(res)

  await globalThis.webR.evalR("library(dplyr)");
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();


app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR('sample_n(mtcars, 10)');
  let output = await result.toJs();
  res.send(output.values)
});

node index.js

And in another console:

$ curl http://localhost:3000
[{"type":"double","names":null,"values":[21,24.4,14.3,32.4,18.7,17.8,22.8,10.4,30.4,21.4]},{"type":"double","names":null,"values":[6,4,8,4,8,6,4,8,4,4]},{"type":"double","names":null,"values":[160,146.7,360,78.7,360,167.6,108,460,75.7,121]},{"type":"double","names":null,"values":[110,62,245,66,175,123,93,215,52,109]},{"type":"double","names":null,"values":[3.9,3.69,3.21,4.08,3.15,3.92,3.85,3,4.93,4.11]},{"type":"double","names":null,"values":[2.62,3.19,3.57,2.2,3.44,3.44,2.32,5.424,1.615,2.78]},{"type":"double","names":null,"values":[16.46,20,15.84,19.47,17.02,18.9,18.61,17.82,18.52,18.6]},{"type":"double","names":null,"values":[0,1,0,1,0,1,1,0,1,1]},{"type":"double","names":null,"values":[1,0,0,1,0,0,1,0,1,1]},{"type":"double","names":null,"values":[4,4,3,4,3,4,4,3,4,4]},{"type":"double","names":null,"values":[4,2,4,1,2,4,1,4,2,2]}]

🚀 Yeay (ter) 🚀

And now for the Dockerfile

FROM r-base:4.3.1

# Create app directory
WORKDIR /usr/src/app

RUN apt update && \
  apt install nodejs npm -y

COPY package*.json ./

RUN npm install

RUN Rscript ./node_modules/webrtools/r/install.R dplyr

COPY . .

EXPOSE 3000

CMD [ "node", "index.js" ]

You can find the code here, and see it live at srv.colinfay.me/webr-preloadr.

You can also try it with

docker run -it -p 3000:3000 colinfay/webr-preloadr

Final notes

Should I use it in prod?

Probably not — for now it’s more of an experiment than a real bullet-proof approach. But if you’re a reader from the future, many things may have happen, and maybe the webrtools node module is now prod ready. Who knows!

Further work

After some discussion on the issue on the webR repo, it seems that a promising path to do what I’d like to do is to use the FS.mount interface from WASM, but it needs to be expose by webR. Keep an eye on the repo!

Finally, I’m a bit displeased with writing the installR fun in R and having to install R in the Dockerfile to do it, so I should probably re-write it in full NodeJS. We could read the package tree in R with webR and then download/untar with Node.

Food for thoughts.

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Note 2: webR and the tools I’ve been writing are moving fast and if you’re reading this from the future, some of the things here might be obsolete.

In this post, I’ll attempt to recreate a version of the famous Old Faithful Geyser Data {shiny} app using webR, Bootstrap & ExpressJS. (If you really don’t know which app I’m talking about, it’s this one.)

Introductory notes

Before starting, here are two notes regarding the app built in this post:

• this app could have been build in full “web mode” (no server required, just an HTML page), but this is not the approach I’m currently experimenting with. Going full browser-based doesn’t work for all cases, and most of the time with production apps you’ll need some part of your code to be computed by the server (because of resources, because you’ll connect to API with token, because you need access to DB with passwords, because you don’t want the full data to be available in the brower, or many other good reason…).

• In order to recreate the app, my first approach was to try to draw the histogram in base R and send it back to the browser. boB has a great blogpost about how to do exactly that, but after a lot of bad code and good 4 letter words, I realized there was no sane reason for me to display a base R plot instead of a JavaScript based one. That’s why I chose to return the data for the barplot (using the cut() function from R) and draw with chart.js, instead of trying to display the “not so user friendly” base plot.

Project init

Let’s start by creating a new Express app:

mkdir express-webr-old-faithful
cd express-webr-old-faithful
npm init -y
# Installing the deps we'll need
npm i @r-wasm/webr express
# Creating a server, and the front page
touch index.js
touch index.html

Server

Let’s move to the server side first (index.js ). We’ll start by taking the file from the previous blog post, and modify it to:

• serve index.html on /
• create a route that returns the data of the bins for our histogram

Let’s start with our code to init webR.

'use strict';
const express = require("express")
// For serving the html
const path = require("path")
const app = express()
const { WebR } = require('@r-wasm/webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  // Given that we will reuse this value,
  // we assign it at launch
  await globalThis.webR.evalR('x <- faithful[, 2]')
  console.log("webR is ready");
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();

Then, an endpoint that serves index.html:

app.get("/", (req, res) => {
  res.sendFile(path.join(__dirname, "index.html"))
})

And an endpoint that sends the bins for our plot, using ExpressJS parameter notation (path/:n). The value is sent to webR via the env option of evalR

app.get("/hist-data/:n", async (req, res) => {
  let result = await globalThis.webR.evalR(
    'table(cut(x, seq(min(x), max(x), length.out = n + 1)))',
    { env: { n: parseInt(req.params.n) } }
  );
  let output = await result.toJs();
  res.send(output)
})

Now that our backend is ready, let’s check that we can now call it from the command line:

curl http://localhost:3000/hist-data/10
{"type":"integer","names":["(43,48.3]","(48.3,53.6]","(53.6,58.9]","(58.9,64.2]","(64.2,69.5]","(69.5,74.8]","(74.8,80.1]","(80.1,85.4]","(85.4,90.7]","(90.7,96]"],"values":[15,28,26,24,9,23,62,55,23,6]}

Front

Now, time to build the front.

We’ll start with the Boostrap boilerplate from https://getbootstrap.com/docs/5.3/getting-started/introduction/#quick-start.

Note also that I’ve chosen (for simplicity’s sake) to use the CDN version of the external deps, instead of installing them in my Node project, which would be what I would do in a normal context.

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Bootstrap demo</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-9ndCyUaIbzAi2FUVXJi0CjmCapSmO7SnpJef0486qhLnuZ2cdeRhO02iuK6FUUVM" crossorigin="anonymous">
  </head>
  <body>
    <h1>Hello, world!</h1>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js" integrity="sha384-geWF76RCwLtnZ8qwWowPQNguL3RmwHVBC9FhGdlKrxdiJJigb/j/68SIy3Te4Bkz" crossorigin="anonymous"></script>
  </body>
</html>

Let’s change the title, and add:

• the range input
• a div to receive the chart

<div class="container">
  <h1>Old Faithful Geyser Data</h1>
  <!-- Bootstrap grid system -->
  <div class="row align-items-start">
    <div class="col-4">
      <label for="customRange1" class="form-label">Number of bins:</label>
      <input type="range" class="form-range" id="customRange1" min=1 max=30 value=10>
      <div id="bins">Selected: 10</div>
    </div>
    <div class="col-8">
      <div>
        <!-- Where we'll get the chart drawn -->
        <canvas id="myChart"></canvas>
      </div>
    </div>
  </div>
</div>

In order to draw the graph, I’ll rely on a dead simple (yet powerful) JavaScript lib called Chart.js. It has a great bar chart graph that will work perfectly for our case.

Let’s start by adding the lib with <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>.

We then need some JavaScript to:

• Initiate the chart at launch

// https://colinfay.me/api-from-client-shiny/
// Default to 10 bins
fetch("hist-data/10")
      .then((data) =>{
        // Convert the data to json and
        // create a chart
        data.json().then((res) => {
          // Keeping a global object with the chart
          globalThis.chart = new Chart(
          // This is where the chart will go
          document.getElementById('myChart'), {
          type: 'bar',
          data: {
            // webR returns the names
            labels: res.names,
            datasets: [{
              label: "Histogram of waiting times",
              data: res.values
            }]
          }
        });
        })
        .catch((error) => {
          alert("Error catchin result from R")
        })
      })
      .catch((error) => {
        alert("Error catchin result from R")
      })

• Update the chart when the slider is moved

    const update = function (n = 10) {
      fetch(`hist-data/${n}`).then((data) => {
        data.json().then((res) => {
          globalThis.chart.data.labels = res.names;
          globalThis.chart.data.datasets.forEach(dataset => {
            dataset.data = res.values;
          })
          globalThis.chart.update();
        })
      })
      document.querySelector('#bins').innerHTML = `Selected: ${n}`;

    }
document.querySelector('#customRange1').addEventListener(
  'change',
  function() { update(this.value); }
 );

And here it is! You can see the app live at srv.colinfay.me/express-webr-old-faithful. You can find the code here.

You can also try it with:

docker run -it -p 3000:3000 colinfay/express-webr-old-faithful

Note: webR and the tools I’ve been writing are moving fast and if you’re reading this from the future, some of the things here might be obsolete.

webR? wat again?

As described in the doc:

WebR is a version of the statistical language R compiled for the browser and Node.js using WebAssembly, via Emscripten.

In this post, I won’t go into details into what webR is, as if you want to know more, you’ll probably get a better understanding by going through the documentation.

But if like a normal person you don’t read the doc, let’s sum up what webR is. Simply put, WebAssembly (shorten wasm) is a format that can be used to encode a programming language to something that can then be called from a JavaScript runtime (in the browser or in NodeJS). Schematically, you take one language, you translate it to wasm, and the output can be read and used from JavaScript.

If you still have no idea what wasm is but want to get a better grasp on what it is, I suggest reading this blogpost that really sums it well, especially this quote:

WASM is a way to use non-Javascript code and run it in your browser.

I also suggest reading the blogposts by boB on the subject.

The rest of this post assume that you understand what wasm & webR are.

Playing with webR

I’ve been interested with WebAssembly for a while now, and can remember talking about it late during the night at R conferences around 2018 and 2019… but that’s a story for another time 😅

Today’s story is about exploring how to build tools in JavaScript that will call webR.

I’ll try to document my experimentations, and this post is the first one of a series that I hope I’ll find time to continue over the summer.

Hello World! - Creating an API where we call webR

I’ll start with a simple Express JS REST API that uses webR to run R code.

# Initiating a projet and opening it with VSCode
mkdir webrexpresshelloworld
code webrexpresshelloworld

# Quick init of a node project
npm init -y
touch index.js
npm install express

We’ll now install the webR nodejs package

npm i @r-wasm/webr

And now, let’s move to our index.js file. We’ll start with the basics of Express JS: creating the app object.

// Creating the express app
const app = require('express')()

Then, we’ll import WebR from @r-wasm/webr.

// importing the `WebR` class inside the runtime
const { WebR } = require('@r-wasm/webr');

We’ll now be initiating the webR session by:

• creating an instance of webR
• binding it to globalThis so that it’s available everywhere
• calling the init method
• Once webR is ready, we’ll launch the app

As webR communicates via the worker thread asynchronously, we need to use async/await.

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  console.log("webR is ready");
  // Starting the express app
  // only after webR is ready
  app.listen(3000, () => {
    console.log('http://localhost:3000')
  })
})();

Finally, we add a route that returns an hello world:

// Creating a route for the express app
// that will return a titleCase hello world from r
app.get('/', async (req, res) => {
  // Evaluating the R code inside the R worker
  let result = await globalThis.webR.evalR('tools::toTitleCase("hello from r!")');
  // Converting the result to a JS object
  let output = await result.toJs();
  // Sending the result back to the client,
  // (webR outputs an object with types/names/values
  // and here we only want the values)
  res.send(output.values)
});

And now, let’s run our app!

node index.js

You can find the code here, and see it live at srv.colinfay.me/express-webr-hello-world.

You can also try it with

docker run -it -p 3000:3000 colinfay/express-webr-hello-world

In this talk, I reflect on how ultra running has helped me become a better software engineer, and how building software has helped me learning how to run long distances.

Giving this talk has been something quite hard for me (probably harder than running 50 miles 😅) — I usually give very technical talks, but for one time decided I should try telling a bit more about me and my story, about why I love running long distances and building software that are used in production. I hope watching this one will inspire some of you, and feel free to share your thoughts about it with me!

Thanks again to the Apsillon team for inviting me to talk, to Vincent and Diane from ThinkR for their support, and to Maxime for coaching me and pushing me toward being more open about myself.

Like yesterday, trying to write JS code that looks like R code.

Disclaimer

Disclaimer Obviously, this post contains a big spoiler about Advent of Code, as it gives solutions for solving 2021-02.

[Disclaimer bis] I’m no JavaScript expert so this might not be the perfect solution. TBH, that’s also the case for the R solution.

About the JavaScript code

The JavaScript code has been written in the same RMarkdown as the R code. It runs thanks to the {bubble} package: https://github.com/ColinFay/bubble.

bubble::set_node_engine()

Instructions

One

• We start at a position [0, 0] (horizontal, depth), then
  • forward X increases the horizontal position by X units
  • down X increases the depth by X units
  • up X decreases the depth by X units

Result is horizontal * depth.

Two

• We start at a position [0, 0, 0] (horizontal, depth,), then
  • down X increases your aim by X units
  • up X decreases your aim by X units
  • forward X does two things :
    • It increases your horizontal position by X units
    • It increases your depth by your aim multiplied by X

Result is horizontal * depth.

Find the complete instructions at: https://adventofcode.com/2021/day/2.

Reading inputs

# Read
ipt <- read.delim("2021-02-aoc.txt", header = FALSE)$V1

const fs = require('fs')
var ipt = fs.readFileSync("2021-02-aoc.txt", 'utf8').split("\n").filter(x => x.length != 0);

Part one

R solution

# Horizontal and depth
start_pos <- c(horizontal = 0, depth = 0)
for (moves in ipt) {
  move <- strsplit(moves, " ")[[1]]

  if (move[1] == "up") {
    start_pos[2] <- start_pos[2] - as.numeric(move[2])
  }
  if (move[1] == "down") {
    start_pos[2] <- start_pos[2] + as.numeric(move[2])
  }
  if (move[1] == "forward") {
    start_pos[1] <- start_pos[1] + as.numeric(move[2])
  }
}

start_pos[1] * start_pos[2]

## horizontal
##    1727835

JS solution

var start_pos = [0, 0]

// Gotcha: R for(x in y) is JS for(x of y)

for (moves of ipt) {
  // In JS, object have methods, so the split function
  // is containes inside the string object
  move = moves.split(' ')

  // Small gotcha here, JS i 0 based index
  if (move[0] == "up") {
    // R as.numeric is parseInt()
    start_pos[1] = start_pos[1] - parseInt(move[1])
  }
  if (move[0] == "down") {
    start_pos[1] = start_pos[1] + parseInt(move[1])
  }
  if (move[0] == "forward") {
    start_pos[0] = start_pos[0] + parseInt(move[1])
  }
}

start_pos[0] * start_pos[1]

## 1727835

Part two

R solution

# Horizontal, depth, and aim
start_pos <- c(horizontal = 0, depth = 0, aim = 0)
for (moves in ipt) {
  move <- strsplit(moves, " ")[[1]]

  if (move[1] == "up") {
    start_pos[3] <- start_pos[3] - as.numeric(move[2])
  }
  if (move[1] == "down") {
    start_pos[3] <- start_pos[3] + as.numeric(move[2])
  }
  if (move[1] == "forward") {
    start_pos[1] <- start_pos[1] + as.numeric(move[2])
    start_pos[2] <- start_pos[2] + (start_pos[3] * as.numeric(move[2]))
  }
}

start_pos[1] * start_pos[2]

## horizontal
## 1544000595

JS solution

var start_pos = [0, 0, 0]

for (moves of ipt) {
  move = moves.split(' ')
  if (move[0] == "up") {
    start_pos[2] = start_pos[2] - parseInt(move[1])
  }
  if (move[0] == "down") {
    start_pos[2] = start_pos[2] + parseInt(move[1])
  }
  if (move[0] == "forward") {
    start_pos[0] = start_pos[0] + parseInt(move[1])
    start_pos[1] = start_pos[1] + (start_pos[2] * parseInt(move[1]))
  }
}

start_pos[0] * start_pos[1]

## 1544000595

I’ll try in this post to write R code that looks like the JavaScript one, using {purrr}, to show how both syntaxes can look alike. The idea is to highlight how you can leverage your {purrr} knowledge when writting JavaScript code.

Disclaimer

Disclaimer Obviously, this post contains a big spoiler about Advent of Code, as it gives solutions for solving day 1.

[Disclaimer bis] I’m no JavaScript expert so this might not be the perfect solution. TBH, that’s also the case for the R solution.

About the JavaScript code

The JavaScript code has been written in the same RMarkdown as the R code. It runs thanks to the {bubble} package: https://github.com/ColinFay/bubble

Instructions

• Count the number of times n is larger that n-1

• Count the number of times n + n-1 + n-2 is larger that n-1 + n-2 + n-3

Find the complete instructions at: https://adventofcode.com/2021/day/1.

Reading inputs

# Read
library(purrr)
ipt <- read.delim("2021-01-aoc.txt", header = FALSE)$V1

// Reading the file and converting it to int

const fs = require('fs')
var ipt = fs.readFileSync("2021-01-aoc.txt", 'utf8').split("\n").filter(x => x.length != 0);
ipt = ipt.map(x => parseInt(x));

Part one

R solution

2:length(ipt) |>
  map_dbl(
    ~ ipt[.x] - ipt[.x - 1]
  ) |>
  keep(
    ~ .x > 0
  ) |>
  length()

## [1] 1154

JS solution

ipt.map(
  // inside map() in JS, the second argument to the anonymous
  // function is the index inside the array.
  // It mimics what we have in map(2:length(ipt)) in R
  (x, y) => ipt[y] - ipt[y-1]
  ).filter(
    // filter() will keep the element that matches the anonymous
    // predicate function
    x => x >= 0
    ).length;

## 1154

Part two

R solution

tmp <- 3:length(ipt) |>
  map_dbl(
    ~ ipt[.x] + ipt[.x - 1] + ipt[.x - 2]
  )

2:length(tmp) |>
  map_dbl(
    ~ tmp[.x] - tmp[.x - 1]
  ) |>
  keep(
    ~ .x > 0
  ) |>
  length()

## [1] 1127

JS solution

var tmp = ipt.
  map(
    // Creating n + n-1 + n-2
    (x, y) => ipt[y] + ipt[y-1] + ipt[y-2]
    );

tmp.
  map(
    // Same as part 1
    (x, y) => tmp[y] - tmp[y-1]
    ).
  filter(
    // keeping the ones that have raised
    x => x > 0
  ).
  length;

## undefined
## undefined
## 1127