makeClusterPSOCK {future} | R Documentation |
The makeClusterPSOCK()
function creates a cluster of R workers
for parallel processing. These R workers may be background R sessions
on the current machine, R sessions on external machines (local or remote),
or a mix of such. For external workers, the default is to use SSH to connect
to those external machines. This function works similarly to
makePSOCKcluster
of the
parallel package, but provides additional and more flexibility options
for controlling the setup of the system calls that launch the background
R workers, and how to connect to external machines.
makeClusterPSOCK(workers, makeNode = makeNodePSOCK, port = c("auto", "random"), ..., verbose = getOption("future.debug", FALSE)) makeNodePSOCK(worker = "localhost", master = NULL, port, connectTimeout = getOption("future.makeNodePSOCK.connectTimeout", 2 * 60), timeout = getOption("future.makeNodePSOCK.timeout", 30 * 24 * 60 * 60), rscript = NULL, homogeneous = NULL, rscript_args = NULL, methods = TRUE, useXDR = TRUE, outfile = "/dev/null", renice = NA_integer_, rshcmd = getOption("future.makeNodePSOCK.rshcmd", NULL), user = NULL, revtunnel = TRUE, rshopts = getOption("future.makeNodePSOCK.rshopts", NULL), rank = 1L, manual = FALSE, dryrun = FALSE, verbose = FALSE)
workers |
The hostnames of workers (as a character vector) or the number of localhost workers (as a positive integer). |
makeNode |
A function that creates a |
port |
The port number of the master used to for communicating with all
the workers (via socket connections). If an integer vector of ports, then a
random one among those is chosen. If |
... |
Optional arguments passed to
|
verbose |
If TRUE, informative messages are outputted. |
worker |
The hostname or IP number of the machine where the worker should run. |
master |
The hostname or IP number of the master / calling machine, as
known to the workers. If NULL (default), then the default is
|
connectTimeout |
The maximum time (in seconds) allowed for each socket connection between the master and a worker to be established (defaults to 2 minutes). See note below on current lack of support on Linux and macOS systems. |
timeout |
The maximum time (in seconds) allowed to pass without the master and a worker communicate with each other (defaults to 30 days). |
rscript, homogeneous |
The system command for launching |
rscript_args |
Additional arguments to |
methods |
If TRUE, then the methods package is also loaded. |
useXDR |
If TRUE, the communication between master and workers, which is binary, will use big-endian (XDR). |
outfile |
Where to direct the stdout and stderr connection output from the workers. |
renice |
A numerical 'niceness' (priority) to set for the worker processes. |
rshcmd, rshopts |
The command (character vector) to be run on the master
to launch a process on another host and any additional arguments (character
vector). These arguments are only applied if |
user |
(optional) The user name to be used when communicating with another host. |
revtunnel |
If TRUE, a reverse SSH tunnel is set up for each worker such
that the worker R process sets up a socket connection to its local port
|
rank |
A unique one-based index for each worker (automatically set). |
manual |
If TRUE the workers will need to be run manually. The command to run will be displayed. |
dryrun |
If TRUE, nothing is set up, but a message suggesting how to launch the worker from the terminal is outputted. This is useful for troubleshooting. |
An object of class c("SOCKcluster", "cluster")
consisting
of a list of "SOCKnode"
or "SOCK0node"
workers.
makeNodePSOCK()
returns a "SOCKnode"
or
"SOCK0node"
object representing an established connection to a worker.
A hostname is considered to be localhost if it equals:
"localhost"
,
"127.0.0.1"
, or
Sys.info()[["nodename"]]
.
It is also considered localhost if it appears on the same line
as the value of Sys.info()[["nodename"]]
in file ‘/etc/hosts’.
rshcmd
and rshopts
Arguments rshcmd
and rshopts
are only used when connecting
to an external host.
The default method for connecting to an external host is via SSH and the
system executable for this is given by argument rshcmd
. The default
is given by option future.makeNodePSOCK.rshcmd
and if that is not
set the default is either of ssh
and plink -ssh
.
Most Unix-like systems, including macOS, have ssh
preinstalled
on the PATH
. This is also true for recent Windows 10
(since version 1803; April 2018).
Furthermore, when running R from RStudio on Windows, the ssh
client that is distributed with RStudio will be used as a fallback if
neither of the above two commands are available on the PATH
.
For Windows systems prior to Windows 10, which do not have RStudio
installed, it is less common to find ssh
. Instead it is more
likely that such systems have the PuTTY
software and its SSH
client plink
installed.
If no SSH-client is found, an informative error message is produced.
It is also possible to specify the absolute path to the SSH client. To do
this for PuTTY, specify the absolute path in the first element and option
-ssh
in the second as in
rshcmd = c("C:/Path/PuTTY/plink.exe", "-ssh")
.
This is because all elements of rshcmd
are individually "shell"
quoted and element rshcmd[1]
must be on the system PATH
.
Additional SSH options may be specified via argument rshopts
, which
defaults to option future.makeNodePSOCK.rshopts
. For instance, a
private SSH key can be provided as
rshopts = c("-i", "~/.ssh/my_private_key")
. PuTTY users should
specify a PuTTY PPK file, e.g.
rshopts = c("-i", "C:/Users/joe/.ssh/my_keys.ppk")
.
Contrary to rshcmd
, elements of rshopts
are not quoted.
The default is to use reverse SSH tunneling (revtunnel = TRUE
) for
workers running on other machines. This avoids the complication of
otherwise having to configure port forwarding in firewalls, which often
requires static IP address as well as privileges to edit the firewall,
something most users don't have.
It also has the advantage of not having to know the internal and / or the
public IP address / hostname of the master.
Yet another advantage is that there will be no need for a DNS lookup by the
worker machines to the master, which may not be configured or is disabled
on some systems, e.g. compute clusters.
rscript
If homogeneous
is FALSE, the rscript
defaults to
"Rscript"
, i.e. it is assumed that the Rscript
executable
is available on the PATH
of the worker.
If homogeneous
is TRUE, the rscript
defaults to
file.path(R.home("bin"), "Rscript")
, i.e. it is basically assumed
that the worker and the caller share the same file system and R installation.
homogeneous
The default value of homogeneous
is TRUE if and only if either
of the following is fulfilled:
worker
is localhost
revtunnel
is FALSE and master
is localhost
worker
is neither an IP number nor a fully qualified domain
name (FQDN). A hostname is considered to be a FQDN if it contains
one or more periods
Argument connectTimeout
does not work properly on Unix and
macOS due to limitation in R itself. For more details on this, please see
R-devel thread 'BUG?: On Linux setTimeLimit() fails to propagate timeout
error when it occurs (works on Windows)' on 2016-10-26
(https://stat.ethz.ch/pipermail/r-devel/2016-October/073309.html).
When used, the timeout will eventually trigger an error, but it won't happen
until the socket connection timeout timeout
itself happens.
If there is no communication between the master and a worker within the
timeout
limit, then the corresponding socket connection will be
closed automatically. This will eventually result in an error in code
trying to access the connection.
## Setup of three R workers on two remote machines are set up workers <- c("n1.remote.org", "n2.remote.org", "n1.remote.org") cl <- makeClusterPSOCK(workers, dryrun = TRUE) ## Same setup when the two machines are on the local network and ## have identical software setups cl <- makeClusterPSOCK( workers, revtunnel = FALSE, homogeneous = TRUE, dryrun = TRUE ) ## Setup of remote worker with more detailed control on ## authentication and reverse SSH tunnelling cl <- makeClusterPSOCK( "remote.server.org", user = "johnny", ## Manual configuration of reverse SSH tunnelling revtunnel = FALSE, rshopts = c("-v", "-R 11000:gateway:11942"), master = "gateway", port = 11942, ## Run Rscript nicely and skip any startup scripts rscript = c("nice", "/path/to/Rscript"), rscript_args = c("--vanilla"), dryrun = TRUE ) ## Setup of 2 Docker workers running rocker/r-base ## (requires installation of future package) cl <- makeClusterPSOCK( rep("localhost", times = 2L), ## Launch Rscript inside Docker container rscript = c( "docker", "run", "--net=host", "rocker/r-base", "Rscript" ), ## Install future package rscript_args = c( "-e", shQuote("install.packages('future')") ), dryrun = TRUE ) ## Setup of a single udocker.py worker running rocker/r-base ## (requires installation of future package and extra quoting) cl <- makeClusterPSOCK( "localhost", ## Launch Rscript inside Docker container (using udocker) rscript = c( "udocker.py", "run", "rocker/r-base", "Rscript" ), ## Install future package and manually launch parallel workers ## (need double shQuote():s because udocker.py drops one level) rscript_args = c( "-e", shQuote(shQuote("install.packages('future')")), "-e", shQuote(shQuote("parallel:::.slaveRSOCK()")) ), dryrun = TRUE ) ## Launching worker on Amazon AWS EC2 running one of the ## Amazon Machine Images (AMI) provided by RStudio ## (http://www.louisaslett.com/RStudio_AMI/) public_ip <- "1.2.3.4" ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem" cl <- makeClusterPSOCK( ## Public IP number of EC2 instance public_ip, ## User name (always 'ubuntu') user = "ubuntu", ## Use private SSH key registered with AWS rshopts = c( "-o", "StrictHostKeyChecking=no", "-o", "IdentitiesOnly=yes", "-i", ssh_private_key_file ), ## Set up .libPaths() for the 'ubuntu' user and ## install future package rscript_args = c( "-e", shQuote("local({ p <- Sys.getenv('R_LIBS_USER') dir.create(p, recursive = TRUE, showWarnings = FALSE) .libPaths(p) })"), "-e", shQuote("install.packages('future')") ), dryrun = TRUE ) ## Launching worker on Google Cloud Engine (GCE) running a ## container based VM (with a #cloud-config specification) public_ip <- "1.2.3.4" user <- "johnny" ssh_private_key_file <- "~/.ssh/google_compute_engine" cl <- makeClusterPSOCK( ## Public IP number of GCE instance public_ip, ## User name (== SSH key label (sic!)) user = user, ## Use private SSH key registered with GCE rshopts = c( "-o", "StrictHostKeyChecking=no", "-o", "IdentitiesOnly=yes", "-i", ssh_private_key_file ), ## Launch Rscript inside Docker container rscript = c( "docker", "run", "--net=host", "rocker/r-base", "Rscript" ), ## Install future package rscript_args = c( "-e", shQuote("install.packages('future')") ), dryrun = TRUE ) ## Connect to remote Unix machine 'remote.server.org' on port 2200 ## as user 'bob' from a Windows machine with PuTTY installed cl <- makeClusterPSOCK( "remote.server.org", user = "bob", rshopts = c("-P", 2200, "-i", "C:/Users/bobby/.ssh/putty.ppk"), dryrun = TRUE )