In this repository I show a practical solution to massive distributed computing in R. I have successfully tested this approach in my research with ~220 remote hosts (and ~800 CPUs) on computers located in [Faculty of Mathematics and Computer Science, Adam Mickiewicz University in Poznań](https://translate.google.com/translate?hl=pl&sl=pl&tl=en&u=https://laboratoria.wmi.amu.edu.pl/Laboratoria). Brief examples concern [caret](https://topepo.github.io/caret/) package for creating predictive models and [foreach](https://cran.r-project.org/web/packages/doParallel/index.html) loop for more general parallel computing.
The bash script `remote-commands.sh` is responsible for the server and hosts preparation for calculations. The script reads list of remote hosts from either stdin or file set in `CONNECTION_LIST_FILE` script variable (default: `remote-connection-list.txt`).
Then the script executes internal procedures; the names and execution order of such functions are passed as script arguments, e.g. `bash remote-commands.sh configure_hosts`.
The `configure_hosts` is a short name for basic execution order:
1.`generate_ssh_keys` - generates ssh keys to communicate with hosts (see `SSH_*` variables),
1.`hosts_push_ssh_key` - pushes the keys to hosts and disables password authentication,
1.`hosts_install_mro` - installs R defined in `MRO_*` variables,
1.`hosts_push_r_libraries_dump` - pushes R project files to hosts,
1.`make_remote_connection_list_nproc` - creates connection list file for R, defined in `HOSTS_FILE` (default: `remote-hosts.txt`).
You can also scan and limit hosts to those currently available through `hosts_scan_available` procedure and check if there are at least `MIN_HOSTS` available hosts.
Instead of pushing to hosts precompiled R libraries (`hosts_push_r_libraries_dump`) you can force compiling them on the hosts (`hosts_install_r_libraries`).
You may customize `MRO_INSTALL_URL` to your own mirror - from my experience the MRAN servers limit the download bandwidth in case.
If you want to make only one connection per node (regardless number of cores), you can execute `make_remote_connection_list_single` instead of `make_remote_connection_list_nproc`. You may also try to detect logical cores through setting `REMOTE_DETECT_LOGICAL_CPUS`.
Alternatively, instead of installing soft on [WMI rescue](http://rescue.wmi.amu.edu.pl), you can create and boot on hosts your own customized distro.
You can investigate hosts install logs through `hosts_check_install_log_*` functions. You can also check and clean remote worker logs through `hosts_check_worker_log` and `hosts_clean_worker_log` procedures, respectively. Moreover, you can investigate dmesg through `hosts_check_worker_dmesg`. The hosts can be invoked to power off through `hosts_power_off`.
In the `config.R` file you may customize `checkpoint` library configuration and the logging system.
In the `config-parallel.R` you can:
* switch between parallel computing methods (`PARALLEL.USED.METHOD` by default is `REMOTE` but for testing you can change it to `LOCAL`, so that all calculations will be done on the server); for `LOCAL` the method defined in `PARALLEL.LOCAL.METHOD` might be `PSOCK` (default) or `FORK`.
* customize remote R session invoking parameters through `PARALLEL.REMOTE.*` variables,
* worker logs file names are defined in `PARALLEL.*.SLAVE.OUT.FILE` and are unique among whole computing cluster.
Currently R has a hard-coded limit for number of connections, which is effectively 125. If you want to work with more connections (nodes), then on the server you have to recompile and install customized R. You may see how to set custom limit in my repository [Microsoft R Open compilation for customized cluster nodes connection limit](https://github.com/andre-wojtowicz/r-compile-customised-mro).
In caret you can control reproducibility by pre-setting seeds; see section [5.4 Notes on Reproducibility](https://topepo.github.io/caret/model-training-and-tuning.html#repro).
By default the Intel MKL support is installed with Microsoft R Open. In `config-parallel.R` you can choose in `PARALLEL.DISABLE.MKL.THREADS` variable either this library should be either disabled (default) or enabled. This can be useful in particular if you make cluster in the one-host-one-connection manner and calculations on remote nodes can benefit from [BLAS routines](https://github.com/andre-wojtowicz/blas-benchmarks).