Parallelisation guidance
Emil Pedersen
2026-11-19
Source:vignettes/parallelisation_guidance.Rmd
parallelisation_guidance.RmdSimple parallelisation setup
LTFGRS bases its parallelisation support on the future suite of packages. please see their documentation for details.
We will use the simulate_under_LTM() function to get
some easy-to-access data:
sim <- simulate_under_LTM(fam_vec = c("m", "f", "s1"), h2 = 0.5, pop_prev = 0.05, n_sim = 100)OBS: We only use observations here to show the necessary steps for parallelisation and its feasibility. A real computational gain is not expected at this scale.
The output of simulate_under_LTM() has the necessary
information for the tibble-based input of
estimate_liability() stored in the named list entry
thresholds. Hence, a minimal parallelised call is given
by:
plan(multisession, workers = 2)
ltfgrs <- estimate_liability(.tbl = sim$thresholds %>% rename(pid = indiv_ID), h2 = 0.5)
#> The number of workers is 2
plan(sequential) # always return to sequential after parallel compute is finished The function plan() from the future suite of packages
sets up the parallel backend. It accepts several different types of
backends. multisession works across all operating systems
and will be a decent choice in most cases. The number of workers to use
is limited by the number of cores available on a user’s CPU. Here we
simply use
for illustrative purposes.
R parallelisation bottleneck
Under the hood, parallelisation with the future package (and others)
copies data and passes it from the parent process (your interactive
session) to the child processes (the processes performing the
computation in the background). The child processes are separate
instances of R that are spun up by the parent process and data is copied
to them. Data copy is generally slow, which means there is a
computational cost to setting up the parallel backend and passing large
data sets to each child process. In many cases, the reduction in
run-time of a loop is enough to offset this initial cost. However, there
are situations where it may not. For example, if data becomes too large,
then the cost of copying data becomes prohibitively large. The future
package has an option called future.globals.maxSize, which
caps the size of the exported items at ~500MiBs by default. This ceiling
is designed to guard against this particular issue. It is possible to
raise this ceiling, but it is generally advised not to.
Parallelisation bottleneck work around
In estimate_liability(), there is no dependency between
individuals. This means we can either run it all in one call, or we can
split it into, e.g. 50, separate calls. The combined results will not be
affected. However, the computational circumstances can be substantially
different.
Imagine a situation where an FGRS is required for 10 million
individuals. Then a naive parallelisation is likely to reach the export
limit imposed by the future.globals.maxSize option.
However, if we split the 10 million into partitions of 200,000
individuals, then the computational circumstances are easier to
handle.
We have introduced a function called partition_indices()
that takes any matrix-like object and returns the indices for roughly
equal partitions of the provided object. This function can help users
partition the input for estimate_liability() into
manageable sizes.
The example code given below partitions the input data into 50 equally sized partitions and loops over each partition sequentially. Computations within each partition are parallelised.
# the input is long, meaning each individual has their own row.
# Selection needs to happen on families, not individuals:
indx_list <- partition_indices(nrow = length(unique(sim$thresholds$fid)), n_partitions = 50)
# For the tibble-style (long format) input, we need to nest data into families
# before we are able to loop over it:
nested_thrs <- tidyr::nest(sim$thresholds, data = -fid)
# setting up parallel backend
plan(multisession, workers = 2)
ltfgrs_partitioned <- lapply(seq_along(indx_list), function(i) {
estimate_liability(.tbl = nested_thrs %>%
slice(indx_list[[i]]) %>%
tidyr::unnest(data) %>%
rename(pid = indiv_ID),
h2 = 0.5)
}) %>% bind_rows
plan(sequential)At large scales, it may also be beneficial to save the partitions in
nested_thrs to disc and load them as needed. Similarly, it
may also be beneficial to store the results of each iteration of
ltfgrs_partitioned to guard against interruptions such that
a partition only needs to be run once.