Distributed compute clusters allow the computing power of heterogeneous (and homogeneous) resources to be utilised to solve large-scale science and engineering problems. One class of problem that has attractive scalability properties, and is therefore often implemented using compute clusters, is task farming (or parameter sweep) applications. A typical characteristic of such applications is that no communication is needed between distributed subtasks during the overall computation. However, interesting large-scale task farming problem instances that do require global communication between subtask sets also exist. We propose a framework called semi-synchronised task farming in order to address problems requiring distributed formulations containing subtasks that alternate between independence and synchronisation. We apply this framework to several large-scale contemporary computer vision problems and present a detailed performance analysis to demonstrate framework scalability. Semi-synchronised task farming splits a given problem into a number of stages. Each stage involves firstly distributing independent subtasks to be completed in parallel. Following subtask set completion, a set of synchronised global decisions, based on information retrieved from the distributed results, is made. The results influence the following subtask distribution stage. This subtask distribution followed by result collation process is iterated until overall problem solutions are obtained. We construct a simplified Bulk Synchronous Parallel (BSP) model to formalise this framework and with this formalisation, we develop a predictive model for overall task completion time. We present experimental benchmark results comparing the performance observed by applying our framework to solve real-world problems on compute clusters with that of solving the tasks in a serial fashion. Furthermore by assessing the predicted time savings that our framework provides in simulation and validating these predictions on a range of complex problems drawn from real-world computer vision tasks, we are able to reliably predict the performance gain obtained when using a compute cluster to tackle resource intensive computer vision tasks.
Applying semi-synchronised task farming to large-scale computer vision problems / McDonagh, S.; Beyan, C.; Huang, P. X.; Fisher, R. B.. - In: INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS. - ISSN 1094-3420. - 29:4(2015), pp. 437-460.
Scheda prodotto non validato
I dati visualizzati non sono stati ancora sottoposti a validazione formale da parte dello Staff di IRIS, ma sono stati ugualmente trasmessi al Sito Docente Cineca (Loginmiur).
|Titolo:||Applying semi-synchronised task farming to large-scale computer vision problems|
|Autori:||McDonagh, S.; Beyan, C.; Huang, P. X.; Fisher, R. B.|
|Titolo del periodico:||INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS|
|Anno di pubblicazione:||2015|
|Numero e parte del fascicolo:||4|
|Codice identificativo Scopus:||2-s2.0-84944329628|
|Codice identificativo WOS:||WOS:000362996700004|
|Digital Object Identifier (DOI):||http://dx.doi.org/10.1177/1094342014532965|
|Citazione:||Applying semi-synchronised task farming to large-scale computer vision problems / McDonagh, S.; Beyan, C.; Huang, P. X.; Fisher, R. B.. - In: INTERNATIONAL JOURNAL OF HIGH PERFORMANCE COMPUTING APPLICATIONS. - ISSN 1094-3420. - 29:4(2015), pp. 437-460.|
|Appare nelle tipologie:||03.1 Articolo su rivista (Journal article)|