MAJOR EDIT
Suppose I have a function that does internal parallelization, and I want to parallelize this function. How do I do this in Python?
Let's consider a very simple concrete example.
The function fun1
takes two numbers as input and adds them up.
def fun1(a, b):
return a+b
The function fun2
takes two integer inputs m
and n
. It generates a list of size mn containing tuples as the following code shows. Then it parallelizes fun1
on this list.
def fun2(m, n):
# create list of mn tuples
list_tuple = [(i, j) for i in range(m) for j in range(n)]
# apply fun1 parallelly to list_tuple
pool = multiprocessing.Pool()
return pool.starmap(fun1, list_tuple)
Thus, fun2
returns a list of size mn. We want to apply fun2
parallelly to a list of (m,n) pairs.
if __name__ == '__main__':
# create the list of (m, n) tuples
list_tuple = [(m, n) for m in range(10) for n in range(10)]
# parallellize fun2
pool = multiprocessing.Pool()
pool.starmap(fun2, list_tuple)
My questions:
- Can I write everything above in the same script?
- If answer to 1 is yes, what is the correct way to write it? For example should there be
if __name__ == 'main'
inside the definition offun2
as well? - If answer to 1 is no, then how to write this correctly in two scripts?
I am a beginner at parallelization and don't yet understand how the threads are connected so help is highly appreciated.
Thanks!
source https://stackoverflow.com/questions/75820729/calling-a-function-parallelly-that-does-internal-parallelization
Comments
Post a Comment