Python 3 concurrent.futures - process for loop in parallel

If I have understood correctly how the concurrent.futures module in Python 3 works, the following code:

import concurrent.futures
import threading
# Simple function returning a value
def test(i): a = 'Hello World\n' return a
def main(): output1 = list() with concurrent.futures.ThreadPoolExecutor() as executor: # psdd iterator to test function for out1 in executor.map(test, range(0, 10)): # append returned result output1.append(out1) # confirm output print(output1) print("Task Executed {}".format(threading.current_thread()))
if __name__ == '__main__': main()

...performs the following functions:

1. Passes a for loop to the function called test().
2. Processes the loop in parallel, rather than in serial.

However, what I really want is to process the loop in parallel in my main() function as so:

import concurrent.futures
import threading
def main(): output1 = list() with concurrent.futures.ThreadPoolExecutor() as executor: # psdd iterator to test function for out1 in executor.submit(range(0, 10)): a = 'Hello World\n' # append returned result output1.append(a) # confirm output print(output1) print("Task Executed {}".format(threading.current_thread()))
if __name__ == '__main__': main()

...this however produces the following error:

Traceback (most recent call last): File "G:\HTPC Scripts\WebGrab Plus\TESTTESTTEST2.py", line 221, in <module> main() File "G:\HTPC Scripts\WebGrab Plus\TESTTESTTEST2.py", line 209, in main for out1 in executor.submit(range(0, 10)):
TypeError: 'Future' object is not iterable

What do I need to amend in my code?

Thanks

1 Answer

It doesn't matter whether you use submit or map, you always have to use a callable (such as a function) as the first argument.

Python does allow nested functions (also take note of the way to use Futures);

import concurrent.futures
def main(): def worker(arg): return str(arg) + ' Hello World!' with concurrent.futures.ThreadPoolExecutor() as e: fut = [e.submit(worker, i) for i in range(10)] for r in concurrent.futures.as_completed(fut): print(r.result())
if __name__ == '__main__': main()

The only way to define a callable in-place is with a lambda expression, but those have significant limitations.