cutcutcodec.core.opti.parallel.buffer¶

Buffer management in threading loop.

Functions

`imap`(func, args[, maxsize])	Like `cutcutcodec.core.opti.parallel.starimap()` with one argument.
`map`(func, args[, maxsize])	Like `cutcutcodec.core.opti.parallel.starmap()` with one argument.
`starimap`(func, args[, maxsize])	Like `multiprocessing.pool.ThreadPool.imap` but with limited buffer and stared args.
`starmap`(func, args[, maxsize])	Like `multiprocessing.pool.ThreadPool.map` but with limited buffer and stared args.

Details

cutcutcodec.core.opti.parallel.buffer.imap(func: callable, args: Iterable, maxsize: Integral = 0)[source]

Like cutcutcodec.core.opti.parallel.starimap() with one argument.

Examples¶

>>> import time
>>> from cutcutcodec.core.opti.parallel.buffer import imap
>>> def foo(t):
...     time.sleep(t)
...     return t
...
>>> list(imap(foo, [1.0, 0.5, 0.0], maxsize=3))  # yield fastest first
[0.0, 0.5, 1.0]
>>>

cutcutcodec.core.opti.parallel.buffer.map(func: callable, args: Iterable, maxsize: Integral = 0)[source]

Like cutcutcodec.core.opti.parallel.starmap() with one argument.

Examples¶

>>> import time
>>> from cutcutcodec.core.opti.parallel.buffer import map
>>> def foo(t):
...     time.sleep(t)
...     return t
...
>>> list(map(foo, [1.0, 0.5, 0.0]))  # keep order
[1.0, 0.5, 0.0]
>>>

cutcutcodec.core.opti.parallel.buffer.starimap(func: callable, args: Iterable, maxsize: Integral = 0)[source]

Like multiprocessing.pool.ThreadPool.imap but with limited buffer and stared args.

Parameters¶

funccallable: The function to evaluate in an over thread.
argsiterable: The parameters to give a the function.
maxsizeint, default=max(2, os.cpu_count()//2): The size of the buffer.

Notes¶

Contrary to multiprocessing functions, args is iterated in the main thread.
If an exception is raised in one of the threads, it is propagated to this function.

Examples¶

>>> import itertools
>>> from cutcutcodec.core.opti.parallel.buffer import starimap
>>> def foo(x, y):
...     return x + y
...
>>> sorted(starimap(foo, itertools.product(["a", "b", "c"], ["1", "2", "3"])))
['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3']
>>>

cutcutcodec.core.opti.parallel.buffer.starmap(func: callable, args: Iterable, maxsize: Integral = 0)[source]

Like multiprocessing.pool.ThreadPool.map but with limited buffer and stared args.

Parameters¶

funccallable: The function to evaluate in an over thread.
argsiterable: The parameters to give a the function.
maxsizeint, default=max(2, os.cpu_count()//2): The size of the buffer.

Notes¶

Contrary to multiprocessing functions, args is iterated in the main thread.
If an exception is raised in one of the threads, it is propagated to this function.

Examples¶

>>> import itertools
>>> from cutcutcodec.core.opti.parallel.buffer import starmap
>>> def foo(x, y):
...     return x + y
...
>>> list(starmap(foo, itertools.product(["a", "b", "c"], ["1", "2", "3"])))
['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3']
>>>