Revisit parallelization in HARK

[sbenthall]

We have some support for parallelization using multi-threading:

https://github.com/econ-ark/HARK/blob/876ede3a8de0ba2606a27f6b9b6d2ae42076f580/HARK/parallel.py

The only place this is used is in the Market solve_agents method:

HARK/HARK/core.py

Lines 1274 to 1301 in 876ede3

	def solve_agents(self):
	"""
	Solves the microeconomic problem for all AgentTypes in this market.
	Parameters
	----------
	None
	Returns
	-------
	None
	"""
	try:
	multi_thread_commands(self.agents, ["solve()"])
	except Exception as err:
	if self.print_parallel_error_once:
	# Set flag to False so this is only printed once.
	self.print_parallel_error_once = False
	print(
	"**** WARNING: could not execute multi_thread_commands in HARK.core.Market.solve_agents() ",
	"so using the serial version instead. This will likely be slower. "
	"The multiTreadCommands() functions failed with the following error:",
	"\n",
	sys.exc_info()[0],
	":",
	err,
	) # sys.exc_info()[0])
	multi_thread_commands_fake(self.agents, ["solve()"])

This is a bit brittle; it causes this failure when there's some reasonable refactoring:

#1240 (comment)

In general, there are many ways to parallelize and it would be better to have options for how to parallelize, rather than hard-coding one method in the model definition code.

[mnwhite]

That's the only place that the HARK code itself uses it, but it's meant to be used in research project code. Here's a quick example from a project I have open right now: [image: image.png] That's in a project that doesn't use Market, because it's entirely "micro"-- there are no equilibrium objects to solve for. The multithreading here is ham-handed in that it uses HD writes and reads to send data among temporarily spawned Python processes. It never needs to worry about memory conflicts (etc) because there isn't any shared memory. HD operations are both relatively slow and latent, even with an SSD. It's best used for "big" chunks of work like solving an entire agent type; even then, its best to only "pass back" the necessary information, thus the delSolution() call above-- it throws out the very large solution object and sends back only the simulated data for that type.

…

On Wed, Mar 22, 2023 at 12:31 PM Sebastian Benthall < ***@***.***> wrote: We have some support for parallelization using multi-threading: https://github.com/econ-ark/HARK/blob/876ede3a8de0ba2606a27f6b9b6d2ae42076f580/HARK/parallel.py The only place this is used is in the Market solve_agents method: https://github.com/econ-ark/HARK/blob/876ede3a8de0ba2606a27f6b9b6d2ae42076f580/HARK/core.py#L1274-L1301 This is a bit brittle; it causes this failure when there's some reasonable refactoring: #1240 (comment) <#1240 (comment)> In general, there are many ways to parallelize and it would be better to have options for how to parallelize, rather than hard-coding one method in the model definition code. — Reply to this email directly, view it on GitHub <#1245>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKRAFMLOSWAML7FXI7DPRDW5MSNXANCNFSM6AAAAAAWECAZAQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>