o i e@sldZddlZddlZddlmmZddZddZ    dd d Z dd d Z ddZ GdddZ dS)zUtils for running models in a distribution setting. Mostly from https://github.com/tensorflow/models/blob/master/official/utils/misc/distribution_utils.py. NcCsBtjjjjtjjjjtjjjjd}||vrtd|||S)aKReturn a CollectiveCommunication based on all_reduce_alg. Args: all_reduce_alg: a string specifying which collective communication to pick, or None. Returns: tf.distribute.experimental.CollectiveCommunication object Raises: ValueError: if `all_reduce_alg` not in [None, "ring", "nccl"] )NringncclzqWhen used with `multi_worker_mirrored`, valid values for all_reduce_alg are [`ring`, `nccl`]. Supplied value: {}) tf distribute experimentalCollectiveCommunicationAUTORINGNCCL ValueErrorformat)all_reduce_alg collective_communication_optionsrU/var/www/myenv/lib/python3.10/site-packages/keras/src/benchmarks/distribution_util.py_collective_communications   rcCsF|durdStjjtjjd}||vrtd|||}||dS)aReturn a CrossDeviceOps based on all_reduce_alg and num_packs. Args: all_reduce_alg: a string specifying which cross device op to pick, or None. num_packs: an integer specifying number of packs for the cross device op. Returns: tf.distribute.CrossDeviceOps object or None. Raises: ValueError: if `all_reduce_alg` not in [None, "nccl", "hierarchical_copy"]. N)rhierarchical_copyzqWhen used with `mirrored`, valid values for all_reduce_alg are [`nccl`, `hierarchical_copy`]. Supplied value: {}) num_packs)rr NcclAllReduceHierarchicalCopyAllReducer r )r rmirrored_all_reduce_optionscross_device_ops_classrrr_mirrored_cross_device_ops7s rmirroredcCs|dkrtd|}|dkr|dkrtd|dS|dkr+tjjjt|dS|d krG|dkr9tjd S|dkrAtd tjd S|d krg|dkrSd g}n ddt |D}tjj |t ||dStd|)aKReturn a DistributionStrategy for running the model. Args: distribution_strategy: a string specifying which distribution strategy to use. Accepted values are "off", "one_device", "mirrored", and "multi_worker_mirrored" -- case insensitive. "off" means not to use Distribution Strategy. num_gpus: Number of GPUs to run this model. Returns: tf.distribute.DistibutionStrategy object. Raises: ValueError: if `distribution_strategy` is "off" or "one_device" and `num_gpus` is larger than 1; or `num_gpus` is negative. rz`num_gpus` can not be negative.offrzNWhen {} GPUs are specified, distribution_strategy flag cannot be set to `off`.Nmulti_worker_mirrored) communication one_devicez device:CPU:0z=`OneDeviceStrategy` can not be used for more than one device.z device:GPU:0rcSsg|]}d|qS)z device:GPU:%dr).0irrr sz-get_distribution_strategy..)devicescross_device_opsz$Unrecognized Distribution Strategy: ) r lowerr rrrMultiWorkerMirroredStrategyrOneDeviceStrategyrangeMirroredStrategyr)distribution_strategynum_gpusr rr"rrrget_distribution_strategyWsF  r+cCsttjdd}|r"t|ddgt|ddg}|S|rS|d}t|}|dkr9|dkr9td |dkr?dn|}td|id|d d tjd<|Sd}|S) zSet multi-worker cluster spec in TF_CONFIG environment variable. Args: worker_hosts: comma-separated list of worker ip:port pairs. Returns: Number of workers in the cluster. TF_CONFIGz{}clusterchiefworker,rrz2Must specify task_index when number of workers > 1)typeindex)r.task) jsonloadsosenvirongetlensplitr dumps) worker_hosts task_index tf_config num_workersworkersrrrconfigure_clusters,   rBcCs|r|}|St}|SN)scopeDummyContextManager)strategystrategy_scoperrrget_strategy_scopes rHc@seZdZddZddZdS)rEcCdSrCr)selfrrr __enter__zDummyContextManager.__enter__cGrIrCr)rJargsrrr__exit__rLzDummyContextManager.__exit__N)__name__ __module__ __qualname__rKrNrrrrrEs rE)rrNr)Nr,) __doc__r5r7tensorflow.compat.v2compatv2rrrr+rBrHrErrrrs! A!