o i ePT @s(dZddlZddlZddlmZmZmZddlZddl m Z ddl m Z ddlmZgdggZdd Zed Gd d d ejZed GdddeZdedededeeeeeffddZdedededeeeeeffddZdddejfde deejdeejdededef dd ZdS)!z Library of TPU helper functions.N)ListOptionalTuple) tf_logging)Topology) tf_export)rrrrc Csi}t|jdD]8}t|jdD].}|||ddf}||}||vr*i||<|||vr7t|||<||||qq i}|D]\}} i} | D] \} } t| | | <qR| ||<qH|S)zDComputes a nested dict which maps task and logical core to replicas.rN)rangeshapetask_ordinal_at_coordinatessetadditemssorted) core_assignmenttopologytask_and_cores_to_replicasreplica logical_core coordinatestask_idtask_to_sorted_replica_idtaskcore_to_replicascore_to_sorted_replicascorereplicasrV/var/www/myenv/lib/python3.10/site-packages/tensorflow/python/tpu/device_assignment.py#_compute_task_and_cores_to_replicass$    rz tpu.experimental.DeviceOrderModec@seZdZdZdZdZdZdS)DeviceOrderModezFThe way of determining device orders when computing device assignment.rrN)__name__ __module__ __qualname____doc__AUTORINGMESHrrrrr 8s r z!tpu.experimental.DeviceAssignmentc@s@eZdZdZdedejfddZedefddZ ede fd d Z ede fd d Z edejfd dZ de de defddZde de dee fddZd&de de de fddZ   d'de de deedefddZ   d'de de deedefddZedddejfded eejd!eejd"e d#eddf d$d%ZdS)(DeviceAssignmentzMapping from logical cores in a computation to the physical TPU topology. Prefer to use the `DeviceAssignment.build()` helper to construct a `DeviceAssignment`; it is easier if less flexible than constructing a `DeviceAssignment` directly. rrcCst|tstdt|tj|tjd}||_|j dkr&td|j |j d|_ |j d|_ |j d|j krFtd|j d |j ||_t|j||_d S) aConstructs a `DeviceAssignment` object. Args: topology: A `Topology` object that describes the physical TPU topology. core_assignment: A logical to physical core mapping, represented as a rank 3 numpy array. See the description of the `core_assignment` property for more details. Raises: ValueError: If `topology` is not `Topology` object. ValueError: If `core_assignment` is not a rank 3 numpy array. z*topology must be a Topology object, got {}dtypez8core_assignment must be a rank 3 numpy array, got shape rrzAcore_assignment.shape[-1] must have size equal to topology rank (z), got core_assignment.shape=N) isinstancer ValueErrorformattypenpasarrayint32 _topologyndimr _num_replicas_num_cores_per_replica mesh_rank_core_assignmentr_task_and_cores_to_replicas)selfrrrrr__init__Ms0     zDeviceAssignment.__init__returncC|jS)z-A `Topology` that describes the TPU topology.)r5r<rrrrrzDeviceAssignment.topologycCr?)z The number of cores per replica.)r8r@rrrnum_cores_per_replicawrAz&DeviceAssignment.num_cores_per_replicacCr?)z*The number of replicas of the computation.)r7r@rrr num_replicas|rAzDeviceAssignment.num_replicascCr?)zThe logical to physical core mapping. Returns: An integer numpy array of rank 3, with shape `[num_replicas, num_cores_per_replica, topology_rank]`. Maps (replica, logical core) pairs to physical topology coordinates. )r:r@rrrrs z DeviceAssignment.core_assignmentrrcCst|j||ddfS)zReturns the name of the TPU device assigned to a logical core.rK)rr5tpu_device_name_at_coordinatesrMrrr tpu_devicerOzDeviceAssignment.tpu_devicercomputation_shapecomputation_striderCdevice_order_modecCst|||||dS)N)rrRrSrCrT)device_assignment)clsrrRrSrCrTrrrbuilds zDeviceAssignment.build)rr)rrN)r"r#r$r%rr2ndarrayr=propertyrintrBrCrrrrrFrHrstrrNrQ classmethodr r&rWrrrrr)Dsp%    r)x_sizey_sizez_coordr>cCsg}t|dD]+}td|D] }||d||fqt|dddD]}||d|d|fq$qt|dddD] }|d||fq<|S)aRing-order of a X by Y mesh, with a fixed Z coordinate. For example, in a 4x4 mesh, this returns the following order. 0 -- 1 -- 2 -- 3 | | | | 15-- 6 -- 5 -- 4 | | | | 14-- 7 -- 8 -- 9 | | | | 13-- 12-- 11-- 10 Note that chip 0 is not included in the output. Args: x_size: An integer represents the mesh size in the x-dimension. Must be larger than 1. y_size: An integer represents the mesh size in the y-dimension. Must be larger than 1. z_coord: An integer represents the z-coordinate to use for the chips in the ring. Returns: A list of (x,y,z) triples in ring order. r!rrr-)r append)r]r^r_retijrrr _open_ring_2dsrdz_sizecs|dkr|dkrddt|DS|dkr"|dkr"ddt|DS|dkr3|dkr3ddt|DS|dkr=|ddksQ|dkrG|ddksQ|dkrw|ddkrwtdg}t|D]t|D]|fd d t|Dqbq\|Sd g}|dkr|t||d|S|dkrd g}|d d t||dD|S|dkrd g}|d d t||dD|Sd g}td|D]}t|||}|ddkr||q|t|qt|dddD] }|dd|fq|S)aRing-order of a X by Y by Z mesh. Constructs the 3d ring from 2d rings that are stacked in the Z dimension and joined in one corner. z == 0: 0 -- 1 -- 2 -- 3 | | | | 15 - 6 -- 5 -- 4 | | | | 14 - 7 -- 8 -- 9 | | | | 13 - 12 - 11 - 10 z == 1: 63 - 30 - 29 - 28 | | | | 16 - 25 - 26 - 27 | | | | 17 - 24 - 23 - 22 | | | | 18 - 19 - 20 - 21 z == 2: 62 - 31 - 32 - 33 | | | | 45 - 36 - 35 - 34 | | | | 44 - 37 - 38 - 39 | | | | 43 - 42 - 41 - 40 z == 3: 61 - 60 - 59 - 58 | | | | 46 - 55 - 56 - 57 | | | | 47 - 54 - 53 - 52 | | | | 48 - 49 - 50 - 51 Args: x_size: An integer represents the mesh size in the x-dimension. Must be larger than 1. y_size: An integer represents the mesh size in the y-dimension. Must be larger than 1. z_size: An integer represents the mesh size in the z-dimension. Must be larger than 1. For example, in a 4x4x4 mesh, this returns the following order. Returns: A list of (x,y,z) triples in ring order. rcSsg|]}dd|fqSrr.0rbrrr 'z_ring_3d..cSsg|]}d|dfqSrfrrgrrrri)rjcSsg|]}|ddfqSrfrrgrrrri+rjr!rz Odd dimensionc3s|]}|fVqdSNr)rhxyzrr 6sz_ring_3d..)rrrcss |] \}}}|||fVqdSrkr)rhrlrornrrrrpBcss |] \}}}|||fVqdSrkr)rhrnrorlrrrrpFrqr-)r loggingwarningextendrdreversedr`)r]r^rerarbrrrmr_ring_3dsJ6        rwrrrRrSrCrTc2 s@t|tr t|d}t|tstdt|t|j|j}dur0tjdgtj dntj tj ddurHtjdgtj dntj tj dj fkratddj j fkrrtddj t dkrtd t dkrtd }t ||krtd ||||}|}t|} || krtd || |d d} |jjdkrCdg} |dkr|} } tfddt|DD])\}}|d}tt| d| }t||| |<| | | |} | d8} q| dkr| dksJtj|gdtj d}dko:d|dko:tdko:|| k}|tjkrS|tjkrN|sNtd|tjk}|rg}tddd}t| d| d| d}t|D]N}||\}}}g}ttD]5}|||d\}}}|d|} |d|}!|d|}"||d}#|| |!|"|#gq||qtnt|D]F}|}$g}%t| dg| ddgD]}&|%|$|&|$|&}$qt|%dd|%dgg}'|'}(|'})|(||)||ddf<qȇfddtD}*tjddtj|*ddiDdd}*|* df}*|*|ddtj!ddf}nWtdksMJ|t|j"|j#ks]J|j$}+g}t},t|D].}-g}.t|,D]}|-|,|}/|/|j#}0|/|j#}1|.|+|0|1ddfqs||.qkt%||dS)aComputes a device_assignment of a computation across a TPU topology. Attempts to choose a compact grid of cores for locality. Returns a `DeviceAssignment` that describes the cores in the topology assigned to each core of each replica. `computation_shape` and `computation_stride` values should be powers of 2 for optimal packing. Args: topology: A `Topology` object that describes the TPU cluster topology. To obtain a TPU topology, evaluate the `Tensor` returned by `initialize_system` using `Session.run`. Either a serialized `TopologyProto` or a `Topology` object may be passed. Note: you must evaluate the `Tensor` first; you cannot pass an unevaluated `Tensor` here. computation_shape: A rank 1 int32 numpy array with size equal to the topology rank, describing the shape of the computation's block of cores. If None, the `computation_shape` is `[1] * topology_rank`. computation_stride: A rank 1 int32 numpy array of size `topology_rank`, describing the inter-core spacing of the `computation_shape` cores in the TPU topology. If None, the `computation_stride` is `[1] * topology_rank`. num_replicas: The number of computation replicas to run. The replicas will be packed into the free spaces of the topology. device_order_mode: An enum of `DeviceOrderMode` class which indicates whether to assign devices to form rings or meshes, or let the library to choose. Returns: A DeviceAssignment object, which describes the mapping between the logical cores in each computation replica and the physical cores in the TPU topology. Raises: ValueError: If `topology` is not a valid `Topology` object. ValueError: If `computation_shape` or `computation_stride` are not 1D int32 numpy arrays with shape [3] where all values are positive. ValueError: If computation's replicas cannot fit into the TPU topology. ) serializedz)`topology` is not a Topology object; got Nrr*z#computation_shape must have shape [z]; got z$computation_stride must have shape [zcomputation_stride must be positive; got computation_stride={}z>computation footprint {} does not fit in TPU topology shape {}zprequested {} replicas but only {} replicas with shape {} and computation_stride {} fit in a TPU mesh of shape {}cSs||d|S)Nrr)nmrrr ceil_of_ratiosz(device_assignment..ceil_of_ratiorc3s$|] \}}||dfVqdS)rNr)rhrbrl) topology_rankrrrpsz$device_assignment..g?r-zdevice_order_mode=DeviceOrderMode.RING is not compatible with the 3D tiling current topology. Try setting device_order_mode=DeviceOrderMode.AUTOr!cs*g|]}td|||qSrf)r2arangerg)rRrSrrris z%device_assignment..cSsg|] }|dtjfqS).)r2newaxisrgrrrrisindexingij)axis)r)&r.bytesrr/r1len mesh_shaper2arrayr4r3r anyr0prodmissing_devicessizer enumeraterZmathceilminfullr r&r'rwr r` concatenatemeshgridreshaper num_tasksnum_tpus_per_taskdevice_coordinatesr))2rrRrSrCrTrcomputation_footprint block_countsreplica_counts max_replicasr{ replica_shaperemaining_replicasremaining_dimsrlnirb target_sizereplica_offsetsenable_3d_tiling assignment inner_ring outer_ringrouter_xouter_youter_zper_replica_assignmentindexinner_xinner_yinner_zpxpypzpitposdim replica_posouterinnerindicesrdevices_per_replicarindexreplica_assignment logical_idrdevicer)rRrSr|rrUVs 0                                 rU)r%enumrtypingrrrnumpyr2tensorflow.python.platformrrrtensorflow.python.tpu.topologyr tensorflow.python.util.tf_exportrSINGLE_CORE_ASSIGNMENTrIntEnumr objectr)rZrdrwr&rXrUrrrrsT       % h