o i eB" @sdZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Zdd l mZdd l mZdd lmZdd lmZdd lmZdZdZdZdZdZdZedGdddedgdZ  d ddZ ddZ!ddZ"ddZ#dS)!z+TPU system metadata and associated tooling.N) config_pb2)session) device_util)contextconfig)device)errors)ops) tf_logging)tpu) tf_exportii  tpu_worker coordinator)localz"tpu.experimental.TPUSystemMetadatacs eZdZdZfddZZS)TPUSystemMetadataaDescribes some metadata about the TPU system. Attributes: num_cores: interger. Total number of TPU cores in the TPU system. num_hosts: interger. Total number of hosts (TPU workers) in the TPU system. num_of_cores_per_host: interger. Number of TPU cores per host (TPU worker). topology: an instance of `tf.tpu.experimental.Topology`, which describes the physical topology of TPU system. devices: a tuple of strings, which describes all the TPU devices in the system. cstt|||||||SN)superr__new__)cls num_cores num_hostsnum_of_cores_per_hosttopologydevices __class__X/var/www/myenv/lib/python3.10/site-packages/tensorflow/python/tpu/tpu_system_metadata.pyr=s zTPUSystemMetadata.__new__)__name__ __module__ __qualname____doc__r __classcell__rrrrr(s  rrrrrrFc CsLd}g}tt}trt}dd|D}nqd} td|z>t .t j |tt|d}|} WdWdWnB1sKwYWdn1sZwYWn(tjyd |} |tkrtd | td |t|d7}nt| Ynwq|D]} tj| j} | jd kr|| j| j|d7}qd} |rtd d|D} t | dkrt!d"|| #} d}|r|st!d"||t$||}dd}t%t&||d}t'|t || ||d}|rtdtd|j(td|j)td|j*|j+D] } td| q|Std||S)zOs z._query_tpu_system_metadata..Tz8Querying Tensorflow master (%s) for TPU system metadata.rNzFailed to connect to the Tensorflow master. The TPU worker may not be ready (still scheduling) or the Tensorflow master address is incorrect: got (%s).z%szRetrying (%d/%d).TPUcSsg|]}t|qSr)len)r+core_idsrrrr-yszHTPU cores on each host is not same. This should not happen!. devices: {}zCannot find any TPU cores in the system (master address {}). This usually means the master address is incorrect or the TPU worker has some problems. Available devices: {}cSs&tj|j}|j|j|j|j|jfSr) tf_device DeviceSpec from_stringr)jobreplicataskr* device_index)rspecrrr _sort_keysz-_query_tpu_system_metadata.._sort_key)keyr%zFound TPU system:z*** Num TPU Cores: %dz*** Num TPU Workers: %dz *** Num TPU Cores Per Worker: %dz*** Available Device: %szFailed to find TPU: %s), collections defaultdictlistrexecuting_eagerlyrlist_logical_deviceslogginginfor Graph as_defaultr&Sessionget_session_config_with_timeout_PINGING_MASTER_TIMEOUT_IN_MS list_devicesr DeadlineExceededError _RETRY_TIMESwarning ValueErrorr2r3r4r)r*r7appendr8setvaluesr0 RuntimeErrorformatpop_obtain_topologytuplesortedrrrrr)master_address cluster_defquery_topologytpu_core_countr device_dictlogical_devices retry_countsessmsgrr9rnum_cores_per_host_setrr:metadatarrr_query_tpu_system_metadataDs          rac CszNtd|t7tt|}tj||d}| t }|WdWdWS1s6wYWdWdS1sGwYWdSt j y\td|w)zObtains TPU fabric topology.zfInitializing TPU system (master: %s) to fetch topology for model parallelism. This might take a while.rNzaFail to initialize TPU system with master (%s). Please double check the TPU system is functional.)rArBr rCrDrF!_INITIAL_TPU_SYSTEM_TIMEOUT_IN_MSr&rErunr initialize_systemr rIrL)rVrWsession_configr]rrrrrSs2&rScCstj||d}|S)z>Returns a session given a timeout and a cluster configuration.)operation_timeout_in_msrW)r ConfigProto)timeout_in_secsrW config_protorrrrFsrFcCs|tvrdS|r |js tStdd|jD}t|vrtdt|dkr+|jdjSt|dkr>t|vr>|t| Std) aReturns the canonical job name to use to place TPU computations on. Args: master: A `string` representing the TensorFlow master to use. cluster_def: A ClusterDef object describing the TPU cluster. Returns: A string containing the job name, or None if no job should be specified. Raises: ValueError: If the user needs to specify a tpu_job_name, because we are unable to infer the job name automatically, or if the user-specified job names are inappropriate. Ncss|]}|jVqdSr)r))r+r5rrr szmaster_job..z1Currently, tpu_worker is not an allowed job name.r.rzCould not infer TPU job name.) _LOCAL_MASTERSr5_DEFAULT_JOB_NAMErNrLr0r)_DEFAULT_COORDINATOR_JOB_NAMEremoverR)masterrW job_namesrrr master_jobs     rr)NF)$r#r<tensorflow.core.protobufrtensorflow.python.clientrr&tensorflow.python.distributertensorflow.python.eagerrtensorflow.python.frameworkrrr2r r tensorflow.python.platformr rAtensorflow.python.tpur tensorflow.python.util.tf_exportr rGrJrbrmrnrl namedtuplerrarSrFrrrrrrs:             c