o i e?@sdZddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd l mZdd lmZdd lmZd d ZddZGddde jZGddde jZddZddZdS)z(Implementation of LoadDataset in Python.N)message) text_format) snapshot_pb2)_pywrap_snapshot_utils) dataset_ops)structured_function)gen_experimental_dataset_ops)gfile)nested_structure_codercsfdd}|durdd}|dur6ttjtjd }|}Wdn1s-wYt|}|}|rHt |||t ||St |||S)z$Loads dataset from tf.data snapshot.c sjz%ttd}t|tWdWS1swYWdStjt j t fy4YdSw)zReads the distributed snapshot metadata. Returns: DistributedSnapshotMetadata if the snapshot is a distributed snapshot. Returns None if it is a non-distributed snapshot. rN) r GFiler TF_DATA_SnapshotMetadataFilePathr ParseLinesrDistributedSnapshotMetadata ParseErrorr DecodeErrorUnicodeDecodeError)fpathQ/var/www/myenv/lib/python3.10/site-packages/tensorflow/python/data/ops/load_op.py"_get_distributed_snapshot_metadata"s (z1_load.._get_distributed_snapshot_metadataNcSs|jddttjdS)NcSs|SNr)xrrr4sz)_load....) cycle_lengthnum_parallel_calls) interleavemultiprocessing cpu_countrAUTOTUNE)datasetsrrrr3s z_load..rb) r r osrjoinrDATASET_SPEC_FILENAMEread_parse_element_spec_validate_snapshot_load_distributed_snapshot _LoadDataset)r element_spec compression reader_funcrr encoded_specdistributed_snapshot_metadatarrr_loads(  r1csHt|fddtD}tj|}|fdd}||S)zLoads a distributed snapshot.csg|] }tj|qSr)r$rr%).0r) chunks_dirrr Ksz._load_distributed_snapshot..cst|tjjdS)N)r,r-)_SnapshotChunkDatasetr(r,r-) chunk_file)metadatarrrOs z,_load_distributed_snapshot..)r TF_DATA_CommittedChunksDirectoryr ListDirectoryrDatasetfrom_tensor_slicesmap)rr7r. chunk_filesdatasetr)r3r7rr*Gs    r*c,eZdZdZfddZeddZZS)r+z.A dataset that loads previously saved dataset.csh||_||_||_tj|dtt|jd|_tj |f|jj j ||jj d|j }t |dS)Nzload())input_structure)reader_func_other_argsr-r.)_path _element_spec _compressionrStructuredFunctionWrapperr DatasetSpec _reader_funcged_ops load_datasetfunctioncaptured_inputs_flat_structuresuper__init__)selfrr,r-r.variant_tensor __class__rrrNYs( z_LoadDataset.__init__cC|jSrrCrOrrrr,lz_LoadDataset.element_spec__name__ __module__ __qualname____doc__rNpropertyr, __classcell__rrrQrr+Vs  r+cr?)r5zAA dataset for one chunk file from a tf.data distributed snapshot.cs4||_||_tj|fd|i|j}t|dS)Nr-) _chunk_filerCrHsnapshot_chunk_datasetrLrMrN)rOr6r,r-rPrQrrrNtsz_SnapshotChunkDataset.__init__cCrSrrTrUrrrr,}rVz"_SnapshotChunkDataset.element_specrWrrrQrr5qs   r5cCst|s td|dt|}t|r2t|d }td|d|1s-wYt|}t|sDtd|dt|j }|r]||kr]td|d|d|d |rs||j krutd|d |d |j d dSdS) a8Validates a tf.data distributed snapshot. Args: path: Root path of the distributed snapshot. metadata: The DistributedSnapshotMetadata of the snapshot. element_spec: Dataset element_spec. compression: Compression method used for saving. Raises: ValueError if the snapshot is invalid. z#Failed to load tf.data snapshot at z(: The snapshot directory does not exist.r z+. The save job failed to write it. Status: Nz5. The save job has not finished writing the snapshot.z. User specified element_spec z!, but the actual element_spec is .z. User specified compression z , but the actual compression is ) r Exists ValueErrorrTF_DATA_SnapshotErrorFilePathr r'TF_DATA_SnapshotDoneFilePathr(r,r-)rr7r,r- error_filer done_filesnapshot_element_specrrrr)sH         r)cCstj}||t|Sr)r struct_pb2StructuredValueParseFromString decode_proto)encoded_element_spec struct_pbrrrr(s   r()r[rr$google.protobufrrtensorflow.core.protobufr+tensorflow.python.data.experimental.servicertensorflow.python.data.opsrrtensorflow.python.opsrrHtensorflow.python.platformr tensorflow.python.saved_modelr r1r* DatasetSourcer+r5r)r(rrrrs$         ( -