o i e#@sddZddlZddlmmZddlmZddlm Z ddl m Z ddl m Z dZGdd d ZdS) zTraining state management.N)backend)distributed_file_utils) mode_keys)%support_on_demand_checkpoint_callbackc@sPeZdZdZdZdZ  dddZddd Zd d Zd d Z ddZ ddZ dS)WorkerTrainingStateaTraining state management class. This class provides apis for backing up and restoring the training state. This allows model and epoch and batch information to be saved periodically and restore for fault-tolerance, also known as preemption-recovery purpose. epochNcCs*|ot|j|_||_||_tjtj|jtj ddd|_ tjtj|j tj ddd|_ t |j |jt |j |j tjj|j|j |j |jjd}tjj|tj|dtd|_t||jj}|jjjjrm|j|_n tjj||td|_|jrtjj|jjj |j|_!|j|j!_"|j!|j_#dSdS)N)dtypeckpt_saved_epoch) initial_valuenameckpt_saved_batch)modelr r train_counterchief) directory max_to_keep)$rdistribute_strategy_enable_save_before_preemption_model _save_freqtfVariableconstantCKPT_SAVED_EPOCH_UNUSED_VALUEint64_ckpt_saved_epochCKPT_SAVED_BATCH_UNUSED_VALUE_ckpt_saved_batchr set_valuetrain Checkpoint_train_counterCheckpointManagerospathjoinMAX_CHECKPOINT_TO_KEEPread_checkpoint_managerr write_dirpathextendedshould_checkpointwrite_checkpoint_manager distribute experimentalPreemptionCheckpointHandlercluster_resolverpreemption_handler_read_checkpoint_manager_preemption_handler)selfrcheckpoint_dir save_freqsave_before_preemption_arg checkpointwrite_checkpoint_dirr;Y/var/www/myenv/lib/python3.10/site-packages/keras/src/distribute/worker_training_state.py__init__0sl   zWorkerTrainingState.__init__rcCs&|jrt|jj|jjdSdS)zBack up the current state of training into a checkpoint file. Args: epoch: The current epoch information to be saved. batch: The current batch(step) information to be saved. N)r-saverremove_temp_dirpathrrr)r5r batchr;r;r<back_ups zWorkerTrainingState.back_upcCs(|jr|jjd7_|jdSdS)Nr)rr2 _run_counter&_check_preemption_and_maybe_checkpointr5r;r;r<backup_if_preemptedsz'WorkerTrainingState.backup_if_preemptedcCs|js |jdSdS)aRestore the training state from the backed up checkpoint file. Returns: True if the training state is successfully restored. False if the training state doesn't need to be restored, or error occurred so it can't. N)rr)restore_or_initializerDr;r;r<restores zWorkerTrainingState.restorecCsB|j|jurz tjj|jjWdStjjyYdSwdS)zDelete the backup directories. Delete the backup directories which should not exist after `fit()` successfully finishes. N) r-r)riogfilermtreererrors NotFoundErrorrDr;r;r< delete_backups z!WorkerTrainingState.delete_backupcCsd}t|j}t|j}|tjjkrC|jst|j t r;|dkr7||dkr1|d}d}||fS|}|d}||fS|dkrC|d}||fS)aMaybe load 1st epoch from checkpoint, considering worker recovery. When `_ckpt_saved_epoch` attribute exists and is not `CKPT_SAVED_EPOCH_UNUSED_VALUE`, this is under multi-worker training setting and indicates the worker is recovering from previous failure. In this case, infer `initial_epoch` from `self._ckpt_saved_epoch` to continue previous unfinished training from certain epoch. Args: steps_per_epoch: The number of steps per epoch value. initial_epoch: The original initial_epoch user passes in in `fit()`. mode: The mode for running `model.fit()`. Returns: If the training is recovering from previous failure under multi-worker training setting, return the (epoch, step) the training is supposed to continue at. Otherwise, return the `initial_epoch, initial_step` the user passes in. rr) revalrrrModeKeysTRAINr isinstancerint)r5steps_per_epoch initial_epochmode initial_stepr r@r;r;r<%maybe_load_initial_counters_from_ckpts$     z9WorkerTrainingState.maybe_load_initial_counters_from_ckpt)r N)r) __name__ __module__ __qualname____doc__rrr=rArErGrMrWr;r;r;r<r"s  T r)r[r%tensorflow.compat.v2compatv2r keras.srcrkeras.src.distributerkeras.src.utilsr+keras.src.distribute.distributed_file_utilsrr(rr;r;r;r<s