o i e@shdZddlZddlZddlZddlmmZddl m Z ddl m Z ddl mZddl mZddlmZddlmZdd lmZdd lmZdd d dddd dd d d d dejddfddZejeejdZejeejd dZejeejd dZddZ ddZ! d%ddZ" d&ddZ#d'ddZ$ddZ%Gdd d ej&Z'Gd!d"d"ej&Z(Gd#d$d$ej&Z)dS)(zNPart of the Keras training engine related to Python generators of array data. N)backend) callbacks)training_utils)training_utils_v1) data_utils) generic_utils)ModeKeys) tf_logging Fstepsc+KsHd|vr|d}d}d}t|tjjtjjjjf}|r-|}|dur-d}tj|||||d}t|||||| d\}}|du}t|t j }t ||| | ||||| t ||| d}d}|sft || | | | d\}}t||\}}|rqdnd }tj||||||||||d }|tjkrtjd|d }ntjd|d }to|j}|rt|tjkrd nd } | d|j_|||||}t||D]9}!|jjrn1| i}"|tjkr|!|!|"|durt"j#}#n|}#d }$|$|#krt$|}%|%dur)|r|rd|j_t%&d|||fn|$d kr|$}||_'nd|j_t%&d|||fnit(tj)*|%d j+d }|$|d}&|,|d|$|&|j- }'||%}(t|(t.sT|(g}(|$d kro|/|(|'rotj0|||||||||d |1|(|2||&|(|}&|,|d|$|&|$d 7}$|jjrn|$|#ks|3|j4})|2||"|)|}"t5|)d kr|)d })|rt6||!r|jjst7||||| | | | ||tj8dd }*t|*t.s|*g}*|j2||"|*|dd}"|tjkr|9|!|"|r|!|d krtjjj:|}qd|_;|<||dur|=|r| >ddd|tjkr"|j?S|)S)aLoop function for arrays of data with modes TRAIN/TEST/PREDICT. Args: model: Keras Model instance. data: Either a tuple of NumPy/Tensor inputs (i.e. `(x,)` or `(x, y)` or `(x, y, sample_weights)`) or a generator or `keras.utils.data_utils.Sequence` object or Eager Iterator or Dataset. steps_per_epoch: Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. Ignored with the default value of `None`. epochs: Number of times to iterate over the data. verbose: 0, 1, or 2. Verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch. Note that the progress bar is not particularly useful when logged to a file, so verbose=2 is recommended when not running interactively (eg, in a production environment). callbacks: List of callbacks to be called during training. validation_data: Either a tuple of NumPy/Tensor inputs (i.e. `(x,)` or `(x, y)` or `(x, y, sample_weights)`) or a generator or `keras.utils.data_utils.Sequence` object or Eager Iterator or Dataset. validation_steps: Total number of steps (batches of samples) before declaring validation finished. validation_freq: Only relevant if validation data is provided. Integer or `collections.abc.Container` instance (e.g. list, tuple, etc.). If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. `validation_freq=2` runs validation every 2 epochs. If a Container, specifies the epochs on which to run validation, e.g. `validation_freq=[1, 2, 10]` runs validation at the end of the 1st, 2nd, and 10th epochs. class_weight: Dictionary mapping class indices to a weight for the class. max_queue_size: Integer. Maximum size for the generator queue. If unspecified, `max_queue_size` will default to 10. workers: Integer. Maximum number of processes to spin up when using process-based threading. If unspecified, `workers` will default to 1. If 0, will execute the generator on the main thread. use_multiprocessing: Boolean. If `True`, use process-based threading. If unspecified, `use_multiprocessing` will default to `False`. Note that because this implementation relies on multiprocessing, you should not pass non-pickleable arguments to the generator as they can't be passed easily to children processes. shuffle: Boolean. Whether to shuffle the order of the batches at the beginning of each epoch. Only used with instances of `Sequence` (`keras.utils.Sequence`). Has no effect when `steps_per_epoch` is not `None`. initial_epoch: Epoch at which to start training (useful for resuming a previous training run). mode: One of ModeKeys.TRAIN/ModeKeys.TEST/ModeKeys.PREDICT. batch_size: Integer batch size or None if unknown. Will only be used if `data` is in NumPy/Tensor format. steps_name: The string name of the steps argument, either `steps`, `validation_steps`, or `steps_per_epoch`. Only used for error message formatting. **kwargs: Additional arguments for backwards compatibility. `steps` is accepted as an alias for `steps_per_epoch`. Returns: - In TRAIN mode: `History` object. - In TEST mode: Evaluation metrics. - In PREDICT mode: Outputs of the Model called on inputs. Raises: ValueError: in case of invalid arguments. r FNT)epochs steps_name)steps_per_epoch batch_sizer shuffle class_weight)workersuse_multiprocessingmax_queue_sizersamples) do_validationr rrr count_modeverbosemoder r rzYour dataset ran out of data; interrupting training. Make sure that your dataset can generate at least `%s * epochs` batches (in this case, %d batches). You may need to use the repeat() function when building your dataset.zYour dataset iterator ran out of data; interrupting training. Make sure that your iterator can generate at least `%s * epochs` batches (in this case, %d batches). You may need touse the repeat() function when building your dataset.)batchsizebegin)rrr rrrrendvalidation_steps) rrrrrrrrrrval_)prefix)@ isinstancetfdataDatasetcompatv1rinfer_steps_for_datasetconvert_to_generator_likerSequence_validate_arguments_make_execution_function_make_enqueued_generator_get_num_samples_or_stepscbksconfigure_callbacksrPREDICTOutputsAggregatorMetricsAggregatorexecuting_eagerly run_eagerlyreager_learning_phase_scopeTRAIN __enter__model stop_training_call_begin_hook#_maybe_load_initial_epoch_from_ckptrange reset_metricson_epoch_beginnpinf_get_next_batchloggingwarningr intnestflattenshape_call_batch_hook _is_compiledlistcreateset_callback_parameters aggregate make_logsfinalizeresultslenshould_run_validationmodel_iterationTEST on_epoch_endmake_one_shot_iterator_successful_loop_finish_call_end_hookstop__exit__history)+r;r&rr rrvalidation_datar!validation_freqrrrrr initial_epochrrrkwargsreset_dataset_after_each_epochoriginal_dataset is_dataset generatorr is_sequencebatch_functionenqueuernum_samples_or_steps use_stepsr aggregatorshould_set_learning_phaselearning_phase_scopeepoch epoch_logs target_stepsstep batch_data batch_logs is_deferred batch_outsrS val_resultsrxU/var/www/myenv/lib/python3.10/site-packages/keras/src/engine/training_generator_v1.pyrV$s~U                     P         rV)r)rrc CsXzt|}WnttjjfyYdSwt|ts|f}t|dvr*td ||S)z'Retrieves the next batch of input data.N)r zOutput of generator should be a tuple of 1 or 2 or 3 elements: (input,) or (input, target) or (input, target, sample_weights). Received {}) next StopIterationr%errorsOutOfRangeErrorr$tuplerT ValueErrorformat)rfgenerator_outputrxrxryrDs   rDc Cs|s|r|dkrttd|dur&|s&|tjkrdnd} td| dt|p1t|t j j } | r@t|tj s@|s@tdt d d |DrUtd d d |DdS)aURaises errors if arguments are invalid. Args: is_sequence: Boolean, whether data is a `keras.utils.data_utils.Sequence` instance. is_dataset: Boolean, whether data is a dataset instance. use_multiprocessing: Boolean. If `True`, use process-based threading. If unspecified, `use_multiprocessing` will default to `False`. Note that because this implementation relies on multiprocessing, you should not pass non-pickleable arguments to the generator as they can't be passed easily to children processes. workers: Integer. Maximum number of processes to spin up when using process-based threading. If unspecified, `workers` will default to 1. If 0, will execute the generator on the main thread. steps_per_epoch: Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. Ignored with the default value of `None`. validation_data: Either a tuple of NumPy/Tensor inputs (i.e. `(x,)` or `(x, y)` or `(x, y, sample_weights)`) or a generator or `keras.utils.data_utils.Sequence` object or Eager Iterator or Dataset. validation_steps: Total number of steps (batches of samples) before declaring validation finished. mode: One of ModeKeys.TRAIN/ModeKeys.TEST/ModeKeys.PREDICT. kwargs: Additional arguments for backwards compatibility. Raises: ValueError: If `steps_per_epoch` or `validation_steps` are not passed for data types that require them, or if unrecognized keyword arguments are passed. r zUsing a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence` class.Nrr z,Please specify the number of steps via the `z ` argument.z/Please specify the `validation_steps` argument.css|]}|dkVqdS)r Nrx.0krxrxry z&_validate_arguments..zInvalid arguments passed: cSsg|]}|dkr|qSrrxrrxrxry sz'_validate_arguments..)rErF UserWarningrr9rris_generator_or_sequencer$r%r&Iteratorr,any) rgrerrrr_r!rrbarg_nameval_genrxrxryr-s8)    r-cst|trtdd|D}t|st|tjjr,t|tjr(|dur(t|}||fSt|tjj r=tj j j ||fSt tj|djddurRtdt t}fdd}|||fS)aMake a generator out of NumPy or EagerTensor inputs. Args: data: Either a generator or `keras.utils.data_utils.Sequence` object or `Dataset`, `Iterator`, or a {1,2,3}-tuple of NumPy arrays or EagerTensors. If a tuple, the elements represent `(x, y, sample_weights)` and may be `None` or `[None]`. batch_size: Used when creating a generator out of tuples of NumPy arrays or EagerTensors. steps_per_epoch: Steps of the generator to run each epoch. If `None` the number of steps will be read from the data (for `keras.utils.data_utils.Sequence` types). epochs: Total number of epochs to run. shuffle: Whether the data should be shuffled. Returns: - Generator, `keras.utils.data_utils.Sequence`, or `Iterator`. Raises: - ValueError: If `batch_size` is not provided for NumPy or EagerTensor inputs. css.|]}tddtj|Ds|VqdS)css|]}|duVqdSNrx)rerxrxryrrz6convert_to_generator_like...N)allr%rHrI)relerxrxryrs z,convert_to_generator_like..NrzvWhen passing input data as arrays, do not specify `steps_per_epoch`/`steps` argument. Please use `batch_size` instead.c3s~t}tD]2}rtj|t}|D]\}}|||}tjt j || d}t j ||Vqq dS)z;Makes a generator out of a structure of NumPy/EagerTensors.) contiguousN) rBaranger?randomrr make_batchesr slice_arraysr%rHrIpack_sequence_as)r& index_array_batches batch_start batch_end batch_idsflat_batch_datarr num_samplesrrxry_gens      z'convert_to_generator_like.._gen)r$rrrr%r&rr,rTr'r(r)rYrGrHrIrJrmathceil)r&rrr rrrxrryr+s*    r+cCs|t|tj}d}|dkr-|rtj|||d}ntj||d}|j||d|}||fS|r8t|}||fS|}||fS)z:Create a buffered queue of next elements of the generator.Nr)rr)r)rr)r$rr,OrderedEnqueuerGeneratorEnqueuerstartgetiter_sequence_infinite)rfrrrrrgrioutput_generatorrxrxryr/&s(  r/cs\|tjkrtjj|d}n|tjkrj}n dfdd }|}|tjkr,tj|dd}|S)z2Makes function to run one step of model execution.rNcs |Sr)predict_on_batch)xysample_weightsr;rxryrNs z2_make_execution_function..predict_on_batchF)r@)NN)rr9 functoolspartialtrain_on_batchrW test_on_batchr3)r;rrfrrxrryr.Es   r.cCs8tj|}t|ddrt|djddfS|dfS)zLReturns number of samples or steps, and whether to use steps count mode.rrJFT)r%rHrIhasattrrGrJ)r&r flat_inputsrxrxryr0Zs r0c@steZdZdZ                  dd d Z          dd d Z       dd dZdS)GeneratorOrSequenceTrainingLoopzGenerator-like. Input is Python generator, or Sequence object. The difference between this class and `GeneratorLikeTrainingFunction` is that this class only handles inputs that with x, y and sample_weight fused into one param. Nr Trr FcCsF||||tj|| |dt||||||| ||| |||| | ddS)N)validation_splitr)rr rrr_r!r`rrrrrrar)_validate_or_infer_batch_sizercheck_generator_arguments fit_generator)selfr;rrrr rrrr_rr sample_weightrarr!r`rrrrxrxryfitms,z#GeneratorOrSequenceTrainingLoop.fitc Cs2||||t||t|||||| | | dSN)r rrrrr)rrrevaluate_generator) rr;rrrrrr rrrrrxrxryevaluates z(GeneratorOrSequenceTrainingLoop.evaluatec Cs&||||t|||||||| dSrrpredict_generator) rr;rrrr rrrrrxrxrypredicts z'GeneratorOrSequenceTrainingLoop.predict)NNNr r NrNTNNrNNr r r F) NNNr NNNr r F)NrNNr r F__name__ __module__ __qualname____doc__rrrrxrxrxryrcsP  0 rc@beZdZdZ               d ddZ       dd d Z    dd d ZdS)"EagerDatasetOrIteratorTrainingLoopz9A non-distributed Dataset or iterator in eager execution.Nr rTrcKsj||||t||| |t|tjjjjtjjfr#| r#t |t ||||||| ||| d| | ddS)Nrr) rr rrr_r!r`rrrrar) rrvalidate_dataset_inputr$r%r(r)r&r'verify_dataset_shuffledr)rr;rrrr rrrr_rrrrarr!r`rbrxrxryrs2 z&EagerDatasetOrIteratorTrainingLoop.fitc Ks0||||t|||t||||d|dSNr)r rrr)rrrr) rr;rrrrrr rrbrxrxryrs z+EagerDatasetOrIteratorTrainingLoop.evaluatecKs"||||t||||d|dSrr)rr;rrrr rrbrxrxryrs z*EagerDatasetOrIteratorTrainingLoop.predictNNNr r NrNTNNrNNr NNNr NNNNrNNrrxrxrxryrs> 3 rc@r)GeneratorLikeTrainingLoopaGTrainingLoop that handle inputs like python generator. This is the default handler for most of the input data types, includes symbolic tensors or Numpy array-like, Datasets and iterators in graph mode (since they generate symbolic tensors). This Function is used to handle model with `run_eagerly` = True. Nr rTrcKs||||}|j||| | |dd||| d \}}}| r#|| ||} n(|rEd|kr/dkrEnnt||||\}}}}}}|||f} n|rKtdt||||f|||||| ||d| | ddS) NTr)rrr check_stepsrr rrrg?zH`validation_steps` should not be specified if `validation_data` is None.r) rrr rrr_r!r`rrrar)r_standardize_user_data_prepare_validation_datar"split_training_and_validation_datarr)rr;rrrr rrrr_rrrrarr!r`rbrval_xval_yval_sample_weightsrxrxryr7sf   zGeneratorLikeTrainingLoop.fitc KsH||||}|j||||dd|d\}}} t|||| f|||d|dS)NTr )rrrrr rr rrrr)rrr) rr;rrrrrr rrbrrxrxryrs&   z"GeneratorLikeTrainingLoop.evaluatec Ks<||||}|j|dd|d\}}}t|||||d|dS)NTr )rrr rr)rrr) rr;rrrr rrbrrxrxryrs  z!GeneratorLikeTrainingLoop.predictrrrrrxrxrxryr.s>  O $r)NNr F)r Fr Fr)*rrrnumpyrBtensorflow.compat.v2r(v2r% keras.srcrrr1keras.src.enginerrkeras.src.utilsrrkeras.src.utils.mode_keysrtensorflow.python.platformr rEr9rVrrrWrr3rrDr-r+r/r.r0 TrainingLooprrrrxrxrxrysj         VJ I  k`