o i eV@s~dZddlmmZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZGd d d Zd dZddZddZddZGdddejjjZdZdZ Gddde!Z"edGddde"dZ#Gdd d ejjj$ej%e#Z&Gd!d"d"ejjj$e j'e#Z(Gd#d$d$ejjjZ)d%d&Z*ejj+,ej%e*e&d'd(Z-d)d*Z.d+d,Z/dS)-z*Contains the loss scaling optimizer class.N)backend) optimizers)utils) optimizer) optimizer_v2)serialization_lib) tf_logging) keras_exportc@seZdZdZdgZddZdS)_UnwrapPreventeraWrapper that DistributionStrategy will not unwrap. Typically, DistributionStrategy will unwrap values when going from a cross- replica context to a replica context via `call_for_each_replica`. This class is a wrapper that DistributionStrategy will not unwrap, so it can be used to prevent it from unwrapping a value. TODO(reedwm): Find/implement a better way of preventing values from being unwrapped by DistributionStrategy valuecC ||_dSN)r selfr r]/var/www/myenv/lib/python3.10/site-packages/keras/src/mixed_precision/loss_scale_optimizer.py__init__. z_UnwrapPreventer.__init__N)__name__ __module__ __qualname____doc__ __slots__rrrrrr s r cs$ddfdd|D}t|S)zKReturns a scalar boolean tensor indicating if all gradients are finite.cSst|tjr |jS|Sr ) isinstancetf IndexedSlicesvalues)grrr raw_values6sz"_is_all_finite..raw_valuescs*g|]}|durttj|qSr )r reduce_allmath is_finite.0rrrr 9s z"_is_all_finite..)rr)gradsis_finite_per_gradrr$r_is_all_finite2s   r(cCstr|S|jS)a6Returns the tensor's op in graph mode, or the tensor in eager mode. This is useful because sometimes an op is needed in graph mode instead of a tensor. In eager mode, there are no ops. Args: tensor: A tensor. Returns: The tensor's op in graph mode. The tensor in eager mode. )rexecuting_eagerlyop)tensorrrr_op_in_graph_modeAs r,cs"ttjfddtjS)z5Assigns a value to a variable if the value is finite.cstSr )r,assignrr varrrVsz#_assign_if_finite..)rcondr r!no_op)r/r rr.r_assign_if_finiteRs   r3cCsf|r|rdSd}|s|std|ddS|s$td|ddS|s1td|ddSdS)z=Warn if the loss or gradients hasn't been scaled or unscaled.Nz with tf.GradientTape() as tape: loss = loss_fn() scaled_loss = opt.get_scaled_loss(loss) scaled_grads = tape.gradient(scaled_loss, vars) grads = opt.get_unscaled_gradients(scaled_grads) opt.apply_gradients([(grads, var)])aYou forgot to call LossScaleOptimizer.get_scaled_loss() and LossScaleOptimizer.get_unscaled_gradients() before calling LossScaleOptimizer.apply_gradients(). This will likely result in worse model quality, so please call them in the correct places! For example:zq For more information, see https://www.tensorflow.org/api_docs/python/tf/keras/mixed_precision/LossScaleOptimizeraYou forgot to call LossScaleOptimizer.get_scaled_loss() before calling LossScaleOptimizer.apply_gradients() (you did call get_unscaled_gradients() however). This will likely result in worse model quality, so please call get_scaled_loss() in the correct place! For example:aYou forgot to call LossScaleOptimizer.get_unscaled_gradients() before calling LossScaleOptimizer.apply_gradients() (you did call get_scaled_loss() however). This will likely result in worse model quality, so please call get_unscaled_gradients() in the correct place! For example:)rwarning)loss_has_been_scaledgradients_have_been_unscaled example_coderrr_maybe_warn_about_scaling[s, r8cseZdZdZfddZdddZdfdd Zdfd d Zed d Z eddZ eddZ eddZ eddZ ddZddZZS)_DynamicLossScaleStatez"The state of a dynamic loss scale.cs\tt||_t||_t||_i|_|jdt j |jd|_ |jdt j dd|_ dS)zCreates the dynamic loss scale.current_loss_scale)namedtype initial_value good_stepsrN)superrfloat_initial_loss_scaleint _growth_steps _multiplier_weights _add_weightrfloat32_current_loss_scaleint64_counter)rinitial_loss_scale growth_steps multiplier __class__rrrs     z_DynamicLossScaleState.__init__NcCsltj|||dtjjtjjd}trd}n tjj }|j }||f}||j |<|j ||dt ||S)a;Adds a weight to this loss scale. Args: name: Variable name. initial_value: The variable's initial value. dtype: The type of the variable. Returns: A variable. Raises: RuntimeError: If a weight with `name` has already been added. F)r=r;r< trainablesynchronization aggregationN)r; trackable)rVariableVariableSynchronizationAUTOVariableAggregationNONEr)compatv1get_default_graph _graph_keyrE_handle_deferred_dependenciesrtrack_variable)rr;r=r<variable graph_keygraphkeyrrrrFs"    z"_DynamicLossScaleState._add_weight checkpointc  svtrd}n tjj}|j}i}t|jdddD]\\}}}||kr+|||<q| t j |fi||S)z6From Trackable. Gather graph-specific weights to save.NcSs |ddSNrr)irrrr0s z<_DynamicLossScaleState._trackable_children..)rb) rr)rYrZr[r\sortedrEitemsupdater?_trackable_children) r save_typekwargsr`raweightsr;rvrNrrris z*_DynamicLossScaleState._trackable_childrencs^|dur ||}nt|}|dur|Strd}n tjj}|j}|j ||fdS)z3From Trackable. Find a weight in the current graph.N) getr?_lookup_dependencyrr)rYrZr[r\rE)rr;cached_dependencies unconditionalr`rarNrrros   z)_DynamicLossScaleState._lookup_dependencycC|jSr )rArrrrrKz)_DynamicLossScaleState.initial_loss_scalecCrrr )rCrsrrrrLrtz#_DynamicLossScaleState.growth_stepscCrrr )rDrsrrrrMrtz!_DynamicLossScaleState.multipliercCrr)z:Returns the current loss scale as a float32 `tf.Variable`.)rHrsrrrr:z)_DynamicLossScaleState.current_loss_scalecCrr)z/Returns the counter as a float32 `tf.Variable`.)rJrsrrrcounterruz_DynamicLossScaleState.countercCs t|jS)z.update_if_finite_grads..incr_loss_scalecstjdS)Nr)r,rv assign_addrrsrrr0*szO_DynamicLossScaleState.update..update_if_finite_grads..)rr1rvrL)rrsrrupdate_if_finite_gradss  z=_DynamicLossScaleState.update..update_if_finite_gradscs0tjjd}tjdj|S)z,Update assuming the gradients are nonfinite.rr)rmaximumr:rMr|rvr-r}rsrrupdate_if_not_finite_grads-s   zA_DynamicLossScaleState.update..update_if_not_finite_grads) rnestflatten distribute has_strategyin_cross_replica_context get_strategyextendedcall_for_each_replicar(experimental_local_resultsr1) rr& distributionis_finite_per_replicar!rr update_opshould_apply_gradientsrrsrrhs,     z_DynamicLossScaleState.updater )rc)rrrrrrFriropropertyrKrLrMr:rvrxrh __classcell__rrrNrr9s$  $     r9iics eZdZdZfddZZS)LossScaleOptimizerMetaclassaMetaclass that delegates LossScaleOptimizer instance creation. This metaclass causes a LossScaleOptimizer or LossScaleOptimizerV3 to be created when a BaseLossScaleOptimizer is constructed. As a result, when a user creates a loss scale optimizer with `tf.keras.mixed_precision.LossScaleOptimizer(opt)`, either a LossScaleOptimizer or LossScaleOptimizerV3 will be created, depending on the type of `opt`. cs~|turtt|j|g|Ri|St|tjr$t|g|Ri|St|tj r5t |g|Ri|Sd|d}t |)Nz"inner_optimizer" must be an instance of `tf.keras.optimizers.Optimizer` or `tf.keras.optimizers.experimental.Optimizer`, but got: .) BaseLossScaleOptimizerr?rrxrr OptimizerV2LossScaleOptimizerr OptimizerLossScaleOptimizerV3 TypeError)clsinner_optimizerr{rkmsgrNrrrxPs"   z$LossScaleOptimizerMetaclass.__call__)rrrrrxrrrrNrrEs rz(keras.mixed_precision.LossScaleOptimizerc@sheZdZdZeddZeddZeddZedd Zed d Z ed d Z ddZ ddZ dS)raiAn optimizer that applies loss scaling to prevent numeric underflow. Loss scaling is a technique to prevent numeric underflow in intermediate gradients when float16 is used. To prevent underflow, the loss is multiplied (or "scaled") by a certain factor called the "loss scale", which causes intermediate gradients to be scaled by the loss scale as well. The final gradients are divided (or "unscaled") by the loss scale to bring them back to their original value. `LossScaleOptimizer` wraps another optimizer and applies loss scaling to it. By default, the loss scale is dynamically updated over time so you do not have to choose the loss scale. The `minimize` method automatically scales the loss, unscales the gradients, and updates the loss scale so all you have to do is wrap your optimizer with a `LossScaleOptimizer` if you use `minimize`. For example: >>> opt = tf.keras.optimizers.experimental.SGD(0.25) >>> opt = tf.keras.mixed_precision.LossScaleOptimizer(opt) >>> var = tf.Variable(1.) >>> loss_fn = lambda: var ** 2 >>> # 'minimize' applies loss scaling and updates the loss sale. >>> opt.minimize(loss_fn, var_list=[var]) >>> var.numpy() 0.5 If a `tf.GradientTape` is used to compute gradients instead of `minimize`, you must scale the loss and gradients manually. This can be done with the `LossScaleOptimizer.get_scaled_loss` and `LossScaleOptimizer.get_unscaled_gradients` methods. For example: >>> with tf.GradientTape() as tape: ... loss = loss_fn() ... scaled_loss = opt.get_scaled_loss(loss) >>> scaled_grad = tape.gradient(scaled_loss, var) >>> (grad,) = opt.get_unscaled_gradients([scaled_grad]) >>> opt.apply_gradients([(grad, var)]) # Loss scale is updated here >>> var.numpy() 0.25 Warning: If you forget to call `get_scaled_loss` or `get_unscaled_gradients` (or both) when using a `tf.GradientTape`, the model will likely converge to a worse quality. Please make sure you call each function exactly once. When mixed precision with float16 is used, there is typically no risk of underflow affecting model quality if loss scaling is properly used. See [the mixed precision guide]( https://www.tensorflow.org/guide/keras/mixed_precision) for more information on how to use mixed precision. Args: inner_optimizer: The `tf.keras.optimizers.Optimizer` or `tf.keras.optimizers.experimental.Optimizer` instance to wrap. dynamic: Bool indicating whether dynamic loss scaling is used. If `True`, the loss scale will be dynamically updated over time using an algorithm that keeps the loss scale at approximately its optimal value. If False, a single fixed loss scale is used and `initial_scale` must be specified, which is used as the loss scale. Recommended to keep as True, as choosing a fixed loss scale can be tricky. Currently, there is a small performance overhead to dynamic loss scaling compared to fixed loss scaling. Defaults to `True`. initial_scale: The initial loss scale. If `dynamic` is True, this defaults to `2 ** 15`. If `dynamic` is False, this must be specified and acts as the sole loss scale, as the loss scale does not change over time. When dynamic loss scaling is used, is better for this to be a very high number, because a loss scale that is too high gets lowered far more quickly than a loss scale that is too low gets raised. dynamic_growth_steps: With dynamic loss scaling, every `dynamic_growth_steps` steps with finite gradients, the loss scale is doubled. If a nonfinite gradient is encountered, the count is reset back to zero, gradients are skipped that step, and the loss scale is halved. The count can be queried with `LossScaleOptimizer.dynamic_counter`. This argument can only be specified if `dynamic` is True. Defaults to `2000`. `LossScaleOptimizer` will occasionally skip applying gradients to the variables, in which case the trainable variables will not change that step. This is done because the dynamic loss scale will sometimes be raised too high, causing overflow in the gradients. Typically, the first 2 to 15 steps of the model are skipped as the initial loss scale is very high, but afterwards steps will only be skipped on average 0.05% of the time (the fraction of steps skipped is `1 / dynamic_growth_steps`). `LossScaleOptimizer` delegates all public `Optimizer` methods to the inner optimizer. Additionally, in methods `minimize` and `get_gradients`, it scales the loss and unscales the gradients. In methods `minimize` and `apply_gradients`, it additionally updates the loss scale and skips applying gradients if any gradient has a nonfinite value. ### Hyperparameters If wrapping a `tf.keras.optimizers.Optimizer`, hyperparameters can be accessed and set on the LossScaleOptimizer, which will be delegated to the wrapped optimizer. >>> opt = tf.keras.optimizers.legacy.Adam(beta_1=0.8, epsilon=1e-5) >>> opt = tf.keras.mixed_precision.LossScaleOptimizer(opt) >>> opt.beta_1 # Equivalent to `opt.inner_optimizer.beta_1` 0.8 >>> opt.beta_1 = 0.7 # Equivalent to `opt.inner_optimizer.beta_1 = 0.7` >>> opt.beta_1 0.7 >>> opt.inner_optimizer.beta_1 0.7 However, accessing or setting non-hyperparameters is not delegated to the LossScaleOptimizer. In an Adam optimizer, `beta_1` is a hyperparameter but `epsilon` is not, as the Adam optimizer only calls `Optimizer._set_hyper` on `beta_1`. >>> opt.inner_optimizer.epsilon 1e-5 >>> opt.epsilon Traceback (most recent call last): ... AttributeError: 'LossScaleOptimizer' object has no attribute 'epsilon' >>> opt.epsilon = 1e-4 # This does NOT set epsilon on `opt.inner_optimizer` >>> opt.inner_optimizer.epsilon >>> 1e-5 In the above example, despite epsilon being set on the LossScaleOptimizer, the old epsilon value will still be used when training as epsilon was not set on the inner optimizer. cCt)z5Bool indicating whether dynamic loss scaling is used.NotImplementedErrorrsrrrdynamiczBaseLossScaleOptimizer.dynamiccCr)z2The current loss scale as a float32 scalar tensor.rrsrrr loss_scalerz!BaseLossScaleOptimizer.loss_scalecCr)aThe number of steps since the loss scale was last increased or decreased. This is None if `LossScaleOptimizer.dynamic` is False. The counter is incremented every step. Once it reaches `LossScaleOptimizer.dynamic_growth_steps`, the loss scale will be doubled and the counter will be reset back to zero. If nonfinite gradients are encountered, the loss scale will be halved and the counter will be reset back to zero. rrsrrrdynamic_counters z&BaseLossScaleOptimizer.dynamic_countercCr)zThe initial loss scale. If `LossScaleOptimizer.dynamic` is False, this is the same number as `LossScaleOptimizer.loss_scale`, as the loss scale never changes. rrsrrr initial_scalesz$BaseLossScaleOptimizer.initial_scalecCr)zThe number of steps it takes to increase the loss scale. This is None if `LossScaleOptimizer.dynamic` is False. Every `dynamic_growth_steps` consecutive steps with finite gradients, the loss scale is increased. rrsrrrdynamic_growth_stepss z+BaseLossScaleOptimizer.dynamic_growth_stepscCr)z7The optimizer that this LossScaleOptimizer is wrapping.rrsrrrrrz&BaseLossScaleOptimizer.inner_optimizercCr)a;Scales the loss by the loss scale. This method is only needed if you compute gradients manually, e.g. with `tf.GradientTape`. In that case, call this method to scale the loss before passing the loss to `tf.GradientTape`. If you use `LossScaleOptimizer.minimize` or `LossScaleOptimizer.get_gradients`, loss scaling is automatically applied and this method is unneeded. If this method is called, `get_unscaled_gradients` should also be called. See the `tf.keras.mixed_precision.LossScaleOptimizer` doc for an example. Args: loss: The loss, which will be multiplied by the loss scale. Can either be a tensor or a callable returning a tensor. Returns: `loss` multiplied by `LossScaleOptimizer.loss_scale`. r)rlossrrrget_scaled_losssz&BaseLossScaleOptimizer.get_scaled_losscCr)aUnscales the gradients by the loss scale. This method is only needed if you compute gradients manually, e.g. with `tf.GradientTape`. In that case, call this method to unscale the gradients after computing them with `tf.GradientTape`. If you use `LossScaleOptimizer.minimize` or `LossScaleOptimizer.get_gradients`, loss scaling is automatically applied and this method is unneeded. If this method is called, `get_scaled_loss` should also be called. See the `tf.keras.mixed_precision.LossScaleOptimizer` doc for an example. Args: grads: A list of tensors, each which will be divided by the loss scale. Can have None values, which are ignored. Returns: A new list the same size as `grads`, where every non-None value in `grads` is divided by `LossScaleOptimizer.loss_scale`. rrr&rrrget_unscaled_gradients0sz-BaseLossScaleOptimizer.get_unscaled_gradientsN) rrrrrrrrrrrrrrrrrrgs |       r) metaclasscseZdZdZdZ   dQddZeddZedd Zed d Z ed d Z eddZ eddZ ddZ ddZdRddZddZddZ dSddZddZd d!ZedTd"d#Zed$d%Zejd&d%Zd'd(Zd)d*Zed+d,Zd-d.Zd/d0Zed1d2Zejd3d2Zed4d5Zejd6d5Zed7d8Z e jd9d8Z d:d;Z!dd?Z#d@dAZ$dUdCdDZ%dEdFZ&fdGdHZ'fdIdJZ(edKdLZ)e)jdMdLZ)edNdOZ*e*jdPdOZ*Z+S)VrzDAn optimizer that applies loss scaling to prevent numeric underflow.TNcCs0t|tjst|tjrtd|d|}t|t|ts'td|ft|tr3td|ftt |ddr@t d||_ d|j _ t jjj||j |ro|durYt}|dur_t}t||d d |_||jd n|durwt d t||_|durt d |fd|_d|_|t|j ddS)NaYou passed an instance of the new experimental optimizer, `optimizer.Optimizer`, to LossScaleOptimizer, but only the classic optimizers subclassing from `tf.keras.optimizers.Optimizer` can be passed. Please use `loss_scale_optimizer.LossScaleOptimizerV3` instead of `tf.keras.mixed_precision.LossScaleOptimizer`, as the former supports wrapping instances of the new experimental optimizer. Got optimizer: zW"inner_optimizer" must be an instance of `tf.keras.optimizers.Optimizer`, but got: %s. zM"dynamic" argument to LossScaleOptimizer.__init__ must be a bool, but got: %rzFLossScaleOptimizer cannot wrap another LossScaleOptimizer, but got: %s#_is_wrapped_by_loss_scale_optimizerF}"inner_optimizer" is already wrapped by a LossScaleOptimizer. An optimizer can only be wrapped by a single LossScaleOptimizerTrMr7"initial_scale" must be specified if "dynamic" is FalsezF"dynamic_growth_steps" must be None if "dynamic" is False, but got: %sbase_optimizer)rrrrrrboolr_raise_if_strategy_unsupportedgetattr ValueError _optimizerrr __internal__trackingDelegatingTrackableMixinr_DEFAULT_INITIAL_SCALE_DEFAULT_GROWTH_STEPSr9 _loss_scale_track_trackabler@_loss_has_been_scaled_gradients_have_been_unscaledFakeOptimizerForRestoration)rrrrrrrrrrTsz        zLossScaleOptimizer.__init__cC t|jtSr rrr9rsrrrrryzLossScaleOptimizer.dynamiccC&t|jtr t|jjSt|jSr rrr9rrwr:rsrrrr  zLossScaleOptimizer.loss_scalecCt|jtr |jjSdSr rrr9rvrsrrrr z"LossScaleOptimizer.dynamic_countercCt|jtr |jjS|jSr rrr9rKrsrrrr z LossScaleOptimizer.initial_scalecCrr rrr9rLrsrrrrrz'LossScaleOptimizer.dynamic_growth_stepscCrrr rrsrrrrrtz"LossScaleOptimizer.inner_optimizerc4d_trfdd}|StjjS)NTc}|tj|jSr rcastrr<loss_valrrrrnew_lossz4LossScaleOptimizer.get_scaled_loss..new_lossrcallablerrrr<rrrrrrr z"LossScaleOptimizer.get_scaled_lossc"d|_d|jfdd|DS)NT?c"g|] }|dur t|ndqSr _multiply_gradientr"loss_scale_reciprocalrrr%  z=LossScaleOptimizer.get_unscaled_gradients..rrrrrrr   z)LossScaleOptimizer.get_unscaled_gradientsc Cs|durtn|}| ||}Wdn1swY|jj||||d}dd|D}dd|D}||}tt||S)NtapecSg|]\}}|qSrrr#r_rrrr%z9LossScaleOptimizer._compute_gradients..cSg|]\}}|qSrrr#rrmrrrr%r)r GradientTaperr_compute_gradientsrlistzip) rrvar_list grad_lossrgrads_and_varsr&rlunscaled_gradsrrrrs  z%LossScaleOptimizer._compute_gradientscCs"||}|j||}||Sr )rr get_gradientsr)rrparamsr&rrrrs  z LossScaleOptimizer.get_gradientscC|j|dSr )r_create_all_weightsrrrrrrz&LossScaleOptimizer._create_all_weightsc stjr tdttjjt |}|r&j |}j |}t |}dd|Dtdd|Dfddfddtjjrj\}}fd d }tjj||}t||Sfd d }tjj|fd S)N6apply_gradients() must be called in a replica context.cSrrrrrrrr% rz6LossScaleOptimizer.apply_gradients..cSrrrrrrrr%&rcsjjjdddSNrF) read_valuer iterationsrrrsrrdo_not_apply_fn(sz;LossScaleOptimizer.apply_gradients..do_not_apply_fncs$tjtr j|StdfSNT)rrr9rhrr2)r&rsrr_if_should_apply_grads.s   zBLossScaleOptimizer.apply_gradients.._if_should_apply_gradscsSr _apply_gradientsr)r&r;r wrapped_varsrrapply_fn9z4LossScaleOptimizer.apply_gradients..apply_fncs>\}}fdd}tjj||}t||S)NcsjjjfdSNrzrrrr)rr&r;rrrrrKs z\LossScaleOptimizer.apply_gradients.._apply_gradients_cross_replica..apply_fn)rr smart_condr|)rr&rr;loss_scale_update_opshould_apply_gradsrmaybe_apply_oprrr)rr&r;rr_apply_gradients_cross_replicaCs  zJLossScaleOptimizer.apply_gradients.._apply_gradients_cross_replicarz)rrrrrr8rroptimizer_utilsfilter_empty_gradientsr!_transform_unaggregated_gradients_aggregate_gradientstupler rstrategy_supports_no_merge_callr r|get_replica_context merge_call) rrr; experimental_aggregate_gradientsr r rrrr)rrr&r;rrrapply_gradientssD        z"LossScaleOptimizer.apply_gradientscCs|jjtt||j|ddS)NF)r;rrrrrr )rr&rr;rrrr]s z#LossScaleOptimizer._apply_gradientscC t|j}||j|j|jdSN)rrrrr serializerrrrrserialized_optimizerrrr get_configi zLossScaleOptimizer.get_configcCs|}d|vr`tj|dtjjjjtjjjj ddd}t |tjjjjr1d|d<|j |d<n(t |tjjjj rUd|d<|j |d<|j |d <|jd krTtd |fntd |d |d<t |dtjrm|d}n tj|d|dd}|d=||fi|S)Nr)FixedLossScaleDynamicLossScalez loss scale)module_objectsprintable_module_nameFrrTrrzqCannot deserialize LossScaleOptimizer with a DynamicLossScale whose multiplier is not 2. Got DynamicLossScale: %szSerialized LossScaleOptimizers with a LossScale that is neither a FixedLossScale nor a DynamicLossScale can no longer be deserializedrrcustom_objectsuse_legacy_optimizer)copyrdeserialize_keras_objectpoprrYrZmixed_precisionr$r%r_loss_scale_valuerKincrement_periodrMrrrr deserialize)rconfigr)rrrrr from_configrsP         zLossScaleOptimizer.from_configcC|jjSr rrrsrrrrzLossScaleOptimizer.iterationscC ||j_dSr r5rr_rrrrrycC |jSr rget_slot_namesrsrrrr;rz!LossScaleOptimizer.get_slot_namescCr9r r variablesrsrrrr=rzLossScaleOptimizer.variablescCr4r )rrlrsrrrrlr6zLossScaleOptimizer.weightscCr9r )r get_weightsrsrrrr>rzLossScaleOptimizer.get_weightscC |j|Sr )r set_weights)rrlrrrr@ zLossScaleOptimizer.set_weightscCr4r rclipnormrsrrrrCr6zLossScaleOptimizer.clipnormcCr7r rBrvalrrrrCrycCr4r rglobal_clipnormrsrrrrGr6z"LossScaleOptimizer.global_clipnormcCr7r rFrDrrrrGrycCr4r r clipvaluersrrrrIr6zLossScaleOptimizer.clipvaluecCr7r rHrDrrrrIrycCr?r )rr)rrrrrrrAz'LossScaleOptimizer._aggregate_gradientscC|j|||Sr )r_restore_slot_variable)r slot_namer_ slot_variablerrrrKs z)LossScaleOptimizer._restore_slot_variablecCrJr r _create_or_restore_slot_variablerslot_variable_positionrLr_rrrrOz3LossScaleOptimizer._create_or_restore_slot_variablecCs|j||Sr )rget_slot)rr/rLrrrrSrzLossScaleOptimizer.get_slotzeroscCrJr )radd_slot)rr/rL initializerrrrrUrzLossScaleOptimizer.add_slotc Cspzt||WSty7}z$|dks|dkr||dkrd}||jjvr1|j|WYd}~S|d}~ww)Nr_hyperlr learning_rate)object__getattribute__AttributeErrorrrW _get_hyper)rr;errrr[s z#LossScaleOptimizer.__getattribute__csHtt}d|vr ||jjO}d|jjvr |dt|S)NrrYrX)setr?__dir__rrWkeysaddr)rresultrNrrr`s  zLossScaleOptimizer.__dir__cs||dkrd}z|dkrt||d}Wn tyd}Ynw|dkr5||jjvr5|s5|j||dSt||dS)NrXrYrTFr)rZr[r\rrW _set_hyperr? __setattr__)rr;r has_attributerNrrres   zLossScaleOptimizer.__setattr__cCr4r rrYrsrrrrY!r6z LossScaleOptimizer.learning_ratecCr7r rgrrrrrY%rycCr4r rgrsrrrrX)r6zLossScaleOptimizer.lrcCr7r )rrXrrrrrX-ryTNN)NNrr )rT),rrrr_HAS_AGGREGATE_GRADrrrrrrrrrrrrrrrr" classmethodr3rsetterr;r=rlr>r@rCrGrIrrKrOrSrUr[r`rerYrXrrrrNrrKs `         [  5               rc@s:eZdZdZejjj   d3ddZe ddZ e dd Z e d d Z e d d Z e ddZe ddZddZddZd4ddZ d5ddZddZddZed4dd Ze d!d"Zejd#d"Ze d$d%Zd&d'Ze d(d)Zejd*d)Ze d+d,Zejd-d,Ze d.d/Zejd0d/Zd1d2ZdS)6raPAn optimizer that applies loss scaling to prevent numeric underflow. This is a copy of the `mixed_precision.LossScaleOptimizer` class defined above, except it subclasses and wraps the new experimental Optimizer class instead of the `tf.keras.optimizers.Optimizer` class. Some of the methods this class defines and calls are different compared to LossScaleOptimizer due to the differences between the two Optimizer base classes. Additionally, this class does not support the legacy graph mode, but LossScaleOptimizer does. Since the new experimental Optimizer does not have a hyperparameter concept, LossScaleOptimizerV3 does not delegate arbitrary hyperparameter accesses to the inner optimizer, unlike LossScaleOptimizer. LossScaleOptimizerV3 does delegate the "learning_rate" attribute, however. TNcCs"t|tjst|tjrtd|td|dt|ts)tdt|t|tr5td|t t |ddrBt d||_ d |j _ tjjj||j |rq|dur[t}|durat}t||d d |_||jd n|duryt d t||_|durt d|d|_d|_dS)Na]You passed a `tf.keras.optimizers.Optimizer` instance to LossScaleOptimizerV3, but only the new experimental optimizer defined in keras/optimizer_expeirmental/optimizer.py can be passed. Please use `tf.keras.mixed_precision.LossScaleOptimizer` instead of LossScaleOptimizerV3, as the former supports `tf.keras.optimizers.Optimizer`s. Got optimizer: z="inner_optimizer" must be an instance of Optimizer, but got: rzK"dynamic" argument to LossScaleOptimizer.__init__ must be a bool, but got: zDLossScaleOptimizer cannot wrap another LossScaleOptimizer, but got: rFrTrrrrzD"dynamic_growth_steps" must be None if "dynamic" is False, but got: )rrrrrrrreprrrrrrrrrrrrrrr9rrr@rr)rrrrrrrrrUst        zLossScaleOptimizerV3.__init__cCrr rrsrrrrryzLossScaleOptimizerV3.dynamiccCrr rrsrrrrrzLossScaleOptimizerV3.loss_scalecCrr rrsrrrrrz$LossScaleOptimizerV3.dynamic_countercCrr rrsrrrrrz"LossScaleOptimizerV3.initial_scalecCrr rrsrrrrrz)LossScaleOptimizerV3.dynamic_growth_stepscCrrr rrsrrrrrtz$LossScaleOptimizerV3.inner_optimizercr)NTcrr rrrrrrrz6LossScaleOptimizerV3.get_scaled_loss..new_lossrrrrrrrz$LossScaleOptimizerV3.get_scaled_losscr)NTrcrr rr"rrrr%rz?LossScaleOptimizerV3.get_unscaled_gradients..rrrrrrrz+LossScaleOptimizerV3.get_unscaled_gradientscCs|durtn|}| ||}Wdn1swY|jj|||d}dd|D}dd|D}||}tt||S)NrcSrrrrrrrr%rz:LossScaleOptimizerV3.compute_gradients..cSrrrrrrrr%r)rrrrcompute_gradientsrrr)rrrrrr&rlrrrrrms  z&LossScaleOptimizerV3.compute_gradientsFc  sPt|}t|\}t |Wdn1swYtjr,tdtt j j t |}|dd}jjpHjj}|sU|rU|sUj|}t|}dd|Dtdd|Dfddfd d tjjr}fd d }tjj||dSfd d} tjj| fddS)NrrTcSrrrrrrrr%$rz8LossScaleOptimizerV3.apply_gradients..cSrrrrrrrr%*rcsjjjddddSrrrrsrrr,sz=LossScaleOptimizerV3.apply_gradients..do_not_apply_fncs$tjtrj|\}}|SdSr)rrr9rh)r&rshould_apply_gradrsrrr2s zDLossScaleOptimizerV3.apply_gradients.._if_should_apply_gradscs Sr rr)r&rrrrr<rAz6LossScaleOptimizerV3.apply_gradients..apply_fncs0}fdd}tjj||dS)NcsjjjfddSr r r)rr&rrrrrIs  z^LossScaleOptimizerV3.apply_gradients.._apply_gradients_cross_replica..apply_fn)rrr )rr&rr rr)rr&rrrDs  zLLossScaleOptimizerV3.apply_gradients.._apply_gradients_cross_replicarz)rrr init_scopebuildrrrrr8rrrrr-r_run_with_dtensor_meshaggregate_gradientsrr rrr rr) rrskip_gradients_aggregationrktrainable_variablesrrun_with_dtensorr rrr)rrr&rrrrsT            z$LossScaleOptimizerV3.apply_gradientscCs |jjtt||jdddS)NT)rtr)rr&rrrrrZs z%LossScaleOptimizerV3._apply_gradientscCrrrr rrrr"br#zLossScaleOptimizerV3.get_configcCsL|}t|dtjr|d}n tj|d|dd}|d=||fi|S)NrFr()r+rrrrr1)rr2r)rrrrr3ks z LossScaleOptimizerV3.from_configcCr4r r5rsrrrryr6zLossScaleOptimizerV3.iterationscCr7r r5r8rrrr}rycCr4r r<rsrrrr=r6zLossScaleOptimizerV3.variablescCr?r )rrprrrrrprAzLossScaleOptimizerV3.buildcCr4r rgrsrrrrYr6z"LossScaleOptimizerV3.learning_ratecCr7r rg)rrYrrrrYrycCr4r ruse_emarsrrrrxr6zLossScaleOptimizerV3.use_emacCr7r rw)rrxrrrrxrycCr4r r ema_momentumrsrrrrzr6z!LossScaleOptimizerV3.ema_momentumcCr7r ry)rrzrrrrzrycCrr )rfinalize_variable_valuesrrrrr{rz-LossScaleOptimizerV3.finalize_variable_valuesrhr )F) rrrrrrr no_automatic_dependency_trackingrrrrrrrrrrrmrrr"rjr3rrkr=rprYrxrzr{rrrrr@s` W        g           rc@s(eZdZdZddZddZddZdS) ra;A fake optimizer used to support restoring TensorFlow 2.2 checkpoints. The checkpoint format for LossScaleOptimizers changed after TF 2.2. This class exists to support restoring TF 2.2 checkpoints in newer version of TensorFlow. In TF 2.2, LossScaleOptimizer would track the wrapped optimizer by calling the following in LossScaleOptimizer.__init__ ``` self._track_trackable(self._optimizer, 'base_optimizer') ``` This means a dependency from the LossScaleOptimizer to the wrapped optimizer would be stored in the checkpoint. However now, the checkpoint format with a LossScaleOptimizer is the same as the format without a LossScaleOptimizer, except the loss scale is also stored. This means there is no dependency from the LossScaleOptimizer to the wrapped optimizer. Instead, the LossScaleOptimizer acts as if it is the wrapped optimizer, from a checkpoint's perspective, by overriding all Trackable methods and delegating them to the wrapped optimizer. To allow restoring TF 2.2. checkpoints, LossScaleOptimizer adds a dependency on this class instead of the inner optimizer. When restored, this class will instead restore the slot variables of the inner optimizer. Since this class has no variables, it does not affect the checkpoint when saved. cCr r r)rrrrrrrz$FakeOptimizerForRestoration.__init__cCr9r r:rsrrrr;rz*FakeOptimizerForRestoration.get_slot_namescCrJr rNrPrrrrOrRzt d|t||j |jdSt|tjjjjrWtd|t d|) aFCreates an LSO from a tf.compat.v1.mixed_precision.LossScale. This is only used to pass to `tf.__internal__.mixed_precision.register_loss_scale_wrapper` below, which is called so that `tf.compat.v1.mixed_precision.enable_mixed_precision_graph_rewrite` can wrap a Keras optimizer with a LossScaleOptimizer. Args: optimizer: An OptimizerV2 instance. loss_scale: A `tf.compat.v1.mixed_precision.LossScale` instance Returns: A LossScaleOptimizer that wraps `optimizer` and uses the same loss scaling algorithm as `loss_scale`. F)rrrrz]When passing a DynamicLossScale to "loss_scale", DynamicLossScale.multiplier must be 2. Got: )rrz^Passing a LossScale that is not a FixedLossScale or a DynamicLossScale is not supported. Got: zInvalid value passed to loss_scale. loss_scale must be the string "dynamic" (recommended), an int, a float, a FixedLossScale, or a DynamicLossScale. Got value: )rrBr@rrrYrZr.r$r/r%rMrrKr0 LossScaler)rrls_valrrr/_create_loss_scale_optimizer_from_v1_loss_scalesD rcCs<t||j}t|tjrtj|j||j|jdS||S)z@Multiply a (possibly sparse) gradient by the given scale factor.) dense_shape)rrr<rrrindicesr)gradientscalerrrr s rcCs\tjsdStj}t|tjjtjjjjjtjj tjjjj tjj tjjjj fp-t S)z;Returns True if the current Strategy supports loss scaling.T) rrrrrMultiWorkerMirroredStrategyrYrZ experimentalOneDeviceStrategyMirroredStrategy dtensor_utilsrunning_with_dtensor_strategystrategyrrrstrategy_supports_loss_scalings       rcCsTts(tj}t|tjjjtjjjjjtjjfrt dt d|j j ddS)zLRaise an exception if the current strategy doesn't support loss scaling.zLoss scaling is not supported with TPUStrategy. Loss scaling is unnecessary with TPUs, since they support bfloat16 instead of float16 and bfloat16 does not require loss scaling. You should remove the use of the LossScaleOptimizer when TPUs are used.z?Loss scaling is not supported with the tf.distribute.Strategy: z9. Try using a different Strategy, e.g. a MirroredStrategyN) rrrrrr TPUStrategyrYrZrrOrrrrrr6s$  r)0rtensorflow.compat.v2rYv2r keras.srcrrkeras.src.dtensorrrkeras.src.optimizersrrkeras.src.optimizers.legacyrkeras.src.savingrtensorflow.python.platformr tensorflow.python.util.tf_exportr r r(r,r3r8rr Trackabler9rrtyperrrrrrrrrr.register_loss_scale_wrapperrrrrrrrsb          .8" d xf+6