o i eF&@svdZddlmmZddlmZddlmZddl m Z ee ddgdGd d d ej Z e j d eje _dS) zFTRL optimizer implementation.N) optimizer)register_keras_serializable) keras_exportz"keras.optimizers.experimental.Ftrlzkeras.optimizers.Ftrl)v1csbeZdZdZ               dfd d Zfd d ZddZfddZZS)Ftrla Optimizer that implements the FTRL algorithm. "Follow The Regularized Leader" (FTRL) is an optimization algorithm developed at Google for click-through rate prediction in the early 2010s. It is most suitable for shallow models with large and sparse feature spaces. The algorithm is described by [McMahan et al., 2013](https://research.google.com/pubs/archive/41159.pdf). The Keras version has support for both online L2 regularization (the L2 regularization described in the paper above) and shrinkage-type L2 regularization (which is the addition of an L2 penalty to the loss function). Initialization: ```python n = 0 sigma = 0 z = 0 ``` Update rule for one variable `w`: ```python prev_n = n n = n + g ** 2 sigma = (n ** -lr_power - prev_n ** -lr_power) / lr z = z + g - sigma * w if abs(z) < lambda_1: w = 0 else: w = (sgn(z) * lambda_1 - z) / ((beta + sqrt(n)) / alpha + lambda_2) ``` Notation: - `lr` is the learning rate - `g` is the gradient for the variable - `lambda_1` is the L1 regularization strength - `lambda_2` is the L2 regularization strength - `lr_power` is the power to scale n. Check the documentation for the `l2_shrinkage_regularization_strength` parameter for more details when shrinkage is enabled, in which case gradient is replaced with a gradient with shrinkage. Args: learning_rate: A `Tensor`, floating point value, a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use. The learning rate. Defaults to `0.001`. learning_rate_power: A float value, must be less or equal to zero. Controls how the learning rate decreases during training. Use zero for a fixed learning rate. initial_accumulator_value: The starting value for accumulators. Only zero or positive values are allowed. l1_regularization_strength: A float value, must be greater than or equal to zero. Defaults to `0.0`. l2_regularization_strength: A float value, must be greater than or equal to zero. Defaults to `0.0`. l2_shrinkage_regularization_strength: A float value, must be greater than or equal to zero. This differs from L2 above in that the L2 above is a stabilization penalty, whereas this L2 shrinkage is a magnitude penalty. When input is sparse shrinkage will only happen on the active weights. beta: A float value, representing the beta value from the paper. Defaults to 0.0. {{base_optimizer_keyword_args}} MbP?皙?NFGz?Tc stjd ||| | | | | ||d ||dkrtd|d|dkr+td|d|dkr7td|d|dkrCtd|d|dkrOtd|d|||_||_||_||_||_||_ ||_ dS) N) name weight_decayclipnorm clipvalueglobal_clipnormuse_ema ema_momentumema_overwrite_frequency jit_compiler z^`initial_accumulator_value` needs to be positive or zero. Received: initial_accumulator_value=.zR`learning_rate_power` needs to be negative or zero. Received: learning_rate_power=z``l1_regularization_strength` needs to be positive or zero. Received: l1_regularization_strength=z``l2_regularization_strength` needs to be positive or zero. Received: l2_regularization_strength=zt`l2_shrinkage_regularization_strength` needs to be positive or zero. Received: l2_shrinkage_regularization_strength=) super__init__ ValueError_build_learning_rate_learning_ratelearning_rate_powerinitial_accumulator_valuel1_regularization_strengthl2_regularization_strength$l2_shrinkage_regularization_strengthbeta)self learning_raterrrrr r!r rrrrrrrr kwargs __class__rH/var/www/myenv/lib/python3.10/site-packages/keras/src/optimizers/ftrl.pyrdsf    z Ftrl.__init__c st|t|dr|jrdSg|_g|_|D]&}|j|j|dtj tj |j |j d|j dd|j|j|ddqd |_dS) zInitialize optimizer variables. Args: var_list: list of model variables to build Ftrl variables on. _builtN accumulator)dimsvalue)dtype)model_variable variable_name initial_valuelinear)r-r.T)rbuildhasattrr( _accumulators_linearsappendadd_variable_from_referencetfcastfillshaperr,)r"var_listvarr%rr'r1s0   z Ftrl.buildc Cst|j|j}||}|j|j|}|j|j|}|j}|j }||j d|}|d|j |} |t |d} | | t | | t || ||t | | |d|} t||j |j} || || || dS)z=Update step given gradient and the associated model variable.g@N)r7r8r#r,_var_keyr3 _index_dictr4rrr!r pow assign_add clip_by_valuerassign) r"gradientvariablelrvar_keyaccumr0lr_powerl2_reg grad_to_use new_accum quadraticlinear_clippedrrr' update_steps6 zFtrl.update_stepc s<t}|||j|j|j|j|j|j |j d|S)N)r#rrrrr r!) r get_configupdate_serialize_hyperparameterrrrrrr r!)r"configr%rr'rPs  zFtrl.get_config)rrr r r r r NNNNFr NTr) __name__ __module__ __qualname____doc__rr1rOrP __classcell__rrr%r'rs,G I!rz{{base_optimizer_keyword_args}})rWtensorflow.compat.v2compatv2r7keras.src.optimizersr$keras.src.saving.object_registrationr tensorflow.python.util.tf_exportr Optimizerrreplacebase_optimizer_keyword_argsrrrr's   c