o i eS5 @sdZddlZddlZddlZddlZddlmZddlZddl m Z ej ej ej ej ej ej eZej ejZdZdZddZd d Zd d Zd dZddZddZddZ    dddZ  dddZ    dddZdS)zJClasses and functions that help to inspect Python source w.r.t. TF graphs.N)app) profilingz.py)z.pycz.pyocCstjtj|SN)ospathnormpathabspath) file_pathr W/var/www/myenv/lib/python3.10/site-packages/tensorflow/python/debug/lib/source_utils.py_norm_abs_path'sr cCtj|\}}|tvSr)rrsplitextlowerUNCOMPILED_SOURCE_SUFFIXESr _ extensionr r r %is_extension_uncompiled_python_source+ rcCr r)rrrrCOMPILED_SOURCE_SUFFIXESrr r r #is_extension_compiled_python_source0rrcCs|d|dS)N:)rfind) watch_keyr r r !_convert_watch_key_to_tensor_name5srcCsTt|s t|s dSt|}|ts|to)|d o)tj dtj |vS)aGuess whether a Python source file is a part of the tensorflow library. Special cases: 1) Returns False for unit-test files in the library (*_test.py), 2) Returns False for files under python/debug/examples. Args: py_file_path: full path of the Python source file in question. Returns: (`bool`) Whether the file is inferred to be a part of the tensorflow library. Fz_test.pyz tensorflow/python/debug/examples) rrr startswith_TENSORFLOW_BASEDIR _ABSL_BASEDIRendswithrrr) py_file_pathr r r guess_is_tensorflow_py_library9s    r!cCstj|r(t|d}|d}Wdn1swY|d}nt|}|dur6td|t t t t |d}||fS)aLoad the content of a Python source code file. This function covers the following case: 1. source_file_path points to an existing Python (.py) file on the file system. 2. source_file_path is a path within a .par file (i.e., a zip-compressed, self-contained Python executable). Args: source_file_path: Path to the Python source file to read. Returns: A length-2 tuple: - Lines of the source file, as a `list` of `str`s. - The width of the string needed to show the line number in the file. This is calculated based on the number of lines in the source file. Raises: IOError: if loading is unsuccessful. rbutf-8N z?Source path neither exists nor can be loaded as a .par file: %s)rrisfileopenreaddecodesplit_try_load_par_sourceIOErrorintnpceillog10len)source_file_pathf source_text source_linesline_num_widthr r r load_sourceRs   r7c Cs|} tj|\}}|sdStjtjj||d}|drwtj|rwt|D}dd| D}||vrh| | | |}| d}|dWdWdS1scwYWdn1srwYq) a[Try loading the source code inside a .par file. A .par file is a zip-compressed, self-contained Python executable. It contains the content of individual Python source files that can be read only through extracting from the zip file. Args: source_file_path: The full path to the file inside the .par file. This path should include the path to the .par file itself, followed by the intra-par path, e.g., "/tmp/my_executable.par/org-tensorflow/tensorflow/python/foo/bar.py". Returns: If successful, lines of the source file as a `list` of `str`s. Else, `None`. T)startz.parcSsg|]}tj|qSr )rrr).0namer r r sz(_try_load_par_source..r#r$N)rrr*rrelpathrr&zipfileZipFilenamelistr'indexr(r))r2 prefix_pathbasename suffix_pathz norm_nameszfr4r r r r+ws* r+FcCs|j}|s tdt|}i}|D]S}t||jD]H\} } } } |dur+| |ks3|dur4| |kr4qt| |kr;q|rM||j} tt t t | } n|jg} | |vr]||  | n| || <|renqq|S)aAnnotate a Python source file with a list of ops created at each line. (The annotation doesn't change the source file itself.) Args: dump: (`DebugDumpDir`) A `DebugDumpDir` object of which the Python graph has been loaded. source_file_path: (`str`) Path to the source file being annotated. do_dumped_tensors: (`str`) Whether dumped Tensors, instead of ops are to be used to annotate the source file. file_stack_top: (`bool`) Whether only the top stack trace in the specified source file is to be annotated. min_line: (`None` or `int`) The 1-based line to start annotate the source file from (inclusive). max_line: (`None` or `int`) The 1-based line number to end the annotation at (exclusive). Returns: A `dict` mapping 1-based line number to a list of op name(s) created at that line, or tensor names if `do_dumped_tensors` is True. Raises: ValueError: If the dump object does not have a Python graph set. zUCannot perform source annotation due to a lack of set Python graph in the dump objectN) python_graph ValueErrorr get_operationsreversednode_tracebackr:debug_watch_keyslistsetmaprextend)dumpr2do_dumped_tensorsfile_stack_topmin_linemax_linepy_graphline_to_op_namesopr line_numberr watch_keysitems_to_appendr r r annotate_sources4    r\c s|j}|s tdtt}tt}i}i|rt|nd}|r't|nd}t} |D]} |r;|| j s;q0| | j D]v\} } } } t | } | | vs\|rV|| r\t j | sb| | qA|| | j | |vry|| | krx| || <n| || <| jD] }|j }|| |q|| j }|D]!}|d\}}}d||f}|vrt||t|||<qqAq0i}|D]}tfdd||D||<qg}|D]!} || t| t|| it|| i|| d|| fqt|dd d S) a@Generate a list of source files with information regarding ops and tensors. Args: dump: (`DebugDumpDir`) A `DebugDumpDir` object of which the Python graph has been loaded. path_regex_allowlist: A regular-expression filter for source file path. node_name_regex_allowlist: A regular-expression filter for node names. Returns: A list of tuples regarding the Python source files involved in constructing the ops and tensors contained in `dump`. Each tuple is: (source_file_path, is_tf_library, num_nodes, num_tensors, num_dumps, first_line) is_tf_library: (`bool`) A guess of whether the file belongs to the TensorFlow Python library. num_nodes: How many nodes were created by lines of this source file. These include nodes with dumps and those without. num_tensors: How many Tensors were created by lines of this source file. These include Tensors with dumps and those without. num_dumps: How many debug Tensor dumps were from nodes (and Tensors) that were created by this source file. first_line: The first line number (1-based) that created any nodes or Tensors in this source file. The list is sorted by ascending order of source_file_path. Raises: ValueError: If the dump object does not have a Python graph set. zPCannot generate source list due to a lack of set Python graph in the dump objectNrz%s:%sc3s|] }|dVqdS)rN)get)r9 tensor_nametensor_name_to_num_dumpsr r /s  z1list_source_files_against_dump..rcSs|dS)Nrr )xr r r =sz0list_source_files_against_dump..)key)rGrH collections defaultdictrNrecompilerImatchr:rKr rrr&addoutputsrLr*r1 get_tensorsr-sumappendr!r]sorted)rQpath_regex_allowlistnode_name_regex_allowlistrVpath_to_node_namespath_to_tensor_namespath_to_first_line path_regexnode_name_regexto_skip_file_pathsrXr rYr output_tensorr^rZr node_name output_slotdebug_oppath_to_num_dumpsroutputr r_r list_source_files_against_dumps"            r~c Cst|}|r t|nd}|rt|nd}i}|D]J} | js qt| j|kr(q|dur1| j|ks:|dur;| j|kr;q|rE|| jjsEq|rN|| jsNq| j|vr\t | || j<q|| j | q|S)aAnnotate a Python source file with profiling information at each line. (The annotation doesn't change the source file itself.) Args: profile_data: (`list` of `ProfileDatum`) A list of `ProfileDatum`. source_file_path: (`str`) Path to the source file being annotated. node_name_filter: Regular expression to filter by node name. op_type_filter: Regular expression to filter by op type. min_line: (`None` or `int`) The 1-based line to start annotate the source file from (inclusive). max_line: (`None` or `int`) The 1-based line number to end the annotation at (exclusive). Returns: A `dict` mapping 1-based line number to a the namedtuple `profiling.LineOrFuncProfileSummary`. N) r rgrhr rYrinode_exec_statsryop_typerAggregateProfilerj) profile_datar2node_name_filterop_type_filterrTrUrv op_type_regexline_to_profile_summary profile_datumr r r annotate_source_against_profile@s.   r)FFNN)NN)NNNN)__doc__rerrgr=abslrnumpyr.tensorflow.python.debug.librrdirnamerr__file__rrrrr rrrr!r7r+r\r~rr r r r sH  %# D g