o i er@sdZddlZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z ddl mZddlmZd ZGd d d eZed d dZedgdddZeje_edgdddZedddZdiddZddZedgddd Zed!d"d#Zed$gdd%d&Zed'd(d)Zed*gdd+d,Zed-d.d/Zed0did1d2Z ed3gddid4d5Z!e je!_ed6gddid7d8Z"ed9did:d;Z#djd=d>Z$ed?gdd@dAZ%edBdCdDZ&edEgddFdGZ'edHdIdJZ(dKdLZ)edMgddNdOZ*edPdQdRZ+edSdTdUZedVgddjdWdXZ,edYdkdZd[Z-ed\gdd]d^Z.ed_d`daZ/dbdcZ0efdddeZ1edfdgdhZ2dS)lz1File IO methods that wrap the C++ FileSystem API.N)join)errors)_pywrap_file_io)compat) deprecation) tf_exportic@seZdZdZd2ddZeddZeddZd d Zd d Z d dZ ddZ ddZ d3ddZ edddd4ddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1ZdS)5FileIOa FileIO class that exposes methods to read / write to / from files. The constructor takes the following arguments: name: [path-like object](https://docs.python.org/3/glossary.html#term-path-like-object) giving the pathname of the file to be opened. mode: one of `r`, `w`, `a`, `r+`, `w+`, `a+`. Append `b` for bytes mode. Can be used as an iterator to iterate over lines in the file. The default buffer size used for the BufferedInputStream used for reading the file line by line is 1024 * 512 bytes. utf-8cCsb||_||_||_d|_d|_d|v|_|dd}|dvr%tddd|dv|_ |dv|_ dS)Nb)rwar+w+a+z5mode is not 'r' or 'w' or 'a' or 'r+' or 'w+' or 'a+')r rrr)rr rrr) _FileIO__name _FileIO__mode_FileIO__encoding _read_buf_writable_file _binary_modereplacerInvalidArgumentError_read_check_passed_write_check_passed)selfnamemodeencodingr O/var/www/myenv/lib/python3.10/site-packages/tensorflow/python/lib/io/file_io.py__init__0s   zFileIO.__init__cC|jS)zReturns the file name.)rrr r r!r>z FileIO.namecCr#)z.Returns the mode in which the file was opened.)rr$r r r!rCr%z FileIO.modecCs8|js|js tdddtt|jd|_dSdS)NzFile isn't open for readingi) rrrPermissionDeniedErrorrBufferedInputStreamr path_to_strrr$r r r!_preread_checkHs  zFileIO._preread_checkcCs@|js|js tdddtt|jt |j |_dSdS)NzFile isn't open for writing) rrrr&r WritableFiler path_to_bytesras_bytesrr$r r r!_prewrite_checkPs zFileIO._prewrite_checkcCs&|jr tj||jdStj||jdS)Nr)rrr,r as_str_any)rvalr r r!_prepare_valueXszFileIO._prepare_valuecCs t|jjS)zReturns the size of the file.)statrlengthr$r r r!size^ z FileIO.sizecCs$||jtj||jddS)z@Writes file_content to the file. Appends to the end of the file.r.N)r-rappendrr,r)r file_contentr r r!writebsz FileIO.writecCs8||dkr||}n|}||j|S)a=Returns the contents of a file as a string. Starts reading from current position in file. Args: n: Read `n` bytes if `n != -1`. If `n = -1`, reads to end of file. Returns: `n` bytes of the file (or whole file) in bytes mode or `n` bytes of the string if in string (regular) mode. r9)r)r4tellr1rread)rnr3r r r!r;hs  z FileIO.readNz7position is deprecated in favor of the offset argument.positionrcCs||dur|durtd|dur|durtd|dur"|}|dkr'n |dkr2||7}n|dkr=||7}n tddd||j|dS)a>Seeks to the offset in the file. Args: offset: The byte count relative to the whence argument. whence: Valid values for whence are: 0: start of the file (default) 1: relative to the current position of the file 2: relative to the end of file. `offset` is usually negative. Nz seek(): offset argument requiredz:seek(): offset and position may not be set simultaneously.rz9Invalid whence argument: {}. Valid values are 0, 1, or 2.) r) TypeErrorr:r4rrformatrseek)roffsetwhencer=r r r!rB{s(z FileIO.seekcCs|||jS)z4Reads the next line, keeping \n. At EOF, returns ''.)r)r1rreadliner$r r r!rEszFileIO.readlinecCs,|g} |}|s |S||q)z*Returns all lines from the file in a list.)r)rEr6)rlinessr r r! readliness zFileIO.readlinescCs*|jr ||jS||jS)z)Returns the current position in the file.)rr)rr:r-rr$r r r!r:s   z FileIO.tellcC|S)"Make usable with "with" statement.r r$r r r! __enter__zFileIO.__enter__cCs |dS)rJN)close)r unused_type unused_valueunused_tracebackr r r!__exit__r5zFileIO.__exit__cCrINr r$r r r!__iter__szFileIO.__iter__cCs|}|s t|SrR)rE StopIteration)rretvalr r r!__next__szFileIO.__next__cCs|SrR)rVr$r r r!nextsz FileIO.nextcCs|jr |jdSdS)aFlushes the Writable file. This only ensures that the data has made its way out of the process without any guarantees on whether it's written to disk. This means that the data would survive an application crash but not necessarily an OS crash. N)rflushr$r r r!rXsz FileIO.flushcCs$d|_|jr|jd|_dSdS)aCloses the file. Should be called for the WritableFile to be flushed. In general, if you use the context manager pattern, you don't need to call this directly. >>> with tf.io.gfile.GFile("/tmp/x", "w") as f: ... f.write("asdf\n") ... f.write("qwer\n") >>> # implicit f.close() at the end of the block For cloud filesystems, forgetting to call `close()` might result in data loss as last write might not have been replicated. N)rrrMr$r r r!rMs   z FileIO.closecCsdS)zBReturns True as FileIO supports random access ops of seek()/tell()Tr r$r r r!seekablerLzFileIO.seekable)r )r9)NrN)__name__ __module__ __qualname____doc__r"propertyrrr)r-r1r4r8r;rdeprecated_argsrBrErHr:rKrQrSrVrWrXrMrYr r r r!r"s:     )   rzio.gfile.existscCs.z tt|WdStjyYdSw)aDetermines whether a path exists or not. >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.exists("/tmp/x") True You can also specify the URI scheme for selecting a different filesystem: >>> # for a GCS filesystem path: >>> # tf.io.gfile.exists("gs://bucket/file") >>> # for a local filesystem: >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.exists("file:///tmp/x") True This currently returns `True` for existing directories but don't rely on this behavior, especially if you are using cloud filesystems (e.g., GCS, S3, Hadoop): >>> tf.io.gfile.exists("/tmp") True Args: path: string, a path Returns: True if the path exists, whether it's a file or a directory. False if the path does not exist and there are no filesystem errors. Raises: errors.OpError: Propagates any errors reported by the FileSystem API. FT)r FileExistsrr+r NotFoundErrorpathr r r!file_exists_v2s (rdz gfile.Exists)v1cCt|SrR)rdfilenamer r r! file_exists(sriz gfile.RemovecC t|dS)zDeletes the file located at 'filename'. Args: filename: string, a filename Raises: errors.OpError: Propagates any errors reported by the FileSystem API. E.g., `NotFoundError` if the file does not exist. N)delete_file_v2rgr r r! delete_file0s rlzio.gfile.removecCtt|dS)zDeletes the path located at 'path'. Args: path: string, a path Raises: errors.OpError: Propagates any errors reported by the FileSystem API. E.g., `NotFoundError` if the path does not exist. N)r DeleteFilerr+rbr r r!rk>s rkFcCs,|r t|dd}|St|dd}|S)ayReads the entire contents of a file to a string. Args: filename: string, path to a file binary_mode: whether to open the file in binary mode or not. This changes the type of the object returned. Returns: contents of the file as a string or bytes. Raises: errors.OpError: Raises variety of errors that are subtypes e.g. `NotFoundError` etc. rbrr )rr;)rh binary_modefr r r!read_file_to_stringLs   rscCs<t|dd}||WddS1swYdS)zWrites a string to a given file. Args: filename: string, path to a file file_content: string, contents that need to be written to the file Raises: errors.OpError: If there are errors during the operation. r rpN)rr8)rhr7rrr r r!write_string_to_filebs "rtz gfile.GlobcCrf)aReturns a list of files that match the given pattern(s). Args: filename: string or iterable of strings. The glob pattern(s). Returns: A list of strings containing filenames that match the given pattern(s). Raises: * errors.OpError: If there are filesystem / directory listing errors. * errors.NotFoundError: If pattern to be matched is an invalid directory. )get_matching_files_v2rgr r r!get_matching_filespsrvz io.gfile.globcCs4t|tjrddtt|DSdd|DS)aReturns a list of files that match the given pattern(s). The patterns are defined as strings. Supported patterns are defined here. Note that the pattern can be a Python iteratable of string patterns. The format definition of the pattern is: **pattern**: `{ term }` **term**: * `'*'`: matches any sequence of non-'/' characters * `'?'`: matches a single non-'/' character * `'[' [ '^' ] { match-list } ']'`: matches any single character (not) on the list * `c`: matches character `c` where `c != '*', '?', '\\', '['` * `'\\' c`: matches character `c` **character range**: * `c`: matches character `c` while `c != '\\', '-', ']'` * `'\\' c`: matches character `c` * `lo '-' hi`: matches character `c` for `lo <= c <= hi` Examples: >>> tf.io.gfile.glob("*.py") ... # For example, ['__init__.py'] >>> tf.io.gfile.glob("__init__.??") ... # As above >>> files = {"*.py"} >>> the_iterator = iter(files) >>> tf.io.gfile.glob(the_iterator) ... # As above See the C++ function `GetMatchingPaths` in [`core/platform/file_system.h`] (../../../core/platform/file_system.h) for implementation details. Args: pattern: string or iterable of strings. The glob pattern(s). Returns: A list of strings containing filenames that match the given pattern(s). Raises: errors.OpError: If there are filesystem / directory listing errors. errors.NotFoundError: If pattern to be matched is an invalid directory. cSg|]}t|qSr rr/).0matching_filenamer r r! sz)get_matching_files_v2..cSs,g|]}tt|D]}t|q qSr )rGetMatchingFilesrr,r/)rysingle_filenamerzr r r!r{s) isinstancesix string_typesrr|rr,)patternr r r!rus 4ruz gfile.MkDircCrj)a=Creates a directory with the name `dirname`. Args: dirname: string, name of the directory to be created Notes: The parent directories need to exist. Use `tf.io.gfile.makedirs` instead if there is the possibility that the parent dirs don't exist. Raises: errors.OpError: If the operation fails. N) create_dir_v2dirnamer r r! create_dirs rzio.gfile.mkdircCrm)a@Creates a directory with the name given by `path`. Args: path: string, name of the directory to be created Notes: The parent directories need to exist. Use `tf.io.gfile.makedirs` instead if there is the possibility that the parent dirs don't exist. Raises: errors.OpError: If the operation fails. N)r CreateDirrr+rbr r r!rs rzgfile.MakeDirscCrj)zCreates a directory and all parent/intermediate directories. It succeeds if dirname already exists and is writable. Args: dirname: string, name of the directory to be created Raises: errors.OpError: If the operation fails. N)recursive_create_dir_v2rr r r!recursive_create_dirs rzio.gfile.makedirscCrm)zCreates a directory and all parent/intermediate directories. It succeeds if path already exists and is writable. Args: path: string, name of the directory to be created Raises: errors.OpError: If the operation fails. N)rRecursivelyCreateDirrr+rbr r r!rs rz io.gfile.copycCtt|t||dS)aCopies data from `src` to `dst`. >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.exists("/tmp/x") True >>> tf.io.gfile.copy("/tmp/x", "/tmp/y") >>> tf.io.gfile.exists("/tmp/y") True >>> tf.io.gfile.remove("/tmp/y") You can also specify the URI scheme for selecting a different filesystem: >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.copy("/tmp/x", "file:///tmp/y") >>> tf.io.gfile.exists("/tmp/y") True >>> tf.io.gfile.remove("/tmp/y") Note that you need to always specify a file name, even if moving into a new directory. This is because some cloud filesystems don't have the concept of a directory. >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.mkdir("/tmp/new_dir") >>> tf.io.gfile.copy("/tmp/x", "/tmp/new_dir/y") >>> tf.io.gfile.exists("/tmp/new_dir/y") True >>> tf.io.gfile.rmtree("/tmp/new_dir") If you want to prevent errors if the path already exists, you can use `overwrite` argument: >>> with open("/tmp/x", "w") as f: ... f.write("asdf") ... 4 >>> tf.io.gfile.copy("/tmp/x", "file:///tmp/y") >>> tf.io.gfile.copy("/tmp/x", "file:///tmp/y", overwrite=True) >>> tf.io.gfile.remove("/tmp/y") Note that the above will still result in an error if you try to overwrite a directory with a file. Note that you cannot copy a directory, only file arguments are supported. Args: src: string, name of the file whose contents need to be copied dst: string, name of the file to which to copy to overwrite: boolean, if false it's an error for `dst` to be occupied by an existing file. Raises: errors.OpError: If the operation fails. N)rCopyFilerr+srcdst overwriter r r!copy_v2sArz gfile.CopycCst|||dSrR)r)oldpathnewpathrr r r!copyIsrz gfile.RenamecCst|||dS)a7Rename or move a file / directory. Args: oldname: string, pathname for a file newname: string, pathname to which the file needs to be moved overwrite: boolean, if false it's an error for `newname` to be occupied by an existing file. Raises: errors.OpError: If the operation fails. N) rename_v2)oldnamenewnamerr r r!renameQ rzio.gfile.renamecCr)a+Rename or move a file / directory. Args: src: string, pathname for a file dst: string, pathname to which the file needs to be moved overwrite: boolean, if false it's an error for `dst` to be occupied by an existing file. Raises: errors.OpError: If the operation fails. N)r RenameFilerr+rr r r!ras rTcCs`t|s t||dS|dtj}t||z t|||WdStjy/t|w)a!Writes to `filename` atomically. This means that when `filename` appears in the filesystem, it will contain all of `contents`. With write_string_to_file, it is possible for the file to appear in the filesystem with `contents` only partially written. Accomplished by writing to a temp file and then renaming it. Args: filename: string, pathname for a file contents: string, contents that need to be written to the file overwrite: boolean, if false it's an error for `filename` to be occupied by an existing file. z.tmpN) has_atomic_movertuuiduuid4hexrrOpErrorrl)rhcontentsr temp_pathnamer r r!atomic_write_string_to_filers rzgfile.DeleteRecursivelycCrj)zDeletes everything under dirname recursively. Args: dirname: string, a path to a directory Raises: errors.OpError: If the operation fails. N)delete_recursively_v2rr r r!delete_recursivelys rzio.gfile.rmtreecCrm)zDeletes everything under path recursively. Args: path: string, a path Raises: errors.OpError: If the operation fails. N)rDeleteRecursivelyrr+rbr r r!rs rzgfile.IsDirectorycCrf)zReturns whether the path is a directory or not. Args: dirname: string, path to a potential directory Returns: True, if the path is a directory; False otherwise )is_directory_v2rr r r! is_directorys rzio.gfile.isdircC*z tt|WStjyYdSw)zReturns whether the path is a directory or not. Args: path: string, path to a potential directory Returns: True, if the path is a directory; False otherwise F)r IsDirectoryrr+rrrbr r r!rs  rcCr)auChecks whether the file system supports atomic moves. Returns whether or not the file system of the given path supports the atomic move operation for a file or folder. If atomic move is supported, it is recommended to use a temp location for writing and then move to the final location. Args: path: string, path to a file Returns: True, if the path is on a file system that supports atomic move False, if the file system does not support atomic move. In such cases we need to be careful about using moves. In some cases it is safer not to use temporary locations in this case. T)r HasAtomicMoverr+rrrbr r r!rs rzgfile.ListDirectorycCrf)aGReturns a list of entries contained within a directory. The list is in arbitrary order. It does not contain the special entries "." and "..". Args: dirname: string, path to a directory Returns: [filename1, filename2, ... filenameN] as strings Raises: errors.NotFoundError if directory doesn't exist )list_directory_v2rr r r!list_directorysrzio.gfile.listdircCs8t|stjddd|dddtt|DS)aDReturns a list of entries contained within a directory. The list is in arbitrary order. It does not contain the special entries "." and "..". Args: path: string, path to a directory Returns: [filename1, filename2, ... filenameN] as strings Raises: errors.NotFoundError if directory doesn't exist NzCould not find directory {})node_defopmessagecSrwr rx)ryrhr r r!r{sz%list_directory_v2..)rrrarAr GetChildrenrr+rbr r r!rsrz io.gfile.joincGsDtt|}d|ddvrt|g|RStjj|g|RS)aJoin one or more path components intelligently. TensorFlow specific filesystems will be joined like a url (using "/" as the path seperator) on all platforms: On Windows or Linux/Unix-like: >>> tf.io.gfile.join("gcs://folder", "file.py") 'gcs://folder/file.py' >>> tf.io.gfile.join("ram://folder", "file.py") 'ram://folder/file.py' But the native filesystem is handled just like os.path.join: >>> path = tf.io.gfile.join("folder", "file.py") >>> if os.name == "nt": ... expected = "folder\\file.py" # Windows ... else: ... expected = "folder/file.py" # Linux/Unix-like >>> path == expected True Args: path: string, path to a directory paths: string, additional paths to concatenate Returns: path: the joined path. z://r>N)rr/r(urljoinosrcr)rcpathspath_r r r!r s rz gfile.WalkcCs t||S)aRecursive directory tree generator for directories. Args: top: string, a Directory name in_order: bool, Traverse in order if True, post order if False. Errors that happen while listing directories are ignored. Yields: Each yield is a 3-tuple: the pathname of a directory, followed by lists of all its subdirectories and leaf files. That is, each yield looks like: `(dirname, [subdirname, subdirname, ...], [filename, filename, ...])`. Each item is a string. )walk_v2)topin_orderr r r!walk3s rz io.gfile.walkc csdd}tt|}zt|}Wn!tjy4}z|r#||nWYd}~dSWYd}~nd}~wwg}g}|D]}|||} t| rL||q;||q;|||f} |r\| V|D]} t||| ||dD]} | Vqjq^|sx| VdSdS)aRecursive directory tree generator for directories. Args: top: string, a Directory name topdown: bool, Traverse pre order if True, post order if False. onerror: optional handler for errors. Should be a function, it will be called with the error as argument. Rethrowing the error aborts the walk. Errors that happen while listing directories are ignored. Yields: Each yield is a 3-tuple: the pathname of a directory, followed by lists of all its subdirectories and leaf files. That is, each yield looks like: `(dirname, [subdirname, subdirname, ...], [filename, filename, ...])`. Each item is a string. cSs,|dtjkrdt|d|gSt||S)Nrr )rsepr)parentitemr r r!_make_full_pathWs z walk_v2.._make_full_pathN)onerror) rr/r(rrrarr6r) rtopdownrrlistingerrfilessubdirsr full_pathheresubdirsubitemr r r!rEs>          rz gfile.StatcCrf)zReturns file statistics for a given path. Args: filename: string, path to a file Returns: FileStatistics struct that contains information about the path Raises: errors.OpError: If the operation fails. )stat_v2rgr r r!r2s r2z io.gfile.statcCstt|S)zReturns file statistics for a given path. Args: path: string, path to a file Returns: FileStatistics struct that contains information about the path Raises: errors.OpError: If the operation fails. )rStatrr(rbr r r!rrrcCs@t|d}t|d}||krdSt|}t|}||kS)aCompare two files, returning True if they are the same, False otherwise. We check size first and return False quickly if the files are different sizes. If they are the same size, we continue to generating a crc for the whole file. You might wonder: why not use Python's `filecmp.cmp()` instead? The answer is that the builtin library is not robust to the many different filesystems TensorFlow runs on, and so we here perform a similar comparison with the more robust FileIO. Args: filename_a: string path to the first file. filename_b: string path to the second file. Returns: True if the files are the same, False otherwise. roF)rr4 file_crc32) filename_a filename_bsize_asize_bcrc_acrc_br r r!filecmpsrcCshd}t|dd}|j|d}|rt||}|j|d}|sWdn1s)wYt|d@S)aGet the crc32 of the passed file. The crc32 of a file can be used for error checking; two files with the same crc32 are considered equivalent. Note that the entire file must be read to produce the crc32. Args: filename: string, path to a file block_size: Integer, process the files by reading blocks of `block_size` bytes. Use -1 to read the file as once. Returns: hexadecimal as string, the crc32 of the passed file. rrorp)r<Nl)rr;binasciicrc32r)rh block_sizecrcrrchunkr r r!rs    rzio.gfile.get_registered_schemescCstS)aReturns the currently registered filesystem schemes. The `tf.io.gfile` APIs, in addition to accepting traditional filesystem paths, also accept file URIs that begin with a scheme. For example, the local filesystem path `/tmp/tf` can also be addressed as `file:///tmp/tf`. In this case, the scheme is `file`, followed by `://` and then the path, according to [URI syntax](https://datatracker.ietf.org/doc/html/rfc3986#section-3). This function returns the currently registered schemes that will be recognized by `tf.io.gfile` APIs. This includes both built-in schemes and those registered by other TensorFlow filesystem implementations, for example those provided by [TensorFlow I/O](https://github.com/tensorflow/io). The empty string is always included, and represents the "scheme" for regular local filesystem paths. Returns: List of string schemes, e.g. `['', 'file', 'ram']`, in arbitrary order. Raises: errors.OpError: If the operation fails. )rGetRegisteredSchemesr r r r!get_registered_schemessr)F)T)TN)3r]rr posixpathrrrrtensorflow.python.frameworkrtensorflow.python.lib.iortensorflow.python.utilrr tensorflow.python.util.tf_exportr_DEFAULT_BLOCK_SIZEobjectrrdrirlrkrsrtrvrurrrrrrrrrrrrrrrrrrr2rrrrr r r r!s      X  .         D       D               %   9