The effectiveness of a product especially software, relies on how organised the basic building blocks are. With Python being used for all complex mathematical and machine learning processes, the aspect of file finding and handling might seem trivial or unattractive from the birds eye view. But documentation becomes crucial when the content of the product has to be worked on by a different team.

For instance, specific file documentations– these are usually generated through a third party script which will parse a file and, based on the comment blocks, will create an explicit PDF. Afterwards there should be information regarding the code repository, where the file updates are found, and where they need to be moved.

Here are few important file system methods that make Python easier to use:

f.write(string)

This method is used to write the contents of string to the file, returning the number of characters written.

os.path()

To test if a path exists (be it a file, directory or even link), use os.path.exists() is used.

import os.path

path_to_file = “/path/to/file”

if os.path.isfile(path_to_file):

This method returns True if the given path exists and it’s a file.

os.getcwd()

This method returns the current working directory as a string.

os.listdir()

os.listdir() returns the contents of the current working directory as a list of strings.

os.walk()

Creates a generator that can return information about the current directory and subdirectories. It works through the directories in the specified starting directory.

os.walk() returns the following items for each directory it traverses:

• current directory path as a string
• subdirectory names in the current directory as lists of strings
• filenames in current directory as a list of strings

It’s often useful to use os.walk() with a for loop to iterate over the contents of a directory and its subdirectories.

import os

cwd = os.getcwd()

for dir_path, dir_names, file_names in os.walk(cwd):

  for f in file_names:

      print(f)

os.chdir()

This method changes the current working directory to either the absolute or relative path provided.

It’s a good idea to handle any exceptions raised when using this method with try-except. Otherwise there is a chance of deleting directories or files you don’t want deleted.

os.path.join()

The os.path module has a number of useful methods for common pathname manipulations. You can use it to find information about directory names and parts of directory names. The module also has methods to check whether a file or directory exists.

os.path.join() is designed to create a path that will work on most any operating system by joining multiple strings into one file path.

os.makedirs()

os.makedirs() makes directories. The mkdir() method also makes directories, but it does not make intermediate directories. So I suggest you use os.makedirs().

shutil.copy2()

This method can be used to copy a file or directory .shutil.copy2()is a good choice because it tries to preserve as much of the source file’s metadata as possible.

shutil.copy2(“source_file_path”,”destination_directory_path”) 

shutil.move()

To move a file or directory — mv

shutil.move() can be used to change a file’s location. It uses copy2 as the default under the hood.

shutil.move(“source_file_path”,”destination_directory_path”) 

os.remove()

To remove a directory and all files and directories in it:

os.remove(“my_file_path”)

Recursive Looping

To loop over all files recursively under some path (i.e. including subdirectories) as follows:

import os

path = ‘/tmp/’

for root, directories, filenames in os.walk(path):
   for directory in directories:

       directory_path = os.path.join(root, directory)
    
   for filename in filenames:

       file_path = os.path.join(root,filename)

Read more about file systems here