HackerNews – customFunctionsGeneral Part 1: write_log

news, daily newspaper, press-1172463.jpg

When doing automated tasks, recording what is going on and executed is a necessity, so we know if there are any errors and where exactly they occur. For this HackerNews project, we will write a log in txt using a function that we will call from the main file. So, let’s create a function to write log in the customFunctionsGeneral file.

import sharedvars as sv
from dotenv import load_dotenv
import os as os
from sqlalchemy import create_engine
import pandas as pd
import sys

def write_log(message=''):

    """ Retrieves message given as parameter and logs execution details and
    timestamp in file specified in sharedVars.py. If file hasn't been created,
    then create a new file designated to the current time of execution.
    # Initiate a file name to write
    if sv.file_name_for_log is None:
        sv.file_name_for_log = dt.datetime.now().strftime('%Y_%m_%d_%H_%M_%S')

    # Check if a text is provided
    if message == '':
        print('Hey you need to provide a message to write in a log file')
    # If provided, we will write text message to the file
        with open(f'{sv.file_name_for_log}.txt', 'a') as f:
            f.write('On ' + dt.datetime.now().strftime('%b %d, %Y') + ' at ' +
                    dt.datetime.now().strftime('%H:%M:%S') + ', '
                    + message + '\n')

Whenever the write_log function is called by running the main.py file, with specific parameters (message) in my main file, the function will create a new log file and record whatever is put into write_log. Therefore, when the main.py file is run, there should be one write_log file containing all the time and executions of the ETL process.