Everything a Dev Needs to Know about Python Logging

Python logging is a fundamental element in the development and maintenance of Python applications. 

Start logging & monitoring your Python applications in 60 seconds.

Get started free

When building software, it’s essential to understand what’s happening under the hood, especially when things go wrong. Python’s built-in logging library equips developers with the ability to capture and preserve valuable information about their application’s behavior.

This invaluable data serves as a reference point when diagnosing and addressing issues, streamlining the troubleshooting process.

In this article, we delve into the multifaceted world of Python logging, exploring various use cases, best practices, and proven strategies for overcoming the challenges of performance enhancement. 

What is Python Logging?

Python logging is a built-in logging module specific to applications built with Python. Developers use it to capture valuable information about code execution and software behavior during runtime. 

Python logging also provides multiple ways to configure the logging system, such as using basicConfig() for simple setups and dictConfig() for more advanced configurations.

The Python logging architecture consists of loggers, handlers, filters, and formatters. Let’s provide an overview of each component.

• Loggers are where logging begins. You create instances that emit log records. Loggers invoke functions on preconfigured logger instances and objects that contain event-relevant information, including logger name, message, and function.

• Handlers determine where log messages go after generation by loggers. Users can configure multiple built-in handlers to send logs to various destinations, such as log files, consoles, emails, and external services. These handlers include StreamHandler, FileHandler, NullHandler, and HTTPHandler.

• Filters are used to decide which log records are processed by handlers. They encompass context, logger, and appender filters. Each filter must return a True or False value, and all filters must return True values before any record can be outputted by handlers.

• Formatters specify the layout of log records and enable users to format messages with timestamps, log levels, or any other desired information. To enable log formatting and LogRecord conversion into strings, you configure a formatter and add it to each Handler.

However, it’s important to note that logging is not the only library available for capturing information about the performance of Python applications; developers also utilize print statements. 

So, what’s the difference between the two?

Printing vs. Logging

Although printing and logging are both built-in libraries, they offer distinct advantages in various contexts.

Printing is easy to use when quick code testing is required. Users can employ print() statements during development to display variable values or intermediate results for debugging. However, printing does not provide timestamps for pinpointing the exact moments when errors occur. 

In contrast, logging helps users track events in a more flexible manner, providing comprehensive information, including timestamps, stack traces, and exceptions. We will take a closer look at both in the table below.

Python Logging Levels

Python logging levels indicate the severity of a log message, with each level representing a different degree of importance. When you log a message with a specific level, the system records all messages at that level or higher, while filtering out others. 

For example, if users log a message with the ERROR level, it will include logs from both the Error and Critical levels. Let’s explore each level:

Debug: This level is used to trace and diagnose problems in the program’s execution, providing insights into the application’s internal workings.

Info: It provides general information about the program’s regular operation, ensuring that it functions normally. This level communicates various milestones and important events within the application.

Warning: This level indicates that something unexpected has occurred and alerts that there may be a potential issue in the application (e.g., an unauthorized login).

Error: It is used to report software malfunctions that cause the application to be unable to process a request.

Critical: This level represents the most severe logging level, indicating a serious error that may lead to the application’s termination. 

The logging level is set to ‘WARNING’ by default but can be reconfigured to your desired logging level.

Now, let’s look at an example:

To implement logging using all of these levels, set up a basic logger. You can use ‘DEBUG’ to receive outputs for all levels.

python  
import logging

# Create a logger object 
logger = logging.getLogger(__name__)

# Set the log level 
logger.setLevel(logging.DEBUG)

# Create a file handler to write logs to a file 
file_handler = logging.FileHandler('mylog.log')

# Create a formatter 
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

# Set the formatter for file handler 
file_handler.setFormatter(formatter)

# Add the file handler to the logger 
logger.addHandler(file_handler)

# Log messages at different levels 
logger.debug('Debug message')
logger.info('Info message')
logger.warning('Warning message')
logger.error('Error message')
logger.critical('Critical message')

Below are log message outputs for each logging level.

Output for ‘DEBUG’ Level Logging:

2023-10-07 10:00:00,000 - DEBUG - [PYTHON APP] Initializing request...
2023-10-07 10:00:01,234 - DEBUG - [PYTHON APP] Received user input: "Where is Mount Everest located?"
2023-10-07 10:00:01,236 - DEBUG - [PYTHON APP] Generating response...
2023-10-07 10:00:01,238 - DEBUG - [PYTHON APP] Sending response to the user: "Mount Everest is located between Nepal and Tibet, a region of China..."

Output for ‘INFO’ Level Logging:

2023-10-07 10:00:00,000 - INFO - [PYTHON APP] App is up and running.
2023-10-07 10:00:05,678 - INFO - [PYTHON APP] User requested the weather forecast for New York City.
2023-10-07 10:00:10,543 - INFO - [PYTHON APP] Successfully fetched weather data for New York City.
2023-10-07 10:00:10,544 - INFO - [PYTHON APP] Displaying weather forecast to the user: "Today's weather in New York City: sunny with a high of 25°C."

Output for ‘WARNING’ Level Logging:

2023-10-07 10:00:00,000 - WARNING - [PYTHON APP] Connection to the database is taking longer than usual.
2023-10-07 10:00:10,987 - WARNING - [PYTHON APP] Failed to retrieve user preferences. Using default settings.
2023-10-07 10:00:30,123 - WARNING - [PYTHON APP] Low disk space detected. Free up some space to avoid performance issues.

Output for ‘ERROR’ Level Logging:

2023-10-07 10:00:00,000 - ERROR - [PYTHON APP] An internal server error occurred. Please contact administrator.
2023-10-07 10:00:05,432 - ERROR - [PYTHON APP] Failed to process user input: "Where did COVID-19 start?"
2023-10-07 10:00:10,876 - ERROR - [PYTHON APP] Unable to establish a connection to the external API.

Output for ‘CRITICAL’ Level Logging:

2023-10-07 10:00:00,000 - CRITICAL - [PYTHON APP] System overload! Shutting down to prevent damage.
2023-10-07 10:00:05,123 - CRITICAL - [PYTHON APP] Critical security breach detected. Taking immediate action to protect sensitive data.
2023-10-07 10:00:10,987 - CRITICAL - [PYTHON APP] Server crashed unexpectedly. Restarting the system.

How to Log to a File in Python

To log events to a file in Python, take the following steps.

1. Import the logging module.

python
import logging

2. Set up the logger.

    python
# Create a logger object
logger = logging.getLogger(__name__)

# Set the log level (optional)
logger.setLevel(logging.DEBUG)

3. Create a file handler.

    python
# Create a file handler to write logs to a file
file_handler = logging.FileHandler('app.log')

4. Set the log message format.

   
   python
# Create a formatter for the log messages
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

# Set the formatter for the file handler
file_handler.setFormatter(formatter)

In this log message format, we use `logging.Formatter` with a specific format string, which includes the timestamp, log level, and the actual log message.

5. Add the file handler to the logger to ensure that log messages are formatted according to the desired format.

  
   python
# Add the file handler to the logger
logger.addHandler(file_handler) 

6. Log messages.

  
   python
# Log messages at different levels
logger.debug  ('Debug message')
logger.info  ('Info message')
logger.warning  ('Warning message')
logger.error  ('Error message')
logger.critical  ('Critical message')

How to Log Variables and Exceptions in Python?

Variables are containers for storing data values of different types (e.g., numbers, strings, lists, and objects). In Python, you do not need to declare variables beforehand. Users can perform various operations on variables, such as mathematical calculations, string concatenations, or list manipulations.

Exceptions are events that occur during the execution of a program and disrupt the normal flow of events. For example, when errors—such as division by zero—occur, Python raises an exception to prevent the program from crashing.

To log variables and exceptions in Python, set up the basic steps discussed in previous subsections.

After completing the basic setup, log variables using string formatting.

  
   python
variable = 42
logger.debug ('The value of the variable is %s', variable)

Then, log an exception within a `try`-`except` block.

   python
try:
    result = 10 / 0
except Exception as e:
    logger.exception('An exception occurred: %s', e)

After running the code, you’ll have the output below.

An exception occurred: division by zero
Traceback (most recent call last):
  File "<ipython-input-1-783ac74dcee8>", line 16, in <module>
    result = 10 / 0
ZeroDivisionError: division by zero

The log file (`mylog.log`) will contain the following log messages.

  
2023-10-07 00:00:00,000 - DEBUG - The value of the variable is 42
2023-10-07 00:00:00,000 - ERROR - An exception occurred: division by zero
Traceback (most recent call last):
  File "", line 16, in 
    result = 10 / 0
ZeroDivisionError: division by zero

Use Cases of Python Logging

Python logging is an effective tool for developers involved in building and monitoring applications across various industries, including technology, finance, healthcare, streaming, and gaming service providers. Developers typically rely on Python logging for three key use cases.

Logging with Context

Python logging is valuable in situations where specific contextual information, such as the message’s origin, timestamp, and logging levels, is essential. This information facilitates efficient debugging and troubleshooting.

Using this contextual information, developers can pinpoint the thread, module, system function, and line number from which a log originated.

They can also utilize timestamps to determine when an error message was recorded, when a task failed to complete, or to trace the path of an error before it led to a system crash.

Configuration Over Code

In complex software monitoring environments with multiple modules and threads, Python logging provides developers with the ability to adjust logging behavior on the fly.

For instance, with the dictConfig() function, they can display “DEBUG” logs in the console during app development, switch to “INFO” logs during testing, and reconfigure the system to send “WARNING,” “ERROR,” and “CRITICAL” log outputs via SMTPHandler to a specified email address once the app is running or at any time during the software development life cycle (SDLC).

Whatever your use case, Middleware has got you covered.

Start logging

Fine-Grained Control

When developers need precise control over logging and output destinations, Python logging is the ideal solution. Depending on their specific requirements, developers can use module and class functions to create loggers, handlers, and formatters. 

They can also utilize the fileConfig() function to create and read configuration files or the dictConfig() function to manage dictionary-based configuration information.

LogRecords generated can then be filtered based on logging levels and displayed in the console (through StreamHandler) or any other output destination.

Benefits of Logging

Logging provides several advantages in application development and monitoring, including the following.

Error Detection and Root Cause Analysis

Python logging allows users to capture error messages, stack traces, and other relevant information when an error occurs. These logs are invaluable for root cause analysis, aiding in the understanding of how and where an error originated.

Below is an example of an error log message:

 
2023-10-07 10:00:00,000 - ERROR - [PYTHON APP] An internal server error occurred. Please contact administrator.
2023-10-07 10:00:05,432 - ERROR - [PYTHON APP] Failed to process user input: "Where did COVID-19 start?"
2023-10-07 10:00:10,876 - ERROR - [PYTHON APP] Unable to establish a connection to the external API.

Audit and Compliance

Industry regulations, such as GDPR and PCI-DSS, often mandate that organizations safeguard sensitive data and establish audit trails through the maintenance of logs for specific events, including user access, system changes, and data modifications.

Python logging offers a standardized mechanism for capturing and recording these events, thereby facilitating compliance.

Reduced Incident Response Time

By strategically placing log statements at critical code points, developers can trace the flow of execution and gain insights into the program’s behavior.

Log messages frequently contain variable values, function call information, and other contextual details, such as time and location, which expedite the identification of the root cause of an incident. This reduces the time required to investigate and resolve issues.

Key Parameter Tracking with Performance Monitoring

By logging specific parameters like response times, database query execution times, or resource utilization, developers can gain insights into application performance. This information is invaluable for optimizing software performance.

Alerting

Python logging can be seamlessly integrated with an alerting system to configure custom alerts or notifications for the CRITICAL logging level. Middleware serves as an example of such alerting systems, where developers can establish alerts to notify them when the number of logged errors surpasses a specific threshold.

Middleware also lets developers assign alerts to specific people or teams.

Python Logging Best Practices

Here are six Python logging best practices that every Dev should incorporate:

Use Appropriate Log Levels

Choose the right log level for each message to strike a balance between generating enough information and avoiding excessive noise. This will help you focus on important information.

Log Meaningful Messages

Write descriptive and concise log messages that provide valuable information about the state of your application, making troubleshooting and understanding the code easier.

Limit Sensitive Information

Be mindful of data protection regulations. Avoid logging sensitive data or user-specific information that could compromise data privacy and security.

Implement Log Rotation

Manage log file size and prevent excessive disk usage by periodically archiving or deleting older log files to make space for new logs.

Continuously Improve Logging Approach

Regularly review your logging approach as your project evolves. Analyze the effectiveness of your logs, review their levels and message formats, and adapt them to changing requirements.

Use a Logging Solution

Consider using a log management solution to automate the logging process. Some logging solutions also provide dashboards that simplify log visibility alongside features for customizing log message displays. With an ideal logging solution, you won’t need to run command after command to log your Python applications.

Python Logging with Middleware

For real-time monitoring and error tracking, developers should integrate Middleware into their Python application.

Why?

Middleware’s log monitoring system can help them collect logs from their Python application into a single dashboard. 

How?

To see how this is possible, follow the steps below:

1. Create an account.
2. Install the Middleware agent on your Python application using our easy one-step installation or one-step auto instrumentation commands. For more about Middleware configuration, check our documentation.

Once successfully installed, the Python APM will add severity-based logging support and data export rules so that data can be forwarded to the Middleware Agent.

3. Install the Python package by running the command below in your terminal.

 pip install middleware-apm

4. Add the query below at the very start of your project to the import tracker.

 
from apmpythonpackage import apmpythonclass
tracker=apmpythonclass()
tracker.mw_tracer({APM-PROJECT-NAME}, {APM-SERVICE-NAME})

5. Run this command to enable the tracker.

 
handler = tracker.mw_handler()
logging.getLogger().addHandler(handler)
logging.error("error log sample")

6. To ingest custom logs, run this command based on desired log severity levels.

 
logging.info(“”)
logging.warn(“”)
logging.error(“”)

7. Sign in to Middleware to start.

8. Navigate to “log” to see your log information.

Welcome to a new logging experience!

Closing Thoughts

Developers can seamlessly debug and troubleshoot issues while ensuring compliance with industry standards by implementing effective logging practices and selecting the right logging tool. 

Python logging will remain essential for Python APM. With AI-enabled logging solutions like Middleware in the ring, the future looks promising and reassuring for developers

FAQs