Python logging is a fundamental element in the development and maintenance of Python applications.
When building software, it’s important to understand what’s happening inside the application, especially when things go wrong. Python’s built-in logging library helps developers with the ability to capture and preserve valuable information about their application’s performance and behavior.
This invaluable data serves as a reference point when diagnosing and addressing issues, streamlining the troubleshooting process.
In this article, we explore the multifaceted world of Python logging, exploring various use cases, best practices, and proven strategies for overcoming the challenges of performance enhancement.
What is Python Logging?
Python’s logging is a built-in logging module specific to applications built with Python. Developers use it to capture valuable information about code execution and software or application behavior during runtime.
Python logging also provides multiple ways to configure the logging system, such as using basicConfig() for simple setups and dictConfig() for more advanced configurations.
The Python logging architecture consists of loggers, handlers, filters, and formatters. Let’s provide an overview of each component.
- Loggers are where logging begins. You create instances that emit log records. Loggers invoke functions on preconfigured logger instances and objects that contain event-relevant information, including logger name, message, and function.
- Handlers determine where log messages go after generate by loggers. Users can configure multiple built-in handlers to send logs to various destinations, such as log files, consoles, emails, and external services. These handlers include:
- StreamHandler: sends log messages to the console
- FileHandler: sends log messages to a file.
- NullHandler: does nothing with log messages (used for testing).
- RotatingFileHandler: sends log messages to a file, rotating the file when it reaches a certain size.
- TimedRotatingFileHandler: sends log messages to a file, rotating the file at a specified interval.
- Filters are used to decide which log records are processed by handlers. They encompass context, logger, and appender filters. Each filter must return a “True” or “False” value, and all filters must return “True” values before any record can be outputted by handlers.
- Formatters specify the layout of log records and enable users to format messages with timestamps, log levels, or any other desired information. To enable log formatting and LogRecord conversion into strings, you configure a formatter and add it to each Handler.
However, it’s important to note that logging is not the only library available for capturing information about the performance of Python applications; developers also utilize print statements.
So, what’s the difference between the two?
Printing vs. Logging
Although printing and logging are both built-in libraries, they offer distinct advantages in various contexts.
Printing is easy to use when quick code testing is required. Users can employ print() statements during development to display variable values or intermediate results for debugging.
However, printing has several limitations. It does not provide timestamps for pinpointing the exact moments when errors occur. This makes it difficult to track the sequence of events leading up to an error. It is also not suitable for long-term logging or analysis, as it can clutter the console with unnecessary output and make it challenging to identify errors. Generally, print statements can be scattered throughout the code, making it hard to maintain and update.
In contrast, logging helps users track events in a more flexible manner, providing comprehensive information, including timestamps, stack traces, and exceptions. As it is more structured, developers can categorize log messages by severity (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL) and customize the logging output to suit their needs. And with the timestamps, stack traces, and exceptions, they can easily identify and diagnose issues. By design, logging is for long-term logging and analysis. As such, developers can track errors and performance issues over time and improvise.
Let’s will take a closer look at both in the table below.
| Functionalities | Printing | Logging |
| Purpose | Printing provides quick information display during program execution. | Logging captures log messages for debugging and analysis. |
| Output | Printing outputs to the console only. | Logging offers flexible output destinations via handlers. |
| Flexibility | Printing provides limited control and formatting options. | Logging is highly flexible, offering advanced customizable functionalities, including handlers, log levels, formatters, and filters. |
| Duration | Printing is not designed for long-term log retention or analysis. | Logging is designed for long-term log retention and analysis. |
| Integration | Printing has limited integration with external libraries and frameworks. | Logging seamlessly integrates with libraries and frameworks for centralized logging. |
| Log Importance | Printing simply displays records sent to it, placing less emphasis on the importance or severity of messages. | Logging allows users to assign log levels to indicate the importance or severity of messages. It offers several log levels, including Debug, Info, Warning, Error, and Critical, with the highest being Critical. |
Python Logging Levels
Python logging levels indicate the severity of a log message, with each level representing a different degree of importance. When you log a message with a specific level, the system records all messages at that level or higher, while filtering out others.
For example, if users log a message with the ERROR level, it will include logs from both the Error and Critical levels. Let’s explore each level:
Debug: This level is used to trace and diagnose problems in the program’s execution, providing insights into the application’s internal workings.
Info: It provides general information about the program’s regular operation, ensuring that it functions normally. This level communicates various milestones and important events within the application.
Warning: This level indicates that something unexpected has occurred and alerts that there may be a potential issue in the application (e.g., an unauthorized login).
Error: This is used to report software malfunctions that cause the application to be unable to process a request.
Critical: This level represents the most severe logging level, indicating a serious error that may lead to the application’s termination.
The logging level is set to ‘WARNING’ by default but can be reconfigured to your desired logging level.
Now, let’s look at an example:
To implement logging using all of these levels, set up a basic logger. You can use ‘DEBUG’ to receive outputs for all levels.use ‘DEBUG’ to receive outputs for all levels.
python
import logging
# Create a logger object
logger = logging.getLogger(__name__)
# Set the log level
logger.setLevel(logging.DEBUG)
# Create a file handler to write logs to a file
file_handler = logging.FileHandler('mylog.log')
# Create a formatter
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# Set the formatter for file handler
file_handler.setFormatter(formatter)
# Add the file handler to the logger
logger.addHandler(file_handler)
# Log messages at different levels
logger.debug('Debug message')
logger.info('Info message')
logger.warning('Warning message')
logger.error('Error message')
logger.critical('Critical message')
Below are the log message outputs for each logging level.
Output for ‘DEBUG’ Level Logging:
2023-10-07 10:00:00,000 - DEBUG - [PYTHON APP] Initializing request...
2023-10-07 10:00:01,234 - DEBUG - [PYTHON APP] Received user input: "Where is Mount Everest located?"
2023-10-07 10:00:01,236 - DEBUG - [PYTHON APP] Generating response...
2023-10-07 10:00:01,238 - DEBUG - [PYTHON APP] Sending response to the user: "Mount Everest is located between Nepal and Tibet, a region of China..."
Output for ‘INFO’ Level Logging:
2023-10-07 10:00:00,000 - INFO - [PYTHON APP] App is up and running.
2023-10-07 10:00:05,678 - INFO - [PYTHON APP] User requested the weather forecast for New York City.
2023-10-07 10:00:10,543 - INFO - [PYTHON APP] Successfully fetched weather data for New York City.
2023-10-07 10:00:10,544 - INFO - [PYTHON APP] Displaying weather forecast to the user: "Today's weather in New York City: sunny with a high of 25°C."
Output for ‘WARNING’ Level Logging:
2023-10-07 10:00:00,000 - WARNING - [PYTHON APP] Connection to the database is taking longer than usual.
2023-10-07 10:00:10,987 - WARNING - [PYTHON APP] Failed to retrieve user preferences. Using default settings.
2023-10-07 10:00:30,123 - WARNING - [PYTHON APP] Low disk space detected. Free up some space to avoid performance issues.
Output for ‘ERROR’ Level Logging:
2023-10-07 10:00:00,000 - ERROR - [PYTHON APP] An internal server error occurred. Please contact administrator.
2023-10-07 10:00:05,432 - ERROR - [PYTHON APP] Failed to process user input: "Where did COVID-19 start?"
2023-10-07 10:00:10,876 - ERROR - [PYTHON APP] Unable to establish a connection to the external API.
Output for ‘CRITICAL’ Level Logging:
2023-10-07 10:00:00,000 - CRITICAL - [PYTHON APP] System overload! Shutting down to prevent damage.
2023-10-07 10:00:05,123 - CRITICAL - [PYTHON APP] Critical security breach detected. Taking immediate action to protect sensitive data.
2023-10-07 10:00:10,987 - CRITICAL - [PYTHON APP] Server crashed unexpectedly. Restarting the system.
How to Log to a File in Python
To log events to a file in Python, take the following steps.
1. Import the logging module.
python import logging2. Set up the logger.
python
# Create a logger object
logger = logging.getLogger(__name__)
# Set the log level (optional)
logger.setLevel(logging.DEBUG)
3. Create a file handler.
python
# Create a file handler to write logs to a file
file_handler = logging.FileHandler('app.log')
4. Set the log message format.
python
# Create a formatter for the log messages
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# Set the formatter for the file handler
file_handler.setFormatter(formatter)
In this log message format, we use `logging.Formatter` with a specific format string, which includes the timestamp, log level, and the actual log message.
5. Add a file handler to the logger to ensure that log messages are formatted according to the desired format.
6. Log messages.
python
# Log messages at different levels
logger.debug ('Debug message')
logger.info ('Info message')
logger.warning ('Warning message')
logger.error ('Error message')
logger.critical ('Critical message')
How to Log Variables and Exceptions in Python?
Variables are containers for storing data values of different types (e.g., numbers, strings, lists, and objects). In Python, you do not need to declare variables beforehand. Users can perform various operations on variables, such as mathematical calculations, string concatenations, or list manipulations.
Exceptions are events that occur during the execution of a program and disrupt the normal flow of events. For example, when errors—such as division by zero—occur, Python raises an exception to prevent the program from crashing.
To log variables and exceptions in Python, set up the basic steps discussed in previous subsections.
After completing the basic setup, log variables using string formatting.
Then, log an exception within a `try`-`except` block.
python try: result = 10 / 0 except Exception as e: logger.exception(‘An exception occurred: %s’, e)After running the code, you’ll have the output below.
An exception occurred: division by zero Traceback (most recent call last): File “The log file (`mylog.log`) will contain the following log messages.
2023-10-07 00:00:00,000 - DEBUG - The value of the variable is 42
2023-10-07 00:00:00,000 - ERROR - An exception occurred: division by zero
Traceback (most recent call last):
File "", line 16, in
result = 10 / 0
ZeroDivisionError: division by zero
Use Cases of Python Logging
Python logging is an effective tool for developers involved in building and monitoring applications across various industries, including technology, finance, healthcare, streaming, and gaming service providers. Developers typically rely on Python logging for three key use cases.
Logging with Context
Python logging is valuable in situations where specific contextual information, such as the message’s origin, timestamp, and logging levels, is essential. This information facilitates efficient debugging and troubleshooting.
Using this contextual information, developers can pinpoint the thread, module, system function, and line number from which a log originated.
They can also utilize timestamps to determine:
- Trace the path of an error before it led to a system crash.
- When an error message was recorded.
- When a task fails to be completed.
Configuration Over Code
In complex software monitoring environments with multiple modules and threads, Python logging provides developers with the ability to adjust logging behavior on the fly.
For instance, with the dictConfig() function, they can display “DEBUG” logs in the console during app development, switch to “INFO” logs during testing, and reconfigure the system to send “WARNING,” “ERROR,” and “CRITICAL” log outputs via SMTPHandler to a specified email address once the app is running or at any time during the software development life cycle (SDLC).
Fine-Grained Control
When developers need precise control over logging and output destinations, Python logging is the ideal solution. Depending on their specific requirements, developers can use module and class functions to create loggers, handlers, and formatters.
They can also utilize the fileConfig() function to create and read configuration files or the dictConfig() function to manage dictionary-based configuration information.
LogRecords generated can then be filtered based on logging levels and displayed in the console (through StreamHandler) or any other output destination.
Benefits of Logging
Logging provides several advantages in application development and monitoring, including the following.
Error Detection and Root Cause Analysis
Python logging allows users to capture error messages, stack traces, and other relevant information when an error occurs. These logs are invaluable for root cause analysis, aiding in the understanding of how and where an error originated.
Below is an example of an error log message:
2023-10-07 10:00:00,000 - ERROR - [PYTHON APP] An internal server error occurred. Please contact administrator.
2023-10-07 10:00:05,432 - ERROR - [PYTHON APP] Failed to process user input: "Where did COVID-19 start?"
2023-10-07 10:00:10,876 - ERROR - [PYTHON APP] Unable to establish a connection to the external API.
Audit and Compliance
Industry regulations, such as GDPR and PCI-DSS, often mandate that organizations safeguard sensitive data and establish audit trails through the maintenance of logs for specific events, including user access, system changes, and data modifications.
Python logging offers a standardized mechanism for capturing and recording these events, thereby facilitating compliance.
Reduced Incident Response Time
By strategically placing log statements at critical code points, developers can trace the flow of execution and gain insights into the program’s behavior.
Log messages frequently contain variable values, function call information, and other contextual details, such as time and location, which expedite the identification of the root cause of an incident. This reduces the time required to investigate and resolve issues.
Key Parameter Tracking with Performance Monitoring
By logging specific parameters like response times, database query execution times, or resource utilization, developers can gain insights into application performance. This information is invaluable for optimizing software performance.
Alerting
Python logging can be seamlessly integrated with an alerting system to configure custom alerts or notifications for the CRITICAL logging level. Middleware serves as an example of such alerting systems, where developers can establish alerts to notify them when the number of logged errors surpasses a specific threshold.
Middleware also lets developers assign alerts to specific people or teams.
Python Logging Best Practices
Here are 10 Python logging best practices that every Dev should incorporate:
Use Appropriate Log Levels
Choose the right log level for each message to strike a balance between generating enough information and avoiding excessive noise. This will help you focus on important information.
Log Meaningful Messages
Write descriptive and concise log messages that provide valuable information about the state of your application, making troubleshooting and understanding the code easier.
Limit Sensitive Information
Be mindful of data protection regulations. Avoid logging sensitive data or user-specific information that could compromise data privacy and security.
Implement Log Rotation
Manage log file size and prevent excessive disk usage by periodically archiving or deleting older log files to make space for new logs.
Continuously Improve Logging Approach
Regularly review your logging approach as your project evolves. Analyze the effectiveness of your logs, review their levels and message formats, and adapt them to changing requirements.
Use log correlation IDs
Log correlation IDs are unique identifiers that can be used to track related log messages across different components or services. This is particularly useful in distributed systems where a single request may generate log messages in multiple services.
To implement log correlation IDs, you can first generate a unique ID for each request or transaction. Then include the correlation ID in each log message and use the correlation ID to track related log messages across different services.
Log Exceptions and Errors
Logging exceptions and errors is critical for diagnosing and fixing issues. When logging exceptions and errors, include relevant information such as the error message, stack trace, and any relevant context (e.g., user ID, request parameters).
To log exceptions and errors, use a try-except block to catch exceptions, log the exception using the logging module, and include relevant context in the log message.
Use a Standard Log Format
Using a standard log format across your application makes it easier to parse and analyze logs. A standard log format should include:
- A consistent date and time format
- A log level (e.g., DEBUG, INFO, WARNING, ERROR, CRITICAL)
- A message structure (e.g., key-value pairs)
To implement a standard log format, you need to first define a log format string using the logging module. Then, use this log format string consistently across your application. You can also consider using a logging library or framework that provides a standard log format.
Monitor Log Files
Regularly monitoring log files is critical for identifying issues and preventing problems. To monitor log files:
- Use a log monitoring tool or library
- Set up alerts for critical log messages (e.g., errors, exceptions)
- Regularly review log files to identify trends and patterns
Use a Logging Solution
Consider using a log management solution to automate the logging process. Some logging solutions also provide dashboards that simplify log visibility alongside features for customizing log message displays. With an ideal logging solution, you won’t need to run command after command to log your Python applications.
Python Logging with Middleware
For real-time monitoring and error tracking, developers should integrate Middleware into their Python application.
Why?
Middleware’s log monitoring system can help them collect logs from their Python application into a single dashboard.
How?
To see how this is possible, follow the steps below:
- Create an account.
- Install the Middleware agent on your Python application using our easy one-step installation or one-step auto instrumentation commands. For more about Middleware configuration, check our documentation.
Once successfully installed, the Python APM will add severity-based logging support and data export rules so that data can be forwarded to the Middleware Agent.
- Install the Python package by running the command below in your terminal.
pip install middleware-apm- Add the query below at the very start of your project to the import tracker.
from apmpythonpackage import apmpythonclass
tracker=apmpythonclass()
tracker.mw_tracer({APM-PROJECT-NAME}, {APM-SERVICE-NAME})
- Run this command to enable the tracker.
handler = tracker.mw_handler()
logging.getLogger().addHandler(handler)
logging.error("error log sample")
- To ingest custom logs, run this command based on the desired log severity levels.
- Sign in to Middleware to get started.
- Navigate to “log” to see your log information.
Welcome to the new logging experience!
Closing Thoughts
Developers can seamlessly debug and troubleshoot issues while ensuring compliance with industry standards by implementing effective logging practices and selecting the right logging tool. Python logging will remain essential for Python APM. With AI-enabled logging solutions like Middleware in the ring, the future looks promising and reassuring for developers.
FAQs
What is Python logging used for?
Python logging records events, messages, and errors that occur during program execution. It provides a standardized way to gather information about an application’s behavior for ease of debugging and monitoring.
What is the Python logging library?
The Python logging library is a built-in module in Python that offers a framework for generating log messages in applications.
It provides various features such as different logging levels, log formatters, log handlers, and configuration options.
Does the Python logging module use log4j?
While the Python logging module does not originally use log4j (as log4j is a logging library specific to Java), you can configure the Python logging module to use log4j.
However, this configuration requires a significant amount of code.
What is the fastest way to log in Python?
The speed of logging in Python depends on various factors, including the log level, log handlers used, and the configuration. However, for faster logging, it is best to use a reputable log management solution.
