{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title",
   "metadata": {},
   "source": [
    "# Logging to files\n",
    "\n",
    "In this tutorial, you will learn how to direct log output to files using handlers.\n",
    "File-based logging is essential for production applications, where you need a persistent\n",
    "record of what happened.\n",
    "\n",
    "**Time commitment:** 15–20 minutes\n",
    "\n",
    "**Prerequisites:**\n",
    "\n",
    "- [Your first log message](https://agilearn.co.uk/guides/logging-and-debugging/learn/01-your-first-log-message) tutorial completed\n",
    "- [Log levels and formatting](https://agilearn.co.uk/guides/logging-and-debugging/learn/02-log-levels-and-formatting) tutorial completed\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "By the end of this tutorial, you will be able to:\n",
    "\n",
    "- Use `logging.FileHandler` to write log messages to a file\n",
    "- Use `logging.handlers.RotatingFileHandler` for size-limited log files\n",
    "- Combine multiple handlers to send log output to different destinations\n",
    "- Set different log levels for different handlers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "why-file-logging",
   "metadata": {},
   "source": [
    "## Why log to files?\n",
    "\n",
    "Console output disappears when the terminal is closed. For production applications,\n",
    "you need log messages stored persistently so that you can:\n",
    "\n",
    "- Review what happened after the fact (post-mortem analysis)\n",
    "- Track application behaviour over time\n",
    "- Share logs with team members for debugging\n",
    "- Analyse patterns and trends in application events\n",
    "\n",
    "The `logging` module provides several **handler** classes for writing to files.\n",
    "A handler is an object that sends log records to a specific destination."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "filehandler-section",
   "metadata": {},
   "source": [
    "## Using `logging.FileHandler`\n",
    "\n",
    "The simplest way to log to a file is with `logging.FileHandler`. It writes all\n",
    "log messages to a single file.\n",
    "\n",
    "Let us create a logger with a file handler. We will use a temporary file so the\n",
    "example cleans up after itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "filehandler-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "# Create a temporary file for our log output\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"app.log\")\n",
    "\n",
    "# Create a logger\n",
    "logger = logging.getLogger(\"file_demo\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# Create a file handler\n",
    "file_handler = logging.FileHandler(log_file)\n",
    "file_handler.setLevel(logging.DEBUG)\n",
    "\n",
    "# Create a formatter and attach it to the handler\n",
    "formatter = logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n",
    "file_handler.setFormatter(formatter)\n",
    "\n",
    "# Add the handler to the logger\n",
    "logger.addHandler(file_handler)\n",
    "\n",
    "# Log some messages\n",
    "logger.info(\"Application started\")\n",
    "logger.debug(\"Loading configuration\")\n",
    "logger.warning(\"Configuration file not found, using defaults\")\n",
    "\n",
    "# Flush and read the file to see what was written\n",
    "file_handler.flush()\n",
    "\n",
    "with open(log_file, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(\"Contents of log file:\")\n",
    "    print(f.read())\n",
    "\n",
    "# Clean up the handler\n",
    "logger.removeHandler(file_handler)\n",
    "file_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "filehandler-explanation",
   "metadata": {},
   "source": [
    "The key steps for setting up file logging are:\n",
    "\n",
    "1. Create a logger with `logging.getLogger()`\n",
    "2. Create a `logging.FileHandler` with the path to the log file\n",
    "3. Create a `logging.Formatter` with your desired format string\n",
    "4. Attach the formatter to the handler with `handler.setFormatter()`\n",
    "5. Add the handler to the logger with `logger.addHandler()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dual-handler-section",
   "metadata": {},
   "source": [
    "## Combining console and file handlers\n",
    "\n",
    "A logger can have multiple handlers attached at the same time. A common pattern is\n",
    "to send log output to both the console and a file. This way, you can see messages\n",
    "in real time on the console while also keeping a persistent record in a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dual-handler-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"dual.log\")\n",
    "\n",
    "# Create a logger\n",
    "logger = logging.getLogger(\"dual_demo\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# Console handler\n",
    "console_handler = logging.StreamHandler()\n",
    "console_handler.setLevel(logging.DEBUG)\n",
    "console_formatter = logging.Formatter(\"%(levelname)s: %(message)s\")\n",
    "console_handler.setFormatter(console_formatter)\n",
    "\n",
    "# File handler (with more detail)\n",
    "file_handler = logging.FileHandler(log_file)\n",
    "file_handler.setLevel(logging.DEBUG)\n",
    "file_formatter = logging.Formatter(\n",
    "    \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n",
    ")\n",
    "file_handler.setFormatter(file_formatter)\n",
    "\n",
    "# Add both handlers\n",
    "logger.addHandler(console_handler)\n",
    "logger.addHandler(file_handler)\n",
    "\n",
    "# Log some messages\n",
    "logger.info(\"Starting the application\")\n",
    "logger.warning(\"Low disk space\")\n",
    "\n",
    "# Show the file contents\n",
    "file_handler.flush()\n",
    "print(\"\\n--- File contents ---\")\n",
    "with open(log_file, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(f.read())\n",
    "\n",
    "# Clean up\n",
    "logger.removeHandler(console_handler)\n",
    "logger.removeHandler(file_handler)\n",
    "console_handler.close()\n",
    "file_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dual-explanation",
   "metadata": {},
   "source": [
    "Notice that each handler can have its own formatter. In this example, the console\n",
    "shows a brief format, while the file stores a more detailed format including timestamps\n",
    "and the logger name."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "different-levels-section",
   "metadata": {},
   "source": [
    "## Different levels for different handlers\n",
    "\n",
    "Each handler can have its own log level, independent of the logger level. This is\n",
    "useful when you want detailed logging in files but only important messages on\n",
    "the console.\n",
    "\n",
    "**Important:** The logger level acts as a first gate. If a message does not pass the\n",
    "logger level, it is not sent to any handler. Handler levels provide a second gate\n",
    "for each individual handler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "different-levels-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"levels.log\")\n",
    "\n",
    "logger = logging.getLogger(\"level_demo\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# Console handler – only show WARNING and above\n",
    "console_handler = logging.StreamHandler()\n",
    "console_handler.setLevel(logging.WARNING)\n",
    "console_handler.setFormatter(logging.Formatter(\"%(levelname)s: %(message)s\"))\n",
    "\n",
    "# File handler – capture everything from DEBUG upwards\n",
    "file_handler = logging.FileHandler(log_file)\n",
    "file_handler.setLevel(logging.DEBUG)\n",
    "file_handler.setFormatter(\n",
    "    logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n",
    ")\n",
    "\n",
    "logger.addHandler(console_handler)\n",
    "logger.addHandler(file_handler)\n",
    "\n",
    "# Log at all levels\n",
    "logger.debug(\"Detailed diagnostic information\")\n",
    "logger.info(\"General information\")\n",
    "logger.warning(\"Something unexpected happened\")\n",
    "logger.error(\"An error occurred\")\n",
    "\n",
    "# Show what went to the file\n",
    "file_handler.flush()\n",
    "print(\"\\n--- File contents (all levels) ---\")\n",
    "with open(log_file, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(f.read())\n",
    "\n",
    "# Clean up\n",
    "logger.removeHandler(console_handler)\n",
    "logger.removeHandler(file_handler)\n",
    "console_handler.close()\n",
    "file_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "different-levels-explanation",
   "metadata": {},
   "source": [
    "In this example:\n",
    "\n",
    "- The **console** only shows `WARNING` and `ERROR` (two messages)\n",
    "- The **file** captures all four messages, including `DEBUG` and `INFO`\n",
    "\n",
    "This is a common production pattern: keep detailed logs in files for later analysis,\n",
    "but only show important messages on the console to avoid clutter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rotating-section",
   "metadata": {},
   "source": [
    "## `RotatingFileHandler`\n",
    "\n",
    "A plain `FileHandler` writes to a single file that grows without limit. For\n",
    "long-running applications, this can become a problem as the file grows very large.\n",
    "\n",
    "`logging.handlers.RotatingFileHandler` solves this by automatically rotating\n",
    "log files when they reach a specified size. It keeps a configurable number of\n",
    "backup files.\n",
    "\n",
    "Parameters:\n",
    "\n",
    "- `maxBytes` -- Maximum size of each log file in bytes (0 means no limit)\n",
    "- `backupCount` -- Number of backup files to keep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "rotating-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import logging.handlers\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"rotating.log\")\n",
    "\n",
    "logger = logging.getLogger(\"rotating_demo\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# Create a rotating file handler with a small max size for demonstration\n",
    "rotating_handler = logging.handlers.RotatingFileHandler(\n",
    "    log_file,\n",
    "    maxBytes=200,\n",
    "    backupCount=3\n",
    ")\n",
    "rotating_handler.setFormatter(\n",
    "    logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n",
    ")\n",
    "logger.addHandler(rotating_handler)\n",
    "\n",
    "# Write enough messages to trigger rotation\n",
    "for i in range(20):\n",
    "    logger.info(\"Log message number %s\", i)\n",
    "\n",
    "# Show what files were created\n",
    "print(\"Files in log directory:\")\n",
    "for filename in sorted(os.listdir(log_dir)):\n",
    "    filepath = os.path.join(log_dir, filename)\n",
    "    size = os.path.getsize(filepath)\n",
    "    print(\"  %s (%s bytes)\", filename, size)\n",
    "\n",
    "# Clean up\n",
    "logger.removeHandler(rotating_handler)\n",
    "rotating_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rotating-explanation",
   "metadata": {},
   "source": [
    "When the log file reaches the `maxBytes` limit, it is renamed with a `.1` suffix,\n",
    "and a new log file is started. The previous `.1` file becomes `.2`, and so on.\n",
    "Files beyond `backupCount` are deleted automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "timed-section",
   "metadata": {},
   "source": [
    "## `TimedRotatingFileHandler`\n",
    "\n",
    "If you prefer to rotate log files based on time rather than size, use\n",
    "`logging.handlers.TimedRotatingFileHandler`. It can rotate logs at regular intervals\n",
    "such as every hour, every day, or every week.\n",
    "\n",
    "Common `when` values:\n",
    "\n",
    "| Value | Interval |\n",
    "|-------|----------|\n",
    "| `\"S\"` | Every second |\n",
    "| `\"M\"` | Every minute |\n",
    "| `\"H\"` | Every hour |\n",
    "| `\"D\"` | Every day |\n",
    "| `\"midnight\"` | At midnight |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "timed-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import logging.handlers\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"timed.log\")\n",
    "\n",
    "logger = logging.getLogger(\"timed_demo\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "# Rotate every second (for demonstration purposes)\n",
    "timed_handler = logging.handlers.TimedRotatingFileHandler(\n",
    "    log_file,\n",
    "    when=\"S\",\n",
    "    interval=1,\n",
    "    backupCount=3\n",
    ")\n",
    "timed_handler.setFormatter(\n",
    "    logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n",
    ")\n",
    "logger.addHandler(timed_handler)\n",
    "\n",
    "logger.info(\"This message goes to the timed rotating log\")\n",
    "\n",
    "print(\"Log file created at:\", log_file)\n",
    "print(\"In production, this handler would rotate files based on time.\")\n",
    "\n",
    "# Clean up\n",
    "logger.removeHandler(timed_handler)\n",
    "timed_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "practical-section",
   "metadata": {},
   "source": [
    "## Practical example\n",
    "\n",
    "Let us combine what you have learned into a complete, practical function that sets\n",
    "up logging for a small application. This function creates a logger with both console\n",
    "and file output, with different levels for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "practical-demo",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "\n",
    "def setup_application_logging(\n",
    "    name: str, log_file: str, console_level: int = logging.WARNING,\n",
    "    file_level: int = logging.DEBUG\n",
    ") -> logging.Logger:\n",
    "    \"\"\"Set up a logger with console and file handlers.\n",
    "\n",
    "    Args:\n",
    "        name: The name for the logger.\n",
    "        log_file: The path to the log file.\n",
    "        console_level: Minimum level for console output.\n",
    "        file_level: Minimum level for file output.\n",
    "\n",
    "    Returns:\n",
    "        A configured Logger instance.\n",
    "    \"\"\"\n",
    "    logger = logging.getLogger(name)\n",
    "    logger.setLevel(min(console_level, file_level))\n",
    "\n",
    "    if not logger.handlers:\n",
    "        console_handler = logging.StreamHandler()\n",
    "        console_handler.setLevel(console_level)\n",
    "        console_handler.setFormatter(\n",
    "            logging.Formatter(\"%(levelname)s: %(message)s\")\n",
    "        )\n",
    "\n",
    "        file_handler = logging.FileHandler(log_file)\n",
    "        file_handler.setLevel(file_level)\n",
    "        file_handler.setFormatter(\n",
    "            logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n",
    "        )\n",
    "\n",
    "        logger.addHandler(console_handler)\n",
    "        logger.addHandler(file_handler)\n",
    "\n",
    "    return logger\n",
    "\n",
    "\n",
    "# Use the function\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_path = os.path.join(log_dir, \"application.log\")\n",
    "\n",
    "app_logger = setup_application_logging(\"my_app\", log_path)\n",
    "\n",
    "app_logger.debug(\"Debug information (file only)\")\n",
    "app_logger.info(\"Info message (file only)\")\n",
    "app_logger.warning(\"Warning message (console and file)\")\n",
    "app_logger.error(\"Error message (console and file)\")\n",
    "\n",
    "# Show file contents\n",
    "for handler in app_logger.handlers:\n",
    "    handler.flush()\n",
    "\n",
    "print(\"\\n--- File contents ---\")\n",
    "with open(log_path, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(f.read())\n",
    "\n",
    "# Clean up\n",
    "for handler in app_logger.handlers[:]:\n",
    "    handler.close()\n",
    "    app_logger.removeHandler(handler)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exercises-section",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "Try these exercises to reinforce what you have learned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exercise-1",
   "metadata": {},
   "source": [
    "### Exercise 1: Basic file logging\n",
    "\n",
    "Create a logger called `\"exercise_logger\"` that writes `INFO` level and above to a\n",
    "temporary file. Log three messages, then read and print the file contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exercise-1-solution",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 1: Solution\n",
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"exercise1.log\")\n",
    "\n",
    "logger = logging.getLogger(\"exercise_logger\")\n",
    "logger.setLevel(logging.INFO)\n",
    "\n",
    "handler = logging.FileHandler(log_file)\n",
    "handler.setFormatter(logging.Formatter(\"%(levelname)s - %(message)s\"))\n",
    "logger.addHandler(handler)\n",
    "\n",
    "logger.info(\"Application started\")\n",
    "logger.warning(\"Disk space is low\")\n",
    "logger.error(\"Failed to connect to server\")\n",
    "\n",
    "handler.flush()\n",
    "with open(log_file, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(f.read())\n",
    "\n",
    "logger.removeHandler(handler)\n",
    "handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exercise-2",
   "metadata": {},
   "source": [
    "### Exercise 2: Dual output\n",
    "\n",
    "Create a logger with two handlers: a `StreamHandler` at `ERROR` level and a\n",
    "`FileHandler` at `DEBUG` level. Log messages at all five levels. Verify that\n",
    "only `ERROR` and `CRITICAL` appear on the console, but all messages are in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exercise-2-solution",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 2: Solution\n",
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"exercise2.log\")\n",
    "\n",
    "logger = logging.getLogger(\"exercise2_logger\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "console_handler = logging.StreamHandler()\n",
    "console_handler.setLevel(logging.ERROR)\n",
    "console_handler.setFormatter(logging.Formatter(\"CONSOLE: %(levelname)s - %(message)s\"))\n",
    "\n",
    "file_handler = logging.FileHandler(log_file)\n",
    "file_handler.setLevel(logging.DEBUG)\n",
    "file_handler.setFormatter(logging.Formatter(\"FILE: %(levelname)s - %(message)s\"))\n",
    "\n",
    "logger.addHandler(console_handler)\n",
    "logger.addHandler(file_handler)\n",
    "\n",
    "logger.debug(\"Debug message\")\n",
    "logger.info(\"Info message\")\n",
    "logger.warning(\"Warning message\")\n",
    "logger.error(\"Error message\")\n",
    "logger.critical(\"Critical message\")\n",
    "\n",
    "file_handler.flush()\n",
    "print(\"\\n--- File contents (all five levels) ---\")\n",
    "with open(log_file, \"r\", encoding=\"utf-8\") as f:\n",
    "    print(f.read())\n",
    "\n",
    "logger.removeHandler(console_handler)\n",
    "logger.removeHandler(file_handler)\n",
    "console_handler.close()\n",
    "file_handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exercise-3",
   "metadata": {},
   "source": [
    "### Exercise 3: Rotating file handler\n",
    "\n",
    "Create a logger with a `RotatingFileHandler` that limits each file to 300 bytes\n",
    "and keeps two backup files. Write 15 log messages and then list the files in the\n",
    "log directory to see the rotation in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "exercise-3-solution",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 3: Solution\n",
    "import logging\n",
    "import logging.handlers\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "log_dir = tempfile.mkdtemp()\n",
    "log_file = os.path.join(log_dir, \"rotating_exercise.log\")\n",
    "\n",
    "logger = logging.getLogger(\"rotating_exercise\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "\n",
    "handler = logging.handlers.RotatingFileHandler(\n",
    "    log_file,\n",
    "    maxBytes=300,\n",
    "    backupCount=2\n",
    ")\n",
    "handler.setFormatter(\n",
    "    logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n",
    ")\n",
    "logger.addHandler(handler)\n",
    "\n",
    "for i in range(15):\n",
    "    logger.info(\"Message number %s with some content\", i)\n",
    "\n",
    "print(\"Log files created:\")\n",
    "for filename in sorted(os.listdir(log_dir)):\n",
    "    filepath = os.path.join(log_dir, filename)\n",
    "    print(\"  %s (%s bytes)\" % (filename, os.path.getsize(filepath)))\n",
    "\n",
    "logger.removeHandler(handler)\n",
    "handler.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "summary",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, you learned the following:\n",
    "\n",
    "- `logging.FileHandler` writes log messages to a file\n",
    "- A logger can have multiple handlers, each with its own formatter and level\n",
    "- A common pattern is to use a `StreamHandler` for the console (at a higher level)\n",
    "  and a `FileHandler` for files (at a lower level)\n",
    "- `logging.handlers.RotatingFileHandler` rotates log files when they reach a specified size\n",
    "- `logging.handlers.TimedRotatingFileHandler` rotates log files based on time intervals\n",
    "- Always clean up handlers when you are done (close and remove them from the logger)\n",
    "\n",
    "**Next tutorial:** Continue to [Debugging with pdb](https://agilearn.co.uk/guides/logging-and-debugging/learn/04-debugging-with-pdb) to\n",
    "learn how to use the built-in Python debugger for stepping through code and inspecting\n",
    "variables."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}