How to Read a Large File Line by Line in Python

By the end of this page, you will understand how to process very large files in Python safely and efficiently using file iteration. You will learn why reading the whole file at once causes memory problems, how Python handles files lazily, and which patterns are commonly used in real programs to work with large logs, CSVs, and text datasets.

When working with large files, the main idea is streaming rather than loading everything at once.

If you use methods like read() or readlines() on a very large file, Python tries to bring all file contents into memory. For a 1 GB file, that can easily exceed available RAM, especially because the data in memory may use more space than the raw file size on disk.

A better approach is to read one line at a time. Python file objects are iterable, which means you can loop over them directly. In this pattern, Python only keeps a small part of the file in memory at any given moment.

This matters because real programs often process:

• server logs
• CSV exports
• JSON lines files
• system reports
• imported datasets

Reading line by line makes your program more memory-efficient and often simpler to reason about. It also lets you start processing immediately instead of waiting for the whole file to load.

Imagine a huge book in a library.

• Loading the whole file into memory is like photocopying the entire book onto your desk before reading it.
• Reading line by line is like keeping the book open and reading one line at a time.

You do not need every page in front of you at once if your job is to process the book sequentially. Python can act like that careful reader, bringing in just enough data to continue.

In Python, the most common way to read a file line by line is:

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        print(line.rstrip())

Why this works

• open(...) opens the file.
• with ensures the file is closed automatically.
• for line in file reads the file lazily, one line at a time.
• rstrip() removes the newline at the end when printing.

Example: counting lines

count = 0

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        count += 1

print("Total lines:", count)

This is memory-efficient because only the current line is processed.

Example: find matching lines

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        if "ERROR" in line:
            print(line.rstrip())

This is a common pattern for scanning logs.

Avoid this for huge files

with open("large_file.txt", "r", encoding="utf-8") as file:
    data = file.read()

and:

with open("large_file.txt", "r", encoding="utf-8") as file:
    lines = file.readlines()

Both of these load too much data into memory for large files.

Consider this code:

with open("numbers.txt", "r", encoding="utf-8") as file:
    for line in file:
        number = int(line.strip())
        print(number * 2)

Suppose numbers.txt contains:

10
20
30

Step by step

1. Python opens numbers.txt.
2. The for line in file loop starts.
3. Python reads the first line: "10\n".
4. line.strip() becomes "10".
5. int("10") becomes 10.
6. print(number * 2) prints 20.
7. Python reads the next line: "20\n".
8. The same process repeats and prints 40.
9. Python reads the final line: "30\n".
10. It prints 60.
11. End of file is reached.
12. The with block ends, and Python closes the file automatically.

At no point does Python need to hold the entire file in memory.

Reading large files line by line is common in many practical situations:

• Log processing: scan server logs for warnings, errors, or suspicious activity.
• CSV imports: process very large exports one row at a time.
• ETL scripts: transform data before loading it into a database.
• Batch jobs: clean or validate records from text files.
• Monitoring tools: inspect output files generated by other systems.
• Data migration: move old records into a new system without memory spikes.

Example: simple log counter

error_count = 0

with open("app.log", "r", encoding="utf-8") as file:
    for line in file:
        if "ERROR" in line:
            error_count += 1

print("Errors found:", error_count)

Example: filtering rows into another file

with open("input.txt", "r", encoding="utf-8") as source, open("filtered.txt", "w", encoding="utf-8") as target:
    for line in source:
        if "approved" in line:
            target.write(line)

This pattern scales well because it streams from one file to another.

In real projects, developers usually combine line-by-line reading with a few important patterns.

Guard clauses

Skip bad or empty lines early:

with open("data.txt", "r", encoding="utf-8") as file:
    for line in file:
        line = line.strip()
        if not line:
            continue
        print(line)

Validation

Validate each line before processing:

with open("ids.txt", "r", encoding="utf-8") as file:
    for line in file:
        value = line.strip()
        if not value.isdigit():
            continue
        print(int(value))

Error handling

Handle malformed lines without crashing the whole job:

with open("numbers.txt", "r", encoding="utf-8") as file:
    for line_number, line in enumerate(file, start=1):
        try:
            number = int(line.strip())
            print(number)
        except ValueError:
            print(f"Skipping invalid number on line {line_number}")

Transformation pipelines

A common pattern is read -> clean -> validate -> transform -> save.

with open("users.txt", "r", encoding="utf-8") as source, open("clean_users.txt", "w", encoding="utf-8") as target:
    for line in source:
        cleaned = line.strip().lower()
        if cleaned:
            target.write(cleaned + "\n")

This approach keeps memory usage low and makes the code easier to maintain.

1. Using read() on a huge file

Broken approach:

with open("large_file.txt", "r", encoding="utf-8") as file:
    data = file.read()

Why it is a problem:

• It loads the whole file into memory.
• Large files can cause memory errors.

Use this instead:

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        process(line)

2. Using readlines() for large input

Broken approach:

with open("large_file.txt", "r", encoding="utf-8") as file:
    lines = file.readlines()

Why it is a problem:

• It creates a full list of lines in memory.

3. Forgetting to strip newlines

with open("names.txt", "r", encoding="utf-8") as file:
    for line in file:
        print("Hello", line)

This may print unexpected blank lines because line usually ends with \n.

Better:

with open("names.txt", "r", encoding="utf-8") as file:
    for line in file:
        print("Hello", line.strip())

4. Not using with

Broken approach:

file = open("large_file.txt", "r", encoding="utf-8")
for line in file:
    print(line)

This can leave files open if an error happens before close() is called.

Better:

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        print(line)

5. Assuming every line is valid

If you convert data directly, one bad line can crash the program.

with open("numbers.txt", "r", encoding="utf-8") as file:
    for line in file:
        print(int(line.strip()))

Safer:

with open("numbers.txt", "r", encoding="utf-8") as file:
    for line in file:
        value = line.strip()
        if value.isdigit():
            print(int(value))

6. Ignoring encoding issues

Some files are not UTF-8. If needed, specify the correct encoding used by the file.

for line in file vs readline()

with open("large_file.txt", "r", encoding="utf-8") as file:
    for line in file:
        print(line)

Compared with:

with open("large_file.txt", "r", encoding="utf-8") as file:
    line = file.readline()
    while line:
        print(line)
        line = file.readline()

Both are memory-efficient, but for line in file is usually simpler and more Pythonic.

Text mode vs binary mode

• Use text mode ("r") for normal text files.
• Use binary mode ("rb") when handling raw bytes or non-text data.

For line-by-line text processing, text mode is usually the correct choice.

with open("file.txt", "r", encoding="utf-8") as file:
    for line in file:
        print(line.strip())

Best practice

• Use with open(...) to auto-close files.
• Use for line in file for memory-efficient reading.
• Use strip() or rstrip() when needed.
• Validate lines before converting them.

Avoid for huge files

file.read()
file.readlines()

Useful patterns

Count lines:

count = 0
with open("file.txt", "r", encoding="utf-8") as file:
    for _ in file:
        count += 1

Track line numbers:

with open("file.txt", "r", encoding="utf-8") as file:
    for line_number, line in enumerate(file, start=1):
        print(line_number, line.strip())

Filter lines:

with open("file.txt", "r", encoding="utf-8") as file:
    for line in file:
        if "ERROR" in line:
            print(line.strip())

Edge cases

• Lines usually include a trailing newline.
• Files may use a different encoding.
• Some lines may be empty or malformed.
• Very long single lines can still use significant memory because one full line must be read at a time.

Why is reading line by line more memory-efficient in Python?

Because Python only keeps a small portion of the file in memory at a time instead of storing the entire file contents.

Is for line in file better than readlines() for large files?

Yes. for line in file streams the file, while readlines() loads all lines into memory.

Can Python handle a 1 GB text file?

Yes, if you process it incrementally. Reading it line by line is a common and safe approach.

Should I use readline() or a for loop?

Usually use a for loop. It is cleaner and is the standard Python style for iterating through a file.

What if the file has invalid text encoding?

Pass the correct encoding to open(). If needed, handle decoding errors with options such as errors="ignore" or errors="replace".

Does line-by-line reading mean only one line is ever in memory?

In practice, Python uses internal buffering, but memory usage stays low compared with reading the entire file.

What if one line in the file is extremely long?

That single line still needs to be read, so one unusually large line can still use a lot of memory.

Can I write to another file while reading a large file?

Yes. This is a very common streaming pattern for filtering, transforming, or exporting data.

• File handling — the broader topic of opening, reading, writing, and closing files.
• Context managers — with statements safely manage resources like files.
• Iteration — file objects are iterable, which is why for line in file works.
• Generators — another memory-efficient way to process data lazily.
• CSV processing — large CSV files are often read row by row.
• Exception handling — useful for skipping malformed lines without crashing.
• Text encoding — important when reading files with different character formats.
• Buffering — explains how file reading stays efficient internally.