Python Merge Multiple CSVs Using File I/O

It is common for applications to use CSV files for recording data such as performance values or execution logs. And, there could be more than one CSV file which can give you a meaningful outcome only after their consolidation. In this example, we are demonstrating how to merge multiple CSV files using basic Python File I/O without losing any data.

A CSV file, as the name suggests, combines multiple fields separated by commas. The first row contains the name or title of each column, and the remaining rows contain the actual data values.

Also Read: Multiple Ways to Merge Python Dictionaries

Merging Multiple CSV Files in Python using File I/O

Below is the complete code to merge CSV files. We used a CSV format that records web service performance data. While using this example, you may need to modify the HEADER variable (with the names of actual fields) according to the format of your CSV.

Try One of Our Free Tools: Merge PDF Files for Free

It is essential to understand the header fields we’ve used in this example. It includes the following:

• Timestamp – The time of a web service request
• Client IP – The IP address of the client system making the calls
• Web Service – The name of the web service used
• Status – The return code reflecting success or failure
• GOOD – The total no. of successful requests
• BAD – The total no. of failure calls

import os

csv_header = 'Timestamp,Client IP,Web Service,Status,Good,Bad'
csv_out = 'consolidated.csv'

csv_dir = os.getcwd()

dir_tree = os.walk(csv_dir)
for dirpath, dirnames, filenames in dir_tree:
   pass

csv_list = []
for file in filenames:
   if file.endswith('.csv'):
      csv_list.append(file)

csv_merge = open(csv_out, 'w')
csv_merge.write(csv_header)
csv_merge.write('\n')

for file in csv_list:
   csv_in = open(file)
   for line in csv_in:
      if line.startswith(csv_header):
         continue
      csv_merge.write(line)
   csv_in.close()
   csv_merge.close()
print('Verify consolidated CSV file : ' + csv_out)

The above Python example provides fully runnable code to merge multiple CSV files. First, try copying/pasting and running this code. After that, you may extend or modify it according to your use case.

Also, Read – Pandas to Combine Multiple CSV Files

A Quick Wrap

In this example, we covered “How to Merge Multiple CSV Files in Python.” Our code doesn’t use any special Python package to combine the CSV files and can save you a lot of time from going through each one individually.

If you’ve learned something from this example, please share it with your colleagues. You can also connect to our social media accounts to receive timely updates.

Lastly, our site needs your support to remain free. If you gained some knowledge from this tutorial, share this post on social media (LinkedIn/Twitter).

Enjoy coding,
TechBeamers.