business-science.io

Create A Pandas Dataframe AI Agent With Generative AI, Python And OpenAI

2025-12-08 06:00:00

Hey guys, this is the first article in my NEW GenAI / ML Tips Newsletter. Today, we’re diving into the world of Generative AI and exploring how it can help companies automate common data science tasks. Specifically, we’ll learn how to create a Pandas dataframe agent that can answer questions about your dataset using Python, Pandas, LangChain, and OpenAI’s API. Let’s get started!

Here’s what you’ll learn in this article:

Why Generative AI is Transforming Data Science
What is a Pandas Data Frame Agent?
Create a Pandas DataFrame Agent
- Setting Up the Environment
- Loading and Exploring the Dataset
- Creating the Data Analysis Agent with LangChain
- Interacting with the Agent
- Visualizing the Results
Before You Go Any Further: Join the Free GenAI/ML Tips Newsletter to get the Data and Code so you can follow along

This is what you are making today

We’ll use this Generative AI Workflow to combine data (from CSVs or SQL databases) with a Pandas Data Frame Agent that helps us produce common analytics outputs like visualizations and reports.

Get the Code (In the AI-Tip 001 Folder)

SPECIAL ANNOUNCEMENT: AI for Data Scientists Workshop on December 18th

Inside the workshop I’ll share how I built a SQL-Writing Business Intelligence Agent with Generative AI:

What: GenAI for Data Scientists

When: Wednesday December 18th, 2pm EST

How It Will Help You: Whether you are new to data science or are an expert, Generative AI is changing the game. There’s a ton of hype. But how can Generative AI actually help you become a better data scientist and help you stand out in your career? I’ll show you inside my free Generative AI for Data Scientists workshop.

Price: Does Free sound good?

How To Join: 👉 Register Here

GenAI/ML-Tips Weekly

This article is part of GenAI/ML Tips Weekly, a weekly video tutorial that shows you step-by-step how to do common Data Science and Generative AI coding tasks. Pretty cool, right?

Here is the link to get set up. 👇

Get the Code (In the GenAI/ML Tip 001 Folder)

This Tutorial is Available in Video (9-minutes)

I have a 9-minute video that walks you through setting up the Pandas Data Frame Agent and running data analysis with it. 👇

Why Generative AI is Transforming Data Science

Generative AI, powered by models like OpenAI’s GPT series, is reshaping the data science landscape. These models can understand and generate human-like text, making it possible to interact with data in more intuitive ways. By integrating Generative AI into data science, you can:

Automate Data Insights: Quickly generate summaries and insights from complex datasets.
Enhance Decision Making: Obtain answers to specific questions without manually sifting through data.
Improve Accessibility: Make data science more accessible to non-technical stakeholders.

Creating a Pandas dataframe agent combines the power of AI with data science, enabling you to unlock new possibilities in data exploration and interpretation from Natural Language.

What is a Pandas Data Frame Agent?

A Pandas Data Frame Agent automates common Pandas operations from Natural Language inputs.

It can be used to perform:

GroupBy + Aggregate
Math calculations (that normal LLMs struggle with)
Filters
Pivots
Window calculations
Resampling (Time Series)
Binning
Log Transformations
Summary Statistics (Mean, Median, IQR, Min/Max, Count (frequency), etc)

All from Natural Language prompts.

Make A Pandas Data Frame Agent

Let’s walk through the steps to create a Pandas data frame agent that can answer questions about a dataset using Python, OpenAI’s API, Pandas, and LangChain.

Quick Reminder: You can get all of the code and datasets shown in a Python Script and Jupyter Notebook when you join my GenAI/ML Tips Newsletter.

Code Location: /001_pandas_dataframe_agent

Step 1: Setting Up the Python Environment

First, you’ll need to set up your Python environment and install the required libraries.

pip install openai langchain langchain_openai langchain_experimental pandas plotly pyyaml

Next, import the libraries.

Then run this to access our utility function, parse_json_to_dataframe().

The last part is to set up your OpenAI API Key. Make sure to get an API Key from OpenAI’s API website.

Note: Replace ‘credentials.yml’ with the path to your YAML file containing the OpenAI API key or set the ‘OPENAI_API_KEY’ environment variable directly.

Step 2: Loading and Exploring the Dataset

Load your dataset into a Pandas DataFrame. For this tutorial, we’ll use a sample customer data CSV file. But you could easily use any data that you can get into a Pandas Data Frame:

SQL Database
CSV
Excel File

Run this code to load the customer dataset:

This dataset contains customer information, including sales and geography data.

Step 3: Create the Pandas Data Analysis Agent with LangChain

Initialize the language model and create the Pandas data analysis agent using LangChain.

This is what’s happening:

ChatOpenAI: Initializes the OpenAI language model.
create_pandas_dataframe_agent: Creates an agent that can interact with the Pandas DataFrame.
agent_type: Specifies the type of agent (using OpenAI functions).
suffix: Instructs the agent to return results in JSON format for easy parsing.

Pro-Tip: The secret sauce is to use the suffix parameter to specify the output format. Under the hood, this appends the agent’s default prompt template with additional information that describes how to return the information.

Step 4: Interacting with the Pandas Data Frame Agent

Now, you can ask the agent questions about your data. Try running this code with a Natural Language analysis question:

“What are the total sales by geography?”

The agent processes the query and returns a response.

This is where Post Processing comes into play. Remember when I added the suffix parameter to return JSON. The Agent actually burries the JSON in a string.

That’s OK, because I have created a handy little parsing tool that extracts the JSON from the string and converts it to a Pandas Data Frame for us.

Step 5: Visualizing the Results

With a pandas data frame we can then report the results. I’ll do this manually with Plotly, but a great challenge is to extend the code to create an AI agent that makes the visualization code and executes it automatically.

This visualization provides a clear view of sales distribution across different geographical regions.

Quick Reminder: You can get all of the code and datasets shown in a Python Script and Jupyter Notebook when you join my GenAI/ML Tips Newsletter.

Conclusion

By integrating Generative AI with data science, you’ve created a powerful tool that can interact with your data in natural language. This Pandas data analysis agent simplifies the process of extracting insights and can help non-technical stakeholders automate common data manipulations to help them make data-driven decisions.

But there’s so much more to learn in Generative AI and data science.

If you’re excited to become a Generative AI Data Scientist with Python, then keep reading…

Become A Generative AI Data Scientist

The future of data science is AI / ML.

I’ve helped 6,107+ students learn data science and now I’m helping them become Generative AI Data Scientists, skilled in combining Generative AI / ML. With this system they have:

Landed Promotions to Manager of AI/ML Teams ($200,000+ Role)
Made Proof-Of-Concepts for Clients ($25,000+ Consulting Projects)
Grew their data science skills with Generative AI (Career Growth)

Here’s the system they are taking to become Generative AI Data Scientists:

This is a Live 8-Week Generative AI Bootcamp for Data Scientists that covers:

Week 1: Live Kickoff Clinic + Local LLM Training + AI Fast Track
Week 2: Retrieval Augmented Generation (RAG)
Week 3: Business Intelligence AI Copilot (SQL + Pandas Tools)
Week 4: Customer Analytics Team (Multi-Agent Workflows)
Week 5: Time Series Forecasting Team (Multi-Agent Machine Learning Workflows)
Week 6: LLM Model Deployment AWS Bedrock
Week 7: Fine-Tuning LLM Models AWS Bedrock
Week 8: AI App Deployment With AWS Cloud

Enroll In The Next Cohort Here
(And Become A Generative AI Data Scientist in 2026)

3 Ways to Read Multiple CSV Files: For-Loop, Map, List Comprehension

2021-09-21 02:01:00

Reading many CSV files is a common task for a data scientist. In this free tutorial, we show you 3 ways to streamline reading CSV files in Python. You’ll read and combine 15 CSV Files using the top 3 methods for iteration.

Python Tips Weekly

This article is part of Python-Tips Weekly, a bi-weekly video tutorial that shows you step-by-step how to do common Python coding tasks.

Here are the links to get set up. 👇

Video Tutorial
Follow along with our Full YouTube Video Tutorial.

This 5-minute video covers reading multiple CSV in python.

(Click image to play tutorial)

Read 15 CSV Files [Tutorial]

This FREE tutorial showcases the awesome power of python for reading CSV files. We’ll read 15 CSV files in this tutorial.

Before we get started, get the Python Cheat Sheet

The Python Ecosystem is LARGE. To help, I’ve curated many of the 80/20 Python Packages, those I use most frequently to get results. Simply Download the Ultimate Python Cheat Sheet to access the entire Python Ecosystem at your fingertips via hyperlinked documentation and cheat sheets.

(Click image to download)

Onto the tutorial.

Project Setup

First, load the libraries. We’ll import pandas and glob.

Pandas: The main data wrangling library in Python
glob: A library for locating file paths using text searching (regular expressions)

Second, use glob to extract a list of the file paths for each of the 15 CSV files we need to read in.

Get the code.

Method 1: For-Loop

The most common way to repetitively read files is with a for-loop. It’s a great way for beginners but it’s not the most concise. We’ll show this way first.

Get the code.

We can see that this involves 3-steps:

Instantiating an Empty List: We do this to store our results as we make them in the for-loop.
For-Each filename, read and append: We read using pd.read_csv(), which returns a data frame for each path. Then we append each data frame to our list.
Combine each Data Frame: We use pd.concat() to combine the list of data frames into one big data frame.

PRO-TIP: Combining data frames in lists is a common strategy. Don’t forget to use axis=0 to specify row-wise combining.

Method 2: Using Map

The map() function is a more concise way to iterate. The advantage is that we don’t have to instantiate a list. However, it can be more confusing to beginners.

How it works:

Map takes in two general arguments:

func: A function to iteratively apply
*iterables: One or more iterables that are supplied to the function in order of the functions arguments.

Get the code.

Let’s use it.

Ok, so let’s try map().

Get the code.

We use 3-steps:

Make a Lambda Function: This is an anonymous function that we create on the fly with the first argument that will accept our iterable (each filename in our list of csv file paths).
Supply the iterable: In this case, we provide our list of csv files. The map function will then iteratively supply each element to the function in succession.
Convert to List: The map() function returns a map object. We can then convert this to a list using the list() function.

PRO-TIP: Beginners can be confused by the “map object” that is returned. Just simply use the list() function to extract the results of map() in a list structure.

Method 3: List Comprehension

Because we are returning a list, even easier than map(), we can use a List Comprehension. A list comprehension is a streamlined way of making a for-loop that returns a list. Here’s how it works.

Get the code.

Do this: Add the function that you want to iterate. The parameter must match your looping variable name (next).
For each of these: This is your looping variable name that you create inside of the list comprehension. Each of these are elements that will get passed to your function.
In this: This is your iterable. The list containing each of our file paths.

Summary

There you have it. You now know how to read CSV files using 3 methods:

For-Loops
Map
List Comprehension

But there’s a lot more to learning data science. And if you’re like me, you’re interested in a fast track system that will advance you without wasting time on information you don’t need.

The solution is my course, Data Science Automation with Python

Data Science Automation with Python Course

Tired of struggling to learn data science? Getting stuck in a sea of neverending resources? Eliminate the confusion and speed up your learning in the process.

Businesses are transitioning manual processes to Python for automation. We teach you skills that organizations need right now.

Learn how in our new course, Python for Data Science Automation. Perform an end-to-end business forecast automation using pandas, sktime, and papermill, and learn Python in the process.

SweetViz: Automated Exploratory Data Analysis (EDA) in Python

2021-08-03 04:01:00

SweetViz is a Python library that makes exploratory data analysis (EDA) fast and effective. Learn how to investigate feature relationships using correlation and associations in the automated SweetViz report.

Python Tips Weekly

This article is part of Python-Tips Weekly, a bi-weekly video tutorial that shows you step-by-step how to do common Python coding tasks.

Here are the links to get set up. 👇

Video Tutorial
Follow along with our Full YouTube Video Tutorial.

Learn how to use SweetViz to make and investigate an automated EDA Report.

(Click image to play tutorial)

Before we get started, get the Python Cheat Sheet

(Click image to download)

Onto the tutorial.

SweetViz: Automating EDA

Let’s check out how to automate an exploratory data analysis report with SweetViz.

Get the code.

Step 1: Load Libraries and Data

First, let’s load the libraries and data. From the libraries, we’ll import pandas, sweetviz and my favorite plotting library, plotnine.

Get the code.

The mpg_df data set contains information on fuel efficiency (mpg) along with important vehicle attributes for 398 vehicles.

Get the code.

Step 2: Make the SweetViz EDA Report in 2 Lines of Code

Goal: Understand the relationship between Fuel Economy (MPG) and features in this dataset

We can assess the relationship between vehicle fuel economy and the explanatory features using the sweetviz report. SweetViz automates the process of creating the EDA report in two lines of code.

Get the code.

This creates the SweetViz EDA Report.

Step 3: Investigate the Feature Correlation (Associations)

We can investigate the feature associations / correlations and see that number of cylinders (engine size), displacement (engine volume), horsepower, weight have a relationship to vehicle fuel efficiency.

3A: High-Level Correlations (Associations)

We start with an overall view of the high-level relationships.

The squares are categorical features. Their relationships range from 0 to 1 indicating associative strength only. We need to need to inspect tabs for categories like cylinder, which has a high associative strength to determine its effect on MPG.
The circles are numeric features. Their relationships range from -1 to 1 following a Pearson Correlation. We can see positive and negative relationships indicated by the sign (+/-) and strength of relationship (closer to +/-1 is strong, closer to zero is weak).

3B: Distribution Analysis: Individual Features

We can take a step further and investigate individual features to see how each relate to the target by comparing their distributions.

Feature Tabs

For example, we can investigate “cylinders” to see how the distributions co-vary. Just click on the “cylinders” tab.

Exploratory Panels

This opens up an exploratory panel with useful information that compares the distribution of vehicles by cylinder to their average MPG.

We can see that:

4 Cylinder Engines: Have the highest average MPG
8 Cylinder Engines: Have the lowest average MPG

It’s that easy to explore your dataset!

Summary

Exploratory data analysis can be automated with the python SweetViz reporting package. SweetViz makes it fast and easy to explore features and determine relationships to a target. In our case, we saw that 4 cylinder engines have the highest average MPG while 8 cylinder engines have the lowest average MPG.

Learn More: Business Process Automation with Python

Businesses are transitioning manual processes to Python for automation. Learn how in our new course, Python for Data Science Automation. Perform and end-to-end business forecast automation using pandas, sktime, and papermill, and learn Python in the process.

business-science.io

Create A Pandas Dataframe AI Agent With Generative AI, Python And OpenAI

Table of Contents

This is what you are making today

SPECIAL ANNOUNCEMENT: AI for Data Scientists Workshop on December 18th

GenAI/ML-Tips Weekly

This Tutorial is Available in Video (9-minutes)

Why Generative AI is Transforming Data Science

What is a Pandas Data Frame Agent?

Make A Pandas Data Frame Agent

Step 1: Setting Up the Python Environment

Step 2: Loading and Exploring the Dataset

Step 3: Create the Pandas Data Analysis Agent with LangChain

Step 4: Interacting with the Pandas Data Frame Agent

Step 5: Visualizing the Results

Conclusion

Become A Generative AI Data Scientist

Here’s the system they are taking to become Generative AI Data Scientists:

3 Ways to Read Multiple CSV Files: For-Loop, Map, List Comprehension

Python Tips Weekly

Video TutorialFollow along with our Full YouTube Video Tutorial.

Read 15 CSV Files [Tutorial]

Before we get started, get the Python Cheat Sheet

Project Setup

Method 1: For-Loop

Method 2: Using Map

How it works:

Let’s use it.

Method 3: List Comprehension

Summary

Data Science Automation with Python Course

SweetViz: Automated Exploratory Data Analysis (EDA) in Python

Python Tips Weekly

Video TutorialFollow along with our Full YouTube Video Tutorial.

Before we get started, get the Python Cheat Sheet

SweetViz: Automating EDA

Step 1: Load Libraries and Data

Step 2: Make the SweetViz EDA Report in 2 Lines of Code

Step 3: Investigate the Feature Correlation (Associations)

3A: High-Level Correlations (Associations)

3B: Distribution Analysis: Individual Features

Feature Tabs

Exploratory Panels

Summary

Learn More: Business Process Automation with Python

Video Tutorial
Follow along with our Full YouTube Video Tutorial.

Video Tutorial
Follow along with our Full YouTube Video Tutorial.