Data Engineering series, Part 3 of 5. Previous: What Does a Data Engineer Do?

  1. What Is a Data Engineer?
  2. What Does a Data Engineer Do?
  3. Data Engineering Tools: SQL, Python, and Essential Technologies (you are here)
  4. What Is a Data Pipeline?
  5. How to Become a Data Engineer

The two most important data engineering tools are SQL and Python. SQL is how you work with data in databases, and Python is how you automate, clean, and move data. Around them sit databases, data warehouses, orchestration tools, cloud platforms, and everyday engineering tools like Git and Docker.

The list of tools can look overwhelming, because the field has a huge ecosystem. This guide gives you the map: what each category of tool does, popular examples, and what to learn first.

The Data Engineering Toolbox at a Glance

CategoryWhat it doesPopular examples
LanguagesWrite the logic that moves and shapes dataSQL, Python, Scala, Java, Bash
DatabasesStore data for applications and analysisPostgreSQL, MySQL, SQLite, MongoDB
Data warehousesStore large amounts of data optimized for analysisSnowflake, Google BigQuery, Amazon Redshift, Databricks
ProcessingClean and transform data at any scalePandas, Apache Spark, dbt
OrchestrationSchedule and manage pipelinescron, Apache Airflow, Dagster, Prefect
IngestionBring data in from sourcesPython scripts, Airbyte, Fivetran, Apache Kafka (streaming)
Cloud platformsRun storage and compute without owning serversAWS, Microsoft Azure, Google Cloud
Engineering toolsBuild and ship reliable codeGit, Linux, Docker, VS Code, Jupyter
Data quality and monitoringCheck data and watch for failuresCustom checks, dbt tests, Great Expectations, alerts

You don’t need every tool. Most jobs use a handful from each row, and the exact names change from company to company.

1. SQL: The Foundation

SQL (Structured Query Language) is the language for working with relational databases. It is the single most important skill for a data engineer. You use it to create tables, select and filter rows, combine tables, summarize data, and transform it.

Core SQL topics to learn:

2. Python: The Glue

Python is the most common general-purpose language in data engineering. It reads files, calls APIs, cleans data, loads it into databases, and glues the other tools together. A few libraries do much of the work:

See Python for data analysis: accessing your data for the different ways Python reads data. If you are starting from zero, begin with Python basics.

SQL and Python Working Together

Here is a small example that uses both. Suppose you have a file named orders.csv:

order_id,customer,order_date,amount
1,Ana,2026-01-05,35.50
2,Luis,2026-01-06,120.00
3,Ana,2026-01-09,58.25
4,Marta,2026-01-10,19.99
5,Luis,2026-01-12,75.00

This script loads it into a SQLite database with Python, then uses SQL to summarize it:

import sqlite3

import pandas as pd

# Python: read the file and load it into a database
orders = pd.read_csv("orders.csv", parse_dates=["order_date"])
con = sqlite3.connect("shop.db")
orders.to_sql("orders", con, if_exists="replace", index=False)

# SQL: summarize by customer
summary = pd.read_sql(
    """
    SELECT customer,
           COUNT(*)    AS orders,
           SUM(amount) AS total
    FROM orders
    GROUP BY customer
    ORDER BY total DESC
    """,
    con,
)
print(summary)

Output:

  customer  orders   total
0     Luis       2  195.00
1      Ana       2   93.75
2    Marta       1   19.99

You can also use a window function in the same database to calculate a running total for each customer:

running = pd.read_sql(
    """
    SELECT customer,
           DATE(order_date) AS order_date,
           amount,
           SUM(amount) OVER (
               PARTITION BY customer ORDER BY order_date
           ) AS running_total
    FROM orders
    ORDER BY customer, order_date
    """,
    con,
)
print(running)

Output:

  customer  order_date  amount  running_total
0      Ana  2026-01-05   35.50          35.50
1      Ana  2026-01-09   58.25          93.75
2     Luis  2026-01-06  120.00         120.00
3     Luis  2026-01-12   75.00         195.00
4    Marta  2026-01-10   19.99          19.99

This pattern, Python to move data and SQL to shape it, is at the heart of a lot of data engineering work.

3. Other Languages

  • Bash and the command line to automate tasks on Linux. See what Linux is.
  • Scala and Java appear mostly in big data tools built on the Java platform, such as Apache Spark. You can start without them.

4. Databases

Relational databases store data in tables with relationships between them, and they are the starting point for most engineers. Good options to practice with:

  • SQLite: a simple file-based database, perfect for learning.
  • PostgreSQL: a powerful open-source database. See the PostgreSQL guide.
  • MySQL: another popular open-source database. See MySQL with Docker.

You will also meet NoSQL databases such as MongoDB (documents) and Redis (fast key-value storage), which suit different kinds of data. Learn how to design tables well with database design.

5. Data Warehouses and Data Lakes

A data warehouse is a database built for analysis: it stores large amounts of historical data and answers big analytical queries quickly. Popular cloud examples include Snowflake, Google BigQuery, Amazon Redshift, and Databricks. A data lake stores raw files, often in cloud storage, so you can keep data in its original form and process it later.

Many companies use both. You don’t need an account on every platform to learn: the concepts of tables, schemas, and SQL carry over between them.

6. File Formats

Data comes in files, and the format matters:

  • CSV: simple, readable, and universal, but with no data types.
  • JSON: flexible and common for APIs and nested data.
  • Parquet: a compressed columnar format that is much more efficient for large analytical datasets.

7. Processing and Transformation

  • Pandas is great for data that fits in your computer’s memory.
  • Apache Spark processes data across many machines when a dataset is too large for one computer.
  • dbt lets you write transformations as SQL inside the warehouse and adds testing and documentation. It fits the ELT approach explained in ETL vs ELT.

8. Orchestration

Orchestration tools schedule pipelines and manage the order they run in, with retries and alerts.

  • cron is the basic scheduler on Linux, fine for simple jobs.
  • Apache Airflow is a widely used open-source tool where you define workflows in Python.
  • Dagster and Prefect are newer alternatives with a modern developer experience.

Start with cron to understand scheduling, then learn one orchestrator when you need dependencies and monitoring.

9. Ingestion and Streaming

Ingestion tools collect data from sources. You can write your own Python scripts or use connector tools such as Airbyte (open source) and Fivetran (commercial), which provide ready-made connectors to common sources. For data that arrives continuously, such as clicks or sensor readings, Apache Kafka is a common platform for streaming events.

10. Cloud Platforms

Most modern data systems run in the cloud, on AWS, Microsoft Azure, or Google Cloud. They offer storage, databases, warehouses, and managed pipeline services so you don’t run your own servers. Start by understanding the basics in cloud computing explained. Pick one platform to learn deeply, because the ideas transfer to the others.

11. Everyday Engineering Tools

These aren’t specific to data, but you will use them daily:

12. Data Quality, Monitoring, and Security

Pipelines need checks and alerts. You can start with simple checks you write yourself (see Part 2), then add tools like dbt tests or Great Expectations as projects grow. Remember to protect what you handle: read database security and privacy.

13. The Tools Your Users Use

Data engineers don’t usually build dashboards, but you should know what the consumers use, such as Power BI and Python charts (data visualization with Python). Knowing how your data is used helps you design better pipelines.

What to Learn First

OrderLearnWhy
1SQLThe core skill in every data job
2PythonAutomation, cleaning, and glue code
3Git and the command lineNeeded for all real projects
4A relational database (PostgreSQL)Practice storing and modeling data
5DockerRun databases and tools easily
6A cloud platform (basics)Where most data systems live
7An orchestrator (Airflow, Dagster, or Prefect)Schedule and monitor pipelines
8Spark, dbt, or Kafka as the job requiresSpecialized tools for scale and streaming

Don’t try to learn everything at once. Tools change, but the fundamentals (SQL, data modeling, and how pipelines work) stay valuable for years.

Frequently Asked Questions

What are the most important data engineering tools?

SQL and Python are the most important. After them, learn a relational database like PostgreSQL, Git, Docker, a cloud platform, and an orchestration tool.

Do data engineers use Python or SQL more?

Both, and they complement each other. SQL does most of the work inside databases and warehouses, while Python moves data, automates tasks, and handles things SQL can’t.

Do I need to learn Spark?

Not at first. Spark matters when datasets are too big for a single computer. Learn SQL, Python, and Pandas first, and add Spark if a job calls for it.

Which cloud platform should I learn?

Choose one of AWS, Azure, or Google Cloud, ideally the one used in jobs you are targeting. The core ideas transfer between them.

What is the difference between a database and a data warehouse?

A database usually supports applications and day-to-day transactions. A data warehouse is designed to store large amounts of historical data for analysis and reporting.

Are no-code tools enough?

They can handle simple automation, such as Power Automate flows, but professional data engineering relies on code for flexibility, testing, and scale.

Conclusion

Data engineering tools fall into clear categories: languages, databases and warehouses, processing, orchestration, ingestion, cloud, and engineering basics. Start with SQL and Python, add a database, Git, and Docker, and then learn the cloud and orchestration tools your goals require.

In Part 4, you will see how these tools combine into a working system: What Is a Data Pipeline? Understanding ETL, ELT, and Data Integration.

Next in the Data Engineering series: What Is a Data Pipeline?