Repository navigation
Improve example docs around SQLExecuteQueryOperator in all SQLrelated providers #44807
Description
Body
Context: #44707 (comment)
In Sqlite provider we have operators.rst which explains how to leverage SQLExecuteQueryOperator from common.sql provider to be used with Sqlite. We should have similar doc and examples for all SQL related operators that were removed as part of #44559
The Task:
verify that we have operators.rst in each of the above providers that explain how to use SQLExecuteQueryOperator with the specific provider See sqlite provider operators.rst for example template:
.. _howto/operator:SqliteOperator:
SQLExecuteQueryOperator to connect to Sqlite
============================================
Use the :class:`SQLExecuteQueryOperator<airflow.providers.common.sql.operators.sql>` to execute
Sqlite commands in a `Sqlite <https://sqlite.org/lang.html>`__ database.
.. note::
Previously, ``SqliteOperator`` was used to perform this kind of operation. After deprecation this has been removed. Please use ``SQLExecuteQueryOperator`` instead.
Using the Operator
^^^^^^^^^^^^^^^^^^
Use the ``conn_id`` argument to connect to your Sqlite instance where
the connection metadata is structured as follows:
.. list-table:: Sqlite Airflow Connection Metadata
:widths: 25 25
:header-rows: 1
* - Parameter
- Input
* - Host: string
- Sqlite database file
An example usage of the SQLExecuteQueryOperator to connect to Sqlite is as follows:
.. exampleinclude:: /../../providers/tests/system/sqlite/example_sqlite.py
:language: python
:start-after: [START howto_operator_sqlite]
:end-before: [END howto_operator_sqlite]
Furthermore, you can use an external file to execute the SQL commands. Script folder must be at the same level as DAG.py file.
.. exampleinclude:: /../../providers/tests/system/sqlite/example_sqlite.py
:language: python
:start-after: [START howto_operator_sqlite_external_file]
:end-before: [END howto_operator_sqlite_external_file]
If possible accommodate for the specific parameters that can be passed from the provider to the hook.
For example in Postgres we can pass hook_params for enabling logging so we explain about this explicitly with example
The items:
- Exasol (Previously had
ExasolOperator) - apache.drill (Previously had
DrillOperator) - jdbc (Previously had
JdbcOperator) - microsoft.mssql (Previously had
MsSqlOperator) - mysql (Previously had
MySQLOperator) - oracle (Previously had
OracleOperator) - presto Previously had
PrestoOperator) - snowflake (Previously had
SnowflakeOperator) - trino (Previously had
TrinoOperator) - vertica (Previously had
VerticaOperator) - postgres (Previously had
PostgresOperator) Docs already exists but requires tweaks to align the style
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
geraj1010 commentedon Dec 11, 2024
To confirm, this is for all operators that were deprecated (and merged) in #44559? Are you looking for additional
HOW-TOguides for theThe Items?eladkal commentedon Dec 11, 2024
Yes, get what we have for sqlite to the other providers. I assume it's mostly the same with difference of
conn_idgeraj1010 commentedon Dec 15, 2024
For Amazon, I did find mention of
SQLExecuteQueryOperatorfor Redshift and Athena, but not RDS.Redshift: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/redshift/redshift_sql.html
Athena: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/athena/athena_sql.html
Should I replace the existing files with
operators.rstand create a new one for RDS (if needed)?eladkal commentedon Dec 15, 2024
Amazon is not on the list because the docs there are OK (I looked only on Redshift)
If RDS can be read with this operator then you are welcome to add it
aaron-y-chen commentedon Jan 26, 2025
Hi @eladkal could you assign this issue to me? I'd like to give it a try.
eladkal commentedon Apr 22, 2025
@nailo2c are you working on the remaining items?
aaron-y-chen commentedon Apr 22, 2025
@eladkal yes, I'm still working on the remaining items.
aaron-y-chen commentedon Apr 24, 2025
The items:
Once #49679 is merged, all items will use SQLExecuteQueryOperator, so I believe this issue can be closed.
aaron-y-chen commentedon Apr 30, 2025
Hi @eladkal, #49679 is merged, so I believe this issue can be closed.