Apache Flink Create Table

Question

I'm trying to create table in flink using Table API in Java using eclipse using the following code.

EnvironmentSettings settings = EnvironmentSettings
            .newInstance()
            .inStreamingMode()
            .build();

TableEnvironment tEnv = TableEnvironment.create(settings);

tEnv.executeSql("CREATE TABLE sink_table (" +
            "    `user_a` BIGINT," +
            "    `product` VARCHAR," +
            "    `amount` BIGINT," +
            "    `name_list` ARRAY<STRING>," +
            "    `id_list` ARRAY<INT>," +
            "    PRIMARY KEY (user_a) NOT ENFORCED " +
            ") WITH (" +
            ")");

However program exits without creating any table in Flink SQL Client. How do I create table?

Note: It's getting created directly through SQL client. Also my environment is running Word Count Program through same setup of eclipse.

Also check https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/ for similar code.

David Anderson · Accepted Answer · 2024-04-15 17:26:18Z

3

In Apache Flink SQL, a table is nothing more than a description of how to interpret data stored (or to be stored) somewhere else. When you create such a table it's necessary to specify where the data actually is (or is to be written): e.g., a Kafka topic, a file, a PostgreSQL table, etc. This information about the data storage location goes in the WITH (...) part of the CREATE TABLE statement.

If you want to work with this table in more than one application, and/or in the SQL Client, you have a couple of options. You can use a copy of the same "CREATE TABLE ..." in each place you want to work with the same table, or you can save the table (not the underlying data, but just the table metadata) in a catalog.

In the Java code you shared in your question, you have created a table in the temporary catalog for that one job. You will need to save it in a persistent catalog if you want the SQL Client to be able to use it directly, or you can create a duplicate table in the SQL Client (this "duplicate" will refer to the same underlying data; only the table metadata would be duplicated).

Since this table is not associated with any data storage, it's not very usable.

edited Apr 15, 2024 at 17:26

answered Apr 25, 2022 at 13:41

David Anderson

44.3k4 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ajj Over a year ago

So if I create a table in java program, it won't be visible in Flink SQL unless there is some persistent metadata store shared between them, like HiveCatalog or JdbcCatalog? Any other way?

David Anderson Over a year ago

There is no other way. If you want to share the table metadata it must be persisted somewhere.

Collectives™ on Stack Overflow

Apache Flink Create Table

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related