Skip to content

[ SDK PERFORMANCE] The execution of the show tables command takes a long time. #4295

New issue
New issue
Open
Open
[ SDK PERFORMANCE] The execution of the show tables command takes a long time.#4295

Description

@wangxiaopeng0329
wangxiaopeng0329
opened on Mar 27, 2023

image

As shown in the above figure, CarbonShowTablesCommand obtains metadata from metastore for each table. Currently, when there are 180,000 tables, it takes a long time (about 1 hours) to run the show tables command in spark-sql shell, which needs to be optimized.
When the filter function is not invoked, it takes about 12 seconds to obtain 180,000 tables by running the show tables command.As shown in the following figure.
image

Activity

  1. kevinjmh commented on Mar 30, 2023

    @kevinjmh
    kevinjmh
    on Mar 30, 2023
    Member

    Maybe we can try to get tables' info in one batch instead of one by one

      def getTable(db: String, table: String): CatalogTable

  def getTablesByName(db: String, tables: Seq[String]): Seq[CatalogTable]

  2. wangxiaopeng0329 commented on Mar 30, 2023

    @wangxiaopeng0329
    wangxiaopeng0329
    on Mar 30, 2023
    Author

    Maybe we can try to get tables' info in one batch instead of one by one

      def getTable(db: String, table: String): CatalogTable

  def getTablesByName(db: String, tables: Seq[String]): Seq[CatalogTable]

    hi, when and which version can solve the problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions