Superset ingestor is inefficient to ingest large amount of data

[jeff-xu-z]

Describe the bug
Superset ingestor shall have better efficiency to handle large amount of data or access_token could expire before ingestion completes.

To Reproduce
Ingestion context:

• The system I work on has relatively large amount of data.
  • ~17k datasets
  • ~3k dashboards
  • ~30k charts
• My Superset's access token TTL is 15 minutes.

The snippet below represents my superset ingestion

source:
  type: superset
  config:
    # Coordinates
    connect_uri: https://<hostname>

    # Credentials
    username: <username>
    password: <password>

sink:
  type: file
  config:
    filename: /tmp/superset.json

Running "datahub ingest -c /tmp/superset.yml" will end with "401 Unauthorized" roughly 15 minutes after the ingestion starts on my system.

Expected behavior
Superset ingestion should complete successfully.

Desktop (please complete the following information):

• OS: Linux
• Browser Chrome
• Datahub: 0.13.0

Additional context
N/A

[jeff-xu-z]

I shall be able to contribute a proposed fix/improvements.