CreateTable

Create a BigQuery table.

yaml
type: "io.kestra.plugin.gcp.bigquery.CreateTable"

Examples

yaml
id: gcp_bq_create_table
namespace: company.team

tasks:
  - id: create_table
    type: io.kestra.plugin.gcp.bigquery.CreateTable
    projectId: my-project
    dataset: my-dataset
    table: my-table
    tableDefinition:
      type: TABLE
      schema:
        fields:
        - name: id
          type: INT64
        - name: name
          type: STRING
      standardTableDefinition:
        clustering:
        - id
        - name
    friendlyName: new_table

Properties

dataset *string

The dataset's user-defined ID.

table *string

The table's user-defined ID.

description string

The user-friendly description for the table.

encryptionConfiguration EncryptionConfiguration

The encryption configuration.

expirationDuration string

Format duration

Sets the duration, since now, when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName string

The user-friendly name for the table.

impersonatedServiceAccount string

The GCP service account to impersonate.

labels object

SubType string

Return a map for labels applied to the table.

location string

The geographic location where the dataset should reside.

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId string

The GCP project ID.

requirePartitionFilter booleanstring

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

retryAuto Constant Exponential Random

Automatic retry for retryable BigQuery exceptions.

Some exceptions (especially rate limit) are not retried by default by BigQuery client, we use by default a transparent retry (not the kestra one) to handle this case. The default values are exponential of 5 seconds for a maximum of 15 minutes and ten attempts

retryMessages array

SubType string

Default ["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]

The messages which would trigger an automatic retry.

Message is tested as a substring of the full message, and is case insensitive.

retryReasons array

SubType string

Default ["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

The reasons which would trigger an automatic retry.

scopes array

SubType string

Default ["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used.

serviceAccount string

The GCP service account.

tableDefinition TableDefinition

The table definition.

Outputs

creationTime string

Format date-time

The time when this table was created.

datasetId string

The dataset's ID.

definition TableDefinition-Output

The table definition.

description string

The user-friendly description for the table.

encryptionConfiguration EncryptionConfiguration-Output

The encryption configuration.

etag string

The hash of the table resource.

expirationTime string

Format date-time

Returns the time when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName string

The user-friendly name for the table.

generatedId string

The service-generated id for the table.

labels object

SubType string

Return a map for labels applied to the table.

lastModifiedTime string

Format date-time

The time when this table was last modified.

numBytes integer

The size of this table in bytes.

numLongTermBytes integer

The number of bytes considered "long-term storage" for reduced billing purposes.

numRows integer

The number of rows of data in this table.

projectId string

The project's ID.

requirePartitionFilter boolean

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

selfLink string

The URL that can be used to access the resource again. The returned URL can be used for get or update requests.

table string

The table name.

Definitions

io.kestra.core.models.tasks.retrys.Constant

interval *string

Format duration

type *object

behavior string

Default RETRY_FAILED_TASK

Possible Values

RETRY_FAILED_TASKCREATE_NEW_EXECUTION

maxAttempts integer

Minimum >= 1

maxDuration string

Format duration

warningOnRetry boolean

Default false

io.kestra.plugin.gcp.bigquery.models.TableDefinition

externalTableDefinition ExternalTableDefinition

The external table definition if the type is EXTERNAL.

materializedViewDefinition MaterializedViewDefinition

The materialized view definition if the type is MATERIALIZED_VIEW.

schema Schema

The table's schema.

standardTableDefinition StandardTableDefinition

The table definition if the type is TABLE.

type string

Possible Values

TABLEVIEWMATERIALIZED_VIEWEXTERNALMODEL

The table's type.

viewDefinition ViewDefinition

The view definition if the type is VIEW.

io.kestra.plugin.gcp.bigquery.models.EncryptionConfiguration-Output

kmsKeyName string

io.kestra.plugin.gcp.bigquery.models.StandardTableDefinition

clustering array

SubType string

Returns the clustering configuration for this table. If {@code null}, the table is not clustered.

rangePartitioning RangePartitioning

Returns the range partitioning configuration for this table. If {@code null}, the table is not range-partitioned.

streamingBuffer StandardTableDefinition-StreamingBuffer

Returns information on the table's streaming buffer, if exists. Returns {@code null} if no streaming buffer exists.

timePartitioning TimePartitioning

Returns the time partitioning configuration for this table. If {@code null}, the table is not time-partitioned.

io.kestra.core.models.tasks.retrys.Exponential

interval *string

Format duration

maxInterval *string

Format duration

type *object

behavior string

Default RETRY_FAILED_TASK

Possible Values

RETRY_FAILED_TASKCREATE_NEW_EXECUTION

delayFactor number

maxAttempts integer

Minimum >= 1

maxDuration string

Format duration

warningOnRetry boolean

Default false

io.kestra.plugin.gcp.bigquery.models.RangePartitioning-Range-Output

end integer

interval integer

start integer

io.kestra.plugin.gcp.bigquery.models.Schema-Output

fields array

SubType

io.kestra.plugin.gcp.bigquery.models.RangePartitioning-Range

The end of range partitioning. integerstring

The start of range partitioning. integerstring

The width of each interval. integerstring

io.kestra.core.models.tasks.retrys.Random

maxInterval *string

Format duration

minInterval *string

Format duration

type *object

behavior string

Default RETRY_FAILED_TASK

Possible Values

RETRY_FAILED_TASKCREATE_NEW_EXECUTION

maxAttempts integer

Minimum >= 1

maxDuration string

Format duration

warningOnRetry boolean

Default false

io.kestra.plugin.gcp.bigquery.models.TableDefinition-Output

externalTableDefinition ExternalTableDefinition-Output

materializedViewDefinition MaterializedViewDefinition-Output

schema Schema-Output

standardTableDefinition StandardTableDefinition-Output

type string

Possible Values

TABLEVIEWMATERIALIZED_VIEWEXTERNALMODEL

viewDefinition ViewDefinition-Output

io.kestra.plugin.gcp.bigquery.models.PolicyTags

The policy tags' names. array

SubType string

io.kestra.plugin.gcp.bigquery.models.RangePartitioning

The range of range partitioning. RangePartitioning-Range

The range partitioning field. string

io.kestra.plugin.gcp.bigquery.models.MaterializedViewDefinition-Output

enableRefresh boolean

lastRefreshDate string

Format date-time

query string

refreshInterval string

Format duration

io.kestra.plugin.gcp.bigquery.models.Schema

The fields associated with this schema. array

SubType

io.kestra.plugin.gcp.bigquery.models.ViewDefinition

query string

The query whose result is persisted.

viewUserDefinedFunctions array

SubType

User defined functions that can be used by query. Returns {@code null} if not set.

io.kestra.plugin.gcp.bigquery.models.RangePartitioning-Output

field string

range RangePartitioning-Range-Output

io.kestra.plugin.gcp.bigquery.models.EncryptionConfiguration

The KMS key name. string

io.kestra.plugin.gcp.bigquery.models.PolicyTags-Output

names array

SubType string

io.kestra.plugin.gcp.bigquery.models.MaterializedViewDefinition

enableRefresh booleanstring

Whether automatic refresh is enabled for the materialized view when the base table is updated.

lastRefreshDate string

Format date-time

Date when this materialized view was last modified.

query string

The query whose result is persisted.

refreshInterval string

Format duration

The maximum frequency at which this materialized view will be refreshed.

io.kestra.plugin.gcp.bigquery.models.TimePartitioning

expiration string

Format duration

The number of milliseconds for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partition does not expire.

field string

If not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field.

requirePartitionFilter booleanstring

If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified.

type string

Possible Values

DAYHOURMONTHYEAR

The time partitioning type.

io.kestra.plugin.gcp.bigquery.models.Field

description string

The field description.

mode string

Possible Values

NULLABLEREQUIREDREPEATED

The field mode.

By default, Field.Mode.NULLABLE is used.

name string

The field name.

policyTags PolicyTags

The policy tags for the field.

subFields array

SubType

The list of sub-fields if type is a LegacySQLType.RECORD. Returns null otherwise.

type string

Possible Values

BOOLINT64FLOAT64NUMERICBIGNUMERICSTRINGBYTESSTRUCTARRAYTIMESTAMPDATETIMEDATETIMEGEOGRAPHYJSONINTERVALRANGE

The field type.

io.kestra.plugin.gcp.bigquery.models.UserDefinedFunction

The type of user defined function. string

Possible Values

INLINEFROM_URI

Type of UserDefinedFunction string

If type is UserDefinedFunction.Type.INLINE, this method returns a code blob. If type is UserDefinedFunction.Type.FROM_URI, the method returns a Google Cloud Storage URI (e.g. gs://bucket/path).

io.kestra.plugin.gcp.bigquery.models.ExternalTableDefinition

autodetect booleanstring

Whether automatic detection of schema and format options should be performed.

compression string

The compression type of the data source.

formatType string

Possible Values

CSVJSONBIGTABLEDATASTORE_BACKUPAVROGOOGLE_SHEETSPARQUETORC

The source format, and possibly some parsing options, of the external data.

ignoreUnknownValues booleanstring

Whether BigQuery should allow extra values that are not represented in the table schema.

If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.

maxBadRecords integerstring

The maximum number of bad records that BigQuery can ignore when reading data.

If the number of bad records exceeds this value, an invalid error is returned in the job result.

sourceUris array

SubType string

The fully-qualified URIs that point to your data in Google Cloud Storage.

Each URI can

contain one '*' wildcard character that must come after the bucket's name. Size limits related
to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size
across all URIs.

com.google.cloud.bigquery.StandardTableDefinition-StreamingBuffer

estimatedBytes integer

estimatedRows integer

oldestEntryTime integer

io.kestra.plugin.gcp.bigquery.models.Field-Output

description string

mode string

Possible Values

NULLABLEREQUIREDREPEATED

name string

policyTags PolicyTags-Output

subFields array

SubType

type string

Possible Values

BOOLINT64FLOAT64NUMERICBIGNUMERICSTRINGBYTESSTRUCTARRAYTIMESTAMPDATETIMEDATETIMEGEOGRAPHYJSONINTERVALRANGE

io.kestra.plugin.gcp.bigquery.models.ViewDefinition-Output

query string

viewUserDefinedFunctions array

SubType

io.kestra.plugin.gcp.bigquery.models.TimePartitioning-Output

expiration string

Format duration

field string

requirePartitionFilter boolean

type string

Possible Values

DAYHOURMONTHYEAR

io.kestra.plugin.gcp.bigquery.models.UserDefinedFunction-Output

content string

type string

Possible Values

INLINEFROM_URI

io.kestra.plugin.gcp.bigquery.models.ExternalTableDefinition-Output

autodetect boolean

compression string

formatType string

Possible Values

CSVJSONBIGTABLEDATASTORE_BACKUPAVROGOOGLE_SHEETSPARQUETORC

ignoreUnknownValues boolean

maxBadRecords integer

sourceUris array

SubType string

io.kestra.plugin.gcp.bigquery.models.StandardTableDefinition-Output

clustering array

SubType string

rangePartitioning RangePartitioning-Output

streamingBuffer StandardTableDefinition-StreamingBuffer

timePartitioning TimePartitioning-Output

​Create​Table

CreateTable