key: service

description: Service plugin settings


description: User defined services

type: object


key: otel

description: OpenTelemetry configuration

type: boolean

type: string

enum: zipkin stdout stderr otlp jaeger jaeger_agent

enum: http grpc

default: RoadRunner


key: insecure

description: Use insecure endpoint


key: compress

description: Use gzip compressor


key: exporter

description: Provides functionality to emit telemetry to consumers


description: Used for the http client to override the default URL


key: endpoint

description: Consumer's endpoint


key: client

description: Client to send the spans


description: User's service name


description: User's service version


key: headers

description: User defined headers


key: address

description: Host and port to listen on


description: Maximal incoming request size in megabytes. Zero means no limit

type: integer


description: Send raw body (unescaped) to the PHP worker for the application/x-www-form-urlencoded content type


description: HTTP access logs


key: middleware

description: Middleware for the http plugin, order is important

type: array

enum: headers gzip static sendfile http_metrics cache proxy_ip_parser otel


description: Allow incoming requests only from the following subnets


key: cache

description: RFC 7234 Souin cache: https://github.com/darkweak/souin/tree/master/plugins/roadrunner

enum: fastly cloudflare akamai varnish

enum: soft hard

default: GET POST

default: roadrunner

default: no-store

default: INFO

enum: DEBUG INFO WARN DPANIC PANIC ERROR FATAL


key: api

description: The cache-handler API cache management


key: basepath

description: Default route basepath for every additional APIs to avoid conflicts with existing routes


key: prometheus

description: Prometheus exposed metrics


key: souin

description: Souin listing keys and cache management


description: cache keys configuration


key: cdn

description: If Souin is set after a CDN fill these information


description: Your provider API key if mandatory


key: provider

description: The provider placed before Souin (e.g. fastly, cloudflare, akamai, varnish)


key: strategy

description: The strategy to purge the CDN cache based on tags (e.g. soft, hard)


key: dynamic

description: If true, you'll be able to add custom keys than the ones defined under the surrogate_keys key


description: Override the cache name to use in the Cache-Status header


key: distributed

description: Use Olric or Etcd distributed storage


description: Default headers concatenated in stored keys


key: etcd

description: If distributed is set to true, you'll have to define either the etcd or olric section


key: configuration

description: Configure directly the Etcd client


key: endpoints

description: Define multiple endpoints


key: olric


key: url

description: Olric server


key: regex

description: Regex to exclude from cache


key: stale

description: Stale duration


key: timeout

description: Timeout configuration


key: backend

description: Backend timeout before returning an HTTP unavailable response


key: cache

description: Cache provider (badger, etcd, nutsdb, olric, depending the configuration you set) timeout before returning a miss


key: ttl

description: Default TTL


description: Set default value for Cache-Control response header if not set by upstream


description: Logs verbosity


key: dir

description: Directory for file uploads. Empty value means to use $TEMP based on your OS


key: forbid

description: Deny files with the following extensions to upload

examples: .php .exe

default: .php .exe .bat


key: allow

description: Allow files with the following extensions to upload

examples: .html .go


key: headers

description: HTTP headers map


key: cors

description: Allows to control CORS headers


description: Controls 'Access-Control-Allow-Origin' header value


description: Controls 'Access-Control-Allow-Headers' header value


description: Controls 'Access-Control-Allow-Methods' header value


description: Controls 'Access-Control-Allow-Credentials' header value


description: Controls 'Access-Control-Expose-Headers' header value


description: Controls 'Access-Control-Max-Age' header value (in seconds)


key: request

description: Automatically add headers to every request passed to PHP


key: response

description: Automatically add headers to every response


key: static

description: Static assets serving settings

examples: .php .htaccess


description: Path to the directory with static assets


description: File extensions that should not be served


description: File extensions which should be served


description: Turn on etag computation for the static file


key: weak

description: Use a weak generator (/W), it uses only filename to generate a CRC32 sum


description: Custom headers for the static files


key: pool

description: Workers pool settings


key: ssl

description: SSL (Secure Sockets Layer) settings

required: domains email

required: address cert key


key: address

description: Host and port to listen on


key: acme

description: ACME certificates provider (Let's encrypt)


description: Directory to use as a certificate/pk, account info storage


key: email

description: User email, used to create LE account


description: Alternate port for the http challenge. Challenge traffic should be redirected to this port if overridden.


description: Alternate port for the tls-alpn-01 challenge. Challenge traffic should be redirected to this port if overridden.


description: Challenge types


description: Use production or staging endpoint. NOTE, try to use staging endpoint to make sure, that everything works correctly.


key: domains

description: List of your domains to obtain certificates


key: redirect

description: Automatic redirect from http to https schema


key: cert

description: Path to the cert file


key: key

description: Path to the cert key file


description: Path to the root certificate authority file


description: Client auth type


key: fcgi

description: FastCGI frontend support


description: FastCGI connection DSN. Supported TCP and Unix sockets. An empty value disables this


description: HTTP/2 settings


description: HTTP/2 over non-encrypted TCP connection using H2C


description: Maximal concurrent streams count


key: redis

description: Redis section.


key: metrics

description: Application metrics in Prometheus format (docs: https://roadrunner.dev/docs/beep-beep-metrics)

enum: histogram gauge counter summary

type: number


description: Prometheus client address (path /metrics added automatically).


key: collect

description: Application-specific metrics (published using an RPC connection to the server)


key: help

description: Help message


key: labels

description: Metrics labels


key: status

description: Health check endpoint (docs: https://roadrunner.dev/docs/beep-beep-health). If response code is 200 - it means at least one worker ready to serve requests. 500 - there are no workers ready to service requests.


description: Host and port to listen on (eg.: `127.0.0.1:2114`). Use the following URL: http://127.0.0.1:2114/health?plugin=http. Multiple plugins must be separated using & - http://127.0.0.1:2114/health?plugin=http&plugin=rpc where http and rpc are active (connected) plugins.


description: Response status code if a requested plugin not ready to handle requests. Valid for both /health and /ready endpoints


key: reload

description: Automatically detect PHP file changes and reload connected services

examples: .php .json

examples: .php .go .md


key: interval

description: Sync interval


key: patterns

description: Global patterns to sync


key: services

description: List of included for sync services (this is a map, where key name is a plugin name)


key: dirs

description: Directories to sync. If recursive is set to true, recursive sync will be applied only to the directories in 'dirs' section. Dot (.) means 'current working directory'


key: recursive

description: Recursive search for file patterns to add


key: ignore

description: Ignored folders


description: Service specific file pattens to sync


key: amqp

description: AMQP jobs driver


key: addr

description: AMQP Uri to connect to the rabbitmq server https://www.rabbitmq.com/uri-spec.html


key: key

description: Path to the key file


key: cert

description: Path to the cert file


description: Path to the CA certificate, defines the set of root certificate authorities that servers use if required to verify a client certificate. Used with the `client_auth_type` option.


key: beanstalk

description: Beanstalk jobs driver


description: Beanstalk server address


key: timeout

description: Beanstalk connect timeout


key: sqs

description: SQS jobs driver (https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html)


description: AccessKey ID


key: secret

description: Secret Access key


key: region

description: AWS Region


description: AWS Session token


key: endpoint

description: AWS SQS endpoint to connect


key: jobs

description: JOBS plugin

enum: amqp sqs beanstalk boltdb memory nats kafka

default: default

default: roadrunner+uuid

default: direct

default: AllISRAcks

enum: NoAck LeaderAck AllISRAck

enum: none gzip snappy lz4 zstd

enum: AtEnd At AfterMilli AtStart Relative WithEpoch

default: default-stream


description: Number of threads which will try to obtain the job from the priority queue. Default is the number of workers+1


description: JOBS Push operation timeout in seconds


description: Size of the internal priority queue, if the internal PQ reach the max number of elements, the Push operation will be blocked


key: consume

description: list of pipelines to be consumed by the server automatically at the start, keep empty if you want to start consuming manually


key: pool

description: JOBS workers pool


key: pipelines

description: List of broker pipelines associated with the drivers. This option is not required since you can declare pipelines in the runtime. Pipeline driver should exist.


key: driver

description: JOBS plugin driver


key: config

description: driver configurations


key: priority

description: Pipeline priority. If the job pushed to the pipeline has priority set to 0, it will inherit the pipeline's priority


key: prefetch

description: Number of job to prefetch from the driver


key: permissions

description: Permissions for the boltdb database file


key: file

description: BoldDB file to create or DB to use


description: Consume all payloads, even not Job structured


key: queue

description: Queue name


key: exchange

description: Exchange name


description: Redial timeout (in seconds). How long to try to reconnect to the AMQP server


description: Durable exchange (rabbitmq option: https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchanges)


description: The consumer_id is identified by a string that is unique and scoped for all consumers on this channel


description: Auto-delete (exchange is deleted when last queue is unbound from it): https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchanges


description: Auto-delete (queue that has had at least one consumer is deleted when last consumer unsubscribes)


description: Exchange type


description: Routing key for the queue


key: exclusive

description: Declare a queue exclusive at the exchange


description: When multiple is true, this delivery and all prior unacknowledged deliveries on the same channel will be acknowledged. This is useful for batch processing of deliveries


description: Use rabbitmq mechanism to requeue the job on fail


Queue declare args, associated array (hashmap, key:val)


description: Auto create topic for the consumer/producer


description: Kafka producer options


description: Disable_idempotent disables idempotent produce requests, opting out of Kafka server-side deduplication in the face of reissued requests due to transient network problems. Idempotent production is strictly a win, but does require the IDEMPOTENT_WRITE permission on CLUSTER (pre Kafka 3.0), and not all clients can have that permission.


description: Sets the required acks for produced records


description: The maximum duration in seconds the broker will wait the receipt of the number of required_acks.


description: The level of compression to use on messages.


description: If enabled, the producer will ensure that exactly one copy of each message is written.


description: group_options sets the consumer group for the client to join and consume in. This option is required if using any other group options.


description: Kafka group ID


description: Switches the client to block rebalances whenever you poll


description: Kafka consumer options


key: topics

description: List of the topics to consume. Regex also supported


description: consume_regexp sets the client to parse all topics passed to `topics` as regular expressions. When consuming via regex, every metadata request loads *all* topics, so that all topics can be passed to any regular expressions. Every topic is evaluated only once ever across all regular expressions; either it permanently is known to match, or is permanently known to not match.


description: consumer_offset sets the offset to start consuming from, or if OffsetOutOfRange is seen while fetching, to restart consuming from.


key: type

description: Partition offset type


key: value

description: Value for the: At, AfterMilli, Relative and WithEpoch offsets


description: Topic to consume


description: Partition number


description: Pipeline priority. If the job pushed to the pipeline has priority set to 0, it will inherit the pipeline's priority


description: Number of job to prefetch from the driver until ACK/NACK


description: Number of job to prefetch from the driver


description: Consume all payloads, even not Job structured


description: Beanstalk internal tube priority


key: tube

description: Tube name


If no job is available before this timeout has passed, Reserve returns a ConnError recording ErrTimeout


description: Get queue URL instead of declaring it


description: The duration (in seconds) for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than WaitTimeSeconds. If no messages are available and the wait time expires, the call returns successfully with an empty list of messages.


description: The duration (in seconds) for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than WaitTimeSeconds. If no messages are available and the wait time expires, the call returns successfully with an empty list of messages


description: Queue name


key: subject

description: NATS subject


key: stream

description: NATS stream


description: The consumer will only start receiving messages that were created after the consumer was created


description: Consumer rate-limiter in bytes https://docs.nats.io/jetstream/concepts/consumers#ratelimit


description: Delete the stream when after pipeline was stopped


description: Delete message from the stream after successful acknowledge


key: tcp

description: Plugin to handle RAW TCP packets, available since RR 2.6.0


key: servers

description: TCP servers to allocate


description: User defined TCP servers


key: pool

description: PHP static workers pool


key: grpc

description: GRPC plugin


key: listen

description: GRPC address to listen


key: proto

description: Proto file to use, multiply files supported [SINCE 2.6]. As of [v2023.1.4], wilcards are allowed in the proto field


key: tls

description: GRPC TLS configuration


key: key

description: Path to the key file


key: cert

description: Path to the certificate


description: Path to the CA certificate


description: Client auth type


description: Maximum send message size


description: Maximum receive message size


MaxConnectionIdle is a duration for the amount of time after which an idle connection would be closed by sending a GoAway. Idleness duration is defined since the most recent time the number of outstanding RPCs became zero or the connection establishment


MaxConnectionAge is a duration for the maximum amount of time a connection may exist before it will be closed by sending a GoAway. A random jitter of +/-10% will be added to MaxConnectionAge to spread out connection storms


MaxConnectionAgeGrace is an additive period after MaxConnectionAge after which the connection will be forcibly closed


After a duration of this time if the server doesn't see any activity it pings the client to see if the transport is still alive. If set below 1s, a minimum value of 1s will be used instead


key: timeout

description: After having pinged for keepalive check, the server waits for a duration of Timeout and if no activity is seen even after that the connection is closed


description: GRPC workers pool


key: fileserver

description: [SINCE 2.6] File server to serve static files


key: prefix

description: HTTP prefix


key: root

description: Directory to serve


key: compress

description: When set to true, the server tries minimizing CPU usage by caching compressed files


description: Expiration duration for inactive file handlers. Units: seconds. Use negative number to disable


description: The value for the Cache-Control HTTP-header. Units: seconds


description: Enable range requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests


key: BoltDB

description: BoltDB config section


key: file

description: file name for the db


key: permission

description: Access permission for the DB file.


key: interval

description: TTL keys check interval in seconds. It's safe to use 1 second here, but can be a little costly to performance


key: Memcached

description: In-memory config section


key: addr

description: Address of the memcached node


key: Redis

description: Redis config section


key: addrs

description: Redis server addresses


key: db

description: Redis db number


dial timeout


key: Memory

description: In-memory config section


key: Service

description: User defined service


key: command

description: Command to execute. Can be any command here which can be executed.


key: env

description: Environment variables for the process


description: Timeout for the process stop operation


description: Number of copies (processes) to start per command


description: Allowed time before stop


description: Remain process after exit. In other words, restart process after exit with any exit code


description: Number of seconds to wait before process restart


key: WorkersPool

description: Static pool with PHP workers


key: debug

description: Pool debug mode. Worker will be created right before RR passes request to it


description: Command to use for the pool. Will override the server's command


description: How many worker processes will be started. Zero (or nothing) means the number of logical CPUs


description: Maximal count of worker executions. Zero (or nothing) means no limit


Timeout for worker allocation. Zero means the default limit - 60s


Timeout for the pool.Reset operation (./rr reset). Zero means the default limit - 60s


Timeout for worker destroying before process killing. Zero means the default limit - 60s


key: supervisor

description: Supervisor is used to control http workers


How often to check the state of the workers


key: ttl

description: Maximum time worker is allowed to live (soft limit). Zero means no limit


How long worker can spend in IDLE mode after first using (soft limit). Zero means no limit


description: Maximal worker memory usage in megabytes (soft limit). Zero means no limit


Maximal job lifetime (hard limit). Zero means no limit


key: TCPServers

description: TCP server


description: Address to listen


key: delimiter

description: Data packets delimiter. Every send should end either with EOF or with the delimiter


description: Chunks that RR uses to read the data. In MB. If you expect big payloads on a TCP server, to reduce `read` syscalls, would be a good practice to use a fairly big enough buffer


key: Duration

description: Time duration


key: HostAndPortWithTCP

description: Host and port with tcp:// prefix


key: HostAndPort

description: Host and port


key: LogMode

description: Logging mode

enum: development production raw


key: LogLevel

description: Logging level

enum: debug info warn error panic


key: LogEncoding

description: Encoding format

enum: console json


key: Hashmap

description: Hashmap


key: Bucket

description: Hashmap with floats


description: NATS jobs driver


key: addr

description: NATS server address


description: Kafka jobs driver


key: brokers

description: Kafka broker addresses


key: sasl

description: Mechanism used for the authentication


key: mechanism

description: Kafka brokers


key: username

description: Username for authentication


key: password

description: Password for authentication


key: nonce

description: Optional for the SHA auth types. Empty by default


description: If true, suffixes the tokenauth=true extra attribute to the initial authentication message. Set this to true if the user and pass are from a delegation token. Optional for the SHA auth types. Empty by default


key: zid

description: Zid is an optional authorization ID to use in authenticating


description: AWS Access key ID


description: AWS Secret Access key ID


description: SessionToken, if non-empty, is a session / security token to use for authentication. See the following link for more details: https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html


description: UserAgent is the user agent to for the client to use when connecting to Kafka, overriding the default franz-go/<runtime.Version()>/<hostname>. Setting a UserAgent allows authorizing based on the aws:UserAgent condition key; see the following link for more details: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-useragent


description: Boltdb jobs driver


key: standalone

description: Set to true for old standalone roles, or false for new collection roles.


key: full

description: Json schema version for the test unit configuration file. Acceptance criteria: equal major version, less/equal minor and patch version.


description: Include a different file (similar to #include of the c preprocessor). The order of includes does not matter. Circular / multiple includes are resolved correctly.


description: Miscellaneous test unit information can be defined. After the execution they are available in the test report.

prename: Max

surname: S.

department: Department A

prename: Linda

department: Department B

SUT

customerName: Customer A

caption: This is a caption

Software Under Test

CustomerName: My customer


key: caption

description: Caption of the test unit.


key: description

description: Description of the test unit.


key: version

description: Version of the test unit.


key: test-designer

description: Information on the test developer(s). A sequence of key-value pairs that can be freely selected.


key: additional-information

description: Additional key-value pairs. One key 'name' is required, the other keys can be freely selected.


key: examples

description: This is a description

caption: This is a caption


description: Source artifacts of the test unit. Absolute and relative paths are allowed. Relative path specifications are resolved relative to the vtestunit.yaml file.


key: source-file-path

description: Path to a source file (.can/.canencr/.vtesttree.yaml).


key: capl-library-path

description: Path to a CAPL library file (.dll).


key: modeling-library-path

description: Path to a modeling library file (.vmodule).


description: Schema to validate Windows CE JSON files given as input to DL-Config to generate Scan2Deploy barcodes

default: \FlashDisk

enum: stay close warm-boot cold-boot clean-boot

default: cold-boot

enum: active-sync-usb active-sync-com usblan mass-storage

default: active-sync-usb

default: datalogic

default: tsunami

default: open

enum: none peap tls ttls fast leap

enum: password token certificate unknown

default: password

enum: user logon profile auto unknown

default: user

enum: eap-tls eap-MsChapV2 eap-gtc

default: eap-tls

enum: http https

default: http


key: global

description: Section used to configure application scoped settings


key: target-path

description: Base destination folder where archive/folder will be inflated


key: action

description: Specifies final action performed by the application at the end of the staging process


key: ping-timeout

description: The timeout value used when attempting to reach the host server


key: settings

description: Section used to configure inner device settings


key: usb-function

description: USB communication function to use


key: network

description: Section used to configure device Wi-Fi network


key: profile

description: "Friendly" name to associate with the network connection


key: essid

description: Wireless network ESSID


key: mode

description: Wireless connection mode


key: mode-key

description: Wireless network key for 'wep-40', 'wep-104', 'wpa-psk', and 'wpa2-psk' networks


key: index

description: Wireless network key index for 'wep-40' and 'wep-104' networks


key: eap-method

description: Configures the EAP authentication method


key: eap-username

description: Username for the EAP method


key: eap-password

description: Password for EAP method


key: credentials-type

description: Indicates the EAP credentials type


key: credentials-source

description: Indicates the EAP credentials source


key: validate-server

description: Indicates if the server certificate should be validated


key: connect-to-server

description: Indicates if only the servers listed in the 'connect-to-server-names' list should be connected to


key: connect-to-server-names

description: Semi-colon separated list of servers that are valid to connect to during server certificate validation


key: server-cert-name

description: "Friendly" name of the server digital certificate that should be validated


key: outer-identity

description: The outer-identity of the EAP method


key: fast-reconnect

description: Indicate if fast reconnect should be enabled


key: inner-method

description: Inner EAP authentication method to use


key: auto-pac

description: Indicates if automatic PAC provisioning is used for an EAP-FAST network


key: pac-file-path

description: Full path to the PAC file for manual PAC provisioning


key: pac-aid-info

description: PAC authority identity (AID) information used to select PAC from a list of PACs on the device


key: deployment

description: Section used to download a ZIP archive from a server and inflate it to the 'target-path' folder


key: scheme

description: Deployment download scheme to use


key: host

description: Host name or internet protocol address of the server from which the resource is to be fetched


key: port

description: TCP/IP port to be used to contact the server


key: path

description: Path to server resource to download, complete with query string if needed


key: shell

description: Section for executable shell commands to be performed in the final staging phase


key: cmds

description: Each object in 'cmds' is a shell command to execute in the final phase of staging


key: app-name

description: Full path to the application to be executed


key: args

description: Command line arguments


description:
A crawler rule.
 - Anything prefixed as "HTML RegEx" will do Java-flavored pattern-matching on the page's raw (unmodified by javascript) HTML.

required: enabled name rule pattern

examples: cookieName cookieValue

format: regex

enum: DEEPDECRYPT REWRITE DIRECTHTTP FOLLOWREDIRECT SUBMITFORM

const: DEEPDECRYPT

const: REWRITE

const: DIRECTHTTP

const: SUBMITFORM

const: FOLLOWREDIRECT


key: enabled

description:
- Type: Boolean
- Applies to rule: ALL
- Purpose: Enables or disables this rule


key: cookies

description:
- Type: Array of length-2 Arrays
- Applies to rule: DIRECTHTTP, DEEPDECRYPT, SUBMITFORM or FOLLOWREDIRECT
- Purpose: A list of length-2 arrays in the form [["cookieName", "cookieValue"],[..., ...]...]. Here you can put in your personal cookies e.g. login cookies of websites which JD otherwise fails to parse.Also if "updateCookies" is enabled, JD will update these with all cookies it receives from the website(s) that match the "pattern" property.


key: updateCookies

description:
- Type: Boolean
- Applies to rule: DIRECTHTTP, DEEPDECRYPT, SUBMITFORM or FOLLOWREDIRECT
- Purpose: If the target websites returns new cookies, save these inside this rule and update this rule.


key: logging

description:
- Type: Boolean
- Applies to rule: ALL
- Purpose: Enable this for support purposes. Logs of your LinkCrawler Rules can be found in your JD install dir/logs/: LinkCrawlerRule.<RuleID>.log.0 and /LinkCrawlerDeep.*


key: maxDecryptDepth

description:
- Type: Integer
- Applies to rule: ALL
- Purpose: How many layers deep do should your rule crawl (e.g. rule returns URLs matching the same rule again recursively - how often is this chain allowed to happen?)


key: id

description:
- Type: Integer
- Applies to rule: ALL
- Purpose: Auto generated ID of the rule. Normally leave this blank and JD2 will autoinsert.


key: name

description:
- Type: String
- Applies to rule: ALL
- Purpose: name of the rule.


key: pattern

description:
- Type: RegEx
- Applies to rule: ALL
- Purpose: defines on which URLs will this rule apply by matching it to the pattern


key: packageNamePattern

description:
- Type: HTML RegEx
- Applies to rule: DEEPDECRYPT
- Purpose: All URLs crawled by this rule will be grouped into the same package that is the HTML RegEx's first capture


key: passwordPattern

description:
- Type: HTML RegEx or null
- Applies to rule: DEEPDECRYPT
- Purpose: Matches against archive extraction password that may be found as text inside the page's (unmodified by javascript) HTML code. First returned capture must be the password.


key: formPattern

description:
- Type: HTML RegEx
- Applies to rule: DEEPDECRYPT
- Purpose:


key: deepPattern

description:
- Type: HTML RegEx or null
- Applies to rule: DEEPDECRYPT
- Purpose: Which URLs should this rule look for inside the page's (unmodified by javascript) HTML code. null (or blank) = auto scan and return all supported URLs found in HTML code.
Keep in mind that, if the url's found in the html are relative (e.g. starting with a slash / character instead of a protocol like http or a domain (website root) name) you WILL have to enclose the entire expression in quotes (outside of the parentheses that do capturing) like \"(...)\" AND enclose the part that would match the missing part of the url (i.e. from the protocol to the slash-or-other-character it begins with) in an 'OPTIONAL, NON CAPTURING' group (?:...)?. Resulting in the following pattern for pretty much every case \"((?:missing part of full url)?rest of url that is in the html)\"


key: rewriteReplaceWith

description:
- Type: String
- Applies to rule: REWRITE
- Purpose:Pattern for new URL, can use captures from "pattern" in the $1 form


description: Root schema of Datahub Ingestion

required: type config

descritption: Path to client's CA certificate for HTTPS communications

default: MetadataChangeEvent

default: MetadataChangeProposal

default: PROD

format: password

default: mysql+pymysql

default: localhost

default: powerbi


key: filename

description: Path to file to write to.


description: Path to server's CA certificate for verification of HTTPS communications


description: Experimental: Max parallelism for REST API calls


description: Retry HTTP request also on these status codes


key: server

description: URL of DataHub GMS endpoint.


description: Per-HTTP request timeout.


key: token

description: Bearer token used for authentication.


description: Extra headers which will be added to the request.


description: Disable SSL verification for HTTPS communications.


key: bootstrap

description: Kafka bootstrap URL.


description: Passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.SerializingProducer


description: URL of schema registry being used.


description: Passed to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.schema_registry.SchemaRegistryClient


key: MetadataChangeEvent

description: Overridden Kafka topic name for the MetadataChangeEvent


key: MetadataChangeProposal

description: Overridden Kafka topic name for the MetadataChangeProposal


description: Base configuration class for stateful ingestion for source configs to inherit from.


key: env

description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


key: options

description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Deprecated in favour of database_pattern.


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


key: domain

description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


key: username

description: username


key: password

description: password


description: MySQL host URL.


key: database

description: database (catalog)


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


Regex patterns for databases to filter in ingestion.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


description: host URL


description: If using, omit `service_name`.


key: scheme

description: Will be set automatically to default value.


description: Oracle service name. If using, omit `database`.


description: Add oracle database name to urn, default urn is schema.table


description: Any source that produces dataset urns in a single environment should inherit this class


description: The environment that all assets produced by this connector belong to


description: Windows account username


description: Windows account password


description: Workstation name


description: Power BI Report Server host URL


description: Alias for Power BI Report Server host URL


description: [deprecated] Not used


description: Report Virtual Directory URL name


description: Report Server Virtual Directory URL name


description: Whether ownership should be ingested


description: Ownership type of owner


description: Base configuration class for stateful ingestion for source configs to inherit from.

type: datahub

scheme: mysql+pymysql


description: Stateful Ingestion Config


description: MySQL connection config


Kafka connection config


description: If enabled, include all versions of each aspect. Otherwise, only include the latest version of each aspect.


description: Number of records to fetch from MySQL at a time


description: Name of MySQL table containing all versioned aspects


description: Name of kafka topic containing timeseries MCLs


description: Number of records to process before committing state


description: Whether to update createdon timestamp and kafka offset despite parse errors. Enable if you want to ignore the errors.


default: DAY

format: date-time

bucket_duration: DAY

default: bigquery

format: time-delta


description: Enable storing last profile timestamp in store.


description: When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.


description: When enabled, sql parser will run in isolated in a separate process. This can affect processing time but can protect from sql parser's mem leak.


description: Enable checking last lineage extraction date in store.


description: Size of the time window to aggregate usage stats.


description: Latest date of lineage/usage to consider. Default: Current time in UTC


description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.


description: Enable checking last usage timestamp in store.


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: Should we rate limit requests made to API.


description: Used to control number of API calls made per min. Only used when `rate_limit` is set to `True`.


description: If you are creating temp tables in a dataset with a particular prefix you can use this config to set the prefix for the dataset. This is to support workflows from before bigquery's introduction of temp tables. By default we use `_` because of datasets that begin with an underscore are hidden by default https://cloud.google.com/bigquery/docs/datasets#dataset-naming.


description: The regex pattern to match sharded tables and group as one table. This is a very low level config parameter, only change if you know what you are doing,


Regex patterns for project_id to filter in ingestion.


key: usage

description: Usage related configs


description: Generate usage statistic


description: Capture BigQuery table labels as DataHub tag


description: Capture BigQuery dataset labels as DataHub tag


Regex patterns for dataset to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


description: Whether `dataset_pattern` is matched against fully qualified dataset name `<project_id>.<dataset_name>`.


description: Whether to populate BigQuery Console url to Datasets/Tables


description: Include full payload into events. It is only for debugging and internal use.


description: Number of partitioned table queried in batch when getting metadata. This is a low level config property which should be touched with care. This restriction is needed because we query partitions system view which throws error if we try to touch too many tables.


description: Maximum number of columns to process in a table. This is a low level config property which should be touched with care. This restriction is needed because excessively wide tables can result in failure to ingest the schema.


description: [deprecated] Use project_id_pattern or project_ids instead.


description: Ingests specified project_ids. Use this property if you want to specify what projects to ingest or don't want to give project resourcemanager.projects.list to your service account. Overrides `project_id_pattern`.


description: [Advanced] The BigQuery project in which queries are executed. Will be passed when creating a job. If not passed, falls back to the project associated with the service account.


description: Use sql parser to resolve view/table lineage.


description: Sql parse view ddl to get lineage.


description: This parameter ignores the lowercase pattern stipulated in the SQLParser. NOTE: Ignored if lineage_use_sql_parser is False.


description: If enabled, generate column level lineage. Requires lineage_use_sql_parser to be enabled. This and `incremental_lineage` cannot both be enabled.


description: This flag enables the data lineage extraction from Data Lineage API exposed by Google Data Catalog. NOTE: This extractor can't build views lineage. It's recommended to enable the view's DDL parsing. Read the docs to have more information about: https://cloud.google.com/data-catalog/docs/concepts/about-data-lineage


description: Convert urns to lowercase.


description: Use the legacy sharded table urn suffix added.


description: The number of log item will be queried per page for lineage collection


key: credential

description: BigQuery credential informations


description: Additional options to pass to google.cloud.logging_v2.client.Client.


description: Option to enable/disable lineage generation. Is enabled by default.


description: Correction to pad start_time and end_time with. For handling the case where the read happens within our time range but the query completion event is delayed and happens after the configured end time.


description: A list of datasets that contain a table named cloudaudit_googleapis_com_data_access which contain BigQuery audit logs, specifically, those containing BigQueryAuditMetadata. It is recommended that the project of the dataset is also specified, for example, projectA.datasetB.


description: When configured, use BigQueryAuditMetadata in bigquery_audit_metadata_datasets to compute lineage information.


description: Whether to read date sharded tables or time partitioned tables when extracting usage from exported audit logs.


description: Useful for debugging lineage information. Set to True to see the raw lineage created internally.


description: Base configuration class for stateful ingestion for source configs to inherit from.

default: email

default: name


Okta Stateful Ingestion Config.


description: The location of your Okta Domain, without a protocol. Can be found in Okta Developer console. e.g. dev-33231928.okta.com


description: An API token generated for the DataHub application inside your Okta Developer Console. e.g. 00be4R_M2MzDqXawbWgfKGpKee0kuEOfX1RCQSRx00


description: Whether users should be ingested into DataHub.


description: Whether groups should be ingested into DataHub.


description: Whether group membership should be ingested into DataHub. ingest_groups must be True if this is True.


description: Which Okta User Profile attribute to use as input to DataHub username mapping. Common values used are - login, email.


description: A regex used to parse the DataHub username from the attribute specified in `okta_profile_to_username_attr`.


description: Which Okta Group Profile attribute to use as input to DataHub group name mapping.


description: A regex used to parse the DataHub group name from the attribute specified in `okta_profile_to_group_name_attr`.


description: Whether to ingest users in the DEPROVISIONED state from Okta.


description: Whether to ingest users in the SUSPENDED state from Okta.


description: The number of entities requested from Okta's REST APIs in one request.


description: Number of seconds to wait between calls to Okta's REST APIs. (Okta rate limits). Defaults to 10ms.


description: Okta filter expression (not regex) for ingesting users. Only one of `okta_users_filter` and `okta_users_search` can be set. See (https://developer.okta.com/docs/reference/api/users/#list-users-with-a-filter) for more info.


description: Okta search expression (not regex) for ingesting users. Only one of `okta_users_filter` and `okta_users_search` can be set. See (https://developer.okta.com/docs/reference/api/users/#list-users-with-search) for more info.


description: Okta filter expression (not regex) for ingesting groups. Only one of `okta_groups_filter` and `okta_groups_search` can be set. See (https://developer.okta.com/docs/reference/api/groups/#filters) for more info.


description: Okta search expression (not regex) for ingesting groups. Only one of `okta_groups_filter` and `okta_groups_search` can be set. See (https://developer.okta.com/docs/reference/api/groups/#list-groups-with-search) for more info.


description: Any source that is a primary producer of Dataset metadata should inherit this class

default: looker

format: directory-path


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


Pattern for providing dataset names to explores. Allowed variables are ['platform', 'env', 'project', 'model', 'name']


Pattern for providing browse paths to explores. Allowed variables are ['platform', 'env', 'project', 'model', 'name']


Pattern for providing dataset names to views. Allowed variables are ['platform', 'env', 'project', 'model', 'name']


Pattern for providing browse paths to views. Allowed variables are ['platform', 'env', 'project', 'model', 'name']


description: When enabled, attaches tags to measures, dimensions and dimension groups to make them more discoverable. When disabled, adds this information to the description of the column.


description: Default platform name. Don't change.


description: When enabled, extracts column-level lineage from Views and Explores


Reference to your git location. If present, supplies handy links to your lookml on the dataset entity page.


description: Required if not providing github configuration and deploy keys. A pointer to a local directory (accessible to the ingestion system) where the root of the LookML repo has been checked out (typically via a git clone). This is typically the root folder where the `*.model.lkml` and `*.view.lkml` files are stored. e.g. If you have checked out your LookML repo under `/Users/jdoe/workspace/my-lookml-repo`, then set `base_folder` to `/Users/jdoe/workspace/my-lookml-repo`.


description: A map of project_name to local directory (accessible to the ingestion system) or Git credentials. Every local_dependencies or private remote_dependency listed in the main project's manifest.lkml file should have a corresponding entry here. If a deploy key is not provided, the ingestion system will use the same deploy key as the main project.


description: A mapping of [Looker connection names](https://docs.looker.com/reference/model-params/connection-for-model) to DataHub platform, database, and schema values.


List of regex patterns for LookML models to include in the extraction.


List of regex patterns for LookML views to include in the extraction.


description: See note below.


description: See note below.


description: Required if you don't specify the `api` section. The project name within which all the model files live. See (https://docs.looker.com/data-modeling/getting-started/how-project-works) to understand what the Looker project name should be. The simplest way to see your projects is to click on `Develop` followed by `Manage LookML Projects` in the Looker application.


Populates the [TransportOptions](https://github.com/looker-open-source/sdk-codegen/blob/94d6047a0d52912ac082eb91616c1e7c379ab262/python/looker_sdk/rtl/transport.py#L70) struct for looker client


description: When extracting the view definition from a lookml file, the maximum number of characters to extract.


description: When enabled, only views that are reachable from explores defined in the model files are emitted


description: When enabled, field descriptions will include the sql logic for computed fields if descriptions are missing


description: When enabled, sql parsing will be executed in a separate process to prevent memory leaks.


description: When enabled, looker refinement will be processed to adapt an existing view.


description: Base configuration class for stateful ingestion for source configs to inherit from.

default: dev

default: public


description: Enable storing last profile timestamp in store.


description: When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.


description: When enabled, sql parser will run in isolated in a separate process. This can affect processing time but can protect from sql parser's mem leak.


description: Enable checking last lineage extraction date in store.


description: Size of the time window to aggregate usage stats.


description: Latest date of lineage/usage to consider. Default: Current time in UTC


description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.


description: Enable checking last usage timestamp in store.


description: Number of top queries to save to each table.


regex patterns for user emails to filter in usage.


description: Whether to display operational stats.


description: Whether to report read operational stats. Experimental.


description: Whether to format sql queries


description: Whether to ingest the top_n_queries.


description: Email domain of your organisation so users can be displayed on UI appropriately.


Common config for S3 lineage generation


description: The environment that all assets produced by this connector belong to


description: A holder for platform -> platform_instance mappings to generate correct dataset urns


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: username


description: password


description: host URL


description: database


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


description: The default schema to use if the sql parser fails to parse the schema with `sql_based` lineage collector


description: Whether table lineage should be ingested.


description: Whether lineage should be collected from copy commands


description: Generate usage statistic. email_domain config parameter needs to be set if enabled


description: Whether lineage should be collected from unload commands


description: Whether to capture lineage query parser errors with dataset properties for debugging


Which table lineage collector mode to use. Available modes are: [stl_scan_based, sql_based, mixed]


description: Whether `schema_pattern` is matched against fully qualified schema name `<database>.<schema>`.


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: The environment that all assets produced by this connector belong to


description: Size of the time window to aggregate usage stats.


description: Latest date of lineage/usage to consider. Default: Current time in UTC


description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.


description: Number of top queries to save to each table.


regex patterns for user emails to filter in usage.


description: Whether to display operational stats.


description: Whether to report read operational stats. Experimental.


description: Whether to format sql queries


description: Whether to ingest the top_n_queries.


Common config for S3 lineage generation


description: A holder for platform -> platform_instance mappings to generate correct dataset urns


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to SQLAlchemy's create_engine as kwargs.See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details.


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: username


description: password


description: host URL


description: database (catalog)


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


description: The default schema to use if the sql parser fails to parse the schema with `sql_based` lineage collector


description: Whether table lineage should be ingested.


description: Whether lineage should be collected from copy commands


description: Whether lineage should be collected from unload commands


description: Whether to capture lineage query parser errors with dataset properties for debuggings


Which table lineage collector mode to use. Available modes are: [stl_scan_based, sql_based, mixed]


description: Email domain of your organisation so users can be displayed on UI appropriately.


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: username


description: password


description: host URL


description: database (catalog)


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


Regex patterns for ml models to filter in ingestion.


description: Whether projections should be ingested.


description: Whether Models should be ingested.


description: If the source supports it, include view lineage to the underlying storage location.


description: Redash base URL.


description: Redash user API key.


regex patterns for dashboards to filter for ingestion.


regex patterns for charts to filter for ingestion.


description: Only ingest published dashboards and charts.


description: Limit on number of items to be queried at once.


description: Limit on number of pages queried for ingesting dashboards and charts API during pagination.


key: parallelism

description: Parallelism to use while processing.


description: See note below.


description: custom SQL parser. See note below for details.


description: Environment to use in namespace when constructing URNs.


description: Base configuration class for stateful ingestion for source configs to inherit from.

models: YES

sources: YES

seeds: YES

snapshots: YES

test_definitions: YES

test_results: YES

default: dbt:

default: PATCH


description: When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.


description: When enabled, sql parser will run in isolated in a separate process. This can affect processing time but can protect from sql parser's mem leak.


description: The instance of the platform that all assets produced by this recipe belong to


DBT Stateful Ingestion Config.


description: The platform that dbt is loading onto. (e.g. bigquery / redshift / postgres etc.)


description: The platform instance for the platform that dbt is operating on. Use this if you have multiple instances of the same platform (e.g. redshift) and need to distinguish between them.


description: Use model identifier instead of model name if defined (if not, default to model name).


description: Controls for enabling / disabling metadata emission for different dbt entities (models, test definitions, test results, etc.)


description: Prefix added to tags during ingestion.


regex patterns for dbt model names to filter in ingestion.


description: mapping rules that will be executed against dbt meta properties. Refer to the section below on dbt meta automated mappings.


description: mapping rules that will be executed against dbt column meta properties. Refer to the section below on dbt meta automated mappings.


description: mapping rules that will be executed against dbt query_tag meta properties. Refer to the section below on dbt meta automated mappings.


description: Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be "PATCH" or "OVERRIDE"


description: Whether or not to strip email id while adding owners using dbt meta actions.


description: When enabled, ownership info will be extracted from the dbt meta


description: Regex string to extract owner from the dbt node using the `(?P<name>...) syntax` of the [match object](https://docs.python.org/3/library/re.html#match-objects), where the group name must be `owner`. Examples: (1)`r"(?P<owner>(.*)): (\w+) (\w+)"` will extract `jdoe` as the owner from `"jdoe: John Doe"` (2) `r"@(?P<owner>(.*))"` will extract `alice` as the owner from `"@alice"`.


description: Prior to version 0.9.4.2, the assertion GUIDs did not include the environment. If you're using multiple dbt ingestion that are only distinguished by env, then you should set this flag to True.


description: When enabled, converts column URNs to lowercase to ensure cross-platform compatibility. If `target_platform` is Snowflake, the default is True.


description: When enabled, applies the mappings that are defined through the meta_mapping directives.


description: When enabled, applies the mappings that are defined through the `query_tag_mapping` directives.


description: Path to dbt manifest JSON. See https://docs.getdbt.com/reference/artifacts/manifest-json Note this can be a local file or a URI.


description: Path to dbt catalog JSON. See https://docs.getdbt.com/reference/artifacts/catalog-json Note this can be a local file or a URI.


description: Path to dbt sources JSON. See https://docs.getdbt.com/reference/artifacts/sources-json. If not specified, last-modified fields will not be populated. Note this can be a local file or a URI.


description: Path to output of dbt test run as run_results file in JSON format. See https://docs.getdbt.com/reference/artifacts/run-results-json. If not specified, test execution results will not be populated in DataHub.


When fetching manifest files from s3, configuration for aws connection details


Reference to your git location to enable easy navigation from DataHub to your dbt files.


description: Base configuration class for stateful ingestion for source configs to inherit from.

default: presto-on-hive


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: username


description: password


description: Host URL and port to connect to. Example: localhost:3306


description: database (catalog)


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


description: Where clause to specify what Presto views should be ingested.


description: Where clause to specify what Hive tables should be ingested.


description: Where clause to specify what Hive schemas should be ingested.


Regex patterns for hive/presto database to filter in ingestion. Specify regex to only match the database name. e.g. to match all tables in database analytics, use the regex 'analytics'


description: Name of the Hive metastore's database (usually: metastore). For backward compatibility, if this field is not provided, the database field will be used. If both the 'database' and 'metastore_db_name' fields are set then the 'database' field will be used to filter the hive/presto/trino database


key: mode

description: The ingested data will be stored under this platform. Valid options: ['hive', 'presto', 'presto-on-hive', 'trino']


description: Container Subtype name to be 'Database' or 'Catalog' Valid options: ['True', 'False']


description: Dataset Subtype name to be 'Table' or 'View' Valid options: ['True', 'False']


description: Add the Presto catalog name (e.g. hive) to the generated dataset urns. `urn:li:dataset:(urn:li:dataPlatform:hive,hive.user.logging_events,PROD)` versus `urn:li:dataset:(urn:li:dataPlatform:hive,user.logging_events,PROD)`


description: By default, the connector overwrites properties every time. Set this to True to enable merging of properties with what exists on the server.


description: Simplify v2 field paths to v1 by default. If the schema has Union or Array types, still falls back to v2


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


PowerBI Stateful Ingestion Config.


description: PowerBI tenant identifier


Regex patterns to filter PowerBI workspaces in ingestion


description: A mapping of PowerBI datasource's server i.e host[:port] to Data platform instance. :port is optional and only needed if your datasource server is running on non-standard port.For Google BigQuery the datasource's server is google bigquery project name


description: Azure app client identifier


description: Azure app client secret


description: timeout for PowerBI metadata scanning


description: batch size for sending workspace_ids to PBI, 100 is the limit


description: Highly recommend changing this to True, as you can have the same workspace nameTo maintain backward compatability, this is set to False which uses workspace name


description: Whether ownership should be ingested. Admin API access is required if this setting is enabled. Note that enabling this may overwrite owners that you've added inside DataHub's web application.


description: Whether reports should be ingested


key: ownership

description: Configure how is ownership ingested


description: Get only recently modified workspaces based on modified_since datetime '2023-02-10T00:00:00.0000000Z', excludePersonalWorkspaces and excludeInActiveWorkspaces limit to last 30 days


description: Whether to ingest PBI Dashboard and Tiles as Datahub Dashboard and Chart


description: Whether to ingest PBI Dataset Table columns and measures


description: Whether lineage should be ingested between X and Y. Admin API access is required if this setting is enabled


description: Whether to extract endorsements to tags, note that this may overwrite existing tags. Admin API access is required is this setting is enabled


Filter and ingest datasets which are 'Certified' or 'Promoted' endorsement. If both are added, dataset which are 'Certified' or 'Promoted' will be ingested . Default setting allows all dataset to be ingested


description: Extract workspaces to DataHub containers


description: PBI tables will be grouped under a Datahub Container, the container reflect a PBI Dataset


description: Whether PowerBI native query should be parsed to extract lineage


description: Whether to convert the PowerBI assets urns to lowercase


description: Whether to convert the urns of ingested lineage dataset to lowercase


description: Retrieve metadata using PowerBI Admin API only. If this is enabled, then Report Pages will not be extracted. Admin API access is required if this setting is enabled


description: Whether to extract datasets not used in any PowerBI visualization


description: Any source that connects to a platform should inherit this class


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


key: host

description: The elastic search host URI.


description: The username credential.


description: The password credential.


description: Whether to use SSL for the connection or not.


description: Whether to verify SSL certificates.


description: Path to a certificate authority (CA) certificate.


description: Path to the file containing the private key and the certificate, or cert only if using client_key.


description: Path to the file containing the private key if using separate cert and key files.


description: Use hostname verification if not False.


description: Verify the supplied certificate fingerprint if not None.


description: There are cases where an enterprise would have multiple elastic search clusters. One way for them to manage is to have a single endpoint for all the elastic search clusters and use url_prefix for routing requests to different clusters.


regex patterns for indexes to filter in ingestion.


description: Ingests ES index templates if enabled.


The regex patterns for filtering index templates to ingest.


description: Base configuration class for stateful ingestion for source configs to inherit from.

default: snowflake

default: skip


key: classification

description: For details, refer [Classification](../../../../metadata-ingestion/docs/dev_guides/classification.md).


description: Enable storing last profile timestamp in store.


description: Size of the time window to aggregate usage stats.


description: Latest date of lineage/usage to consider. Default: Current time in UTC


description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.


description: Enable checking last usage timestamp in store.


description: Number of top queries to save to each table.


regex patterns for user emails to filter in usage.


description: Whether to display operational stats.


description: Whether to report read operational stats. Experimental.


description: Whether to format sql queries


description: Whether to ingest the top_n_queries.


description: Email domain of your organisation so users can be displayed on UI appropriately.


description: Whether to apply view's usage to its base tables. If set to True, usage is applied to base tables only.


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: Snowflake username.


description: Snowflake password.


description: Private key in a form of '-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n' if using key pair authentication. Encrypted version of private key will be in a form of '-----BEGIN ENCRYPTED PRIVATE KEY-----\nencrypted-private-key\n-----END ECNCRYPTED PRIVATE KEY-----\n' See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html


description: The path to the private key if using key pair authentication. Ignored if `private_key` is set. See: https://docs.snowflake.com/en/user-guide/key-pair-auth.html


description: Password for your private key. Required if using key pair authentication with encrypted private key.


oauth configuration - https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-with-oauth


description: The type of authenticator to use when connecting to Snowflake. Supports "DEFAULT_AUTHENTICATOR", "OAUTH_AUTHENTICATOR", "EXTERNAL_BROWSER_AUTHENTICATOR" and "KEY_PAIR_AUTHENTICATOR".


description: Snowflake account identifier. e.g. xy12345,  xy12345.us-east-2.aws, xy12345.us-central1.gcp, xy12345.central-us.azure, xy12345.us-west-2.privatelink. Refer [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#format-2-legacy-account-locator-in-a-region) for more details.


key: warehouse

description: Snowflake warehouse.


key: role

description: Snowflake role.


description: If enabled, populates the snowflake table-to-table and s3-to-snowflake table lineage. Requires appropriate grants given to the role and Snowflake Enterprise Edition or above.


description: If enabled, populates the snowflake view->table and table->view lineages. Requires appropriate grants given to the role, and include_table_lineage to be True. view->table lineage requires Snowflake Enterprise Edition or above.


description: Connect args to pass to Snowflake SqlAlchemy driver


description: If enabled, populates the snowflake usage statistics. Requires appropriate grants given to the role.


description: If enabled, populates the snowflake technical schema and descriptions.


description: Populates table->table and view->table column lineage. Requires appropriate grants given to the role and the Snowflake Enterprise Edition or above.


description: Populates view->view and table->view column lineage.


description: Optional. Allowed values are `without_lineage`, `with_lineage`, and `skip` (default). `without_lineage` only extracts tags that have been applied directly to the given entity. `with_lineage` extracts both directly applied and propagated tags, but will be significantly slower. See the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/object-tagging.html#tag-lineage) for information about tag lineage/propagation.


description: Whether to populate Snowsight url for Snowflake Objects


description: Whether `schema_pattern` is matched against fully qualified schema name `<catalog>.<schema>`.


description: Whether to use the legacy lineage computation method. By default, uses new optimised lineage extraction method that requires less ingestion process memory. Table-to-view and view-to-view column-level lineage are not supported with the legacy method.


description: Whether to validate upstream snowflake tables against allow-deny patterns


List of regex patterns for tags to include in ingestion. Only used if `extract_tags` is enabled.


description: [Advanced] Regex patterns for temporary tables to filter in lineage ingestion. Specify regex to match the entire table name in database.schema.table format. Defaults are to set in such a way to ignore the temporary staging tables created by known ETL tools. Not used if `use_legacy_lineage_method=True`


description: Format user urns as an email, if the snowflake user's email is set. If `email_domain` is provided, generates email addresses for snowflake users with unset emails, based on their username.


description: Base configuration class for stateful ingestion for source configs to inherit from.

default: db


Superset Stateful Ingestion Config.


description: Superset host URL.


description: optional URL to use in links (if `connect_uri` is only for ingestion)


description: Superset username.


description: Superset password.


key: provider

description: Superset provider.


key: env

description: Environment to use in namespace when constructing URNs

default: PROD


description: Can be used to change mapping for database names in superset to what you have in datahub


description: Enable storing last profile timestamp in store.


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: Size of the time window to aggregate usage stats.


description: Latest date of lineage/usage to consider. Default: Current time in UTC


description: Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on `bucket_duration`). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.


description: Number of top queries to save to each table.


regex patterns for user emails to filter in usage.


description: Whether to display operational stats.


description: Whether to report read operational stats. Experimental.


description: Whether to format sql queries


description: Whether to ingest the top_n_queries.


Unity Catalog Stateful Ingestion Config.


key: token

description: Databricks personal access token


description: Databricks workspace url. e.g. https://my-workspace.cloud.databricks.com


description: Name of the workspace. Default to deployment name present in workspace_url


Regex patterns for catalogs to filter in ingestion. Specify regex to match the full `metastore.catalog` name.


Regex patterns for schemas to filter in ingestion. Specify regex to the full `metastore.catalog.schema` name. e.g. to match all tables in schema analytics, use the regex `^mymetastore\.mycatalog\.analytics$`.


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in `catalog.schema.table` format. e.g. to match all tables starting with customer in Customer catalog and public schema, use the regex `Customer\.public\.customer.*`.


key: domain

description: Attach domains to catalogs, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Option to enable/disable lineage generation.


description: Option to enable/disable ownership generation for metastores, catalogs, schemas, and tables.


description: Option to enable/disable lineage generation. Currently we have to call a rest call per column to get column level lineage due to the Databrick api which can slow down ingestion.


description: Generate usage statistics.


key: profiling

description: Data profiling configuration


key: path

description: Path to Feast repository


description: Path to the `feature_store.yaml` file used to configure the feature store


key: environment

description: Environment to use when constructing URNs


description:
Any non-Dataset source that produces lineage to Datasets should inherit this class.
e.g. Orchestrators, Pipelines, BI Tools etc.

required: token password


description: The environment that all assets produced by this connector belong to


description: A holder for platform -> platform_instance mappings to generate correct dataset urns


description: Mode host URL.


description: Mode user token.


key: password

description: Mode password for authentication.


description: Default schema to use when schema is not provided in an SQL query


description: Use username for owner URN instead of Email


Retry/Wait settings for Mode API to avoid "Too many Requests" error. See Mode API Options below


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: The instance of the platform that all assets produced by this recipe belong to


key: options

description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for schemas to filter in ingestion. Specify regex to only match the schema name. e.g. to match all tables in schema analytics, use the regex 'analytics'


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


key: platform

description: Name of platform being ingested, used in constructing URNs.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls


default: hana+hdbcli


key: username

description: username


description: password


key: database

description: database (catalog)


description: [Deprecated] Alias to apply to database when ingesting.


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


default: owner


description: The environment that all assets produced by this connector belong to


Iceberg Stateful Ingestion Config.


key: adls

description: [Azure Data Lake Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) to crawl for Iceberg tables.  This is one filesystem type supported by this source and **only one can be configured**.


key: localfs

description: Local path to crawl for Iceberg tables. This is one filesystem type supported by this source and **only one can be configured**.


description: Maximum folder depth to crawl for Iceberg tables.  Folders deeper than this value will be silently ignored.


Regex patterns for tables to filter in ingestion.


description: Iceberg table property to look for a `CorpUser` owner.  Can only hold a single user value.  If property has no value, no owner information will be emitted.


description: Iceberg table property to look for a `CorpGroup` owner.  Can only hold a single group value.  If property has no value, no owner information will be emitted.


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: The instance of the platform that all assets produced by this recipe belong to


see Stateful Ingestion


description: The web URL for the cluster.


key: timeout

description: Timout setting, how long to wait for the Pulsar rest api to send data before giving up


description: The complete URL for a Custom Authorization Server. Mandatory for OAuth based authentication.


description: The application's client ID


description: The application's client secret


description: The access token for the application. Mandatory for token based authentication.


description: Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use.


List of regex patterns for tenants to include/exclude from ingestion. By default all tenants are allowed.


List of regex patterns for namespaces to include/exclude from ingestion. By default the functions namespace is denied.


List of regex patterns for topics to include/exclude from ingestion. By default the Pulsar system topics are denied.


description: Extract each individual partitioned topic. e.g. when turned off a topic with 100 partitions will result in 100 Datasets.


key: tenants

description: Listing all tenants requires superUser role, alternative you can set a list of tenants you want to scrape using the tenant admin role


description: Domain patterns


description: Placeholder for OpenId discovery document


description: Any source that connects to a platform should inherit this class

default: delta-lake

const: delta-lake


description: The environment that all assets produced by this connector belong to


description: Path to table (s3 or local file system). If path is not a delta table path then all subfolders will be scanned to detect and ingest delta tables.


description: If set, delta-tables will be searched at location '<base_path>/<relative_path>' and URNs will be created using relative_path only.


description: The platform that this source connects to


regex patterns for tables to filter in ingestion.


description: Number of previous version histories to be ingested. Defaults to 1. If set to -1 all version history will be ingested.


description: Whether DeltaTable should track files. Consider setting this to `False` for large delta tables, resulting in significant memory reduction for ingestion process.When set to `False`, number_of_files in delta table can not be reported.


description: Config to create a token and connect to Azure AD instance

default: userPrincipalName

default: displayName


description: The instance of the platform that all assets produced by this recipe belong to


Azure AD Stateful Ingestion Config.


description: Application ID. Found in your app registration on Azure AD Portal


description: Directory ID. Found in your app registration on Azure AD Portal


description: Client secret. Found in your app registration on Azure AD Portal


key: authority

description: The authority (https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-client-application-configuration) is a URL that indicates a directory that MSAL can request tokens from.


description: The token URL that acquires a token from Azure AD for authorizing requests.  This source will only work with v1.0 endpoint.


key: redirect

description: Redirect URI.  Found in your app registration on Azure AD Portal.


description: [Microsoft Graph API endpoint](https://docs.microsoft.com/en-us/graph/use-the-api)


description: Which Azure AD User Response attribute to use as input to DataHub username mapping.


description: A regex used to parse the DataHub username from the attribute specified in `azure_ad_response_to_username_attr`.


description: Which Azure AD Group Response attribute to use as input to DataHub group name mapping.


description: A regex used to parse the DataHub group name from the attribute specified in `azure_ad_response_to_groupname_attr`.


description: Whether users should be ingested into DataHub.


description: Whether groups should be ingested into DataHub.


description: Whether group membership should be ingested into DataHub. ingest_groups must be True if this is True.


description: This option is useful only when `ingest_users` is set to False and `ingest_group_membership` to True. As effect, only the users which belongs to the selected groups will be ingested.


regex patterns for users to filter in ingestion.


regex patterns for groups to include in ingestion.


description: If enabled, report will contain names of filtered users and groups.


description: Whether workunit ID's for groups should be masked to avoid leaking sensitive information.


description: Whether workunit ID's for users should be masked to avoid leaking sensitive information.


description: Base configuration class for stateful ingestion for source configs to inherit from.


key: env

description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


description: A map of domain names to allow deny patterns. Domains can be urn-based (`urn:li:domain:13ae4d85-d955-49fc-8474-9004c663a810`) or bare (`13ae4d85-d955-49fc-8474-9004c663a810`).


description: Provides the mapping for the `key` and the `value` schemas of a topic to the corresponding schema registry subject name. Each entry of this map has the form `<topic_name>-key`:`<schema_registry_subject_name_for_key_schema>` and `<topic_name>-value`:`<schema_registry_subject_name_for_value_schema>` for the key and the value schemas associated with the topic, respectively. This parameter is mandatory when the [RecordNameStrategy](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#how-the-naming-strategies-work) is used as the subject naming strategy in the kafka schema registry. NOTE: When provided, this overrides the default subject name resolution even when the `TopicNameStrategy` or the `TopicRecordNameStrategy` are used.


description: The fully qualified implementation class(custom) that implements the KafkaSchemaRegistryBase interface.


description: Disables warnings reported for non-AVRO/Protobuf value or key schemas if set.


description: Disables the utilization of the TopicRecordNameStrategy for Schema Registry subjects. For more information, visit: https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#handling-differences-between-preregistered-and-client-derived-schemas:~:text=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy


key: file

description: File path or URL to lineage file to ingest.


description: Whether we want to query datahub-gms for upstream data. False means it will hard replace upstream data for a given entity. True means it will query the backend for existing upstreams and include it in the ingestion run


default: postgres


description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.


Regex patterns for tables to filter in ingestion. Specify regex to match the entire table name in database.schema.table format. e.g. to match all tables starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns for views to filter in ingestion. Note: Defaults to table_pattern if not specified. Specify regex to match the entire view name in database.schema.view format. e.g. to match all views starting with customer in Customer database and public schema, use the regex 'Customer.public.customer.*'


Regex patterns to filter tables (or specific columns) for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered.


key: domain

description: Attach domains to databases, schemas or tables during ingestion using regex patterns. Domain key can be a guid like *urn:li:domain:ec428203-ce86-4db3-985d-5a8ee6df32ba* or a string like "Marketing".) If you provide strings, then datahub will attempt to resolve this name to a guid, and will error out if this fails. There can be multiple domain keys specified.


description: Whether views should be ingested.


description: Whether tables should be ingested.


description: If the source supports it, include table lineage to the underlying storage location.


description: username


description: password


description: host URL


description: database (catalog). If set to Null, all databases will be considered for ingestion.


description: [Deprecated] Alias to apply to database when ingesting.


key: scheme

description: database scheme


description: URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.


Regex patterns for databases to filter in ingestion. Note: this is not used if `database` or `sqlalchemy_uri` are provided.


description: Initial database used to query for the list of databases, when ingesting multiple databases. Note: this is not used if `database` or `sqlalchemy_uri` are provided.


description: Include table lineage for views


description: Base configuration class for stateful ingestion for source configs to inherit from.


description: List of PathSpec. See [below](#path-spec) the details about PathSpec


description: The environment that all assets produced by this connector belong to


description: The instance of the platform that all assets produced by this recipe belong to


key: credential

description: Google cloud storage [HMAC keys](https://cloud.google.com/storage/docs/authentication/hmackeys)


description: Maximum number of rows to use when inferring schemas for TSV and CSV files.


description: Number of files to list to sample for schema inference. This will be ignored if sample_files is set to False in the pathspec.


key: filename

description: File path or URL of CSV file to ingest.


description: Whether the new tags, terms and owners to be added will override the existing ones added only by this source or not. Value for this config can be "PATCH" or "OVERRIDE"


key: delimiter

description: Delimiter to use when parsing CSV


description: Delimiter to use when parsing array fields (tags, terms and owners)


description:
Any non-Dataset source that produces lineage to Datasets should inherit this class.
e.g. Orchestrators, Pipelines, BI Tools etc.


description: A holder for platform -> platform_instance mappings to generate correct dataset urns


description: Metabase host URL.


description: Metabase username.


key: password

description: Metabase password.


description: Database name map to use when constructing dataset URN.


description: Custom mappings between metabase database engines and DataHub platforms


description: Custom mappings between metabase database id and DataHub platform instance


description: Default schema name to use when schema is not provided in an SQL query


default: awsathena+rest

default: awsdatacatalog


key: options

description: Any options specified here will be passed to [SQLAlchemy.create_engine](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine) as kwargs.