标签：ase any tab port modified 模式 expressed ica mode
发现模式提供了一种描述tap 支持数据流的方式，使用了json schema 做为描述数据的结构以及每个数据流的
类型，发现模式的实现依赖tap 的数据源，有些taps 将硬编码每个流的模式，而其他的将连接到提供可用流的
描述的api，当运行发现模式时，tap 应该写如stdout 流列表，称为目录，每个条目包含关于流的一些基本信息和
描述流的json schema
发现模式下运行tap， 使用--discover
tap --config CONFIG --discover
我们可以在运行的时候将输出重定向到一个文件
tap --config CONFIG --discover > catalog.json
对于一些遗留的taps ，会使用properties.json 做为目录
JSON用于表示数据，因为它无处不在，可读，并且特别适用于将数据公开为JSON（如Web API）的大量源。但是，
JSON远非完美：
Stitch Target和Stitch API使用schema如下：
{
"type": [
"null",
"object"
],
"additionalProperties": false,
"properties": {
"id": {
"type": [
"null",
"string"
],
},
"name": {
"type": [
"null",
"string"
],
},
"date_modified": {
"type": [
"null",
"string"
],
"format": "date-time",
}
}
}
发现模式的输出应该是Tap支持的数据流列表。此JSON格式的列表称为目录。顶层是一个对象，其中一个被调用的键"streams"指向一个对象数组，
每个对象都有以下字段：
tap_stream_id 字符串 需要 流的唯一标识符。允许这与流的名称不同，以允许具有重复流名称的源。
schema 对象 需要 流的JSON模式。
table_name 字符串 可选的 对于数据库源，表的名称。
metadata 元数据数组 可选的 请参阅下面的元数据以获取解释
参考：
{
"streams": [
{
"tap_stream_id": "users",
"stream": "users",
"schema": {
"type": ["null", "object"],
"additionalProperties": false,
"properties": {
"id": {
"type": [
"null",
"string"
],
},
"name": {
"type": [
"null",
"string"
],
},
"date_modified": {
"type": [
"null",
"string"
],
"format": "date-time",
}
}
}
}
]
}
元数据是关联模式中节点的额外信息的首选机制。
应该通过tap 来写入和读取某些元数据。此元数据称为discoverable元数据。其他元数据将由其他系统（如UI）编写
，因此只能通过tap读取。这种类型的元数据称为non-discoverable元数据
参考的字段信息：
| Keyword | Tap Type | Discoverable? | Description |
|---|---|---|---|
selected |
any | non-discoverable | Either true or false. Indicates that this node in the schema has been selected by the user for replication. |
replication-method |
any | non-discoverable | Either FULL_TABLE, INCREMENTAL, or LOG_BASED. The replication method to use for a stream. |
replication-key |
any | non-discoverable | The name of a property in the source to use as a "bookmark". For example, this will often be an "updated-at" field or an auto-incrementing primary key (requires replication-method). |
view-key-properties |
database | non-discoverable | List of key properties for a database view. |
inclusion |
any | discoverable | Either available, automatic, or unsupported. available means the field is available for selection, and the tap will only emit values for that field if it is marked with "selected": true. automatic means that the tap will emit values for the field. unsupported means that the field exists in the source data but the tap is unable to provide it. |
selected-by-default |
any | discoverable | Either true or false. Indicates if a node in the schema should be replicated if a user has not expressed any opinion on whether or not to replicate it. |
valid-replication-keys |
any | discoverable | List of the fields that could be used as replication keys. |
schema-name |
any | discoverable | The name of the stream. |
forced-replication-method |
any | discoverable | Used to force the replication method to either FULL_TABLE or INCREMENTAL. |
table-key-properties |
database | discoverable | List of key properties for a database table. |
is-view |
database | discoverable | Either true or false. Indicates whether a stream corresponds to a database view. |
row-count |
database | discoverable | Number of rows in a database table/view. |
database-name |
database | discoverable | Name of database. |
sql-datatype |
database | discoverable | Represents the datatype of a database column. |
参考的数据格式
{
"metadata" : {
"selected" : true,
"some-other-metadata" : "whatever"
},
"breadcrumb" : ["properties", "some-field-name"]
}
上面的breadcrumb对象定义了到元数据所属节点的模式的路径。流的元数据将具有空的面包屑。
参考完整例子
{
"streams": [
{
"tap_stream_id": "users",
"stream": "users",
"schema": {
"type": ["null", "object"],
"additionalProperties": false,
"properties": {
"id": {
"type": [
"null",
"string"
],
},
"name": {
"type": [
"null",
"string"
],
},
"date_modified": {
"type": [
"null",
"string"
],
"format": "date-time",
}
}
},
"metadata": [
{
"metadata": {
"inclusion": "available",
"table-key-properties": ["id"],
"selected-by-default": true,
"valid-replication-keys": ["date_modified"],
"schema-name": "users",
},
"breadcrumb": []
},
{
"metadata": {
"inclusion": "automatic",
},
"breadcrumb": ["properties", "id"]
},
{
"metadata": {
"inclusion": "available",
"selected-by-default": true,
},
"breadcrumb": ["properties", "name"]
},
{
"metadata": {
"inclusion": "automatic",
},
"breadcrumb": ["properties", "date_modified"]
}
]
}
]
}
https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md
标签：ase any tab port modified 模式 expressed ica mode
原文地址：https://www.cnblogs.com/rongfengliang/p/10247300.html
