# DataX TDengineWriter

[简体中文](./tdenginewriter-CN.md) | English

## 1 Quick Introduction

The TDengineWriter plugin enables writing data to the target table of the TDengine database. At the bottom level, TDengineWriter connects TDengine through JDBC, executes insert statement /schemaless statement according to TDengine SQL syntax, and writes data to TDengine.

TDengineWriter can be used as a data migration tool for DBAs to import data from other databases into TDengine.



## 2 Implementation

TDengineWriter obtains the protocol data generated by Reader through DataX framework, connects to TDengine through JDBC Driver, executes insert statement /schemaless statement, and writes the data to TDengine.

In TDengine, table can be divided into super table, sub-table and ordinary table. Super table and sub-table include Colum and Tag. The value of tag column of sub-table is fixed value. (details please refer to: [data model](https://www.taosdata.com/docs/cn/v2.0/architecture#model))

The TDengineWriter can write data to super tables, sub-tables, and ordinary tables using the following methods based on the type of the table and whether the column parameter contains TBName:

1. Table is a super table and column specifies tbname: use the automatic insert statement to create the table and use tbname as the name of the sub-table.

2. Table is a super table and column does not contain tbname: use schemaless to write the table. TDengine will auto-create a tbname based on the super table name and tag value.

3. Table is a sub-table: Use insert statement to write, ignoreTagUnmatched parameter is true, ignore data in record whose tag value is inconsistent with that of table.

4. Table is a common table: use insert statement to write data.


## 3 Features Introduction
### 3.1 Sample
Configure a job to write to TDengine

Create a supertable on TDengine:

```sql
create database if not exists test;
create table test.weather (ts timestamp, temperature int, humidity double) tags(is_normal bool, device_id binary(100), address nchar(100));
```

Write data to TDengine using the following Job configuration:

```json
{
  "job": {
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "type": "string",
                "value": "tb1"
              },
              {
                "type": "date",
                "value": "2022-02-20 12:00:01"
              },
              {
                "type": "long",
                "random": "0, 10"
              },
              {
                "type": "double",
                "random": "0, 10"
              },
              {
                "type": "bool",
                "random": "0, 50"
              },
              {
                "type": "bytes",
                "value": "abcABC123"
              },
              {
                "type": "string",
                "value": "北京朝阳望京"
              }
            ],
            "sliceRecordCount": 1
          }
        },
        "writer": {
          "name": "tdenginewriter",
          "parameter": {
            "username": "root",
            "password": "taosdata",
            "column": [
              "tbname",
              "ts",
              "temperature",
              "humidity",
              "is_normal",
              "device_id",
              "address"
            ],
            "connection": [
              {
                "table": [
                  "weather"
                ],
                "jdbcUrl": "jdbc:TAOS-RS://192.168.56.105:6041/test"
              }
            ],
            "batchSize": 100,
            "ignoreTagsUnmatched": true
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 1
      }
    }
  }
}
```

### 3.2 Configuration

* jdbcUrl
  * Descrption: Data source JDBC connection information, TDengine JDBC information please refer to: [Java connector](https://www.taosdata.com/docs/cn/v2.0/connector/java#url)
  * Required: yes
  * Default: none
* username
  * Descrption: username
  * Required: yes
  * Default: none

* password
  * Descrption: password of username
  * Required: yes
  * Default: none

* table
  * Descrption: A list of table names that should contain all of the columns in the column parameter (except tbname). Note that tbname in column is used as the TDengine sub-table name.
  * Required: yes
  * Default: none

* column
  * Descrption: A list of field names, the order of the fields should be the column in the record
  * Required: yes
  * Default: none

* batchSize
  * Descrption: Each batchSize record is written to a batch
  * Required: no
  * Default: 1

* ignoreTagsUnmatched
  * Descrption: When table is a sub-table in TDengine, table has a tag value. If the tag value of the data and the tag value of the table are not equal, the data is not written to the table.
  * Required: no
  * Default: false


#### 3.3 Type Convert

Data types in datax that can be mapped to data types in TDengine

| DataX Type | TDengine Type                             |
| ---------- | ----------------------------------------- |
| INT        | TINYINT, SMALLINT, INT                    |
| LONG       | TIMESTAMP, TINYINT, SMALLINT, INT, BIGINT |
| DOUBLE     | FLOAT, DOUBLE                             |
| STRING     | TIMESTAMP, BINARY, NCHAR                  |
| BOOL       | BOOL                                      |
| DATE       | TIMESTAMP                                 |
| BYTES      | BINARY                                    |

### 3.2 From MongoDB to TDengine

Here are some examples of data sources migrating to TDengine

| Sample               | Configuration                                                |
| -------------------- | ------------------------------------------------------------ |
| TDengine to TDengine | [super table to super table with tbname](../src/test/resources/t2t-1.json) |
| TDengine to TDengine | [super table to super table without tbname](../src/test/resources/t2t-2.json) |
| TDengine to TDengine | [super table to sub-table](../src/test/resources/t2t-3.json) |
| TDengine to TDengine | [table to table](../src/test/resources/t2t-4.json)           |
| RDBMS to TDengine    | [table to super table with tbname](../src/test/resources/dm2t-1.json) |
| RDBMS to TDengine    | [table to super table without tbname](../src/test/resources/dm2t-2.json) |
| RDBMS to TDengine    | [table to sub-table](../src/test/resources/dm2t-3.json)      |
| RDBMS to TDengine    | [table to table](../src/test/resources/dm2t-4.json)          |
| OpenTSDB to TDengine | [metric to table](../src/test/resources/o2t-1.json)          |

## 4 Restriction



## FAQ

### Do columns in source table and columns in target table must be in the same order?

Yes, TDengineWriter parses the data from the Datax in the order of the fields in the column.
