码迷,mamicode.com
首页 > 其他好文 > 详细

drill 集成开源s3 存储minio

时间:2018-10-25 11:05:30      阅读:413      评论:0      收藏:0      [点我收藏+]

标签:server   mat   with   ini   htm   display   match   csv   art   

drill 支持s3数据的查询,同时新版的通过简单配置就可以实现minio 的集成

测试使用docker 运行drill 参考 https://www.cnblogs.com/rongfengliang/p/9846899.html,minio 安装也可以使用docker
minio bucket 创建以及文件上传可以使用UI 或者mc

配置

  • s3 配置说明
{
  "type": "file",
  "connection": "s3a://app/",
  "config": {
    "fs.s3a.access.key": "<key>",
    "fs.s3a.secret.key": "<secret>",
    "fs.s3a.endpoint": "<minio server>",
    "fs.s3a.connection.ssl.enabled": "false", // default isn‘t  ssl
    "fs.s3a.path.style.access": "true" 
  },
  "workspaces": {
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  },
  "enabled": true
}
  • 查询
select * from s3.`root`.`user.json`
  • minio 添加数据
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}
{"name":"dalong","age":33}

效果

技术分享图片
技术分享图片

参考资料

https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5?_branch_match_id=520116032687566918
https://www.cnblogs.com/rongfengliang/p/9846899.html
http://drill.apache.org/docs/s3-storage-plugin/

drill 集成开源s3 存储minio

标签:server   mat   with   ini   htm   display   match   csv   art   

原文地址:https://www.cnblogs.com/rongfengliang/p/9847726.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!