TDengine在量化交易中提供数据分析[更新中]。

2020/02/24 TDengine

记录一下量化交易的开始

准备工具

  • DB TDengine
  • 编程语言 Golang + Python
  • 环境 Unix Linux
  • 交易所 Bitmex Huobi Binance Okex
  • 每天增量300w数据
  • 机器配置 I7-8700 + 32GB DDR4 + GTX2060 + GTX1080 (Xgboost) 磁盘 6TB(SATA6GB/s)

编译 TDengine (修改表字段限制)

  • 下载或者clone 官方发布的Release

  • 编译

yum install gcc gcc-c++ make maven  -y # centos 7

git clone https://github.com/taosdata/TDengine

cd TDengine

mkdir build && cd build

cmake .. && cmake --build .

make install

#Install the project...
#/usr/bin/cmake -P cmake_install.cmake
#-- Install configuration: "Debug"
#make install script: /root/TDengine-ver-1.6.5.5-fnk/packaging/tools/make_install.sh
#this is centos system
#source directory: /root/TDengine-ver-1.6.5.5-fnk
#binary directory: /root/TDengine-ver-1.6.5.5-fnk/build
#Start to install TDEngine...
#Created symlink from /etc/systemd/system/multi-user.target.wants/taosd.service to /etc/systemd/system/taosd.service.

#TDengine is installed successfully!

#To configure TDengine : edit /etc/taos/taos.cfg
#To start TDengine     : sudo systemctl start taosd
#To access TDengine    : use taos in shell

#TDengine is installed successfully!

# 出现这个说明已经安装好了

修改Taos DB 配置文件

# 将数据存储地址修改成 数据盘 日志改为 日志盘
# data file's directory
dataDir               /data

# log file's directory
logDir                /data/log

创建数据库

create database bitmex;
use bitmex;
--- 订单订阅数据表
CREATE TABLE bitmex.trade
 (
   ts                   TIMESTAMP,
   systemc_time         BIGINT,
   bitmex_timestamp     TIMESTAMP,
   side                 BINARY(4),
   size                 FLOAT,
   price                FLOAT,
   symbol               BINARY(16)
);


--- 交易深度数据
CREATE TABLE bitmex.order_l1_10 
  ( 
     ts               TIMESTAMP, 
     systemc_time     BIGINT, 
     symbol           BINARY(16), 
     bitmex_timestamp TIMESTAMP, 
     asks_1_p         FLOAT, 
     asks_1_s         FLOAT, 
     asks_2_p         FLOAT, 
     asks_2_s         FLOAT, 
     asks_3_p         FLOAT, 
     asks_3_s         FLOAT, 
     asks_4_p         FLOAT, 
     asks_4_s         FLOAT, 
     asks_5_p         FLOAT, 
     asks_5_s         FLOAT, 
     asks_6_p         FLOAT, 
     asks_6_s         FLOAT, 
     asks_7_p         FLOAT, 
     asks_7_s         FLOAT, 
     asks_8_p         FLOAT, 
     asks_8_s         FLOAT, 
     asks_9_p         FLOAT, 
     asks_9_s         FLOAT, 
     asks_10_p        FLOAT, 
     asks_10_s        FLOAT, 
     bids_1_p         FLOAT, 
     bids_1_s         FLOAT, 
     bids_2_p         FLOAT, 
     bids_2_s         FLOAT, 
     bids_3_p         FLOAT, 
     bids_3_s         FLOAT, 
     bids_4_p         FLOAT, 
     bids_4_s         FLOAT, 
     bids_5_p         FLOAT, 
     bids_5_s         FLOAT, 
     bids_6_p         FLOAT, 
     bids_6_s         FLOAT, 
     bids_7_p         FLOAT, 
     bids_7_s         FLOAT, 
     bids_8_p         FLOAT, 
     bids_8_s         FLOAT, 
     bids_9_p         FLOAT, 
     bids_9_s         FLOAT, 
     bids_10_p        FLOAT, 
     bids_10_s        FLOAT 
  ); 

启动采集工具 (开始实时的通过WebSocket接收数据)

# 采集部分两部分

# 采集端 Agent(通过SSL GRPC 将数据传输到 Server)多个 Agent 同时传输

# Server端 将通过GRPC 接收到的数据 通过http接口落库(这里用http接口是因为开发和部署的机器都是Unix的机器,数据库在内网的一台mac上运行。)

  • 采集服务目录结构

  • 采集服务日志恢复

    下图所看到的是 GRPC服务断掉了,数据将会写入log文件并且有明确的标记,当grpc再次可用,将会启动日志恢复。

通过Middware将数据实时的放入TDEngine(这里存在一层日志备份未来可能存储到Hdfs走FLink训练)

确认数据

{
    "status": "succ",
    "head": [
        "ts",
        "systemc_time",
        "symbol",
        "bitmex_timestamp",
        "asks_1_p",
        "asks_1_s",
        "asks_2_p",
        "asks_2_s",
        "asks_3_p",
        "asks_3_s",
        "asks_4_p",
        "asks_4_s",
        "asks_5_p",
        "asks_5_s",
        "asks_6_p",
        "asks_6_s",
        "asks_7_p",
        "asks_7_s",
        "asks_8_p",
        "asks_8_s",
        "asks_9_p",
        "asks_9_s",
        "asks_10_p",
        "asks_10_s",
        "bids_1_p",
        "bids_1_s",
        "bids_2_p",
        "bids_2_s",
        "bids_3_p",
        "bids_3_s",
        "bids_4_p",
        "bids_4_s",
        "bids_5_p",
        "bids_5_s",
        "bids_6_p",
        "bids_6_s",
        "bids_7_p",
        "bids_7_s",
        "bids_8_p",
        "bids_8_s",
        "bids_9_p",
        "bids_9_s",
        "bids_10_p",
        "bids_10_s"
    ],
    "data": [
        [
            "2020-02-25 21:02:56.000",
            1582635776179,
            "XBTUSD",
            "2020-02-25 13:02:55.729",
            9625,
            49906,
            9625.5,
            82752,
            9626,
            10282,
            9626.5,
            1112,
            9627,
            109168,
            9627.5,
            47425,
            9628,
            84425,
            9628.5,
            114773,
            9629,
            69830,
            9629.5,
            34412,
            9624.5,
            2225271,
            9624,
            3024,
            9623.5,
            11051,
            9623,
            468523,
            9622.5,
            6502,
            9622,
            9985,
            9621.5,
            550000,
            9621,
            9295,
            9620.5,
            726837,
            9620,
            647086
        ]
    ],
    "rows": 1
}

提供Web界面 空和多 多家交易所对比,以及瞬时趋势

Python 调用数据开启分析 基础分析 EMA 交叉 (四家交易所同时10秒计算一次)

每天计算趋势图

钉钉通知。

纠正模型

转载请注明出处,本文采用 CC4.0 协议授权

Affiliate

Search

    Post Directory