Rust Native Spark Sample Install

2019-12-26 10:34:51 +0800

Native Spark for rust fast spark

介绍 Native Spark For Rust

* github 地址 ： https://github.com/rajasekarv/native_spark

Sample

## clone project 
git clone https://github.com/rajasekarv/native_spark
## Use Ubuntu18.04

    curl -ssf https://sh.rustup.rs | sh  # 如果失败 curl https://sh.rustup.rs > a.sh && ./a.sh -y 强制执行
     apt install openssl
     apt install openssl-dev
     apt install openssl-sys
     apt install libssl-dev
## clone capnproto 
## 序列化工具包
git clone https://github.com/sandstorm-io/capnproto.git
cd capnproto
sudo  apt install autoreconf
sudo  apt install autoconf
sudo  apt install automake
sudo  apt install libtool
autoreconf -i
./configure 
make -j4 check
sudo make install 

## 进入项目测试

cd native_spark
## 安装nightly 版本
rustup install nightly
## 强行覆盖一波
rustup override set nightly
## run example 
cargo run --example make_rdd

## cd native_spark/docker
# build docker 镜像
./build_image.sh
# 启动测试节点 并且将当前的 target 目录映射到 容器内的 /home/dev
./test_cluster.sh

图示

看一下代码

启动测试节点

进入Master

测试native_spark

在容器内测试native_spark 集群 `./file_read --deployment_mode distributed --local_ip=0.0.0.0`

RUST_BACKTRACE=full ./file_read --deployment_mode distributed --local_ip=0.0.0.0

未完结

转载请注明出处，本文采用 CC4.0 协议授权