通过Serverless 获取 hadoop 文件系统的[原创]

通过Serverless (Kubeless) 进行操作Hdfs , 实现下载文件.

code

Python

from hdfs import Client
from bottle import route,HTTPResponse
client = Client("http://hdfs-web-svc.cloudera:50070", root="/", timeout=100, session=False)

headers = {}
headers[str("content-type")] = 'application/octet-stream'
headers['Content-Disposition'] = 'attachment;filename="data.csv"'
@route('/download')
def getfile():
    with client.read("/public/csv/parking_ths_car_record/part-m-00000") as reader:
        content = reader.read()
        content = content.decode('utf-8', 'ignore').encode('gbk')
        return HTTPResponse(body=content, status=200, headers=headers)

serverless.yml

# Welcome to Serverless!
#
# For full config options, check the kubeless plugin docs:
#    https://github.com/serverless/serverless-kubeless
#
# For documentation on kubeless itself:
#    http://kubeless.io

# Update the service name below with your own service name
service: getfile

# Please ensure the serverless-kubeless provider plugin is installed globally.
# $ npm install -g serverless-kubeless
#
# ...before installing project dependencies to register this provider.
# $ npm install

provider:
  name: kubeless
  namespace: ${env:K8S_NAMESPACE, 'cloudera'}
  runtime: python2.7

plugins:
  - serverless-kubeless

functions:
  getfile:
    handler: handler.getfile

requirements.txt

bottle==0.12.13
hdfs==2.1.0

kubernetes Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: getfile-ing
  namespace: cloudera
spec:
  rules:
  - host: getfile.kube.ibm-testing.com
    http:
      paths:
      - backend:
          serviceName: getfile
          servicePort: 8080

通过Serverless (Kubeless) 进行操作Hdfs , 实现下载文件.

code

相关地址

通过Serverless (Kubeless) 进行操作Hdfs , 实现下载文件.

code

相关地址