6.StreamSets_Data_Collector_deploy_to_docker.md · Hu Lyndon/streamsets-start-asset

## StreamSets Data Collector deploy to docker

# Basic Usage

```
docker run --restart on-failure -p 18630:18630 -d --name streamsets-dc streamsets/datacollector dc
```

The default login is: admin / admin.

# Detailed Usage

- You can specify a custom configs by mounting them as a volume to /etc/sdc or /etc/sdc/
- Configuration properties in `sdc.properties` can also be overridden at runtime by specifying them env vars prefixed with SDC_CONF
- For example http.port would be set as SDC_CONF_HTTP_PORT=12345
- You *should at a minimum* specify a data volume for the data directory and stage libraries. The default configured location is /data for $SDC_DATA. You can override this location by passing a different value to the environment variable SDC_DATA. Creating a volume for additional stage libraries is described in more detail below.
- You can also specify your own explicit port mappings, or arguments to the streamsets command.

For example to run with a customized sdc.properties file, a local filsystem path to store pipelines, and statically map the default UI port you could use the following:

```
docker run -v $PWD/sdc.properties:/etc/sdc/sdc.properties:ro -v $PWD/sdc-data:/data:rw -p 18630:18630 -d streamsets/datacollector dc
```

# Creating a Data Volumes

To create a dedicated data volume for the pipeline store issue the following command:

要创建一个专用的数据卷，为了pipeline store，发出如下命令

```
docker volume create --name sdc-data
```

You can then use the `-v` (volume) argument to mount it when you start the data collector.

然后你可以使用 -v 参数挂载它，当你在启动SDC时候

```
docker run -v sdc-data:/data -P -d streamsets/datacollector dc
```

**Note:** There are two different methods for managing data in Docker. The above is using *data volumes* which are empty when created. You can also use *data containers* which are derived from an image. These are useful when you want to modify and persist a path starting with existing files from a base container, such as for configuration files. We'll use both in the example below. See [Manage data in containers](https://docs.docker.com/engine/tutorials/dockervolumes/) for more detailed documentation.

**注意：**有两种不同的方法管理Docker中的数据。上面被使用的data volume在被创建的时候是空的。你也可以使用从一个镜像中派生出来的data container。当您要修改并保留从基本容器中的现有文件开始的路径时，这些功能非常有用，例如配置文件。在下面的示例中，我们将同时使用它们。 请参阅管理容器中的数据以获取更多详细文档。

# Pre-configuring Data Collector

#### Option 1 - Volumes (Recommended)

First we create a data container for our configuration. We'll call ours `sdc-conf`

```shell
docker create -v /etc/sdc --name sdc-conf streamsets/datacollector
docker run --rm -it --volumes-from sdc-conf ubuntu bash
```

**Tip:** You can substitute `ubuntu` for your favorite base image. This is only a temporary container for editing the base configuration files.

提示：你可以使用喜欢的基础image来替换ubuntu。这仅仅是为了编辑基本配置文件而创建的临时容器。

Edit the configuration of SDC to your liking by modifying the files in `/etc/sdc`

通过修改/etc/sdc中的文件，根据自己的喜好编辑SDC的配置

You can choose to create separate data containers using the above procedure for `$SDC_DATA` (`/data`) and other locations, or you can add all of the volumes to the same container. For multiple volumes in a single data container you could use the following syntax:

您可以选择使用上面的过程为$SDC_DATA`(/data)`和其他位置创建单独的数据容器，也可以将所有卷添加到同一容器中。 对于单个数据容器中的多个卷，可以使用以下语法：

```
docker create -v /etc/sdc -v /data -v --name sdc-volumes streamsets/datacollector
```

If you find it easier to edit the configuration files locally you can, instead of starting the temporary container above, use the `docker cp` command to copy the configuration files back and forth from the data container.

如果发现在本地编辑配置文件比较容易，则可以使用docker cp命令从数据容器来回复制配置文件，而不是启动上面的临时容器。

To install stage libs using the CLI or Package Manager UI you'll need to create a volume for the stage libs directory. It's also recommended to use a volume for the data directory at a minimum.

要使用CLI或Package Manager UI安装Stage库，您需要为stage libs目录创建一个卷。 还建议至少为数据目录使用一个卷。

`docker volume create --name sdc-stagelibs` (If you didn't create a data container for `/data` then run the command below) `docker volume create --name sdc-data`

The volume needs to then be mounted to the correct directory when launching the container. The example below is for Data Collector version 3.9.1.

然后在启动容器时需要将卷挂载到正确的目录。 下面的示例适用于Data Collector版本3.9.1。

```shell
docker run --name sdc -d -v sdc-stagelibs:/opt/streamsets-datacollector-3.9.1/streamsets-libs -v sdc-data:/data -P streamsets/datacollector dc -verbose
```

To get a list of available libs you could do: 要获取可用libs的列表，可以执行以下操作：

```shell
docker run --rm streamsets/datacollector:3.9.1 stagelibs -list
```

For example, to install the JDBC lib into the sdc-stagelibs volume you created above, you would run:

例如，要将JDBC库安装到上面创建的sdc-stagelibs卷中，可以运行：

```shell
docker run --rm -v sdc-stagelibs:/opt/streamsets-datacollector-3.9.1/streamsets-libs streamsets/datacollector:3.9.1 stagelibs -install=streamsets-datacollector-jdbc-lib
```

#### Option 2 - Deriving a new image

One disadvantage of the first method is that we can't commit data in a volume and distribute it via a docker registry. Instead we must create the volume, backup the data, restore the data if we need to recreate or move the container to another host.

第一种方法的缺点是我们无法提交卷中的数据，也无法通过Docker注册表分发数据。 相反，如果我们需要重新创建容器或将容器移动到另一台主机，则必须创建卷，备份数据，还原数据。

This second option will allow us to make modifications to the original base image, creating a new one which can be pushed to a docker registry and easily distributed.

第二个选项将允许我们对原始基础映像进行修改，创建一个新映像，该映像可以推送到Docker注册表并轻松分发。

The simplest and recommended way is simply to create your own Dockerfile with the official streamsets/datacollector image as the base! This provides a repeatable process for building derived images.

最简单和推荐的方法是简单地以官方streamsets/datacollector image为基础创建自己的Dockerfile！ 这为构建派生image提供了可重复的过程。

For example this derived Dockerfile:

例如，此派生的Dockerfile：

```
FROM streamsets/datacollector:3.9.1
# My custom configured sdc.properties
COPY sdc.properties /etc/sdc/sdc.properties
docker build -t mycompany/datacollector:3.9.1-abc .
docker push mycompany/datacollector:3.9.1abc
```

I've now created a new image with a customized sdc.properties file and am able to distribute it from a docker registry with ease!

现在，我使用自定义的`sdc.properties`文件创建了一个新image，并能够轻松地从Docker注册表中分发它！

You can also launch a default container, modify it while it is running and use the `docker commit` command, but this isn't recommended.

您还可以启动默认容器，在运行时对其进行修改，并使用docker commit命令，但这是不推荐的。

Hu Lyndon/streamsets-start-asset

简介

发行版

贡献者

近期动态

Hu Lyndon/streamsets-start-asset .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

Hu Lyndon/streamsets-start-asset