[JAWS-UG CLI] Elasticsearch服务: #1 创建和更新域名
我們將使用AWS CLI來創建Elasticsearch Service域。
前提
对于Elasticsearch Service的访问权限
具有对Elasticsearch Service的完全权限。
AWS命令行界面的版本
已确认在以下版本中进行过运行测试。
- AWS CLI 1.10.19
aws --version
aws-cli/1.10.19 Python/2.7.10 Darwin/15.4.0 botocore/1.4.10
0. 准备好了
0.1. 区域的确定
我们将决定要使用的区域。（当前用户所使用的当前区域也会改变。）
export AWS_DEFAULT_REGION='ap-northeast-1'
0.2. 确认个人资料
我会确认个人资料是否符合预期。
aws configure list
Name Value Type Location
---- ----- ---- --------
profile es-prjz-mbp13 env AWS_DEFAULT_PROFILE
access_key ****************XXXX shared-credentials-file
secret_key ****************XXXX shared-credentials-file
region ap-northeast-1 env AWS_DEFAULT_REGION
如果正在使用AssumeRole，则会显示 ”作为配置文件。如果在其他情况下也显示”作为配置文件，请执行以下步骤。
export AWS_DEFAULT_PROFILE=<IAMユーザ名>
获取AWS ID
AWS_ID=$( \
aws sts get-caller-identity \
--query 'Account' \
--output text \
) \
&& echo ${AWS_ID}
XXXXXXXXXXXX
需要使用 aws cli 在版本1.10.18或更高版本。
1. 预先准备
1.1. 确定域名
我们将确定要创建的域名的名称。
ES_DOMAIN_NAME="handson-esdomain-$(date +%Y%m%d)" \
&& echo ${ES_DOMAIN_NAME}
handson-esdomain-20160411
在该地区，必须要有独特的特点。
确保已经不存在具有相同名称的域名。
aws es list-domain-names \
--query "DomainNames[?DomainName == \`${ES_DOMAIN_NAME}\`]"
[]
2. 创建域名
2.1. EBS配置
EBS_STORAGE_CLASS="gp2"
EBS_SIZE="10"
EBS_IOPS="0"
FILE_EBS_OPTIONS="${ES_DOMAIN_NAME}-ebs-options.json"
2.2. 设置Elasticsearch实例
ES_INSTANCE_TYPE="t2.micro.elasticsearch"
ES_INSTANCE_COUNT="1"
ES_DEDICATED_MASTER="false"
ES_ZONE_AWARENESS="false"
FILE_ES_CLUSTER_CONFIG="${ES_DOMAIN_NAME}-cluster-config.json"
2.3. 创建域名
我将创建一个域。
cat << ETX
EBS_STORAGE_CLASS: ${EBS_STORAGE_CLASS}
EBS_SIZE: ${EBS_SIZE}
EBS_IOPS: ${EBS_IOPS}
ES_DOMAIN_NAME: ${ES_DOMAIN_NAME}
ES_INSTANCE_TYPE: ${ES_INSTANCE_TYPE}
ES_INSTANCE_COUNT: ${ES_INSTANCE_COUNT}
ES_DEDICATED_MASTER: ${ES_DEDICATED_MASTER}
ES_ZONE_AWARENESS: ${ES_ZONE_AWARENESS}
FILE_ES_CLUSTER_CONFIG: ${FILE_ES_CLUSTER_CONFIG}
FILE_EBS_OPTIONS: ${FILE_EBS_OPTIONS}
ETX
cat << EOF > ${FILE_ES_CLUSTER_CONFIG}
{
"InstanceType": "${ES_INSTANCE_TYPE}",
"InstanceCount": ${ES_INSTANCE_COUNT},
"DedicatedMasterEnabled": ${ES_DEDICATED_MASTER},
"ZoneAwarenessEnabled": ${ES_ZONE_AWARENESS}
}
EOF
cat ${FILE_ES_CLUSTER_CONFIG}
{
"InstanceType": "t2.micro.elasticsearch",
"InstanceCount": 1,
"DedicatedMasterEnabled": false,
"ZoneAwarenessEnabled": false
}
cat << EOF > ${FILE_EBS_OPTIONS}
{
"EBSEnabled": true,
"VolumeType": "${EBS_STORAGE_CLASS}",
"VolumeSize": ${EBS_SIZE},
"Iops": ${EBS_IOPS}
}
EOF
cat ${FILE_EBS_OPTIONS}
{
"EBSEnabled": true,
"VolumeType": "gp2",
"VolumeSize": 10,
"Iops": 0
}
aws es create-elasticsearch-domain \
--domain-name ${ES_DOMAIN_NAME} \
--elasticsearch-cluster-config file://"${FILE_ES_CLUSTER_CONFIG}" \
--ebs-options file://"${FILE_EBS_OPTIONS}"
{
"DomainStatus": {
"ElasticsearchClusterConfig": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
},
"DomainId": "XXXXXXXXXXXX/handson-esdomain-20160411",
"Created": true,
"Deleted": false,
"EBSOptions": {
"Iops": 0,
"VolumeSize": 10 ,
"VolumeType": "gp2",
"EBSEnabled": true
},
"Processing": true,
"DomainName": "handson-esdomain-20160411",
"SnapshotOptions": {
"AutomatedSnapshotStartHour": 0
},
"AccessPolicies": "",
"AdvancedOptions": {
"rest.action.multi.allow_explicit_index": "true"
},
"ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411"
}
}
2.2. 确认域名
让我们来展示一下域名列表。
aws es list-domain-names \
--query "DomainNames[?DomainName == \`${ES_DOMAIN_NAME}\`]"
[
{
"DomainName": "handson-esdomain-20160411"
}
]
2.3. 确认域名内容
让我们确认创建的域名。
aws es describe-elasticsearch-domain \
--domain-name ${ES_DOMAIN_NAME}
{
"DomainStatus": {
"ElasticsearchClusterConfig": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
},
"DomainId": "XXXXXXXXXXXX/handson-esdomain-20160411",
"Created": true,
"Deleted": false,
"EBSOptions": {
"Iops": 0,
"VolumeSize": 10,
"VolumeType": "gp2",
"EBSEnabled": true
},
"Processing": true,
"DomainName": "handson-esdomain-20160411",
"SnapshotOptions": {
"AutomatedSnapshotStartHour": 0
},
"AccessPolicies": "",
"AdvancedOptions": {
"rest.action.multi.allow_explicit_index": "true"
},
"ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411"
}
}
2.4. 确认域名设置
aws es describe-elasticsearch-domain-config \
--domain-name ${ES_DOMAIN_NAME}
{
"DomainConfig": {
"ElasticsearchClusterConfig": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1457742577.891,
"UpdateVersion": 1,
"UpdateDate": 1457742577.891
},
"Options": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
}
},
"AdvancedOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1457742577.891,
"UpdateVersion": 1,
"UpdateDate": 1457742577.891
},
"Options": {
"rest.action.multi.allow_explicit_index": "true"
}
},
"EBSOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1457742577.891,
"UpdateVersion": 2,
"UpdateDate": 1457742577.891
},
"Options": {
"Iops": 0,
"VolumeSize": 10,
"VolumeType": "gp2",
"EBSEnabled": true
}
},
"AccessPolicies": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1457742577.891,
"UpdateVersion": 1,
"UpdateDate": 1457742577.891
},
"Options": ""
},
"SnapshotOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1457742577.891,
"UpdateVersion": 1,
"UpdateDate": 1457742577.891
},
"Options": {
"AutomatedSnapshotStartHour": 0
}
}
}
}
检查域名的启动状态。
ES_CONFIG_STAT=$( \
aws es describe-elasticsearch-domain-config \
--domain-name ${ES_DOMAIN_NAME} \
--query 'DomainConfig.ElasticsearchClusterConfig.Status.State' \
--output text \
) \
&& echo ${ES_CONFIG_STAT}
Processing
一旦处理变为活动状态，就可以使用了（需要大约10分钟）。
ES_CONFIG_STAT=$( \
aws es describe-elasticsearch-domain-config \
--domain-name ${ES_DOMAIN_NAME} \
--query 'DomainConfig.ElasticsearchClusterConfig.Status.State' \
--output text \
) \
&& echo ${ES_CONFIG_STAT}
Active
2.5. 检查域名设置
aws es describe-elasticsearch-domains \
--domain-name ${ES_DOMAIN_NAME}
{
"DomainStatusList": [
{
"ElasticsearchClusterConfig": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
},
"Endpoint": "search-handson-esdomain-20160411-xxxxxxxxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com",
"Created": true,
"Deleted": false,
"DomainName": "handson-esdomain-20160411",
"EBSOptions": {
"Iops": 0,
"VolumeSize": 10,
"VolumeType": "gp2",
"EBSEnabled": true
},
"SnapshotOptions": {
"AutomatedSnapshotStartHour": 0
},
"DomainId": "XXXXXXXXXXXX/handson-esdomain-20160411",
"AccessPolicies": "",
"Processing": false,
"AdvancedOptions": {
"rest.action.multi.allow_explicit_index": "true",
},
"ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411"
}
]
}
获取域名的ARN。
使用Elasticsearch服务操作域时，需要确定目标域的ARN。
ES_DOMAIN_ARN=$( \
aws es describe-elasticsearch-domain \
--domain-name ${ES_DOMAIN_NAME} \
--query 'DomainStatus.ARN' \
--output text \
) \
&& echo ${ES_DOMAIN_ARN}
arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411
3. 访问域名终端点
让我们尝试通过浏览器访问Elasticsearch服务。
首先，获取域名的终端点（URL）。
ES_DOMAIN_ENDPOINT=$( \
aws es describe-elasticsearch-domains \
--domain-name ${ES_DOMAIN_NAME} \
--query "DomainStatusList[?ARN == \`${ES_DOMAIN_ARN}\`].Endpoint" \
--output text \
) \
&& echo ${ES_DOMAIN_ENDPOINT}
search-handson-esdomain-20160411-xxxxxxxxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com
下一步是获取Kibana的URL。
ES_KIBANA_ENDPOINT=$( \
echo "${ES_DOMAIN_ENDPOINT}/_plugin/kibana/" \
) \
&& echo ${ES_KIBANA_ENDPOINT}
search-handson-esdomain-20160411-xxxxxxxxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com/_plugin/kibana/
让我们在浏览器中访问Kibana。
应该会出现以下的错误。
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411/_plugin/kibana/"}
目前，由于没有获得权限，任何人都无法访问。
4. 域名的更新
我将尝试更改域名的访问策略。
4.1. 制定政策
FILE_INPUT="${ES_DOMAIN_NAME}-AccessPolicy".json
cat << ETX
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
AWS_ID: ${AWS_ID}
ES_DOMAIN_NAME: ${ES_DOMAIN_NAME}
FILE_INPUT: ${FILE_INPUT}
ETX
cat << EOF >> ${FILE_INPUT}
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "es:*",
"Principal":"*",
"Effect": "Allow",
"Resource":"arn:aws:es:${AWS_DEFAULT_REGION}:${AWS_ID}:domain/${ES_DOMAIN_NAME}/*"
}
]
}
EOF
cat ${FILE_INPUT}
创建JSON文件后，务必确认格式是否损坏。
jsonlint -q ${FILE_INPUT}
如果没有输出错误的话，就可以了。
4.2. 设置策略
cat << ETX
ES_DOMAIN_NAME: ${ES_DOMAIN_NAME}
FILE_INPUT: ${FILE_INPUT}
ETX
aws es update-elasticsearch-domain-config \
--domain-name ${ES_DOMAIN_NAME} \
--access-policies file://${FILE_INPUT}
{
"DomainConfig": {
"ElasticsearchClusterConfig": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1459853258.411,
"UpdateVersion": 6,
"UpdateDate": 1459853734.055
},
"Options": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
}
},
"AdvancedOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1459861451.767,
"UpdateVersion": 9,
"UpdateDate": 1459861451.767
},
"Options": {
"rest.action.multi.allow_explicit_index": "true"
}
},
"EBSOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1459853258.411,
"UpdateVersion": 6,
"UpdateDate": 1459853734.055
},
"Options": {
"VolumeSize": 10,
"VolumeType": "gp2",
"EBSEnabled": true
}
},
"AccessPolicies": {
"Status": {
"PendingDeletion": false,
"State": "Processing",
"CreationDate": 1459861451.633,
"UpdateVersion": 9,
"UpdateDate": 1459861451.633
},
"Options": "{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"es:ESHttp*","Resource":"arn:aws:es:us-west-2:961382088619:domain/example-esdomain-20160401/>>*<<"}]}"
},
"SnapshotOptions": {
"Status": {
"PendingDeletion": false,
"State": "Active",
"CreationDate": 1459853258.411,
"UpdateVersion": 6,
"UpdateDate": 1459853734.055
},
"Options": {
"AutomatedSnapshotStartHour": 0
}
}
}
}
aws es describe-elasticsearch-domain \
--domain-name ${ES_DOMAIN_NAME}
{
"DomainStatus": {
"ElasticsearchClusterConfig": {
"DedicatedMasterEnabled": false,
"InstanceCount": 1,
"ZoneAwarenessEnabled": false,
"InstanceType": "t2.micro.elasticsearch"
},
"DomainId": "XXXXXXXXXXXX/handson-esdomain-20160411",
"Created": true,
"Deleted": false,
"EBSOptions": {
"Iops": 0,
"VolumeSize": ||EBS_SIZE|,
"VolumeType": "gp2",
"EBSEnabled": true
},
"Processing": true,
"DomainName": "handson-esdomain-20160411",
"SnapshotOptions": {
"AutomatedSnapshotStartHour": 0
},
"AccessPolicies": "{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"*"},"Action":"es:*","Resource":"arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411/*"}]}",
"AdvancedOptions": {
"rest.action.multi.allow_explicit_index": "true",
"indices.fielddata.cache.size": ""
},
"ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/handson-esdomain-20160411"
}
}
等待AccessPolicies的状态变为Active（大约需要10分钟）。
ES_ACCESS_POLICIES_STATUS=$( \
aws es describe-elasticsearch-domain-config \
--domain-name ${ES_DOMAIN_NAME} \
--query 'DomainConfig.AccessPolicies.Status.State' \
--output text \
) \
&& echo ${ES_ACCESS_POLICIES_STATUS}
Active
5. 确认对 Kibana 的访问
请再次使用浏览器访问以下URL。
echo ${ES_KIBANA_ENDPOINT}
如果显示出“Kibana正在加载”的页面就可以了。
请在创建索引后进行设置操作，现在请将其保持不变。
结束
我已经尝试创建和配置Elasticsearch Service域，并确认了访问策略的更改。
