I was addicted to not being able to connect to AWS-S3 from the Docker container

I was unexpectedly addicted to trying to get the files on AWS-S3 from the application (JVM) in the Docker container launched on the AWS-EC2 instance. I'll note that this information may be useful to someone. If you are in a hurry, please see only the final summary.

Cause

The error I was getting was something like this.

Unable to load AWS credentials from any provider in the chain
        at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3820)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3778)
        at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:649)

I was angry without the credential.

What I tried part 1

If this did not pass, I could not even start the application, so for the time being, I set the environment variables as follows in the container and tried restarting.

export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxx

As a result, I got the same error. It doesn't seem to read the environment variables. ʻEnv | grep AWS` Then, it seems that the environment variables are set properly.

What I tried part 2

Next, for the time being, I created a credential file in the container. As the application launch user, I created the following file with the name $ HOME / .aws / credentials.properties.

accessKey=AKIAJZJ4UAXAC4QVVW5Q=xxxxxxxxxxxxxxxx
secretKey=FXNnRbZjAlg5Z9SWkFbpKXKLRpGVKLTlwNoKDHuP=xxxxxxxxxxxxxxxx

As a result, I got the same error. It doesn't seem to read credentials.properties. I also prepared credentials just in case, but it didn't work.

What I tried part 3

Finally, when I came here, I decided to set it properly when creating the container. I remember the word hurry.

I modified docker-compose.yml to look like the following.

version: '2.0'
services:
  my-app:
    command: /bin/bash
    container_name: my-app
    image: xxxx/xxxx/my-app:my-app-1.2.3
    network_mode: bridge
    ports:
      - 9000:9000/tcp
    stdin_open: true
    tty: true
    volumes:
      - /var/log/my-app:/var/log/my-app:rw
environment:                              #← Add
  - AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxx    #← Add
  - AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxx #← Add

As a result, I got the same error. It doesn't seem to read the environment variables. I also tried creating a ʻenv file, but it didn't work. When I do ʻenv | grep AWS in the container, it looks like the environment variables are set properly.

What I tried part 4

At this point, I thought, "It doesn't matter because I'm angry that there is no credential, "but can I connect to the S3 bucket I want to see from this EC2 instance in the first place? I started to worry about that. There is no help for it, so install ʻaws-clion the host machine and check the connection. Create thecredentials.properties` file mentioned above on the host machine.

# credentials.Properties creation
touch $HOME/.aws/credentials.properties
(Write and save the access key and secret key with vi)

#Introduced because pip was not included
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py

# aws-cli introduction
sudo pip install awscli

#Verification
aws s3 ls s3://xxxxx/xxxxx

I was able to get the file list from S3 safely. It seems that the host machine can connect without any problem.

What I tried part 5

Try mounting credentials.properties on the host machine when creating the container. I rewrote the docker-compose.yml file as follows. The directory on the container side should be the .aws directory under the home directory of the application startup user.

version: '2.0'
services:
  my-app:
    command: /bin/bash
    container_name: my-app
    image: xxxx/xxxx/my-app:my-app-1.2.3
    network_mode: bridge
    ports:
      - 9000:9000/tcp
    stdin_open: true
    tty: true
    volumes:
      - /var/log/my-app:/var/log/my-app:rw
      - $HOME/.aws/credentials.properties:/home/my-app/.aws/credentials.properties #← Add

As a result, I was able to successfully retrieve the file from S3 and launch the application! Why didn't the other method work? Has not been properly investigated and is unknown.

Summary

Below is a summary of my personal situation

What I tried result
Set environment variables inside the container
In the containercredentials.propertiesCreate
Set environment variables when creating a container
When creating a containercredentials.propertiesMount

--Premise: You can connect to S3 from the host machine --Why it didn't work in other ways → I don't know (not investigated)

I'm sorry for the dragonfly that is cut off, but that's all from the scene. If you have any idea of the cause, I would appreciate it if you could tell me.

Recommended Posts

I was addicted to not being able to connect to AWS-S3 from the Docker container
I was addicted to the roll method
I was addicted to the Spring-Batch test
I was able to deploy the Docker + laravel + MySQL app to Heroku!
What I was addicted to when updating the PHP version of the development environment (Docker) from 7.2.11 to 7.4.x
Problems I was addicted to when building the digdag environment with docker
[Rails] The cause of not being able to post was in form_with
I was addicted to the NoSuchMethodError in Cloud Endpoints
I was addicted to the roll method
I was addicted to the Spring-Batch test
I was addicted to not being able to connect to AWS-S3 from the Docker container
I was swallowed by the darkness of the romaji, trying to convert my name to romaji
What I was addicted to when introducing the JNI library
I was addicted to looping the Update statement on MyBatis
I was addicted to the setting of laradock + VSCode + xdebug
What I was addicted to with the Redmine REST API
[Docker] How to access the host from inside the container. http://host.docker.internal:
The story I was addicted to when setting up STS
I was addicted to starting sbt
About the matter that I was addicted to how to use hashmap
I was addicted to the API version min23 setting of registerTorchCallback
[Rough explanation] Causes and remedies for not being able to obtain the name from the ActiveHash model
Memorandum: What I was addicted to when I hit the accounting freee API
Investigate the replacement from Docker to Podman.
[Rails] I was addicted to the nginx settings when using Action Cable.
A story I was addicted to when testing the API using MockMVC
I was addicted to rewriting to @SpringApplicationConfiguration-> @SpringBootTest
[Docker] Copy files from docker container to host
I was addicted to scrollview because I couldn't tap the variable size UIView
[CircleCI] I was addicted to the automatic test of CircleCI (rails + mysql) [Memo]
I was a little addicted to ssh connection from mac to linux (ubuntu)
[Docker] Does not connect to MySQL workbench ... DB container port forwarding settings
I was addicted to unit testing with the buffer operator in RxJava
I was a little addicted to the S3 Checksum comparison, so I made a note.
I was addicted to using RXTX on Sierra
How to get a heapdump from a Docker container
Copy files from docker container to host (docker cp)
I was addicted to installing Ruby/Tk on MacOS
The code I used to connect Rails 3 to PostgreSQL 10
I made a Docker container to run Maven
I was addicted to doing onActivityResult () with DialogFragment
The CSV file that I was able to download suddenly started to appear on the page.
The part I was addicted to in "Introduction to Ajax in Java Web Applications" of NetBeans
A memorandum because I was addicted to the setting of the Android project of IntelliJ IDEA
Docker for Windows: MySQL container does not start when migrating from Hyper-v to WSL2
[Java] I want to calculate the difference from the date
Tokoro I rewrote in the migration from Wicket 7 to 8
I tried to summarize the state transition of docker
Communicate from the outside to the container launched by docker-compose
Command to try using Docker for the time being
Push Docker images from GitHub Actions to GitHub Container Registry
Ssh connect from container to git with VSCode Remote Container
I want to connect to Heroku MySQL from a client
When I started ansible + docker now, I stumbled from the beginning, but I managed to start it
From inexperienced, I was able to change jobs to an in-house developed company even with Corona
It's not a big deal if you understand that I was addicted to receiving emails with Java Mail from Exchange Online
What I fixed when updating to Spring Boot 1.5.12 ・ What I was addicted to
[JDBC] I tried to access the SQLite3 database from Java.
What I was addicted to while using rspec on rails
Wait for the container service to start with docker healthcheck
I was addicted to setting default_url_options with Rails devise introduction
What I did in the version upgrade from Ruby 2.5.2 to 2.7.1
A story I was addicted to in Rails validation settings
I tried to build the environment little by little using docker