github actions workflow - ssh into server, download single file via curl, unable to open file

Question

I have a github workflow which is supposed to login via SSH into a remote server and then download the docker-compose file from a private github repository.

I was first struggling to get the multi-line string syntax right for the run section in the workflow yaml file, in particular the curl command, since it requires quotes for the header. The solution i found is to split the curl command into environment variables.

download-file:
    name: download file
    runs-on: ubuntu-latest

    steps:
      - name: install ssh keys
        # check this thread to understand why its needed:
        # <https://stackoverflow.com/a/70447517>
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          host='${{ secrets.SSH_HOST }}'
          hosts="$(dig +short "$host" | grep -v '\.$' | sed -z 's|\n|,|g')$host"
          ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts

      - name: download docker compose file from private repo to server
        run: ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}
          # the first curl call to the public repo works. 
          # the file can be opened. the second curl call also 
          # finishes but the file can NOT be opened. 
          "ls -la
          && cd test
          && ls -la
         
          && curl curl $CURL_PUBLIC_URL -o $URL_PUBLIC_O
          && curl -v -H $CURL_H1 -H $CURL_H2 -H $CURL_H3 -o $CURL_O $CURL_URL
          && exit"

        env:
          CURL_H1: '"\"Authorization: Bearer ${{secrets.PULL_ACCESS_TOKEN}}\""'
          CURL_H2: '"\"Accept: application/vnd.github.v3.raw\""'
          CURL_H3: '"\"Content-Type: text/html; charset=UTF-8\""'
          CURL_O: "docker-compose.yml"
          CURL_URL: "https://raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"
          CURL_PUBLIC_URL: "https://raw.githubusercontent.com/thnk2wn/rasp-cat-siren/master/siren/Dockerfile"
          CURL_PUBLIC_O: "Dockerfile"

      - name: cleanup
        run: rm -rf ~/.ssh

PROBLEM:

When I execute the curl command after logging in manually into the server, the file is downloaded and I can open it, no problem.

Within the github workflow, the curl request also executes without any error, but when I try to open the downloaded file docker-compose.yml via cat docker-compose.yml after ssh manually into the server, the file can not be opened and the message found an invalid character in header name is displayed instead.

My guess is that there is some kind of issue with the string formatting of the variables. It already took me hours experimenting with various string formats escaping the double quote with the backslash to keep the double quote. The string syntax looks already quite overly complicated to me to be honest.

EDIT 1: When I use wget instead of curl in the actions run command, then i get a 400 response. When I execute the wget request manually on the server, the file is downloaded without any issue.

EDIT 2: Downloading a file from a public github repository works without any problems and can be opened with no issues.

QUESTION:

How can I run the curl request after ssh into the server downloading the single docker-compose file from the private github repo without corrupting the file so that the file can actually be opened? Respectively, is there an issue with the formatting of the syntax for the curl command within runconsidering the string formatting of the envvariables and if so, what would be a better/ correct way?

Thanks for your help!

Could you run git archive instead of curl? If yes, I have a working example. — Mate
– Mate, Commented Oct 19, 2024 at 11:38
of course, I just need to get this file from the private github repo, no matter how :) please provide your working example. — Lucien Chardon
– Lucien Chardon, Commented Oct 19, 2024 at 11:43

Mate · Accepted Answer · 2024-10-19 12:01:57Z

0

git example, maybe need some minor to adjust env names

cat $sshKey > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
# ssh-keyscan -t rsa,dsa github.com >> ~/.ssh/known_hosts

git archive --format=tar [email protected] "+YOUR-BRANCH+" docker-compose.prod.yml | tar -O -xf - > docker-compose.yml

Note: I don't know what you'll do next. But it's never a good idea to use docker-compose in prod. It's just to start, you need to avoid it when your project grow up

edited Oct 19, 2024 at 12:01

answered Oct 19, 2024 at 11:55

Mate

5,3272 gold badges36 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Lucien Chardon Over a year ago

thanks! why is docker compose not good for production in your opinion?

Mate Over a year ago

Because they said: docs.docker.com/compose/how-tos/production .

Lucien Chardon Over a year ago

How can it never be a good idea to use compose for production? In the link you provided they literally explain considerations on how to setup compose for production :)

Lucien Chardon Over a year ago

btw: the project I use this setup for is for an MVP/ proof of concept running on a single server. I am not sure yet about kubernetes which seems to me like something way too overcomplicated for most use cases.

Mate Over a year ago

Yes, please, read again. "Compose has traditionally been focused on development and testing workflows, but with each release we're making progress on more production-oriented features..." They did a lot of work, but over time it becomes difficult to maintain. Docker + k8s could be a better approach

|

Lucien Chardon · Accepted Answer · 2024-10-19 12:40:32Z

It seems the culprit was somehow the header information in the curl request. So I found this solution where the access token is placed inside the URL. The critical piece here is the syntax of the env variable CURL_URL:

# ...
download-file:
    name: download file
    runs-on: ubuntu-latest

    env:
      CURL_O: "docker-compose.yml"
      CURL_PATH: "raw.githubusercontent.com/<username>/<repository name>/refs/heads/main/docker-compose.prod.yml"

    steps:
      - name: install ssh keys
        # check this thread to understand why its needed:
        # <https://stackoverflow.com/a/70447517>
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          host='${{ secrets.SSH_HOST }}'
          hosts="$(dig +short "$host" | grep -v '\.$' | sed -z 's|\n|,|g')$host"
          ssh-keyscan -H "$hosts" > ~/.ssh/known_hosts

      - name: test pull single file from private repo
        env:
          CURL_URL: "https://${{secrets.PULL_ACCESS_TOKEN}}@${{env.CURL_PATH}}"
        run: |
          ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }} <<'ENDSSH'
          cd test
          ls -la
          curl ${{env.CURL_URL}} -o ${{env.CURL_O}}
          ENDSSH

      - name: cleanup
        run: rm -rf ~/.ssh

Collectives™ on Stack Overflow

github actions workflow - ssh into server, download single file via curl, unable to open file

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related