14

I have to debug an existing project about .net Core and AWS. Our project runs well on our AWS instance but we can't run the project in local.

Firstly we got the AmazonServiceException: Unable to find credentials, but now we have the message: AmazonClientException: No RegionEndpoint or ServiceURL configured. I think it's better.

Our configuration: In our application we have 3 appsettings.{env.EnvironmentName}.json (Development, Local and Production). We know by default VS use the development file. In our development appsettings file we have no AWS object but in the local appsettings file we have only that: 

"AWS": {
      "Region": "ap-southeast-2"
}

We don't have any web.config or other json config file.

We tried to create a credential file as:

[name of my IAM profile]
aws_access_key_id=accesskey
aws_secret_access_key=secretkey
region=ap-southeast-2

But we didn't find how we can use it.

We also try to run the project with the dotnet core run command and to specify some environment variables as :

export AWS_Region=ap-southeast-2
export AWS_ACCESS_KEY_ID=id
export AWS_SECRET_ACCESS_KEY=secret
export AWS_SESSION_TOKEN=token
export 
AWS_CREDENTIAL_FILE=/Users/user/Developer/exemple/nameproject/G$

But same error.

Program.cs file:

var host = new WebHostBuilder()
    .UseKestrel(options => options.AddServerHeader = false)
    .UseContentRoot(Directory.GetCurrentDirectory())
    .UseIISIntegration()
    .UseStartup<Startup>()
    .UseUrls("http://*:9000")
    .CaptureStartupErrors(true)
    .Build();

host.Run();

Startup file (first function):

public Startup(IHostingEnvironment env) {
        // Configuration override https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration
        var builder = new ConfigurationBuilder()
            .SetBasePath(env.ContentRootPath)
            .AddJsonFile($"appsettings.{env.EnvironmentName}.json", false, true)
            .AddEnvironmentVariables();

        Configuration = builder.Build();

        // Shared startup configurator
        CommonStartup = new CommonStartup(env, Configuration);
    }

Here is our question: Where or how are configured the credentials of our project?

Thank you in advance for your answer.

Improve this question
4
  • The AWS SDK automatically tries to find the Access/Secret key so having them in your app.config should be enough. The keys in app.config they need to be as "AWSAccessKey" and "AWSSecretKey" Commented Dec 21, 2017 at 16:37
  • Thank you for your answer Rajesh. The problem is there is no appconfig in this project except the 3 appsettings.{env.EnvironmentName}.json. Can you give me an example of this file? And by default the project use the development config file, how can I tell it to use the local version of appsettings? Thank you in advance for your answers. Commented Dec 21, 2017 at 22:32
  • As you are using configurationbuilder just add them to your json and they should flow through. Commented Dec 22, 2017 at 10:32
  • I tried to add that on my appsettings file: "AWS": { "AccessKeyId": "myaccesskey", "SecretAccessKey": "mysecretkey", "Region": "ap-southeast-2" } But i got the message: donet quit unexpectedly (I am on mac)... Commented Dec 27, 2017 at 0:28

5 Answers 5

Reset to default
14

In AWS, you can create an IAM role and configure it to only have access to the resources it needs (S3 read/write, SES etc). You can then attach this role to the EC2 instance.

If you're using the AWS SDK, use something like:

services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
services.AddAWSService<IAmazonS3>();

It will automatically take care of the permissions for you.

For local development, you probably want to use a credentials file. In your appsettings.Local.json you would have an setting similar to this:

"AWS": {
  "Profile": "myprofilename",
  "Region": "eu-west-1",
  "ProfilesLocation": "C:\\Credentials.txt"
}

You might want to store the credentials file outside of your project so you don't accidentally check it in to source control.

Credentials.txt would look like:

[myprofilename]
aws_access_key_id=MY_ACCESS_KEY
aws_secret_access_key=MY_ACCESS_SECRET

Setting the Environment

You probably want the code sitting on the server to be identical for each environment - it makes deployment and other tasks so much easier. You can achieve this by using the Parameter Store to store all your configuration for each AWS environment.

The way I go about this is to use a "tag" on the EC2 instance to specify the environment name. I then use the tag to get correct configuration from the Parameter Store.

In your case, the tag would either be environment=development or environment=production

The parameter names/keys in the store should match the property names in your JSON appsettings file that you want to override.

They would look similar to:

/development/ConnectionStrings/Database
/development/MySettingGroup/MySetting
/production/ConnectionStrings/Database
/production/MySettingGroup/MySetting

I've added some code to github that checks for the tag and parameters etc - if it's running locally it defaults to an environment name of "LocalDevelopment" (that's the convention I use - so you would need to change that to "Local") and loads the correct appsettings file.

https://github.com/secretorange/aws-aspnetcore-environment-startup

The files you would need to use in your project are here:

https://github.com/secretorange/aws-aspnetcore-environment-startup/tree/master/AWSBoot/Boot

Using the BootHelper your Startup code would look similar to this:

public static IWebHost BuildWebHost()
{
    // ===================================
    // Get the boot config from the server
    // ===================================
    var bootConfig = Task.Run(() => BootHelper.GetConfig()).Result;

    var webHost = new WebHostBuilder()
        .UseKestrel()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .ConfigureAppConfiguration((context, config) =>
        {
            // !!! IMPORTANT !!!
            // Set the environment from boot config
            context.HostingEnvironment.EnvironmentName = bootConfig.Environment;

            config.AddJsonFile("appsettings.json", optional: true)
                    .AddJsonFile($"appsettings.{context.HostingEnvironment.EnvironmentName}.json", optional: true);

            // !!! IMPORTANT !!!
            // If there are any parameters from the server
            // then we'll use them to override anything in the JSON files
            config.AddInMemoryCollection(bootConfig.Parameters);
        })
        .UseIISIntegration()
        .UseStartup<Startup>()
        .Build();

    return webHost;
}

The IAM Role(s) in AWS will need policies attached to grant access to tags and parameters etc. They will look similar to:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "ec2:DescribeInstances",
              "tag:GetResources",
              "tag:GetTagValues",
              "tag:GetTagKeys"
          ],
          "Resource": "*"
      }
  ]
}


{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": "ssm:GetParametersByPath",
          "Resource": [
              "arn:aws:ssm:YOUR_ARN_HERE:parameter/development",
              "arn:aws:ssm:YOUR_ARN_HERE:parameter/development/*"
          ]
      }
  ]
}

It might seem like a bit of a hassle to setup - but once it's up an running it means you can easily keep all your secrets out of source control and also create a new environment (Staging for example) simply by creating a new tag and parameters. No code changes required.

If you prefer, you can keep some config in appsettings.Production.json - it just means that if you want to create a new environment, you'll need to create a new JSON file and deploy new code etc. It's perhaps a bit cleaner if all environment information is in AWS (i.e. Parameter Store).

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

7

I figured out a way to configure AWS Credentials using only values in the appsettings.json file. I'll detail it below in case it might help someone. Heads up! This is Not the recommended way by AWS, I just needed it for this specific use case.

In this example I needed the AWS Credentials (Access Key Id and Access Secret) and also the Region, and some other configuration for an SQS Queue client I needed. The application is a .Net 5 Worker Service (it has dependency injection and configuration files set-up out of the box the same way an ASP.Net Core web app would).

Here is the appsettings.json file:

{
  "AwsSqsConfiguration": {
    "AWSAccessKey": "ACCESSKEYID",
    "AWSSecretKey": "SECRETKEY",
    "AWSRegion": "us-west-2",
    "AWSQueueUrl": "https://sqs.us-east-1.amazonaws.com/rest-of-queue-url",
    "AWSQueueServiceUrl": "http://sqs.us-east-1.amazonaws.com"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  }
}

Here's the corresponding C# 9 record for the configuration section 'AwsSqsConfiguration':

public record AwsSqsConfiguration (
        string AWSAccessKey = null,
        string AWSSecretKey = null,
        string AWSRegion = null,
        string AWSQueueUrl = null,
        string AWSQueueServiceUrl = null);

Here's the Program.cs class (similar to an ASP.Net Core web app's Startup.cs class). Notice the use of Amazon.Runtime.BasicAWSCredentials to pass in the Access Key and the Secret Key:

using Amazon.Extensions.NETCore.Setup;
using Amazon.SQS;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using PlatformDeploymentService.Core.Factories;
using PlatformDeploymentService.Core.Interfaces;
using PlatformDeploymentService.Core.Models.Configuration;
using PlatformDeploymentService.Core.Services;

namespace PlatformDeploymentService.WorkerService
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((hostingContext, configuration) => {
                    configuration.Sources.Clear();
                    IHostEnvironment env = hostingContext.HostingEnvironment;
                    configuration
                        .SetBasePath(env.ContentRootPath)
                        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", true, true)
                        .AddEnvironmentVariables();
                })
                .ConfigureServices((hostContext, services) =>
                {
                    services.AddHostedService<Worker>();

                    var awsSqsConfiguration = new AwsSqsConfiguration();
                    hostContext.Configuration.GetSection(nameof(AwsSqsConfiguration)).Bind(awsSqsConfiguration);
                    AWSOptions awsOptions = new AWSOptions
                    {
                        Credentials = new Amazon.Runtime.BasicAWSCredentials(awsSqsConfiguration.AWSAccessKey, awsSqsConfiguration.AWSSecretKey),
                        Region = Amazon.RegionEndpoint.GetBySystemName(awsSqsConfiguration.AWSRegion)
                    };
                    services.AddDefaultAWSOptions(awsOptions);
                    services.AddSingleton<IAmazonSQS>(sp => new AmazonSQSClient(awsOptions.Credentials, new AmazonSQSConfig { ServiceURL = awsSqsConfiguration.AWSQueueServiceUrl }));
                });
    }
}

And the package references for the Nuget packages used in that Program.cs class:

<ItemGroup>
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.3.101" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="5.0.0" />
    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.10.9" />
  </ItemGroup>

And then just wherever you need the SQS Client to queue messages or whatever, you receive an instance of IAmazonSQS in the constructor so the dependency injection can inject it.

References:

Improve this answer

2 Comments

I forgot to mention that the Nuget package AWSSDK.SQS (version 3.5.1.3 as of this writing) is also used, to reference IAmazonSQS and AmazonSQSClient, AmazonSQSConfig, etc.
This seems like a lot of undoing of defaults, only to rebuild some of the same things for very little benefit.
3

Here is the blog post I wrote up about configuring AWS credentials for .NET Core using the AWSSDK.Extensions.NETCore.Setup

https://aws.amazon.com/blogs/developer/configuring-aws-sdk-with-net-core/

Improve this answer

4 Comments

But what is this? "Profile": "local-test-profile", . How to set local-test-profile on windows pc?
@norm instead of storing credentials in a file isnt there a way to directly reading from appsettings.
Unlike .NET Framework there is no Global singleton to get a hold of the configuration object created for your application. The purpose of AWSSDK.Extensions.NETCore.Setup is to bridge the gap from .NET Core Configuration system with the AWS SDK which can't take a dependency on the Configuration packages or have a handle to the Configuration object.
@NormJohanson what to do in production? Are environment variables the way forward?
2

I target .net Core 2.0 instead of 1.1 and also to add environment variable in VS (ASPNETCORE_ENVIRONMENT, AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID, AWS_DEFAULT_REGION) and it works!

Thank you for your help Rajesh.

But if someone know why it works now, thank you in advance to write your answer here.

Improve this answer

2 Comments

How did you add those environment variables please?
You can do it in a number of ways. Visual Studio has a place in the properties under debug where you can set the environment variables to be pulled. Be sure that you are under the correct project. Another way is to add them to your system's environment variables to run. This is a good option if you are using docker or a non windows environment and still want it to be running on CI (Jenkins)
0

For me the environment variable was missing. Just added it and it worked fine.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.