Alestic.com - A Personal AWS Blog by Eric Hammond - Alestic.com

Time 2020-11-08 14:05:33

Web Name: Alestic.com - A Personal AWS Blog by Eric Hammond - Alestic.com

WebSite: http://alestic.com

ID:109067

Keywords:

Personal,AWS,Alestic,

Description:

Amazon recently announced the AWS IAM Access Analyzer,a useful tool to help discover if you have granted unintended accessto specific types of resources in your AWS account.At the moment, an Access Analyzer needs to be created in each regionof each account where you want to run it.Since this manual requirement can be a lot of work, it is a commoncomplaint from customers. Given that Amazon listens to customerfeedback and since we currently have to specify a type of ACCOUNT ,I expect at some point Amazon may make it easier to run AccessAnalyzer across all regions and maybe in all accounts in an AWSOrganization. Until then This article shows how I created an AWS IAM Access Analyzer in allregions of all accounts in my AWS Organization using the aws-cli.PrerequisitesTo make this easy, I use the bash helper functions that I definedin last week s blog post here:Please read the blog post to see what assumptions I make about the AWSOrganization and account setup. You may need to tweak things if yoursetup differs from mine.Here is my GitHub repo that makes it more convenient for me to installthe bash functions. If your AWS account structure matches minesufficiently, it might work for you, too:To start, let s show how to create an IAM Access Analyzer in allregions of a single account.Here s a simple command to get all the regions in the current AWSaccount:aws ec2 describe-regions \ --output text \ --query 'Regions[][RegionName]'This command creates an IAM Access Analyzer in a specificregion. We ll tack on a UUID because that s what Amazon does, though Isuspect it s not really necessary.region=us-east-1uuid=$(uuid -v4 -FSIV || echo 1 ) # may need to install uuid commandanalyzer= accessanalyzer-$uuid aws accessanalyzer create-analyzer \ --region $region \ --analyzer-name $analyzer \ --type ACCOUNTBy default, there is a limit of a single IAM Access Analyzer peraccount region. The fact that this is a default limit implies thatit may be increased by request, but for this guide, we ll just notcreate an IAM Access Analyzer if one already exists.This command lists the name of any IAM Access Analyzers that mightalready have been created in a region:region=us-east-1aws accessanalyzer list-analyzers \ --region $region \ --output text \ --query 'analyzers[][name]'We can put the above together, iterating over the regions, checking tosee if an IAM Access Analyzer already exists, and creating one if itdoesn t: Read More… by generating a temporary IAM STS session with MFA then assuming cross-account IAM rolesI recently had the need to run some AWS commands across all AWSaccounts in my AWS Organization. This was a bit more difficult toaccomplish cleanly than I had assumed it might be, so I present thesteps here for me to find when I search the Internet for it in thefuture.You are also welcome to try out this approach, though if your accountstructure doesn t match mine, it might require some tweaking.Assumptions And Background(Almost) all of my AWS accounts are in a single AWS Organization. Thisallows me to ask the Organization for the list of account ids.I have a role named admin in each of my AWS accounts. It has a lotof power to do things. The default cross-account admin role name foraccounts created in AWS Organizations is OrganizationAccountAccessRole .I start with an IAM principal (IAM user or IAM role) that the aws-clican access through a source profile . This principal has the power toassume the admin role in other AWS accounts. In fact, that principalhas almost no other permissions.I require MFA whenever a cross-account IAM role is assumed.You can read about how I set up AWS accounts here, including the aboveconfiguration:I use and love the aws-cli and bash. You should, too, especially ifyou want to use the instructions in this guide.I jump through some hoops in this article to make sure that AWScredentials never appear in command lines, in the shell history, or infiles, and are not passed as environment variables to processes thatdon t need them (no export).SetupFor convenience, we can define some bash functions that will improveclarity when we want to run commands in AWS accounts. These freely usebash variables to pass information between functions.The aws-session-init function obtains temporary session credentialsusing MFA (optional). These are used to generate temporary assume-rolecredentials for each account without having to re-enter an MFA tokenfor each account. This function will accept optional MFA serial numberand source profile name. This is run once.aws-session-init() { # Sets: source_access_key_id source_secret_access_key source_session_token local source_profile=${1:-${AWS_SESSION_SOURCE_PROFILE:?source profile must be specified}} local mfa_serial=${2:-$AWS_SESSION_MFA_SERIAL} local token_code= local mfa_options= if [ -n $mfa_serial ]; then read -s -p Enter MFA code for $mfa_serial: token_code echo mfa_options= --serial-number $mfa_serial --token-code $token_code read -r source_access_key_id \ source_secret_access_key \ source_session_token \ $(aws sts get-session-token \ --profile $source_profile \ $mfa_options \ --output text \ --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]') test -n $source_access_key_id return 0 || return 1 Read More… A collection of AWS workshop links built by Jennine Townsend, expert sysadmin and cloud intelligence analystCaution: This is not an official list of AWS workshops. It is notpossible to verify that some of these links are controlled andmaintained by Amazon. You should examine the source of theinstructions and code to decide if you trust the source and want torun what the guide is suggesting you try. Use new, AWS dedicatedaccounts to run workshops and labs instead of running them in accountswith existing valuable resources.Most of these AWS workshops seem to be from or updated for AWS re:Invent 2019:DOP306 - Building a Serverless Application with the AWS Cloud Development Kit (AWS CDK)https://github.com/aws-samples/aws-modern-application-workshop/tree/python-cdkService Catalog Toolshttps://service-catalog-tools-workshop.com/reinvent2019/SEC404 - Building Secure APIs in the Cloudhttps://workshop.reinvent.awsdemo.me Slides:http://files.reinvent.awsdemo.me/building_secure_apis_in_the_cloud.pdfSVS203 - Wild Rydes: In this workshop you’ll deploy a simple web application that enables users to request unicorn rides from the Wild Rydes fleet https://webapp.serverlessworkshops.ioSumerian AR Webapp Workshophttp://workshop-sumerian-ar-webapp.s3-website-eu-west-1.amazonaws.com/en/A co-worker at Archer asked if there was a way to schedule messagespublished to an Amazon SNS topic.I know that scheduling messages to SQS queues is possible to someextent using the DelaySeconds message timer, whichallows postponing visibility in the queue up to 15 minutes, but SNSdoes not currently have native support for delays.However, since AWS Step Functions has built-in integration with SNS,and since it also has a Wait state that can schedule or delayexecution, we can implement a fairly simple Step Functions statemachine that puts a delay in front of publishing a message to an SNStopic, without any AWS Lambda code.OverviewThis article uses an AWS CloudFormation template to create a sampleAWS stack with one SNS topic and one Step Functions state machine withtwo states.This is the CloudFormation template, if you d like to review it:Here is the Step Functions state machine definition from the aboveCloudFormation template:{ StartAt : Delay , Comment : Publish to SNS with delay , States : { Delay : { Type : Wait , SecondsPath : $.delay_seconds , Next : Publish to SNS Publish to SNS : { Type : Task , Resource : arn:aws:states:::sns:publish , Parameters : { TopicArn : ${SNSTopic} , Subject.$ : $.subject , Message.$ : $.message End : trueThe Delay state waits for delay_seconds provided in the input tothe state machine execution (as we ll see below).The Publish to SNS task uses the Step Functions integration with SNSto call the publish API directly with the parameters listed, some ofwhich are also passed in to the state machine execution.Now let s take it for a spin! Read More…Amazon recently announced AWS Solutions, a centralcatalog of well-designed, well-documented, CloudFormation templatesthat solve common problems, or create standard solutionframeworks. My tweet about this announcement garnered moreinterest than I expected.One common request was to have a way to be alerted when Amazonpublishes new AWS Solutions to this catalog. Kira Hammond (yesrelation) has used AWS to built and launched a public service thatfills this need.Kira s AWS Solutions Update Feed monitors the AWS Solutions catalogand posts a message to an SNS topic whenever new solutions areadded. The SNS topic is public, so anybody in the world can subscribeto receive these alerts through email, AWS Lambda, or SQS.DesignHere s an architecture diagram showing how Kira constructed thismonitoring and alerting service using serverless technologies on AWS:A scheduled trigger from a CloudWatch Event Rule runs an AWS Lambdafunction every N hours.The AWS Lambda function, written in Python, makes an HTTPS requestto the AWS Solutions catalog to download the current list of solutions.The function retrieves the last known list of solutions from an S3bucket.The function compares the previous list with the current list,generating a list of any new AWS Solutions.If there are any new solutions, a message is posted to a public SNStopic, sending the message to all subscribers.The current list of solutions is saved to S3 for comparison in thefuture runs.If you want to receive alerts when Amazon adds entries to the AWSSolutions catalog, you can subscribe to this public SNS topic: Read More…At Archer, we have been moving credentials into AWS SystemsManager (SSM) Parameter Store and AWS SecretsManager. One of the more interesting credentials isan SSH key that is used to clone a GitHub repository into anenvironment that has IAM roles available (E.g., AWS Lambda, Fargate,EC2).We d like to treat this SSH private key as a secret that is storedsecurely in SSM Parameter Store, with access controlled by AWS IAM, andonly retrieve it briefly when it is needed to be used. We don t evenwant to store it on disk when it is used, no matter how temporarily.After a number of design and test iterations with Buddy, hereis one of the approaches we ended up with. This is one I like for howclean it is, but may not be what ends up going into the final code.This solution assumes that you are using bash to run your Gitcommands, but could be converted to other languages if needed.Using The SolutionHere is the bash function that retrieves the SSH private key from SSMParameter Store, adds it to a temporary(!) ssh-agent process, and runsthe desired git subcommand using the same temporary ssh-agentprocess:git-with-ssm-key() ssm_key= $1 shift ssh-agent bash -o pipefail -c ' if aws ssm get-parameter \ --with-decryption \ --name '$ssm_key' \ --output text \ --query Parameter.Value | ssh-add -q - then git $@ else echo 2 ERROR: Failed to get or add key: '$ssm_key' exit 1 ' bash $@ Here is a sample of how the above bash function might be used to clonea repository using a Git SSH private key stored in SSM Parameter Storeunder the key /githubkeys/gitreader :git-with-ssm-key /githubsshkeys/gitreader clone git@github.com:alestic/myprivaterepo.gitOther git subcommands can be run the same way. The SSH private keyis only kept in memory and only during the execution of the gitcommand.How It Works Read More… A guest post authored by Jennine Townsend, expert sysadmin and AWS aficionadoThere were so many sessions at re:Invent! Now that it s over, I wantto watch some sessions on video, but which ones?Of course I ll pick out those that are specific to my interests, but Ialso want to know the sessions that had good buzz, so I made a listthat s kind of mashed together from sessions that I heard good thingsabout on Twitter, with those that had lots of repeats and overflowsessions, figuring those must have been popular. Read More…There are a number of great guides to AWS re:Invent withexcellent recommendations on how to prepare, what to bring, and how toget the most out of your time. This is not that. In this article, I amgoing to focus only on specific things that I recommend every attendeeexperience at least once while at AWS re:Invent.You may not want to do these things every day (if available) or evenevery year when you return to re:Invent, but I recommend arrangingyour first year schedule to fit as many as you can so you don t gohome missing out.You are taking a week off and making a long trip to Las Vegas. Don tleave without having seen some of the impressive, large scale,in-person experiences the Amazon team has organized.1. Attend a Live AWS re:Invent KeynoteAttending a keynote in person is one of the best ways to get a feelfor the excitement, energy, and shear scale of this movement.You could go to a satellite location and watch the keynote on a screenwith fellow attendees, but there s something special about being inthe big room live.Tip: Get in line early to get a decent seat and enjoy the liveDJ. There are giant screens that show the speakers and presentationmaterial so you won t miss out on content wherever you sit.Tip: It takes a while for the entire crowd to get out of thekeynote space, so don t schedule a session or important lunch meetingright after.Tip: Werner Vogel s keynote is slotted for 2 hours instead of the2.5 hours for Andy Jassy s, but I m not sure if Werner has ever endeda re:Invent keynote on schedule, so sit back and enjoy the informationand enthusiasm.Tip: If new product/service/feature announcements are what exciteyou, then make sure you hit the Andy Jassy keynote.Veterans often watch the streamed keynote on their phone orlaptop while getting ready in their hotel room, or eating breakfast ata cafe. But you flew halfway around the world to be here, so youshould go the last mile (perhaps literally) to get the live AWSre:Invent keynote experience at least once. Read More… I was in the audience when Amazon announced the AWS SecretsManager at the AWS Summit San Francisco. My first thought wasthat we already have a way to store secrets in SSM ParameterStore. In fact, I tweeted: Just as we were all working out the details of using SSM Parameter Store to manage our secrets Another tool to help build securely (looking forward to learning about it).#AWSsummit San Francisco @esh 2018-04-04 11:10So, I started poring over the AWS Secrets Manager documentation andslowly strarted to gain possible enlightenment.I have archived below three stream of consciousness threads that Ioriginally posted to Twitter.Thoughts 1: Secret Rotation Is The Value In AWS Secrets ManagerAfter reading the new AWS Secrets Manager docs, it looks like there isa lot of value in the work Amazon has invested into the design ofrotating secrets.There are a number of different ways systems support secrets, andvarious failure scenarios that must be accounted for.Though RDS secret rotation support is built in to AWS Secrets Manager,customers are going to find more value in the ability to plug incustom code to rotate secrets in any service using AWS Lambda,naturally.Customers write the code that performs the proper steps, and AWSSecrets Manager will drive the steps.It almost looks like we could take the secret rotation frameworkdesign and develop AWS Step Functions and CloudWatch Events Scheduleto drive rotation for secrets in SSM Parameter Store,but for such a critical piece of security infrastructure execution, itmakes sense to lean on Amazon to maintain this part and drive therotation steps reliably.There are ways to create IAM policies that are fine-tuned with justthe AWS Secrets Manager permissions needed, including conditions oncustom tags on each secret.When designing AWS Secrets Manager, I suspect there were discussionsinside Amazon about whether ASM itself should perform the steps tomove the current/pending/previous version labels around during secretrotation, to reduce the risk of customer code doing thisincorrectly.I think this may have required giving the ASM service too muchpermission to manipulate the customer s secrets, so the decision seemsright to keep this with the customer s AWS Lambda function, eventhough there is some added complexity in development.The AWS Secrets Manager documentation is impressively clear forcreating custom AWS Lambda functions for secret rotation, especiallyfor how complex the various scenarios can be.Here s a link to the AWS Secrets Manager Use Guide.https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.htmlThoughts 2: Rotating Secrets Frequently Is ImportantI m starting to understand that it s AWS Secrets Manager not AWSSecrets Store , and that the biggest part of that management seem tobe the automated, transparent, reliable secret rotation.Now that I can see a smooth path to regular secret rotation with AWSSecrets Manager, I m starting to feel like I ve been living inprimitive times letting my database passwords sit at the same valuefor months (ok, years). Read More… A guest post authored by Jennine Townsend, sysadminextraordinaire, and perhaps the AWS Doc team s biggest fan.I ve always believed in reading vendor documentation and now that AWSis by far my largest vendor I m focussed on their documentation. It snot enough to just read it, though, since AWS is constantly makingchanges and releasing features, so even the documentation that I vealready read and think I know will be different tomorrow.Putting together a list of my favorite or most-hated documentationwould just result in a list of what I worked on last week, so insteadI thought it would be more interesting to point out some meta documentation docs about the docs, and docs about what I think ofas the meta services, and a few pages that you ve read butthey ve changed since you last read them!AWS meta documentation:Policy Variables Know what they are, because theycan save a bunch of workGrammar of the IAM JSON Policy Language A bit esotericbut useful, and the examples are real-world useful andthought-provoking but note that if you use CloudFormation for IAMpolicies, you can write them in YAML!Comparing the AWS STS APIs this table is key if youwant to write code that uses rolesDemystifying EC2 Resource-Level Permissions Blog post that s a bit old but still a good walkthrough of iteratingon nontrivial permissionsFive of the seven links on the CloudFormation TemplateReference page were orange (visited recently) for mejust now, but these in particular are always open in a tab while I mwriting CloudFormation: with an SMS text warning two minutes before interruption, using CloudWatch Events Rules And SNSThe EC2 Spot instance marketplace has had a number ofenhancements in the last couple months that have madeit more attractive for more use cases. Improvements include:You can run an instance like you normally do for on-demand instancesand add one option to make it a Spot instance! The instance startsup immediately if your bid price is sufficient given spot marketconditions, and will generally cost much less than on-demand.Spot price volatility has been significantly reduced. Spot pricesare now based on long-term trends in supply and demand instead ofhour-to-hour bidding wars. This means that instances are much lesslikely to be interrupted because of short-term spikes in Spotprices, leading to much longer running instances on average.You no longer have to specify a bid price. The Spot Request willdefault to the instance type s on-demand price in that region. Thissaves looking up pricing information and is a reasonable defaultif you are using Spot to save money over on-demand.CloudWatch Events can now send a two-minutewarning before a Spot instance is interrupted,through email, text, AWS Lambda, and more.Putting these all together makes it easy to take instances youformerly ran on-demand and add an option to turn them into new Spotinstances. They are much less likely to be interrupted than with theold spot market, and you can save a little to a lot in hourly costs,depending on the instance type, region, and availability zone.Plus, you can get a warning a couple minutes before the instance isinterrupted, giving you a chance to save work or launch analternative. This warning could be handled by code (e.g., AWS Lambda)but this article is going to show how to get the warning by email andby SMS text message to your phone.WARNING!You should not run a Spot instance unless you can withstand having theinstance stopped for a while from time to time.Make sure you can easily start a replacement instance if the Spotinstance is stopped or terminated. This probably includes regularlystoring important data outside of the Spot instance (e.g., S3).You cannot currently re-start a stopped or hibernated Spot instancemanually, though the Spot market may re-start it automatically if youconfigured it with interruption behavior stop (or hibernate ) andif the Spot price comes back down below your max bid.If you can live with these conditions and risks, then perhaps givethis approach a try.Start An EC2 Instance With A Spot RequestAn aws-cli command to launch an EC2 instance can be turned into a SpotRequest by adding a single parameter: --instance-market-options ...The option parameters we will use do not specify a max bid, so itdefaults to the on-demand price for the instance type in theregion. We specify stop and persistent so that the instance willbe restarted automatically if it is interrupted temporarily by arising Spot market price that then comes back down.Adjust the following options to suite. The important part for thisexample is the instance market options.ami_id=ami-c62eaabe # Ubuntu 16.04 LTS Xenial HVM EBS us-west-2 (as of post date)region=us-west-2instance_type=t2.smallinstance_market_options= MarketType='spot',SpotOptions={InstanceInterruptionBehavior='stop',SpotInstanceType='persistent'} instance_name= Temporary Demo $(date +'%Y-%m-%d %H:%M') instance_id=$(aws ec2 run-instances \ --region $region \ --instance-type $instance_type \ --image-id $ami_id \ --instance-market-options $instance_market_options \ --tag-specifications \ 'ResourceType=instance,Tags=[{Key= Name ,Value= ' $instance_name ' }]' \ --output text \ --query 'Instances[*].InstanceId')echo instance_id=$instance_idOther options can be added as desired. For example, specify an ssh keyfor the instance with an option like: --key $USERand a user-data script with: --user-data file:///path/to/user-data-script.shIf there is capacity, the instance will launch immediately and beavailable quickly. It can be used like any other instance that islaunched outside of the Spot market. However, this instance has therisk of being stopped, so make sure you are prepared for this.The next section presents a way to get the early warning before theinstance is interrupted. Read More…If you keep creating AWS accounts for every project, as I do, then youwill build up a large inventory of accounts. Occasionally, you mightwant to get a list of all of the accounts for easy review.The following simple aws-cli command pipeline: --output text \ --query 'Accounts[?Status==`ACTIVE`][Status,JoinedTimestamp,Id,Email,Name]' | sort | cut -f2- | column -t -n -e -s$'\cI'Here is a sample of what the output might look like: Read More… instead of connecting to the DeepLens with HDMI micro cable, monitor, keyboard, mouseCredit for this excellent idea goes to Ernie Kim. Thank you!Instructions without sshThe standard AWS DeepLens instructions recommendconnecting the device to a monitor, keyboard, and mouse. Theinstructions provide information on how to view the video streams inthis mode:If you are connected to the DeepLens using a monitor, you can view theunprocessed device stream (raw camera video before being processed bythe model) using this command on the DeepLens device:mplayer –demuxer lavf /opt/awscam/out/ch1_out.h264If you are connected to the DeepLens using a monitor, you can view theproject stream (video after being processed by the model on theDeepLens) using this command on the DeepLens device:mplayer –demuxer lavf -lavfdopts format=mjpeg:probesize=32 /tmp/results.mjpegInstructions with sshYou can also view the DeepLens video streams over ssh, without havinga monitor connected to the device. To make this possible, you need toenable ssh access on your DeepLens. This is available as a checkboxoption in the initial setup of the device. I m working toget instructions on how to enable ssh access afterwards andwill update once this is available.To view the video streams over ssh, we take the same mplayer commandoptions above and the same source stream files, but send the streamover ssh, and feed the result to the stdin of an mplayer processrunning on the local system, presumably a laptop.All of the following commands are run on your local laptop (not on theDeepLens device).You need to know the IP address of your DeepLens device on your localnetwork:ip_address=[IP ADDRESS OF DeepLens]You will need to install the mplayer software on your locallaptop. This varies with your OS, but for Ubuntu:sudo apt-get install mplayerYou can view the unprocessed device stream (raw camera video beforebeing processed by the model) over ssh using the command:ssh aws_cam@$ip_address cat /opt/awscam/out/ch1_out.h264 | mplayer –demuxer lavf -cache 8092 - You can view the project stream (video after being processed by themodel on the DeepLens) over ssh with the command:ssh aws_cam@$ip_address cat /tmp/\*results.mjpeg | mplayer –demuxer lavf -cache 8092 -lavfdopts format=mjpeg:probesize=32 -Note: The AWS Lambda function running in Greengrass on the AWSDeepLens can send the processed video anywhere it wants. Some of thesamples that Amazon provides send to /tmp/results.mjpg, some send to/tmp/ssd_results.mjpeg, and some don t write processed videoanywhere. If you are unsure, perhaps find and read the AWS Lambdafunction code on the device or in the AWS Lambda web console.Benefits of using ssh to view the video streams include:You don t need to have an extra monitor, keyboard, mouse, andmicro-HDMI adapter cable.You don t need to locate the DeepLens close to a monitor, keyboard,mouse.You don t need to be physically close to the DeepLens when you areviewing the video streams.For those of us sitting on a couch with a laptop, a DeepLens acrossthe room, and no extra micro-HDMI cable, this is great news!BonusTo protect the security of your sensitive DeepLens video feeds: Read More… Copy+paste some aws-cli commands to add a new AWS account to your AWSOrganizationThe AWS Organizations service was introduced at AWSre:Invent 2016. The service has some advanced features, but at aminimum, it is a wonderful way to create new accounts easily, with:no need to enter a phone number, answer a call, and key in aconfirmation that you are humanautomatic, pre-defined cross-account IAM role assumable by masteraccount IAM usersno need to pick and securely store a passwordI create new AWS accounts at the slightest provocation. They don tcost anything as long as you aren t using resources, and they are anice way to keep unrelated projects separate for security, costmonitoring, and just keeping track of what resources belong where.I will create an AWS account for a new, independent, side project. Iwill create an account for a weekend hackathon to keep that mess awayfrom anything else I care about. I will even create an account just totest a series of AWS commands for a blog post, making sure that I amnot depending on some earlier configurations that might not be inreaders accounts.By copying and pasting commands based on the examples in this article,I can create and start using a new AWS account in minutes.Before You Start Read More… If you are using and depending on the TimerCheck.ioservice, please be aware that the entire code base will be swapped outand replaced with new code before the end of May, 2017.Ideally, consumers of the TimerCheck.io API will notice no changes,but if you are concerned, you can test out the new implementationusing this temporary endpoint: https://new.timercheck.io/For example:This new endpoint uses the same timer database, so all timers can bequeried and set using either endpoint.At some point before the end of May, the new code will be activated bythe standard https://timercheck.io endpoint. Read More…

TAGS:Personal AWS Alestic 

<<< Thank you for your visit >>>

A Personal AWS Blog by Eric Hammond

Hot Websites