AWS Morning Brief for the week of September 20, 2021 with Corey Quinn.

AWS Morning Brief

Billed on AWS For Startups

Links: Entirely optional for attackers: https://osamaelnaggar.com/blog/aws_waf_dangerous_defaults/ Worst Case: https://www.tbray.org/ongoing/When/202x/2021/10/08/The-WOrst-Case Are looking to change that: https://www.theregister.com/2021/10/11/cyan_zero_day_legislative_project/ Introducing Security at the Edge: https://aws.amazon.com/blogs/security/introducing-the-security-at-the-edge-core-principles-whitepaper/ Password reuse: https://www.hypr.com/password-reuse/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter. Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud observability; it’s more than just hipster monitoring.Corey: I must confess, I didn’t expect to see an unpatched AWS vulnerability being fodder for this podcast so early in the security lifespan here, but okay. Yes, yes, before I get letters, it’s not a vulnerability as AWS would define it, but it’s a pretty crappy default that charges customers money while giving them a false sense of security.Past that, it’s going to be a short podcast this week, and that’s just fine by me because the point of it is, “The things you should know as someone who has to care about security.” On slow news weeks like last week that means I’m not here to give you pointless filler. Onward.Now, AWS WAF is expensive and apparently, as configured by default, entirely optional for attackers. Only the first 8KB of a request are inspected by default. That means that any malicious payload that starts after the 8KB limit in a POST request will completely bypass AWS WAF unless you’ve explicitly added a rule to block any POST request greater than 8KB in size, which you almost assuredly have not done. Even their managed rule that addresses size limits only kicks in at 10KB. This is—as the kids say—less than ideal.I had a tweet recently that talked about the horror of us-east-1 being globally unavailable for ages. Tim Bray took this and ran with the horrifying concept in a post he called, “Worst Case.” It’s really worth considering things like this when it comes to disaster and continuity planning. How resilient are our apps and infrastructure really when all is said and done? What dependencies do we take on third parties who in turn rely on the same infrastructure that we’re trying to guard against failure from?An unfortunate reality is that many cybersecurity researchers don’t have much in the way of legal protections; some folks are looking to change that through legislation. Here’s some good advice: if a security researcher reports a vulnerability to you or your company in good faith, perhaps not acting like a raging jackhole is an option that’s on the table. Bug bounties are hilariously small; they could make many times as much money by selling vulnerabilities to the highest bidder. Instead they’re reporting bugs to you in good faith. Word spreads. If you’re a hassle to deal with, other researchers won’t report things to you in the future. “Be a nice person,” is surprisingly undervalued when it comes to keeping yourself and your company out of trouble.Now, only one interesting thing came out of the mouth of AWS horse last week in a security context, and it’s a Core Principles whitepaper: “Introducing Security at the Edge.” Setting aside entirely the fact that neither contributor to this has the job title of “EdgeLord,” I like it. Rather than focusing on specific services—although of course there’s some of that because vendors are going to vendor—it emphasizes how to think about the various considerations of edge locations that aren’t deep within hardened data centers. “How should I think about this problem,” is the kind of question that really deserves to be asked a lot more than it is.and lastly, let’s end up with a tip of the week. If you have a multi-cloud anything, ensure that credentials are not shared between two cloud providers. I’m talking about passwords, keys, et cetera. This is a step beyond the standard password reuse warning of not using the same password for multiple accounts. Think it through; if one of your providers happens to be Azure, and they Azure up the security yet again, you really don’t want that to grant an attacker or other random Azure customers access to your AWS account as well, do you? I thought not.Corey: This episode is sponsored in part by Liquibase. If you’re anything like me, you’ve screwed up the database part of a deployment so severely that you’ve been banned from ever touching anything that remotely sounds like SQL at least three different companies. We’ve mostly got code deployment solved for, but when it comes to databases, we basically rely on desperate hope, with a rollback plan of keeping our resumes up to date. It doesn’t have to be that way. Meet Liquibase. It’s both an open-source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails that ensure you’ll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: And that is what happened last week in AWS security. I have been your host, Corey Quinn, and if you remember nothing else, it’s that when you don’t get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

AWS W(T)AF

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Turbotax-of-AWS-BillingNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

The Turbotax of AWS Billing

AWS Morning Brief for the week of October 18, 2021 with Corey Quinn.

AWS Butt Computing

Links: Disclosed a nasty auto-delete bug: https://arstechnica.com/information-technology/2021/10/researcher-refuses-telegrams-bounty-award-discloses-auto-delete-bug/ Enroll basically all of it’s users: https://blog.google/technology/safety-security/making-sign-safer-and-more-convenient/ Worth taking a look: https://labs.bishopfox.com/tech-blog/IAM-vulnerable-assessing-the-aws-assessment-tools Enumerate those yourself: https://www.hezmatt.org/~mpalmer/blog/2021/10/07/enumerating-aws-iam-accounts.html AWS Access Keys: https://www.nojones.net/posts/aws-access-keys-a-reference/ Routes billions of text messages: https://www.vice.com/en/article/z3xpm8/company-that-routes-billions-of-text-messages-quietly-says-it-was-hacked “Enabling Data Classification for Amazon RDS database with Amazon Macie”: https://aws.amazon.com/blogs/security/enabling-data-classification-for-amazon-rds-database-with-amazon-macie/ “How to set up a two-way integration between AWS Security Hub and Jira Service Management”: https://aws.amazon.com/blogs/security/how-to-set-up-a-two-way-integration-between-aws-security-hub-and-jira-service-management/ “Update the alternate security contact across your AWS accounts for timely security notifications”: https://aws.amazon.com/blogs/security/update-the-alternate-security-contact-across-your-aws-accounts-for-timely-security-notifications/ CloudSploit: https://github.com/aquasecurity/cloudsploit TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example, and the only thing that these things do is alert you whenever someone attempts to use them. It’s an awesome approach to detecting breaches. I’ve used something similar for years myself before I found them. Check them out. But wait, there’s more because they also have an enterprise option that you should be very much aware of: canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It’s awesome. If you don’t do something like this, instead you’re likely to find out that you’ve gotten breached the very hard way. So, check it out. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You’ll know which one of those you fall into. Take a look. I’m a big fan. More to come from Thinkst Canary in the weeks ahead.Corey: To begin with, the big news is that week is the week of the year in which the Last Week in AWS charity shirt is available for sale. All proceeds to benefit 826 National. To get your snarky, sarcastic shirt, “The AWS Status Page,” this year, visit lastweekinaws.com/charityshirt and thank you in advance for your support.Now, last week’s big security news was about Amazon’s subsidiary, Twitch—or Twetch, depending upon pronunciation. It had a bunch of its code repos and streamer payouts leaked. Given that they are in fact an Amazon company largely hosted on AWS, you know, except for the streaming parts; are you a lunatic? That would cost ALL the money—this makes it tricky for AWS to message this as not their problem as per their vaunted Shared Responsibility Model. What’s the takeaway? Too soon to say but, ouch.From the community. Telegram offered a researcher a €1,000 bounty, which is just insultingly small. The researcher said, “Not so much,” and disclosed a nasty auto-delete bug. If you’re going to run a bug bounty program, ensure that you’re paying researchers enough money to incentivize them to come forward and deal with your no-doubt obnoxious disclosure process.You can expect a whole bunch of people who don’t care about security to suddenly be asking fun questions as Google prepares to enroll basically all of its users into two-factor-auth. Good move, but heads up, support folks.I found a detailed analysis of AWS account assessment tools. These use things like CloudSploit, which I’ll talk about in a bit, IAM Vulnerable, et cetera. Fundamentally, they all look at slightly different things; they’re also all largely the same, but it might be worth taking a look.AWS has made statements indicating that they don’t believe that enumerating which IAM accounts exist in a given AWS account is a security risk, so someone has put out a great technique you can use to enumerate those yourself. Why not, since Amazon doesn’t find this to be a problem.A reference to the various kinds of AWS Access Keys is also something I found relatively handy because I hadn’t seen this ever explained before. It taught me a lot about the different kinds of key nonsense that I encounter in the wild from time to time. Take a look, it’s worth the read.It didn’t get a lot of attention in the press due to, you know, things last week, but a company that routes billions of text messages said that it was hacked. It’s worth pointing out that SMS is a garbage second-factor, just because how lax security around it is. I’m a big believer in hardware keys like Yubikeys for important stuff, and an app like Authy or Google Authenticator for less important or shared accounts.I know, you shouldn’t be sharing accounts; as soon as you come up with a better way for multiple people in different locations to do things that require root credentials in an AWS account, do let me know. Back to my point; treat SMS as a second factor only as better than nothing, not a serious security bulwark when it matters.Three things came out from the mouth of AWS horse last week. “Enabling Data Classification for Amazon RDS database with Amazon Macie.” While the idea of streaming from a relational database through a bunch of wildly expensive AWS services is of course ludicrous, the actual value of knowing what the data classification in your database is can’t be understated.The best practice pattern here is to make sure that you’re bounding the truly sensitive stuff to its own location. For instance, instead of storing credit card information in ‘the database’; have a token that references a completely separate database that contains that information that’s severely locked down; that way any random business query doesn’t return sensitive data, and you can restrict access to that data to only the queries or groups or situations that require it. Note that this is only an example and you should not in fact be storing credit card numbers yourself. Good God.Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.Corey: “How to set up a two-way integration between AWS Security Hub and Jira Service Management.” Now, I’m not a big fan of either Jira or Security Hub, but integrating whatever it is that finds alerts into something that reports them to someone empowered to do something about them is kind of important. You’ve got to tune it, though. “Someone visited your website,” showing up 3000 times in an hour is going to be very noisy, and mask alerts of the form, “Your database is open to the world.”They also talk about how to “Update the alternate security contact across your AWS accounts for timely security notifications.” You definitely want to ensure that every AWS account in your cloud estate has the right addresses here configured, and hope that someone who’s compromised your accounts doesn’t use this API to simply change them back again. It’ll stop you from doing that, right? Right? Hello?And finally, MetaSploit is famous as an exploitation toolkit for systems. CloudSploit is attempting to be the same thing, only for cloud accounts. It’s not something you’ll likely use day-to-day, but it is a great way to spend an afternoon tinkering while also learning new things. And that’s what happened Last Week in AWS: Security. Thank you for listening and once again, I ask you, go ahead and visit lastweekinaws.com/charityshirt and get yours today.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it’s that when you don’t get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Edition.

AWS Security is Twitching

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/why-i-turned-down-an-aws-job-offerNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

Why I Turned Down an AWS Job Offer Revisited

AWS Morning Brief for the week of October 11, 2021 with Corey Quinn.

Charity T-Shirt Week

Links: Let’s Encrypt’s root certificate has expired, and it might break your devices: https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/ Slack was bitten by DNSSEC: https://Twitter.com/tqbf/status/1443654964556013569 Prepare For Cybersecurity Assessments From Your Customers: https://www.securitysystemsnews.com/article/prepare-for-cybersecurity-assessments-from-your-customers AWS Lambda now supports triggering Lambda functions from an Amazon SQS queue in a different account: https://aws.amazon.com/about-aws/whats-new/2021/09/aws-lambda-lambda-function-amazon-sqs-queue/ Migrating custom Landing Zone with RAM to AWS Control Tower: https://aws.amazon.com/blogs/mt/migrating-custom-landing-zone-with-ram-to-aws-control-tower/ Introducing the Ransomware Risk Management on AWS Whitepaper: https://aws.amazon.com/blogs/security/introducing-the-ransomware-risk-management-on-aws-whitepaper/ Validate IAM policies in CloudFormation templates using IAM Access Analyzer: https://aws.amazon.com/blogs/security/validate-iam-policies-in-cloudformation-templates-using-iam-access-analyzer/ Pacu: The Open Source AWS Exploitation Framework: https://rhinosecuritylabs.com/aws/pacu-open-source-aws-exploitation-framework/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example, and the only thing that these things do is alert you whenever someone attempts to use them. It’s an awesome approach to detecting breaches. I’ve used something similar for years myself before I found them. Check them out. But wait, there’s more because they also have an enterprise option that you should be very much aware of: canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It’s awesome. If you don’t do something like this, instead you’re likely to find out that you’ve gotten breached the very hard way. So, check it out. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You’ll know which one of those you fall into. Take a look. I’m a big fan. More to come from Thinkst Canary in the weeks ahead.Corey: Somehow we made it through an entire week without a major vendor having a headline-level security breach. You know, I could get used to this; I’ll take, “It’s harder for me to figure out what to talk about here,” over, “A bunch of customers are scrambling because their providers have failed them,” every time.So, let’s see what the community had to say. Last week, as you’re probably aware, Let’s Encrypt’s root certificate expiredwhich caused pain for a bunch of folks. Any device or configuration that hadn’t been updated for a few years is potentially going to see things breaking. The lesson here is to be aware that certificates do expire. The antipattern is to do super-long registrations for thing, but that just makes it worse.One of the things Let’s Encrypt got very right is forcing 90-day certificate rotations for client certs. When you’ve got to do that every three months, you know where all of your certificates are. If you’ve got to replace it once every ten years, you’ll have no clue; that was six employees ago.In bad week news, Slack was bitten by DNSSEC when they attempted and failed to roll it out. DNSSEC is a bag of pain it’s best not to bother with, as a general rule. DNS is always a bag of pain because of caching and TTL issues. In effect, Slack tried to roll out DNSSEC—probably due to a demand by some big corporate customer—had it fail, panicked and rolled back the change, and was in turn bitten by outages as a bunch of DNS resolvers had the DS key cached, but the authoritative nameservers stopped publishing it. This is a mess and a great warning to those of us who might naively assume that anything like DNSSEC that offers improved security comes without severe tradeoffs. Measure twice, cut once because mistakes are going to show.I also found a somewhat alarmist article talking about cybersecurity assessments from your customers and fine, but it brings up a good point. If you’re somehow responsible for security but don’t have security in your job title—which, you know, this show is aimed at—you may one day be surprised to have someone from sales pop up and ask you to fill out a form from a prospective customer. Ignore the alarm and the panic but you’re going to want to get towards something approaching standardization around how you handle those.The first time you get one of these, it’s a novel exercise; by the tenth, you just want to have a prepared statement you can hand them so you can move on with things. Well, those prepared statements are often called things like, “SOC 2 certifications.” There’s a spectrum and where you fall on it depends upon who you work for and what you do. So, take them seriously and don’t be surprised when you get one.AWS had a few interesting security-related announcements. AWS Lambda now supports triggering Lambda functions from an Amazon SQS queue in a different account. That doesn’t sound like a security announcement, so why am I talking about it? Because until recently, it wasn’t possible so a lot of folks scoped their IAM policies very broadly; what do you care if any random SQS queue in your own account can invoke a Lambda? With this change, suddenly internet randos can invoke Lambda functions, and you should probably go check production immediately.Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.Corey: Migrating custom Landing Zone with RAM to AWS Control Tower. It’s worth considering the concept here because, “Using the polished thing” is usually better than building and then maintaining something yourself. You wind up off in the wilderness; then AWS shows up and acts befuddled, “Why on earth would you build things the way that we told you to build them at the time you set up your environment?” It’s obnoxious and they need to stop talking and own their mistakes, but keeping things current with the accepted way of doing things is usually worth at least considering.AWS has a whitepaper on Ransomware Risk Management out and I’m honestly conflicted about it. There are gems but it talks about a pile of different services they offer to offset the risk. Some of them—like AWS Backup—are great.Others—“Use Systems Manager State Manager”—present as product pitches for products of varying quality and low adoption. On balance, it’s worth reading but retain a healthy skepticism if you do. It should be noted that the points that the address and the framework they lay out is exactly how risk management folks think, and that’s helpful.Validate IAM policies in CloudFormation templates using IAM Access Analyzer. I like that one quite a bit. It does what it says on the tin, and applies a bunch of more advanced linting rules than you’d find in something like cfn-lint.Note that this costs nothing for a change, even though it does communicate with AWS to run its analysis. Note that as AWS improves the Access Analyzer, findings will likely change, so be aware that this may well result in a regression should you have it installed as part of a CI/CD pipeline.And as far as tools go, if you’re not a security researcher, good; you’re in the right place. But that said, if you have a spare afternoon at some point, you may want to check out Pacu—that’s P-A-C-U. It’s an open-source AWS exploitation framework that lets you see just how insecure your AWS accounts might be. I generally leave playing with those sorts of things to security professionals, but this is a fun way to just take a quick check and see if there’s a burning fire that jumps out that might arise for you down the road. And I’ll talk to you more about all this stuff next week.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it’s that when you don’t get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Edition.Announcer: This has been a HumblePod production. Stay humble.

DNSSEC Inspired Outages

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/The-Compelling-Economics-of-Cloudflare-R2Never miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

The Compelling Economics of Cloudflare R2

AWS Morning Brief for the week of September 3, 2021 with Corey Quinn.

Cloudflare's Object Storage Lesson

Links: “I Trust AWS IAM to Secure my Applications. I Don’t Trust the IAM Docs to Tell Me How”: https://ben11kehoe.medium.com/i-trust-aws-iam-to-secure-my-applications-i-dont-trust-the-iam-docs-to-tell-me-how-f0ec4c119e79 “Introduction to Zero Trust on AWS ECS Fargate”: https://omerxx.com/identity-aware-proxy-ecs/ Threat Stack Aquired by F5: https://techcrunch.com/2021/09/20/f5-acquires-cloud-security-startup-threat-stack-for-68-million/ AWS removed from CVE-2021-38112: https://rhinosecuritylabs.com/aws/cve-2021-38112-aws-workspaces-rce/ Ransomware that encrypts the contents of S3 buckets: https://rhinosecuritylabs.com/aws/s3-ransomware-part-1-attack-vector/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example, and the only thing that these things do is alert you whenever someone attempts to use them. It’s an awesome approach to detecting breaches. I’ve used something similar for years myself before I found them. Check them out. But wait, there’s more because they also have an enterprise option that you should be very much aware of: canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It’s awesome. If you don’t do something like this, instead you’re likely to find out that you’ve gotten breached the very hard way. So, check it out. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You’ll know which one of those you fall into. Take a look. I’m a big fan. More to come from Thinkst Canary weeks ahead.Corey: This podcast seems to be going well. The Meanwhile in Security podcast has been fully rolled over and people are chiming in with kind things, which kind of makes me wonder, is this really a security podcast? Because normally people in that industry are mean.Let’s dive into it. What happened last week in security? touching AWS, Ben Kehoe is on a security roll lately. His title of the article in full reads,  “I Trust AWS IAM to Secure My Applications. I Don’t Trust the IAM Docs to Tell Me How”, and I think he’s put his finger on the pulse of something that’s really bothered me for a long time. IAM feels arcane and confusing. The official doc just made that worse For me. My default is assuming that the problem is entirely with me, But that’s not true at all. I suspect I’m very far from the only person out there who feels this way.An “Introduction to Zero Trust on AWS ECS Fargate” is well-timed. Originally when Fargate launched, the concern was zero trust of AWS ECS Fargate, But we’re fortunately past that now. The article is lengthy and isn’t super clear as to the outcome that it’s driving for and also forgets that SSO was for humans and not computers, But it’s well documented and it offers plenty of code to implement such a thing yourself. It’s time to move beyond static IAM roles for everything.Threat Stack has been a staple of the Boston IT scene for years; they were apparently acquired by F5 for less money than they’d raised, which seems unfortunate. I’m eagerly awaiting to see how they find F5 for culture. I bet it’s refreshing.and jealous of Azure as attention in the past few episodes of this podcast, VMware wishes to participate by including a critical severity flaw that enables ransomware in vCenter or vSphere. I can’t find anything that indicates whether or not VMware on AWS is affected, So those of you running that thing you should probably validate that everything’s patched. reach out to your account manager, which if you’re running something like that, you should be in close contact with anyway.Corey: Now from AWS themselves, what do they have to say? not much last week on the security front, their blog was suspiciously silent. scuttlebutt on Twitter has it that they’re attempting to get themselves removed from an exploit, a CVE-2021-38112, which is a remote code execution vulnerability. If you have the Amazon workspaces client installed, update it because a malicious URL could cause code to be executed in the client’s machine. It’s been patched, but I think AWS likes not having public pointers to pass security lapses lurking around. I don’t blame them, I mean, who wants that? The reason I bring it up is Not to shame them for it, but to highlight that all systems have faults in them. AWS is not immune to security problems, nor is any provider. It’s important, to my mind, to laud companies for rapid remediation and disclosure and to try not to shame them for having bugs in the first place. I don’t always succeed at it, But I do try. But heaven help you if you try to blame an intern for a security failure.And instead of talking about a tool, Let’s do a tip of the week. Ransomware is in the news a lot, But so far, all that I’ve seen with regard to ransomware that encrypts the contents of S3 buckets is theoretical proofs—or proves—of concept. That said, for the data you can’t afford to lose, you’ve got a few options that stack together neatly. The approach distills down to some combination of enabling MFA delete, enabling versioning on the bucket, and setting up replication rules to environments that are controlled by different credential sets entirely. This will of course become both maintenance-intensive and extremely expensive for some workloads, But it’s always a good idea to periodically review your use of S3 and back up the truly important things.Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it’s that when you don’t get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Editionwith the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

F5's Refreshing Culture

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Actual-Next-1-Million-Cloud-CustomersNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

The Actual Next 1 Million Cloud Customers

AWS Morning Brief for the week of September 27,2021 with Corey Quinn.

Old Zealand's Data Center Migration

Links: WTF? Microsoft makes fixing deadly OMIGOD flaws on Azure your job: https://www.theregister.com/2021/09/17/microsoft_manual_omigod_fixes/ Travis CI flaw exposed secrets of thousands of open source projects: https://arstechnica.com/information-technology/2021/09/travis-ci-flaw-exposed-secrets-for-thousands-of-open-source-projects/ How to Build Strong Security Guardrails in the AWS Cloud With Minimal Effort: https://markn.ca/2021/how-to-build-strong-security-guardrails-in-the-aws-cloud-with-minimal-effort/ Introduction to OWASP Top 10 2021: https://owasp.org/Top10/ AWS SIGv4 and SIGv4A: https://shufflesharding.com/posts/aws-sigv4-and-sigv4a Inside Figma: getting out of the (secure) shell: https://www.figma.com/blog/inside-figma-getting-out-of-the-secure-shell/ AWS Firewall Manager now supports AWS WAF rate-based rules: https://aws.amazon.com/about-aws/whats-new/2021/09/aws-firewall-manager-waf-rate-based-rules/ How to automate incident response to security events with AWS Systems Manager Incident Manager: https://aws.amazon.com/blogs/security/how-to-automate-incident-response-to-security-events-with-aws-systems-manager-incident-manager/ New Standard Contractual Clauses now part of the AWS GDPR Data Processing Addendum for customers: https://aws.amazon.com/blogs/security/new-standard-contractual-clauses-now-part-of-the-aws-gdpr-data-processing-addendum-for-customers/ Protect your remote workforce by using a managed DNS firewall and network firewall: https://aws.amazon.com/blogs/security/protect-your-remote-workforce-by-using-a-managed-dns-firewall-and-network-firewall/ AWS Security Hub Automated Response and Remediation: https://github.com/awslabs/aws-security-hub-automated-response-and-remediation Checkov: https://github.com/bridgecrewio/checkov TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me. I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example, and the only thing that these things do is alert you whenever someone attempts to use them. It’s an awesome approach to detecting breaches. I’ve used something similar for years myself before I found them. Check them out. But wait, there’s more because they also have an enterprise option that you should be very much aware of: canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files that it presents on a fake file store, you get instant alerts. It’s awesome. If you don’t do something like this, instead you’re likely to find out that you’ve gotten breached the very hard way. So, check it out. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I am so glad I found them. I love it.” Again, those URLs are canarytokens.org and canary.tools. And the first one is free because of course it is. The second one is enterprise-y. You’ll know which one of those you fall into. Take a look. I’m a big fan. More to come from Thinkst Canary weeks ahead.Corey: Oh, for th—this is the third episode of the Last Week in AWS slash AMB: Security Edition, and instead of buying a sponsorship like a reasonable company, Microsoft Azure is once again forcing me to talk about their cloud instead, via completely blowing it when it comes to security. Again. Not only did they silently install an agent onto virtual machines in Azure that add a handful of trivially exploitable vulnerabilities, it’s also apparently your job to fix it for them. I have to confess, I take Azure a lot less seriously than I did a month ago.Now, let’s dive in here. Speaking of terrible things, it’s honestly difficult for me to imagine a company screwing the pooch harder than TravisCI did this month. They had a bug that started leaking private credentials into public build logs; this is bad. They fixed it; this is good. And then only begrudgingly disclosed it in a buried release with remarkably little public messaging; this is unfathomable. At this point, if you’re using TravisCI, get the hell off of it. Mistakes happen to every vendor. The ones that try to hide their mistakes are absolutely not companies you can trust.If you put up a slide deck and accompanying notes entitled How to Build Strong Security Guardrails in the AWS Cloud With Minimal Effort, I’m probably going to take a look at it because strong guardrails are important and minimal effort is critical if you expect it to actually get done. If you’re also my longtime friend Mark Nunnikhoven, then I’m going to default to treating it as gospel because Mark frankly does not miss when it comes to AWS concepts explained in an easily approachable way. Security has got to be aligned with the way engineers work within your environment. Remember, it’s not that hard to spin up a new AWS account on someone’s corporate credit card; you absolutely do not want to incentivize that behavior.Corey: I periodically say the OWASP Top 10, which is a list of the most critical security risks for applications on the web, has not meaningfully changed in ten years. Well, apparently it just did. It’s worth reviewing the changes; broken configurations top the list. The Open Web Application Security Project—OWASP—is a foundation that’s remained surprisingly free of capture by security vendors. It’s a good starting point to frame your risk exposure and what to think about.AWS VP and Distinguished Engineer Colm MacCárthaigh has an article on AWS’s new signing protocol, along with the differences between AWS SIGv4 and SIGv4A. As a quick primer, all requests to AWS are signed for authentication reasons. The new SIGv4A isn’t region-locked—and the recent release of the S3 Multi-Region Access Points is why it makes it a bit of a problem—there’s no key exchange, and it’s more computationally expensive. You don’t really need to know the details as a practitioner, but you should be aware that AWS very much does put stupendous thought into this, and they sweat the details something fierce. This is why we trust cloud providers like AWS, and Google Cloud, and absolutely not Azure.Figma has a great post up, talking about how they stopped using SSH via bastion host and started using Systems Manager Session Manager instead. Bad name, wonderful service. More to the point, what I like about this post isn’t just the, “Here’s how the technology works,” parts, but also dives into the nuts and bolts of how they handled the migration without stopping work for folks. Communicating changes like this is tricky; don’t lose sight of that.Now, from the mouth of AWS horse itself, let’s dive in. AWS Firewall Manager now supports AWS WAF rate-based rules. This is pretty awesome if for no other reason than it’s aware both of multiple regions as well as multiple accounts.An awful lot of security services that are both first and third-party alike tend to go for addressing only one of those at best. Anything that lets you manage things centrally in a holistic way when it comes to security is generally going to be a win, but you also don’t want a giant single point of failure. It’s a bit of a balancing act, but that’s why our field needs us. It’s why they pay us.How to automate incident response to security events with AWS Systems Manager Incident Manager. And I’m genuinely torn on this. I like automation, but it strikes me as a way to end up automating the responses to fairly common things rather than addressing the actual cause so you get fewer false alarms. You really don’t want the security pager going off frequently, if for no other reason than you’ll be training the people carrying it to ignore it.Announcer: Have you implemented industry best practices for securely accessing SSH servers, databases, or Kubernetes? It takes time and expertise to set up. Teleport makes it easy. It is an identity-aware access proxy that brings automatically expiring credentials for everything you need, including role-based access controls, access requests, and the audit log. It helps prevent data exfiltration and helps implement PCI and FedRAMP compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.Corey: AWS is harping about its New Standard Contractual Clauses now part of the AWS GDPR Data Processing Addendum for customers, blah, blah, blah—look, if you have compliance obligations, here’s what you do. Check the documents in AWS Artifact, reach out to your account manager for additional resources, and whatever you do, do not attempt to YOLO it yourself from first principles. AWS has piles and piles of documents ready and waiting to satisfy regulators and auditors alike. I tried to do it myself once, and a financial institution attempted to set up a tour of us-east-one. Trust me when I say you don’t want to go down that path.Protect your remote workforce by using a managed DNS firewall and network firewall. Look, the post can safely be discarded; it’s chock full of complexity lurking deep in the weeds, but I bring it up instead so that you think for a moment about the threat model of a remote workforce, read as most of them these days. Does having a DNS firewall protect against threats that they’re likely to encounter? Does a network firewall make sense in a zero-trust world? Consider those things in the context of your environment rather than in the context of a company that has things it needs to sell you. Good decisions are rarely sourced from vendors.A couple of tools as well. Automating response and remediation is one of those delicate balances. The unimaginatively named AWS Security Hub Automated Response and Remediation GitHub repo has ways to handle this but it’s going to be super easy to automate away things that really shouldn’t be automated. You are definitely going to want to think through edge and corner cases.And lastly, I tripped over checkov last week. It analyzes your Terraform slash CloudFormation slash whatever configurations for various misconfigurations. It caught a couple of things that I’ve been ignoring for a while, and while it missed another couple of problems in my environment, it’s definitely going to be something I integrate into my deployment pipelines in the future, once I have deployment pipelines. That’s checkov—C-H-E-C-K-O-V—open-source projects. Take a look. I’m a fan.And that’s what happened to the world of AWS security last week. Enjoy not having to care about the rest of it.Corey: I have been your host, Corey Quinn, and if you remember nothing else, it’s that when you don’t get what you want, you get experience instead. Let my experience guide you with the things you need to know in the AWS security world, so you can get back to doing your actual job. Thank you for listening to the AWS Morning Brief: Security Editionwith the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcasts, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

OMIGOD, Get it Together Already

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/17-more-ways-to-tun-containers-on-aws Never miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

17 More Ways to Run Containers on AWS

The latest in AWS news, sprinkled with snark. Posts about AWS come out over sixty times a day. We filter through it all to find the hidden gems, the community contributions--the stuff worth hearing about! Then we summarize it with snark and share it with you--minus the nonsense