From a30a943a6648eb5a6535ccada7d4c4ea60b96c11 Mon Sep 17 00:00:00 2001 From: James Brechtel Date: Thu, 5 Jan 2023 21:57:29 -0500 Subject: [PATCH] Adding Allure post --- content/posts/the-allure-of-cloud-services.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 content/posts/the-allure-of-cloud-services.md diff --git a/content/posts/the-allure-of-cloud-services.md b/content/posts/the-allure-of-cloud-services.md new file mode 100644 index 0000000..063095d --- /dev/null +++ b/content/posts/the-allure-of-cloud-services.md @@ -0,0 +1,109 @@ ++++ +title = "The allure of cloud services - AWS Transfer Server" +date = "2023-01-05" ++++ + +## Build vs buy + +I like cloud services as much as the next person. Definitely not _too_ much but +definitely, umm, much? + +Anyway, I think cloud specific services can _sometimes_ be worth the sort of +vendor lock-in that they facilitate. + +Vendor lock in sucks, but so does re-inventing the wheel. As mentioned in an +[earlier post](/posts/cloudwatch-metric-filters/) it can be pretty nice to +create alerts and application metrics passively from your log files. + +Knowing when to use vendor-specific services and APIs versus rolling your own +is, in some ways, more art than science. Reasonable teams won't change their +infrastructure provider very many times and the difference between the two +choices is often less about lock-in and more about whether or not the problem +you're solving is really the special unicorn that you want it to be or not. + +Sometimes, the cloud vendor solution is just bad, though. + +## Case in point - AWS Transfer Server + +On my team, we need to accept SFTP file transfers from about 20-30 vendors. + +Sadly we can't avoid SFTP for this. It's an industry standard for the kind of +data we're receiving and we aren't in a position to dictate otherwise. + +AWS offers a suite of products which include a hosted SFTP solution, [AWS Transfer](https://aws.amazon.com/aws-transfer-family/). + +SFTP is simple enough and I'd ultimately like the uploaded files in an S3 +bucket so seems like a great fit, right? + +Wrong. + +You have to jump through [a hundred](https://aws.amazon.com/secrets-manager/) +[different](https://aws.amazon.com/lambda/) +[hoops](https://aws.amazon.com/iam/) to event attempt to have password-based +authentication working on this service. + +The services involved to make password based authentication work here aren't in +and of themselves a problem. The problem is the multiple points of failure they +represent. Each service can fail on its own, the permissions between the +services can fail, or the code in the Lambda itself can fail. + +Debugging it was a nightmare. It ultimately worked but I wasn't sure _why_. I +had a strange permissions error about the IAM Role being used to either launch +the Lambda or the role used to access the S3 bucket for uploads. I don't +remember at this point. But it was really clear that if it ever broke again +then myself or whatever poor soul had to work on it was going to going to +regret the choice of technologies to make this work. + +You know what is easy and simple and well understood? OpenSSH and Linux user +accounts. + +In the end, the AWS Transfer option proved to be just complicated and enciting +enough for me to waste about a day and a half on it before I realize, in a +drunken stupor, that I'd be better off creating a snow-flaked EC2 instance and +calling it a day. + +## Resisting temptation + +I do think I could have identified that this was a bad idea before I wasted +over a day. One doesn't have to be doomed to repeat this mistake to learn the +lesson here. + +The way to avoid this is to interrogate the path before you start. + +Empathy is a good tool, here. + +Picture the thing working as intended. Then imagine someone else (or future you +a year from now) having to search for how to change something or track down an +error. + +- Are they likely to find many other people using these services? (In this case, no) +- Are the services involved purpose build for the task at hand? (In this case, no none of them) +- Does your team have pre-existing expertise around the services involved? (Again, no) +- What benefits does this approach yield over a traditional solution? (Upload to S3 is nice) +- Are those benefits important to your use case? (Not for us) + +I didn't ask myself of those questions. I _really_ should have. Because the +answers are easy to get and they clearly indicate AWS Transfer Server is not +the best answer. + +# Timeboxes +Another tool that could have helped here is a timebox. + +A timebox is when you decided to give yourself a fixed, and often pretty short, +amount of time to accomplish something or to at least get meaningful insight +into a potential solution. + +Sometimes I will demand that a working solution can be done inside a given +timebox or then the approach is abandoned. But other times I might just say +that I'm going to spend X amount of time on an approach and then make a point +to re-evaluate how long the complete solution will take. + +It's important to remember how valuable timeboxes can be. The more time we +waste on things like this then the more attached we become to it. [Sunk +cost](https://en.wikipedia.org/wiki/Escalation_of_commitment) can get even the +best of us. + +For me the simplest tool for avoiding sunk cost fallacy is to timebox risky +endeavors like this one. The time inside a timebox starts out as a write off. I +never feel bad about discarding failed results inside a timebox. I should use +them more aggressively.