Highlights

[!note] a multi-threading deadlock in a database vendor’s ODBC driver. She had to manually disassemble the vendor library and then generate a binary patch. By hand. (Location 421)

[!note] She doesn’t want to see brochures for the car they’ve promised to build—she wants to see the engineering plans and the actual parts that they’re going to assemble the car from. (Location 469)

[!note] Find someone who’s actually doing integration testing so I can shoulder-surf while they work. (Location 479)

[!note] need a small army. (Location 515) We have a test army for SA, exactly

[!note] a bunch of contractor developers brought in two months ago. They can’t do builds, either. Shocking. I took them out to lunch. What a disappointment. They know even less than I do. (Location 525) 太上头了

[!note] Kurt Reznick, QA Manager (Location 49)

[!note] Dwayne Cox, Lead Infrastructure Engineer (Location 51)

[!note] Brent Geller, Ops Lead (Location 52)

[!note] William Mason, Director of QA (Location 55)

[!note] Wes Davis, Director of Distributed Technology Operations (Location 56)

[!note] Developers cannot be productive without a great build, integration, and test process. (Location 667)

[!note] Maybe it’s because when friends do favors for friends, we don’t require them to open a ticket first. (Location 928)

[!note] Maxine knows that agility is never free. Over time, without this type of investment, software often becomes more and more difficult to change. (Location 1033)

[!note] But in almost every other domain, especially when you have customers, change is a fact of life. (Location 1036)

[!note] able to build things with focus, flow, and joy. (Location 1052)

[!note] People were able to do what they wanted without being dependent on scores of other people. This is what great architecture enables. (Location 1053)

[!note] Kurt, (Location 1087)

[!note] Bill (Location 1087)

[!note] “How many transactions per second are we expecting for product displays and orders? And how many transactions per second are the current builds capable of handling right now? That will tell us how many servers we need for the horizontally scalable portions, as well as how far we’re off for the vertically scaled components, like the database.” (Location 1113) This is something that are from the architect’s vocabulary

[!note] Since when have you ever cared what people in authority think about you? (Location 1146) This should be the attitude of an architect!!! Never care about the people in authority think about you!

[!note] Just scanning the section headings makes her heart leap—they’re the painstakingly assembled Phoenix build instructions, complete with links to documents, license keys, step-by-step tutorials, and even links to a bunch of videos. One is titled “Getting your uberjar to run in our (very) crazy, screwed up production web cluster (8 min),” and another is “How to monitor your apps despite our Ops groups (12 min).” (Location 1237)

[!note] Best of all, there’s a link to a four-node virtual machine cluster with administrative access! That means Maxine will be able to do whatever she wants without having to fill out another service desk ticket! (Location 1241)

[!note] But she can definitely see the different personalities and tech stacks of the different teams working on Phoenix. She had no idea there were so many. (Location 1283)

[!note] Maxine isn’t usually a fan of rigid standardization, but she’s not a fan of everyone getting to choose whatever they fancy in the moment. (Location 1286)

[!note] Each decision is a commitment to support it for years or even decades—these are decisions that go far beyond just one team. (Location 1287)

[!note] Kurt? He’s not the sharpest knife in the drawer, which is why he’s stuck in QA. He’s also really nosy. Why do you ask? (Location 1305)

[!note] “I want to build and maintain something for the long haul, instead of shipping the ‘feature of the day’ and dragging all this technical debt around.” (Location 1427)

[!note] You clearly don’t understand how this game works. We’re QA. We protect the organization from developers. (Location 1453) OMG, is this really a QA directors’ thought? I am scared of this out-dated engineering culture.

[!note] You give developers an inch, and they’ll take a mile.’” (Location 1455) QA, SRE and Devs need to have a common OKR - improve the effectiveness of the CI/CD pipeline

[!note] “He only cares about growing his union membership dues, not about what’s right for the business. (Location 1461) That is bad..

[!note] it. A BOM is an invisible first character that some programs put in a CSV file to indicate whether it’s big-endian or little-endian. She’s been bitten by this before. (Location 1590) Me as well as someone worked with csv for a very long time!!

[!note] “You have no idea,” Brent says, rolling his eyes. “Different problem every time, depending on who generated the file. The most common problems lately are zero-length files, or files with no rows in them. And it’s not just the pricing team—we have data problems like this all over the place.” (Location 1596) We need a sophisticated csv parser program here!

[!note] The Phoenix rollout was more than just a spectacular outage … it was the most amazing example of production data loss Maxine has ever seen. (Location 1614)

[!note] Maybe we explore outsourcing IT? (Location 1664) Speechlesssssssss I hate contractors !!!!!!!

[!note] “You should be able to create value by changing one file, one module, one service, one component, one API call, one container, one app, or whatever! (Location 1918) Create value by changing one thing!

[!note] “Build responsibility moved from Dev to QA to interns. Tech giants like Facebook, Amazon, Netflix, Google, and Microsoft give Dev productivity responsibilities to only the most senior and experienced engineers. But here at Parts Unlimited, it’s the exact opposite.” (Location 1928)

[!note] “Sometimes it’s when simple components become complected, and you can no longer reason about it or change it without immense effort or risk of catastrophe. Sometimes it’s when decision-making processes or the organizational structure loses locality, forcing even small decisions to be escalated—your infamous ‘Square.’ (Location 1937) I like the word - locality!

[!note] “I’ve started calling all of these things ‘complexity debt,’ because they’re not just technical issues—they’re business issues. And it’s always a choice,” (Location 1940) Tech debt and product debt, and biz debt!!! Are the team organized in an effective way? Are the processes optimized?

[!note] “You can choose to build new features or you can choose to pay down complexity debt. When a fool spends all their time on features, the inevitable outcome is that even easy tasks become difficult and take longer to execute. And no matter how hard you try or how many people you have, it eventually collapses under its own weight , forcing you to start over from scratch.” (Location 1941) Restart from scratch…

[!note] the First Ideal of Locality and Simplicity. (Location 1973)

[!note] “The Second Ideal is Focus, Flow, and Joy. (Location 1977)

[!note] The Third Ideal is Improvement of Daily Work. (Location 1986)

[!note] The Fourth Ideal is Psychological Safety, where we make it safe to talk about problems, because solving problems requires prevention, which requires honesty, and honesty requires the absence of fear. (Location 1987)

[!note] Fifth Ideal is Customer Focus, (Location 1989)

[!note] The First Ideal—Locality and Simplicity The Second Ideal—Focus, Flow, and Joy The Third Ideal—Improvement of Daily Work The Fourth Ideal—Psychological Safety The Fifth Ideal—Customer Focus (Location 1992)

[!note] QA is often viewed as an underclass, but at least they’re above Ops. All of which is crap (Location 2011) Change your perception of QA role

[!note] “This is our moment to shine. Here’s our chance to show people what engineering greatness looks like,” (Location 2056)

[!note] byzantine processes (Location 2069)

[!note] “Nope, not any more. We used to be able to do that too. But now someone else deploys it for us. ‘Stay in your lane,’ they told us.” (Location 2174) Let Dev test, let Dev deploy!

[!note] That’s just not good enough for her. In her MRP team, any developer could test their own code and even push code into production themselves. (Location 2192)

[!note] In her MRP team, any developer could test their own code and even push code into production themselves. They didn’t have to wait weeks for other people to do that work for them. (Location 2192) We need to enable the Devs of SA to do that!

[!note] “So, someone has to respond to the ticket and copy the logs off the server for us?” (Location 2235)

[!note] Four hours later, after reviewing the production logs, they confirm that the problem isn’t Data Hub. Two hours after that, everyone finally agrees. As Tom had suspected, it was an internal networking (Location 2246) Remind me that i spent an hour to dig into various applications metrics and logs to indentify a low level service mesh infra issue(networking connection issue), in order to conduct a root cause analysis of a production issue. I am totally fine with my current team and company!

[!note] Chad Stone in network engineering is no longer with the company. (Location 2255) Holy shit…fire anyone who made mistakes…

[!note] But over the years, we’ve hard-coded some things that we shouldn’t have, updated some things here, updated other things there, never quite documenting all of it … and now? It’s a mess.” (Location 2274) Hardcoding No documentation ….

[!note] “We need developers to be able to focus their best energies on building features, not trying to get builds to work. I have a ton of passion for this too and would love your help. (Location 2292)

Buy Me A Coffee